Meta tags:
description=Log;

Headings (most frequently used words):

log, writes, reads, deletes, guarantees, tag, cloud, tags,

Text of the page (most frequently used words):
the (105), message (28), and (24), log (24), file (18), kafka (16), offset (15), this (14), that (13), which (13), are (11), messages (11), bytes (11), for (10), data (10), with (10), each (10), size (9), format (8), from (8), apache (7), configuration (7), use (7), implementation (6), consumer (6), security (5), can (5), disk (5), segment (5), length (5), deletes (5), byte (5), design (5), license (4), last (4), documentation (4), all (4), two (4), block (4), written (4), case (4), provides (4), number (4), than (4), reads (4), value (4), buffer (4), read (4), producer (4), structure (4), partition (4), streams (4), configs (4), corruption (3), must (3), crash (3), write (3), but (3), before (3), crc (3), though (3), maximum (3), flush (3), over (3), entry (3), valid (3), its (3), event (3), time (3), files (3), binary (3), search (3), messagesetsend (3), getting (3), start (3), when (3), client (3), given (3), first (3), will (3), gives (3), guid (3), broker (3), api (3), topic (3), integer (3), docs (3), version (3), started (3), connect (3), page (3), operations (3), privacy (2), events (2), donate (2), software (2), foundation (2), either (2), trademarks (2), distribution (2), previous (2), info (2), source (2), versioned (2), unwritten (2), lost (2), nonsense (2), makes (2), guarantee (2), between (2), inode (2), actual (2), losing (2), new (2), containing (2), parameter (2), forcing (2), process (2), payload (2), stored (2), detected (2), guarantees (2), allows (2), policy (2), any (2), more (2), days (2), could (2), also (2), useful (2), while (2), still (2), list (2), allow (2), view (2), result (2), total (2), error (2), code (2), sent (2), most (2), clients (2), consume (2), specified (2), reading (2), requires (2), global (2), done (2), simple (2), maintained (2), giving (2), bit (2), max (2), larger (2), multiple (2), server (2), ever (2), get (2), likely (2), seconds (2), after (2), writes (2), maintain (2), mapping (2), since (2), random (2), lookup (2), per (2), counter (2), uniquely (2), named (2), 2106 (2), developer (2), guide (2), apis (2), create (2), ecosystem (2), cases (2), introduction (2), thanks, contents, website, 2026, under, terms, logo, registered, united, states, other, countries, next, modified, december, 2025, add, every, handle, html, links, paths, 764, 379dba2230, note, kinds, handled, truncation, due, added, reason, general, order, addition, gain, updated, occurs, not, detects, corner, prevents, corrupting, course, controls, startup, recovery, run, iterates, newest, verifies, sum, less, crc32, matches, truncated, deleted, one, manager, pluggable, delete, policies, choose, eligible, deletion, current, modification, ago, retained, avoid, locking, allowing, modify, copy, style, consistent, views, proceed, immutable, static, snapshot, segments, progressing, fetch, multimessagesetsend, multifetch, following, results, capability, recently, subscribing, right, now, fails, within, sla, attempts, non, existant, outofrangeexception, reset, itself, fail, appropriate, locating, calculating, specific, then, variation, against, memory, range, logical, chunk, return, iterator, contained, intended, single, abnormally, large, retried, times, doubling, until, successfully, make, reject, some, give, bound, need, complete, ends, partial, easily, delimiting, serial, appends, always, rolled, fresh, reaches, configurable, say, 1gb, takes, forced, durability, system, unusual, our, original, idea, was, generated, uniqueness, furthermore, complexity, maintaining, heavy, weight, index, synchronized, essentially, requiring, full, persistent, access, thus, simplify, decided, atomic, coupled, node, identify, simpler, seeks, request, however, once, settled, jump, directly, using, seemed, natural, both, monotonically, increasing, integers, unique, hidden, decision, ultimately, detail, went, efficient, approach, magic, exact, standard, interface, sets, transfered, without, recopying, conversion, desirable, follows, my_topic, partitions, consists, directories, namely, populated, sequence, entries, storing, followed, identified, position, stream, below, contains, created, 00000000000, additional, have, name, roughly, where, my_topic_1, my_topic_0, tags, you, viewing, older, date, see, latest, ops, tag, cloud, print, entire, section, issue, child, edit, zookeeper, monitoring, hardware, java, datacenters, basic, network, layer, upgrading, quick, download, releases, sponsors, org, contact, trademark, project, summit, blog, community, powered, key, concepts, podcasts, videos, books, papers, quickstart,

Text of the page (random words):
log apache kafka get started introduction quickstart use cases books and papers videos podcasts docs key concepts apis configuration design implementation operations security clients kafka connect kafka streams powered by community blog kafka summit project info trademark ecosystem events contact us apache apache org license donate sponsors security privacy releases 4 3 4 2 4 1 4 0 3 9 3 8 3 7 3 6 3 5 3 4 3 3 3 2 3 1 3 0 2 8 2 7 2 6 2 5 2 4 2 3 2 2 2 1 2 0 1 1 1 0 0 11 0 0 10 2 0 10 1 0 10 0 0 9 0 0 8 2 0 8 1 0 8 0 0 7 download kafka ak 0 8 2 x getting started introduction use cases quick start ecosystem upgrading api configuration broker configs consumer configs producer configs new producer configs design design implementation api design network layer messages message format log distribution operations basic kafka operations datacenters kafka configuration java version hardware and os monitoring zookeeper security kafka connect kafka streams streams developer guide view page source edit this page create child page create documentation issue print entire section writes reads deletes guarantees tag cloud apis 1 configuration 34 connect 34 design 34 developer guide 34 docs 2106 getting started 34 implementation 34 kafka 2106 ops 34 security 34 streams 68 you are viewing documentation for an older version 0 8 2 of kafka for up to date documentation see the latest version ak 0 8 2 x implementation log log log tags kafka docs a log for a topic named my_topic with two partitions consists of two directories namely my_topic_0 and my_topic_1 populated with data files containing the messages for that topic the format of the log files is a sequence of log entries each log entry is a 4 byte integer n storing the message length which is followed by the n message bytes each message is uniquely identified by a 64 bit integer offset giving the byte position of the start of this message in the stream of all messages ever sent to that topic on that partition the on disk format of each message is given below each log file is named with the offset of the first message it contains so the first file created will be 00000000000 kafka and each additional file will have an integer name roughly s bytes from the previous file where s is the max log file size given in the configuration the exact binary format for messages is versioned and maintained as a standard interface so message sets can be transfered between producer broker and client without recopying or conversion when desirable this format is as follows on disk format of a message message length 4 bytes value 1 4 n magic value 1 byte crc 4 bytes payload n bytes the use of the message offset as the message id is unusual our original idea was to use a guid generated by the producer and maintain a mapping from guid to offset on each broker but since a consumer must maintain an id for each server the global uniqueness of the guid provides no value furthermore the complexity of maintaining the mapping from a random id to an offset requires a heavy weight index structure which must be synchronized with disk essentially requiring a full persistent random access data structure thus to simplify the lookup structure we decided to use a simple per partition atomic counter which could be coupled with the partition id and node id to uniquely identify a message this makes the lookup structure simpler though multiple seeks per consumer request are still likely however once we settled on a counter the jump to directly using the offset seemed natural both after all are monotonically increasing integers unique to a partition since the offset is hidden from the consumer api this decision is ultimately an implementation detail and we went with the more efficient approach writes the log allows serial appends which always go to the last file this file is rolled over to a fresh file when it reaches a configurable size say 1gb the log takes two configuration parameter m which gives the number of messages to write before forcing the os to flush the file to disk and s which gives a number of seconds after which a flush is forced this gives a durability guarantee of losing at most m messages or s seconds of data in the event of a system crash reads reads are done by giving the 64 bit logical offset of a message and an s byte max chunk size this will return an iterator over the messages contained in the s byte buffer s is intended to be larger than any single message but in the event of an abnormally large message the read can be retried multiple times each time doubling the buffer size until the message is read successfully a maximum message and buffer size can be specified to make the server reject messages larger than some size and to give a bound to the client on the maximum it need ever read to get a complete message it is likely that the read buffer ends with a partial message this is easily detected by the size delimiting the actual process of reading from an offset requires first locating the log segment file in which the data is stored calculating the file specific offset from the global offset value and then reading from that file offset the search is done as a simple binary search variation against an in memory range maintained for each file the log provides the capability of getting the most recently written message to allow clients to start subscribing as of right now this is also useful in the case the consumer fails to consume its data within its sla specified number of days in this case when the client attempts to consume a non existant offset it is given an outofrangeexception and can either reset itself or fail as appropriate to the use case the following is the format of the results sent to the consumer messagesetsend fetch result total length 4 bytes error code 2 bytes message 1 x bytes message n x bytes multimessagesetsend multifetch result total length 4 bytes error code 2 bytes messagesetsend 1 messagesetsend n deletes data is deleted one log segment at a time the log manager allows pluggable delete policies to choose which files are eligible for deletion the current policy deletes any log with a modification time of more than n days ago though a policy which retained the last n gb could also be useful to avoid locking reads while still allowing deletes that modify the segment list we use a copy on write style segment list implementation that provides consistent views to allow a binary search to proceed on an immutable static snapshot view of the log segments while deletes are progressing guarantees the log provides a configuration parameter m which controls the maximum number of messages that are written before forcing a flush to disk on startup a log recovery process is run that iterates over all messages in the newest log segment and verifies that each message entry is valid a message entry is valid if the sum of its size and offset are less than the length of the file and the crc32 of the message payload matches the crc stored with the message in the event corruption is detected the log is truncated to the last valid offset note that two kinds of corruption must be handled truncation in which an unwritten block is lost due to a crash and corruption in which a nonsense block is added to the file the reason for this is that in general the os makes no guarantee of the write order between the file inode and the actual block data so in addition to losing written data the file can gain nonsense data if the inode is updated with a new size but a crash occurs before the block containing that data is not written the crc detects this corner case and prevents it from corrupting the log though the unwritten messages are of course lost last modified december 19 2025 add license info to every source file and handle documentation html links for all versioned paths 764 379dba2230 previous message format next distribution the contents of this website are 2026 apache software foundation under the terms of the apache license v2 apache kafka kafka and the kafka logo are either registered trademarks or trademarks of the apache software foundation in the united states and other countries security donate thanks events license privacy