Meta tags:
description=Distribution;

Headings (most frequently used words):

consumer, registry, broker, zookeeper, node, offsets, registration, algorithm, distribution, offset, tracking, directories, notation, topic, consumers, and, groups, id, partition, owner, rebalancing, migrating, from, to, kafka, tag, cloud, tags,

Text of the page (most frequently used words):
the (123), consumer (62), #consumers (38), #broker (36), and (33), offsets (29), offset (28), kafka (25), group (25), topic (24), each (20), this (18), zookeeper (18), for (17), partition (17), registry (16), are (15), manager (15), that (14), all (13), its (13), which (13), node (13), rebalancing (11), partitions (11), under (10), topics (10), will (10), can (10), commit (10), within (9), given (9), your (9), you (9), from (8), apache (7), when (7), same (7), also (7), new (7), brokers (7), ephemeral (7), streams (7), consumed (6), design (6), any (6), itself (6), registers (6), register (6), directory (6), storage (6), znode (6), configuration (6), security (5), owner (5), following (5), implementation (5), registration (5), group_id (5), value (5), with (5), use (5), fetch (5), distribution (5), license (4), other (4), documentation (4), let (4), may (4), need (4), nodes (4), among (4), would (4), connect (4), algorithm (4), groups (4), they (4), only (4), logical (4), ids (4), then (4), set (4), case (4), started (4), high (4), level (4), configs (4), registered (3), triggered (3), one (3), about (3), assign (3), releases (3), ownership (3), into (3), single (3), allowed (3), there (3), using (3), watch (3), changes (3), change (3), existing (3), their (3), consumer_id (3), used (3), possible (3), example (3), store (3), xyz (3), indicate (3), migrating (3), these (3), fail (3), after (3), automatically (3), simple (3), docs (3), version (3), page (3), operations (3), privacy (2), events (2), donate (2), contents (2), software (2), foundation (2), trademarks (2), log (2), last (2), add (2), info (2), source (2), should (2), consume (2), sort (2), size (2), try (2), does (2), during (2), algorithms (2), addition (2), multiple (2), some (2), get (2), data (2), such (2), way (2), number (2), has (2), creates (2), message (2), filter (2), trigger (2), available (2), joining (2), leaving (2), triggers (2), belongs (2), information (2), have (2), writes (2), host (2), name (2), port (2), list (2), broker_id (2), partition_id (2), must (2), establish (2), consumption (2), maximum (2), stored (2), simply (2), currently (2), disappear (2), process (2), dies (2), shared (2), unique (2), form (2), foobar (2), order (2), migrate (2), since (2), provides (2), named (2), containing (2), hello (2), world (2), notation (2), directories (2), back (2), steps (2), rolling (2), bounce (2), verify (2), healthy (2), config (2), dual (2), enabled (2), receives (2), request (2), vector (2), cache (2), just (2), retry (2), offsetfetchrequest (2), backing (2), off (2), done (2), offsetcommitrequest (2), periodically (2), fetches (2), commits (2), manage (2), manually (2), java (2), look (2), issue (2), tracking (2), 2106 (2), getting (2), developer (2), guide (2), apis (2), create (2), api (2), producer (2), ecosystem (2), cases (2), introduction (2), thanks, website, 2026, terms, logo, either, united, states, countries, previous, modified, december, 2025, every, file, handle, html, links, versioned, paths, 764, 379dba2230, time, subscribes, producing, clustered, together, index, position, remove, current, entries, owned, newly, assigned, until, original, allows, come, consensus, consuming, removal, both, divided, evenly, always, simplifies, had, concurrently, contention, kind, locking, required, more, than, won, reduces, force, rebalance, stream, being, added, evaluation, determine, changed, starts, basically, independent, publish, what, joins, dynamically, created, owners, consumer_node_id, before, begin, own, particular, claiming, offset_counter_value, persistent, track, valued, contains, map, identify, active, topic1, topicn, transient, hostname, uuid, identification, purposes, divide, fairly, exactly, jointly, run, across, three, machines, might, provided, tell, themselves, coordinate, balance, setting, however, mechanism, deprecated, future, release, therefore, recommended, maintains, stores, npartions, znodes, dynamic, shutdown, thus, notifying, longer, present, identifies, part, startup, creating, purpose, allow, moved, different, physical, machine, without, affecting, attempt, already, say, because, two, servers, configured, error, element, path, denoted, means, not, fixed, fact, sub, numerical, ranges, mldr, subdirectories, arrow, gives, structures, ordination, between, roll, performed, above, false, true, earlier, default, returns, committed, was, became, becoming, leader, load, offsetsloadinprogress, exception, appends, special, compacted, sends, successful, response, replicas, receive, replicate, configurable, timeout, compact, needs, maintain, most, recent, per, caches, memory, table, serve, quickly, consumer_offsets, tracks, resume, those, event, restart, option, designated, called, instance, send, handles, unsupported, scala, discover, explicitly, issuing, consumermetadatarequest, reading, consumermetadataresponse, contain, proceed, moves, rediscover, wish, take, code, samples, explain, how, tags, viewing, older, date, see, latest, ops, tag, cloud, print, entire, section, child, edit, view, monitoring, hardware, datacenters, basic, format, messages, network, layer, upgrading, quick, start, download, sponsors, org, contact, trademark, project, summit, blog, community, powered, clients, key, concepts, podcasts, videos, books, papers, quickstart,

Text of the page (random words):
distribution apache kafka get started introduction quickstart use cases books and papers videos podcasts docs key concepts apis configuration design implementation operations security clients kafka connect kafka streams powered by community blog kafka summit project info trademark ecosystem events contact us apache apache org license donate sponsors security privacy releases 4 3 4 2 4 1 4 0 3 9 3 8 3 7 3 6 3 5 3 4 3 3 3 2 3 1 3 0 2 8 2 7 2 6 2 5 2 4 2 3 2 2 2 1 2 0 1 1 1 0 0 11 0 0 10 2 0 10 1 0 10 0 0 9 0 0 8 2 0 8 1 0 8 0 0 7 download kafka ak 0 8 2 x getting started introduction use cases quick start ecosystem upgrading api configuration broker configs consumer configs producer configs new producer configs design design implementation api design network layer messages message format log distribution operations basic kafka operations datacenters kafka configuration java version hardware and os monitoring zookeeper security kafka connect kafka streams streams developer guide view page source edit this page create child page create documentation issue print entire section consumer offset tracking migrating offsets from zookeeper to kafka zookeeper directories notation broker node registry broker topic registry consumers and consumer groups consumer id registry consumer offsets partition owner registry broker node registration consumer registration algorithm consumer rebalancing algorithm tag cloud apis 1 configuration 34 connect 34 design 34 developer guide 34 docs 2106 getting started 34 implementation 34 kafka 2106 ops 34 security 34 streams 68 you are viewing documentation for an older version 0 8 2 of kafka for up to date documentation see the latest version ak 0 8 2 x implementation distribution distribution distribution tags kafka docs consumer offset tracking the high level consumer tracks the maximum offset it has consumed in each partition and periodically commits its offset vector so that it can resume from those offsets in the event of a restart kafka provides the option to store all the offsets for a given consumer group in a designated broker for that group called the offset manager i e any consumer instance in that consumer group should send its offset commits and fetches to that offset manager broker the high level consumer handles this automatically if you use the simple consumer you will need to manage offsets manually this is currently unsupported in the java simple consumer which can only commit or fetch offsets in zookeeper if you use the scala simple consumer you can discover the offset manager and explicitly commit or fetch offsets to the offset manager a consumer can look up its offset manager by issuing a consumermetadatarequest to any kafka broker and reading the consumermetadataresponse which will contain the offset manager the consumer can then proceed to commit or fetch offsets from the offsets manager broker in case the offset manager moves the consumer will need to rediscover the offset manager if you wish to manage your offsets manually you can take a look at these code samples that explain how to issue offsetcommitrequest and offsetfetchrequest when the offset manager receives an offsetcommitrequest it appends the request to a special compacted kafka topic named __ consumer_offsets the offset manager sends a successful offset commit response to the consumer only after all the replicas of the offsets topic receive the offsets in case the offsets fail to replicate within a configurable timeout the offset commit will fail and the consumer may retry the commit after backing off this is done automatically by the high level consumer the brokers periodically compact the offsets topic since it only needs to maintain the most recent offset commit per partition the offset manager also caches the offsets in an in memory table in order to serve offset fetches quickly when the offset manager receives an offset fetch request it simply returns the last committed offset vector from the offsets cache in case the offset manager was just started or if it just became the offset manager for a new set of consumer groups by becoming a leader for a partition of the offsets topic it may need to load the offsets topic partition into the cache in this case the offset fetch will fail with an offsetsloadinprogress exception and the consumer may retry the offsetfetchrequest after backing off this is done automatically by the high level consumer migrating offsets from zookeeper to kafka kafka consumers in earlier releases store their offsets by default in zookeeper it is possible to migrate these consumers to commit offsets into kafka by following these steps set offsets storage kafka and dual commit enabled true in your consumer config do a rolling bounce of your consumers and then verify that your consumers are healthy set dual commit enabled false in your consumer config do a rolling bounce of your consumers and then verify that your consumers are healthy a roll back i e migrating from kafka back to zookeeper can also be performed using the above steps if you set offsets storage zookeeper zookeeper directories the following gives the zookeeper structures and algorithms used for co ordination between consumers and brokers notation when an element in a path is denoted xyz that means that the value of xyz is not fixed and there is in fact a zookeeper znode for each possible value of xyz for example topics topic would be a directory named topics containing a sub directory for each topic name numerical ranges are also given such as 0 mldr 5 to indicate the subdirectories 0 1 2 3 4 an arrow is used to indicate the contents of a znode for example hello world would indicate a znode hello containing the value world broker node registry brokers ids 0 n host port ephemeral node this is a list of all present broker nodes each of which provides a unique logical broker id which identifies it to consumers which must be given as part of its configuration on startup a broker node registers itself by creating a znode with the logical broker id under brokers ids the purpose of the logical broker id is to allow a broker to be moved to a different physical machine without affecting consumers an attempt to register a broker id that is already in use say because two servers are configured with the same broker id is an error since the broker registers itself in zookeeper using ephemeral znodes this registration is dynamic and will disappear if the broker is shutdown or dies thus notifying consumers it is no longer available broker topic registry brokers topics topic 0 n npartions ephemeral node each broker registers itself under the topics it maintains and stores the number of partitions for that topic consumers and consumer groups consumers of topics also register themselves in zookeeper in order to coordinate with each other and balance the consumption of data consumers can also store their offsets in zookeeper by setting offsets storage zookeeper however this offset storage mechanism will be deprecated in a future release therefore it is recommended to migrate offsets storage to kafka multiple consumers can form a group and jointly consume a single topic each consumer in the same group is given a shared group_id for example if one consumer is your foobar process which is run across three machines then you might assign this group of consumers the id foobar this group id is provided in the configuration of the consumer and is your way to tell the consumer which group it belongs to the consumers in a group divide up the partitions as fairly as possible each partition is consumed by exactly one consumer in a consumer group consumer id registry in addition to the group_id which is shared by all consumers in a group each consumer is given a transient unique consumer_id of the form hostname uuid for identification purposes consumer ids are registered in the following directory consumers group_id ids consumer_id topic1 streams topicn streams ephemeral node each of the consumers in the group registers under its group and creates a znode with its consumer_id the value of the znode contains a map of topic streams this id is simply used to identify each of the consumers which is currently active within a group this is an ephemeral node so it will disappear if the consumer process dies consumer offsets consumers track the maximum offset they have consumed in each partition this value is stored in a zookeeper directory if offsets storage zookeeper this valued is stored in a zookeeper directory consumers group_id offsets topic broker_id partition_id offset_counter_value persistent node partition owner registry each broker partition is consumed by a single consumer within a given consumer group the consumer must establish its ownership of a given partition before any consumption can begin to establish its ownership a consumer writes its own id in an ephemeral node under the particular broker partition it is claiming consumers group_id owners topic broker_id partition_id consumer_node_id ephemeral node broker node registration the broker nodes are basically independent so they only publish information about what they have when a broker joins it registers itself under the broker node registry directory and writes information about its host name and port the broker also register the list of existing topics and their logical partitions in the broker topic registry new topics are registered dynamically when they are created on the broker consumer registration algorithm when a consumer starts it does the following register itself in the consumer id registry under its group register a watch on changes new consumers joining or any existing consumers leaving under the consumer id registry each change triggers rebalancing among all consumers within the group to which the changed consumer belongs register a watch on changes new brokers joining or any existing brokers leaving under the broker id registry each change triggers rebalancing among all consumers in all consumer groups if the consumer creates a message stream using a topic filter it also registers a watch on changes new topics being added under the broker topic registry each change will trigger re evaluation of the available topics to determine which topics are allowed by the topic filter a new allowed topic will trigger rebalancing among all consumers within the consumer group force itself to rebalance within in its consumer group consumer rebalancing algorithm the consumer rebalancing algorithms allows all the consumers in a group to come into consensus on which consumer is consuming which partitions consumer rebalancing is triggered on each addition or removal of both broker nodes and other consumers within the same group for a given topic and a given consumer group broker partitions are divided evenly among consumers within the group a partition is always consumed by a single consumer this design simplifies the implementation had we allowed a partition to be concurrently consumed by multiple consumers there would be contention on the partition and some kind of locking would be required if there are more consumers than partitions some consumers won t get any data at all during rebalancing we try to assign partitions to consumers in such a way that reduces the number of broker nodes each consumer has to connect to each consumer does the following during rebalancing 1 for each topic t that ci subscribes to 2 let pt be all partitions producing topic t 3 let cg be all consumers in the same group as ci that consume topic t 4 sort pt so partitions on the same broker are clustered together 5 sort cg 6 let i be the index position of ci in cg and let n size pt size cg 7 assign partitions from i n to i 1 n 1 to consumer ci 8 remove current entries owned by ci from the partition owner registry 9 add newly assigned partitions to the partition owner registry we may need to re try this until the original partition owner releases its ownership when rebalancing is triggered at one consumer rebalancing should be triggered in other consumers within the same group about the same time last modified december 19 2025 add license info to every source file and handle documentation html links for all versioned paths 764 379dba2230 previous log the contents of this website are 2026 apache software foundation under the terms of the apache license v2 apache kafka kafka and the kafka logo are either registered trademarks or trademarks of the apache software foundation in the united states and other countries security donate thanks events license privacy