Meta tags:
description=Kafka® is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. What does all that mean? First let’s review some basic messaging terminology: Kafka maintains feeds of messages in categories called topics. We’ll call processes that publish messages to a Kafka topic producers. We’ll call processes that subscribe to topics and process the feed of published messages consumers.. Kafka is run as a cluster comprised of one or more servers each of which is called a broker.;

Headings (most frequently used words):

introduction, topics, and, logs, distribution, producers, consumers, guarantees, tag, cloud, tags,

Text of the page (most frequently used words):
the (94), kafka (34), #consumer (31), and (26), messages (22), for (21), that (20), this (18), consumers (17), partition (16), each (16), are (15), topic (13), log (12), partitions (11), server (10), order (10), message (10), one (10), all (9), with (9), can (9), group (9), cluster (9), will (8), more (8), they (8), which (8), apache (7), design (7), have (7), over (7), data (7), topics (7), instances (7), guarantees (6), offset (6), only (6), has (6), within (6), two (6), publish (6), leader (6), servers (6), security (5), use (5), documentation (5), producer (5), then (5), load (5), many (5), queue (5), messaging (5), but (5), subscribe (5), published (5), producers (5), introduction (5), license (4), any (4), sent (4), first (4), than (4), high (4), ordering (4), however (4), process (4), processes (4), like (4), some (4), called (4), started (4), streams (4), configuration (4), configs (4), cases (3), without (3), level (3), provides (3), not (3), different (3), key (3), you (3), parallelism (3), consumed (3), there (3), traditional (3), consume (3), from (3), out (3), may (3), means (3), groups (3), number (3), single (3), done (3), followers (3), distribution (3), example (3), clients (3), docs (3), getting (3), version (3), implementation (3), connect (3), page (3), operations (3), privacy (2), events (2), donate (2), contents (2), software (2), foundation (2), trademarks (2), other (2), info (2), source (2), handle (2), these (2), section (2), instance (2), stored (2), appended (2), same (2), total (2), between (2), per (2), achieved (2), mean (2), does (2), having (2), notion (2), provide (2), both (2), balancing (2), pool (2), retains (2), hands (2), delivered (2), effectively (2), consumption (2), system (2), four (2), subscriber (2), fault (2), tolerance (2), works (2), broadcast (2), name (2), separate (2), read (2), them (2), abstraction (2), second (2), acts (2), act (2), requests (2), new (2), its (2), distributed (2), replicated (2), configurable (2), size (2), fit (2), what (2), fact (2), position (2), controlled (2), older (2), days (2), after (2), available (2), performance (2), commit (2), feed (2), maintains (2), partitioned (2), let (2), logs (2), java (2), network (2), broker (2), call (2), basic (2), 2106 (2), developer (2), guide (2), apis (2), create (2), api (2), ecosystem (2), thanks, website, 2026, under, terms, logo, either, registered, united, states, countries, next, last, modified, december, 2025, add, every, file, html, links, versioned, paths, 764, 379dba2230, replication, factor, tolerate, failures, losing, committed, details, given, sees, particular, lower, appear, earlier, gives, following, combined, ability, sufficient, most, applications, require, though, better, able, assigning, exactly, doing, ensure, reader, consumes, since, still, balances, note, cannot, multiple, although, asynchronously, arrive, lost, presence, parallel, systems, often, work, around, exclusive, allows, course, processing, stronger, too, hosting, commonly, found, small, logical, composed, scalability, nothing, semantics, where, instead, just, label, themselves, subscribing, machines, traditionally, models, goes, offers, generalizes, queuing, their, choice, responsible, choosing, assign, round, robin, fashion, simply, balance, according, semantic, function, say, based, partitioning, zero, handles, write, while, passively, replicate, fails, automatically, become, follower, others, well, balanced, handling, share, across, serve, several, purposes, allow, scale, beyond, individual, must, host, arbitrary, amount, unit, bit, combination, features, very, cheap, come, much, impact, our, command, line, tools, tail, changing, existing, metadata, retained, basis, normally, advance, linearly, reads, likes, reset, reprocess, whether, been, period, time, retention, set, discarded, free, space, constant, respect, retaining, lots, problem, ordered, immutable, sequence, continually, assigned, sequential, uniquely, identifies, category, looks, dive, into, communication, simple, language, agnostic, client, languages, tcp, protocol, run, comprised, send, turn, serves, feeds, categories, review, terminology, service, functionality, unique, tags, viewing, date, see, latest, ops, tag, cloud, print, entire, issue, child, edit, view, zookeeper, monitoring, hardware, datacenters, format, layer, upgrading, quick, start, download, releases, sponsors, org, contact, trademark, project, summit, blog, community, powered, concepts, podcasts, videos, books, papers, quickstart, get,

Text of the page (random words):
introduction apache kafka get started introduction quickstart use cases books and papers videos podcasts docs key concepts apis configuration design implementation operations security clients kafka connect kafka streams powered by community blog kafka summit project info trademark ecosystem events contact us apache apache org license donate sponsors security privacy releases 4 3 4 2 4 1 4 0 3 9 3 8 3 7 3 6 3 5 3 4 3 3 3 2 3 1 3 0 2 8 2 7 2 6 2 5 2 4 2 3 2 2 2 1 2 0 1 1 1 0 0 11 0 0 10 2 0 10 1 0 10 0 0 9 0 0 8 2 0 8 1 0 8 0 0 7 download kafka ak 0 8 2 x getting started introduction use cases quick start ecosystem upgrading api configuration broker configs consumer configs producer configs new producer configs design design implementation api design network layer messages message format log distribution operations basic kafka operations datacenters kafka configuration java version hardware and os monitoring zookeeper security kafka connect kafka streams streams developer guide view page source edit this page create child page create documentation issue print entire section topics and logs distribution producers consumers guarantees tag cloud apis 1 configuration 34 connect 34 design 34 developer guide 34 docs 2106 getting started 34 implementation 34 kafka 2106 ops 34 security 34 streams 68 you are viewing documentation for an older version 0 8 2 of kafka for up to date documentation see the latest version ak 0 8 2 x getting started introduction introduction tags kafka docs kafka is a distributed partitioned replicated commit log service it provides the functionality of a messaging system but with a unique design what does all that mean first let s review some basic messaging terminology kafka maintains feeds of messages in categories called topics we ll call processes that publish messages to a kafka topic producers we ll call processes that subscribe to topics and process the feed of published messages consumers kafka is run as a cluster comprised of one or more servers each of which is called a broker so at a high level producers send messages over the network to the kafka cluster which in turn serves them up to consumers like this communication between the clients and the servers is done with a simple high performance language agnostic tcp protocol we provide a java client for kafka but clients are available in many languages topics and logs let s first dive into the high level abstraction kafka provides the topic a topic is a category or feed name to which messages are published for each topic the kafka cluster maintains a partitioned log that looks like this each partition is an ordered immutable sequence of messages that is continually appended to a commit log the messages in the partitions are each assigned a sequential id number called the offset that uniquely identifies each message within the partition the kafka cluster retains all published messages whether or not they have been consumed for a configurable period of time for example if the log retention is set to two days then for the two days after a message is published it is available for consumption after which it will be discarded to free up space kafka s performance is effectively constant with respect to data size so retaining lots of data is not a problem in fact the only metadata retained on a per consumer basis is the position of the consumer in the log called the offset this offset is controlled by the consumer normally a consumer will advance its offset linearly as it reads messages but in fact the position is controlled by the consumer and it can consume messages in any order it likes for example a consumer can reset to an older offset to reprocess this combination of features means that kafka consumers are very cheap they can come and go without much impact on the cluster or on other consumers for example you can use our command line tools to tail the contents of any topic without changing what is consumed by any existing consumers the partitions in the log serve several purposes first they allow the log to scale beyond a size that will fit on a single server each individual partition must fit on the servers that host it but a topic may have many partitions so it can handle an arbitrary amount of data second they act as the unit of parallelism more on that in a bit distribution the partitions of the log are distributed over the servers in the kafka cluster with each server handling data and requests for a share of the partitions each partition is replicated across a configurable number of servers for fault tolerance each partition has one server which acts as the leader and zero or more servers which act as followers the leader handles all read and write requests for the partition while the followers passively replicate the leader if the leader fails one of the followers will automatically become the new leader each server acts as a leader for some of its partitions and a follower for others so load is well balanced within the cluster producers producers publish data to the topics of their choice the producer is responsible for choosing which message to assign to which partition within the topic this can be done in a round robin fashion simply to balance load or it can be done according to some semantic partition function say based on some key in the message more on the use of partitioning in a second consumers messaging traditionally has two models queuing and publish subscribe in a queue a pool of consumers may read from a server and each message goes to one of them in publish subscribe the message is broadcast to all consumers kafka offers a single consumer abstraction that generalizes both of these the consumer group consumers label themselves with a consumer group name and each message published to a topic is delivered to one consumer instance within each subscribing consumer group consumer instances can be in separate processes or on separate machines if all the consumer instances have the same consumer group then this works just like a traditional queue balancing load over the consumers if all the consumer instances have different consumer groups then this works like publish subscribe and all messages are broadcast to all consumers more commonly however we have found that topics have a small number of consumer groups one for each logical subscriber each group is composed of many consumer instances for scalability and fault tolerance this is nothing more than publish subscribe semantics where the subscriber is cluster of consumers instead of a single process a two server kafka cluster hosting four partitions p0 p3 with two consumer groups consumer group a has two consumer instances and group b has four kafka has stronger ordering guarantees than a traditional messaging system too a traditional queue retains messages in order on the server and if multiple consumers consume from the queue then the server hands out messages in the order they are stored however although the server hands out messages in order the messages are delivered asynchronously to consumers so they may arrive out of order on different consumers this effectively means the ordering of the messages is lost in the presence of parallel consumption messaging systems often work around this by having a notion of exclusive consumer that allows only one process to consume from a queue but of course this means that there is no parallelism in processing kafka does it better by having a notion of parallelism the partition within the topics kafka is able to provide both ordering guarantees and load balancing over a pool of consumer processes this is achieved by assigning the partitions in the topic to the consumers in the consumer group so that each partition is consumed by exactly one consumer in the group by doing this we ensure that the consumer is the only reader of that partition and consumes the data in order since there are many partitions this still balances the load over many consumer instances note however that there cannot be more consumer instances than partitions kafka only provides a total order over messages within a partition not between different partitions in a topic per partition ordering combined with the ability to partition data by key is sufficient for most applications however if you require a total order over messages this can be achieved with a topic that has only one partition though this will mean only one consumer process guarantees at a high level kafka gives the following guarantees messages sent by a producer to a particular topic partition will be appended in the order they are sent that is if a message m1 is sent by the same producer as a message m2 and m1 is sent first then m1 will have a lower offset than m2 and appear earlier in the log a consumer instance sees messages in the order they are stored in the log for a topic with replication factor n we will tolerate up to n 1 server failures without losing any messages committed to the log more details on these guarantees are given in the design section of the documentation last modified december 19 2025 add license info to every source file and handle documentation html links for all versioned paths 764 379dba2230 next use cases the contents of this website are 2026 apache software foundation under the terms of the apache license v2 apache kafka kafka and the kafka logo are either registered trademarks or trademarks of the apache software foundation in the united states and other countries security donate thanks events license privacy