What Is Apache Kafka? Core Concepts Explained
Apache Kafka is a distributed event streaming platform that is used for building real-time data pipelines and streaming applications.
At its core, Kafka is an append-only log. Producers add records to the end of the log, and consumers read them at their own pace. Unlike a traditional message queue, reading a record does not delete it, so several independent applications can process the same stream.
Core concepts#
| Concept | Description |
|---|---|
| Record | A single event: a key, a value, a timestamp, and optional headers |
| Topic | A named stream of records, such as orders |
| Partition | One ordered log. A topic is split into one or more partitions |
| Offset | The position of a record within a partition |
| Broker | A Kafka server that stores partitions and serves clients |
| Producer | An application that writes records to a topic |
| Consumer | An application that reads records from a topic |
| Consumer group | A set of consumers that share the work of reading a topic |
How it works#
Partitions and ordering#
Partitions are how Kafka scales: a topic can be spread across many brokers, so it is not limited by the capacity of one machine.
Ordering is guaranteed only within a partition, not across a whole topic. Records with the same key go to the same partition, so using orderId as the key keeps all events of an order in sequence.
Consumer groups#
Within a consumer group, each partition is read by only one consumer at a time. Adding consumers to a group spreads the partitions between them, so a group with more consumers than partitions leaves some consumers idle.
Different groups are independent. A billing group and an analytics group can both read the orders topic, and each tracks its own offsets.
Retention#
Kafka keeps records for a configured time or size limit, whether they have been read or not. This means a consumer can rewind to an earlier offset and process old records again, for example after fixing a bug. For topics that hold the latest state of each key, log compaction can be used instead of deleting records by age.
Replication#
Each partition has a configurable number of replicas on different brokers. One replica is the leader and handles writes, and the followers copy its data. If the leader's broker fails, a follower that is in sync takes over.
Since Kafka 4.0, clusters manage their metadata with the built-in KRaft protocol, and ZooKeeper is no longer needed.
Try it from the command line#
With a local broker running on port 9092, create a topic with three partitions:
bin/kafka-topics.sh --create --topic orders --partitions 3 \
--bootstrap-server localhost:9092
Write some records. Each line you type becomes a record:
bin/kafka-console-producer.sh --topic orders \
--bootstrap-server localhost:9092
Read them in another terminal:
bin/kafka-console-consumer.sh --topic orders --from-beginning \
--group billing --bootstrap-server localhost:9092
When to use Kafka#
Kafka is a good fit for:
- Event-driven communication between services
- Streaming data into warehouses, search indexes, and other systems
- Activity tracking, logs, and metrics at high volume
- Replaying history to rebuild state or test a new consumer
It is usually overkill for a small application with low traffic, or when you only need a simple job queue. Kafka needs a cluster to run and monitor, and a database table or a lighter broker is often enough.
Kafka Visualization#
Kafka Visualization is an interactive tool that shows how producers, partitions, and consumers behave. It is a good way to see these concepts in action.