by Timur Dautov

What Is Apache Kafka? Core Concepts Explained

Apache Kafka is a distributed event streaming platform that is used for building real-time data pipelines and streaming applications.

At its core, Kafka is an append-only log. Producers add records to the end of the log, and consumers read them at their own pace. Unlike a traditional message queue, reading a record does not delete it, so several independent applications can process the same stream.

Core concepts#

Concept Description
Record A single event: a key, a value, a timestamp, and optional headers
Topic A named stream of records, such as orders
Partition One ordered log. A topic is split into one or more partitions
Offset The position of a record within a partition
Broker A Kafka server that stores partitions and serves clients
Producer An application that writes records to a topic
Consumer An application that reads records from a topic
Consumer group A set of consumers that share the work of reading a topic

How it works#

Partitions and ordering#

Partitions are how Kafka scales: a topic can be spread across many brokers, so it is not limited by the capacity of one machine.

Ordering is guaranteed only within a partition, not across a whole topic. Records with the same key go to the same partition, so using orderId as the key keeps all events of an order in sequence.

Consumer groups#

Within a consumer group, each partition is read by only one consumer at a time. Adding consumers to a group spreads the partitions between them, so a group with more consumers than partitions leaves some consumers idle.

Different groups are independent. A billing group and an analytics group can both read the orders topic, and each tracks its own offsets.

Retention#

Kafka keeps records for a configured time or size limit, whether they have been read or not. This means a consumer can rewind to an earlier offset and process old records again, for example after fixing a bug. For topics that hold the latest state of each key, log compaction can be used instead of deleting records by age.

Replication#

Each partition has a configurable number of replicas on different brokers. One replica is the leader and handles writes, and the followers copy its data. If the leader's broker fails, a follower that is in sync takes over.

Since Kafka 4.0, clusters manage their metadata with the built-in KRaft protocol, and ZooKeeper is no longer needed.

Try it from the command line#

With a local broker running on port 9092, create a topic with three partitions:

bin/kafka-topics.sh --create --topic orders --partitions 3 \
  --bootstrap-server localhost:9092

Write some records. Each line you type becomes a record:

bin/kafka-console-producer.sh --topic orders \
  --bootstrap-server localhost:9092

Read them in another terminal:

bin/kafka-console-consumer.sh --topic orders --from-beginning \
  --group billing --bootstrap-server localhost:9092

When to use Kafka#

Kafka is a good fit for:

  • Event-driven communication between services
  • Streaming data into warehouses, search indexes, and other systems
  • Activity tracking, logs, and metrics at high volume
  • Replaying history to rebuild state or test a new consumer

It is usually overkill for a small application with low traffic, or when you only need a simple job queue. Kafka needs a cluster to run and monitor, and a database table or a lighter broker is often enough.

Kafka Visualization#

Kafka Visualization is an interactive tool that shows how producers, partitions, and consumers behave. It is a good way to see these concepts in action.

Comments