Apache Kafka Architecture
The internals of the beast
Apache Kafka Architecture
Apache Kafka is a powerful distributed stream processing platform originally developed by LinkedIn, written in Scala and Java. In this article, I walk through the foundational concepts, design and architecture, concluding with hands-on coding examples using Node.js and the KafkaJS library.
Introduction to Kafka
Kafka enables the processing of real-time data streams in a distributed and scalable manner. It’s widely used in systems requiring reliable communication between components, especially for event-driven architectures and microservices.
However, these words sounds very markety and I like to deconstruct things to its basic first principles so that is what I’m going to do. This guide outlines the essential components, explains the core concepts like brokers, producers, consumers, and topics, and dives into more complex abstractions like partitions, consumer groups, and Kafka’s distributed nature.
Kafka Core Components
Here we discuss the fundamental components of Kafka.
Topics and Messages
Messages in Kafka are organized into logical categories called topics. Each topic is append-only and immutable (ie you cannot go back and edit a message). New data is added sequentially at the end. Disks love that.
Kafka Broker (Server)
The central server that receives and stores messages sent by producers and allow consumers to read the messages. Clients communicate with brokers via TCP, typically on port 9092.
Producers and Consumers:
Producers connects to a broker, sends messages to the Kafka broker by specifying the topic.
Consumers connects to a broker, polls messages from a topic for a given broker for processing.
Partitions
I always say the best way to work with billions of rows is to avoid working with billions of row. Kafka is no different.
The topics in Kafka can be divided into partitions to allow data distribution and scalability. Each partition acts as a log that consumers read sequentially. Messages within partitions are identified by their offset, which is the message’s position in the log.
Partitioning enables parallel processing, you can have two consumers each processing a partition in parallel. While this allows scaling and improve reading throughput, it introduces complexity as producers and consumers must be aware of partitioning logic. A topic may have multiple partitions based on partition keys (e.g., A–M in partition 0, N–Z in partition 1).
Pub/Sub vs Queue Model
Kafka supports both pub/sub and queue-based messaging patterns:
Queue: The topic can be treated as a queue, ie any message published to the queue will be consumed only once and popped from the queue. No two consumers can get the same message.
Pub/Sub: The topic can be in publish-subscribe entity, ie any message published to the topic can be consumed by different consumers.
Kafka achieves this dual functionality using consumer groups. Each consumer in a group is assigned specific partitions, ensuring messages are processed only once per group.
To simulate a queue , all consumers are placed in the same consumer group (they all have the same group name). Whereas to simulate pub/sub mode you would place all consumers in unique group names.
Consumer Groups and Rebalancing
Each partition in a topic is assigned one consumer in a group. When multiple consumers join the same group, Kafka rebalances partitions among them. Each partition is consumed by only one consumer in a group. This allows horizontal scaling while ensuring order within each partition is maintained. Will demonstrate that in the code section.
Distributed Kafka Clusters and Zookeeper
Kafka operates as a distributed system with multiple brokers. Each broker can be a leader or follower for different partitions. The leader handles reads and writes, while followers replicate data.
Kafka originally relied on Zookeeper to manage broker coordination, metadata, and leader election. Zookeeper tracks which broker is the leader of which partition. However, Zookeeper introduces complexity and is being phased out in newer Kafka versions. Kafka 4.0 no longer has Zookeeper.
On Long Polling
You may ask, how does the consumer get the data the moment it is published to the topic/partition? I encourage you to first think of how could this be implemented.
One approach is push, ie the broker pushes the message to the consumer TCP connection. This approach is real-time, but overwhelms the consumer, as it needs to process the message.
Another approach is polling, ie the consumer every X milliseconds asks the broker, is there anything for me and the broker replies, yes here you go, or no nothing, this approach does allow the consumer to read at its own pace but may introduce severe network chatter, which lead to network bandwidth, it also may causes consumer to miss the real-time message, since the message may have came in to the broker between polls.
The final approach which Kafka went with is long polling which I talked extensively on my backend course. The consumer connects to the broker and sends a normal poll, if there is a message the broker replies back, if there isn’t the broker doesn’t respond with “no”, but instead just wait little bit more (configurable), this increases the chances that messages may arrive during this wait. Once we get a message the broker writes the response. Meanwhile the consumer feels like its poll request just took little longer.
Kafka, Docker and Node Example
We will use docker to install Kafka broker and Zookeeper single mode cluster. Then create a topic called “Users” with two partitions, produce some messages on both partitions and have consumers read from the partitions.
My docker host IP is 192.168.7.179, replace it with your own docker host IP.
You will need to have docker and Node installed to proceed
First we clone the repo and change IP address to your docker host.
#clone the repo
git clone https://github.com/hnasr/javascript_playground.git
#change to the kafka directory
cd javascript_playground/kafka
#edit docker-compose.yml, producer.js, topics.js and consumer.js change the 192.168.7.179 with your docker host ip.
cat starthere.txtSpin up Zookeeper and Kafka containers using Docker commands.
services:
zookeeper:
image: zookeeper
hostname: zookeeper
ports:
- 2181:2181
volumes:
- zookeeper_data:/data
- zookeeper_datalog:/datalog
kafka:
image: ches/kafka
restart: always
hostname: localhost
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 192.168.7.179
ZOOKEEPER_IP: 192.168.7.179
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
ALLOW_PLAINTEXT_LISTENER: 1
depends_on:
- zookeeper
volumes:
- kafka_data:/kafka
volumes:
zookeeper_data:
zookeeper_datalog:
kafka_data:docker-compose up
Create a topic called `MyUsers` with two partitions, we will use kafkajs module to connect to the kafka broker.
//const Kafka = require("kafkajs").Kafka
const {Kafka} = require("kafkajs")
run();
async function run(){
try
{
const kafka = new Kafka({
"clientId": "myapp",
"brokers" :["192.168.7.179:9092"]
})
const admin = kafka.admin();
console.log("Connecting.....")
await admin.connect()
console.log("Connected!")
//A-M, N-Z
await admin.createTopics({
"topics": [{
"topic" : "MyUsers",
"numPartitions": 2
}]
})
console.log("Created Successfully!")
await admin.disconnect();
}
catch(ex)
{
console.error(`Something bad happened ${ex}`)
}
finally{
process.exit(0);
}
}node topics.jsProduce some messages with a producer, the messages are user names. I have the messages starting with letters <N goes to partition 1 anything ≥N goes to partition 2.
//const Kafka = require("kafkajs").Kafka
const {Kafka} = require("kafkajs")
const msg = process.argv[2];
run();
async function run(){
try
{
const kafka = new Kafka({
"clientId": "myapp",
"brokers" :["192.168.7.179:9092"]
})
const producer = kafka.producer();
console.log("Connecting.....")
await producer.connect()
console.log("Connected!")
//A-M 0 , N-Z 1
const partition = msg[0] < "N" ? 0 : 1;
const result = await producer.send({
"topic": "Users",
"messages": [
{
"value": msg,
"partition": partition
}
]
})
console.log(`Send Successfully! ${JSON.stringify(result)}`)
await producer.disconnect();
}
catch(ex)
{
console.error(`Something bad happened ${ex}`)
}
finally{
process.exit(0);
}
}node producer.js Adam
node producer.js ZackHusseinMac:kafka HusseinNasser$ node producer.js Adam
Connecting.....
Connected!
Send Successfully! [{"topicName":"MyUsers","partition":0,"errorCode":0,"offset":"0","timestamp":"-1"}]
HusseinMac:kafka HusseinNasser$ node producer.js Zack
Connecting.....
Connected!
Send Successfully! [{"topicName":"MyUsers","partition":1,"errorCode":0,"offset":"0","timestamp":"-1"}]Implement a consumer that subscribes to the topic and handles messages with long polling.
Note that I hard coded a consumer group name “test” in the consumer.js file. Running the consumer.js file will create a new consumer on the MyUsers topic and it will be responsible for both partitions. You can note this from the response"memberAssignment”:{“MyUsers”:[0,1]}
HusseinMac:kafka HusseinNasser$ node consumer.js
Connecting.....
Connected!
{"level":"INFO","timestamp":"2025-04-13T14:29:36.102Z","logger":"kafkajs","message":"[Consumer] Starting","groupId":"test"}
{"level":"INFO","timestamp":"2025-04-13T14:29:58.411Z","logger":"kafkajs","message":"[Runner] Consumer has joined the group","groupId":"test","memberId":"myapp-901679c6-034c-4efd-bc4d-cb5fc5bc9e6e","leaderId":"myapp-901679c6-034c-4efd-bc4d-cb5fc5bc9e6e","isLeader":true,"memberAssignment":{"MyUsers":[0,1]},"groupProtocol":"RoundRobinAssigner","duration":22307}
RVD Msg Adam on partition 0
RVD Msg Zack on partition 1Now assume we run another consumer, which will also have the same group. Kafka will rebalance the group and assign a partition for each consumer so they parallel process.
Note the memberAssignment property for each consumer.
HusseinMac:kafka HusseinNasser$ node consumer.js (consumer 1)
{"level":"ERROR","timestamp":"2025-04-13T14:31:09.360Z","logger":"kafkajs","message":"[Connection] Response Heartbeat(key: 12, version: 0)","broker":"192.168.7.179:9092","clientId":"myapp","error":"The group is rebalancing, so a rejoin is needed","correlationId":39,"size":6}
{"level":"ERROR","timestamp":"2025-04-13T14:31:09.361Z","logger":"kafkajs","message":"[Runner] The group is rebalancing, re-joining","groupId":"test","memberId":"myapp-901679c6-034c-4efd-bc4d-cb5fc5bc9e6e","error":"The group is rebalancing, so a rejoin is needed","retryCount":0,"retryTime":308}
{"level":"INFO","timestamp":"2025-04-13T14:31:09.382Z","logger":"kafkajs","message":"[Runner] Consumer has joined the group","groupId":"test","memberId":"myapp-901679c6-034c-4efd-bc4d-cb5fc5bc9e6e","leaderId":"myapp-901679c6-034c-4efd-bc4d-cb5fc5bc9e6e","isLeader":true,"memberAssignment":{"MyUsers":[1]},"groupProtocol":"RoundRobinAssigner","duration":21}
HusseinMac:kafka HusseinNasser$ node consumer.js (consumer 2)
Connecting.....
Connected!
{"level":"INFO","timestamp":"2025-04-13T14:31:07.599Z","logger":"kafkajs","message":"[Consumer] Starting","groupId":"test"}
{"level":"INFO","timestamp":"2025-04-13T14:31:09.383Z","logger":"kafkajs","message":"[Runner] Consumer has joined the group","groupId":"test","memberId":"myapp-6c861d03-879d-4b7b-81a1-1e2cd1c15c83","leaderId":"myapp-901679c6-034c-4efd-bc4d-cb5fc5bc9e6e","isLeader":false,"memberAssignment":{"MyUsers":[0]},"groupProtocol":"RoundRobinAssigner","duration":1783}Let us publish one message for each partition and see what happens.
node producer.js Adamy
node producer.js Zackery(consumer 1)
RVD Msg Zackery on partition 1
(consumer 2)
RVD Msg Adamy on partition 0
What happens if more consumers than partitions are available? As per testing, new consumers won’t get any partitions assigned. But a consumer group rebalance may reassign partitions.
Kafka Pros and Cons
By understanding the architecture and design of Apache Kafka we know the fact it is performs well for writes (producing messages), because of the append-only nature. And having the consumer aware of the position to where to read from makes reads also fast. Kafka scales well with partitioning and handle fault tolerant with the distributed nature.
However, no software is free of limitations, with all these features added to Kafka, complexity is unavoidable. Producers needs to be aware of the partitions which introduces complexity in producing messages. The need for distributed architecture, replication introduces depedancies on an external component (Zookeeper) which adds more complexity and even latency. All of this makes Kafka difficult to install, maintain and manage.
Note that Kafka 4.0 that was release in March 2025 has completely removed Zookeeper in favor of a raft based replication between brokers.
Conclusion
Now that you know the basics of Apache Kafka, you will understand its power and its limitations. That is the most important thing. Anything on top of Kafka uses these fundamentals.
If you enjoyed this article, check out my backend and database courses.
Good explanation
amazing article!