Kafka Interview Prep

A comprehensive collection of Apache Kafka interview questions and answers, covering architecture, producers, consumers, reliability guarantees, and production operations.

Topics covered:

• Core architecture (brokers, topics, partitions, ZooKeeper/KRaft)
• Producers (acks, idempotency, partitioning)
• Consumers (consumer groups, offsets, rebalancing)
• Reliability (delivery semantics, replication, ISR)
• Performance (zero-copy, batching, compression)
• Kafka Connect & Streams
• Operations (retention, compaction, monitoring)

Q: What is Apache Kafka and why would you use it?

Answer:

Apache Kafka is an open-source distributed event streaming platform originally developed at LinkedIn and donated to the Apache Software Foundation. It is designed for high-throughput, fault-tolerant, real-time data streaming.

What Kafka Does

At its core, Kafka is a distributed commit log. Producers write messages (events) to topics, and consumers read those messages. Unlike traditional message queues, messages in Kafka are persisted to disk and can be replayed.

Why Use Kafka?

1. Decoupling of Systems Instead of service A calling service B directly (tight coupling), A publishes an event to Kafka. Any service interested in that event subscribes independently.

Without Kafka:  OrderService → PaymentService → EmailService → AnalyticsService
                (chain of synchronous calls, one failure breaks everything)

With Kafka:     OrderService → [Kafka Topic: "orders"]
                                    ├── PaymentService (consumer)
                                    ├── EmailService (consumer)
                                    └── AnalyticsService (consumer)

2. Extreme Throughput Kafka handles millions of messages per second with latency under 10ms. LinkedIn processes over 7 trillion messages/day through Kafka.

3. Durability & Replayability Messages are written to disk and replicated across brokers. Consumers can re-read old messages (e.g., replay the last 7 days of events to rebuild a search index).

4. Ordering Guarantees Messages within a single partition are strictly ordered — essential for event sourcing and log-based architectures.

Common Use Cases

• Event-driven microservices — publish domain events between services.
• Real-time analytics — stream clickstream data to dashboards.
• Log aggregation — centralize logs from thousands of servers.
• Change Data Capture (CDC) — stream database changes (via Debezium + Kafka Connect).
• ETL pipelines — replace batch processing with real-time streaming.

[!NOTE] Kafka is NOT a traditional message queue (like RabbitMQ or SQS). It's a distributed log that happens to be excellent at messaging. The key difference: messages aren't deleted after consumption — they're retained based on a time or size policy.

Q: Explain Brokers, Topics, and Partitions in Kafka.

Answer:

These are the three fundamental building blocks of Kafka's architecture.

Broker

A broker is a single Kafka server. A Kafka cluster consists of multiple brokers (typically 3+). Each broker:

• Stores data on disk
• Serves producer and consumer requests
• Participates in replication
• Is identified by a unique integer ID

Brokers are designed so that no single broker holds all the data for a topic — data is distributed across brokers via partitions.

Topic

A topic is a named category/feed to which messages are published. Think of it as a table in a database or a folder in a filesystem.

Topics: "user-signups", "order-events", "payment-transactions"

Topics are multi-subscriber — many consumer groups can read from the same topic independently without affecting each other.

Partition

Each topic is split into one or more partitions. A partition is an ordered, immutable sequence of messages (an append-only log). Each message within a partition gets a sequential ID called an offset.

Topic: "orders" (3 partitions)

Partition 0: [msg0] [msg1] [msg2] [msg3] [msg4] → 
Partition 1: [msg0] [msg1] [msg2] →
Partition 2: [msg0] [msg1] [msg2] [msg3] →

Why Partitions Matter

1. Parallelism Each partition can be consumed by a different consumer in a consumer group. More partitions = more consumers = higher throughput.

2. Ordering Messages are strictly ordered WITHIN a partition, but there is no ordering guarantee ACROSS partitions. If ordering matters for a specific entity (e.g., all events for user X), you must ensure all events for that entity go to the same partition using a partition key.

3. Distribution Partitions are spread across brokers. For a topic with 6 partitions on a 3-broker cluster, each broker holds ~2 partitions.

How They Relate

Kafka Cluster
├── Broker 0
│   ├── orders-partition-0 (Leader)
│   └── orders-partition-2 (Follower)
├── Broker 1
│   ├── orders-partition-1 (Leader)
│   └── orders-partition-0 (Follower)
└── Broker 2
    ├── orders-partition-2 (Leader)
    └── orders-partition-1 (Follower)

[!IMPORTANT] Choosing the right number of partitions is a critical design decision. Too few = throughput bottleneck. Too many = increased memory usage, slower leader elections, and longer recovery times. A common starting point is number of partitions = desired throughput / throughput per partition (usually a few MB/s per partition).

Q: What is the difference between ZooKeeper and KRaft mode?

Answer:

This is a hot interview topic because Kafka is in the middle of a major architectural transition.

ZooKeeper Mode (Legacy)

Historically, Kafka relied on Apache ZooKeeper — a separate distributed coordination service — to manage cluster metadata:

• Broker registration (which brokers are alive)
• Controller election (one broker is the "controller" that manages partition leadership)
• Topic configuration (partition count, replication factor, ACLs)
• Consumer group offsets (in older versions; now stored in Kafka itself)

┌─────────────────────┐
│  ZooKeeper Ensemble  │  (3-5 separate servers)
│  ┌───┐ ┌───┐ ┌───┐  │
│  │ZK1│ │ZK2│ │ZK3│  │
│  └───┘ └───┘ └───┘  │
└─────────┬───────────┘
          │ metadata
┌─────────▼───────────┐
│    Kafka Cluster     │
│  ┌──┐ ┌──┐ ┌──┐     │
│  │B0│ │B1│ │B2│     │
│  └──┘ └──┘ └──┘     │
└─────────────────────┘

KRaft Mode (New, ZooKeeper-Free)

Starting with Kafka 3.3 (production-ready in 3.5+), Kafka can run without ZooKeeper using an internal Raft-based consensus protocol called KRaft (Kafka Raft).

In KRaft mode, a subset of Kafka brokers act as controllers and manage all metadata internally using a replicated metadata log (__cluster_metadata topic).

┌─────────────────────────────┐
│        Kafka Cluster        │
│  ┌────────┐  ┌──┐  ┌──┐    │
│  │B0 (ctrl)│  │B1│  │B2│   │
│  │B1 (ctrl)│  └──┘  └──┘   │
│  │B2 (ctrl)│                │
│  └────────┘                 │
│  (controllers embedded)     │
└─────────────────────────────┘

Why the Migration?

Current Status

• ZooKeeper mode: Deprecated as of Kafka 3.5. Will be removed entirely in Kafka 4.0.
• KRaft mode: Production-ready. All new deployments should use KRaft.

[!TIP] In interviews, mentioning the KRaft migration shows you're up-to-date with the Kafka ecosystem. If asked "how does Kafka manage metadata?", mention both modes and note that ZooKeeper is being phased out.

Q: How does Replication work in Kafka? What is the ISR?

Answer:

Replication is how Kafka achieves fault tolerance. Each partition is replicated across multiple brokers.

Key Concepts

Replication Factor: The number of copies of each partition. A replication factor of 3 means every partition has 3 replicas across 3 different brokers.

Leader Replica: One replica is designated the leader. All producer writes and consumer reads go through the leader.

Follower Replicas: The remaining replicas continuously fetch new messages from the leader to stay in sync. They don't serve client requests (by default).

Topic: "payments" (replication-factor=3)

Broker 0: [Partition 0 - LEADER]    [Partition 1 - Follower]
Broker 1: [Partition 0 - Follower]  [Partition 1 - LEADER]
Broker 2: [Partition 0 - Follower]  [Partition 1 - Follower]

What is the ISR (In-Sync Replicas)?

The ISR is the set of replicas that are "caught up" with the leader — they have replicated all messages within the allowed lag threshold (replica.lag.time.max.ms, default 30s).

Partition 0:
  Leader (Broker 0):   offset 100
  Follower (Broker 1): offset 99   ← In ISR (close enough)
  Follower (Broker 2): offset 85   ← NOT in ISR (too far behind)

ISR = {Broker 0, Broker 1}

Why ISR Matters

The ISR directly affects data durability and availability:

1. With acks=all: The producer considers a write successful only when ALL replicas in the ISR have acknowledged it. If the ISR shrinks to just the leader, acks=all effectively becomes acks=1.

2. Leader Election: When a leader fails, the new leader is chosen from the ISR (by default). This ensures no data loss because ISR members have all committed messages.

3. min.insync.replicas: A critical safety net. If set to 2 (with replication-factor=3), the producer will refuse to write if the ISR drops below 2 replicas. This prevents data loss scenarios.

Common Production Configuration

# Topic-level
replication.factor=3
min.insync.replicas=2

# Producer-level
acks=all

This means:

• 3 copies of every partition.
• At least 2 must acknowledge before a write is confirmed.
• If 2 brokers die, writes are rejected (protecting data integrity over availability).

[!CAUTION] Setting unclean.leader.election.enable=true allows an out-of-sync replica to become leader when all ISR members are dead. This guarantees availability but risks data loss because the new leader may be missing messages. In most production systems, this is set to false.

Q: What is Kafka Tiered Storage (KIP-405) and when should you use it?

Answer:

Tiered Storage decouples Kafka's compute (brokers) from its storage (long-term retention) by moving older log segments off broker-local disks into cheaper remote object storage (S3, GCS, Azure Blob, HDFS).

The Problem It Solves

Pre-KIP-405, every byte you wanted to retain lived on the broker's local disk. That forced uncomfortable tradeoffs:

• Want 30-day retention for replay? Buy 30 days of SSD on every broker.
• Replication factor 3 multiplies that storage cost by 3x.
• Adding a broker triggers expensive partition reassignment to rebalance terabytes of cold data.
• A broker failure means hours of re-replication for data nobody is actively reading.

The Architecture

        ┌─────────────────────────────────────────┐
        │              Kafka Broker               │
        │                                         │
        │   ┌─────────────┐   ┌──────────────┐    │
        │   │ Local Tier  │   │ Remote Log   │    │
        │   │ (hot data)  │   │ Manager      │    │
        │   │  SSD/NVMe   │   │ (RLM)        │    │
        │   └─────────────┘   └──────┬───────┘    │
        └─────────────────────────────┼───────────┘
                                      │
                              ┌───────▼────────┐
                              │ Remote Tier    │
                              │ S3 / GCS / ... │
                              │ (cold data)    │
                              └────────────────┘

• Local tier: Active segment + recent closed segments. Serves the tail of the log — what producers write and consumers usually read.
• Remote tier: Closed segments older than local.retention.ms are uploaded by the Remote Log Manager. The local copy is then deleted.
• Reads for offsets in remote segments are fetched transparently — the consumer doesn't know or care where bytes live.

Configuration

Cluster level:

remote.log.storage.system.enable=true
remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.RemoteLogManager
remote.log.metadata.manager.class.name=org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager

Topic level:

kafka-configs.sh --alter --topic events \
  --add-config remote.storage.enable=true,\
local.retention.ms=86400000,\
retention.ms=2592000000

• retention.ms: total retention (local + remote). Here, 30 days.
• local.retention.ms: how much stays on broker disk. Here, 1 day.

When to Use It

Good fit:

• Long-retention topics (compliance, audit, replay, ML training data).
• Topics where 95% of reads are tail reads and historical reads are rare.
• You want elastic brokers — adding/removing nodes shouldn't shuffle terabytes.

Bad fit:

• Low-retention, high-throughput pipelines (you'd just round-trip to S3 for no reason).
• Workloads with frequent historical scans (S3 GET latency >> NVMe).
• Compacted topics — KIP-405 currently only supports delete-cleanup-policy topics.

Tradeoffs

[!NOTE] Confluent Cloud and AWS MSK both ship Tiered Storage with proprietary remote tiers. KIP-405 is the open-source baseline — production-ready in Apache Kafka 3.6+.

Interview Follow-ups

• "What happens to remote segments if a topic is deleted?" — RLM issues asynchronous delete of remote objects; orphans are possible during failure, hence lifecycle policies.
• "How is metadata about remote segments stored?" — In an internal topic __remote_log_metadata (default impl).
• "Does it affect producer throughput?" — No; producers only write to local tier. Upload is async.

Q: How is a Kafka partition stored on disk — segments, indexes, and the page cache trick?

Answer:

A Kafka partition is an append-only sequence of bytes, materialized on disk as a series of segment files plus index files. The on-disk layout is intentionally boring — that's why Kafka is fast.

Per-Partition Directory Layout

/var/kafka/orders-0/
├── 00000000000000000000.log         <- segment data (messages)
├── 00000000000000000000.index       <- offset → file position
├── 00000000000000000000.timeindex   <- timestamp → offset
├── 00000000000019234567.log
├── 00000000000019234567.index
├── 00000000000019234567.timeindex
├── leader-epoch-checkpoint
└── partition.metadata

The number in the filename is the base offset — the offset of the first message in that segment.

Segments

A partition is split into segments to make rolling and deletion cheap. Only the active segment (the one with the highest base offset) is being written. Closed segments are immutable.

A segment is rolled when:

• segment.bytes (default 1 GB) reached.
• segment.ms (default 7 days) elapsed.

Why segments matter:

• Retention: delete entire old segments — no defragmentation, no rewriting.
• Compaction: produce a new segment from old ones, swap atomically.
• Tiered storage: closed segments can be uploaded to S3.

The Log File Format

Each segment .log is a sequence of record batches:

RecordBatch:
  baseOffset, batchLength
  partitionLeaderEpoch
  magic (record format version)
  CRC
  attributes (compression, transactional, control)
  lastOffsetDelta
  baseTimestamp, maxTimestamp
  producerId, producerEpoch, baseSequence
  records: [varint-encoded records]

Batches preserve producer-side batching all the way to disk. Compression (lz4, zstd) is per-batch. Consumers receive batches without re-encoding.

.index — Sparse Offset Index

Maps offset → physical position. Sparse, not per-message:

relative_offset:int32   position:int32
        0                0
       128            4096       <- one entry per ~4 KB
       256            8192
       384           12288

Entry every index.interval.bytes (default 4096). To find offset N:

1. Binary search the sparse index for the largest entry ≤ N.
2. Scan forward in the .log from that file position.

Sparse index = small enough to mmap entirely. Linear scan from there = fast because of sequential disk reads.

.timeindex — Time-to-Offset Index

timestamp:int64   offset:int64

Used for --from-timestamp consumer rewinds and retention by time. Also sparse.

Why Kafka Is Fast — The Page Cache Story

Kafka writes to a normal file. The OS caches recently written pages in page cache (RAM). Consumers reading the tail of the log read from page cache, not disk.

Producer  ──►  write()  ──►  Page Cache  ──►  Disk (async by OS)
                                  │
                                  ▼
Consumer  ──◄── read()  ◄──  Page Cache (still warm)

Two consequences:

1. Producer doesn't fsync per record. It writes to the OS, returns. The OS flushes on its own schedule. Crash-safe via replication, not fsync.
2. Tail-reading consumers hit RAM, not disk. Throughput is bounded by network, not IOPS.

Zero-Copy Send (sendfile)

When delivering to a consumer, Kafka uses sendfile() (Linux syscall) to copy bytes directly from the page cache to the socket buffer:

Without sendfile:
  page cache → user buffer → kernel socket buffer → NIC   (4 copies, 4 context switches)

With sendfile:
  page cache → NIC                                       (1 copy in kernel, 2 switches)

This works only if no transformation is needed — which is why Kafka stores compressed batches unchanged and never decompresses on the broker.

Replication

Each partition has a leader and N replicas. Followers fetch from the leader exactly the same way consumers do. The leader tracks each replica's high-water mark. A record is committed once it's in all in-sync replicas (min.insync.replicas).

Leader:        offset 1000 (latest)
Follower A:    offset 998
Follower B:    offset 999
HW (committed): 998   ← min of in-sync followers

Consumers can only read up to the high-water mark.

Log Compaction (vs Delete)

cleanup.policy=delete (default): retention by time or size, drop oldest segments.

cleanup.policy=compact: keep at least one record per key. The log cleaner periodically rewrites segments removing superseded records:

Before:    k1:v1, k2:v2, k1:v3, k3:v4, k1:v5
After:                          k2:v2, k3:v4, k1:v5

Useful for changelog topics, Kafka Streams state, KRaft metadata.

Both can be combined: cleanup.policy=compact,delete.

Operational Implications

• Disk choice: sequential I/O dominates. SATA SSDs/NVMe are great. Spinning disk works if you have enough partitions to parallelize writes. RAID-10 over RAID-5/6.
• No need to size the heap large. Kafka uses ~4–8 GB heap; the rest of RAM is page cache. -Xmx4g is normal even on a 64 GB box.
• fsync isn't durability in Kafka. Replication is. Don't tune flush.messages / flush.ms aggressively — let the OS handle it.
• Tail-read consumers ≠ historical-read consumers: tail = RAM, historical = disk. A consumer that lags by hours pulls from disk and competes with the page cache.

Examining a Segment

# Dump batches with the tool
kafka-dump-log.sh --files 00000000000019234567.log --print-data-log

# Output:
# offset: 19234567 position: 0 batchSize: 1234 ...
# offset: 19234568 timestamp: 1700000000000 key: "abc" payload: {...}

Common Mistakes

[!NOTE] The whole architecture is "let the OS be the database." Kafka stores bytes in order, lets Linux cache them, and ships them with sendfile. The cleverness is in not being clever.

Interview Follow-ups

• "Why doesn't Kafka use a B-tree like a database?" — Append-only sequential writes are the access pattern. Sparse index + binary search is enough; trees would slow writes.
• "Why does deleting old data not fragment the log?" — Segments are entire files. Deletion = unlink(). No fragmentation possible.
• "How does Tiered Storage interact with this?" — Closed segments + their indexes are uploaded to remote (S3). Local copy is then deleted. Reads of old offsets transparently fetch from remote.

Q: How does KRaft mode work, and what changed when ZooKeeper was removed?

Answer:

KRaft (KIP-500) replaces ZooKeeper with a built-in Raft consensus quorum running inside Kafka brokers. As of Kafka 3.5+, KRaft is the default and ZooKeeper mode is deprecated; Kafka 4.0 removes ZK entirely.

Why Remove ZooKeeper

ZK-based Kafka had two distinct systems with different operational profiles:

• ZK stored cluster metadata (topics, partitions, ACLs, configs).
• Kafka brokers stored data.

Pain points:

• Two systems to deploy, monitor, secure, upgrade.
• Metadata changes went through ZK's notification model — slow for many partitions.
• Controller fail-over was O(N) in topic count (had to reload every partition state).
• ZK's write throughput capped the maximum partition count at ~200k cluster-wide.

KRaft replaces this with a Raft log of metadata events, kept inside Kafka itself.

Architecture

ZooKeeper mode:                    KRaft mode:

  ┌──────────┐                     ┌────────────────────────┐
  │  ZK      │                     │  Controllers (Raft)    │
  │ ensemble │                     │  c1   c2   c3          │
  └────┬─────┘                     └──────────┬─────────────┘
       │ metadata                              │ metadata via Raft log
       │                                       │
  ┌────┴───────┐                     ┌────────┴─────────────┐
  │  Brokers   │                     │  Brokers             │
  │ b1 b2 b3   │                     │  b1   b2   b3        │
  └────────────┘                     └──────────────────────┘

• Controller quorum: typically 3 (or 5) nodes that own metadata.
• Brokers: handle data, replicate from each other, fetch metadata from the controller quorum.
• Same process can run as process.roles=broker,controller (combined mode, for small clusters) or split (recommended for production).

The Metadata Log

Cluster metadata is stored on a special internal topic: __cluster_metadata. Single partition. Replicated across all controller nodes via Raft.

Every metadata change is an event in this log:

• Create topic → TopicRecord.
• Change partition leader → PartitionChangeRecord.
• Register broker → RegisterBrokerRecord.
• Update ACL → AccessControlEntryRecord.

Each broker replays the log into an in-memory image. On startup, brokers consume from offset 0 (or from a snapshot) and end at the high-water mark.

Snapshots

Replaying the entire log on every restart would be slow on big clusters. KRaft periodically writes a snapshot of the current metadata image:

__cluster_metadata segments:
  00000000000000000000.log
  00000000000000000000.checkpoint           <-- snapshot
  00000000000019234567.log

New brokers load the snapshot, then replay only the delta. Startup goes from minutes (with ZK) to seconds.

Configuration

server.properties:

process.roles=broker,controller             # combined; or just broker / controller
node.id=1
controller.quorum.voters=1@c1:9093,2@c2:9093,3@c3:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER
log.dirs=/var/kafka/data
metadata.log.dir=/var/kafka/metadata

Bootstrap a new cluster:

kafka-storage.sh random-uuid
# returns cluster id, e.g. abc123...

kafka-storage.sh format -t abc123... -c server.properties
# initializes log dirs

kafka-server-start.sh server.properties

Operational Differences

Combined vs Isolated Mode

Combined (process.roles=broker,controller):

• Same node is both. Saves machines.
• Production-supported for small clusters (≤ 5 nodes).
• Failure of one node loses both broker capacity and controller voter.

Isolated (separate broker and controller nodes):

• Recommended for production at scale.
• Controllers are small (a few hundred MB RAM, low CPU) — cheap nodes.
• Brokers don't share resources with consensus traffic.
• Independent scaling.

Migration from ZK to KRaft

Kafka 3.4+ supports a documented migration:

1. Upgrade cluster to a KRaft-capable Kafka version.
2. Provision a KRaft controller quorum.
3. Set zookeeper.metadata.migration.enable=true on brokers and controllers.
4. Wait for migration to complete (metadata copied from ZK to KRaft log).
5. Roll brokers to remove ZK config.
6. Decommission ZK.

Non-trivial; test in non-prod first.

What Changed for Operators

• Admin CLIs now require --bootstrap-server <broker> not --zookeeper <zk>. Older scripts break.
• Monitoring: ZK-specific metrics gone. New KRaft metrics: kafka.controller:type=KafkaController,name=....
• Backup: ZK snapshots no longer relevant. Back up metadata.log.dir from controllers (or rely on Raft replication).
• DNS / connection strings: clients only ever needed broker addresses (ZK was admin-only).

KRaft Failure Modes

• Lose minority of controllers: cluster keeps operating; voter quorum still met.
• Lose majority of controllers: metadata operations halt (Raft can't make progress). Brokers keep serving cached metadata; producers/consumers continue until they need a metadata update.
• Disagreement (split brain): impossible — Raft ensures one leader at a time.

Common Mistakes

[!NOTE] KRaft is not a feature you "enable" — it's the new default mode of running Kafka. Treat ZK mode as legacy and plan migrations within the support window.

Interview Follow-ups

• "Why Raft over Paxos?" — Raft's understandability and existing tooling. Plus Kafka already had append-only log semantics — a natural fit.
• "What's the controller leader election?" — Raft leader election. One controller is the active leader; others are followers/voters.
• "Can KRaft be used standalone like ZK was?" — No, it's embedded in Kafka. Other systems that needed ZK (HBase, Solr) are not affected by Kafka's KRaft move.

Q: What are the different acks settings and how do they affect durability?

Answer:

The acks (acknowledgements) producer configuration controls how many brokers must confirm receipt of a message before the producer considers the write successful. It's the primary knob for trading off between durability and latency.

acks=0 (Fire and Forget)

The producer does not wait for any acknowledgement. It sends the message and immediately considers it delivered.

• Durability: None. Messages can be lost silently.
• Latency: Lowest possible.
• Use case: Metrics, logs, or any data where occasional loss is acceptable.

acks=1 (Leader Acknowledgement)

The producer waits for the leader replica to write the message to its local log and acknowledge. Followers may not have replicated it yet.

• Durability: Message is lost if the leader crashes before followers replicate.
• Latency: Low.
• Use case: General-purpose, acceptable for most non-critical workloads.

acks=all (or acks=-1) (Full ISR Acknowledgement)

The producer waits for all replicas in the ISR to acknowledge. This is the strongest durability guarantee.

• Durability: Message survives as long as at least one ISR replica survives.
• Latency: Highest (waiting for multiple replicas).
• Use case: Financial transactions, order processing, anything where data loss is unacceptable.

Visual Comparison

Producer → [Broker 0: Leader] → [Broker 1: Follower] → [Broker 2: Follower]

acks=0:  Producer sends, doesn't wait.        Risk: Total loss
acks=1:  Producer waits for Leader ACK.        Risk: Leader dies before replication
acks=all: Producer waits for ALL ISR ACKs.     Risk: Only if ALL replicas die

The min.insync.replicas Safety Net

acks=all alone has a subtle trap: if the ISR shrinks to just the leader (all followers are lagging), then acks=all effectively becomes acks=1.

The fix is combining it with min.insync.replicas:

acks=all
min.insync.replicas=2  # At least 2 replicas must ACK
replication.factor=3

If fewer than 2 replicas are in the ISR, the producer receives a NotEnoughReplicasException and the write is rejected — preventing the silent durability downgrade.

[!TIP] The gold standard production config is acks=all + min.insync.replicas=2 + replication.factor=3. This tolerates one broker failure while guaranteeing no data loss.

Q: What is an Idempotent Producer in Kafka?

Answer:

An idempotent producer guarantees that even if a message is sent multiple times (due to retries), it is written to the Kafka log exactly once per partition. This eliminates duplicate messages caused by network errors.

The Problem Without Idempotency

1. Producer sends message A to broker.
2. Broker writes message A and sends an ACK.
3. The ACK is lost due to a network glitch.
4. Producer thinks the write failed, so it retries message A.
5. Broker writes message A again → duplicate.

How Idempotency Works

When enabled, Kafka assigns each producer a unique Producer ID (PID) and each message gets a sequence number per partition.

The broker tracks the latest sequence number for each PID+partition pair. If a message arrives with a sequence number that has already been written, the broker silently discards the duplicate and returns a success ACK.

Producer (PID=42) → Partition 0:
  Msg(seq=0) → Written ✅
  Msg(seq=1) → Written ✅
  Msg(seq=1) → Duplicate, discarded! (but ACK sent) ✅
  Msg(seq=2) → Written ✅

Enabling Idempotency

# Producer config
enable.idempotence=true

# These are automatically set when idempotence is enabled:
acks=all
retries=Integer.MAX_VALUE
max.in.flight.requests.per.connection=5  # (was 1 in older versions)

[!NOTE] Since Kafka 3.0, enable.idempotence=true is the default. You don't need to explicitly set it in newer versions.

Scope and Limitations

Idempotency alone does NOT provide exactly-once semantics across multiple partitions or consumer-producer chains. For that, you need transactions (covered in the Reliability section).

[!TIP] In interviews, the key insight is: idempotency prevents duplicates from retries within a single producer session. It does NOT prevent duplicates from application-level retries (e.g., your service crashes and replays the same business logic). For that, you need application-level deduplication or Kafka transactions.

Q: How does Kafka decide which partition a message goes to?

Answer:

The partition assignment strategy determines message ordering and parallelism.

Partitioning Strategies

1. Key-Based Partitioning (Default when key is provided) When a message has a key, Kafka applies murmur2(key) % numPartitions to determine the partition. All messages with the same key always go to the same partition, guaranteeing ordering for that key.

producer.send(new ProducerRecord<>("orders", "user-123", orderEvent));
// All events for "user-123" go to the same partition → strict ordering

2. Round-Robin (Default when key is null, Kafka < 2.4) Messages without a key are distributed across partitions in a round-robin fashion.

3. Sticky Partitioning (Default when key is null, Kafka ≥ 2.4) Instead of round-robin per message, the producer "sticks" to one partition for the duration of a batch, then switches. This significantly improves batching efficiency and throughput.

Round-Robin:  P0, P1, P2, P0, P1, P2  (small batches, many network calls)
Sticky:       P0, P0, P0, P1, P1, P1  (full batches, fewer network calls)

4. Custom Partitioner You can implement your own partitioning logic:

public class GeoPartitioner implements Partitioner {
    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                         Object value, byte[] valueBytes, Cluster cluster) {
        String region = extractRegion(key);
        if ("us-east".equals(region)) return 0;
        if ("eu-west".equals(region)) return 1;
        return 2; // default
    }
}

The Repartitioning Trap

[!CAUTION] If you add partitions to an existing topic, the key-to-partition mapping changes (murmur2(key) % newNumPartitions). Messages for the same key may suddenly go to a different partition, breaking ordering guarantees for in-flight data. This is why you should plan partition counts carefully upfront.

Choosing the Right Strategy

Q: How do Batching and Compression work in Kafka producers?

Answer:

Batching and compression are Kafka's two most impactful performance optimizations on the producer side.

Batching

Instead of sending each message individually over the network, the producer accumulates messages into batches and sends them together. This dramatically reduces network overhead.

Key Configuration:

batch.size=16384          # Max batch size in bytes (16 KB default)
linger.ms=5               # Max time to wait for a batch to fill before sending

How it works:

1. Producer receives a send() call.
2. The message is added to a batch buffer for the target partition.
3. The batch is sent when either batch.size is reached or linger.ms expires — whichever comes first.

linger.ms=0 (default):  Send immediately, tiny batches, many network calls.
linger.ms=5:            Wait up to 5ms to fill the batch, fewer calls, higher throughput.
linger.ms=100:          Wait up to 100ms, maximum batching, added latency.

[!TIP] For high-throughput systems, set linger.ms=5-20 and batch.size=65536 (64KB) or higher. The small latency increase is usually negligible compared to the throughput gain.

Compression

Kafka supports compressing message batches before sending them over the network. This reduces:

• Network bandwidth (often 50-80% reduction)
• Disk storage on brokers (compressed data stays compressed on disk)

Producer Config:

compression.type=snappy   # Options: none, gzip, snappy, lz4, zstd

Comparison:

How They Work Together

Application: send(msg1), send(msg2), send(msg3), send(msg4)
                      │
        ┌─────────────▼──────────────┐
        │      Batch Accumulator     │
        │  [msg1, msg2, msg3, msg4]  │
        │   (wait for linger.ms or   │
        │    batch.size reached)     │
        └─────────────┬──────────────┘
                      │ compress batch
        ┌─────────────▼──────────────┐
        │   Compressed Batch         │
        │   (e.g., snappy: 60% smaller) │
        └─────────────┬──────────────┘
                      │ single network call
                      ▼
                   Broker

Broker-Side Compression

The broker stores batches in the same compressed format they were received. It does NOT decompress and recompress. This means compression set by the producer extends to both network transfer AND disk storage — a double win.

[!NOTE] If the broker's compression.type differs from the producer's, the broker will decompress and recompress, causing significant CPU overhead. It's best to let the producer control compression and set the broker to compression.type=producer (the default).

Q: How do Consumer Groups and Offsets work in Kafka?

Answer:

Consumer groups are Kafka's mechanism for parallel consumption and load balancing.

Consumer Groups

A consumer group is a set of consumers that cooperate to consume messages from a topic. Each partition is assigned to exactly one consumer within a group. This ensures each message is processed once per group.

Topic "orders" (4 partitions)
Consumer Group "payment-service" (3 consumers):

  Consumer A ← Partition 0, Partition 1
  Consumer B ← Partition 2
  Consumer C ← Partition 3

Key Rule: If the number of consumers exceeds the number of partitions, the extra consumers sit idle.

4 partitions, 6 consumers:
  Consumer A ← Partition 0
  Consumer B ← Partition 1
  Consumer C ← Partition 2
  Consumer D ← Partition 3
  Consumer E ← IDLE ❌
  Consumer F ← IDLE ❌

Multiple Consumer Groups

Different consumer groups consume the same topic independently. Each group maintains its own offset position — they don't interfere with each other.

Topic "orders" (3 partitions) →
  Consumer Group "payment-service"  → reads all 3 partitions independently
  Consumer Group "email-service"    → reads all 3 partitions independently
  Consumer Group "analytics"        → reads all 3 partitions independently

This is what makes Kafka a publish-subscribe system, not just a queue.

Offsets

An offset is a sequential integer that uniquely identifies each message within a partition. Offsets are how consumers track what they've already read.

Partition 0: [0] [1] [2] [3] [4] [5] [6] [7] [8]
                                  ↑
                          Consumer's current offset = 5
                          (has read 0-4, will read 5 next)

Where Are Offsets Stored?

Consumer offsets are committed to an internal Kafka topic called __consumer_offsets (50 partitions by default). This is a regular compacted topic managed by Kafka itself.

# Check committed offsets for a group
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
    --describe --group payment-service

Output:

GROUP             TOPIC    PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
payment-service   orders   0          1024            1030            6
payment-service   orders   1          987             987             0

[!IMPORTANT] The LAG column is critical for monitoring. It shows how many unprocessed messages are waiting. Increasing lag = consumer can't keep up = potential backpressure issue.

Q: What are the different Offset Commit Strategies?

Answer:

How and when a consumer commits its offset determines what happens when the consumer crashes and restarts. This directly affects delivery semantics.

Auto-Commit (Default)

Offsets are committed automatically at a fixed interval, regardless of whether messages have been processed.

enable.auto.commit=true        # Default
auto.commit.interval.ms=5000   # Every 5 seconds

The Problem:

1. Consumer fetches messages at offset 100-110.
2. Auto-commit fires, committing offset 110.
3. Consumer crashes while processing message 105.
4. Consumer restarts, reads from offset 110 → messages 105-109 are lost.

This creates at-most-once semantics: messages can be lost, but are never reprocessed.

Manual Commit (Synchronous)

The application explicitly commits after successfully processing messages.

consumer.poll(Duration.ofMillis(100));
// Process messages...
consumer.commitSync(); // Blocks until commit is confirmed

Trade-off: If the consumer crashes after processing but before committing, it will reprocess those messages on restart → at-least-once semantics (duplicates possible, but no data loss).

Manual Commit (Asynchronous)

Same as synchronous but non-blocking:

consumer.commitAsync((offsets, exception) -> {
    if (exception != null) {
        log.error("Commit failed", exception);
    }
});

Trade-off: Higher throughput, but if the commit fails silently, you may reprocess messages.

Best Practice: Sync + Async Hybrid

try {
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
        for (ConsumerRecord<String, String> record : records) {
            processRecord(record);
        }
        consumer.commitAsync(); // Fast, non-blocking for normal flow
    }
} catch (Exception e) {
    log.error("Consumer error", e);
} finally {
    consumer.commitSync(); // Guaranteed commit on shutdown
    consumer.close();
}

Commit Granularity

You can also commit offsets for specific partitions:

Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();
offsets.put(new TopicPartition("orders", 0), new OffsetAndMetadata(lastOffset + 1));
consumer.commitSync(offsets);

Summary

[!TIP] Most production systems use manual commits with at-least-once semantics and design their consumers to be idempotent (processing the same message twice produces the same result). This is simpler and more reliable than attempting exactly-once.

Q: What is Consumer Rebalancing and why can it be problematic?

Answer:

A rebalance is the process of redistributing partition assignments among consumers in a group. It's triggered when the group membership changes.

What Triggers a Rebalance?

1. A consumer joins the group (new instance deployed).
2. A consumer leaves the group (instance crashes or shuts down).
3. A consumer fails to send a heartbeat within session.timeout.ms.
4. A consumer's poll() calls take longer than max.poll.interval.ms.
5. Partitions are added to the subscribed topic.

Why Rebalancing is Problematic

During a rebalance, all consumers in the group stop processing. This causes a processing pause (sometimes called "stop the world") that can last from milliseconds to minutes depending on the group size.

Normal operation:
  Consumer A ← P0, P1  (processing)
  Consumer B ← P2, P3  (processing)

Consumer B crashes → REBALANCE triggered:
  All consumers STOP processing
  Coordinator reassigns partitions
  Consumer A ← P0, P1, P2, P3 (resumes)
  
Total pause: seconds to minutes

Rebalance Strategies

1. Eager Rebalancing (Default in older versions) All consumers give up ALL partition assignments, then get new ones. Maximum disruption.

2. Cooperative (Incremental) Rebalancing (Kafka ≥ 2.4) Only the partitions that need to move are revoked and reassigned. Other consumers continue processing without interruption.

partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

Consumer B crashes:
  Consumer A: keeps P0, P1 (no pause!) + receives P2, P3
  Only P2, P3 are "moved" — minimal disruption

Avoiding Unnecessary Rebalances

Tune these consumer configs:

# Time before a consumer is considered dead (default: 45s)
session.timeout.ms=45000

# Heartbeat interval (should be ~1/3 of session.timeout)
heartbeat.interval.ms=15000

# Max time between poll() calls before consumer is evicted
max.poll.interval.ms=300000   # 5 minutes

# Reduce records per poll if processing is slow
max.poll.records=500

[!CAUTION] The most common cause of unnecessary rebalances is slow message processing. If processing a batch of messages takes longer than max.poll.interval.ms, Kafka assumes the consumer is dead and triggers a rebalance — even though it's still alive and processing. Either speed up processing, reduce max.poll.records, or increase max.poll.interval.ms.

Static Group Membership (Kafka ≥ 2.3)

Assign a fixed identity to each consumer using group.instance.id. When a consumer restarts, it rejoins with the same identity and gets its previous partitions back — no rebalance triggered during brief restarts.

group.instance.id=consumer-host-1

Q: What is Cooperative Rebalancing and how does KIP-848 change consumer groups?

Answer:

Rebalancing is how a consumer group redistributes partitions when membership changes (consumer joins/leaves, topic gains partitions). For years, Kafka used eager rebalancing, which had a stop-the-world problem. Cooperative rebalancing fixes that, and KIP-848 rewrites the protocol entirely to push coordination off the clients.

Eager Rebalancing (the old default)

t0: consumers A, B, C own partitions [p0..p8]
t1: consumer D joins
t2: ALL consumers REVOKE all partitions  <-- stop-the-world
t3: leader computes new assignment
t4: ALL consumers receive new assignment, resume

Every rebalance — even adding a single consumer — forced every member to drop all partitions and reprocess from the last committed offset. For a group consuming 200 partitions, that meant a multi-second processing gap on every scale event.

Cooperative Rebalancing (KIP-429, default since 3.0)

Set via partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor.

t0: A=[p0,p1,p2], B=[p3,p4,p5], C=[p6,p7,p8]
t1: D joins
t2: Plan: D gets [p2, p5, p8]. A,B,C revoke ONLY those.
t3: A=[p0,p1], B=[p3,p4], C=[p6,p7]   <-- still consuming!
t4: D picks up [p2,p5,p8]

Key properties:

• Incremental: members keep partitions not being reassigned.
• Two-phase: first revoke just the moving partitions, then assign.
• Sticky: assignor tries to preserve previous ownership to minimize state warm-up (important for Kafka Streams).

KIP-848: The Next Consumer Rebalance Protocol

Available as preview in 3.7+, GA in 4.0. It moves group coordination from a client-side leader to the group coordinator broker.

Old protocol problems:

• Group "leader" is one of the consumers — it must download metadata and compute assignments. Slow with many partitions.
• Heartbeats, sync, and join are tangled — slow members stall the whole group.
• Static membership and cooperative are bolt-ons.

KIP-848 changes:

• Broker computes assignments using a server-side assignor.
• Consumers send a single ConsumerGroupHeartbeat RPC — no JoinGroup/SyncGroup dance.
• Reconciliation is per-member and asynchronous — slow consumers don't block fast ones.
• Rebalance time drops dramatically for large groups (hundreds of consumers, thousands of partitions).

Old:                                    New (KIP-848):
JoinGroup → SyncGroup → Heartbeat       Heartbeat (carries everything)
  (synchronous, leader-driven)            (async, broker-driven)

Comparison

Migration Caveats

You can't flip mid-flight from eager to cooperative without care. The supported path:

1. Roll consumers with partition.assignment.strategy=[CooperativeStickyAssignor, RangeAssignor] (cooperative and an old strategy).
2. Wait for entire group to converge.
3. Roll again with only CooperativeStickyAssignor.

[!NOTE] Kafka Streams uses its own assignor (StreamsPartitionAssignor) which has been cooperative for longer than the plain consumer. Same eventual destination, slightly different lineage.

Interview Follow-ups

• "What's the difference between sticky and cooperative?" — Sticky is about minimizing partition movement; cooperative is about not stopping the world. The default CooperativeStickyAssignor does both.
• "What is static membership?" — group.instance.id makes a consumer's identity survive restarts, avoiding a rebalance for transient outages (within session.timeout.ms).
• "How does KIP-848 affect client compatibility?" — Old clients keep using the classic protocol against the same coordinator; brokers support both.

Q: How do you tune a Kafka consumer for throughput, latency, and safety?

Answer:

Consumer tuning is mostly the interplay of four properties: how much you fetch per poll, how often you poll, how often you commit, and what your session/heartbeat timeouts look like. Misconfiguring any of them produces familiar failures: rebalance storms, duplicate processing, lag spikes, or silent data loss.

The Poll Loop

while (running) {
    var records = consumer.poll(Duration.ofMillis(500));
    for (var r : records) process(r);
    consumer.commitSync();
}

Three timers run during this loop:

• session.timeout.ms: heartbeat thread keeps group membership. If no heartbeat within this window, the broker considers the consumer dead.
• heartbeat.interval.ms: how often heartbeats are sent. Must be < session.timeout/3.
• max.poll.interval.ms: max wall time between two poll() calls. If processing takes longer, the broker kicks you out → rebalance.

Throughput Tuning

Bigger batches = fewer round trips, fewer commits, more throughput. Trade: higher end-to-end latency, bigger spike on rebalance (have to re-process the in-flight batch).

Latency Tuning

Latency and throughput pull in opposite directions. Pick the workload's priority.

max.poll.interval.ms — the Most Misunderstood Setting

Default: 5 minutes (Kafka 2.x+). If your processing for one batch takes longer, you're rebalanced.

Common scenarios:

• Heavy per-record work (image processing, ML inference) + big batches → exceed limit.
• External API call inside the loop with long timeout.
• GC pause / app stall.

Fixes (in order):

1. Reduce max.poll.records so each batch is smaller.
2. Increase max.poll.interval.ms (last resort — masks real problems).
3. Move heavy work to a thread pool, but careful: you still must keep polling to maintain group membership.

The Background-Processing Pattern

If you must process slowly:

while (running) {
    var records = consumer.poll(Duration.ofMillis(100));
    for (var r : records) workerPool.submit(() -> process(r));
    // Don't commit yet — wait for workers to finish.
}

Caveats:

• Lose ordering across records in a partition.
• Must track which offsets are safe to commit (only commit offsets whose work is fully done).
• Must pause() the partition while workers are saturated (unsubscribe() would lose membership).

Better: increase partition count and run more consumers — keep the simple poll loop.

Offset Commit Strategies

enable.auto.commit: false           # production default
auto.commit.interval.ms: 5000       # only used if auto-commit is true

Manual sync commit (safest):

consumer.commitSync();

Blocks until brokers ack. Adds latency but you know offsets are durably committed.

Manual async commit (fast):

consumer.commitAsync((offsets, ex) -> {
    if (ex != null) log.warn("commit failed", ex);
});

Fire-and-forget. On shutdown, do a final commitSync() to ensure latest offsets stick.

Auto commit (don't use in production):

enable.auto.commit=true

Offsets advance every 5 seconds regardless of whether processing succeeded. Silent data loss / duplicate processing on crash.

Rebalance Behavior

When a consumer joins/leaves, partitions are reassigned. Use a callback to flush state and commit:

consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
    public void onPartitionsRevoked(Collection<TopicPartition> tp) {
        consumer.commitSync();             // flush before yielding
    }
    public void onPartitionsAssigned(Collection<TopicPartition> tp) {
        // optionally seek
    }
});

With partition.assignment.strategy=CooperativeStickyAssignor (default Kafka 3+), revocation is incremental — only re-assigned partitions are revoked, not all of them. Massively reduces rebalance pain.

Static Membership

group.instance.id: ${HOSTNAME}
session.timeout.ms: 60000

Set a persistent instance ID and the broker keeps your slot reserved for session.timeout.ms after you disappear. Restart within that window = no rebalance.

Use for stateful consumers (Kafka Streams especially).

Consumer Lag

Lag = log end offset - committed offset per partition. The single most important metric.

kafka-consumer-groups.sh --bootstrap-server b1:9092 \
  --group orders-app --describe
# TOPIC      PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# orders     0          12345           12345           0
# orders     1          12000           12500           500

Alert on:

• lag increasing trend (consumer can't keep up).
• lag > N absolute threshold per business SLA.
• Lag concentrated on one partition (hot partition).

Per-consumer JMX:

kafka.consumer:type=consumer-fetch-manager-metrics,
  client-id=*,name=records-lag-max

auto.offset.reset for New Groups

• latest (default): start at end of topic. Skip historical data.
• earliest: start from beginning. Reprocess everything.
• none: throw if no committed offset.

This only applies to new consumer groups or expired offsets. Don't expect it to matter on a running app.

Diagnostic Checklist for Lag Spikes

1. Is one partition lagging more than others? → hot key, look upstream.
2. Is processing CPU-bound? → profile, scale consumers (up to partition count).
3. Is downstream (DB, API) slow? → it's downstream, not Kafka.
4. Is GC pause > heartbeat? → tune G1/ZGC, increase heap.
5. Is max.poll.interval.ms getting exceeded? → batch too big.

Common Mistakes

commitSync semantics: commit offset = "next offset to read". Use record.offset() + 1.

[!NOTE] The single most useful guideline: poll fast, process bounded, commit explicitly. Almost every "Kafka is unreliable" story traces back to violating one of these three.

Interview Follow-ups

• "Why does my consumer lag at the same time every day?" — Compaction or retention deletes / off-peak production lulls / scheduled batch jobs upstream. Look at producer rate per partition.
• "commitSync vs commitAsync — which?" — Async in the hot path; sync on shutdown and inside rebalance callback. Together they minimize duplication.
• "What is fetch.max.bytes vs max.partition.fetch.bytes?" — First caps total fetch response. Second caps per-partition contribution. Keep total ≥ partitions × per-partition.

Q: What are the different delivery semantics in Kafka?

Answer:

This is one of the most important Kafka interview questions. There are three delivery guarantees, and understanding the trade-offs is essential.

1. At-Most-Once

Messages may be lost but are never reprocessed. The consumer commits the offset before processing the message.

1. Fetch message at offset 42
2. Commit offset 43 ✅
3. Process message... CRASH 💥
4. Restart → reads from offset 43 → message 42 is LOST

When to use: Metric collection, logging — where occasional loss is acceptable and speed matters most.

2. At-Least-Once

Messages are never lost but may be duplicated. The consumer commits the offset after processing the message.

1. Fetch message at offset 42
2. Process message ✅
3. Commit offset 43... CRASH 💥 (commit failed)
4. Restart → reads from offset 42 → message 42 is PROCESSED AGAIN

When to use: Most production systems. Design consumers to be idempotent (safe to process twice).

Idempotent consumer pattern:

void processOrder(OrderEvent event) {
    // Check if already processed using a deduplication store
    if (processedIds.contains(event.getId())) {
        return; // Skip duplicate
    }
    executeBusinessLogic(event);
    processedIds.add(event.getId());
}

3. Exactly-Once Semantics (EOS)

Messages are processed exactly once — no loss, no duplicates. This is the hardest to achieve and requires specific Kafka features.

How Kafka achieves EOS:

1. Idempotent Producer (enable.idempotence=true) — prevents duplicate writes.
2. Transactions (transactional.id) — atomic writes across multiple partitions.
3. Consumer read_committed isolation — consumers only see committed transactional messages.

# Producer
enable.idempotence=true
transactional.id=my-transaction-id

# Consumer
isolation.level=read_committed

When to use: Financial systems, inventory management, or when consuming from one topic, processing, and producing to another topic atomically (the "consume-transform-produce" pattern).

Summary

[!IMPORTANT] Exactly-once in Kafka is scoped to the Kafka ecosystem (producer → broker → consumer within Kafka). It does NOT guarantee exactly-once when writing to external systems (like a database). For end-to-end exactly-once with external systems, you need idempotent consumers or two-phase commit patterns.

Q: How do Kafka Transactions work?

Answer:

Kafka transactions enable atomic writes across multiple partitions and topics. They are the foundation for exactly-once semantics (EOS) in the "consume-transform-produce" pattern.

The Problem

Imagine a stream processing pipeline that reads from topic A, transforms the data, and writes to topic B while also committing consumer offsets. Without transactions, a crash mid-pipeline could result in:

• Data written to topic B but offset not committed → duplicates on retry
• Offset committed but data not written to topic B → data loss

How Transactions Work

// 1. Configure transactional producer
Properties props = new Properties();
props.put("transactional.id", "order-processor-1");
props.put("enable.idempotence", "true");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

// 2. Initialize transactions (called once)
producer.initTransactions();

try {
    // 3. Begin transaction
    producer.beginTransaction();

    // 4. Consume from input topic
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));

    for (ConsumerRecord<String, String> record : records) {
        // 5. Process and produce to output topic
        String result = transform(record.value());
        producer.send(new ProducerRecord<>("output-topic", record.key(), result));
    }

    // 6. Commit consumer offsets AS PART OF the transaction
    producer.sendOffsetsToTransaction(
        getOffsetsToCommit(records),
        consumer.groupMetadata()
    );

    // 7. Commit transaction (atomic: either ALL writes + offset commit succeed, or NONE)
    producer.commitTransaction();

} catch (Exception e) {
    // 8. Abort transaction (all writes are rolled back)
    producer.abortTransaction();
}

What Happens Atomically

When commitTransaction() succeeds, ALL of the following are committed together:

1. All send() messages to output topics.
2. The consumer offset commit.

If anything fails, abortTransaction() rolls back everything — the output messages are marked as "aborted" and the consumer offsets are not updated.

Consumer Side: read_committed

For consumers to properly participate in transactions:

isolation.level=read_committed

The transactional.id

• Must be unique per producer instance but stable across restarts.
• When a producer with the same transactional.id restarts, Kafka "fences" the old producer — any pending transactions from the old instance are aborted.
• This prevents "zombie" producers from causing duplicates.

[!TIP] Kafka Streams uses transactions internally to provide exactly-once semantics out of the box. You just set processing.guarantee=exactly_once_v2 and the framework handles all the transactional plumbing automatically.

Q: How do you implement Dead Letter Queues (DLQ) and retry topics in Kafka?

Answer:

Unlike RabbitMQ or SQS, Kafka has no native DLQ primitive. A DLQ in Kafka is just another topic you publish to when processing fails — but the patterns around routing, retries, and replay are what make or break production systems.

The Core Problem

A consumer pulls a poison message — bad schema, downstream 500, programmer error. Options:

• Block forever: retry in place. Partition stops moving. Lag grows. Other partitions keep going. Eventually pages someone.
• Skip and commit: data loss.
• Park it: write to a DLQ topic, commit offset, keep moving.

DLQ = "park it, deal with it later."

Topology

              ┌─────────────────────────┐
              │       orders            │
              └──────────┬──────────────┘
                         │
                  ┌──────▼───────┐
                  │  Consumer    │
                  └──┬─────────┬─┘
                     │ ok      │ fail
                     │         │
                ┌────▼───┐  ┌──▼──────────────┐
                │ commit │  │ orders.retry.5s │──┐
                └────────┘  └─────────────────┘  │
                                                 │ still fails
                            ┌─────────────────┐  │
                            │ orders.retry.30s│◀─┘
                            └────────┬────────┘
                                     │ still fails
                            ┌────────▼────────┐
                            │   orders.dlq    │
                            └─────────────────┘

Tiered retry topics let you back off without blocking the main topic.

Pattern 1: Plain DLQ (no retry)

try {
    process(record);
} catch (RetryableException e) {
    // also goes to DLQ in this simple variant
    producer.send(new ProducerRecord<>("orders.dlq", record.key(), record.value()));
} catch (NonRetryableException e) {
    producer.send(new ProducerRecord<>("orders.dlq", record.key(), record.value()));
}
consumer.commitSync();

Always attach metadata in headers — why it failed, which consumer, original offset/partition, attempt count.

record.headers().add("error-class", e.getClass().getName().getBytes());
record.headers().add("error-message", e.getMessage().getBytes());
record.headers().add("original-topic", "orders".getBytes());
record.headers().add("attempt", String.valueOf(attempt).getBytes());

Pattern 2: Non-blocking Tiered Retry (Confluent / Spring style)

Spring Kafka's @RetryableTopic or Confluent's parallel-consumer auto-create topics like:

orders
orders-retry-0    (delay 5s)
orders-retry-1    (delay 30s)
orders-retry-2    (delay 5m)
orders-dlt        (terminal)

Each retry topic has a consumer that sleeps until record.timestamp + delay, then re-attempts. Lower partitions don't get blocked by one bad record.

@RetryableTopic(
    attempts = "4",
    backoff = @Backoff(delay = 5000, multiplier = 6.0),
    dltTopicSuffix = "-dlt",
    autoCreateTopics = "true"
)
@KafkaListener(topics = "orders")
public void consume(Order o) { ... }

@DltHandler
public void dlt(Order o, @Header(KafkaHeaders.EXCEPTION_MESSAGE) String err) { ... }

Pattern 3: Kafka Connect Sink DLQ

Built in. Set:

errors.tolerance=all
errors.deadletterqueue.topic.name=connect.dlq
errors.deadletterqueue.context.headers.enable=true

Connect routes failed records with full error context in headers.

Common Mistakes

Replay Strategy

Don't auto-replay. A typical replay pipeline:

1. Engineer reads DLQ headers, identifies root cause.
2. Fix consumer code, deploy.
3. Run a replay job that reads orders.dlq and re-produces matching records to orders.
4. Move the replayed records to orders.dlq.replayed.<date> for audit.

[!NOTE] A DLQ that nobody monitors is a data leak with extra steps. Treat DLQ depth as a first-class SLI.

Interview Follow-ups

• "Why not just pause() the partition?" — Works for transient failures, but lag balloons and the consumer becomes a state machine you have to babysit. DLQ trades latency for liveness.
• "How to preserve ordering with DLQ?" — You can't perfectly. Tiered retries reorder by design. If strict ordering matters, block (pause) and page; don't use DLQ.
• "What about exactly-once?" — Wrap produce-to-DLQ + commit-offset in a transaction (isolation.level=read_committed on downstream).

Q: Why is Kafka so fast?

Answer:

Kafka achieves extraordinary throughput (millions of messages/second) through several deliberate architectural decisions.

1. Sequential I/O (Append-Only Log)

Kafka writes messages to disk in a strictly sequential, append-only fashion. It never does random disk seeks.

Sequential disk writes are shockingly fast — often 600 MB/s+ on modern SSDs, compared to ~100 KB/s for random writes. This is because the OS can fully leverage disk write-ahead buffers and avoid head movement on HDDs.

2. Zero-Copy (sendfile)

When a consumer reads data, the normal path involves 4 copies:

Disk → Kernel Buffer → User Space → Socket Buffer → NIC

Kafka uses the Linux sendfile() system call to skip user space entirely:

Disk → Kernel Buffer → NIC  (zero-copy, 2 copies instead of 4)

This eliminates context switches and memory copies, reducing CPU usage and increasing throughput dramatically.

3. Page Cache (OS-Level Caching)

Kafka does NOT manage its own in-memory cache. Instead, it relies on the OS page cache. When data is written to disk, the OS caches it in RAM. When consumers read recent data, it's served directly from the page cache — essentially a free, automatically managed in-memory read cache.

Hot data (recent):  Served from OS page cache (RAM speed)
Cold data (old):    Read from disk (still fast due to sequential reads)

This is why Kafka's performance is often counter-intuitive: it writes to "disk" but reads from "memory."

4. Batching + Compression

Producers batch many messages together and optionally compress them. This means:

• Fewer network round trips
• Less disk I/O (one write for many messages)
• Smaller on-disk footprint

5. Partitioning (Horizontal Scaling)

Each partition is an independent log. Multiple partitions can be read/written in parallel across different brokers and consumers. Adding partitions and brokers scales throughput linearly.

6. No Per-Message Acknowledgment to Consumers

Unlike RabbitMQ (which tracks ACK per message), Kafka consumers simply track their offset position. There's no per-message bookkeeping on the broker side, which eliminates enormous overhead.

Summary

[!TIP] In interviews, the two killer points are sequential I/O and zero-copy. These are what fundamentally differentiate Kafka's performance from traditional message brokers that rely on random I/O and per-message routing.

Q: What is Consumer Lag and how do you handle backpressure?

Answer:

Consumer lag is the difference between the latest message offset in a partition (log-end offset) and the consumer's current committed offset. It tells you how far behind a consumer is.

Measuring Lag

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
    --describe --group my-service

GROUP       TOPIC    PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
my-service  orders   0          4500            5000            500
my-service  orders   1          3200            3200            0
my-service  orders   2          2800            4100            1300

Healthy: LAG = 0 or near-zero and stable. Unhealthy: LAG is increasing over time = consumer can't keep up with producers.

Causes of Growing Lag

1. Slow processing — each message takes too long (external API calls, heavy computation).
2. Insufficient consumers — fewer consumers than partitions.
3. Frequent rebalances — consumers pausing during rebalancing.
4. GC pauses — long garbage collection stops in JVM-based consumers.
5. Skewed partitions — one partition has significantly more data due to hot keys.

Strategies to Handle Backpressure

1. Scale consumers horizontally Add more consumer instances (up to the number of partitions):

Before: 2 consumers for 6 partitions (3 partitions each)
After:  6 consumers for 6 partitions (1 partition each)

2. Increase partitions More partitions = more parallelism. But be cautious of the repartitioning trap (breaks key ordering for existing data).

3. Optimize processing

• Process messages asynchronously (decouple consumption from processing).
• Use batch processing instead of one-at-a-time.
• Cache external service responses.

4. Tune consumer configs

max.poll.records=100        # Fewer records per poll = less time per batch
fetch.min.bytes=1           # Don't wait for large fetches
max.poll.interval.ms=600000 # More time allowed between polls

5. Dead Letter Queue (DLQ) If a specific message consistently fails processing, send it to a DLQ topic instead of blocking the consumer:

try {
    processMessage(record);
} catch (Exception e) {
    producer.send(new ProducerRecord<>("orders.dlq", record.key(), record.value()));
    // Continue processing next message
}

Monitoring Lag

Critical metrics to alert on:

• kafka.consumer.lag — absolute lag (messages behind).
• kafka.consumer.lag_rate — rate of lag change (is it growing?).
• Consumer group state — STABLE, REBALANCING, DEAD.

Tools: Burrow (LinkedIn), Kafka Lag Exporter (Prometheus), or built-in kafka-consumer-groups.sh.

[!CAUTION] A sudden spike in consumer lag often precedes a production incident. Set up alerts for when lag exceeds a threshold (e.g., >10,000 messages) OR when lag is consistently increasing over a 5-minute window.

Q: What are hot partitions, how do you detect them, and how do you fix them?

Answer:

A hot partition is one that receives or serves a disproportionate share of traffic. Because a single partition is consumed by exactly one consumer in a group, a hot partition becomes the parallelism ceiling of your entire pipeline — adding consumers won't help.

Why Hot Partitions Happen

Default Kafka partitioner: hash(key) % numPartitions. Three failure modes:

1. Skewed key distribution. One tenant, one device ID, one product ID generates 10x traffic. Hash doesn't help — every record from that key lands on the same partition.
2. Low-cardinality keys. Keying by country_code with 200 countries and 50 partitions means just a handful of partitions carry most traffic.
3. Bad partitioner. Custom partitioner that doesn't spread well, or routes based on a poor predicate (e.g., timestamp bucket).

Symptoms

• Consumer lag is high on a few partitions, near zero on others.
• Broker disk I/O / network is unbalanced across leaders.
• p99 end-to-end latency is bad even though average is fine.
• Adding consumers doesn't reduce lag.

Metrics to watch:

kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=*    # per-partition variant
kafka.consumer:type=consumer-fetch-manager-metrics,records-lag-max

A "tail/median ratio" of bytes-in per partition > 3 is a smell. > 10 is on fire.

Detection (CLI)

Per-partition message rate via kafka-run-class kafka.tools.GetOffsetShell:

kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list b1:9092 --topic orders --time -1
# orders:0:1234567
# orders:1:1234890
# orders:2:9876543   <-- hot

Sample twice with a delta to get rate. Or pull JMX MessagesInPerSec tagged by partition.

Remediation Patterns

1. Increase partition count. Only helps if key distribution is already good. Won't help with a single dominant key.

kafka-topics.sh --alter --topic orders --partitions 64

Caveat: only new keys benefit (old keys hash to old partitions if you didn't reshuffle); ordering guarantees per key are preserved across the change only because hash is stable on key.

2. Key salting (for a hot tenant). Append a salt to the key for the hot tenant only, then aggregate downstream:

String key = isHot(tenantId) ? tenantId + "#" + (counter++ % 8) : tenantId;

This splits one logical key across 8 partitions. Downstream must handle out-of-order or use a windowed aggregator.

3. Custom partitioner with two-level routing.

public int partition(String topic, Object key, byte[] keyBytes, ...) {
    if (isHotKey(key)) {
        return ThreadLocalRandom.current().nextInt(hotPartitions);
    }
    return Math.abs(Utils.murmur2(keyBytes)) % (totalPartitions - hotPartitions) + hotPartitions;
}

Reserve a partition range for hot keys, round-robin within it; cold keys hash into the rest.

4. Sticky Partitioner / KIP-794 Uniform Sticky. For null-key records, the default since 2.4 batches many records to one partition until a batch closes, then picks another. This trades per-record fairness for throughput. If your data has no key, just rely on this — don't write a custom partitioner.

5. Re-key upstream. If customer_id is hot but (customer_id, order_id) would be uniform, re-key in a stream processor:

orders --[KStream]--> rekey to order_id --> orders.rekeyed

Tradeoff Summary

[!NOTE] Ordering is the price you pay to fix hot partitions. Decide first whether per-key order matters. For analytics, usually not. For ledgers, almost always.

Interview Follow-ups

• "Can you add partitions without losing ordering?" — For future records you cannot guarantee old-key→old-partition. The hash mapping changes. If ordering matters, drain consumers, copy to a new topic with the new partition count via MirrorMaker, switch over.
• "Why is one consumer at 100% CPU while others idle?" — Almost always a hot partition.
• "How does this interact with exactly-once?" — Transactions don't help with skew; they make it worse (transaction coordinator on the producing partition's broker can become a bottleneck).

Q: What is Kafka Connect?

Answer:

Kafka Connect is a framework for reliably streaming data between Kafka and external systems (databases, search indexes, filesystems, cloud services) without writing any code.

How It Works

Kafka Connect runs as a separate, scalable cluster of worker processes. You configure data pipelines using JSON configurations — no Java code required.

                    Kafka Connect
External Source ──▶ [Source Connector] ──▶ Kafka Topic
Kafka Topic    ──▶ [Sink Connector]   ──▶ External Sink

Source Connectors

Read data from an external system and write it to Kafka topics.

{
    "name": "postgres-source",
    "config": {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "database.hostname": "db.example.com",
        "database.port": "5432",
        "database.dbname": "orders_db",
        "topic.prefix": "cdc"
    }
}

This captures every INSERT/UPDATE/DELETE from Postgres and streams it to topics like cdc.public.orders, cdc.public.users.

Sink Connectors

Read data from Kafka topics and write it to an external system.

{
    "name": "elasticsearch-sink",
    "config": {
        "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
        "topics": "orders",
        "connection.url": "http://es.example.com:9200",
        "type.name": "_doc"
    }
}

Popular Connectors

Standalone vs Distributed Mode

In distributed mode, if a worker dies, its connectors are automatically reassigned to surviving workers.

Why Not Just Write a Custom Producer/Consumer?

• Built-in offset tracking — Connect tracks source positions automatically.
• Fault tolerance — automatic failover in distributed mode.
• Schema evolution — integrates with Schema Registry.
• Configurable transforms — Single Message Transforms (SMTs) for lightweight data manipulation.
• No code to maintain — just JSON config.

[!TIP] In interviews, Debezium + Kafka Connect for CDC is a particularly strong topic. It's the industry standard for streaming database changes (e.g., syncing a PostgreSQL write-model to an Elasticsearch read-model in real-time).

Q: What is the difference between Kafka Streams, Apache Flink, and Apache Spark Streaming?

Answer:

All three are stream processing frameworks, but they serve different niches.

Kafka Streams

A lightweight client library (not a cluster/framework) for building stream processing applications that read from and write to Kafka.

StreamsBuilder builder = new StreamsBuilder();
builder.stream("input-topic")
    .filter((key, value) -> value.contains("important"))
    .mapValues(value -> value.toUpperCase())
    .to("output-topic");

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

Key characteristics:

• Runs as a regular Java application — no separate cluster to deploy.
• Exactly-once semantics built in.
• Supports stateful operations (aggregations, joins, windowing) with local state stores (RocksDB).
• Scales by simply running more instances of the application.

Apache Flink

A distributed stream processing framework designed for complex, low-latency event processing at massive scale.

Key characteristics:

• Runs on its own cluster (JobManager + TaskManagers).
• True event-time processing with watermarks.
• Advanced windowing (tumbling, sliding, session windows).
• Exactly-once semantics via checkpointing.
• Can process both streams and batch data (unified model).
• Supports multiple languages (Java, Scala, Python, SQL).

Apache Spark Streaming (Structured Streaming)

A micro-batch streaming engine built on top of Spark. It processes data in small batches rather than true record-at-a-time streaming.

Key characteristics:

• Runs on a Spark cluster.
• Processes streams as a series of small batch jobs.
• Shares Spark's batch processing ecosystem (MLlib, SQL, DataFrames).
• Higher latency than Flink (seconds vs milliseconds).

Comparison

When to Use What?

• Kafka Streams: Your data is in Kafka and goes back to Kafka. You want simplicity and don't want to manage a separate cluster.
• Flink: You need sub-millisecond latency, complex event processing (CEP), or reading from non-Kafka sources.
• Spark Streaming: Your team already uses Spark for batch and wants to add streaming. Latency of seconds is acceptable.

[!NOTE] Apache Flink is increasingly becoming the industry standard for large-scale stream processing. Many companies are migrating from Spark Streaming to Flink for its true streaming model and lower latency.

Q: What is Schema Registry and why is Avro commonly used with Kafka?

Answer:

The Problem: Schema Evolution

In a microservices architecture, producers and consumers are developed by different teams and deployed at different times. What happens when the producer changes the message format (adds a field, renames one, changes a type)?

Without schema management, the consumer breaks because it can't deserialize the new format.

Schema Registry

The Confluent Schema Registry is a centralized service that stores and manages schemas for Kafka message keys and values. It ensures that producers and consumers agree on the data format.

Producer → Schema Registry: "Here's my schema, give me an ID"
           Schema Registry → "Schema ID: 42"
Producer → Kafka: [Schema ID: 42] + [Serialized Data]
                       ...
Consumer ← Kafka: [Schema ID: 42] + [Serialized Data]
Consumer → Schema Registry: "What schema is ID 42?"
           Schema Registry → Returns the schema
Consumer: Deserializes data using the schema

Why Avro?

Apache Avro is a binary serialization format that is the dominant choice for Kafka messages. It pairs perfectly with Schema Registry.

Avro schema example:

{
    "type": "record",
    "name": "OrderEvent",
    "namespace": "com.example",
    "fields": [
        {"name": "orderId", "type": "string"},
        {"name": "amount", "type": "double"},
        {"name": "currency", "type": "string", "default": "USD"},
        {"name": "timestamp", "type": "long"}
    ]
}

Why not JSON?

Avro messages are typically 50-70% smaller than JSON because they don't include field names — only the values, referenced by schema position.

Compatibility Modes

Schema Registry enforces compatibility rules when a schema evolves:

Example of backward-compatible change:

// v1
{"name": "orderId", "type": "string"}
{"name": "amount", "type": "double"}

// v2 (backward compatible: new field has a default)
{"name": "orderId", "type": "string"}
{"name": "amount", "type": "double"}
{"name": "currency", "type": "string", "default": "USD"}  // ← NEW

[!TIP] In interviews, mentioning Avro + Schema Registry together shows you understand production Kafka. The key insight: Schema Registry acts as a contract between services, preventing breaking changes from deploying to production.

Q: How do you evolve schemas safely with Schema Registry?

Answer:

Schema Registry stores Avro/Protobuf/JSON schemas keyed by subject (usually <topic>-value). Compatibility rules determine which schema changes are allowed. Picking the wrong compatibility mode is how you ship a deserialization-error incident.

The Five Compatibility Modes

The default BACKWARD is the right choice for most consumer-led rollouts. BACKWARD_TRANSITIVE is safer but stricter.

What Each Mode Actually Allows

BACKWARD (consumer with new schema reads data written with old schema):

• ✅ Add an optional field (with default).
• ✅ Remove an optional field.
• ❌ Add a required field (consumer can't read old data missing it).
• ❌ Rename a field (different name = different field).
• ❌ Change a field's type incompatibly.

FORWARD (consumer with old schema reads data written with new schema):

• ✅ Add a required field (old consumer just ignores it).
• ✅ Remove an optional field with default.
• ❌ Remove a required field (old consumer fails to find it).

FULL = intersection of both. The strictest practical rule.

Worked Examples

Initial schema:

{
  "type": "record", "name": "Order", "fields": [
    {"name": "id",     "type": "string"},
    {"name": "amount", "type": "double"}
  ]
}

BACKWARD: add an optional field

{
  "type": "record", "name": "Order", "fields": [
    {"name": "id",     "type": "string"},
    {"name": "amount", "type": "double"},
    {"name": "tax",    "type": ["null", "double"], "default": null}
  ]
}

✅ Old records (no tax) → new consumer reads tax=null. ✅ Schema Registry accepts.

Failure: required field with no default

{"name": "tax", "type": "double"}    // no default

❌ Backward-incompatible. Old data has no tax field; new consumer can't supply one.

Schema Registry rejects with 409 Conflict at registration time.

Subject Naming Strategies

• TopicNameStrategy (default): <topic>-key, <topic>-value. One schema per topic.
• RecordNameStrategy: subject = fully-qualified record name. Lets one topic carry multiple record types.
• TopicRecordNameStrategy: <topic>-<recordName>. Hybrid.

Use RecordNameStrategy when you want event-typed topics (e.g., OrderCreated, OrderShipped on the same orders topic).

The Magic Byte Wire Format

Avro/Proto-over-Kafka uses Confluent's wire format:

| 0x00 | schema_id (4 bytes, big-endian) | payload bytes |
   ^         ^                                ^
   magic     schema fetched from Registry     Avro/Proto bytes

Consumer sees the schema ID, fetches the writer's schema from Registry (cached), then deserializes against its own reader's schema. No schema travels with each message — that's why Registry is mandatory.

Schema Registry Operations

# Register a schema
curl -X POST -H "Content-Type: application/json" \
  --data '{"schema": "{\"type\": \"record\", ...}"}' \
  http://registry:8081/subjects/orders-value/versions

# Get latest schema
curl http://registry:8081/subjects/orders-value/versions/latest

# Set compatibility mode for a subject
curl -X PUT -H "Content-Type: application/json" \
  --data '{"compatibility": "BACKWARD_TRANSITIVE"}' \
  http://registry:8081/config/orders-value

CI Gates

The single most valuable thing you can do: register schemas as a CI step before merge.

# Maven plugin
mvn schema-registry:test-compatibility \
  -Dschema.registry.url=https://registry \
  -DsubjectNamingStrategy=TopicNameStrategy

If the proposed schema is incompatible, the build fails. Production never sees the bad schema.

Removing or Renaming a Field

You don't. You deprecate:

1. Make the field optional with a default (BACKWARD-compatible change).
2. Update consumers to stop relying on it.
3. Update producers to stop writing it.
4. Leave the field in the schema indefinitely (or remove on a coordinated "version 2" topic migration).

To rename, add the new name as an optional field, dual-write, switch readers, retire old.

Avro aliases let you rename without breaking readers:

{"name": "amount", "type": "double", "aliases": ["price"]}

Reader looking for price finds amount. Limited usefulness; not all serializers honor aliases.

Reference Schemas

Avoid duplicating common types. Register a Money schema, reference it:

{
  "type": "record", "name": "Order",
  "fields": [
    {"name": "total", "type": "com.example.Money"}
  ]
}

Registered with references = [{"name": "com.example.Money", "subject": "money-value", "version": 1}].

Protobuf vs Avro vs JSON Schema

Avro is the historical Kafka default; Protobuf has caught up; JSON Schema is convenient but bulky.

Operational Concerns

• Registry HA: run ≥ 2 instances. Schema IDs are global; ID allocation is leader-only.
• Cache schemas client-side: client.cache.capacity (default 1000). A single schema fetch per ID per consumer lifetime is cheap.
• Schema deletion: soft-delete removes from API but keeps the ID alive (consumers can still deserialize). Hard-delete loses the ability to deserialize old data — almost never the right call.

Common Mistakes

[!NOTE] Avro/Proto defaults rule: every new field should have a default. Treat removal as "make optional, leave forever." Treat rename as "add new, deprecate old."

Interview Follow-ups

• "What happens if Schema Registry is down?" — Producers can use cached schemas; new schemas can't register. Consumers can deserialize cached IDs but new ones fail. Multi-AZ HA is essential.
• "How do you delete a topic that uses a schema?" — Delete topic in Kafka, optionally delete subject in Registry. ID assignments are still tracked.
• "Can two producers race-register the same schema?" — Registry deduplicates by content hash — same schema = same ID, no race.

Q: What are the core Kafka Streams patterns — KStream, KTable, joins, windowing?

Answer:

Kafka Streams is a library (not a cluster) for building stateful event-processing apps. The hard part is the conceptual model: stream vs table duality, time, joins, and how state is stored and recovered.

Stream vs Table Duality

KStream<K, V>     KTable<K, V>
record-by-record  changelog of state per key
"events"          "current value per key"

• KStream: every record is independent. Appending logs.
• KTable: per-key, latest value wins. Conceptually a materialized view of a compacted topic.

Same topic can be read as either:

KStream<String, Order> orders = builder.stream("orders");
KTable<String, Order>  latestOrders = builder.table("orders");

Convert:

KTable<String, Long> counts = stream
    .groupByKey()
    .count();             // KStream → KTable via aggregation

KStream<String, Order> changes = latestOrders.toStream();   // KTable → KStream of changes

Topology

builder.stream("input")
       .filter((k, v) -> v.amount > 0)
       .mapValues(v -> enrich(v))
       .to("output");

Every operator is a node in a topology DAG. The runtime executes the topology across N stream threads in your app instance, with state stored in local RocksDB.

State Stores

Aggregations and joins need state. Streams keeps state in RocksDB on local disk and backs it up to a changelog topic in Kafka.

KTable<String, Long> counts = stream.groupByKey().count(
    Materialized.<String, Long, KeyValueStore<Bytes, byte[]>>as("counts-store"));

On crash, a new instance restores state by reading the changelog from offset 0 (with the help of standby replicas if configured).

Joins

Three join modes:

1. KStream-KStream join (windowed).

KStream<String, OrderShipped> joined = orders.join(
    shipments,
    (o, s) -> new OrderShipped(o, s),
    JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5))
);

Records within 5 minutes of each other (by event time) are joined. Without a window, the join couldn't terminate.

2. KStream-KTable join (lookup).

KStream<String, EnrichedOrder> enriched = orders.join(
    customers,                                    // KTable
    (order, customer) -> enrich(order, customer)
);

For each order, look up the current customer record. No window. Updates to the table see future orders enriched with new values.

3. KTable-KTable join (relational).

KTable<String, View> view = users.join(profiles, (u, p) -> new View(u, p));

Like a SQL inner join. Changes on either side update the result.

Left, outer, and inner variants exist for each.

Co-Partitioning Requirement

KStream-KStream and KStream-KTable joins require co-partitioning:

• Same number of partitions.
• Same key.
• Same partitioner.

If the input topics aren't co-partitioned, you'll see runtime exceptions. Fix: repartition() (creates an internal repartition topic with the right key/partition count).

orders.selectKey((k, v) -> v.customerId)
      .repartition(Repartitioned.as("orders-by-customer").withNumberOfPartitions(12))
      .join(customers, ...);

Time Semantics

Kafka Streams distinguishes:

• Event time: timestamp inside the record (when the event happened).
• Processing time: when the app sees the record.
• Ingestion time: when the broker received it.

Default: event time, extracted from the record's timestamp field. Change with a TimestampExtractor.

Windowing

// Tumbling: fixed-size, non-overlapping
TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1));

// Hopping: fixed-size, overlap
TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(1))
           .advanceBy(Duration.ofMinutes(1));

// Session: gap-based
SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(30));

// Sliding: window slides per event
SlidingWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5));

Pattern:

KTable<Windowed<String>, Long> counts = stream
    .groupByKey()
    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1)))
    .count();

The key becomes Windowed<String> — original key plus window bounds.

Grace Period and Late Events

A late event for a window that already closed normally gets dropped. The grace period keeps the window open longer:

TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(2))

Trade: lower grace = faster window close; higher grace = handles out-of-order events at memory cost.

Suppress (Emit Once Per Window)

By default, aggregations emit on every update — many intermediate values per window.

.suppress(Suppressed.untilWindowCloses(unbounded()))

Emits one result per window after it closes. Useful for downstream consumers that don't want noise.

Interactive Queries

You can query state stores directly from your app:

ReadOnlyKeyValueStore<String, Long> store = streams.store(
    StoreQueryParameters.fromNameAndType("counts-store", QueryableStoreTypes.keyValueStore()));
Long c = store.get("user-1");

Useful for serving low-latency reads from materialized state without going through a database.

Exactly-Once

props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);

Wraps consume → process → produce → commit in a Kafka transaction. Each input record contributes to outputs atomically. EOS_V2 (Kafka 2.5+) is much cheaper than V1 — one producer per stream thread, not per partition.

Common Patterns

1. Event Enrichment.

KStream(orders) JOIN KTable(customers) → KStream(enriched-orders)

2. Real-Time Counters.

KStream(clicks) → groupByKey → windowed-count → KTable(clicks-per-min)

3. Sessionization.

KStream(events) → groupByKey → session-window → KTable(user-sessions)

4. Materialized View / CQRS.

KStream(domain-events) → fold/aggregate → KTable(view) → interactive queries

Common Mistakes

Streams vs ksqlDB vs Flink

[!NOTE] Kafka Streams is great when your inputs and outputs are Kafka, your state fits on the app instances, and you want zero external coordinators. For multi-source pipelines or massive state, Flink is the better tool.

Interview Follow-ups

• "What's the difference between aggregate and reduce?" — reduce requires the same type in/out. aggregate lets the result type differ from the input.
• "How are stream tasks scheduled?" — One task per input partition. Threads run multiple tasks. Adding instances rebalances tasks across instances.
• "What happens to state on rolling deploy?" — RocksDB stays on disk; instance picks up where it left off. Standby replicas keep warm state on other instances for fast failover.

Q: What is the difference between Retention and Log Compaction in Kafka?

Answer:

Kafka keeps data on disk and provides two distinct retention strategies for controlling how long messages are stored.

Time/Size-Based Retention (Default)

Messages are deleted after a configured time period or when the log exceeds a size limit.

# Time-based: Delete messages older than 7 days
log.retention.hours=168        # (default: 168 hours = 7 days)

# Size-based: Delete oldest messages when partition log exceeds 1GB
log.retention.bytes=1073741824

# Segment file size (retention is applied per segment)
log.segment.bytes=1073741824   # 1GB per segment file

How it works: Kafka stores messages in segment files. When a segment's age exceeds retention.hours (or the total log size exceeds retention.bytes), the entire segment file is deleted.

Partition 0:
  segment-0.log (2 days old) ← DELETED when > 7 days
  segment-1.log (1 day old)
  segment-2.log (current, active)

Log Compaction

Instead of deleting messages by time, Kafka keeps only the latest value for each unique key. It's as if you have a table where each key's row is updated in-place.

log.cleanup.policy=compact

Before compaction:

Offset  Key      Value
0       user-1   {"name": "Alice"}
1       user-2   {"name": "Bob"}
2       user-1   {"name": "Alice Smith"}  ← newer value for user-1
3       user-3   {"name": "Charlie"}
4       user-2   null                     ← tombstone (delete marker)

After compaction:

Offset  Key      Value
2       user-1   {"name": "Alice Smith"}  ← latest value kept
3       user-3   {"name": "Charlie"}
                                          ← user-2 deleted (tombstone)

When to Use Which?

Real-World Examples

Compaction:

• __consumer_offsets — Kafka's internal topic for consumer offsets (only latest offset per group/partition matters).
• CDC topics (Debezium) — latest row state per primary key.
• User profile cache — latest profile per user ID.

Time retention:

• Clickstream events, application logs, order events.

[!IMPORTANT] Compaction is not immediate. A background thread called the "log cleaner" periodically compacts segments. Between compactions, both old and new values for a key may exist. Never rely on compaction for real-time deduplication — it's an eventual cleanup mechanism.

Q: What are the key metrics to monitor in a Kafka cluster?

Answer:

Monitoring is essential for maintaining a healthy Kafka cluster. Here are the critical metrics organized by component.

Broker Metrics

Producer Metrics

Consumer Metrics

Monitoring Stack

Kafka (JMX Metrics)
    ↓
Prometheus (JMX Exporter)
    ↓
Grafana (Dashboards + Alerts)

Popular Tools:

• Prometheus + JMX Exporter: Industry standard for metric collection.
• Grafana: Visualization and alerting.
• Burrow: LinkedIn's tool specifically for consumer lag monitoring.
• Kafka Manager / AKHQ: Web UI for cluster management.
• Confluent Control Center: Commercial monitoring (Confluent Platform).

Critical Alerts to Set Up

# Example Prometheus alerting rules
groups:
  - name: kafka-alerts
    rules:
      - alert: KafkaOfflinePartitions
        expr: kafka_server_replicamanager_offline_partitions_count > 0
        for: 1m
        labels:
          severity: critical

      - alert: KafkaConsumerLagHigh
        expr: kafka_consumer_group_lag > 10000
        for: 5m
        labels:
          severity: warning

      - alert: KafkaUnderReplicatedPartitions
        expr: kafka_server_replicamanager_under_replicated_partitions > 0
        for: 5m
        labels:
          severity: warning

[!TIP] In interviews, the most impactful metrics to mention are UnderReplicatedPartitions (replication health), consumer lag (processing health), and OfflinePartitionsCount (availability). These cover the three biggest operational concerns: data durability, throughput, and uptime.

Q: When would you choose Kafka over RabbitMQ or SQS?

Answer:

This is a common architectural decision question. Each messaging system serves different primary use cases.

Apache Kafka

A distributed event streaming platform designed as a durable commit log.

Best for:

• High-throughput event streaming (millions of msg/sec)
• Event sourcing and CQRS architectures
• Log aggregation
• Real-time analytics pipelines
• When consumers need to replay old messages
• When multiple independent consumers need the same data

RabbitMQ

A traditional message broker that implements AMQP (Advanced Message Queuing Protocol).

Best for:

• Request/reply patterns (RPC over messaging)
• Complex routing logic (topic exchanges, headers, fanout)
• Priority queues (some messages should be processed first)
• When messages should be deleted after consumption
• Smaller scale (thousands, not millions, of msg/sec)
• When you need per-message acknowledgement and fine-grained delivery control

Amazon SQS

A fully managed message queue on AWS.

Best for:

• Teams that don't want to operate infrastructure
• Simple producer-consumer patterns
• Variable/bursty workloads (auto-scales transparently)
• Dead letter queue support out of the box
• When tight AWS integration is needed (Lambda triggers, IAM)

Comparison Table

Decision Framework

Need replay / event sourcing?          → Kafka
High throughput (>100K msg/sec)?       → Kafka
Multiple independent consumers?       → Kafka
Complex routing (headers, priorities)? → RabbitMQ
Request/reply (RPC) pattern?           → RabbitMQ
Don't want to manage infrastructure?  → SQS (or Confluent Cloud/CloudAMQP)
Simple queue, AWS ecosystem?           → SQS

[!NOTE] These are not mutually exclusive. Many production architectures use Kafka for event streaming (inter-service communication) AND SQS/RabbitMQ for task queues (background job processing). Using the right tool for each specific use case is the mark of a mature architecture.

Q: How do you size a Kafka cluster — partitions, brokers, disk, network?

Answer:

Sizing is the most common "system design" Kafka question. Walk through it as a back-of-envelope calculation grounded in four numbers: throughput, retention, replication factor, partition count.

Inputs You Need First

• Peak ingress (MB/s, not avg).
• Retention (hours or days).
• Replication factor (almost always 3).
• Consumer fanout (how many independent consumer groups read each byte).
• Compression ratio (typical 3–5x for JSON, 2x for already-compact binary).

Step 1: Storage

disk_per_broker = (peak_ingress * retention * RF) / num_brokers

Example: 50 MB/s peak, 7 days retention, RF=3, 6 brokers.

total = 50 * 86400 * 7 * 3 = 90,720,000 MB ≈ 90 TB
per broker ≈ 15 TB

Add 30% headroom for compaction, indexes, OS, log roll, and operational margin → ~20 TB per broker.

Step 2: Network

Each byte produced is:

• Written once over network to leader.
• Replicated RF-1 times.
• Read fanout times by consumers.

broker_egress = ingress_share * (RF - 1 + fanout)

Example: 50 MB/s peak, 3 consumer groups, RF=3, 6 brokers.

ingress per broker  = 50/6   ≈ 8.3 MB/s
egress per broker   = 8.3 * (2 + 3) ≈ 42 MB/s

At 10 GbE NIC = 1.25 GB/s. You're fine. At 1 GbE (125 MB/s) and a heavier fanout, you're not.

Step 3: Partitions

Lower bound: enough partitions for peak consumer parallelism.

min_partitions >= max_consumers_in_one_group

If you ever want 24 parallel consumers, you need at least 24 partitions.

Upper bound considerations:

• Each partition has open file handles, a leader, replication threads, metadata.
• ZK-mode: keep total partitions per broker under ~4,000.
• KRaft: practical limits are much higher (10s of thousands per broker), but rebalance time still scales with partition count.
• More partitions = larger end-to-end latency floor (more batches to flush).

Rule of thumb:

partitions ≈ max(target_throughput / partition_throughput, consumer_parallelism)

Single-partition sustainable throughput depends on disk and replication but 10–30 MB/s is a realistic planning number on commodity hardware.

Step 4: Broker Count

Constraints:

• Replicas per broker: keep under ~4,000 (ZK) or ~10–20k (KRaft).
• Disk per broker: tied to retention. SSD/NVMe strongly preferred.
• Failure domain: RF=3 needs at least 3 brokers across at least 3 racks (set broker.rack).

For meaningful workloads, minimum useful cluster is 3 brokers; production typically 5–6+ for headroom during node loss + maintenance.

Worked Example

"We ingest 200k events/sec, avg 1 KB. Retention 14 days. 4 consumer groups. Plan it."

• Ingress: 200,000 × 1 KB = 200 MB/s peak (assume = avg for simplicity).
• Storage: 200 * 86400 * 14 * 3 = 725 TB. With 30% headroom → ~940 TB.
• 6 brokers → ~157 TB/broker. Too much. → 12 brokers, 80 TB/broker, or compress (~3x typical) → 4 brokers viable, plan 6 for HA.
• Partitions: enough for consumer parallelism. If each group runs 32 consumers, ≥ 32 partitions, round up to 64 for headroom.
• Network/broker: 200/6 * (2 + 4) = 200 MB/s egress. 10 GbE sufficient.

Common Mistakes

[!NOTE] Always plan for N+1 brokers — you must be able to lose a broker (or take one down for maintenance) without losing capacity.

Interview Follow-ups

• "What happens if you set RF higher than broker count?" — Topic creation fails. RF ≤ broker count, always.
• "Why not 1000 partitions per topic?" — Producer batches per-partition; tiny partitions = tiny batches = bad throughput. Also, controller fail-over time scales with total partitions.
• "How does KRaft change sizing?" — Removes ZK as a metadata bottleneck, lets you scale partitions higher, faster controller fail-over. Storage/network math unchanged.

Q: How do you secure a Kafka cluster — TLS, SASL, ACLs?

Answer:

Kafka security has three independent dimensions: encryption in transit (TLS), authentication (who are you?), and authorization (what can you do?). You configure them per listener, and most production clusters layer all three.

Listeners

Brokers expose one or more listeners with distinct security profiles:

listeners=INTERNAL://:9092,EXTERNAL://:9093,CONTROLLER://:9094
listener.security.protocol.map=\
  INTERNAL:PLAINTEXT,\
  EXTERNAL:SASL_SSL,\
  CONTROLLER:SSL
inter.broker.listener.name=INTERNAL

• INTERNAL: broker-to-broker, often PLAINTEXT inside a VPC.
• EXTERNAL: client traffic — encrypted + authenticated.
• CONTROLLER: KRaft controller plane.

TLS (Encryption + Optional mTLS Auth)

listeners=SSL://:9093
ssl.keystore.location=/etc/kafka/broker.keystore.jks
ssl.keystore.password=...
ssl.truststore.location=/etc/kafka/truststore.jks
ssl.truststore.password=...
ssl.client.auth=required        # mTLS — clients also present a cert

With ssl.client.auth=required, the cert's CN/SAN becomes the principal — that is the authentication. No separate SASL handshake.

SASL (Authentication)

SCRAM example (server side):

listeners=SASL_SSL://:9093
sasl.enabled.mechanisms=SCRAM-SHA-512
listener.name.sasl_ssl.scram-sha-512.sasl.jaas.config=...

Client jaas.conf:

KafkaClient {
  org.apache.kafka.common.security.scram.ScramLoginModule required
  username="orders-svc"
  password="...";
};

[!NOTE] Never use SASL_PLAINTEXT with PLAIN outside a closed test network — credentials cross the wire reversibly.

Authorization (ACLs)

Set authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer (KRaft) or kafka.security.authorizer.AclAuthorizer (ZK).

# Allow service "orders-svc" to produce to "orders"
kafka-acls.sh --bootstrap-server b1:9093 --add \
  --allow-principal User:orders-svc \
  --operation Write --operation Describe \
  --topic orders

# Allow consumer group "orders-consumer"
kafka-acls.sh --bootstrap-server b1:9093 --add \
  --allow-principal User:orders-consumer \
  --operation Read --operation Describe \
  --topic orders --group orders-consumer

Operations: Read, Write, Create, Delete, Alter, Describe, ClusterAction, AlterConfigs, DescribeConfigs, IdempotentWrite.

Resources: Topic, Group, Cluster, TransactionalId, DelegationToken.

Default policy: deny when authorizer is enabled — explicit allow required. Be careful enabling on a running cluster without first auditing required principals.

Common Production Setup

Clients  ──SASL_SSL (SCRAM or OAUTH)──>  Brokers
                                          │
                                  ─SSL (mTLS)─> KRaft controllers
                                          │
                                  ─SSL─> Inter-broker

Authorization via ACLs, with one principal per service, automated by Terraform or a topic registry.

Encryption at Rest

Kafka doesn't encrypt log segments itself. Two options:

• Disk-level encryption (LUKS, EBS encryption) — protects only against disk theft.
• Field-level / message-level encryption in the producer (envelope encryption with KMS) — protects in the broker filesystem too. Required for PCI/PII often.

Common Pitfalls

Interview Follow-ups

• "How do you rotate a SCRAM password without downtime?" — Add new credentials, deploy client with both old+new in jaas (or just new), remove old SCRAM entry in metadata. SCRAM creds live in __cluster_metadata (KRaft) or ZK.
• "How do mTLS and ACLs interact?" — mTLS gives you the principal (User:CN=...); ACLs apply against that principal. No SASL needed.
• "Is there a way to do row-level / field-level authorization?" — Not natively. Use client-side encryption with KMS keys gated by an external policy engine, or a privacy proxy in front.

Q: How do you replicate Kafka across data centers — MirrorMaker 2 vs Cluster Linking?

Answer:

A single Kafka cluster lives in one fault domain. To survive a region outage, support active-active multi-region traffic, or aggregate edge clusters into a central one, you need inter-cluster replication. Two main tools: open-source MirrorMaker 2 and Confluent's proprietary Cluster Linking.

Why You Replicate

Common drivers:

• Disaster recovery (DR): failover to a standby cluster if primary region dies.
• Aggregation: many edge clusters → one central analytics cluster.
• Geo-distribution: producers/consumers near their users; mirror for cross-region reads.
• Migration: move workloads between clusters with no downtime.
• Compliance: pin a copy of data in a specific region.

MirrorMaker 2 (MM2)

A Kafka Connect–based replicator (replaces MM1). Runs as a fleet of Connect workers.

   ┌──────────────┐                          ┌──────────────┐
   │  Cluster A   │  ───  MirrorMaker 2  ──► │  Cluster B   │
   │  topic foo   │                          │  topic A.foo │
   └──────────────┘                          └──────────────┘

Topics are renamed by default — foo on A becomes A.foo on B — to make active-active safe (no infinite loops). Configurable via replication.policy.

# mm2.properties
clusters = primary, dr
primary.bootstrap.servers = primary:9092
dr.bootstrap.servers = dr:9092

primary->dr.enabled = true
primary->dr.topics = orders|payments|inventory
primary->dr.replication.factor = 3

replication.policy.separator = .
sync.topic.acls.enabled = true
sync.topic.configs.enabled = true

offset-syncs.topic.replication.factor = 3

MM2 also:

• Mirrors topic configs (retention, partition count delta).
• Mirrors ACLs (optional).
• Translates consumer offsets so a consumer can resume on the DR cluster (using RemoteClusterUtils.translateOffsets).

Cluster Linking (Confluent)

Native broker-side replication — no separate Connect workers. The destination cluster pulls bytes directly from the source.

   ┌──────────────┐                          ┌──────────────┐
   │  Cluster A   │  ◄── byte-for-byte ───   │  Cluster B   │
   │  topic foo   │                          │  topic foo   │
   └──────────────┘                          └──────────────┘

Properties:

• No topic renaming. Same topic name on both sides (one-way).
• Same offsets. Consumers can fail over without offset translation.
• Lower latency, fewer hops. Brokers fetch directly.
• Confluent Platform / Cloud only.

Comparison

Active-Passive (DR) Topology

Region A (active)               Region B (standby)
┌───────────────┐               ┌───────────────┐
│  Producers ───┼──┐    MM2     │   (idle)      │
│  Consumers ───┘  │ ─────────► │  Consumers    │
└───────────────┘                └───────────────┘
                                  ▲
                                  └─ used on failover

On failover:

1. Stop producers in A.
2. Wait for MM2 to drain.
3. Translate consumer offsets to DR cluster.
4. Point producers + consumers at B.

RPO (recovery point objective) ≈ replication lag, typically seconds.

Active-Active Topology

Region A                      Region B
┌──────────┐                  ┌──────────┐
│   foo    │ ──── MM2 ──────► │ A.foo    │   <- reads from both
│ A.foo  ◄─┼────── MM2 ───────┤   foo    │
└──────────┘                  └──────────┘

Producers write to local foo. Consumers subscribe to both foo and A.foo (or B.foo).

Renaming prevents loops: A.foo on B is not re-replicated to A (MM2's default policy skips already-replicated topics).

Caveats:

• Consumer applications must merge two streams. Per-key ordering across regions is impossible without conflict resolution.
• Use case fits independent streams (per-region orders) better than shared state (global inventory).

Aggregation Topology

edge1 ┐
edge2 ┼─► MM2 ───► central
edge3 ┘

Many small clusters → one big cluster for analytics. Each edge ships its local topic to a region-prefixed topic centrally (edge1.orders, edge2.orders...). Central jobs consume all of them.

Operational Concerns

Lag monitoring:

mm2-MirrorSourceConnector.records.lag
mm2-MirrorCheckpointConnector.offset.lag

Alert if lag grows beyond a few seconds of replication.

Throughput:

MM2 scales horizontally — add Connect workers. Each task handles a set of partitions. Tune tasks.max and producer.linger.ms for throughput vs latency.

Cycle prevention:

DefaultReplicationPolicy rejects topics already named with a source prefix. If you use a custom policy, be sure to encode "this came from cluster X, don't bounce back."

Schema sync:

Schemas (Avro, Protobuf) must also be replicated if you use Schema Registry. Use a separate Schema Registry per cluster or a federated setup; otherwise consumers in DR can't deserialize.

Common Mistakes

Network Cost

Inter-region bandwidth is expensive. Estimate:

bytes/sec = ingress_to_replicate × compression_ratio
$/month  = bytes/sec × seconds × cloud_egress_rate

For a 100 MB/s replicated stream cross-region at $0.02/GB egress: ~$5k/month. Compression matters — lz4/zstd cuts cost ~3x.

Migration Pattern: Cluster Linking Promotion

For zero-downtime migration:

1. Establish Cluster Link from old → new.
2. Wait until lag = 0.
3. Pause producers on old; wait for drain.
4. Cut producers to new.
5. Cut consumers to new (same offsets — no translation).
6. Decommission old.

Interview Follow-ups

• "What's the difference between MM2 and Confluent Replicator?" — Replicator predates MM2, similar idea, Confluent-licensed. MM2 has feature parity for most use cases.
• "Does replication preserve transactions?" — Aborted records are filtered if you set read.isolation.level=read_committed on the source side. Transactional state itself doesn't cross cluster boundaries.
• "Why not use Kafka's built-in replication for DR?" — Native replication requires synchronous ISR — too slow across regions. Cross-cluster replication is async and decoupled.