Key Concepts
| Topic | Sub-Topics | UseCase |
|---|---|---|
| Apache Kafka | Event Streaming, Messaging, Pub/Sub, Architecture | Understand distributed event streaming |
| Kafka Architecture | Broker, Cluster, Topic, Partition, Replica | Learn Kafka internals |
| Installation & Setup | Single Node, Multi-Broker, Docker, KRaft Mode | Set up a Kafka cluster |
| Kafka Components | Producer, Consumer, Broker, Controller | Understand component interactions |
| Topics | Create, Delete, Describe, List Topics | Manage event streams |
| Partitions | Partitioning Strategy, Ordering, Scaling | Scale message processing |
| Replication | Leader, Follower, ISR, Replication Factor | Ensure high availability |
| Offsets | Consumer Offset, Commit, Reset | Track message consumption |
| Producers | KafkaProducer API, Configuration, Acknowledgments | Publish business events |
| Producer Configurations | acks, retries, batch.size, linger.ms, compression.type | Optimize producer performance |
| Partitioners | Default Partitioner, Custom Partitioner | Route messages efficiently |
| Message Keys | Key-based Routing, Ordering | Preserve customer transaction order |
| Consumers | KafkaConsumer API, Poll Loop | Consume event streams |
| Consumer Groups | Group Coordination, Rebalancing | Scale event processing |
| Consumer Configurations | auto.offset.reset, enable.auto.commit, max.poll.records | Tune consumer behavior |
| Offset Management | Auto Commit, Manual Commit, CommitSync, CommitAsync | Reliable message processing |
| Serialization | StringSerializer, JSON, Avro, Protobuf | Serialize business objects |
| Deserialization | JSONDeserializer, AvroDeserializer | Convert messages into objects |
| Schema Registry | Avro Schema, Compatibility, Versioning | Manage evolving message schemas |
| Message Compression | GZIP, Snappy, LZ4, ZSTD | Reduce network and storage costs |
| Retention Policies | Time-based, Size-based Retention | Store historical events |
| Log Compaction | Cleanup Policy, Tombstone Records | Maintain latest entity state |
| Delivery Guarantees | At Most Once, At Least Once, Exactly Once | Reliable financial transactions |
| Idempotent Producer | enable.idempotence, Duplicate Prevention | Prevent duplicate messages |
| Transactions | Transactional Producer, Atomic Writes | Multi-topic financial updates |
| Error Handling | Retries, Dead Letter Topics (DLQ), Exception Handling | Handle failed messages |
| Retry Patterns | Retry Topics, Exponential Backoff | Recover from transient failures |
| Dead Letter Queue (DLQ) | Failed Event Handling | Isolate problematic records |
| Kafka Streams | Streams API, Stateless & Stateful Processing | Build streaming applications |
| KTables | Materialized Views, Aggregation | Maintain real-time summaries |
| Stream Joins | Stream-Stream, Stream-Table, Table-Table | Merge real-time datasets |
| Windowing | Tumbling, Hopping, Sliding, Session Windows | Time-based analytics |
| Interactive Queries | Query Local State Stores | Build real-time dashboards |
| Kafka Connect | Source Connectors, Sink Connectors | Integrate databases and cloud storage |
| Source Connectors | JDBC Source, Debezium, File Source | Capture changes from source systems |
| Sink Connectors | JDBC Sink, Elasticsearch, S3, Iceberg | Deliver data to downstream systems |
| CDC | Debezium, Database Events | Stream database changes |
| MirrorMaker 2 | Cluster Replication, Disaster Recovery | Replicate events across regions |
| Security | SSL, SASL, ACLs, Authentication, Authorization | Secure Kafka clusters |
| Monitoring | JMX Metrics, Prometheus, Grafana | Monitor Kafka health |
| Performance Tuning | Batch Size, Compression, Partition Count | Optimize throughput |
| Capacity Planning | Broker Sizing, Storage, Network | Design enterprise Kafka clusters |
| Spring Boot Integration | Spring for Apache Kafka, KafkaTemplate, @KafkaListener | Build event-driven microservices |
| Spring Cloud Stream | Binder, Functional Programming Model | Simplify event-driven applications |
| Spark Integration | Structured Streaming with Kafka | Real-time ETL pipelines |
| Databricks Integration | Kafka &†‚’ Delta Lake | Lakehouse streaming architecture |
| Apache Iceberg Integration | Kafka Connect Iceberg Sink | Build transactional data lakes |
| Kubernetes Deployment | Strimzi, Helm Charts, Operators | Run Kafka on Kubernetes |
| CI/CD & Operations | Docker, Kubernetes, Rolling Upgrades | Deploy and maintain Kafka clusters |
Interview question
| What is Apache Kafka, and why is it used in distributed systems? |
| Explain Kafka architecture and its major components. |
| What is the difference between Kafka and traditional message queues? |
| Explain Kafka topics, partitions, and offsets. |
| Why does Kafka use partitions? |
| How does Kafka achieve high throughput? |
| How does Kafka achieve fault tolerance? |
| What is a Kafka broker? |
| What happens when a Kafka broker goes down? |
| What is a Kafka cluster? |
| What is the role of a Kafka controller? |
| What is KRaft mode in Kafka? |
| How is KRaft different from ZooKeeper-based Kafka? |
| What is a Kafka topic and how should topics be designed? |
| What is partition count, and how do you decide the number of partitions? |
| What is replication factor in Kafka? |
| Explain leader and follower replicas. |
| What is ISR in Kafka? |
| What causes a replica to leave the ISR? |
| What happens when a partition leader fails? |
| What is an unclean leader election? |
| What is the difference between leader replica and preferred replica? |
| What is a Kafka offset? |
| How does Kafka maintain message ordering? |
| Can Kafka guarantee ordering across multiple partitions? |
| How does Kafka partition a message when a key is provided? |
| What happens when a Kafka producer sends a message without a key? |
| Explain Kafka producer architecture. |
| What are Kafka producer acknowledgements? |
| Explain the difference between acks=0, acks=1, and acks=all. |
| What is idempotent producer in Kafka? |
| How does Kafka prevent duplicate messages with an idempotent producer? |
| What is the role of producer retries? |
| What is the difference between retries and idempotence? |
| What are linger.ms and batch.size in Kafka producer? |
| How does compression improve Kafka producer performance? |
| Which Kafka compression algorithms are available? |
| What is max.in.flight.requests.per.connection? |
| How would you tune a Kafka producer for high throughput? |
| How would you configure a Kafka producer for strong durability? |
| Explain Kafka consumer architecture. |
| What is a Kafka consumer group? |
| How are partitions assigned to consumers in a consumer group? |
| What happens when the number of consumers exceeds the number of partitions? |
| Can two consumers in the same consumer group read the same partition simultaneously? |
| Can consumers from different consumer groups read the same message? |
| What is consumer group rebalancing? |
| What causes Kafka consumer rebalancing? |
| What is the difference between eager and cooperative rebalancing? |
| Explain Kafka consumer partition assignment strategies. |
| What are Range, RoundRobin, Sticky, and CooperativeSticky assignors? |
| What is consumer lag in Kafka? |
| How do you monitor and troubleshoot consumer lag? |
| What causes Kafka consumer lag? |
| How can consumer lag be reduced? |
| What is auto.offset.reset? |
| Explain earliest, latest, and none offset reset policies. |
| What is enable.auto.commit? |
| What is the difference between auto commit and manual offset commit? |
| When should you use commitSync() versus commitAsync()? |
| What happens if a consumer crashes before committing an offset? |
| What happens if a consumer commits an offset before processing the message? |
| How would you implement at-least-once processing in Kafka? |
| How would you implement at-most-once processing in Kafka? |
| How can exactly-once processing be achieved in Kafka? |
| What is Kafka's exactly-once semantics? |
| What is a Kafka transaction? |
| What is a transactional.id? |
| What is read_committed versus read_uncommitted? |
| How does Kafka handle duplicate messages? |
| What is Kafka Streams? |
| What is the difference between Kafka Consumer API and Kafka Streams? |
| Explain KStream and KTable. |
| What is a GlobalKTable? |
| Explain stateful and stateless Kafka Streams processing. |
| What are state stores in Kafka Streams? |
| What is a repartition topic in Kafka Streams? |
| How does Kafka Streams handle failures? |
| What is Kafka Connect? |
| What is the difference between Source Connector and Sink Connector? |
| What are Kafka Connect workers? |
| Explain standalone and distributed Kafka Connect modes. |
| How do you implement a Kafka-to-database pipeline using Kafka Connect? |
| What is Schema Registry and why is it used with Kafka? |
| Explain Avro, JSON Schema, and Protobuf with Kafka. |
| What is schema evolution in Kafka? |
| Explain backward, forward, and full schema compatibility. |
| How do you secure Kafka in an enterprise environment? |
| Explain SSL/TLS, SASL, and ACLs in Kafka security. |
| How do Kafka ACLs control producer and consumer access? |
| How do you monitor Kafka brokers in production? |
| Which Kafka metrics are critical for production monitoring? |
| How would you troubleshoot high broker CPU and disk usage? |
| How would you troubleshoot high Kafka network utilization? |
| How do you handle disk failures in a Kafka cluster? |
| How do you perform Kafka partition reassignment? |
| How do you increase partitions for an existing Kafka topic, and what are the implications? |
| How do retention policies and log compaction work in Kafka? |
| How would you design a highly available Kafka cluster for a banking or financial application? |
| Design a production-grade Kafka architecture for millions of messages per second with fault tolerance, ordering, security, monitoring, and exactly-once processing. |
