#Apache_Kafka

Last updated    - V4 (14-Sep-2025)

Topic - SubTopic	Basic	Intermediate	Advanced	Expert
Kafka Basics - Overview	What is Kafka?, Features of Kafka, Difference between Kafka and traditional messaging systems	Kafka architecture overview, Brokers, Topics, Partitions	Kafka use cases in real-time streaming, Kafka ecosystem components, Kafka vs RabbitMQ	Designing enterprise-grade Kafka solutions, Multi-cluster Kafka deployments, Scalability considerations
Kafka Topics & Partitions	Topics, Partitions, Replication factor, Offset basics	Log compaction, Partition key, Consumer group assignment	Custom partitioners, Partition reassignment, Efficient topic design	Large-scale topic management, Multi-tenant topics, High-throughput topic optimization
Kafka Producers	Producer API basics, Sending messages, Keyed messages	Producer configuration, Acknowledgments, Idempotent producer	Transactional producer, Compression, Batching messages	High-performance producers, Multi-threaded producers, Optimizing producer throughput
Kafka Consumers	Consumer API basics, Polling messages, Group management	Consumer offsets, Auto commit vs manual commit, Rebalancing	Kafka consumer performance tuning, Exactly-once semantics, Multi-threaded consumers	Advanced consumer patterns, Scaling consumers, Consumer-side optimizations for large-scale deployments
Kafka Broker & Cluster	Broker responsibilities, Controller node, Zookeeper basics	Broker configuration, Leader and follower replication, ISR	Cluster scaling, High availability, Broker fault tolerance	Multi-datacenter Kafka, Cross-cluster replication, Kafka governance & operations
Kafka Streams & Connect	Kafka Streams basics, Stateless vs stateful operations	Stream processing, KTable, KStream joins	Kafka Streams topology, Windowed operations, Fault tolerance	Advanced stream processing patterns, Kafka Connect for ETL, Custom connectors
Kafka Security	SSL/TLS basics, Authentication, Simple ACLs	SASL authentication, User/group access control, ACLs for topics	Role-based access control, Kerberos integration, Encryption at rest	Enterprise security strategies, Multi-tenant security, Compliance for Kafka clusters
Kafka Performance & Optimization	Throughput basics, Latency considerations, Simple monitoring	Producer/consumer tuning, Compression, Batch size optimization	JVM tuning, Log segment management, Partition balancing	High-throughput Kafka, Multi-cluster optimization, Disaster recovery planning
Kafka Monitoring & Tools	Kafka logs, Metrics basics, Consumer lag	JMX metrics, Broker monitoring, Topic monitoring	Prometheus/Grafana integration, Alerting, Cluster health checks	Enterprise monitoring, Real-time alerting, Automated scaling and tuning
Kafka Advanced Features	N/A	Exactly-once semantics, Transactions, Idempotent producer	Log compaction, Custom partitioner, Kafka Streams joins	Cross-datacenter replication, Kafka multi-cluster management, Optimized high-volume pipelines

1. Kafka Fundamentals

1. What is Apache Kafka and why is it used?
2. Explain the difference between Kafka and traditional message brokers.
3. What are Kafka Producers, Consumers, and Brokers?
4. Define a Kafka Topic and Partition.
5. How does Kafka achieve fault tolerance?
6. What is a Kafka Cluster?
7. Explain Zookeeper?s role in Kafka (before KRaft).
8. What is KRaft mode in Kafka?
9. What is the difference between Kafka and RabbitMQ?
10. Explain Kafka?s publish-subscribe model.
11. What is a Consumer Group in Kafka?
12. How does Kafka handle backpressure?
13. What are Kafka Offsets?
14. How does Kafka achieve high throughput?
15. What is the retention policy in Kafka?
16. Explain the difference between at-most-once, at-least-once, and exactly-once delivery semantics.
17. What is a Kafka Record?
18. How does Kafka handle ordering of messages?
19. What are the key components of Kafka Architecture?
20. What is the role of a Kafka Controller?
21. Explain Kafka log compaction.
22. How does Kafka compare with AWS Kinesis?
23. What is a Kafka ISR (In-Sync Replica)?
24. Why is Kafka considered a distributed streaming platform?
25. What are some real-world use cases of Kafka?

---

2. Kafka Producers

1. What is a Kafka Producer?
2. How does a producer decide which partition to send a message to?
3. Explain Kafka producer acks setting (0, 1, all).
4. What is Kafka Producer batching?
5. How does Kafka handle producer retries?
6. What is idempotent producer in Kafka?
7. Explain the difference between synchronous and asynchronous send.
8. What is producer linger.ms setting?
9. How does compression work in Kafka producers?
10. What is the purpose of key.serializer and value.serializer?
11. How can producers ensure message ordering?
12. What is max.in.flight.requests.per.connection setting?
13. How does Kafka ensure durability of producer messages?
14. Explain transactional producers in Kafka.
15. What is the role of partitioner class in Kafka?
16. What are producer interceptors?
17. How does buffer.memory impact producers?
18. What is the significance of request.timeout.ms?
19. How does producer throughput tuning work?
20. What is a delivery.timeout.ms parameter?
21. How does Kafka producer handle network failures?
22. Explain the difference between send() and flush().
23. What is a Kafka producer callback?
24. How to configure producer for low latency vs high throughput?
25. What are common producer-related errors and how to handle them?

---

3. Kafka Consumers

1. What is a Kafka Consumer?
2. Explain Consumer Groups.
3. How does Kafka handle load balancing across consumers?
4. What is consumer rebalancing?
5. How are offsets managed in Kafka?
6. What is auto.offset.reset and its options?
7. Explain enable.auto.commit in Kafka.
8. What happens when a consumer in a group fails?
9. How does Kafka ensure at-least-once delivery for consumers?
10. What is the difference between KafkaConsumer and High-Level Consumer API?
11. Explain how consumers poll records from Kafka.
12. What is max.poll.records in Kafka consumer?
13. How to handle consumer lag?
14. What are partition assignment strategies in Kafka?
15. What is cooperative rebalancing?
16. How does Kafka consumer ensure fault tolerance?
17. Explain sticky partition assignment.
18. How to manually commit offsets in Kafka?
19. What happens if a consumer crashes after processing but before committing offset?
20. How does consumer heartbeat mechanism work?
21. What is session.timeout.ms in consumers?
22. Explain max.poll.interval.ms in consumer configs.
23. How does Kafka handle slow consumers?
24. What are common consumer-related errors and fixes?
25. How does Kafka consumer handle deserialization errors?

---

4. Kafka Topics & Partitions

1. What is a Kafka Topic?
2. Explain Kafka partitioning and its benefits.
3. How does Kafka ensure ordering within a partition?
4. What factors determine the number of partitions?
5. How to increase the number of partitions for a topic?
6. Can we decrease partitions in Kafka? Why or why not?
7. What is partition reassignment?
8. Explain Kafka topic replication.
9. How does replication factor impact availability?
10. What is leader and follower partition?
11. How does Kafka elect a new leader partition?
12. Explain partition skew problem.
13. How to monitor partition distribution across brokers?
14. What is topic retention period?
15. Difference between delete and compact cleanup policy.
16. What is minimum.insync.replicas?
17. How does Kafka store topic data on disk?
18. How do partitions affect throughput?
19. Explain how replication ensures fault tolerance.
20. What is rack awareness in Kafka?
21. How do you create a topic using CLI?
22. How to delete a topic safely in Kafka?
23. What happens if replication factor > number of brokers?
24. How does Kafka handle leader unavailability?
25. How to optimize partition assignment strategy?

---

5. Kafka Streams & Connect

1. What is Kafka Streams API?
2. Difference between Kafka Streams and Spark Streaming.
3. What is a KStream vs KTable?
4. Explain Kafka GlobalKTable.
5. How does Kafka Streams ensure fault tolerance?
6. What is stateful vs stateless processing?
7. Explain changelog topics in Kafka Streams.
8. How does windowing work in Kafka Streams?
9. What are stream processors and topologies?
10. What is a repartition topic?
11. Explain the role of RocksDB in Kafka Streams.
12. How does Kafka Streams achieve exactly-once semantics?
13. What is Kafka Connect?
14. Difference between source connector and sink connector.
15. Examples of commonly used Kafka connectors.
16. How does Kafka Connect handle data transformations?
17. What is single message transform (SMT)?
18. How does Connect ensure data consistency?
19. Explain standalone vs distributed mode in Kafka Connect.
20. How does offset storage work in Kafka Connect?
21. What are common challenges in Kafka Connect?
22. How does error handling work in Kafka Connect?
23. Explain dead letter queues in Connect.
24. How do you scale Kafka Connect workers?
25. What are best practices for Kafka Streams deployments?

---

6. Kafka Security

1. What security features does Kafka provide?
2. Explain SSL/TLS encryption in Kafka.
3. How does SASL authentication work in Kafka?
4. What is Kerberos authentication in Kafka?
5. Explain SCRAM mechanism in Kafka.
6. How to configure Kafka for client authentication?
7. What is role-based access control (RBAC) in Kafka?
8. Explain ACLs in Kafka.
9. How to grant permissions for a topic in Kafka?
10. What is the difference between PLAINTEXT and SASL_SSL protocols?
11. How to secure inter-broker communication?
12. Explain how to encrypt data at rest in Kafka.
13. What is quota management in Kafka?
14. How to secure Zookeeper in Kafka (pre-KRaft)?
15. What is delegation token authentication?
16. How does Kafka handle audit logging?
17. What are common misconfigurations in Kafka security?
18. How does Kafka support multi-tenant security?
19. Explain SSL handshake in Kafka communication.
20. What is the impact of enabling security on performance?
21. How to rotate credentials in Kafka?
22. Explain best practices for securing Kafka clusters.
23. How does OAuth2 authentication work in Kafka?
24. What is a principal in Kafka security?
25. How to troubleshoot authentication failures?

---

7. Kafka Administration & Monitoring

1. How do you install and configure Kafka?
2. What is the role of Zookeeper in cluster management?
3. How does KRaft mode replace Zookeeper?
4. Explain Kafka cluster scaling.
5. What are key metrics to monitor in Kafka?
6. What tools are available for Kafka monitoring?
7. How to monitor consumer lag?
8. What is JMX in Kafka monitoring?
9. How does Kafka Manager (or CMAK) help in admin tasks?
10. What is Cruise Control in Kafka?
11. How to balance partitions across brokers?
12. Explain Kafka cluster upgrade process.
13. How to recover from a broker failure?
14. What is broker.id in Kafka config?
15. How to configure Kafka logs retention and cleanup?
16. How to perform rolling restarts in Kafka?
17. What are log segments in Kafka?
18. Explain the role of controller quorum in KRaft.
19. How to handle Kafka disk failures?
20. How to backup and restore Kafka topics?
21. How does Kafka handle broker decommissioning?
22. What is preferred replica election?
23. How to configure Kafka for high availability?
24. Explain monitoring of producer/consumer throughput.
25. What are common operational challenges in Kafka clusters?