Level |
Subtopic |
Topics |
Basic |
Kafka Fundamentals |
Kafka Overview, Event Streaming Basics, Use Cases, Kafka vs Traditional Messaging, Real-time Data Use Cases |
|
Kafka Architecture |
Brokers, Topics, Partitions, Offsets, Producers, Consumers, Zookeeper |
|
Installation & Setup |
Local Setup, Kafka CLI Tools, Cluster Setup Basics, Config Files, Zookeeper Setup |
|
Producers & Consumers |
Producer API, Consumer API, Consumer Groups, Auto Offset Reset, Poll vs Subscribe |
|
Topics & Partitions |
Creating Topics, Partitioning Basics, Replication Factor, Retention Policies, Log Segments |
|
Data Flow Basics |
Write Path, Read Path, Publish-Subscribe, Message Ordering, Delivery Semantics |
|
Serialization |
String Serializer, JSON, Avro Basics, Protobuf Intro, Schema Registry Intro |
|
Kafka CLI & Tools |
kafka-topics.sh, kafka-console-producer.sh, kafka-console-consumer.sh, kafka-consumer-groups.sh, kafka-configs.sh |
|
Monitoring Basics |
JMX Basics, Kafka Metrics, Producer/Consumer Lag, Simple Dashboards, Logs |
|
Common Use Cases |
Messaging Queue, Log Aggregation, Stream Processing Intro, Microservices Integration, Event Sourcing |
Intermediate |
Kafka Producers (Advanced) |
Idempotent Producers, Delivery Guarantees, Batching, Compression, Retry/ACK Configs |
|
Kafka Consumers (Advanced) |
Rebalancing, Static Group Membership, Partition Assignment, Offset Management, Consumer Lag Analysis |
|
Kafka Streams API |
Stream Processing, KStream vs KTable, Stateless Ops, Stateful Ops, Windowing |
|
Kafka Connect |
Connectors Intro, Source vs Sink, JDBC Connector, File Connector, REST API |
|
Schema Management |
Confluent Schema Registry, Backward/Forward Compatibility, Versioning, Schema Evolution, Validation |
|
Security Basics |
SSL, SASL, Authentication, Authorization, ACLs |
|
Multi-Cluster Setup |
MirrorMaker, Cluster Linking, Replication Across DCs, DR Basics, Active-Passive Setup |
|
Scaling Kafka |
Horizontal Scaling, Partition Strategies, Broker Scaling, Data Distribution, Rack Awareness |
|
Reliability & Durability |
Replication, Min In-Sync Replicas, Acknowledgments, Unclean Leader Election, ISR Management |
|
Monitoring & Alerting |
Kafka Manager, Burrow, Consumer Lag Monitoring, Grafana + Prometheus, Alerts Setup |
Advanced |
Kafka Streams Advanced |
Joins, Aggregations, Session Windows, Interactive Queries, Fault Tolerance |
|
Transactions in Kafka |
Exactly Once Semantics, Transactions API, Producer Transactions, Consumer Transactions, Use Cases |
|
Kafka Connect Advanced |
Custom Connectors, SMT (Single Message Transform), Distributed Mode, Error Handling, Scaling Connect |
|
Performance Tuning |
Producer Tuning, Consumer Tuning, Broker Configs, OS Tuning, GC Optimization |
|
Reliability Engineering |
End-to-End Exactly Once, Data Guarantees, Message Ordering Guarantees, Log Compaction, Recovery Mechanisms |
|
Security Advanced |
Kerberos, Role-Based Access Control, Audit Logs, Secret Management, Enterprise Security |
|
Storage & Log Management |
Log Segments, Index Files, Compaction, Tiered Storage, Retention Strategies |
|
Cloud & Managed Kafka |
Confluent Cloud, AWS MSK, Azure Event Hubs, GCP Pub/Sub vs Kafka, Hybrid Deployments |
|
Integrations |
Kafka + Spark, Kafka + Flink, Kafka + Hadoop, Kafka + Cassandra, Kafka + Elasticsearch |
|
High Availability |
Broker Failover, Controller Failover, Zookeeper Failover, Multi-Cluster HA, Zero Data Loss Strategies |
Expert |
Kafka Internals |
Broker Internals, Partition Leader Election, Controller Role, Metadata Quorum (KRaft), Replica Fetching |
|
Zookeeper & KRaft |
Zookeeper Deep Dive, ZK Failures, Migration to KRaft, Metadata Management, Quorum Replication |
|
Large Scale Deployments |
Multi-Region Kafka, Cross-Data Center Replication, 1000+ Brokers, Petabyte Scale, Global Kafka |
|
Governance & Compliance |
GDPR Compliance, Data Retention Governance, Access Policies, Data Lineage, Auditing |
|
Custom Development |
Custom Partitioner, Custom Serializer/Deserializer, Custom Connectors, Custom Streams Processor, Extensions |
|
Enterprise Integrations |
Kafka + Microservices, Kafka + API Gateways, Kafka + Service Mesh, Kafka + ERP/CRM, Legacy Systems Integration |
|
Disaster Recovery & Backup |
Cross-Cluster Replication, Snapshotting, Recovery Testing, RPO/RTO, Backup Strategies |
|
Observability |
Distributed Tracing, OpenTelemetry, Logs + Metrics + Traces Integration, Advanced Lag Analysis, Chaos Testing |
|
Kafka with Containers & K8s |
Kafka on Docker, Strimzi Operator, Helm Charts, StatefulSets, Multi-Tenant Clusters |
|
Future & Trends |
KIP (Kafka Improvement Proposals), KRaft Evolution, Tiered Storage Future, Unified APIs, Kafka Beyond Messaging |
|