Key Concepts
| Day | Topic | Sub Topics |
|---|---|---|
| 1 | Introduction | Spark, Hadoop vs Spark, Spark Architecture, Components, Cluster Manager, Spark Ecosystem, Use Cases |
| 2 | Spark Installation & Setup | Install Spark, Java Setup, Python Setup, Scala Setup, Spark Shell, IDE Setup, Local Mode |
| 3 | Spark Fundamentals | Driver, Executor, Cluster, Worker Nodes, DAG, Lazy Evaluation, Actions vs Transformations |
| 4 | Spark Core API | SparkContext, SparkSession, Creating RDDs, Parallelize, Text File Input, Spark UI |
| 5 | RDD | Immutable Collections, Partitioning, Narrow vs Wide Transformations, Lineage, Fault Tolerance |
| 6 | RDD Transformations | map(), flatMap(), filter(), distinct(), union(), intersection(), sample(), cartesian() |
| 7 | RDD Actions | collect(), count(), first(), take(), reduce(), aggregate(), saveAsTextFile(), foreach() |
| 8 | Pair RDD | Key-Value Pair, reduceByKey(), groupByKey(), combineByKey(), sortByKey(), join(), cogroup() |
| 9 | Partitioning | HashPartitioner, RangePartitioner, Repartition, Coalesce, Partition Strategy, Skew Handling |
| 10 | Shared Variables | Broadcast Variables, Accumulators, Custom Accumulators, Performance Benefits |
| 11 | Spark SQL Introduction | DataFrames, Datasets, Schema, Catalyst Optimizer, Tungsten Engine |
| 12 | DataFrame Operations | select(), filter(), where(), alias(), withColumn(), drop(), distinct(), rename() |
| 13 | DataFrame Functions | Built-in Functions, String Functions, Date Functions, Math Functions, Conditional Functions |
| 14 | DataFrame Aggregation | groupBy(), agg(), sum(), avg(), count(), min(), max(), pivot() |
| 15 | DataFrame Join | Inner Join, Left Join, Right Join, Full Join, Cross Join, Broadcast Join |
| 16 | Window Functions | row_number(), rank(), dense_rank(), lag(), lead(), running totals, partitionBy() |
| 17 | Spark SQL | SQL Queries, Temporary Views, Global Views, SQL Optimization, Explain Plan |
| 18 | Reading Data Sources | CSV, JSON, Parquet, ORC, Avro, XML, JDBC |
| 19 | Writing Data Sources | Save Modes, Partitioned Output, Bucketing, Compression, JDBC Writes |
| 20 | Performance Optimization | Cache(), Persist(), Checkpoint, Serialization, Memory Management, Shuffle Optimization |
| 21 | Spark Performance Tuning | Executor Memory, Core Allocation, Dynamic Allocation, AQE, Broadcast Threshold |
| 22 | Spark Streaming | Streaming Concepts, DStreams, Structured Streaming, Sources, Sinks |
| 23 | Structured Streaming | Watermarking, Event Time, Processing Time, Output Modes, Checkpointing |
| 24 | Spark with Kafka | Kafka Integration, Read Streams, Write Streams, Consumer Groups, Offset Management |
| 25 | Spark with Delta Lake & Iceberg | ACID Tables, Time Travel, Merge, Update, Delete, Schema Evolution |
| 26 | Spark on Cloud | Azure Databricks, AWS EMR, Google Dataproc, Spark on Kubernetes, Cluster Deployment |
| 27 | Spark Testing | Unit Testing, DataFrame Testing, Mock Data, Debugging, Logging, Spark UI Analysis |
| 28 | Spark Security | Authentication, Authorization, Encryption, Kerberos, SSL, Data Governance |
| 29 | Real-Time Projects | ETL Pipeline, Log Processing, Banking Transactions, Recommendation Engine, Data Lake Processing |
| 30 | Interview Preparation | Architecture Questions, Coding Questions, Optimization Scenarios, Performance Debugging, Best Practices |
Interview question
| What is Apache Spark? |
| Why is Apache Spark used for big data processing? |
| What are the key features of Apache Spark? |
| What are the main components of Apache Spark? |
| What is Spark Core? |
| What is Spark SQL? |
| What is Spark Structured Streaming? |
| What is MLlib? |
| What is GraphX? |
| What is PySpark? |
| What is the difference between Spark and Hadoop MapReduce? |
| What is the Spark architecture? |
| What is a Spark Driver? |
| What is a Spark Executor? |
| What is a Spark Cluster Manager? |
| What is a Spark Application? |
| What is a Spark Job? |
| What is a Spark Stage? |
| What is a Spark Task? |
| How does Spark execute an application? |
| What is an RDD? |
| What are the characteristics of RDDs? |
| What is RDD lineage? |
| What is RDD fault tolerance? |
| What is lazy evaluation in Spark? |
| What are transformations in Spark? |
| What are actions in Spark? |
| What is the difference between transformations and actions? |
| What is a narrow transformation? |
| What is a wide transformation? |
| What is a shuffle in Spark? |
| Why is shuffle expensive in Spark? |
| What is partitioning in Spark? |
| What is a partition? |
| How does Spark determine the number of partitions? |
| What is repartition()? |
| What is coalesce()? |
| What is the difference between repartition() and coalesce()? |
| What is caching in Spark? |
| What is persistence in Spark? |
| What is the difference between cache() and persist()? |
| What are Spark storage levels? |
| What is a broadcast variable? |
| What is an accumulator? |
| What are broadcast joins? |
| What is a DataFrame in Spark? |
| What is a Dataset in Spark? |
| What is the difference between RDD, DataFrame, and Dataset? |
| Why are DataFrames preferred over RDDs? |
| What is Spark SQL? |
| What is the Catalyst Optimizer? |
| What is the Tungsten execution engine? |
| What is whole-stage code generation? |
| What is the Spark SQL execution plan? |
| What are logical and physical plans in Spark? |
| What is Adaptive Query Execution? |
| How does AQE improve Spark performance? |
| What is predicate pushdown? |
| What is column pruning? |
| What is partition pruning? |
| What is bucketing in Spark? |
| What is a window function in Spark SQL? |
| How do joins work in Spark? |
| What are the different join strategies in Spark? |
| What is a broadcast hash join? |
| What is a sort-merge join? |
| What is a shuffled hash join? |
| What is a Cartesian join? |
| How do you optimize joins in Spark? |
| What is data skew in Spark? |
| How do you identify data skew in Spark? |
| How do you handle data skew in Spark? |
| What is the salting technique in Spark? |
| What is Spark Structured Streaming? |
| How does Structured Streaming work? |
| What is a streaming query? |
| What is a trigger in Structured Streaming? |
| What is checkpointing in Spark Streaming? |
| What is watermarking in Structured Streaming? |
| What is event-time processing? |
| What is processing-time processing? |
| What is the difference between batch and streaming processing? |
| What is exactly-once processing in Spark? |
| How does Spark handle late-arriving data? |
| How does Spark integrate with Kafka? |
| How do you build a real-time Kafka-to-Spark pipeline? |
| How do you optimize Spark applications? |
| How do you tune Spark executor memory? |
| How do you tune Spark executor cores? |
| What is dynamic resource allocation in Spark? |
| What is the Spark UI? |
| How do you troubleshoot slow Spark jobs? |
| How do you diagnose out-of-memory errors in Spark? |
| How do you reduce shuffle operations in Spark? |
| How do you optimize PySpark applications? |
| What is Apache Arrow in PySpark? |
| How does PySpark communicate with the JVM? |
| What is a Pandas UDF in PySpark? |
| What is a Python UDF? |
| What is the difference between Python UDF and Pandas UDF? |
| How can Spark be used for machine learning? |
| What is Spark MLlib? |
| How can Spark process large AI training datasets? |
| How can Spark be integrated with MLflow? |
| How can Spark be used for feature engineering? |
| How can Spark support large-scale RAG pipelines? |
| How can Spark be used to process documents for RAG? |
| How can Spark generate datasets for LLM applications? |
| How can Spark be used for embedding generation pipelines? |
| How can Spark integrate with vector databases for AI applications? |
| How can Spark support large-scale Agentic AI data pipelines? |
| How can Spark process AI inference workloads at scale? |
| How would you design a production-grade Spark pipeline for AI and real-time analytics? |
