27 August 2026

#Azure Databricks


Key Concepts


S.No Topic Sub-Topics
1 Azure Databricks Fundamentals & Architecture Databricks workspace, Control Plane, Data Plane, Workspace architecture, Azure integration, DBFS, Databricks Runtime, compute architecture, metastore architecture, Unity Catalog architecture
2 Workspace & Environment Management Workspace creation, Workspace configuration, Multiple environments, Dev/Test/Prod, Resource groups, Azure regions, Private workspaces, Workspace networking, Workspace settings, Account-level management
3 Compute & Cluster Management All-purpose compute, Job clusters, Serverless compute, Cluster pools, Cluster policies, Autoscaling, Auto termination, Custom clusters, Single-node clusters, Compute configuration
4 Databricks Runtime LTS Runtime, ML Runtime, Photon Runtime, Runtime upgrades, Runtime compatibility, Runtime libraries, Custom environments, Runtime release management
5 Apache Spark on Databricks Spark architecture, Driver, Executors, SparkSession, DAG, Jobs, Stages, Tasks, Transformations, Actions, Lazy evaluation, Narrow vs wide transformations
6 Spark DataFrame & SQL Engineering DataFrames, Spark SQL, Temporary views, Global views, SQL functions, Window functions, Joins, Aggregations, CTEs, UDFs, Higher-order functions
7 PySpark Engineering PySpark DataFrames, Schema management, Column expressions, Built-in functions, UDF optimization, Pandas UDF, Vectorized UDF, Type hints, Error handling, Modular PySpark
8 Delta Lake Delta tables, ACID transactions, Transaction log, Schema enforcement, Schema evolution, Time travel, OPTIMIZE, VACUUM, RESTORE, MERGE, DELETE, UPDATE, Delta history
9 Delta Lake Advanced Delta Change Data Feed, Deletion vectors, Liquid clustering, Predictive optimization, Data skipping, Z-ordering, Column mapping, Generated columns, Identity columns, Delta protocol
10 Unity Catalog Catalogs, Schemas, Tables, Volumes, External locations, Storage credentials, Managed tables, External tables, Views, Functions, Models, Three-level namespace
11 Data Governance & Security RBAC, ACLs, Table permissions, Column-level security, Row-level security, Dynamic views, Data masking, ABAC, Service principals, Managed identities, Audit logs
12 Azure Data Lake Storage Gen2 Integration ADLS Gen2, Storage accounts, Containers, Access keys, SAS, OAuth, Managed identity, Service principals, External locations, Mount alternatives, Secure storage access
13 Data Ingestion Batch ingestion, Incremental ingestion, Auto Loader, CloudFiles, File notification mode, Directory listing mode, Schema inference, Schema evolution, Checkpointing
14 Streaming & Structured Streaming Structured Streaming, Streaming DataFrames, Checkpoints, Output modes, Watermarks, Stateful processing, Trigger modes, Stream-stream joins, Failure recovery
15 Kafka & Event Streaming Azure Event Hubs, Apache Kafka, Kafka source, Kafka sink, Consumer groups, Offsets, Partitions, Event-time processing, Exactly-once patterns, Kafka-to-Delta pipelines
16 Medallion Architecture Bronze layer, Silver layer, Gold layer, Raw data, Cleansing, Standardization, Business transformation, Aggregation, Data marts, Incremental processing
17 Lakehouse Architecture Data Lakehouse, Data Lake vs Data Warehouse, Delta Lakehouse, Open table formats, Data products, Data domains, Lakehouse reference architecture, Enterprise Lakehouse
18 Databricks SQL & Warehousing SQL Warehouse, Serverless SQL Warehouse, SQL endpoints, Query editor, Dashboards, Alerts, Query history, Query profiles, Materialized views, SQL functions
19 Performance Optimization Spark optimization, Partitioning, Repartition, Coalesce, Broadcast joins, AQE, Predicate pushdown, Data skipping, Caching, Photon, Shuffle optimization, Small-file optimization
20 Spark Memory & Troubleshooting Driver memory, Executor memory, Memory overhead, Garbage collection, OOM errors, Shuffle spill, Skewed joins, Long-running tasks, Executor failures, Spark UI analysis
21 Databricks Workflows & Orchestration Lakeflow Jobs, Jobs, Tasks, Task dependencies, Job parameters, Scheduling, Triggers, Retries, Repair runs, Job clusters, Conditional tasks, For-each tasks
22 Lakeflow Declarative Pipelines Lakeflow Declarative Pipelines, Declarative ETL, Streaming tables, Materialized views, Pipeline expectations, Data quality rules, Incremental processing, Pipeline orchestration
23 Data Quality & Observability Data quality checks, Expectations, Constraint validation, Null checks, Duplicate detection, Schema validation, Data lineage, Pipeline monitoring, Data freshness, SLA monitoring
24 CI/CD & DevOps Databricks Asset Bundles, Git integration, GitHub, Azure DevOps, Branching strategies, CI/CD pipelines, Environment promotion, Deployment automation, Infrastructure as Code
25 Databricks CLI & REST APIs Databricks CLI, REST API, Workspace API, Jobs API, Cluster API, SQL API, Unity Catalog API, Authentication, Service principals, Automation scripts
26 Infrastructure as Code Terraform, Databricks Terraform Provider, Azure Resource Manager, Workspace provisioning, Cluster policies, Unity Catalog resources, Service principals, Environment automation
27 Monitoring & Logging Spark UI, Ganglia metrics, Query profile, Cluster logs, Driver logs, Executor logs, Audit logs, Azure Monitor, Log Analytics, Application Insights, System tables
28 Security & Network Architecture VNet injection, Private Link, Private endpoints, NSGs, Secure cluster connectivity, Public network access, Firewall integration, Key Vault, Customer-managed keys, Encryption
29 ML & AI on Databricks Databricks Runtime ML, MLflow, Experiment tracking, Model registry, Feature engineering, Feature Store, Model serving, Vector Search, Mosaic AI, LLM applications
30 Production Architecture & Expert Practices Enterprise architecture, Multi-workspace strategy, Multi-environment architecture, Cost optimization, Reliability, Disaster recovery, High availability, Data lineage, Governance architecture, Production troubleshooting, Capacity planning

Interview question

What is Azure Databricks?
What are the core components of Azure Databricks architecture?
What is the difference between Azure Databricks Control Plane and Data Plane?
What is Databricks Runtime?
What is the significance of Databricks Runtime LTS?
What is Photon in Azure Databricks?
What is a Databricks workspace?
What is the purpose of a Databricks cluster?
What are the different compute options available in Azure Databricks?
What is the difference between all-purpose compute and job compute?
What is serverless compute in Azure Databricks?
What is a cluster policy?
What is cluster autoscaling?
What is cluster auto-termination?
What are cluster pools in Databricks?
What is Azure Databricks SQL Warehouse?
What is a serverless SQL warehouse?
What is Apache Spark in Azure Databricks?
What are Spark drivers and executors?
What is a Spark DAG?
What are Spark jobs, stages, and tasks?
What is lazy evaluation in Spark?
What are narrow and wide transformations?
What is Spark Adaptive Query Execution?
What is a Spark DataFrame?
What is Spark SQL?
What is PySpark?
What are the major advantages of PySpark in Databricks?
What is Delta Lake?
What are the key features of Delta Lake?
What is the Delta transaction log?
What is ACID transaction support in Delta Lake?
What is schema enforcement in Delta Lake?
What is schema evolution in Delta Lake?
What is Delta Lake time travel?
What is the Delta RESTORE command?
What is the MERGE operation in Delta Lake?
What is Delta Change Data Feed?
What are deletion vectors in Delta Lake?
What is liquid clustering in Delta Lake?
What is OPTIMIZE in Delta Lake?
What is VACUUM in Delta Lake?
What is Z-Ordering?
What is data skipping?
What is predictive optimization?
What is Unity Catalog?
What is the three-level namespace in Unity Catalog?
What are catalogs and schemas in Unity Catalog?
What are managed tables in Unity Catalog?
What are external tables in Unity Catalog?
What are external locations?
What are storage credentials?
What are Unity Catalog volumes?
What is data lineage in Unity Catalog?
What is fine-grained access control in Unity Catalog?
What is row-level security in Databricks?
What is column-level security in Databricks?
What is dynamic data masking?
What is ABAC in Unity Catalog?
What is Auto Loader?
What is the CloudFiles source in Databricks?
What is schema inference in Auto Loader?
What is schema evolution in Auto Loader?
What is file notification mode?
What is directory listing mode?
What is checkpointing in Auto Loader?
What is Structured Streaming?
What are output modes in Structured Streaming?
What is watermarking in Structured Streaming?
What is stateful stream processing?
What are streaming checkpoints?
What are trigger modes in Structured Streaming?
What are stream-stream joins?
What is Azure Event Hubs integration with Databricks?
What is Kafka integration with Databricks?
What is the Medallion Architecture?
What is the purpose of the Bronze layer?
What is the purpose of the Silver layer?
What is the purpose of the Gold layer?
What is Lakehouse architecture?
What is the difference between a Data Lake and Lakehouse architecture?
What is Lakeflow Declarative Pipelines?
What are streaming tables in Lakeflow?
What are materialized views in Lakeflow?
What are data quality expectations in Lakeflow?
What are Databricks Workflows?
What are tasks in Databricks Jobs?
What are task dependencies?
What are job parameters?
What are repair runs?
What are Databricks Asset Bundles?
What is Git integration in Databricks?
What is Databricks CLI?
What are Databricks REST APIs?
What is Terraform integration with Azure Databricks?
What is VNet injection?
What is Secure Cluster Connectivity?
What is Azure Private Link integration with Databricks?
What is Managed Identity integration with Azure Databricks?
What is Azure Key Vault integration?
What is Databricks audit logging?
What are system tables in Databricks?
What is Databricks SQL query profiling?
What is MLflow in Azure Databricks?
What is Mosaic AI?
What is Databricks Vector Search?
What is Model Serving in Databricks?
What is Feature Engineering in Databricks?
Why is Azure Databricks preferred for large-scale data engineering?
Why is Delta Lake important in a Lakehouse architecture?
Why does Delta Lake use a transaction log?
Why is ACID support important for data lakes?
Why should Delta tables be preferred over raw Parquet tables?
Why is schema enforcement important?
Why would you use schema evolution?
Why is Unity Catalog important for enterprise Databricks?
Why should managed tables be preferred in certain scenarios?
Why are external locations required in Unity Catalog?
Why is Auto Loader preferred for incremental file ingestion?
Why is checkpointing required for streaming workloads?
Why are watermarks required in streaming applications?
Why can streaming jobs accumulate state?
Why can Spark jobs suffer from data skew?
Why does a shuffle impact Spark performance?
Why are broadcast joins useful?
Why can excessive repartitioning reduce performance?
Why does Spark create many small files?
Why is OPTIMIZE required for some Delta workloads?
Why can excessive partitioning be harmful?
Why is Z-Ordering useful for selective queries?
Why is liquid clustering preferred for certain modern Delta workloads?
Why should VACUUM be used carefully?
Why is Photon useful for SQL workloads?
Why is Adaptive Query Execution important?
Why can driver memory become a bottleneck?
Why can executor memory become a bottleneck?
Why do Spark jobs experience out-of-memory errors?
Why should UDFs be avoided when built-in Spark functions exist?
Why are Pandas UDFs faster than traditional Python UDFs in many cases?
Why is the Medallion Architecture widely adopted?
Why should Bronze data generally remain immutable?
Why should business transformations generally occur in Silver or Gold?
Why is data quality important in production pipelines?
Why are Databricks Jobs preferred for scheduled workloads?
Why should job clusters be used for production batch processing?
Why is serverless compute useful?
Why should cluster policies be enforced?
Why is auto-termination important for cost optimization?
Why are system tables useful for platform monitoring?
Why is data lineage important for governance?
Why are service principals used for automation?
Why is CI/CD important for Databricks projects?
Why are Databricks Asset Bundles useful for deployments?
Why is Terraform useful for Databricks infrastructure?
Why should secrets not be hardcoded in notebooks?
Why should production workspaces be isolated from development?
Why is disaster recovery important for enterprise Databricks?
Why should Databricks costs be continuously monitored?
When should you use Azure Databricks instead of Azure Synapse?
When should you use serverless compute?
When should you use all-purpose compute?
When should you use job clusters?
When should you configure cluster autoscaling?
When should you use Photon?
When should you use Delta Lake?
When should you use Delta MERGE?
When should you use Delta Change Data Feed?
When should you use time travel?
When should you use RESTORE?
When should you run OPTIMIZE?
When should you use VACUUM?
When should you use Z-Ordering?
When should you use liquid clustering?
When should you use Auto Loader?
When should you use Structured Streaming?
When should you use batch processing instead of streaming?
When should you use watermarks?
When should you use stream-stream joins?
When should you use Kafka instead of file ingestion?
When should you use Azure Event Hubs?
When should you use Unity Catalog managed tables?
When should you use external tables?
When should you use external locations?
When should you use volumes?
When should you implement row-level security?
When should you implement column masking?
When should you use dynamic views?
When should you use ABAC?
When should you use broadcast joins?
When should you repartition a DataFrame?
When should you use coalesce?
When should you cache a DataFrame?
When should you use checkpointing?
When should you investigate Spark skew?
When should you increase executor memory?
When should you increase driver memory?
When should you use Lakeflow Declarative Pipelines?
When should you use materialized views?
When should you use streaming tables?
When should you use Databricks SQL Warehouses?
When should you use Databricks Workflows?
When should you use Databricks Asset Bundles?
When should you use Terraform?
When should you use Databricks REST APIs?
When should you use service principals?
When should you implement Private Link?
When should you use MLflow?
When should you use Vector Search?
Which Databricks Runtime should you select for production?
Which compute type is best for scheduled production jobs?
Which compute type is best for interactive development?
Which cluster policy settings are most important?
Which Spark configuration parameters commonly require tuning?
Which join strategy is best for a small lookup table?
Which optimization technique is best for data skew?
Which Delta optimization technique should be used for frequently filtered columns?
Which Delta operation should be used for upsert processing?
Which Delta feature supports incremental change extraction?
Which Auto Loader mode is suitable for large-scale file ingestion?
Which schema evolution strategy should be used for changing source schemas?
Which streaming trigger should be used for continuous processing requirements?
Which output mode should be used for aggregation streams?
Which checkpoint location strategy should be used in production?
Which Azure storage authentication mechanism is preferred for enterprise workloads?
Which Unity Catalog object should be used to control access to ADLS?
Which Unity Catalog object should be used for file-based data access?
Which table type is preferable for centrally governed data?
Which security mechanism should be used for sensitive columns?
Which security mechanism should be used for tenant-based row filtering?
Which architecture is best for Bronze, Silver, and Gold processing?
Which Databricks feature provides data lineage?
Which Databricks feature provides audit information?
Which system tables are useful for platform monitoring?
Which workflow feature should be used for task dependencies?
Which workflow feature should be used for failure retries?
Which deployment approach is recommended for multi-environment Databricks?
Which CI/CD approach is suitable for Databricks notebooks and jobs?
Which Infrastructure-as-Code tool is commonly used with Databricks?
Which authentication mechanism should automated deployments use?
Which networking configuration provides private connectivity?
Which Azure service can securely store Databricks secrets?
Which Spark UI section should you inspect for slow stages?
Which metrics indicate executor memory pressure?
Which technique reduces small-file problems?
Which technique reduces unnecessary shuffle?
Which technique improves selective Delta queries?
Which technique can dynamically optimize Spark joins?
Which Databricks feature supports machine learning experiment tracking?
Which Databricks feature supports model serving?
Which Databricks feature supports vector-based retrieval?
Which architecture is suitable for real-time Kafka-to-Delta pipelines?
Which ingestion approach is best for millions of files arriving incrementally?
Which Databricks feature is best for declarative ETL?
Which feature is best for SQL-based BI workloads?
Which approach is best for disaster recovery across Azure regions?
Which governance model is best for multiple business domains?
How does Azure Databricks integrate with Azure Data Lake Storage Gen2?
How does Databricks authentication work with Azure resources?
How does Unity Catalog enforce data access?
How does the three-level namespace work?
How does Delta Lake provide ACID transactions?
How does the Delta transaction log work?
How does Delta time travel work?
How does Delta MERGE process upserts?
How does Change Data Feed work?
How does liquid clustering improve Delta performance?
How does Auto Loader detect newly arriving files?
How does Auto Loader handle schema evolution?
How does Structured Streaming achieve fault tolerance?
How does watermarking control streaming state?
How does Spark handle shuffle operations?
How does Adaptive Query Execution optimize Spark queries?
How does a broadcast join improve performance?
How do you identify data skew in Spark?
How do you optimize a slow Spark job?
How do you troubleshoot executor out-of-memory errors?
How do you troubleshoot driver out-of-memory errors?
How do you reduce small files in Delta tables?
How do you optimize a large Delta MERGE operation?
How do you optimize a highly selective Delta query?
How do you design an incremental ingestion pipeline?
How do you implement Bronze-to-Silver transformations?
How do you implement Silver-to-Gold transformations?
How do you implement data quality checks?
How do you quarantine bad records?
How do you implement row-level security?
How do you implement column-level masking?
How do you securely access ADLS from Databricks?
How do you configure Unity Catalog external locations?
How do you implement CI/CD for Databricks?
How do you deploy Databricks Jobs across environments?
How do you use Databricks Asset Bundles?
How do you provision Databricks infrastructure using Terraform?
How do you automate Databricks using REST APIs?
How do you authenticate CI/CD pipelines securely?
How do you monitor Databricks cluster performance?
How do you monitor pipeline failures?
How do you analyze Spark UI for performance problems?
How do you optimize Databricks compute costs?
How do you design a multi-environment Databricks architecture?
How do you design a multi-workspace enterprise architecture?
How do you implement disaster recovery for Databricks?
How do you integrate Kafka with Structured Streaming?
How do you implement Kafka-to-Delta ingestion?
How do you implement MLflow model lifecycle management?
How do you design a production-grade Lakehouse?
Where does Azure Databricks store Delta transaction logs?
Where should Bronze data be stored?
Where should Silver data be stored?
Where should Gold data be stored?
Where should checkpoints be stored for streaming pipelines?
Where should Auto Loader schema information be stored?
Where should Unity Catalog external locations point?
Where are Unity Catalog managed tables stored?
Where are Databricks cluster logs available?
Where can Spark job performance be analyzed?
Where can Databricks SQL query performance be analyzed?
Where can Databricks audit events be accessed?
Where can Unity Catalog lineage be viewed?
Where should secrets be stored?
Where should service principal credentials be managed?
Where should Terraform state be maintained?
Where should Databricks Asset Bundles be executed?
Where should CI/CD deployment logic reside?
Where should production notebooks be maintained?
Where should reusable PySpark code be maintained?
Where should shared libraries be managed?
Where should data quality failures be captured?
Where should rejected records be stored?
Where should streaming state be maintained?
Where should Kafka offsets be managed?
Where should sensitive data be masked?
Where should row-level security logic be implemented?
Where should production monitoring metrics be collected?
Where should Databricks system-table data be analyzed?
Where should cluster policies be configured?
Where should compute permissions be controlled?
Where should Unity Catalog permissions be granted?
Where should storage credentials be configured?
Where should private endpoints be configured?
Where should Key Vault integration be configured?
Where should disaster recovery resources be deployed?
Where should replicated Delta data be maintained?
Where should MLflow experiments be tracked?
Where should registered models be governed?
Where should Vector Search indexes be maintained?
Where should SQL dashboards be created?
Where should Databricks Jobs be scheduled?
Where should workflow notifications be configured?
Where should pipeline alerts be configured?
Where should cost monitoring be performed?
Where should enterprise data governance policies be defined?
Where should development, testing, and production resources be separated?
Where should Azure Databricks networking controls be implemented?
Where should application-level data validation occur?
What is the difference between Azure Databricks and Azure Synapse?
What is the difference between Data Lake and Lakehouse?
What is the difference between managed and external Delta tables?
What is the difference between Unity Catalog and Hive Metastore?
What is the difference between all-purpose and job compute?
What is the difference between serverless and classic compute?
What is the difference between partitioning and clustering?
What is the difference between Z-Ordering and liquid clustering?
What is the difference between OPTIMIZE and VACUUM?
What is the difference between schema enforcement and schema evolution?
What is the difference between batch and streaming processing?
What is the difference between Auto Loader and COPY INTO?
What is the difference between directory listing and file notification modes?
What is the difference between checkpointing and watermarking?
What is the difference between broadcast join and shuffle join?
What is the difference between repartition and coalesce?
What is the difference between cache and persist?
What is the difference between Databricks Jobs and Lakeflow Declarative Pipelines?
What is the difference between Databricks Asset Bundles and Terraform?
What is the difference between row-level security and column-level masking?
Scenario: A Delta table contains billions of records and queries filtering by customer_id are slow. How would you optimize it?
Scenario: A daily Spark job suddenly takes three times longer than normal. How would you investigate it?
Scenario: A Spark join is causing severe data skew. How would you identify and resolve it?
Scenario: A Databricks job fails with executor out-of-memory errors. What steps would you take?
Scenario: A driver crashes while collecting a large DataFrame. How would you redesign the processing?
Scenario: Millions of small files arrive in ADLS every hour. How would you design the ingestion pipeline?
Scenario: Source files arrive continuously with evolving schemas. How would you implement ingestion using Auto Loader?
Scenario: A Kafka topic contains high-volume financial transactions. How would you build a reliable Kafka-to-Delta pipeline?
Scenario: Streaming data arrives several hours late. How would you handle late-arriving records?
Scenario: A streaming query is accumulating excessive state. How would you troubleshoot and optimize it?
Scenario: A business requires near-real-time Bronze, Silver, and Gold processing. How would you design the architecture?
Scenario: Multiple business units need isolated access to shared data. How would you design Unity Catalog governance?
Scenario: A company needs different data visibility for different regions. How would you implement row-level security?
Scenario: Sensitive customer information must be hidden from most users. How would you implement column-level protection?
Scenario: Developers are creating expensive clusters without restrictions. How would you control compute usage?
Scenario: Databricks costs have increased significantly. How would you identify and reduce unnecessary costs?
Scenario: Production Databricks deployments must be fully automated. How would you design the CI/CD architecture?
Scenario: The same Databricks application must run across Dev, QA, and Prod. How would you manage environment-specific configuration?
Scenario: An organization wants all Databricks infrastructure managed as code. How would you implement Terraform?
Scenario: A production pipeline fails halfway through execution. How would you design recovery and retry handling?
Scenario: A Delta MERGE processing billions of records is extremely slow. How would you optimize it?
Scenario: A critical Delta table was accidentally modified. How would you recover the previous version?
Scenario: A company requires private-only access to Azure Databricks. How would you design the network architecture?
Scenario: An enterprise requires disaster recovery for Databricks across Azure regions. How would you design the solution?
Scenario: BI users require fast interactive SQL access over a large Lakehouse. How would you design the Databricks SQL architecture?

Related Topics