Showing posts with label Data Science. Show all posts
Showing posts with label Data Science. Show all posts

27 August 2026

#Azure Databricks


Key Concepts


S.No Topic Sub-Topics
1 Azure Databricks Fundamentals & Architecture Databricks workspace, Control Plane, Data Plane, Workspace architecture, Azure integration, DBFS, Databricks Runtime, compute architecture, metastore architecture, Unity Catalog architecture
2 Workspace & Environment Management Workspace creation, Workspace configuration, Multiple environments, Dev/Test/Prod, Resource groups, Azure regions, Private workspaces, Workspace networking, Workspace settings, Account-level management
3 Compute & Cluster Management All-purpose compute, Job clusters, Serverless compute, Cluster pools, Cluster policies, Autoscaling, Auto termination, Custom clusters, Single-node clusters, Compute configuration
4 Databricks Runtime LTS Runtime, ML Runtime, Photon Runtime, Runtime upgrades, Runtime compatibility, Runtime libraries, Custom environments, Runtime release management
5 Apache Spark on Databricks Spark architecture, Driver, Executors, SparkSession, DAG, Jobs, Stages, Tasks, Transformations, Actions, Lazy evaluation, Narrow vs wide transformations
6 Spark DataFrame & SQL Engineering DataFrames, Spark SQL, Temporary views, Global views, SQL functions, Window functions, Joins, Aggregations, CTEs, UDFs, Higher-order functions
7 PySpark Engineering PySpark DataFrames, Schema management, Column expressions, Built-in functions, UDF optimization, Pandas UDF, Vectorized UDF, Type hints, Error handling, Modular PySpark
8 Delta Lake Delta tables, ACID transactions, Transaction log, Schema enforcement, Schema evolution, Time travel, OPTIMIZE, VACUUM, RESTORE, MERGE, DELETE, UPDATE, Delta history
9 Delta Lake Advanced Delta Change Data Feed, Deletion vectors, Liquid clustering, Predictive optimization, Data skipping, Z-ordering, Column mapping, Generated columns, Identity columns, Delta protocol
10 Unity Catalog Catalogs, Schemas, Tables, Volumes, External locations, Storage credentials, Managed tables, External tables, Views, Functions, Models, Three-level namespace
11 Data Governance & Security RBAC, ACLs, Table permissions, Column-level security, Row-level security, Dynamic views, Data masking, ABAC, Service principals, Managed identities, Audit logs
12 Azure Data Lake Storage Gen2 Integration ADLS Gen2, Storage accounts, Containers, Access keys, SAS, OAuth, Managed identity, Service principals, External locations, Mount alternatives, Secure storage access
13 Data Ingestion Batch ingestion, Incremental ingestion, Auto Loader, CloudFiles, File notification mode, Directory listing mode, Schema inference, Schema evolution, Checkpointing
14 Streaming & Structured Streaming Structured Streaming, Streaming DataFrames, Checkpoints, Output modes, Watermarks, Stateful processing, Trigger modes, Stream-stream joins, Failure recovery
15 Kafka & Event Streaming Azure Event Hubs, Apache Kafka, Kafka source, Kafka sink, Consumer groups, Offsets, Partitions, Event-time processing, Exactly-once patterns, Kafka-to-Delta pipelines
16 Medallion Architecture Bronze layer, Silver layer, Gold layer, Raw data, Cleansing, Standardization, Business transformation, Aggregation, Data marts, Incremental processing
17 Lakehouse Architecture Data Lakehouse, Data Lake vs Data Warehouse, Delta Lakehouse, Open table formats, Data products, Data domains, Lakehouse reference architecture, Enterprise Lakehouse
18 Databricks SQL & Warehousing SQL Warehouse, Serverless SQL Warehouse, SQL endpoints, Query editor, Dashboards, Alerts, Query history, Query profiles, Materialized views, SQL functions
19 Performance Optimization Spark optimization, Partitioning, Repartition, Coalesce, Broadcast joins, AQE, Predicate pushdown, Data skipping, Caching, Photon, Shuffle optimization, Small-file optimization
20 Spark Memory & Troubleshooting Driver memory, Executor memory, Memory overhead, Garbage collection, OOM errors, Shuffle spill, Skewed joins, Long-running tasks, Executor failures, Spark UI analysis
21 Databricks Workflows & Orchestration Lakeflow Jobs, Jobs, Tasks, Task dependencies, Job parameters, Scheduling, Triggers, Retries, Repair runs, Job clusters, Conditional tasks, For-each tasks
22 Lakeflow Declarative Pipelines Lakeflow Declarative Pipelines, Declarative ETL, Streaming tables, Materialized views, Pipeline expectations, Data quality rules, Incremental processing, Pipeline orchestration
23 Data Quality & Observability Data quality checks, Expectations, Constraint validation, Null checks, Duplicate detection, Schema validation, Data lineage, Pipeline monitoring, Data freshness, SLA monitoring
24 CI/CD & DevOps Databricks Asset Bundles, Git integration, GitHub, Azure DevOps, Branching strategies, CI/CD pipelines, Environment promotion, Deployment automation, Infrastructure as Code
25 Databricks CLI & REST APIs Databricks CLI, REST API, Workspace API, Jobs API, Cluster API, SQL API, Unity Catalog API, Authentication, Service principals, Automation scripts
26 Infrastructure as Code Terraform, Databricks Terraform Provider, Azure Resource Manager, Workspace provisioning, Cluster policies, Unity Catalog resources, Service principals, Environment automation
27 Monitoring & Logging Spark UI, Ganglia metrics, Query profile, Cluster logs, Driver logs, Executor logs, Audit logs, Azure Monitor, Log Analytics, Application Insights, System tables
28 Security & Network Architecture VNet injection, Private Link, Private endpoints, NSGs, Secure cluster connectivity, Public network access, Firewall integration, Key Vault, Customer-managed keys, Encryption
29 ML & AI on Databricks Databricks Runtime ML, MLflow, Experiment tracking, Model registry, Feature engineering, Feature Store, Model serving, Vector Search, Mosaic AI, LLM applications
30 Production Architecture & Expert Practices Enterprise architecture, Multi-workspace strategy, Multi-environment architecture, Cost optimization, Reliability, Disaster recovery, High availability, Data lineage, Governance architecture, Production troubleshooting, Capacity planning

Interview question

What is Azure Databricks?
What are the core components of Azure Databricks architecture?
What is the difference between Azure Databricks Control Plane and Data Plane?
What is Databricks Runtime?
What is the significance of Databricks Runtime LTS?
What is Photon in Azure Databricks?
What is a Databricks workspace?
What is the purpose of a Databricks cluster?
What are the different compute options available in Azure Databricks?
What is the difference between all-purpose compute and job compute?
What is serverless compute in Azure Databricks?
What is a cluster policy?
What is cluster autoscaling?
What is cluster auto-termination?
What are cluster pools in Databricks?
What is Azure Databricks SQL Warehouse?
What is a serverless SQL warehouse?
What is Apache Spark in Azure Databricks?
What are Spark drivers and executors?
What is a Spark DAG?
What are Spark jobs, stages, and tasks?
What is lazy evaluation in Spark?
What are narrow and wide transformations?
What is Spark Adaptive Query Execution?
What is a Spark DataFrame?
What is Spark SQL?
What is PySpark?
What are the major advantages of PySpark in Databricks?
What is Delta Lake?
What are the key features of Delta Lake?
What is the Delta transaction log?
What is ACID transaction support in Delta Lake?
What is schema enforcement in Delta Lake?
What is schema evolution in Delta Lake?
What is Delta Lake time travel?
What is the Delta RESTORE command?
What is the MERGE operation in Delta Lake?
What is Delta Change Data Feed?
What are deletion vectors in Delta Lake?
What is liquid clustering in Delta Lake?
What is OPTIMIZE in Delta Lake?
What is VACUUM in Delta Lake?
What is Z-Ordering?
What is data skipping?
What is predictive optimization?
What is Unity Catalog?
What is the three-level namespace in Unity Catalog?
What are catalogs and schemas in Unity Catalog?
What are managed tables in Unity Catalog?
What are external tables in Unity Catalog?
What are external locations?
What are storage credentials?
What are Unity Catalog volumes?
What is data lineage in Unity Catalog?
What is fine-grained access control in Unity Catalog?
What is row-level security in Databricks?
What is column-level security in Databricks?
What is dynamic data masking?
What is ABAC in Unity Catalog?
What is Auto Loader?
What is the CloudFiles source in Databricks?
What is schema inference in Auto Loader?
What is schema evolution in Auto Loader?
What is file notification mode?
What is directory listing mode?
What is checkpointing in Auto Loader?
What is Structured Streaming?
What are output modes in Structured Streaming?
What is watermarking in Structured Streaming?
What is stateful stream processing?
What are streaming checkpoints?
What are trigger modes in Structured Streaming?
What are stream-stream joins?
What is Azure Event Hubs integration with Databricks?
What is Kafka integration with Databricks?
What is the Medallion Architecture?
What is the purpose of the Bronze layer?
What is the purpose of the Silver layer?
What is the purpose of the Gold layer?
What is Lakehouse architecture?
What is the difference between a Data Lake and Lakehouse architecture?
What is Lakeflow Declarative Pipelines?
What are streaming tables in Lakeflow?
What are materialized views in Lakeflow?
What are data quality expectations in Lakeflow?
What are Databricks Workflows?
What are tasks in Databricks Jobs?
What are task dependencies?
What are job parameters?
What are repair runs?
What are Databricks Asset Bundles?
What is Git integration in Databricks?
What is Databricks CLI?
What are Databricks REST APIs?
What is Terraform integration with Azure Databricks?
What is VNet injection?
What is Secure Cluster Connectivity?
What is Azure Private Link integration with Databricks?
What is Managed Identity integration with Azure Databricks?
What is Azure Key Vault integration?
What is Databricks audit logging?
What are system tables in Databricks?
What is Databricks SQL query profiling?
What is MLflow in Azure Databricks?
What is Mosaic AI?
What is Databricks Vector Search?
What is Model Serving in Databricks?
What is Feature Engineering in Databricks?
Why is Azure Databricks preferred for large-scale data engineering?
Why is Delta Lake important in a Lakehouse architecture?
Why does Delta Lake use a transaction log?
Why is ACID support important for data lakes?
Why should Delta tables be preferred over raw Parquet tables?
Why is schema enforcement important?
Why would you use schema evolution?
Why is Unity Catalog important for enterprise Databricks?
Why should managed tables be preferred in certain scenarios?
Why are external locations required in Unity Catalog?
Why is Auto Loader preferred for incremental file ingestion?
Why is checkpointing required for streaming workloads?
Why are watermarks required in streaming applications?
Why can streaming jobs accumulate state?
Why can Spark jobs suffer from data skew?
Why does a shuffle impact Spark performance?
Why are broadcast joins useful?
Why can excessive repartitioning reduce performance?
Why does Spark create many small files?
Why is OPTIMIZE required for some Delta workloads?
Why can excessive partitioning be harmful?
Why is Z-Ordering useful for selective queries?
Why is liquid clustering preferred for certain modern Delta workloads?
Why should VACUUM be used carefully?
Why is Photon useful for SQL workloads?
Why is Adaptive Query Execution important?
Why can driver memory become a bottleneck?
Why can executor memory become a bottleneck?
Why do Spark jobs experience out-of-memory errors?
Why should UDFs be avoided when built-in Spark functions exist?
Why are Pandas UDFs faster than traditional Python UDFs in many cases?
Why is the Medallion Architecture widely adopted?
Why should Bronze data generally remain immutable?
Why should business transformations generally occur in Silver or Gold?
Why is data quality important in production pipelines?
Why are Databricks Jobs preferred for scheduled workloads?
Why should job clusters be used for production batch processing?
Why is serverless compute useful?
Why should cluster policies be enforced?
Why is auto-termination important for cost optimization?
Why are system tables useful for platform monitoring?
Why is data lineage important for governance?
Why are service principals used for automation?
Why is CI/CD important for Databricks projects?
Why are Databricks Asset Bundles useful for deployments?
Why is Terraform useful for Databricks infrastructure?
Why should secrets not be hardcoded in notebooks?
Why should production workspaces be isolated from development?
Why is disaster recovery important for enterprise Databricks?
Why should Databricks costs be continuously monitored?
When should you use Azure Databricks instead of Azure Synapse?
When should you use serverless compute?
When should you use all-purpose compute?
When should you use job clusters?
When should you configure cluster autoscaling?
When should you use Photon?
When should you use Delta Lake?
When should you use Delta MERGE?
When should you use Delta Change Data Feed?
When should you use time travel?
When should you use RESTORE?
When should you run OPTIMIZE?
When should you use VACUUM?
When should you use Z-Ordering?
When should you use liquid clustering?
When should you use Auto Loader?
When should you use Structured Streaming?
When should you use batch processing instead of streaming?
When should you use watermarks?
When should you use stream-stream joins?
When should you use Kafka instead of file ingestion?
When should you use Azure Event Hubs?
When should you use Unity Catalog managed tables?
When should you use external tables?
When should you use external locations?
When should you use volumes?
When should you implement row-level security?
When should you implement column masking?
When should you use dynamic views?
When should you use ABAC?
When should you use broadcast joins?
When should you repartition a DataFrame?
When should you use coalesce?
When should you cache a DataFrame?
When should you use checkpointing?
When should you investigate Spark skew?
When should you increase executor memory?
When should you increase driver memory?
When should you use Lakeflow Declarative Pipelines?
When should you use materialized views?
When should you use streaming tables?
When should you use Databricks SQL Warehouses?
When should you use Databricks Workflows?
When should you use Databricks Asset Bundles?
When should you use Terraform?
When should you use Databricks REST APIs?
When should you use service principals?
When should you implement Private Link?
When should you use MLflow?
When should you use Vector Search?
Which Databricks Runtime should you select for production?
Which compute type is best for scheduled production jobs?
Which compute type is best for interactive development?
Which cluster policy settings are most important?
Which Spark configuration parameters commonly require tuning?
Which join strategy is best for a small lookup table?
Which optimization technique is best for data skew?
Which Delta optimization technique should be used for frequently filtered columns?
Which Delta operation should be used for upsert processing?
Which Delta feature supports incremental change extraction?
Which Auto Loader mode is suitable for large-scale file ingestion?
Which schema evolution strategy should be used for changing source schemas?
Which streaming trigger should be used for continuous processing requirements?
Which output mode should be used for aggregation streams?
Which checkpoint location strategy should be used in production?
Which Azure storage authentication mechanism is preferred for enterprise workloads?
Which Unity Catalog object should be used to control access to ADLS?
Which Unity Catalog object should be used for file-based data access?
Which table type is preferable for centrally governed data?
Which security mechanism should be used for sensitive columns?
Which security mechanism should be used for tenant-based row filtering?
Which architecture is best for Bronze, Silver, and Gold processing?
Which Databricks feature provides data lineage?
Which Databricks feature provides audit information?
Which system tables are useful for platform monitoring?
Which workflow feature should be used for task dependencies?
Which workflow feature should be used for failure retries?
Which deployment approach is recommended for multi-environment Databricks?
Which CI/CD approach is suitable for Databricks notebooks and jobs?
Which Infrastructure-as-Code tool is commonly used with Databricks?
Which authentication mechanism should automated deployments use?
Which networking configuration provides private connectivity?
Which Azure service can securely store Databricks secrets?
Which Spark UI section should you inspect for slow stages?
Which metrics indicate executor memory pressure?
Which technique reduces small-file problems?
Which technique reduces unnecessary shuffle?
Which technique improves selective Delta queries?
Which technique can dynamically optimize Spark joins?
Which Databricks feature supports machine learning experiment tracking?
Which Databricks feature supports model serving?
Which Databricks feature supports vector-based retrieval?
Which architecture is suitable for real-time Kafka-to-Delta pipelines?
Which ingestion approach is best for millions of files arriving incrementally?
Which Databricks feature is best for declarative ETL?
Which feature is best for SQL-based BI workloads?
Which approach is best for disaster recovery across Azure regions?
Which governance model is best for multiple business domains?
How does Azure Databricks integrate with Azure Data Lake Storage Gen2?
How does Databricks authentication work with Azure resources?
How does Unity Catalog enforce data access?
How does the three-level namespace work?
How does Delta Lake provide ACID transactions?
How does the Delta transaction log work?
How does Delta time travel work?
How does Delta MERGE process upserts?
How does Change Data Feed work?
How does liquid clustering improve Delta performance?
How does Auto Loader detect newly arriving files?
How does Auto Loader handle schema evolution?
How does Structured Streaming achieve fault tolerance?
How does watermarking control streaming state?
How does Spark handle shuffle operations?
How does Adaptive Query Execution optimize Spark queries?
How does a broadcast join improve performance?
How do you identify data skew in Spark?
How do you optimize a slow Spark job?
How do you troubleshoot executor out-of-memory errors?
How do you troubleshoot driver out-of-memory errors?
How do you reduce small files in Delta tables?
How do you optimize a large Delta MERGE operation?
How do you optimize a highly selective Delta query?
How do you design an incremental ingestion pipeline?
How do you implement Bronze-to-Silver transformations?
How do you implement Silver-to-Gold transformations?
How do you implement data quality checks?
How do you quarantine bad records?
How do you implement row-level security?
How do you implement column-level masking?
How do you securely access ADLS from Databricks?
How do you configure Unity Catalog external locations?
How do you implement CI/CD for Databricks?
How do you deploy Databricks Jobs across environments?
How do you use Databricks Asset Bundles?
How do you provision Databricks infrastructure using Terraform?
How do you automate Databricks using REST APIs?
How do you authenticate CI/CD pipelines securely?
How do you monitor Databricks cluster performance?
How do you monitor pipeline failures?
How do you analyze Spark UI for performance problems?
How do you optimize Databricks compute costs?
How do you design a multi-environment Databricks architecture?
How do you design a multi-workspace enterprise architecture?
How do you implement disaster recovery for Databricks?
How do you integrate Kafka with Structured Streaming?
How do you implement Kafka-to-Delta ingestion?
How do you implement MLflow model lifecycle management?
How do you design a production-grade Lakehouse?
Where does Azure Databricks store Delta transaction logs?
Where should Bronze data be stored?
Where should Silver data be stored?
Where should Gold data be stored?
Where should checkpoints be stored for streaming pipelines?
Where should Auto Loader schema information be stored?
Where should Unity Catalog external locations point?
Where are Unity Catalog managed tables stored?
Where are Databricks cluster logs available?
Where can Spark job performance be analyzed?
Where can Databricks SQL query performance be analyzed?
Where can Databricks audit events be accessed?
Where can Unity Catalog lineage be viewed?
Where should secrets be stored?
Where should service principal credentials be managed?
Where should Terraform state be maintained?
Where should Databricks Asset Bundles be executed?
Where should CI/CD deployment logic reside?
Where should production notebooks be maintained?
Where should reusable PySpark code be maintained?
Where should shared libraries be managed?
Where should data quality failures be captured?
Where should rejected records be stored?
Where should streaming state be maintained?
Where should Kafka offsets be managed?
Where should sensitive data be masked?
Where should row-level security logic be implemented?
Where should production monitoring metrics be collected?
Where should Databricks system-table data be analyzed?
Where should cluster policies be configured?
Where should compute permissions be controlled?
Where should Unity Catalog permissions be granted?
Where should storage credentials be configured?
Where should private endpoints be configured?
Where should Key Vault integration be configured?
Where should disaster recovery resources be deployed?
Where should replicated Delta data be maintained?
Where should MLflow experiments be tracked?
Where should registered models be governed?
Where should Vector Search indexes be maintained?
Where should SQL dashboards be created?
Where should Databricks Jobs be scheduled?
Where should workflow notifications be configured?
Where should pipeline alerts be configured?
Where should cost monitoring be performed?
Where should enterprise data governance policies be defined?
Where should development, testing, and production resources be separated?
Where should Azure Databricks networking controls be implemented?
Where should application-level data validation occur?
What is the difference between Azure Databricks and Azure Synapse?
What is the difference between Data Lake and Lakehouse?
What is the difference between managed and external Delta tables?
What is the difference between Unity Catalog and Hive Metastore?
What is the difference between all-purpose and job compute?
What is the difference between serverless and classic compute?
What is the difference between partitioning and clustering?
What is the difference between Z-Ordering and liquid clustering?
What is the difference between OPTIMIZE and VACUUM?
What is the difference between schema enforcement and schema evolution?
What is the difference between batch and streaming processing?
What is the difference between Auto Loader and COPY INTO?
What is the difference between directory listing and file notification modes?
What is the difference between checkpointing and watermarking?
What is the difference between broadcast join and shuffle join?
What is the difference between repartition and coalesce?
What is the difference between cache and persist?
What is the difference between Databricks Jobs and Lakeflow Declarative Pipelines?
What is the difference between Databricks Asset Bundles and Terraform?
What is the difference between row-level security and column-level masking?
Scenario: A Delta table contains billions of records and queries filtering by customer_id are slow. How would you optimize it?
Scenario: A daily Spark job suddenly takes three times longer than normal. How would you investigate it?
Scenario: A Spark join is causing severe data skew. How would you identify and resolve it?
Scenario: A Databricks job fails with executor out-of-memory errors. What steps would you take?
Scenario: A driver crashes while collecting a large DataFrame. How would you redesign the processing?
Scenario: Millions of small files arrive in ADLS every hour. How would you design the ingestion pipeline?
Scenario: Source files arrive continuously with evolving schemas. How would you implement ingestion using Auto Loader?
Scenario: A Kafka topic contains high-volume financial transactions. How would you build a reliable Kafka-to-Delta pipeline?
Scenario: Streaming data arrives several hours late. How would you handle late-arriving records?
Scenario: A streaming query is accumulating excessive state. How would you troubleshoot and optimize it?
Scenario: A business requires near-real-time Bronze, Silver, and Gold processing. How would you design the architecture?
Scenario: Multiple business units need isolated access to shared data. How would you design Unity Catalog governance?
Scenario: A company needs different data visibility for different regions. How would you implement row-level security?
Scenario: Sensitive customer information must be hidden from most users. How would you implement column-level protection?
Scenario: Developers are creating expensive clusters without restrictions. How would you control compute usage?
Scenario: Databricks costs have increased significantly. How would you identify and reduce unnecessary costs?
Scenario: Production Databricks deployments must be fully automated. How would you design the CI/CD architecture?
Scenario: The same Databricks application must run across Dev, QA, and Prod. How would you manage environment-specific configuration?
Scenario: An organization wants all Databricks infrastructure managed as code. How would you implement Terraform?
Scenario: A production pipeline fails halfway through execution. How would you design recovery and retry handling?
Scenario: A Delta MERGE processing billions of records is extremely slow. How would you optimize it?
Scenario: A critical Delta table was accidentally modified. How would you recover the previous version?
Scenario: A company requires private-only access to Azure Databricks. How would you design the network architecture?
Scenario: An enterprise requires disaster recovery for Databricks across Azure regions. How would you design the solution?
Scenario: BI users require fast interactive SQL access over a large Lakehouse. How would you design the Databricks SQL architecture?

Related Topics


16 March 2026

#Apache Iceberg

#Apache Iceberg

Key Concepts


S.No Topic Sub-Topics
1Apache Iceberg Data Lake vs Data Warehouse, Table Format Concept, Iceberg Architecture Overview, Iceberg vs Hive Tables
2Iceberg EcosystemIceberg with Apache Spark, Apache Flink, Trino/Presto, Hive, Databricks
4Iceberg Table ConceptsTable Metadata, Data Files, Manifest Files, Manifest Lists, Snapshot Concept
5Catalogs in IcebergHadoop Catalog, Hive Metastore Catalog, REST Catalog, Glue Catalog, Catalog Configuration
6Creating Iceberg TablesSQL Table Creation, Spark API Table Creation, Table Properties, Table Location, Schema Definition
7Schema EvolutionAdd Columns, Rename Columns, Delete Columns, Change Column Types, Backward Compatibility
8PartitioningHidden Partitioning, Partition Transforms, Partition Spec Evolution, Partition Pruning, Best Practices
9Data OperationsInsert Data, Overwrite Data, Append Mode, Batch Writes, Streaming Writes
10Querying Iceberg TablesSQL Queries, Spark SQL Queries, Filtering Data, Aggregation Queries, Query Optimization
11Time TravelSnapshot IDs, Querying Past Data, Snapshot Retention, Use Cases, SQL Examples
12SnapshotsSnapshot Creation, Snapshot Metadata, Snapshot Expiration, Snapshot Rollback, Snapshot History
13Data VersioningVersioned Tables, Commit Operations, Metadata Tracking, Branching Concepts, Tagging Data Versions
14ACID TransactionsAtomic Operations, Consistency Guarantees, Isolation Levels, Concurrent Writes, Failure Recovery
15File FormatsParquet Support, ORC Support, Avro Support, Compression Options, File Size Optimization
16Manifest FilesManifest Structure, Metadata Tracking, File Listings, Performance Benefits, Manifest Merging
17Metadata ManagementMetadata Files, Metadata Evolution, Metadata Size Optimization, Metadata Cleanup, Metadata Storage
18Data CompactionSmall File Problem, Compaction Strategies, Rewrite Data Files, Optimize Table, Scheduling Compaction
19Delete OperationsEquality Deletes, Position Deletes, Delete Files, Row-Level Deletes, Delete Performance
20Update OperationsRow Updates, Update Strategies, Merge Statements, Update Performance, Handling Conflicts
21Merge OperationsMERGE INTO Syntax, Upserts, CDC Processing, Incremental Updates, Merge Optimization
22Streaming IntegrationSpark Structured Streaming, Flink Streaming, Kafka Integration, Streaming Writes, Streaming Reads
23Performance OptimizationPartition Design, File Size Tuning, Metadata Optimization, Query Planning, Caching Strategies
24SecurityAccess Control, Authentication, Authorization, Data Encryption, Governance Policies
25MonitoringQuery Monitoring, Table Metrics, Logging, Alerts Setup, Performance Tracking
26Iceberg MaintenanceExpire Snapshots, Remove Orphan Files, Rewrite Manifests, Table Optimization, Maintenance Scheduling
27Integration with Data LakesAWS S3 Storage, Azure Data Lake Storage, Google Cloud Storage, HDFS Storage, Hybrid Storage
28ComparisonIceberg vs Delta Lake, Iceberg vs Apache Hudi, Performance Comparison, Feature Comparison, Use Case Differences
29Production Best PracticesData Layout Design, Partition Strategies, Governance, Performance Monitoring, Disaster Recovery
30Real World Use CasesData Lakehouse Architecture, Incremental Data Pipelines, CDC Data Processing, Data Warehousing, Machine Learning Data Pipelines

Interview question

    BASIC

  • What is Apache Iceberg?
  • Why was Apache Iceberg created?
  • What problems does Iceberg solve in data lakes?
  • What is a table format?
  • Difference between Iceberg and Hive tables?
  • What are Iceberg snapshots?
  • What is schema evolution?
  • What is hidden partitioning?
  • Which engines support Iceberg?
  • What file formats are supported by Iceberg?
  • What is a catalog in Iceberg?
  • What is Hadoop catalog?
  • What is Hive Metastore catalog?
  • How do you create an Iceberg table?
  • What is append operation?
  • What is overwrite operation?
  • What is time travel in Iceberg?
  • What is partition pruning?
  • What metadata does Iceberg maintain?
  • What is ACID compliance?
  • What is snapshot isolation?
  • How does Iceberg handle large tables?
  • What is a manifest file?
  • What is a manifest list?
  • What is table metadata file?
  • INTERMEDIATE

  • Explain Iceberg table architecture.
  • How does Iceberg enable schema evolution safely?
  • Difference between partition spec and partition field?
  • How does Iceberg support concurrent writes?
  • Explain snapshot lifecycle.
  • How does rollback work in Iceberg?
  • What is metadata versioning?
  • Explain equality deletes.
  • Explain position deletes.
  • How does Iceberg manage small files?
  • What is compaction in Iceberg?
  • How does Iceberg optimize query performance?
  • Explain partition transforms.
  • How does Iceberg support streaming workloads?
  • What is snapshot expiration?
  • Explain orphan file removal.
  • What is metadata cleanup?
  • Explain Iceberg commit process.
  • How does Iceberg guarantee atomic commits?
  • Explain Iceberg table properties.
  • How do you query historical data?
  • What are branches and tags in Iceberg?
  • How does Iceberg integrate with Spark?
  • How does Iceberg integrate with Flink?
  • Explain Iceberg REST catalog.
  • ADVANCED

  • Explain Iceberg metadata tree structure.
  • How does Iceberg avoid partition explosion?
  • Explain write amplification in Iceberg.
  • How are manifests merged?
  • Explain scan planning in Iceberg.
  • What is vectorized reading in Iceberg?
  • How does Iceberg handle schema ID tracking?
  • Explain optimistic concurrency control.
  • How does Iceberg manage multi-engine consistency?
  • Explain incremental reads.
  • How does Iceberg support CDC pipelines?
  • Explain MERGE INTO implementation.
  • How does Iceberg handle updates internally?
  • Explain delete file application during reads.
  • How does Iceberg reduce metadata bottlenecks?
  • Explain file pruning mechanisms.
  • How are statistics stored in Iceberg?
  • Explain snapshot lineage.
  • How does Iceberg support branching workflows?
  • Explain Iceberg table rewrite operations.
  • How does Iceberg interact with object storage?
  • Explain commit retries and conflict resolution.
  • How does Iceberg enable reproducible analytics?
  • Explain metadata caching strategies.
  • How does Iceberg scale to petabyte datasets?
  • EXPERT

  • Design a lakehouse architecture using Iceberg.
  • Explain Iceberg internals during query planning.
  • How would you tune Iceberg for low-latency analytics?
  • Explain metadata file compaction strategy.
  • How does Iceberg compare with Delta Lake internals?
  • How does Iceberg compare with Apache Hudi architecture?
  • Design CDC ingestion using Iceberg + Kafka.
  • How do you implement multi-tenant Iceberg tables?
  • Explain Iceberg failure recovery mechanisms.
  • How do you manage schema governance at scale?
  • Explain Iceberg security integration patterns.
  • How would you optimize cost in cloud object storage?
  • Design real-time analytics using Iceberg.
  • Explain snapshot branching for experimentation.
  • How do you debug corrupted Iceberg metadata?
  • Explain Iceberg upgrade compatibility strategy.
  • How would you implement data governance using Iceberg?
  • Explain cross-engine transaction guarantees.
  • How do you monitor Iceberg tables in production?
  • Design a petabyte-scale Iceberg deployment.
  • Explain Iceberg performance benchmarking methodology.
  • How do you handle massive small-file ingestion pipelines?
  • Explain disaster recovery strategy for Iceberg tables.
  • How would you implement data version auditing?
  • Explain future roadmap and emerging features of Apache Iceberg.

Related Topics


20 November 2025

#Data_Science

#Data_Science

Key Concepts


S.No Topic Sub-Topics
1Data Science OverviewDefinition, Lifecycle, Use Cases, Roles, Skills Required
2Mathematics for Data ScienceLinear Algebra Basics, Probability, Statistics, Calculus Overview, Optimization
3Python for Data SciencePython Basics, NumPy, Pandas, Matplotlib, Jupyter Notebooks
4Data CollectionData Sources, APIs, Web Scraping, Databases, Data Warehouses
5Data CleaningMissing Values, Outlier Detection, Data Imputation, Data Normalization, Deduplication
6Exploratory Data AnalysisSummary Statistics, Data Visualization, Correlation Analysis, Distribution Analysis, Insights
7Data VisualizationMatplotlib, Seaborn, Plotly, Dashboards, Storytelling
8Statistics for Data ScienceDescriptive Statistics, Inferential Statistics, Hypothesis Testing, Confidence Intervals, A/B Testing
9Feature EngineeringFeature Creation, Feature Scaling, Encoding Categorical Data, Feature Selection, Dimensionality Reduction
10Machine Learning BasicsSupervised Learning, Unsupervised Learning, Model Evaluation, Bias-Variance Tradeoff, Pipelines
11Supervised Learning AlgorithmsLinear Regression, Logistic Regression, Decision Trees, Random Forest, Gradient Boosting
12Unsupervised Learning AlgorithmsK-Means, Hierarchical Clustering, DBSCAN, PCA, Association Rules
13Model Training and EvaluationTrain-Test Split, Cross Validation, Metrics, Overfitting, Underfitting
14Model TuningHyperparameter Tuning, Grid Search, Random Search, Bayesian Optimization, Regularization
15Time Series AnalysisTime Series Components, ARIMA, Seasonality, Forecasting, Evaluation
16Natural Language ProcessingText Preprocessing, Tokenization, Vectorization, Topic Modeling, Sentiment Analysis
17Deep Learning BasicsNeural Networks, Activation Functions, Backpropagation, Optimization Algorithms, Frameworks
18Deep Learning ModelsCNNs, RNNs, LSTMs, Transformers, Use Cases
19Big Data for Data ScienceHadoop, Spark, Distributed Computing, Data Lakes, Scalability
20Model DeploymentModel Serialization, REST APIs, Batch vs Real-time Inference, Monitoring, Scaling
21MLOps FundamentalsVersion Control, CI/CD, Experiment Tracking, Model Registry, Automation
22Model MonitoringData Drift, Concept Drift, Performance Metrics, Alerts, Retraining
23Ethics and Responsible AIBias, Fairness, Explainability, Transparency, Privacy
24Data StorytellingBusiness Context, Visual Narratives, Communication, Dashboards, Stakeholder Presentation
25Domain KnowledgeBusiness Understanding, KPIs, Industry Use Cases, Problem Framing, Decision Making
26Cloud for Data ScienceAWS, Azure, GCP, Managed ML Services, Cost Management
27Advanced AnalyticsRecommendation Systems, Anomaly Detection, Graph Analytics, Causal Inference, Optimization
28Data Science ToolsScikit-learn, TensorFlow, PyTorch, MLflow, DVC
29Best PracticesReproducibility, Documentation, Code Quality, Collaboration, Experiment Management
30End-to-End Data Science ProjectProblem Definition, Data Preparation, Modeling, Evaluation, Deployment

Interview question

🟢 Basic Level

  1. What is Data Science?
  2. Difference between Data Science, Data Analytics, and Machine Learning?
  3. What are structured and unstructured data?
  4. What is a dataset?
  5. What is the role of a data scientist?
  6. What is sampling? Why is it used?
  7. Explain mean, median, and mode.
  8. What is standard deviation?
  9. What is variance?
  10. What is correlation?
  11. What is a histogram?
  12. What is a box plot?
  13. What is a scatter plot used for?
  14. Difference between a population and a sample.
  15. What is a null hypothesis?
  16. What are outliers?
  17. What is data cleaning?
  18. What is feature scaling?
  19. What is normalization?
  20. What is a train-test split?
  21. What is supervised learning?
  22. What is unsupervised learning?
  23. What is a classification problem?
  24. What is a regression problem?
  25. What is overfitting?

🟡 Intermediate Level

  1. What is underfitting?
  2. What is cross-validation?
  3. What is k-fold cross-validation?
  4. What is a confusion matrix?
  5. Explain precision and recall.
  6. What is F1-score?
  7. What is ROC-AUC?
  8. What is logistic regression?
  9. How does k-NN work?
  10. What is Naive Bayes?
  11. What is decision tree?
  12. Explain Random Forest.
  13. What is gradient boosting?
  14. What is XGBoost?
  15. What is feature engineering?
  16. What is one-hot encoding?
  17. What is label encoding?
  18. What is PCA (Principal Component Analysis)?
  19. What is multicollinearity?
  20. What is regularization?
  21. Difference between L1 and L2 regularization.
  22. What is a cost function?
  23. What is bias?variance tradeoff?
  24. What is EDA (Exploratory Data Analysis)?
  25. What are missing value handling techniques?

🔵 Advanced Level

  1. What is a neural network?
  2. What is backpropagation?
  3. Explain gradient descent.
  4. What is stochastic gradient descent?
  5. Explain vanishing gradient problem.
  6. Difference between CNN and RNN.
  7. What is LSTM?
  8. What is attention mechanism?
  9. What are word embeddings?
  10. Explain TF-IDF.
  11. What is time-series forecasting?
  12. What is ARIMA?
  13. What is stationarity in time series?
  14. What is autocorrelation?
  15. What is cross-correlation?
  16. What is anomaly detection?
  17. What is k-means clustering?
  18. What is hierarchical clustering?
  19. What is silhouette score?
  20. What is DBSCAN?
  21. What is model drift?
  22. What is data leakage?
  23. What is MLOps?
  24. What is Docker in ML deployment?
  25. Explain REST API for ML model.

🔴 Expert Level

  1. Explain transformer architecture.
  2. What are large language models (LLMs)?
  3. What is reinforcement learning?
  4. Explain Q-learning.
  5. What is Markov Decision Process (MDP)?
  6. What is approximate inference?
  7. Explain Bayesian networks.
  8. What is causal inference?
  9. What is counterfactual prediction?
  10. What is SHAP?
  11. What is LIME?
  12. Explain gradient boosting in detail.
  13. How does CatBoost handle categorical variables?
  14. What is feature store?
  15. Explain Data Lake vs Data Warehouse.
  16. What is delta lake?
  17. What is distributed training?
  18. What is parameter server architecture?
  19. What is vector database?
  20. What is embedding dimensionality reduction?
  21. Explain end-to-end ML lifecycle.
  22. What is model observability?
  23. What is multi-armed bandit algorithm?
  24. Explain optimization using Adam, RMSProp.
  25. Explain real-time ML pipelines (Kafka + Spark + ML model).

Related Topics


   Data_Warehouse   
   Data_Lake   
   Data_Lakehouse