Showing posts with label ETL. Show all posts
Showing posts with label ETL. Show all posts
26 August 2026
18 January 2026
#Databricks
Key Concepts
| S.No | Topic | Sub Topic |
|---|---|---|
| 1 | Databricks | What is Databricks, Lakehouse concept, Databricks vs Hadoop, Use cases, Architecture overview |
| 2 | Databricks Workspace | Workspace UI, Notebooks, Clusters, Jobs, Repos |
| 3 | Databricks Architecture | Control plane, Data plane, Workspace components, Security layers, Execution flow |
| 4 | Clusters in Databricks | All-purpose clusters, Job clusters, Autoscaling, Cluster policies, Init scripts |
| 5 | Databricks Runtime | DBR versions, Photon engine, ML runtime, GPU runtime, Performance tuning |
| 6 | Notebooks | Languages supported, Notebook workflows, Magic commands, Versioning, Collaboration |
| 7 | Databricks Utilities (dbutils) | File system ops, Secrets, Widgets, Notebook workflows, FS mounts |
| 8 | Data Ingestion | Batch ingestion, Streaming ingestion, Auto Loader, File formats, Schema inference |
| 9 | Delta Lake Fundamentals | ACID transactions, Delta log, Schema enforcement, Time travel, File compaction |
| 10 | Delta Lake Advanced | OPTIMIZE, Z-ORDER, Vacuum, Delta constraints, Change Data Feed |
| 11 | Spark SQL in Databricks | SQL editor, ANSI SQL, Views, CTEs, Query optimization |
| 12 | DataFrames & Datasets | API overview, Transformations, Actions, Lazy evaluation, Performance tips |
| 13 | Databricks SQL Warehouses | Serverless SQL, Query execution, Dashboards, Alerts, Access control |
| 14 | Jobs & Workflows | Job types, Task dependencies, Scheduling, Retries, Monitoring |
| 15 | Databricks Repos | Git integration, Branching, CI/CD basics, Repo permissions, Best practices |
| 16 | Security & Access Control | Users & groups, IAM integration, Table ACLs, Cluster policies, Secrets |
| 17 | Unity Catalog | Metastore, Catalogs & schemas, Data lineage, Fine-grained access, Auditing |
| 18 | Streaming with Databricks | Structured Streaming, Triggers, Watermarking, Stateful ops, Fault tolerance |
| 19 | Auto Loader | CloudFiles, Incremental ingestion, Schema evolution, Notifications, Performance tuning |
| 20 | Databricks ML Overview | ML workspace, ML runtime, Experiment tracking, Feature store, Model registry |
| 21 | MLflow in Databricks | Tracking, Projects, Models, Model registry, Deployment |
| 22 | Feature Store | Feature tables, Offline features, Online features, Reusability, Governance |
| 23 | Model Training | Distributed training, Hyperparameter tuning, AutoML, GPUs, Evaluation metrics |
| 24 | Model Deployment | Batch inference, Real-time serving, Model endpoints, A/B testing, Monitoring |
| 25 | Performance Optimization | Partitioning, Caching, Broadcast joins, Skew handling, Photon usage |
| 26 | Monitoring & Logging | Spark UI, Ganglia, Job metrics, Logs, Alerts |
| 27 | Cost Optimization | Cluster sizing, Spot instances, Autoscaling, Job clusters, Usage reports |
| 28 | Databricks on Cloud | AWS architecture, Azure architecture, GCP basics, Networking, Storage integration |
| 29 | CI/CD & DevOps | Repos + pipelines, Databricks CLI, Asset bundles, Environment promotion, Automation |
| 30 | Real-world Use Cases | ETL pipelines, Streaming analytics, ML pipelines, Lakehouse design, Interview prep |
Interview Question
| What is Databricks and what are its core components? |
| What are the key features of the Databricks Lakehouse Platform? |
| What is the Databricks Lakehouse architecture? |
| What is the difference between a data lake, data warehouse, and lakehouse? |
| What is Delta Lake and why is it important in Databricks? |
| What are the key features of Delta Lake? |
| What is Delta Lake ACID transaction support? |
| What is schema enforcement in Delta Lake? |
| What is schema evolution in Delta Lake? |
| What is the Delta Lake transaction log? |
| What is time travel in Delta Lake? |
| What are Delta Lake table versions? |
| What is the difference between managed and external tables in Databricks? |
| What are Unity Catalog and its main capabilities? |
| What is the three-level namespace in Unity Catalog? |
| What are catalogs, schemas, and tables in Unity Catalog? |
| What is data lineage in Unity Catalog? |
| What are Unity Catalog access control mechanisms? |
| What are external locations and storage credentials in Unity Catalog? |
| What are Delta Sharing and its use cases? |
| What are Databricks workspaces? |
| What are Databricks clusters and how do they work? |
| What are the different types of Databricks compute resources? |
| What are all-purpose clusters and job clusters? |
| What are serverless compute resources in Databricks? |
| What are cluster policies in Databricks? |
| What are Databricks Runtime versions? |
| What is Photon in Databricks? |
| What are Databricks SQL warehouses? |
| What are the differences between SQL warehouses and Spark clusters? |
| What is Apache Spark and how is it used by Databricks? |
| What are Spark DataFrames and how are they used in Databricks? |
| What are Spark RDDs and when would you use them? |
| What is lazy evaluation in Apache Spark? |
| What are Spark transformations and actions? |
| What is Spark Catalyst Optimizer? |
| What is Spark Adaptive Query Execution? |
| What are narrow and wide transformations in Spark? |
| What is data partitioning in Databricks? |
| What is data skew and how can it be handled in Databricks? |
| What is repartitioning and coalescing in Spark? |
| What is caching in Spark and when should it be used? |
| What are broadcast joins and when should they be used? |
| What are Databricks jobs and workflows? |
| What are job tasks and dependencies in Databricks Workflows? |
| What are job parameters and dynamic task values? |
| What are Databricks notebooks and how are they used? |
| What are notebook workflows and reusable notebooks? |
| What are Databricks Repos and how do they support Git integration? |
| What is Databricks Asset Bundles? |
| What is Databricks CLI and how is it used? |
| What are Databricks REST APIs and what can they automate? |
| What is Auto Loader in Databricks? |
| What are the advantages of Auto Loader over traditional file ingestion? |
| What are streaming checkpoints in Databricks? |
| What is Structured Streaming in Databricks? |
| What are streaming triggers in Spark Structured Streaming? |
| What is exactly-once processing in Databricks Structured Streaming? |
| What is the difference between batch processing and streaming processing? |
| What is Delta Live Tables and how does it support data pipelines? |
| What is Lakeflow Declarative Pipelines in Databricks? |
| What are expectations and data quality rules in Databricks pipelines? |
| What are medallion architecture layers in Databricks? |
| What are Bronze, Silver, and Gold tables? |
| What are common ETL and ELT patterns in Databricks? |
| What are incremental data processing strategies in Databricks? |
| What are Change Data Capture patterns in Databricks? |
| What is Delta Change Data Feed? |
| What are MERGE operations in Delta Lake? |
| What are OPTIMIZE and Z-ORDER in Databricks? |
| What is liquid clustering in Databricks? |
| What is VACUUM in Delta Lake? |
| What is data skipping in Databricks? |
| What are Delta Lake performance optimization techniques? |
| What are Databricks SQL query optimization techniques? |
| What is the difference between partitioning, Z-ORDER, and liquid clustering? |
| What is Databricks MLflow and how is it used for machine learning? |
| What is the Databricks Machine Learning platform? |
| What are MLflow experiments, runs, and models? |
| What is the Databricks Feature Store? |
| What is Model Serving in Databricks? |
| What is Mosaic AI in Databricks? |
| What is Databricks Vector Search? |
| What are vector indexes in Databricks Vector Search? |
| What is Retrieval-Augmented Generation and how can it be implemented in Databricks? |
| What is the role of Databricks in building RAG applications? |
| What are embeddings and how are they stored or searched in Databricks? |
| What is a vector database and how does Databricks support vector search? |
| What are AI Gateway capabilities in Databricks Model Serving? |
| What are foundation models in Databricks? |
| What is Model Serving for LLM applications in Databricks? |
| What is prompt engineering and how can it be integrated with Databricks AI workloads? |
| What are AI agents and how can they be developed on Databricks? |
| What is Agent Framework in Databricks? |
| How can Databricks support multi-agent AI architectures? |
| How can Unity Catalog govern AI models, features, and vector indexes? |
| How can Databricks be used to build an enterprise RAG pipeline? |
| How can Databricks be integrated with LangChain for AI applications? |
| How can Databricks be integrated with LangGraph for agentic AI workflows? |
| How can Databricks be used with MCP-based AI agents? |
| How can Databricks support real-time AI inference pipelines? |
| How can Databricks handle large-scale LLM training and fine-tuning workloads? |
| How can Databricks be used to monitor and evaluate production AI agents? |
| How can Databricks implement AI governance, security, and responsible AI practices? |
| How can Databricks be integrated with Azure OpenAI for enterprise AI applications? |
| How can Databricks be used to build an end-to-end Agentic AI data platform? |
| What are the best practices for designing a scalable Databricks platform for AI and data engineering? |
Subscribe to:
Posts (Atom)