- Using on-demand clusters for production jobs: Always use job clusters — they terminate after the job ends, saving 60–80% cost.
- Skipping VNet injection: Default deployments expose clusters via public IPs. Enterprise deployments need VNet injection + private endpoints.
- No data partitioning: Reading a 10 TB Delta table without partitioning is slow. Partition by date or high-cardinality field.
- Forgetting to enable Photon: Photon is enabled by default on newer runtimes. Verify it's enabled for production workloads — it accelerates Spark SQL 2–8x.
- No cost monitoring: Use Azure Cost Management + tags to track spend per workspace. Databricks bills add up fast without monitoring.
Quick Reference — Cheatsheet
- Control plane is Databricks-managed; data plane runs in your subscription.
- Photon engine accelerates Spark SQL 2–8x — enabled by default on newer runtimes.
- Use spot instances for non-critical jobs to cut cost 60–80%.
- VNet injection is mandatory for enterprise deployments.
- Unity Catalog centralises governance across workspaces.
Azure Databricks Quick-Wins
Apply these patterns for production Azure Databricks deployments:
- Use Unity Catalog from day one: Don't start with the legacy Hive Metastore — migrate to UC immediately to avoid governance debt.
- Pin library versions: Use %pip install at the top of notebooks and pin versions for reproducibility.
- Cluster policies: Define cluster policies that auto-terminate idle clusters and enforce instance types.
- Use Repos for CI/CD: Sync notebooks from Azure DevOps/GitHub via Databricks Repos for version-controlled deployments.
- Monitor with Azure Monitor: Set up alerts for job failures, cluster utilisation, and DBU spend.
Frequently Asked Questions
What is Azure Databricks?
A first-party Azure service on Apache Spark + Delta Lake. Unified platform for data engineering, data science, and ML, integrated with ADLS, Synapse, and Azure Data Factory.
How is Azure Databricks different from open-source Spark?
Managed Spark with proprietary optimisations (Photon, Delta enhancements), collaborative UI, MLflow, Azure-native identity/networking. Open-source Spark requires manual setup.
What is the architecture of Azure Databricks?
Control plane (Databricks-managed — UI, jobs) and data plane (your Azure subscription — clusters, ADLS). Data stays in your tenant.
What are the main use cases for Azure Databricks?
Lakehouse, ELT pipelines (DLT), ML training, GenAI/LLM, real-time streaming, SQL analytics.
How much does Azure Databricks cost?
DBU + Azure VM costs. ~₹15–25/hour for dev clusters, ₹250–400/hour for medium production clusters.
Does Azure Databricks integrate with Synapse?
Yes — share Unity Catalog, read/write Synapse SQL pools, use Synapse as BI serving layer.