Azure Databricks — When to Choose It
- Yes: Heavy Spark/SQL workloads, ML/AI workloads, lakehouse architecture, large-scale ELT, real-time analytics.
- No: Simple ETL (use Azure Data Factory), small analytics (use Synapse serverless), simple data science notebooks (use Azure ML Studio).
Azure Databricks shines when workloads are compute-heavy and Spark-native. For simple pipelines, Data Factory + Synapse is cheaper.
Security Best Practices
- VNet injection: Deploy clusters in your own VNet — no public IPs.
- Private endpoints: Connect to ADLS, Key Vault, Synapse via private link.
- Customer-managed keys (CMK): Encrypt notebooks and storage with your own keys.
- Unity Catalog: Centralised access control across workspaces.
- IP allow-listing: Restrict workspace access to known IP ranges.
Azure Databricks — Common Pitfalls
Watch out for these traps in production Azure Databricks deployments:
- Using on-demand clusters for production jobs: Always use job clusters — they terminate after the job ends, saving 60–80% cost.
- Skipping VNet injection: Default deployments expose clusters via public IPs. Enterprise deployments need VNet injection + private endpoints.
- No data partitioning: Reading a 10 TB Delta table without partitioning is slow. Partition by date or high-cardinality field.
- Forgetting to enable Photon: Photon is enabled by default on newer runtimes. Verify it's enabled for production workloads — it accelerates Spark SQL 2–8x.
- No cost monitoring: Use Azure Cost Management + tags to track spend per workspace. Databricks bills add up fast without monitoring.
Quick Reference — Cheatsheet
- Control plane is Databricks-managed; data plane runs in your subscription.
- Photon engine accelerates Spark SQL 2–8x — enabled by default on newer runtimes.
- Use spot instances for non-critical jobs to cut cost 60–80%.
- VNet injection is mandatory for enterprise deployments.
- Unity Catalog centralises governance across workspaces.
Azure Databricks Quick-Wins
Apply these patterns for production Azure Databricks deployments:
- Use Unity Catalog from day one: Don't start with the legacy Hive Metastore — migrate to UC immediately to avoid governance debt.
- Pin library versions: Use %pip install at the top of notebooks and pin versions for reproducibility.
- Cluster policies: Define cluster policies that auto-terminate idle clusters and enforce instance types.
- Use Repos for CI/CD: Sync notebooks from Azure DevOps/GitHub via Databricks Repos for version-controlled deployments.
- Monitor with Azure Monitor: Set up alerts for job failures, cluster utilisation, and DBU spend.
Frequently Asked Questions
What is Azure Databricks?
A first-party Azure service on Apache Spark + Delta Lake. Unified platform for data engineering, data science, and ML, integrated with ADLS, Synapse, and Azure Data Factory.
How is Azure Databricks different from open-source Spark?
Managed Spark with proprietary optimisations (Photon, Delta enhancements), collaborative UI, MLflow, Azure-native identity/networking. Open-source Spark requires manual setup.
What is the architecture of Azure Databricks?
Control plane (Databricks-managed — UI, jobs) and data plane (your Azure subscription — clusters, ADLS). Data stays in your tenant.
What are the main use cases for Azure Databricks?
Lakehouse, ELT pipelines (DLT), ML training, GenAI/LLM, real-time streaming, SQL analytics.
How much does Azure Databricks cost?
DBU + Azure VM costs. ~₹15–25/hour for dev clusters, ₹250–400/hour for medium production clusters.
Does Azure Databricks integrate with Synapse?
Yes — share Unity Catalog, read/write Synapse SQL pools, use Synapse as BI serving layer.
Fazlur Rahman
