Unity Catalog in Databricks: Governance, Lineage & Security Explained

Unity Catalog in Databricks — Quick Answer

Unity Catalog is Databricks\' unified governance solution for data and AI. It provides a single place to manage tables, files, functions, and ML models — with fine-grained access control, automated lineage, search, and cross-workspace sharing. It replaces the Hive Metastore as the default governance layer and is included free with every Databricks deployment.

Unity Catalog — Three-Level Namespace

Unity Catalog uses a 3-level namespace:

catalog.schema.table

Example:
main.sales.orders
main.marketing.customer_segments
sandbox.dev.experimental_features

Each level can have its own access policies. This makes it easy to isolate environments (dev vs prod) and apply governance by domain.

Unity Catalog vs Hive Metastore

Unity Catalog Key Features

1. Fine-grained access control: Row-level filters and column-level masks for data privacy.
2. Automated lineage: Column-level upstream and downstream lineage — automatic across jobs.
3. Search and discovery: Tag-based and full-text search across all data assets.
4. Delta Sharing: Share data with external organisations without copying.
5. Lakehouse Federation: Query external sources (PostgreSQL, MySQL, Snowflake) without copying data.
6. ML model governance: Track ML models in the same catalog as data.
7. Audit logs: Every access logged — who, when, what.

Row-Level and Column-Level Security Example

-- Row filter: only show rows for user's region
CREATE FUNCTION sales.filter_region(region STRING)
  RETURN IF(is_account_group_member('sales_americas'), TRUE, region = 'apac');

ALTER TABLE sales.orders
  SET ROW FILTER sales.filter_region(region);

-- Column mask: hide salary from non-privileged users
CREATE FUNCTION sales.mask_salary(salary DECIMAL(10,2))
  RETURN IF(is_account_group_member('hr_managers'), salary, '****');

ALTER TABLE hr.employees
  ALTER COLUMN salary SET MASK sales.mask_salary;

Unity Catalog Integrations

Unity Catalog Best Practices

• Catalog per environment: main (prod), dev, sandbox. Each with different access policies.
• Schema per domain: sales, marketing, finance, hr. Domain owners manage their own schema.
• Tag everything: PII tags, retention tags, ownership tags. Tags power search and policy.
• Enable column-level lineage: Essential for compliance and impact analysis.
• Use Delta Sharing for external: Don't copy data to external orgs — share securely.
• Audit access: Review access logs monthly. Flag unexpected access patterns.

Common Unity Catalog

Real-World Example — Migrating to Unity Catalog

A typical migration from Hive Metastore to Unity Catalog follows this pattern:

• Step 1 — Inventory: Run a script to list all tables in the legacy metastore. Group by database and identify ownership.
• Step 2 — Catalogue Mapping: Map legacy databases to Unity Catalog schemas. For example, prod.sales.orders → main.sales.orders.
• Step 3 — Permissions Translation: Convert legacy GRANT statements to Unity Catalog ACLs. Map users to groups via SCIM.
• Step 4 — External Locations: Configure storage credentials and external locations for S3/ADLS access.
• Step 5 — Cutover: Migrate table-by-table with a parallel-run period. Update notebook references from hive_metastore.db.table to main.db.table.
• Step 6 — Decommission: Once everything runs on Unity Catalog, disable Hive Metastore.

A full migration takes 2–4 weeks for a medium data team. Unity Catalog tools (terraform-databricks-uc) automate much of this.

Pitfalls

Avoid these traps when adopting Unity Catalog:

• Skipping catalog per environment: One catalog for everything is a governance nightmare. Use separate catalogs for prod, dev, sandbox.
• Not tagging tables: Tags power search and policy. Tables without tags are invisible to the data discovery workflow.
• Over-granting access: Grant only what is needed. "ALL PRIVILEGES" on a catalog is rarely justified.
• Ignoring audit logs: Every access is logged. Review the audit log monthly for unexpected access patterns.
• Mixing Hive Metastore and Unity Catalog: Mixing the two creates governance gaps. Migrate fully or stay fully legacy.

Quick Reference — Cheatsheet

• catalog.schema.table is the 3-level namespace — use it for env isolation.
• Row filters and column masks implement row/column-level security.
• Automatic column-level lineage flows across all jobs.
• Delta Sharing enables secure cross-org data exchange.
• Unity Catalog is free — included with all Databricks deployments.

Frequently Asked Questions

What is Unity Catalog in Databricks?

Databricks' unified governance for data and AI — fine-grained access, lineage, search, cross-workspace sharing.

How is Unity Catalog different from Hive Metastore?

Hive Metastore is legacy — table-level ACLs only, no lineage. Unity Catalog is next-gen with row/column security, sharing, and lineage.

What are the three-level namespace in Unity Catalog?

catalog.schema.table (e.g., main.sales.orders). Each level can have its own access policies.

Does Unity Catalog support row-level and column-level security?

Yes — row filters and column masks via SQL functions.

Does Unity Catalog provide data lineage?

Yes — automatic column-level lineage across Spark, DLT, and SQL warehouses.

Is Unity Catalog free?

Yes — included free with all Databricks deployments.