Getting Started with Unity Catalog: A Practical Walkthrough for Data Teams

If your team has been using Databricks for a while, you probably started with multiple workspaces — one for dev, one for prod, maybe one per squad. Each workspace has its own Hive Metastore, its own set of tables, its own permissions. It works until it doesn’t. Someone creates a table in dev that a prod job depends on. Another team can’t find the customer-level data your team owns. Permissions are a mess of instance profiles and table ACLs spread across workspaces.

Unity Catalog is Databricks’ answer to this. It gives you one place to govern all your data, across all workspaces. In this article, we’ll walk through what Unity Catalog actually is, set one up step by step, create catalogs and schemas, register tables, manage permissions, and cover the rough edges you’ll hit along the way.

What Unity Catalog Actually Is

Unity Catalog is a centralised metastore and governance layer that sits above your Databricks workspaces. Instead of each workspace having its own Hive Metastore, all workspaces attached to a Unity Catalog metastore share the same catalog hierarchy.

The hierarchy works like this:

• Metastore — top-level container for a region. One per region, shared across workspaces.
• Catalog — groups schemas. Think of it as a database container. Usually one per domain or team.
• Schema — groups tables, views, volumes, and functions. Same as a database in SQL terms.
• Table / View / Volume / Function — the actual data objects.

If you have multiple Databricks workspaces in the same region, you attach them all to the same metastore. That means a table created in prod_catalog.sales.orders is visible — permissions permitting — from your dev workspace, your analyst workspace, or wherever else you’ve attached.

Before You Start: What You Need

Unity Catalog needs a few things in place before you can turn it on:

1. A Databricks account with Unity Catalog enabled. Unity Catalog is enabled at the account level, not the workspace level. You need to be an account admin to do this. If you’re on the legacy free trial without account-level access, you’re out of luck here — check with your Databricks rep.
2. A storage bucket for the metastore root. Unity Catalog stores managed table data in its own bucket, separate from whatever external locations you might use. This bucket is the metastore’s root.
3. A service principal or identity that Databricks can use to access that bucket. On AWS, this means an IAM role with a trust policy that Databricks provides during setup. On Azure, it’s an access connector. On GCP, a service account.

Let’s walk through the setup on AWS, since that’s what I work with day to day.

Step 1: Create the Metastore Root Bucket

Create an S3 bucket for your metastore root. Nothing goes in it yet — this is where managed tables will live later:

aws s3api create-bucket \
  --bucket my-company-unity-catalog-metastore \
  --region us-east-1 \
  --create-bucket-configuration LocationConstraint=us-east-1

Enable bucket versioning. Unity Catalog uses it under the hood:

aws s3api put-bucket-versioning \
  --bucket my-company-unity-catalog-metastore \
  --versioning-configuration Status=Enabled

Step 2: Create the IAM Role for Databricks

Databricks needs to read and write to this bucket. You’ll create a role and attach a trust policy that Databricks gives you during setup. The policy looks roughly like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-xxxxx"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "databricks-unity-catalog-xxxxx"
        }
      }
    }
  ]
}

Attach an S3 policy that allows read and write on the metastore bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my-company-unity-catalog-metastore",
        "arn:aws:s3:::my-company-unity-catalog-metastore/*"
      ]
    }
  ]
}

The exact role ARN and external ID come from the Databricks account console during metastore creation. You copy them from there, don’t guess.

Step 3: Create the Metastore in Databricks

Go to your Databricks account console (not the workspace). Under Catalog & Schemas, click Create Metastore:

• Name: something descriptive like us-east1-metastore
• Region: match your bucket region
• Storage root: s3://my-company-unity-catalog-metastore/metastore-root (note the subfolder — Databricks creates it)
• IAM role ARN: the role you created in step 2

Once created, attach your workspaces to this metastore. An individual workspace can only attach to one metastore, and once attached, it cannot be detached without recreating the workspace. Think before clicking.

Step 4: Create Catalogs and Schemas

With the metastore attached, you can create catalogs from any attached workspace. Run this in a Databricks notebook or SQL editor:

CREATE CATALOG IF NOT EXISTS sales_catalog
  COMMENT 'Sales and revenue data';

Then create a schema inside it:

CREATE SCHEMA IF NOT EXISTS sales_catalog.orders
  COMMENT 'Order transaction data';

Now let’s create a managed table:

CREATE TABLE sales_catalog.orders.daily_orders (
  order_id STRING,
  customer_id STRING,
  order_date DATE,
  amount DECIMAL(10, 2)
) USING DELTA
  LOCATION 's3://my-data-lake/sales/orders/daily/';

A managed table stores data inside the metastore root by default. An external table uses LOCATION pointing to your own bucket. I prefer external tables for anything that existed before Unity Catalog — it means I’m not copying terabytes of data.

Step 5: Grant Permissions

This is where Unity Catalog earns its keep. Instead of table ACLs in SQL, you grant permissions using SQL directly:

GRANT USAGE ON CATALOG sales_catalog TO `analysts_group`;
GRANT USAGE ON SCHEMA sales_catalog.orders TO `analysts_group`;
GRANT SELECT ON TABLE sales_catalog.orders.daily_orders TO `analysts_group`;

Permissions cascade down but only if you grant them at each level. If a user has SELECT on a table but not USAGE on the schema, they can’t read the table. This trips people up regularly.

If you’re using Python in a notebook:

df = spark.table("sales_catalog.orders.daily_orders")
df.filter("order_date >= '2025-09-01'").show()

The three-level namespace (catalog.schema.table) is mandatory in Unity Catalog. If you’re used to database.table, that’s gone. Old references like default.my_table break unless you explicitly set a default catalog.

External Locations and Storage Credentials

If you’re reading from external S3 buckets, you need to register them in Unity Catalog:

1. Create a storage credential — an IAM role that Databricks assumes to access the bucket.
2. Create an external location — maps a storage credential to a bucket prefix.

CREATE EXTERNAL LOCATION raw_data_location
  URL 's3://my-data-lake/'
  WITH (STORAGE CREDENTIAL `raw_data_cred`);

Without this, even if your cluster has the right instance profile, Unity Catalog won’t let you through. This is by design — it forces all access through one governed path.

Comparison: Hive Metastore vs Unity Catalog

Feature	Hive Metastore	Unity Catalog
Scope	Single workspace	Regional, multi-workspace
Namespace	database.table	catalog.schema.table
Permissions	SQL table ACLs (fragile)	SQL GRANT model (hierarchical)
Auditing	Per-workspace logs	Centralised audit log
Lineage	Manual or add-on	Built-in table lineage
Volumes	Not supported	Native non-tabular data
Lakehouse Federation	Manual setup	Native query federation to other DBs

Things to Watch Out For

The metastore is region-bound. If you have workspaces in us-east-1 and eu-west-1, you need two metastores. They don’t talk to each other.

Attaching is permanent. Once a workspace is attached to a metastore, it’s attached forever. Test this on a non-production workspace first.

Old code breaks. Any spark.sql("USE database") or spark.table("database.table") references need updating to the three-level namespace. Start by setting a default catalog on the cluster or session to ease the transition.

Cost is not free. Managed tables in the metastore root use Databricks-managed storage. External tables on your own buckets are cheaper if you already own the infrastructure. For production, I’d use external tables with storage credentials for anything substantial — it keeps your data where you control it and avoids lock-in on managed storage pricing.

Auto-loader and streaming. If you use Auto Loader with file notification mode, make sure your external location is configured before the stream starts. Unity Catalog will reject the checkpoint if it can’t resolve the location, and recovery isn’t graceful.

What Changes in Production

In a demo, you create one metastore, one catalog, attach your workspace, and call it done. In production, you’ll realistically have:

• Multiple catalogs per domain — sales_catalog, marketing_catalog, finance_catalog — to scope permissions cleanly.
• Service principals owning and running jobs, not user accounts. Create a service principal per pipeline, grant exactly the permissions it needs.
• Terraform or Databricks Asset Bundles managing the metastore, catalogs, and external locations instead of clicking through the console.
• A migration plan for existing tables. You’ll likely register your existing data lake buckets as external locations, create external tables pointing at the existing Delta paths, and cut over incrementally.

Wrapping Up

Unity Catalog solves the governance mess that grows organically in any multi-workspace Databricks environment. The setup process is fairly straightforward — create a bucket, create a role, click through the account console, and you’re up. The real work is migrating existing tables, getting your team comfortable with three-level namespaces, and setting up the permission model properly.

Start small. Pick one domain, create a catalog, move a few tables, and see how it feels in practice. Once the team is comfortable, expanding to other domains is mostly repeating the same steps.