Medallion Architecture Explained Simply: A Practical Guide

If you have spent any time reading about data lakehouses or Databricks, you have probably heard the term “medallion architecture” thrown around. It sounds fancy, but at its heart the idea is simple: organise your data into layers — bronze, silver, and gold — where each layer improves the quality and usability of the data.

In this article, let us walk through what each layer actually means in practice, how to build one using Delta Lake, and the real trade-offs you will hit when you go beyond a demo. No marketing fluff.

What Is the Medallion Architecture?

The medallion architecture is a way of organising data in a lakehouse. Instead of dumping everything into one big folder and hoping for the best, you process data through three stages:

Layer	What Goes In	Who Uses It
Bronze	Raw ingested data, exactly as it arrived	Data engineers debugging ingestion issues
Silver	Cleaned, validated, deduplicated data	Analytics engineers, data scientists doing exploration
Gold	Business-level aggregates and curated datasets	BI tools, dashboards, business users

Each layer is a separate set of Delta tables. You do not copy data blindly — you transform it as it moves from one layer to the next.

Let us go through each layer with a concrete example. Imagine we are ingesting e-commerce order data from a Kafka topic.

Bronze Layer: Keep Everything, Ask Questions Later

The bronze layer is your insurance policy. You land data exactly as it arrived from the source — same schema, same format, warts and all. No transformations, no filtering.

-- Bronze: append raw Kafka messages as they arrive
CREATE OR REPLACE TABLE bronze.orders_raw
USING DELTA
LOCATION '/mnt/datalake/bronze/orders/'

INSERT INTO bronze.orders_raw
SELECT *, current_timestamp() AS _ingested_at
FROM kafka_source

The key things to notice:

• We add an _ingested_at column so we know when each row landed. Useful for debugging.
• We do not drop columns, rename fields, or fix data types. Bronze is for preservation, not perfection.
• The schema evolves naturally — if the source adds a field, Delta Lake handles schema merging (if you enable it).

When bronze goes wrong: If you try to clean data in bronze, you lose the ability to replay from source. Someone will eventually ask “what did the raw data actually look like?” and you will not have it. Keep bronze raw.

Silver Layer: Clean It, Shape It, Make It Usable

The silver layer is where the real engineering work happens. You take the messy bronze data and turn it into something your team can actually query without wanting to quit.

Here is what typically happens in silver:

1. Deduplication. Kafka can deliver messages more than once. You need to dedupe.
2. Schema enforcement. Parse JSON strings into proper columns, cast types correctly.
3. Data quality checks. Drop rows with null IDs, negative amounts, or future timestamps.
4. Join enrichment. Bring in lookup tables — customer segments, product categories.

-- Silver: clean and deduplicate orders
CREATE OR REPLACE TABLE silver.orders
USING DELTA
LOCATION '/mnt/datalake/silver/orders/'

MERGE INTO silver.orders AS target
USING (
  SELECT
    order_id,
    customer_id,
    CAST(amount AS DECIMAL(10,2)) AS amount,
    CAST(order_timestamp AS TIMESTAMP) AS order_ts,
    _ingested_at
  FROM (
    SELECT *,
      ROW_NUMBER() OVER (
        PARTITION BY order_id ORDER BY _ingested_at DESC
      ) AS rn
    FROM bronze.orders_raw
    WHERE order_id IS NOT NULL
      AND amount > 0
  )
  WHERE rn = 1
) AS source
ON target.order_id = source.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *

A few things worth pointing out:

• The ROW_NUMBER() deduplication pattern is the most common approach I have seen in practice. Partition by your business key, order by ingestion time, and keep only the latest.
• We are using MERGE rather than a full overwrite. This matters when your bronze table is large and you only want to process new data incrementally.
• The WHERE clause filters out obviously bad rows. Be careful not to go overboard — you want to catch genuinely broken data, not quietly drop valid edge cases.

Production note: In a real pipeline, you would wrap the silver transformation in an orchestrator (Airflow, Databricks Workflows, whatever you have). You also want to run data quality checks using something like Great Expectations or Delta Live Tables expectations, and alert if the percentage of bad rows spikes above a threshold.

Gold Layer: Business-Ready Views

Gold is where you stop thinking like an engineer and start thinking like the business. These tables are what BI tools, dashboards, and ML models consume.

Gold tables are typically:

• Aggregated (daily sales, customer LTV, product rankings)
• Denormalised (joining facts with all the dimension columns you need)
• Optimised for read performance (Z-ordering, partition pruning)

-- Gold: daily sales summary for the BI dashboard
CREATE OR REPLACE TABLE gold.daily_sales
USING DELTA
LOCATION '/mnt/datalake/gold/daily_sales/'

INSERT OVERWRITE gold.daily_sales
SELECT
  DATE(order_ts) AS sale_date,
  p.product_category,
  c.customer_segment,
  COUNT(DISTINCT o.order_id) AS total_orders,
  SUM(o.amount) AS total_revenue,
  AVG(o.amount) AS avg_order_value
FROM silver.orders o
JOIN silver.products p ON o.product_id = p.product_id
JOIN silver.customers c ON o.customer_id = c.customer_id
GROUP BY DATE(order_ts), p.product_category, c.customer_segment

Notice INSERT OVERWRITE rather than a merge. Gold tables are often rebuilt entirely from silver on each run, especially for daily aggregations. This keeps things simple and avoids dealing with late-arriving data in complex merge logic.

The Real Trade-Offs No One Talks About

The medallion architecture sounds clean on a whiteboard, but here is what you actually run into:

Storage cost. You are keeping the same data in three places. Bronze alone can be massive if you are landing full CDC feeds. Use Delta Lake’s OPTIMIZE and VACUUM commands to clean up old snapshots and small files, but expect your storage bill to be higher than a raw dump.

Latency. Each layer adds processing time. If your dashboard users need sub-minute freshness, a three-layer batch pipeline might frustrate them. In that case you might stream bronze-to-silver and keep gold as batch, or use a streaming gold layer with something like Delta Live Tables.

Governance overhead. Three layers means three sets of access controls, three sets of data retention policies, and three places where someone can query stale data by mistake. Plan your catalog (Unity Catalog, Glue, whatever) before you build.

Not every dataset needs three layers. If you have a small lookup table that never changes, putting it through bronze-silver-gold is pointless. Land it directly in silver or gold and move on.

A Typical Pipeline Flow

Here is how a medallion pipeline looks end-to-end with Spark:

# Pseudocode for a nightly batch pipeline

# Step 1: Ingest to bronze
df_raw = spark.readStream \
    .format("kafka") \
    .option("subscribe", "orders") \
    .load()

df_raw.writeStream \
    .format("delta") \
    .outputMode("append") \
    .trigger(availableNow=True) \
    .table("bronze.orders_raw")

# Step 2: Clean and deduplicate into silver
df_bronze = spark.read.table("bronze.orders_raw") \
    .filter(col("_ingested_at") > max_silver_timestamp)

df_silver = deduplicate_and_clean(df_bronze)

df_silver.write \
    .format("delta") \
    .mode("append") \
    .table("silver.orders")

# Step 3: Rebuild gold aggregates
df_silver_full = spark.read.table("silver.orders")

df_gold = build_aggregates(df_silver_full)

df_gold.write \
    .format("delta") \
    .mode("overwrite") \
    .table("gold.daily_sales")

This is the basic pattern. In production, each step would have error handling, retries, and notifications wired in. You would also parameterise the pipeline so you can backfill easily — something that saves you at 2 AM when someone says “the dashboard numbers have been wrong since Tuesday.”

Things to Be Careful About

• Schema evolution in bronze can break silver. If the source adds a column and you enabled auto-merge in bronze, your silver transformation will silently ignore the new column unless you update it. Set up schema change alerts.
• Do not create too many layers. I have seen teams add a “platinum” layer between silver and gold, then a “diamond” layer, and suddenly nobody knows where the real numbers live. Three is enough for most cases.
• Partitioning in the wrong place. Partition bronze by ingestion date, not business date — you want to reprocess easily. Partition gold by the dates your queries filter on.
• Small file problem. Streaming writes into Delta can create lots of tiny files. Run OPTIMIZE regularly, especially on silver tables that get frequent incremental writes.

Wrapping Up

The medallion architecture is not magic — it is just a sensible way to organise your data so that raw ingestion, cleaning, and business logic do not get tangled together. Bronze keeps you safe when ingestion goes wrong, silver gives you a clean foundation to build on, and gold makes your data actually useful to the business.

Start simple. You can build a solid medallion pipeline with a handful of Delta tables and a notebook. Once you get the pattern down, you will find it much easier to reason about data quality, lineage, and debugging — even when things inevitably break at the worst possible time.