Medallion Architecture in Production: What 15+ Lakehouse Deployments Taught Us

Seventy percent of data teams over-invest in ingestion and under-invest in curation. The result: a Bronze layer that's secretly Silver, a Silver layer nobody trusts, and a Gold layer analysts route around because it's perpetually three weeks stale.

Medallion architecture in production is not a naming convention. It's a governance contract. After deploying Bronze/Silver/Gold across 15+ enterprise lakehouses on Databricks, Delta Lake, and Azure Data Lake Storage, we've watched the same failure modes repeat — regardless of team size, industry, or budget. The pattern itself is sound. Execution is where organizations unravel.

This post is a direct account of what works, what doesn't, and the specific engineering decisions that separate a durable lakehouse from a well-labeled data swamp.

Why this matters if you're running a modern data platform

If you're a data engineering lead who's already past the "should we do medallion?" question, you're probably dealing with one of these right now:

• Analysts distrust Silver and query Bronze directly, defeating the entire point
• Silver has metastasized — 400 tables, unclear owners, no deprecation path
• Gold tables have conflicting definitions across BI and ML teams
• A source schema change upstream cascades into a 3-day incident

These aren't architecture problems. They're governance problems that disguise themselves as architecture problems. And they compound.

At 10 tables, you can explain your model over coffee. At 100, you need written conventions. At 1,000, you need Unity Catalog, ownership metadata, per-layer SLAs, and automated freshness checks — or the lakehouse becomes load-bearing technical debt that blocks every downstream initiative.

The stakes are real: poorly designed medallion means ML teams train on Silver data that was never deduplicated. Finance can't reconcile numbers because two Gold tables compute revenue differently. And your best engineers spend 60% of their week firefighting instead of building.

If any of that resonates, here's what we've learned.

Bronze, Silver, Gold: where each layer actually breaks

Bronze: the immutable contract you're probably violating

Bronze has one job — land raw data, exactly as received, forever. No transformations. No cleaning. No type casting. No filtering.

The most common violation we see in new engagements:

# ❌ Anti-pattern: cleaning inside Bronze
df = (
    spark.read.format("json")
    .load("s3://raw/orders/")
    .dropna(subset=["order_id"])            # filtering destroys auditability
    .withColumn("amount", col("amount").cast("double"))  # type coercion in Bronze
)
df.write.format("delta").mode("append").save("/bronze/orders")

The argument sounds reasonable: "We're just normalizing nulls." But six months later, a compliance audit asks what you actually received from the source system — and you've already destroyed that evidence. Bronze is your paper trail.

The correct pattern uses Databricks Auto Loader with schema evolution enabled:

# ✅ Correct: raw ingestion, no transformations, metadata columns only
checkpoint_path = "/checkpoints/bronze_orders"
 
(
    spark.readStream
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.schemaLocation", checkpoint_path)
    .option("cloudFiles.schemaEvolutionMode", "addNewColumns")
    .load("s3://raw/orders/")
    .withColumn("_ingested_at", current_timestamp())
    .withColumn("_source_file", input_file_name())
    .writeStream
    .format("delta")
    .option("checkpointLocation", checkpoint_path)
    .option("mergeSchema", "true")
    .outputMode("append")
    .table("bronze.orders_raw")
)

Two metadata columns — _ingested_at and _source_file — cost nothing and save hours during incident investigation. schemaEvolutionMode: addNewColumns means Bronze absorbs upstream schema changes without breaking pipelines. That is Bronze's core promise: no matter what the source system does, data lands.

Silver: where architectures go to die

Silver is where real engineering lives — deduplication, type coercion, SCD handling, cross-domain joins, business rule application. It's also where we see the most catastrophic failures.

Anti-pattern 1: Silver as a pass-through. Teams create silver.orders that is 98% identical to bronze.orders_raw, with a few CAST calls and renamed columns. That's not Silver — that's a copy with extra latency and no additional trust.

Anti-pattern 2: Unbounded Silver proliferation. Without naming conventions and ownership, Silver grows sideways. We've inherited environments with 400+ Silver tables in a single workspace, no lineage graph, and zero clarity on which tables are live.

A naming convention that prevents this:

silver.<domain>_<entity>_<grain>

# Examples:
silver.sales_orders_daily
silver.hr_employees_current
silver.finance_invoices_validated

Every Silver table should have one owner, a documented purpose, a freshness SLA, and a downstream consumer count tracked in Unity Catalog tags. Non-negotiable.

For slowly changing dimensions — the transformation that separates real Silver from pass-throughs — this Delta MERGE pattern holds up in production:

-- SCD Type 2 in Silver using Delta Lake MERGE
MERGE INTO silver.customers AS target
USING (
  SELECT
    customer_id,
    email,
    country,
    current_timestamp() AS updated_at,
    true AS is_current
  FROM bronze.customers_raw
  WHERE _ingested_at > (SELECT MAX(updated_at) FROM silver.customers)
) AS source
ON target.customer_id = source.customer_id AND target.is_current = true
WHEN MATCHED AND (
  target.email   != source.email OR
  target.country != source.country
) THEN UPDATE SET
  target.is_current = false,
  target.valid_to   = current_timestamp()
WHEN NOT MATCHED THEN INSERT (
  customer_id, email, country, updated_at, valid_from, valid_to, is_current
) VALUES (
  source.customer_id, source.email, source.country,
  source.updated_at, current_timestamp(), NULL, true
);

Gold: stop putting business logic in Silver

Gold tables are what BI tools, ML feature stores, and reporting APIs actually consume. They should be domain-specific (named after the business process, not the technical entity), aggregated to the consumption grain, and backed by explicit SLAs.

-- Gold: revenue KPI table for Finance, rebuilt on a 60-minute schedule
CREATE OR REPLACE TABLE gold.finance_revenue_monthly AS
SELECT
  DATE_TRUNC('month', order_date)   AS month,
  region,
  product_category,
  SUM(gross_revenue)                AS gross_revenue,
  SUM(net_revenue)                  AS net_revenue,
  COUNT(DISTINCT customer_id)       AS unique_customers,
  current_timestamp()               AS refreshed_at
FROM silver.sales_orders_daily
WHERE is_current = true
GROUP BY 1, 2, 3;

The mistake teams make: putting aggregation logic in Silver because "it's reusable." It isn't. Two business units will define "revenue" differently, and when they do, you want that disagreement surfaced at the Gold layer with explicit ownership — not buried inside a shared Silver table that half the org depends on.

Unity Catalog: the governance layer that makes medallion real

Without Unity Catalog (or an equivalent), medallion is a naming convention, not an architecture. Unity Catalog enforces it:

• Catalog-level isolation: prod_bronze, prod_silver, prod_gold as separate catalogs with IAM-backed access controls — analysts can't reach Bronze even if they want to
• Column-level masking: PII columns masked in Bronze and Silver; only de-identified Gold surfaces to consumers
• Tags for SLA tracking: {"layer": "gold", "sla_minutes": "60", "owner": "data-platform"} on every table, queryable via SHOW TAGS
• Automatic lineage: Unity tracks Bronze → Silver → Gold data lineage for Delta tables at no extra cost

The lineage graph alone justifies the Unity Catalog investment. When a source schema changes at 2am, you see exactly which Silver and Gold tables are downstream before your oncall engineer finds out the hard way.

The honest lesson: technical patterns are the easy part

After 15+ deployments, the hardest thing we've learned is this: Silver layer ownership doesn't emerge naturally, and it degrades faster than any other layer.

You can have perfect Auto Loader ingestion in Bronze, clean MERGE logic in Silver, and well-named Gold tables — and still end up with a Silver layer nobody trusts eighteen months later. It happens when ownership is diffuse. When team A creates silver.orders and team B creates silver.order_events and six months later nobody knows which one is the source of truth.

The fix isn't technical. It's a data contract enforced in code: every Silver table has one owner, a documented purpose, an explicit freshness SLA, and a deprecation path. We implement this as a YAML manifest checked into the same repository as the pipelines, validated in CI with a Great Expectations suite that checks freshness and row counts before any Gold rebuild is allowed to proceed.

The second hard lesson: Bronze schema drift will break you if you defer addressing it. When a source system quietly adds 40 new columns overnight, schemaEvolutionMode: addNewColumns absorbs it without breaking the Bronze stream. But those 40 columns still need to be audited before Silver pipelines can safely promote them. Build a schema change alert into Bronze ingestion on day one — not after the first incident.

Medallion architecture in production is a discipline, not a diagram. The teams that sustain it long-term are the ones who treat layer boundaries as contracts, not suggestions.

If your Bronze/Silver/Gold setup is drifting — Silver that nobody owns, Gold that BI teams rebuild themselves, or a Unity Catalog you haven't wired up yet — bring it to a 30-minute call. We'll look at your current architecture and tell you exactly where the risks are and what to fix first.

Book a discovery call →