more detail
If you have been asked to onboard a new data source into a data platform, you may have been handed a board with a handful of vague tickets and told to “get it into the lake.” What actually needs to happen between that conversation and the point where a business analyst or AI agent can confidently query the data? This post walks through the delivery framework that experienced data engineering teams use, built around the medallion architecture.
The medallion architecture — also called multi-hop or Bronze-Silver-Gold — is a data design pattern that organises data into three progressive layers of quality and readiness.
Bronze holds raw, unmodified data exactly as it arrives from the source. Nothing is changed, nothing is discarded. It is the permanent record of what came in and when.
Silver is the curated layer. Schemas are enforced, business keys are resolved, duplicates removed, and quality validated. A consumer can use Silver data without needing to understand the source system at all.
Gold is shaped for a specific consumer: a KPI dataset, a dashboard aggregate, a model input, or an AI agent feed. Gold exists to serve a committed use case, not to store data generically.
The pattern was introduced by Databricks, who also created Delta Lake — an open-source transactional storage format that adds ACID guarantees, schema enforcement, and time travel on top of cloud object storage. The medallion pattern was designed as the recommended way to organise data across Delta tables. Since then it has been adopted well beyond the Databricks ecosystem: on AWS, the same pattern is implemented with Apache Iceberg on S3, and the layered logic applies regardless of which storage format is underneath. The separation of concerns is what matters — ingestion logic lives in Bronze, business logic lives in Silver, and consumer contracts live in Gold. Each layer can be rebuilt from the one below independently.
The medallion architecture is not an engineering preference — it is a response to a business problem. Organisations that skip this structure typically end up with pipelines that are tightly coupled to their source systems: when the source changes a schema, the dashboard breaks. When a new team wants the same data shaped differently, someone builds a parallel pipeline from scratch. When an auditor asks where a number came from, nobody can answer with confidence.
Separating data into layers solves each of these directly.
Reliability and trust. Because Bronze preserves the raw record and Silver enforces a validated contract, business users know exactly what quality guarantee they are working with. A metric produced from Gold can be traced back through Silver to Bronze and ultimately to the source event. That traceability is what turns data from something people argue about in meetings into something they make decisions from.
Speed to new use cases. Once a source is in Silver, adding a new consumer does not require touching the ingestion pipeline. A new Gold model, a new dashboard, or a new AI agent input can be built on top of the existing curated layer in days rather than weeks. The investment in onboarding a source properly pays compound returns every time a new use case is added.
Cost control. Raw data is stored once in Bronze. Transformations are layered, not duplicated. Without this discipline, organisations frequently find they have the same data copied four times across four pipelines built by four teams who did not know about each other. Each copy has slightly different logic, and none of them agree.
Data classification and compliance. Every data source that enters a platform needs to be classified before it reaches consumers. Classification answers the questions that legal, security, and compliance teams will eventually ask: what kind of data is this, how sensitive is it, who is permitted to access it, and what obligations come with holding it? Common classification tiers range from public and internal through to confidential and restricted, with each tier carrying different access controls, retention rules, and audit requirements.
PII — Personally Identifiable Information — sits at the most sensitive end of that spectrum and deserves specific treatment. PII includes any data that can identify an individual directly or in combination: names, email addresses, device identifiers, location data, and in many cases transactional records that can be linked back to a person. Under GDPR, CCPA, and equivalent frameworks, the moment PII enters your platform you have legal obligations around consent, storage, access, retention periods, and the right to erasure. Those obligations do not disappear because the data is in a lake.
The medallion architecture handles this well because classification happens at Silver, where the data is understood well enough to make the call. Fields can be tagged in the catalog, masked or tokenised before they reach Gold, and access grants can be scoped so that an analyst querying a Gold model never touches raw PII they have no business reason to see. This is significantly harder to enforce in a flat, undifferentiated data lake where everything lands in the same bucket and access is managed with broad permissions.
A data contract is a formal, versioned agreement between a data producer and a data consumer. It specifies what data will be delivered, in what shape, at what quality, and on what schedule. Unlike an informal handshake or a wiki page that drifts out of date, a data contract is machine-readable and enforceable — if the producer breaks the contract, the pipeline fails loudly rather than silently corrupting downstream outputs.
The concept addresses a specific and common failure mode in data platforms: a source team changes a column name or drops a field, the ingestion pipeline keeps running, and three weeks later a finance team discovers their monthly report has been wrong since the change. By the time the issue surfaces, the trust damage is done and the investigation is painful.
In a medallion architecture, data contracts live at two critical points. The Silver contract defines the curated output of the Bronze-to-Silver transformation: the expected keys, schema, data types, null constraints, and quality guarantees that downstream consumers can rely on. It is owned by the data engineering team and represents their commitment to consumers. The consumer contract (or Gold contract) defines what a specific use case — a dashboard, an AI model, a reporting dataset — will receive, and is jointly agreed between the data team and the consumer owner.
Why use contracts instead of documentation? Documentation describes intent; a contract enforces it. When a Silver contract is in place, any transformation that would break it is caught at the validation step in Stage 4 rather than discovered by a consumer after the fact. New consumers onboarding to an existing Silver source can read the contract and know exactly what they are getting without needing to interrogate the source system or ask the engineering team. And when the source itself changes, the contract makes the scope of the impact immediately visible: every downstream consumer whose contract depends on the changed field knows they are affected.
Practically, a data contract includes the schema (field names, types, nullability), business keys and uniqueness constraints, expected row volume and freshness SLA, data classification tier and any PII fields, and the owner and agreed review cadence. It does not need to be a complex document — a well-structured YAML file checked into version control alongside the pipeline code is enough to deliver most of the benefit.
A well-run source onboarding follows seven stages in a fixed order. The most common cause of rework is skipping or merging the early ones.
Stage 1: Discovery and approval. Before a single line of code is written, confirm why the source matters, who owns it, and who will consume the data. Light profiling — row counts, null rates, schema shape — should happen in an isolated environment. The stage ends with a stakeholder approval or a deliberate stop. A source without a committed consumer should not be onboarded.
Stage 2: Secure access and platform setup. Establish the service principal, secrets management, catalog namespace, storage paths, and environment grants. This work is unglamorous but blocking failures here cost days later.
Stage 3: Bronze availability. Build the extractor or loader, land the raw data, schedule the job, and add basic quality and freshness checks. At this point the source is visible in the platform but not yet ready for business use.
Stage 4: Silver availability. Define and publish the Silver data contract — expected keys, schema, data types, null constraints, and quality guarantees — then build the Bronze-to-Silver transformation and add reconciliation checks. The contract is the commitment to consumers; the transformation is how you honour it. This is often the most time-intensive stage because it is where source-specific complexity surfaces: late-arriving records, schema drift, unexpected nulls in business keys.
Stage 5: Governance and operational readiness. Classify all fields by sensitivity tier, identify and tag any PII, assign data ownership in the catalog, and document lineage from source to consumer. Then add the operational layer: alerting, a runbook covering stale data and backfill procedures, and sign-off from the source owner. Teams that defer this until the end of a milestone consistently find they cannot support the source in production and cannot demonstrate compliance if asked.
Stage 6: Gold or consumer readiness. Agree and publish the consumer data contract — what the consumer will receive, in what shape, and on what schedule — then build the output and validate it with the actual consumer. Without this step the source is delivered but not used.
Stage 7: CI/CD promotion. Formalise the pipeline promotion path between layers so the source can be rebuilt or backfilled without manual intervention.
The most common failure mode is starting Bronze implementation before source ownership and access are confirmed. The second is onboarding a source without a committed consumer: the data lands in Silver, nobody queries it, and months later nobody can remember why it was a priority.
Other recurring mistakes: mixing platform foundation work with source ingestion in the same delivery item, writing stories that describe activities rather than observable outcomes, and leaving monitoring and runbook work until the final week of a sprint.
A well-understood source with clear ownership and a defined consumer takes around 20 engineering days end to end. That excludes waiting time — access approvals, vendor responses, source readiness delays — which should be tracked separately. Sources where ownership, access method, volume, or rate limits are unclear should have discovery separated from implementation before the full onboarding is committed.
We can help you unleash your data’s potential. Get in touch with the DataPhoenix team here.