Data contracts: stop breaking the team downstream

Most data breakages start with an upstream change nobody announced. A data contract is a written agreement between the team that produces a dataset and the teams that use it, checked by a machine in three places. This post draws what a contract holds, where it is checked, and which changes break it.

On a Tuesday, the app team ships a new version. In their order file, the column amount is now called order_amount. It's a tidy rename, and nobody thinks to tell the data team.

On Wednesday morning, Ananya opens the sales dashboard. Tuesday's revenue says ₹0. The nightly load ran, and it reported success. So did the one after it.

Nothing was broken in the pipeline. What broke was a promise that was never written down.

A promise, written down

A data contract is a written agreement between the team that produces a dataset and the teams that use it. The producer promises what the data will look like and how it will behave. The consumers can then build on that promise without guessing.

the parcel is the data; the signed note is the contractProducerthe app teamConsumerthe data teamorders v1.3what's insidewhen it arriveshands overbuilds on itChange coming?tell the consumer first
A data contract is the signed note on the parcel. The consumer builds on the note, not on guesses.

Think of a courier delivery. The parcel is the data. The signed note on top says what's inside, how it's packed, when it will arrive, and who to call if something is wrong. If the sender wants to change any of that, they tell you first, before the parcel turns up different.

A contract written once in a wiki is not enough. The useful kind is a file in version control that a machine checks on every change.

What a contract holds

A contract answers the questions a consumer would otherwise have to ask on chat.

one contract per dataset, kept next to the codeorders.contract.yamlschemaorder_id, amount, ...meaningamount: ₹, incl. taxrulesamount >= 0freshnessin by 6 a.m.ownerapp-orders teamchanges30 days' noticeWhat it looks likecolumns and typesWhat it meanseach column in wordsWhat must be truequality rulesWhen it arrivesfreshnessWho answersa team, not a personHow it changesnotice and versions
What a contract holds. Every line answers a question a consumer would otherwise ask on chat.
  1. Schema. The columns, their types, and which ones are required.
  2. Meaning. What each column means in words. amount is the order value in rupees, including tax, before refunds.
  3. Quality rules. order_id is unique. amount is never negative. country is a two-letter code.
  4. Freshness. How late the data may be. "Yesterday's orders are in by 6 a.m."
  5. Owner. A team, not a person, and how to reach them.
  6. Change policy. How changes are announced, and how much notice consumers get.

You don't have to invent a format for this. The Open Data Contract Standard (ODCS) is an open standard for writing data contracts in YAML, maintained under the Linux Foundation's Bitol project. Whether you use it or a simpler file of your own, the point is the same: one file per dataset, kept next to the code, read by people and by machines.

Three places it is checked

A contract nobody checks is a wish. It earns its keep when it is checked in three places, each catching a different kind of problem.

the same contract file, checked in three placesProducerchanges the appCI checkbefore the changeships1Bronzeas it arrivedIngestion checkbefore rowsreach silver2Silveronly good rowsfailsQuarantine+ alert the ownerMonitoringon time?usual row count?3Contract fileone source of truthsame file
Three checkpoints. The earlier the check, the cheaper the break.
  1. In the producer's CI, before the change ships. CI is the automatic test run on every pull request. The app team's pull request that renames amount runs a check against the contract, and fails. The producer finds out at their desk, not the consumer at 9 a.m. This is the cheapest place to catch a break, and the one teams skip most.
  2. At ingestion, on the bronze-to-silver hop. Bronze keeps what arrived, in the source's own shape. Before rows move into silver, the job compares what arrived with the contract. A missing required column stops the load. A row that breaks a quality rule goes to quarantine with the reason. This is the safety net for producers who don't run the CI check, such as an outside vendor's file.
  3. In monitoring, after the data lands. Freshness and volume are promises about time and size, and you can only check them by watching. Did today's file arrive by 6 a.m.? Is the row count close to the usual?

The ingestion check is the same idea as the quality rules in config. The contract is where those rules come from, and who agreed to them.

The rename, replayed

The DataTwin Studio Problem Lab recorded this exact story on a real Spark and Delta Lake run. The numbers below are from that recorded run.

The source file for 27 October arrived with order_amount instead of amount, plus a new coupon column. The job read it with the old schema, found no amount, and filled it with nulls. Three rows had no amount, and 27 October revenue showed ₹0 instead of ₹1,990.

The naive fix was to let the schema evolve automatically. The load succeeded, silver grew two extra columns, and revenue still showed ₹0, because every report reads amount, not order_amount.

The right fix was a contract check before the write. The check saw that a required column was missing, stopped the load, wrote nothing, and alerted the source owner. A one-line mapping from order_amount to amount was added to the ingestion settings by a reviewed pull request. On the re-run, the check passed, the new coupon column was accepted, and 27 October showed ₹1,990.

Breaking or safe?

The rename and the new coupon column arrived in the same file. One was a break, and the other was fine. Learning to tell them apart is most of the skill.

would a consumer who did nothing wrong get a wrong answer?BREAKING: major version, announce firstSAFE: minor version, ship itRename a columnamount to order_amountDrop a columnChange a typenumber to textChange the meaningamount now excludes taxAdd an optional columncoupon, null for old rowsLoosen an unused rulesafe, but get a quick review
Sort every change into one of two bins before it ships. The bin decides the version number.

A change is breaking if a consumer who did nothing wrong would get a wrong answer or an error. Renaming a column, dropping one, changing its type, making an optional column required, or changing what a column means are all breaking. The last one is the sneakiest: amount that quietly starts excluding tax still passes every schema check.

A change is safe if every existing consumer keeps working without touching their code. Adding an optional column is safe. So is loosening a rule in a way no consumer relies on, though that one deserves a quick review.

That sorting gives you versioning:

How it fits with the rest

A contract is not a separate system. It plugs into two things you may already have.

And in the medallion layers, the contract is checked where bronze becomes silver. That's why bronze keeps everything as it came: when the check stops a load, you have the original file to replay once the fix is in.

What goes wrong

Five questions to check your own setup

  1. For your most important dataset, could you show someone the contract file in under a minute?
  2. If the source team renamed a column today, where would it be caught first: their CI, your ingestion, or a user?
  3. Does a missing required column stop the load, or fill it with nulls?
  4. Who gets the alert when freshness slips, and is that a team or one person on leave?
  5. When did a contract last change, and who reviewed it?

The rename will happen again. A contract decides whether it fails one pull request or one morning of your users' trust.