Two ways of keeping a pipeline healthy are competing in many data teams. One is the handshake: the people who own the source system and the people who consume its data know each other, chat on messaging, and warn one another before changing things. The other is the data contract: a written, versioned, partly automated agreement about what the data will look like. I will argue for contracts, but only for the data that deserves them, and I will explain why the handshake loses as a company grows.
The problem a contract answers
The engineers who run an orders service design its database for the application. They rename a column, change a status code from text to a number, or stop populating a field that the app no longer needs. None of this is wrong from the application’s point of view. Downstream, a revenue dashboard breaks, or worse, keeps running and shows a quietly wrong number. The analysts find out when a manager asks why Monday looks odd.
The root cause is that the pipeline treats the source’s internal tables as a stable interface when nobody ever promised that. A data contract turns the promise explicit.
What goes into one
Formats vary; many teams keep the contract as a YAML or JSON file beside the code, and open specifications such as the Open Data Contract Standard exist. The contents are what matter:
- Schema. Column names, types, whether each can be null, and which column uniquely identifies a row.
- Semantics. What “order value” means: before or after discount, with or without GST, in rupees or the order’s currency. Most real disputes are about meaning, not types.
- Quality expectations. A value of status must be one of a stated list; order amounts must be positive; no more than a small percentage of null customer ids.
- Freshness and volume. For example, data lands by 6 a.m. IST, with row counts in an expected range.
- Ownership and change rules. A named owning team, a contact for incidents, and how much notice is given before a breaking change.
- Sensitivity. Which fields hold personal data, so that access and retention follow from the contract.
The part that makes it more than a document
A contract that lives in a wiki is a wish. The value comes from enforcement. Checks run in the producer’s deployment pipeline, so that a change which removes a promised column fails the build before it reaches production, in the same way that a failing unit test blocks a release. Checks also run on the data itself as it is loaded, quarantining rows that break a rule instead of letting them flow into reports. This is closely related to contract testing in software, where a service’s consumers state what they depend on and the provider is verified against it.
Versioning gives a way to evolve. Adding an optional column is usually safe; renaming or removing one is breaking. Breaking changes ship as a new version, with the old one kept alive for an agreed period while consumers migrate.
The case for the handshake
Fairness requires saying what contracts cost. They take time to write. They need producers to accept responsibility for something that used to be someone else’s problem, and that is a hard political ask, particularly when the source team is measured only on application features. A badly run contract programme becomes a pile of stale YAML files that everyone ignores, which is paperwork in the worst sense.
For a team of fifteen, where the engineer who owns a table sits near the analyst who reads it, a handshake works and costs nothing. It stops working when the producers multiply, when people change jobs, and when the consumer is a dashboard whose author the producer has never met. At that point, who you happen to know is no longer a control.
Where I would start
Not everywhere. Pick the few datasets whose failure hurts most: those feeding financial reporting, regulatory submissions, customer-facing numbers or machine-learning models. Write contracts for those, enforce at least the schema and freshness checks automatically, and name an owner on the producing side. Learn from the first handful before extending.
Keep the rest on the handshake, with a lighter habit: a shared channel where schema changes are announced. A contract is a tool for important interfaces, and applying it to every scratch table will only teach people to resent it.
Leave a Reply