Data Governance

A promise
the data keeps

Data quality does not fail loudly. Nothing crashes; the numbers simply become wrong, and somebody notices weeks later in a meeting. The only defence is to write down every assumption about the data as something a machine can check, and to check it on a schedule.

System: Passenger profiling platform
Scale: 43 contracts, hourly and on every write
Role: Sole architect and engineer
Stack: YAML / SQL / CloudWatch

The challenge

The product's output is inference, so a wrong value does not look wrong. Upstream feeds changed formats without warning and nothing objected.

The approach

Move every assumption out of the code and into a declared registry that runs on a schedule, records its verdicts, and reports its own health.

The result

Whole classes of silent corruption are now caught before a customer sees them, and the registry doubles as the written definition of what each field means.

01New dataA batch has just landed. This is the moment to check it, while it is still possible to hold it back, rather than after a customer has already built a decision on top of it.Celery / post-write trigger 02ContractsEach contract is a named, written-down promise: this field only ever holds these values, this table is never older than this many minutes, every record here has a matching record there. They live in one list as configuration, so adding a check is a line of settings rather than new code — which is why there are dozens of them instead of a handful.YAML registry / templated SQL / SET LOCAL statement_timeout 03LedgerFailures are not just alerts that scroll past. Every violation is recorded with its history, so you can ask how long something has been wrong and whether it is getting worse. That history is what turns a surprise into a trend.PostgreSQL / 90-day retention 04AlertThe point of all of it is arriving first. A wrong number that reaches a customer dashboard costs trust; the same wrong number caught at this stage costs an afternoon.CloudWatch / SNS 05Liveness signalThe checker also reports that it ran at all. Without this, a checker that quietly stopped would look exactly like a system with no problems — the most dangerous failure a monitoring system can have.CloudWatch custom metric

Fig. 03 — The checker also reports whether it is still alive

Hover any step to read what it doesTap any step to read what it does

A promisethe data keeps

The challenge

The approach

The result

A promise
the data keeps