Data Governance
A promise
the data keeps
Data contracts and automated data quality checks in production
Data quality does not fail loudly. Nothing crashes; the numbers simply become wrong, and somebody notices weeks later in a meeting. The only defence is to write down every assumption about the data as something a machine can check, and to check it on a schedule.
In brief At E23 I moved every assumption about the data into 43 declarative data contracts, checked hourly and on every write, with failures recorded in a queryable ledger and alerted through CloudWatch. Silent corruption is now caught before a customer sees it.
The challenge
The product's output is inference, so a wrong value does not look wrong. Upstream feeds changed formats without warning and nothing objected.
The approach
Move every assumption out of the code and into a declared registry that runs on a schedule, records its verdicts, and reports its own health.
The result
Whole classes of silent corruption are now caught before a customer sees them, and the registry doubles as the written definition of what each field means.
Hover any step to read what it doesTap any step to read what it does