Two halves of the same job can have completely different limits. Building the data needs processing power; saving it is limited by how fast a single database will accept writes. Join them together and the slower half sets the pace for everything.
System
Passenger profiling platform
Scale
1B+ rows, written continuously
Role
Sole architect and engineer
Stack
Python / Parquet / S3 / PostgreSQL
In brief At E23 I separated profile generation from database writes in the passenger profiling platform: workers write Parquet part files to S3 and publish one atomic manifest, and a single consumer merges them exactly once. Compute and storage now scale independently, across 1B+ rows.
01
The challenge
Profile generation and database writes were bound together, so the fleet spent most of its time waiting on write locks instead of working.
02
The approach
Put a durable buffer between them and make the handoff a single atomic write, so neither side has to trust the other to be available.
03
The result
Compute and storage now scale independently, a failure is contained to one file, and the buffer became the foundation of the analytical layer.
Fig. 01 — The manifest is the only atomic write in the handoff
Hover any step to read what it doesTap any step to read what it does