~$ whoami
A community about the problems in data engineering — not the tools. Ten years of building warehouses from scratch, and the lesson keeps repeating: the hard part is the data and the tradeoffs, not the tooling.
Microservices are a scaling pattern for code. But the moment you split a service, you split its data — and distributed data is where scaling actually gets hard. A tour of the Scale Cube, database-per-service, the dual-write trap, the outbox + CDC fix, sagas, and sharding, with the cost of each named.
We celebrate AI for the easy, well-documented work — apps, websites, CI/CD, boilerplate. The real test is the messy, under-documented problems with slow feedback: keeping something alive. A reflection on intuition over idealism, the limits of a probabilistic model, and a benchmark you can run with five money plants.
AI collapsed the cost of producing code. What's scarce now is the intuition to know what a system must do and what will quietly break — correctness first, scale second, and the judgment to direct AI well.
Spark, Flink, Debezium — tools get you 0 to 1; maintaining them is the headache. In the age of AI, judgment and a sacred problem statement are everything.
My first post. After years building big-data platforms from scratch, the hard part was never the volume — it's bad data, reliable sourcing, and asking the dumb questions first.