In the age of AI, the problem takes center stage

Spark. Flink. Hadoop. Debezium. Kafka Connect. Every one of these came into existence to solve a specific problem at a specific time, and then got open-sourced to attract developers to use it and contribute back. That’s a good thing. Open source is a great place to learn and to give back to the community.

But I’m not going to deep-dive into the histories of these tools, and here’s why: after a tool succeeds and gets adopted, its community tends to paint a beautiful picture of itself — the benefits, the capabilities, the support, the ecosystem. Don’t let any of that drive how you’re going to solve the problem in front of you.

Tools get you 0 to 1. Maintaining them is the headache.

Nothing is above the urgent business requirement. If a tool gets you from zero to one fast, great — use it. But don’t get trapped.

Maintaining tools, especially open-source ones, is a different game from adopting them. Security patches. Comprehensive testing. Migrations. It’s easy when you’re thinking about a handful of pipelines. Once your pipelines are significant in number, it sucks. And if there’s a failure, you’re cooked — because data impacts business decisions, and it has to be fixed then and there, not next sprint.

That’s the part the glossy adoption stories leave out. The demo always works at small scale. The bill comes due at large scale.

My Debezium story

Debezium gets treated as the gold standard for change data capture. And I’ll be honest — there’s an issue with its Postgres connector that was a pain in the ass for me for quite some time, until I finally decided to ditch both Debezium and Kafka Connect and build my own solution.

Here’s the thing nobody tells you up front: things work at small scale. At large scale, managing Kafka and Kafka Connect (where Debezium deploys) is a task on its own. Then security patches land on Kafka and Kafka Connect, and now you’re upgrading and downgrading versions, testing every combination to make sure everything still works, watching for red flags, and fixing them by hand. Woof.

None of that work shows up in the “look how easy CDC is” tutorial. But it’s most of the actual job once you’re past the demo.

To be clear, this isn’t a knock on Debezium or Kafka. They were written, in their time, to solve a specific use case, and they do it well within that envelope. The mistake is mine to avoid: assuming a gold-standard tool is automatically the right answer for my problem at my scale.

Keep the problem statement at the center

I force myself to keep my problem statement at the center of my thought process. I will not tweak the problem statement just so I can use an open-source tool, or some advanced technology that ships N features I may not need — features that aren’t significant for me, no matter how vast the community support is.

That’s the trap, and it’s subtle. You start with a clean problem. A shiny tool almost fits. So you quietly reshape the problem to match the tool. Now you’re maintaining the tool’s worldview instead of solving your actual need. Open source has real signal, but it also has noise and hype — and the hype is loudest exactly when the actual agenda, your problem statement, takes the back seat.

This is the same muscle as backend engineering. Distributed systems theory, reading systems patterns — all of it helps build intuition. But building actual systems is about judgment, the kind that’s built up over years. Intuition helps you build better systems that break less. You’ll have an MVP in less time, and scaling gets easier. But you should be aware: this intuition cannot be grasped via vibe-coding. It takes time, energy, conscious thinking, and real brainstorming — while you’re building the solution and while you’re maintaining it.

Why this matters more now, not less

In the age of AI, judgment and the problem are everything.

I think the communities of the future will be more sophisticated about the problems they solve. Less deliberation over syntax, less arguing about the forty ways to write a piece of code — more solving genuinely hard problems. The low-level stuff is getting automated. What’s left is the part that was always the real work: framing the problem correctly and exercising judgment about the tradeoffs.

And here’s the uncomfortable part. In the age of AI, you’re not really writing software anymore — but AI tends to vomit trash. It’s your responsibility to keep that code from reaching prod. The model doesn’t carry the pager. You do. When a pipeline breaks at 2 a.m. and a business decision is riding on bad data, “the AI generated it” is not an answer.

So the skill that compounds isn’t knowing the tool of the month, and it isn’t typing fast with an assistant. It’s judgment: keeping your problem statement sacred, knowing when a tool earns its maintenance cost, and knowing when to ditch the gold standard and build the thing that actually fits.

Tools may help you get from 0 to 1. Past that, it’s on you.

These are some scribbles of an idea — more to come.