The hardest systems are the boring ones

When most developers picture a hard technical problem, they picture something that looks hard. Novel machine learning. Distributed consensus. Something with a paper behind it.

Years in the field taught me something less flattering: the real complexity usually sits inside systems that sound trivial when you describe them out loud.

"It's just leads"

At Techception we ran a lead distribution platform for US real estate. Ingest leads, store them, hand them out to agents.

Say that in a meeting and nobody blinks. It is CRUD. It is the first thing you build in a tutorial.

Here is what it actually was:

• Insert and replicate leads arriving continuously from multiple sources
• Distribute them to thousands of agents, each with their own filters and entitlements
• Keep every record consistent across 11 related tables
• Do it with effectively zero room for schema changes after deployment

And the number that changes everything:

230,000,000 leads × 11 tables = 2,500,000,000+ records to keep consistent

Two and a half billion rows. No individual operation is interesting. The interesting part is that there are so many of them that every ordinary decision becomes irreversible.

Scale does not add complexity, it removes options

This is the part I did not appreciate early in my career.

At small scale, almost every architectural decision is recoverable. Wrong index? Add one. Bad column type? Migrate it over a coffee break. Denormalized when you should not have? Refactor next sprint. The cost of being wrong is an afternoon.

At 2.5 billion rows, those same decisions calcify:

• An ALTER TABLE on a hot table is not a migration, it is an outage with a changelog entry
• A missing index is not slow, it is a query that never finishes
• The wrong partition key is not suboptimal, it is a rewrite of the entire dataset
• A schema you want to change is a schema you are going to live with

You cannot throw more servers at it either. Horizontal scale helps with throughput. It does nothing for a partitioning strategy that put related records on different shards, and it actively hurts when your consistency requirements span the boundaries you just created.

The constraint stops being performance and becomes reversibility.

Which is why the work happens before the code

The most critical engineering on that platform happened before anyone wrote a line of it. Partitioning strategy. Index design. Which of the 11 tables could tolerate eventual consistency and which absolutely could not. What we were willing to denormalize, knowing we could never cleanly undo it.

Those sessions felt unproductive at the time. No commits, no demo, nothing to show in standup. They were the only part of the project where the decisions were still cheap.

Every one of them either empowered the team for the next two years or haunted it. There was not much in between.

The takeaway

If a system sounds boring, ask what happens to it at a thousand times the volume. The answer is usually that a decision someone made casually in month one becomes the thing the whole team works around in year two.

Novel problems get attention, careful design, and senior people in the room, precisely because they look hard. Boring problems at scale get a junior engineer and a two-week estimate, because everyone has built a CRUD app before.

That asymmetry is where the expensive mistakes live.

When I first wrote about this I promised a part two on how we actually solved it. That took longer than intended, and it turned into a full case study rather than a post: Streamlining data flow for 230 million leads, covering the partitioning scheme, the indexing strategy, and how query time came down from 15 minutes to milliseconds.