165 million requests and a database running out of connections

Context

Techception ran an interactive video wall SaaS. The platform served 165M+ requests at 99.95% uptime, built serverless on Azure Functions with Service Bus and Cosmos DB alongside MongoDB.

Serverless was the right instinct for the traffic shape. Video wall installations are bursty by nature. It was also the source of the problem, for reasons that only appear at volume.

The problem

MongoDB connections had climbed past 1,800 and were heading for the ceiling.

The symptoms arrived well before the diagnosis:

• p95 latency creeping upward with no obvious code change behind it
• Timeouts clustering under load and clearing when load dropped
• Errors that would not reproduce in staging, because staging never generated enough concurrency to matter

The compute layer looked perfect throughout. Functions were scaling exactly as designed, executing successfully, reporting healthy. The failure was in a resource the function-level monitoring did not watch.

This is the structural flaw in serverless against a connection-oriented database, and I wrote about the general shape of it in Serverless scales, your database does not. Each concurrent execution is an isolated environment. Each one opens its own connection. There is no long-lived process to hold a pool, so concurrency and connection count are the same number.

Compute scaled linearly. The database did not scale at all.

Constraints

• The platform was live and serving customers. No rewrite-and-relaunch.
• Multi-tenant, so one tenant's burst could not be allowed to exhaust connections for everyone else.
• Data consistency requirements across tenants were strict.

Approach

Stop the bleeding first, then fix the architecture. The immediate work was reducing connection churn: reusing clients across invocations where the runtime allowed it, tightening pool configuration, and cutting the number of independent connection-openers in the request path. This bought headroom without a re-architecture.

Then move off serverless where it did not fit. The longer-term fix was re-architecting to a Dockerized system we controlled. Long-lived processes, real connection pools, and concurrency we set rather than discovered. That took availability to 99.9%.

This is the tradeoff stated honestly: we gave up scale-to-zero and got back predictability. For steady request-response traffic against a database, that is the right trade. For genuinely spiky event-driven work it would not have been.

Multi-tenancy with active replication. Tenant databases with active replication holding 99.9% data consistency, so tenant isolation was a property of the architecture rather than of application-level care.

RBAC as a first-class system. A custom role-based access control system with 15+ granular roles, built properly rather than accumulated as conditionals.

Outcome

A 78% reduction in connections, and the ceiling stopped being the thing we designed around.

What I took from it

Monitor the constraint, not the component. Every dashboard was green while the system was failing, because we were watching function health instead of the resource that was actually finite. Ask what the scarcest resource in the system is and instrument that.

Staging that cannot reproduce concurrency cannot find concurrency bugs. This class of failure is invisible until you have production traffic, which means production is where you find it. Load testing that mirrors real concurrency is not a nice-to-have for serverless systems, it is the only way this shows up early.

Serverless is a fit judgment, not a default. It is excellent for spiky, unpredictable, event-driven work. It is a poor default for steady traffic against a database, which is most applications. Choosing it because it sounds like less operational work is how you end up doing more, later, under pressure.