Running a production API with 1M requests/day

One million requests a day sounds like a serious infrastructure problem until you do the division. It is about twelve requests per second on average. A single modest API server, configured honestly, can handle that without breaking a sweat. The reason APIs die at a million requests a day is almost never the requests themselves. It is the database, the queue, the cache, or an endpoint that runs a query nobody bothered to look at since the day it shipped.

We run a production API that crossed this threshold, and the interesting part is how uninteresting the solution is. This is the playbook, in the order we applied it, with the failure stories that made us apply it.

The math first, so the panic is proportionate

Twelve requests per second is the average. The averages are a lie in the way averages always are — traffic is not a metronome, and a marketing email or a partner integration can spike you tenfold in a minute. So the real numbers that matter are the p50, the p99, and the worst minute of the day, measured under load, not the dashboard's nice round total.

We track a single headline number: p99 latency at the busiest minute, per endpoint. Everything else — throughput, error rate, concurrency — is diagnostic context around that number. If p99 stays flat while traffic doubles, the system is healthy. If p99 moves, the problem is a specific query or a specific dependency, and the dashboards are organised to make that obvious.

The boring stack that carries it

The API is a stateless HTTP service in front of Postgres, with Redis for caching and a queue for anything that writes or sends. One application image, two process types: a web server and a background worker. No microservices, no service mesh, no Kubernetes.

Statelessness is the load-bearing decision. Because the web process keeps no state, we can run several replicas and treat them as interchangeable. A request can land on any of them and get the same answer. This one property converts "scale the API" from an architecture problem into a replica-count problem, and it is the reason the million-request day did not require an architecture meeting.

The database is the real API

Every request that reaches Postgres is a request that could have been avoided, and our first optimisation pass was a hunt for avoidable ones. The classic offender is the N+1 pattern: an endpoint that lists a hundred items and runs a query per item, because the eager loading was never added. We found four of those in our own code and the p99 dropped by a third just from fixing them.

The second win was honest indexes. Not more indexes — honest ones. Every slow query in the log got one question: is there an index that serves this access pattern? When the answer was yes, we added it and measured the difference. When the answer was no, we changed the query or the schema, because an index that serves a bad design just makes the bad design faster.

Connection pooling deserves a paragraph of its own. At twelve requests a second with a handful of replicas, the number of open connections is trivial — until a spike multiplies the concurrency and every replica tries to hold its own pool. The fix is to route database connections through a pooler so the database sees a stable, bounded number of connections no matter how much traffic arrives. This is the single most common cause of "it worked in staging" at this scale, and it is pure configuration once you know to look.

Caching that does not lie

Caching at this volume is not optional if you want p99 under control, but a bad cache is worse than no cache. The cache stampede is the classic failure: an entry expires, a thousand concurrent requests all miss, and the origin takes a hit that looks exactly like an attack.

Two rules keep our cache honest. First, TTLs are jittered — the expiry time is base + random, so the herd never expires in lockstep. Second, reads through the cache are also reads of the source of truth on a schedule, so a stale-cache incident is a matter of minutes, not weeks. A cache that can go stale silently is not a cache; it is a second source of truth, and we do not run those.

We cache the expensive-to-compute and the frequently-read: session data, feature flags, product catalog reads, and anything an SDK poll loop hits on an interval. We do not cache writes, we do not cache user-specific data with global keys, and we never cache "to be safe." Every cache entry has a stated reason for existing.

The write path goes through a queue

Any request that must write to the database before responding is a request that will not scale past the database's write capacity. The writes in our API — events, audit records, counters, email triggers — go to a queue, and the worker drains the queue. The request responds fast, the write happens when the database has capacity, and a database blip degrades processing instead of failing requests.

The discipline this requires is idempotency. A queue that retries needs handlers that can process the same job twice without double-applying it, so every job carries an idempotency key and the worker deduplicates against it. That sounds like extra work until the first time a network hiccup redelivers a job and the answer is "fine, it was idempotent" instead of a data-corruption investigation.

Observability that answers questions

The million-request day is not a milestone you celebrate with dashboards; it is a milestone you defend with them. Ours are deliberately sparse. One page shows, per endpoint: requests, p99 latency, error rate, and slow-query count, all correlated to the currently deployed release. The deploy correlation is the part that matters most — when a graph goes sideways, the first question is what changed, and the page answers it instead of starting an investigation.

We log every request with its duration, status, and the query time it spent in the database. That last field is the one nobody logs and everyone wishes they had. It is the difference between "the endpoint is slow" and "the endpoint is slow because it spent 400 ms in one query," and the latter is actionable in seconds.

The failure stories we keep pinned

Three incidents live on the team's wall because they taught the whole playbook in one afternoon each. The first was the thundering herd: a scheduled job refreshed a shared cache key at the same instant, and the origin took ten thousand concurrent requests in a minute. The fix was jitter, and the incident taught us that load is shaped by our own cron jobs more than by users.

The second was connection pool exhaustion: a partner integration opened one connection per request and held it, and the pool filled in eleven minutes, taking the whole API down even though CPU and memory were at ten percent. The pooler fixed it, and the lesson is that concurrency, not load, is what kills a database-backed API.

The third was the missing index that everyone saw: a new endpoint filtered a table by a column that had no index, worked fine at a thousand rows, and got slow at a million. It took longer to find than to fix, and the lesson is that "it was fast in staging" is not a data point.

When you actually need to scale

The order of operations, in case you ever face the same question, is: right-size the instance, fix the N+1s, add the honest indexes, pool the database connections, cache the hot reads, move the writes to a queue, and only then add replicas. We did exactly that, in that order, and we never reached the last step for a million requests a day.

A million requests a day is a real milestone, but the engineering that meets it is the same boring engineering that meets a hundred thousand: statelessness, honest queries, a pool, a cache, a queue, and a p99 that you measure under load. The volume does not change the rules; it just punishes the teams that skipped them.