How we built our global deployment pipeline

Our deployment story used to be a shell script that SSHed into a server and ran git pull. It worked, in the sense that a car with no brakes works — until the hill. When we started deploying to multiple regions, the script stopped being a liability and became a wall. This is the pipeline we built to replace it, what each piece is for, and the mistakes we would not repeat.

The problem we started with

We had three regions, two teams shipping daily, and a deployment process that required a human to be awake, logged in, and confident. Every deploy was a separate event in each region, so a release was not one thing but three slightly different things, and "rollback" meant re-running the old deploy in all three regions and hoping nothing drifted.

The symptoms were the usual ones. Staging and production drifted because they were built at different times. Regional differences appeared out of nowhere. Rollbacks took longer than the deploys they reversed. Change failure rate was a running joke in standups. We stopped patching the script and started over.

One artifact, many regions

The first decision set the tone for everything else: a commit produces exactly one artifact, and every environment and region runs that exact artifact. No per-region builds, no environment-specific tags that could be rebuilt differently, no "production build" that differs from the staging build by a flag.

This is the rule that makes multi-region deployment boring, and boring is the goal. The image is referenced by digest, which makes it content-addressable and immutable. When you promote to production, you are not producing anything — you are relabelling something that already exists, in the registry, in the region, on disk.

The control flow

The pipeline is a sequence of stages, each gated by the one before it. Nothing skips a gate, and everything is retried with backoff because the failure modes are boring and therefore common:

1. 01 Webhook receives the push event and records an idempotency key, so a redelivery cannot double-deploy.
2. 02 CI runs tests, lint, and the build, and fails the stage on anything non-zero. A failed build is a failed deploy; we do not allow a broken commit to promote.
3. 03 The image is pushed to the registry with its digest, and the digest is written into the deployment record as the source of truth.
4. 04 Every region pre-pulls the digest in the background, so the rollout never waits for bytes to cross the network.
5. 05 The orchestrator starts a canary in each region, runs health checks and a smoke test, and only then expands to the full rollout.
6. 06 Traffic is cut over region by region, and any region that fails its post-deploy check is rolled back independently while the others continue.

The important property is that each region is an independent transaction. A bad release does not take down the world; it takes down one region's canary, which rolls back on its own, and the pipeline records exactly which regions are on which digest at any moment.

The build stage in detail

The build is where most pipelines lose their reproducibility, so we spent the most time there. Dependencies are resolved from a lockfile, the build runs on an isolated worker with a clean cache per commit, and the output is written to a fresh layer set that is never mutated afterwards.

Cache strategy took a few tries to get right. A shared cache directory across builds made everything fast until a stale cache produced a build that worked locally and broke in production. We settled on two levels: a remote cache keyed by dependency manifest hashes for fast restores, and a clean-room build on any path that touches code that could affect the runtime. The rule is that a full build always succeeds, so caching is an optimisation and never a correctness input.

Every build records its inputs — commit, manifest hashes, toolchain versions — alongside the image. When a weird runtime behaviour shows up months later, we can reproduce the exact environment instead of hoping the logs are enough. Reproducibility is not a virtue we pursued for its own sake; it is the thing that makes the rollback story honest.

Secrets and configuration per region

One artifact does not mean one configuration. Each region needs its own database URL, its own feature flags, its own provider credentials, and the pipeline has to make that true without turning the artifact into a template. We keep configuration and secrets entirely outside the image: they are injected at deploy time from a store each region can read, and they are versioned alongside the release so a rollback restores the configuration of the previous release too.

The worst failure we ever had in this area was a release that was fine, but was deployed with a stale configuration file, because config had drifted out of the pipeline and into a server that had been hand-edited years ago. The pipeline now treats configuration as part of the release, with the same digest-style versioning, and a deploy is not complete until the declared configuration is verified in the region. Hand-editing a server is how you lose the guarantee the whole system is built on, so the pipeline refuses to deploy anywhere it cannot enforce the declared state.

Signing and supply chain

A pipeline that auto-promotes to production needs to be able to trust its own artifacts, or it is a very fast way to ship an attacker's code. Every image is signed at build time by a key that lives in the CI control plane, and the orchestrator verifies the signature before it will start a canary. An unsigned digest — anything that arrived in the registry by a path other than the pipeline — is refused.

This closed the loop on the artifact-first model. The digest is immutable, the signature proves it came from the build pipeline and nowhere else, and the deployment record ties the signed digest to the commit and the environment. The supply chain property fell out of the architecture instead of being bolted on, because once every promotion is just a relabel of an existing, signed artifact, there is no path for untrusted bytes to get a production label.

Rollouts that can stop themselves

The canary step is where the pipeline earns its keep. We run the new digest on a small slice of traffic in a region, with error-rate and p99 latency gates, before trusting it with the rest. The gates are automatic and brutal: if the error rate crosses 0.5% for two consecutive windows, the rollout pauses and the previous digest resumes.

Automation is the point. A human is fine for a judgement call, but a human cannot watch five regions and three metrics simultaneously at 3 a.m. The pipeline can, and it is not proud. It will roll back a release it was excited about five minutes ago.

Pausing is not the same as aborting. A paused rollout keeps the old digest serving and the new digest ready, and a human can either promote it after reviewing the metrics or kill it. The system handles the emergency; the human handles the decision.

Rollback is a first-class operation

Because every release is a digest reference, rollback is a deployment of the previous digest — the same pipeline, the same gates, the same health checks, just a different reference. We removed the word "restore" from the vocabulary. There is no snapshot to reconstruct; the previous artifact still exists, still signed, still on disk in every region.

This changes the psychology of deploying more than any technical detail. When rollback is cheaper than debugging, teams stop treating the pipeline as something to be feared. Deploys get smaller, more frequent, and easier to reason about, which is the virtuous cycle the whole thing was built for.

Observability was not an afterthought

A deployment pipeline without deployment telemetry is a guess with a confirmation button. Every release writes the digest, the region, the gates, and the timing into the same system that stores application logs, so the question "what changed at 14:03?" has a single answer.

We correlate logs and metrics to the deploy that shipped them. When a graph goes sideways, the first query is "which release touched this service, and what did its canary gates say?" — not "can we find the release notes somewhere?" This single change halved our mean time to recovery, because it halved the time spent figuring out what was live.

The operators' view is deliberately boring: a list of releases, each with its digest, regions, gate results, and timings, and one obvious button for the last-known-good release. We resisted adding clever visualisations because the goal was to make the state of the system answerable in one glance.

What we would do differently

Given another pass, three things would change. First, we would introduce the artifact-first model from day one, because retrofitting it onto an environment-tag workflow is weeks of painful migration. Second, we would gate on post-deploy business metrics, not just technical ones — order latency and checkout success, not only HTTP 500s. Third, we would treat the pipeline itself as a product with an owner, because every team repointed it, hacked around it, and wrote shadow tooling until one owner gave it a consistent shape.

We would also spend less time on the first version of the rollout UI. The dashboard did not make anyone a better operator; the automatic gates and the instant rollback did. The interface just had to stop lying about what was live.

The pipeline is not clever. It is a list of stages, an immutable artifact, automatic gates, and independent regional transactions, wired together so that the failure cases are handled by software instead of by people. That is the whole trick, and it is enough. Boring deployments, everywhere, every time.