Featured

Zero-Downtime Deployments: The Architecture That Never Sleeps

by Kai Okafor Architecture 7 min read
Zero-Downtime Deployments: The Architecture That Never Sleeps

Most deployments are a controlled gamble. You push code, hold your breath, and hope the monitoring dashboard stays green. Elite teams don’t operate that way. They architect systems where deployments are so safe, so deterministic, that they become boring.

Zero-downtime deployments are not a feature you add later. They are a consequence of the architectural decisions you make at the foundation.

The Atomic Artifact Model

The first principle: every deployment must be an immutable, versioned artifact. Not a git pull. Not a rsync. An atomic object with a content-addressable hash.

When you deploy version v3.14.159, that exact byte-for-byte artifact — your compiled binary, your static assets, your configuration snapshot — gets pinned to a content hash. The deployment system then orchestrates a traffic cutover, not a file replacement.

$ vertex artifact build --tag=v3.14.159
 Building artifact...
 SHA256: a7f3c2d9e1b4...
 Artifact stored: registry.vertex.io/app@sha256:a7f3...

This means rollbacks are instantaneous. You’re not re-deploying old code — you’re re-pointing traffic to a previously validated artifact.

Traffic Shifting: The Dual-Pointer Model

The most common mistake is treating deployment as a binary event: old version off, new version on. The correct model is a spectrum.

Vertex uses a dual-pointer system. At any point in time, two artifact versions are resident in memory across the edge network: the current stable version and the incoming candidate. Traffic shifts between them using a weighted routing table at the network layer — not the application layer.

routing:
  stable: v3.14.158 @ 100%
  candidate: v3.14.159 @ 0%

deploy_strategy:
  type: canary
  steps:
    - weight: 5%   # 5% of traffic, 30s observation
    - weight: 25%  # 25% of traffic, 60s observation
    - weight: 100% # Full cutover
  rollback_trigger:
    error_rate_threshold: 0.1%
    p99_latency_increase: 20%

Each step is gated. If the candidate version’s error rate exceeds the threshold during any observation window, the system immediately re-weights to zero and pages on-call. The stable version never went anywhere — it was always resident.

Health Check Architecture

Standard health checks are a lie. Returning HTTP 200 from /health means nothing about whether your application can actually process requests correctly.

We use a synthetic transaction model. Before any traffic weight increase, the deployment system executes a real representative transaction against the candidate version in an isolated “shadow mode” — real infrastructure, zero customer impact.

// health-probe.ts
async function validateCandidate(version: string): Promise<HealthResult> {
  const probe = await vertexClient.runSyntheticTransaction({
    version,
    scenario: 'critical_path',
    assertions: [
      { metric: 'p99_latency', threshold: 50 },    // ms
      { metric: 'error_rate', threshold: 0.001 },   // 0.1%
      { metric: 'throughput', min: 10000 }           // req/s
    ]
  });

  return probe.result;
}

Only after synthetic transactions pass do we increment the traffic weight. This eliminates an entire class of silent failures where code “works” but performs catastrophically under real load patterns.

Database Migration Strategy

Migrations are the hardest part of zero-downtime deployments. The naive approach — run migration, then deploy — creates a window where old code runs against a new schema or vice versa.

The correct approach is the expand-contract pattern:

Phase 1 — Expand: Add new columns/tables. Old code ignores them. New code writes to both old and new. Both versions are simultaneously compatible.

Phase 2 — Migrate: Backfill data. Both versions remain operational.

Phase 3 — Contract: Remove old columns only after old code is entirely decommissioned.

This pattern makes the “migration” invisible to the deployment system. There is no risky migration window — only safe, incremental schema evolution.

The Result

With this architecture, our median deployment time is 2.5 seconds from git push to globally live production. Not 2.5 minutes. Not 2.5 hours.

Downtime is a design choice. Stop choosing it.