Edge Computing at Scale: How We Built a 847-Node Global Network

by Priya Subramaniam Infrastructure 9 min read
Edge Computing at Scale: How We Built a 847-Node Global Network

The phrase “edge computing” has been diluted to meaninglessness by marketing departments. Let me be precise about what we mean at Vertex, and why it matters for P99 latency at scale.

The edge, properly understood, is not a location. It is a property of request routing. A request is “at the edge” when it is processed by infrastructure that is geometrically and topologically closest to the originating client — without traversing unnecessary network hops to reach a centralized origin.

Why Centralized Infrastructure Fails at P99

Averages lie. When your CEO asks “what’s our latency?”, they want a number. When your users in São Paulo are waiting 400ms for a response while users in Virginia see 12ms, the average looks fine. The P99 tells the truth.

Centralized infrastructure structurally fails P99 because the speed of light is a hard constraint. A data center in Virginia serving users in Tokyo will always be subject to approximately 160ms of unavoidable propagation delay. No amount of optimization eliminates physics.

The only solution is proximity: move computation closer to where users are.

The Anycast Architecture

The foundation of Vertex’s edge network is IP anycast. Under anycast routing, a single IP address is simultaneously advertised from hundreds of locations via BGP. When a client resolves that IP and connects, the underlying network — not our application — routes the connection to the geometrically nearest announcing point of presence (PoP).

This means a user in Tokyo connects to our Tokyo PoP. A user in Frankfurt connects to our Frankfurt PoP. No DNS-based geolocation. No application-layer routing decisions. Pure network-layer intelligence.

# BGP announcement at each PoP
$ show route 203.0.113.0/24
# Tokyo PoP — 2ms to user
# Frankfurt PoP — 8ms to user
# São Paulo PoP — 4ms to user
# Sydney PoP — 3ms to user

The critical design insight: anycast failover is automatic. If a PoP goes offline, BGP withdraws the route advertisement within seconds and traffic seamlessly routes to the next-closest PoP. No DNS TTL waiting period. No health-check polling. Pure network convergence.

PoP Selection and Latency Scoring

We operate 847 PoPs. Not all of them are equivalent. Latency varies based on interconnect quality, peering relationships, and transit paths. A PoP 200km from a user via a poorly-peered path may perform worse than a PoP 500km away with direct fiber interconnects to every major carrier.

We solve this with a real-time latency scoring system. Every 15 seconds, each PoP probes a sample of recent origin IPs and publishes its observed round-trip latency. The routing table incorporates these scores as weights alongside raw BGP path cost.

def compute_pop_score(pop: PoP, client_region: str) -> float:
    bgp_cost = pop.bgp_path_cost_to(client_region)
    measured_latency = pop.observed_rtt_ms[client_region]
    interconnect_quality = pop.peering_score[client_region]

    # Weighted combination: measured latency dominates
    return (measured_latency * 0.6) + (bgp_cost * 0.25) + (1 - interconnect_quality) * 0.15

This means our routing adapts in real time to network conditions. A transatlantic cable cut doesn’t cause outages — it triggers a routing rebalance within seconds.

Cold Start Elimination

The traditional edge computing problem: you can push code to the edge, but cold starts create latency spikes that undermine the entire purpose. A function that takes 200ms to warm up on first invocation is worse than a centralized server with 150ms propagation delay.

Vertex eliminates cold starts through predictive pre-warming. We model traffic patterns with 24 hours of historical data and proactively initialize worker processes 90 seconds ahead of predicted demand spikes.

For truly sporadic traffic patterns, we maintain a minimum warm pool at every PoP. The cost is marginal compute overhead. The benefit is deterministic, sub-millisecond execution latency on every request, including the first.

The Numbers

After three years of iteration, our global network delivers:

The edge is not a marketing term. It is a measurable competitive advantage — and the gap between edge-native and origin-served infrastructure will only widen as applications become more latency-sensitive.