Multi-Region RPC Failover: Latency-Aware Routing and Health-Based Failover

Why single-endpoint RPC breaks under real production load

A single RPC endpoint is a single point of failure. When that endpoint's provider, region, or node degrades, every request in your application degrades with it — wallet reads stall, transaction submissions time out, and indexers fall behind. The goal of multi-region RPC failover is to make that failure degrade latency rather than break the application.

The mechanism is a client-side router that holds several independent endpoints and picks one per request based on measured health and round-trip time (RTT). This is the same pattern used by generic cloud and API architectures: AWS documents multi-region design patterns and Azure documents global load balancing and Traffic Manager, both of which describe health-based and latency-based routing as first-class concepts. For Web3 RPC specifically, the chainstack engineering reference 'Plasma: RPC failover with eRPC' describes a failover layer that sits in front of multiple RPC endpoints and routes by health and latency.

Before designing, decide which topology matches your requirement. A single endpoint is fine for prototypes. Active-passive with health-based failover is the minimum for production. Active-active across regions with latency-based routing is for latency-sensitive, high-throughput workloads. A client-side router that holds several independent endpoints and picks by measured health and RTT is the most flexible and is what this article builds.

If you are still choosing a provider, start with Choosing an RPC provider (RPC Assistant) and Public RPC endpoints vs dedicated for production. For a concrete chain, see Base and Base RPC latency: measuring and optimizing.

• Single endpoint: simplest, no redundancy, acceptable only for prototypes.
• Active-passive: one primary, one standby, health-based failover on failure.
• Active-active: multiple regions serving traffic, latency-based routing.
• Client-side router: several independent endpoints, picks by measured health and RTT.

Failover, load balancing, and hedging are different tools

Failover reacts to health: when the primary fails a health check, traffic moves to a standby. Load balancing distributes for capacity: requests are spread across endpoints to avoid saturating any one. Hedging races for tail latency: the same request is sent to two endpoints and the first response wins. Production systems usually combine all three because they solve different problems.

A common mistake is to treat load balancing as failover. Round-robin across endpoints does not help if all endpoints share the same provider or region — they fail together. Conversely, a strict active-passive failover does not help when the primary is healthy but slow; you need latency-aware routing or hedging to protect tail latency.

The practical combination is: health-based failover for availability, latency-based routing for performance, and optional hedging for the slowest percentile. The RPC node monitoring, metrics and failover article covers the monitoring side; this article focuses on the routing and failover mechanics.

• Failover: react to health, move traffic on failure.
• Load balancing: distribute for capacity, avoid saturation.
• Hedging: race for tail latency, first response wins.
• Combine all three: availability + performance + tail protection.

Health signal design: passive outcomes and active probes

Health signals come in two forms. Passive health observes real request outcomes: success, error, timeout, and latency. Active health sends probes on a schedule. Passive is cheap and reflects real traffic; active catches problems before user requests hit a bad endpoint. Use both.

A naive liveness probe is not enough. An endpoint can be 'up' — it answers eth_blockNumber or getHealth — but be stale, meaning its head is behind the chain tip. A node behind the chain tip passes a naive liveness check yet returns stale state, which is the most dangerous failure mode because it looks healthy. Always add a head-lag check: compare the endpoint's reported block number against a reference (another endpoint or a known-good source) and treat a lag beyond a threshold as unhealthy.

Concrete probe set: eth_blockNumber for liveness and head height, getHealth where supported, and a head-lag comparison against the pool's maximum observed height. For correctness-critical reads, also pin a commitment or finality level (for example, 'finalized' or a specific block tag) so that a slightly stale endpoint cannot return a different answer.

The RPC latency: causes, measurement and fixes article explains how to measure RTT correctly; reuse that method for the probe loop.

• Passive: observe real request success, error, timeout, latency.
• Active: scheduled probes for liveness and head height.
• Head-lag check: compare endpoint height to pool maximum.
• Pin commitment/finality for correctness-critical reads.

Verifying independence: same provider or region is not redundancy

Two endpoints at the same provider or in the same region share a blast radius. If that provider has an incident or that region has a network event, both endpoints fail together. They are not real redundancy. Independence means different providers, different regions, and ideally different underlying infrastructure.

A practical independence check: list each endpoint's provider, region, and ASN. If two endpoints share any of those, treat them as one failure domain for planning. Your failover pool should have at least two independent failure domains, and preferably three for latency-based routing.

This is also why a client-side router that holds several independent endpoints is more robust than a single managed load balancer: the router can enforce independence and measure health directly. See API service for how managed endpoints are typically exposed.

The health-based routing, circuit-breaker and ejection behaviour described here mirrors the failover router pattern documented by Chainstack in Plasma: RPC failover with eRPC; for the general multi-region topology tradeoffs see the AWS multi-region architectures whitepaper. Both are references for the concepts, not endorsements or OnFinality performance claims.

• Same provider or region = shared blast radius, not redundancy.
• Independence = different provider, region, and infrastructure.
• Plan for at least two independent failure domains, preferably three.
• A client-side router can enforce independence and measure health.

Latency-based routing mechanics and the geo-proxy caveat

Latency-based routing sends each request to the endpoint with the lowest measured RTT. The mechanism is a small probe loop that measures RTT per endpoint and a scoring function that combines RTT with health. This beats static geography because geography is a coarse proxy for the real RTT — network paths, peering, and congestion change the actual latency.

Azure Traffic Manager and AWS Global Accelerator both offer geography-based routing, but the documentation describes them as coarse routing methods. A small measured-probe loop beats static geo because it reflects the real path at the moment of the request. The chainstack 'Plasma: RPC failover with eRPC' reference describes a similar measured-health approach for RPC.

The caveat: measuring RTT from your client measures your client's path, not every user's path. If your users are globally distributed, measure from the edge or from a representative region. For most applications, a client-side router that measures from the application's own region is sufficient and much simpler.

• Latency-based routing = lowest measured RTT wins.
• Geography is a coarse proxy; measured RTT is the real signal.
• Measure from the application's region or edge, not a single laptop.
• Combine RTT with health in the scoring function.

Circuit breakers and ejection: thresholds, half-open, hysteresis

A circuit breaker ejects an endpoint after a threshold of consecutive errors, then periodically allows a half-open trial to test recovery. If the trial succeeds, the endpoint returns to the pool; if it fails, the breaker stays open. Recovery hysteresis (requiring several successes before full re-entry) prevents flapping.

A naively aggressive breaker can flap: it ejects on a single transient error, then re-admits immediately, causing traffic to oscillate. Use a consecutive-error threshold (for example, 3–5), a cooldown period, and a half-open trial with a success requirement. The exact numbers are reader-measured; tune them against your own error profile.

Ejection should also consider head lag: an endpoint that is up but stale should be ejected even if it returns HTTP 200. Treat head lag as a first-class health signal in the breaker.

• Consecutive-error threshold (e.g., 3–5) before ejection.
• Cooldown period before half-open trial.
• Half-open trial with success requirement before full re-entry.
• Treat head lag as a first-class health signal.

State consistency traps around the chain head

Different endpoints, even of the same chain, can disagree around the head. One endpoint may be one or two blocks ahead of another, so a read at 'latest' can return different state depending on which endpoint answers. This is not a bug; it is the nature of a distributed chain head.

The fix is to pin a commitment or finality level for correctness-critical reads. For example, use 'finalized' or a specific block tag, and do not mix levels across the pool. If you mix 'latest' and 'finalized' across endpoints, you can get inconsistent answers for the same logical query.

For non-critical reads (dashboards, estimates), 'latest' is fine, but be aware that a slightly stale endpoint may return a slightly older head. The head-lag check in the health probe is what keeps that staleness bounded.

• Endpoints can disagree around the head; this is normal.
• Pin commitment/finality for correctness-critical reads.
• Do not mix levels across the pool.
• Head-lag check bounds staleness for non-critical reads.

Cutover and traffic-split: introducing or replacing an endpoint safely

Introducing a new endpoint directly into the pool is risky. The safe pattern is a shadow/traffic-split cutover: dual-write to the shadow endpoint, compare responses against the current primary, then shift a percentage of traffic. Start at a small percentage, watch error rate and latency, and increase gradually.

The comparison step is important: for the same request, do the shadow and primary return the same result at the same commitment level? If not, investigate before shifting traffic. This catches configuration differences, chain mismatches, and stale nodes.

Once the shadow endpoint passes comparison and a small traffic percentage is healthy, promote it to full traffic and demote the old endpoint to standby. This is the same canary pattern used in generic API deployments, applied to RPC endpoints.

• Dual-write to shadow endpoint, compare responses.
• Shift a small percentage of traffic, watch error rate and latency.
• Promote to full traffic only after comparison passes.
• Demote the old endpoint to standby, do not delete it immediately.

Runnable TypeScript router: health probes, scoring, circuit breaker, dispatch

The following Node.js/TypeScript example implements a minimal multi-region RPC router. It runs a health-probe loop that measures RTT and head lag per endpoint, scores endpoints, maintains circuit-breaker state, and dispatches requests to the healthiest endpoint while ejecting failing ones. It uses only built-in fetch and no external dependencies.

Run it with Node 18+ (which has global fetch). Replace the endpoint URLs with your own independent endpoints. The expected output shows the probe results and the chosen endpoint for a sample request.

// rpc-router.ts — run with: npx tsx rpc-router.ts (Node 18+)
type Endpoint = {
  url: string;
  name: string;
  rttMs: number;
  head: number;
  healthy: boolean;
  consecutiveErrors: number;
  breakerOpen: boolean;
  lastProbe: number;
};

const endpoints: Endpoint[] = [
  { url: 'https://rpc-a.example.com', name: 'region-a', rttMs: 0, head: 0, healthy: true, consecutiveErrors: 0, breakerOpen: false, lastProbe: 0 },
  { url: 'https://rpc-b.example.com', name: 'region-b', rttMs: 0, head: 0, healthy: true, consecutiveErrors: 0, breakerOpen: false, lastProbe: 0 },
  { url: 'https://rpc-c.example.com', name: 'region-c', rttMs: 0, head: 0, healthy: true, consecutiveErrors: 0, breakerOpen: false, lastProbe: 0 },
];

const ERROR_THRESHOLD = 3;
const HEAD_LAG_THRESHOLD = 5;
const PROBE_INTERVAL_MS = 5000;

async function rpc(url: string, method: string, params: any[] = []): Promise<any> {
  const start = Date.now();
  const res = await fetch(url, {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ jsonrpc: '2.0', id: 1, method, params }),
  });
  const rtt = Date.now() - start;
  if (!res.ok) throw new Error(`HTTP ${res.status}`);
  const json = await res.json();
  if (json.error) throw new Error(json.error.message);
  return { result: json.result, rtt };
}

async function probe(e: Endpoint): Promise<void> {
  try {
    const { result, rtt } = await rpc(e.url, 'eth_blockNumber');
    e.rttMs = rtt;
    e.head = parseInt(result, 16);
    e.healthy = true;
    e.consecutiveErrors = 0;
    e.breakerOpen = false;
  } catch (err) {
    e.consecutiveErrors += 1;
    e.healthy = false;
    if (e.consecutiveErrors >= ERROR_THRESHOLD) e.breakerOpen = true;
  }
  e.lastProbe = Date.now();
}

function score(e: Endpoint, maxHead: number): number {
  if (e.breakerOpen || !e.healthy) return -1;
  const lag = maxHead - e.head;
  if (lag > HEAD_LAG_THRESHOLD) return -1;
  return 1000 - e.rttMs - lag * 10;
}

function pick(): Endpoint | null {
  const maxHead = Math.max(...endpoints.map((e) => e.head));
  const ranked = endpoints
    .map((e) => ({ e, s: score(e, maxHead) }))
    .filter((x) => x.s >= 0)
    .sort((a, b) => b.s - a.s);
  return ranked.length ? ranked[0].e : null;
}

async function dispatch(method: string, params: any[] = []): Promise<any> {
  const maxHead = Math.max(...endpoints.map((e) => e.head));
  const ranked = endpoints
    .map((e) => ({ e, s: score(e, maxHead) }))
    .filter((x) => x.s >= 0)
    .sort((a, b) => b.s - a.s);
  for (const { e } of ranked) {
    try {
      const { result } = await rpc(e.url, method, params);
      return { endpoint: e.name, result };
    } catch {
      e.consecutiveErrors += 1;
      if (e.consecutiveErrors >= ERROR_THRESHOLD) e.breakerOpen = true;
    }
  }
  throw new Error('all endpoints failed');
}

async function main() {
  setInterval(() => endpoints.forEach(probe), PROBE_INTERVAL_MS);
  await Promise.all(endpoints.map(probe));
  console.log('probe results:', endpoints.map((e) => ({ name: e.name, rttMs: e.rttMs, head: e.head, healthy: e.healthy, breakerOpen: e.breakerOpen })));
  const chosen = pick();
  console.log('chosen endpoint:', chosen?.name ?? 'none');
  const out = await dispatch('eth_blockNumber');
  console.log('dispatch result:', out);
}

main().catch(console.error);

Results table: measure your own endpoints

Use the table below to record your own measurements. Do not rely on vendor-published numbers; measure from your application's region. Run the probe loop for at least 10 minutes and record the median and p95 RTT, head lag, and error rate per endpoint.

The 'documented / varies by provider' note applies to any provider-published latency or uptime figure. Your measured values are the ones that matter for routing decisions.

• Endpoint name | Provider | Region | Median RTT (ms) | p95 RTT (ms) | Head lag (blocks) | Error rate (%) | Notes
• region-a | provider-1 | us-east | (measured) | (measured) | (measured) | (measured) | primary
• region-b | provider-2 | eu-west | (measured) | (measured) | (measured) | (measured) | standby
• region-c | provider-3 | ap-south | (measured) | (measured) | (measured) | (measured) | standby

Verification plan: inject a failure and assert traffic moves

Verification is not optional. Inject a failing endpoint and assert that traffic moves to a healthy one, then assert that it recovers when the endpoint returns. Use a controlled failure: point one endpoint URL at a non-routable address or return HTTP 500 from a local proxy.

Steps: (1) run the router with three endpoints; (2) confirm the chosen endpoint is the lowest-RTT healthy one; (3) break the chosen endpoint; (4) confirm the router ejects it after the consecutive-error threshold and dispatches to the next healthy endpoint; (5) restore the endpoint; (6) confirm the half-open trial re-admits it after the cooldown.

Record the observed behavior in the results table. If the router does not move traffic, check the health probe interval, the error threshold, and whether the breaker is stuck open.

• Inject failure: non-routable URL or HTTP 500 proxy.
• Assert traffic moves after the consecutive-error threshold.
• Assert recovery after cooldown and half-open trial.
• Record observed behavior in the results table.

Common failures and troubleshooting

The most common failure is false redundancy: two endpoints at the same provider or region. Check provider, region, and ASN for each endpoint. The second most common is a naive liveness check that misses head lag; add the head-lag comparison.

Other failures: a circuit breaker that flaps because the threshold is too low; mixing commitment levels across the pool, causing inconsistent reads; and a probe loop that measures from the wrong region. For each, the fix is in the health signal design and the scoring function.

If you see intermittent stale reads, check whether the endpoint is behind the chain tip and whether your reads pin a finality level. If you see traffic oscillating between endpoints, increase the consecutive-error threshold and add recovery hysteresis.

• False redundancy: same provider/region/ASN.
• Naive liveness: misses head lag, returns stale state.
• Breaker flapping: threshold too low, no hysteresis.
• Mixed commitment levels: inconsistent reads.
• Probe from wrong region: misleading RTT.

Limitations, tradeoffs, and when to update this design

A client-side router adds complexity: you must maintain the probe loop, the scoring function, and the breaker state. It also adds a small amount of latency for the probe traffic. For very small applications, a managed load balancer or a single provider with a standby may be simpler.

Latency-based routing optimizes for the median but can hurt the tail if the lowest-RTT endpoint is also the most loaded. Hedging can help the tail but doubles request volume. There is no free lunch; measure and tune.

Update this design when your endpoint set changes, when your provider's regions change, or when your traffic pattern shifts. Re-run the verification plan after any change. For pricing and plan implications, see RPC pricing.

• Client-side router adds complexity and probe traffic.
• Latency routing optimizes median, not always tail.
• Hedging helps tail but doubles request volume.
• Re-run verification after any endpoint or provider change.

Next steps: from router to production rollout

Start by measuring your current endpoints with the results table. Then implement the router in a staging environment, run the verification plan, and only then roll it out to production. Keep the old endpoint as standby during the cutover.

For broader context, revisit RPC node monitoring, metrics and failover and Public RPC endpoints vs dedicated for production. For chain-specific latency work, see Base RPC latency: measuring and optimizing and Base.

If you are evaluating providers, use Choosing an RPC provider (RPC Assistant) and RPC pricing. The OnFinality Learn hub collects the full set of performance and optimization guides.

• Measure current endpoints with the results table.
• Implement and verify the router in staging first.
• Keep the old endpoint as standby during cutover.
• Revisit monitoring and provider-selection guides for context.