RPC Request Hedging: Cut Tail Latency Across Endpoints

Why Tail Latency Dominates User-Visible Slowness in RPC Fan-Out

A single page load or transaction simulation often triggers dozens of JSON-RPC reads: balance checks, nonce lookups, contract calls, and block queries. If each call has a median latency of 50 ms but a p99 of 800 ms, the median page load looks fine, yet roughly one in a hundred calls stalls for nearly a second. When you fan out to 20 calls, the probability that at least one hits the tail rises sharply, so the user-visible experience is governed by the slowest call, not the average.

This is the classic tail-at-scale problem described in Google's seminal work on tail latency and in the gRPC 'Request Hedging' design documentation. The median is not the metric that users feel; the p99 is. Reducing the median from 50 ms to 40 ms is barely noticeable, but reducing the p99 from 800 ms to 200 ms transforms perceived responsiveness.

For blockchain RPC specifically, tail latency comes from many sources: garbage collection pauses, mempool spikes, disk I/O on archive nodes, network jitter, and provider-side rate limiting. These are transient and independent across providers, which is exactly the condition that makes hedging effective. For a broader treatment of latency causes, see RPC latency: causes, measurement and fixes.

• Median latency hides the slowest 1% of calls that users actually notice.
• Fan-out multiplies tail probability: 20 calls at p99 each yield a much higher chance of one slow response.
• Independent transient causes across providers are the precondition for hedging to help.

Hedging vs Retries vs Failover: A Decision Table

These three techniques are often conflated, but they solve different problems and compose differently. Retries are sequential: you wait for a failure or timeout, then try again, which adds latency equal to the timeout. Failover is routing: you send to a healthy endpoint based on health checks, but you still wait for that endpoint's response. Hedging is parallel: you send duplicates before you know whether the first will be slow, and you take the first success.

The table below summarizes when each applies. Use hedging only for idempotent reads where duplicate execution is harmless. Use retries for transient errors on any method, but beware of non-idempotent writes. Use failover for endpoint health and regional routing. In practice, production clients combine all three: failover picks the primary, hedging races a secondary for reads, and retries handle explicit errors.

• Hedging: parallel duplicate after a delay; reduces p99; doubles request weight; read-only.
• Retries: sequential after failure; recovers from errors; adds timeout latency; any method with idempotency key.
• Failover: routing to healthy endpoint; handles outages; does not reduce tail from a slow-but-alive endpoint.

The Hedging Mechanism: Timer, Duplicate, First-Success, Cancel

The canonical hedged request, as described in the gRPC 'Request Hedging' documentation, works as follows. The client sends the request to endpoint A immediately. It arms a timer set to a delay derived from the observed p95 latency of that method. If the response arrives before the timer fires, the client returns it and no duplicate is sent. If the timer fires first, the client sends the same request to endpoint B. The first successful response wins; the other in-flight request is cancelled via an abort signal.

The delay is critical. If it is too low, both requests fire almost simultaneously and you pay double cost on nearly every call. If it is too high, the hedge rarely fires and the p99 remains high. A common starting point is the p95 of the method's latency distribution, measured over a rolling window. You can also cap the number of outstanding hedges per method to bound cost.

Cancellation must be cooperative. In Node.js, AbortController propagates an abort signal to fetch, which closes the underlying socket. For JSON-RPC over HTTP, this is safe for reads because the server may still process the request, but the client stops waiting. Never hedge a write method like eth_sendRawTransaction, because both duplicates could be accepted and cause a double submission.

The delay-before-duplicate semantics used here follow the client-side hedging model documented in the gRPC request hedging guide, which is a useful primary reference for the timer-and-cancel design even when your transport is plain HTTP JSON-RPC rather than gRPC.

• Send to A immediately; arm a timer at ~p95 delay.
• On timer fire, send duplicate to B; resolve on first success.
• Cancel the loser with AbortController; cap outstanding hedges.
• Restrict to idempotent reads: eth_call, eth_getBalance, getLatestBlockhash, getAccountInfo.

Request-Budget Consequences: Why Hedging Doubles Method Weight

Every hedged request consumes additional request weight on your provider plan. If you hedge 10% of calls, your effective request volume rises by 10%; if you hedge aggressively with a low timer, it can approach 100% overhead. This matters because most RPC providers meter by compute units or request count, and rate limits are enforced per method. See RPC pricing for how weight is typically calculated.

The budget impact is method-specific. A heavy eth_call with a large gas limit may cost many more units than a lightweight eth_blockNumber. Hedging the heavy call doubles a large cost; hedging the light call doubles a small one. Prioritize hedging for methods that are both frequent and tail-prone, and avoid hedging methods that are already fast or cheap.

Also consider the 429 response. If a provider returns 429 Too Many Requests, that is a signal to back off, not to hedge. Hedging a 429 duplicates the rate-limit violation and can worsen throttling. Treat 429 as a non-hedgeable error and route it to failover or backoff logic instead. For batching strategies that reduce call count, see JSON-RPC batch requests and best practices.

• Hedging adds request volume proportional to hedge rate; budget accordingly.
• Heavy methods cost more per duplicate; hedge selectively.
• Never hedge a 429; back off or failover instead.

Provider-Selection Requirements for Effective Hedging

Hedging only works if the endpoints are genuinely independent. If both endpoints share the same provider, region, or upstream infrastructure, a single outage or congestion event will slow both, and the hedge provides no benefit. The gRPC documentation emphasizes that hedging assumes independent failure domains. For blockchain RPC, this means using different providers or at least different regions and node types.

When selecting endpoints, verify that they are not behind the same load balancer or CDN edge. A quick check is to compare the server IP or TLS certificate issuer across endpoints. If they match, the endpoints may share fate. For guidance on evaluating providers, see Choosing an RPC provider (RPC Assistant).

OnFinality's API service provides multi-region endpoints, but you should still confirm independence if you mix providers. The goal is that a transient slowdown on one endpoint does not affect the other. If you are building on Ethereum, see Ethereum network endpoints for available options.

• Use different providers or regions to ensure independent failure domains.
• Check server IP and TLS issuer to detect shared infrastructure.
• Avoid hedging against a single provider's redundant endpoints if they share fate.

Runnable TypeScript Example: Hedge-to-N with AbortController

The following Node.js example implements a hedged JSON-RPC client. It sends the request to the first endpoint, arms a timer at a configurable p95 delay, then sends duplicates to additional endpoints. The first successful response resolves the promise; all other in-flight requests are aborted. It also caps the number of outstanding hedges per call.

The code uses the native fetch API and AbortController, available in Node.js 18+. It assumes read-only methods and does not retry on 429. You can adapt the endpoint list and delay to your environment. The example logs latency per attempt so you can derive the p95 timer empirically.

import { performance } from 'node:perf_hooks';

interface HedgeOptions {
  endpoints: string[];
  method: string;
  params: any[];
  p95DelayMs: number;
  maxHedges?: number;
}

async function hedgedRpc({ endpoints, method, params, p95DelayMs, maxHedges = 2 }: HedgeOptions): Promise<any> {
  const controllers: AbortController[] = [];
  const attempts: Promise<any>[] = [];
  let settled = false;

  const send = (url: string, index: number) => {
    const controller = new AbortController();
    controllers.push(controller);
    const start = performance.now();
    return fetch(url, {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({ jsonrpc: '2.0', id: index, method, params }),
      signal: controller.signal,
    })
      .then(async (res) => {
        if (res.status === 429) throw new Error('rate-limited');
        const json = await res.json();
        if (json.error) throw new Error(json.error.message);
        const elapsed = performance.now() - start;
        console.log(`endpoint ${index} succeeded in ${elapsed.toFixed(1)} ms`);
        return json.result;
      })
      .catch((err) => {
        if (err.name === 'AbortError') return new Promise(() => {});
        throw err;
      });
  };

  // Send to first endpoint immediately
  attempts.push(send(endpoints[0], 0));

  // Arm timer for hedge
  const timer = setTimeout(() => {
    if (settled) return;
    const remaining = Math.min(maxHedges, endpoints.length - 1);
    for (let i = 1; i <= remaining; i++) {
      attempts.push(send(endpoints[i], i));
    }
  }, p95DelayMs);

  try {
    const result = await Promise.race(attempts);
    settled = true;
    clearTimeout(timer);
    controllers.forEach((c) => c.abort());
    return result;
  } catch (err) {
    clearTimeout(timer);
    throw err;
  }
}

// Example usage
const endpoints = [
  'https://eth-mainnet.example-provider-a.com',
  'https://eth-mainnet.example-provider-b.com',
];

hedgedRpc({
  endpoints,
  method: 'eth_getBalance',
  params: ['0x0000000000000000000000000000000000000000', 'latest'],
  p95DelayMs: 150,
}).then(console.log).catch(console.error);

Deriving the p95 Timer from Per-Request Latency

The hedge delay should be based on your own measured latency, not a guessed constant. Instrument every RPC call with a start and end timestamp, record the method name, endpoint, and outcome, and compute percentiles over a rolling window. The p95 of successful calls is a reasonable initial timer: it means the hedge fires only when the primary is slower than 95% of normal calls.

You can compute percentiles in-process or export metrics to a monitoring system. For a deeper discussion of metrics and failover, see RPC node monitoring, metrics and failover. Start with a conservative timer (e.g., p95) and lower it gradually while watching request volume and cost.

The table below is a template for your own measurements. Fill it with data from your endpoints; do not rely on generic numbers. The goal is to see the p99 drop after enabling hedging, while confirming that the median and request volume change as expected.

• Record method, endpoint, start time, end time, and success/failure for every call.
• Compute p50, p95, and p99 over a rolling window (e.g., 5 minutes).
• Set the hedge timer to the p95 of successful calls, then tune down cautiously.
• Monitor request volume and 429 rate after enabling hedging.

Results Table: Measuring the p99 Drop Against Your Own Endpoints

Use the following table to record your before-and-after measurements. Run a controlled load test that issues the same read method repeatedly (e.g., 10,000 calls) against your endpoint set, first without hedging and then with hedging enabled. Capture p50, p95, p99, and the total request count. The expected outcome is that p99 falls significantly while p50 stays roughly the same and request count rises.

Do not treat any specific latency figure as universal. Latency varies by provider, region, method, and time of day. The value of this exercise is the relative change you observe in your own environment. If p99 does not improve, check that your endpoints are truly independent and that the hedge timer is not too high.

For a broader overview of latency measurement, see RPC latency: causes, measurement and fixes. For timeout-specific issues, see RPC timeout errors: causes and fixes.

• Metric | Without hedging | With hedging | Notes
• p50 latency | fill | fill | should be similar
• p95 latency | fill | fill | may improve slightly
• p99 latency | fill | fill | primary target
• Total requests | fill | fill | expect increase
• 429 rate | fill | fill | monitor for throttling

Common Failures and Troubleshooting

The most dangerous failure is hedging a non-idempotent write. If you hedge eth_sendRawTransaction, both endpoints may accept the transaction, resulting in a double submission or nonce conflict. Always restrict hedging to read-only methods. If you need reliability for writes, use retries with idempotency keys or a transaction manager.

Another common mistake is hedging against a single provider's endpoints that share infrastructure. If both endpoints go through the same load balancer, a single congestion event slows both, and the hedge is useless. Verify independence by checking server IPs and TLS issuers.

A timer set too low causes both requests to fire on nearly every call, doubling cost without meaningful tail reduction. A timer set too high means the hedge rarely fires. Monitor the hedge fire rate and adjust. Finally, never hedge a 429 response; that is a signal to back off, not to duplicate. If you see 429s, reduce your request rate or switch to a provider with higher limits.

• Never hedge eth_sendRawTransaction or any write method.
• Verify endpoint independence; avoid shared infrastructure.
• Tune the timer: too low doubles cost, too high misses the tail.
• Do not hedge 429; back off or failover instead.
• Cancelling a request that already committed is harmless for reads but dangerous for writes.

Limitations and Tradeoffs of Request Hedging

Hedging cannot fix a systematically wrong or lagging node. If one endpoint consistently returns stale data or incorrect results, racing it against a correct endpoint may still return the wrong answer if the stale response arrives first. Hedging reduces latency variance, not correctness errors. For correctness, use block-height checks or quorum reads.

Hedging trades extra request volume for tail reduction. If your workload is already within budget and your p99 is acceptable, hedging adds cost without benefit. It is also pointless when one endpoint is already fast and reliable; the hedge will rarely fire, and the added complexity may not be justified.

Finally, hedging is not a substitute for proper capacity planning. If your provider is rate-limiting you, hedging will worsen the problem. Use hedging as one tool among many, alongside failover, batching, and caching. For a holistic view, see the OnFinality Learn hub.

• Does not fix stale or incorrect data; use quorum or height checks.
• Adds request volume; only worthwhile if p99 matters and budget allows.
• Useless if one endpoint is already fast; adds complexity.
• Not a fix for rate limiting; can worsen 429s.

Next Steps: Integrating Hedging into Your RPC Stack

Start by instrumenting your RPC calls to measure p50, p95, and p99 per method. Identify the read methods that contribute most to user-visible tail latency. Then implement hedging for those methods using the TypeScript example above, with a conservative timer and a cap on outstanding hedges.

Run a controlled load test and fill in the results table. Compare p99 before and after, and monitor request volume and 429 rate. If p99 improves without excessive cost, roll out gradually. If not, revisit endpoint independence and timer tuning.

For production, consider combining hedging with failover and batching. Review RPC pricing to understand cost implications, and explore API service for multi-region endpoints. For provider selection, see Choosing an RPC provider (RPC Assistant).

• Instrument first; hedge only the tail-prone read methods.
• Use a conservative p95 timer and cap hedges.
• Measure p99 improvement and cost impact before full rollout.
• Combine with failover and batching for a complete strategy.