A multi-provider RPC pool must distinguish four failure classes: transport, protocol, semantic (stale head), and rate-limit. Naive try/catch only catches transport failures, leaving stale reads and silent errors unhandled. The core safety rule is a freshness gate: every response is stamped with the endpoint's observed head, and the pool refuses to satisfy a 'latest' read from an endpoint below the pool's high-water mark. Circuit breakers with blockchain-specific ejection thresholds and half-open probes using real cheap reads prevent repeated failures. This article provides a runnable Node.js dispatch layer, a results table for self-measurement, and troubleshooting for split heads, cached responses, and thundering herds.
Failure Classification for RPC Pools
A multi-provider RPC pool must classify failures into four distinct classes, because each requires a different response. Transport failures (connection refused, TLS handshake failure, DNS resolution failure) are the easiest to detect and are the only ones a naive try/catch handles. Protocol failures occur when the endpoint returns a valid JSON-RPC 2.0 response containing an error object with a code, as defined in the JSON-RPC 2.0 Specification. Semantic failures are the most dangerous: the endpoint returns HTTP 200 with a valid result, but the result is stale—for example, eth_blockNumber returns a block height lower than the pool's current high-water mark. Rate-limit failures return HTTP 429, sometimes with a Retry-After header, and must be handled with a wait rather than immediate retry.
The Ethereum JSON-RPC specification defines eth_blockNumber and block tags like latest, safe, and finalized. These are the authoritative signals for freshness. A pool that treats any HTTP 200 as success will serve stale reads during partial failures, exactly when correctness matters most. The multi-endpoint consistency and head lag page covers the read discipline once a pool exists; this article defines the pool contract itself.
- Transport failure: connect, TLS, or DNS error—no HTTP response.
- Protocol failure: JSON-RPC error object with a code (e.g., -32603 internal error).
- Semantic failure: HTTP 200 with a result that is stale or inconsistent with the pool's head.
- Rate-limit failure: HTTP 429, optionally with Retry-After; requires backoff, not immediate failover.
Dispatch Strategies: Weighted, Round-Robin, and Least-Outstanding
Round-robin dispatch is actively harmful for read-after-write workloads on a chain with per-node head lag. If a write is sent to provider A and the subsequent read is dispatched to provider B, which is a few blocks behind, the read may not see the write. Weighted dispatch can help if weights reflect observed head freshness, but static weights become stale. Least-outstanding dispatch—sending to the endpoint with the fewest in-flight requests—balances load but ignores head height. The safest default for read-after-write is to pin reads to the same endpoint that served the write until the write is confirmed, then allow failover only to endpoints at or above the pool's high-water mark.
For read-only workloads, least-outstanding combined with a freshness gate works well. The multi-region RPC failover and latency-aware routing page covers region and RTT scoring; this article focuses on the head-height gate that makes any dispatch strategy safe.
- Round-robin: simple but unsafe for read-after-write when nodes lag.
- Weighted: can incorporate head freshness but requires dynamic weight updates.
- Least-outstanding: good for read-only load, but must be combined with a freshness gate.
- Pinning: for read-after-write, pin to the write endpoint until confirmed.
The Freshness Gate: Head High-Water Mark
The freshness gate is the one rule that makes failover safe: every dispatched response is stamped with the endpoint's observed head (via eth_blockNumber), and the pool refuses to satisfy a 'latest' read from an endpoint whose head is below the pool's high-water mark. The high-water mark is the maximum head observed across all healthy endpoints, updated on each successful head sample. When a read request arrives, the dispatcher selects an endpoint whose last observed head is at or above the high-water mark. If no such endpoint exists, the request waits or fails with a clear error rather than serving stale data.
This gate prevents the classic stale-read failure: a provider that is syncing or lagging answers requests but returns old data. The multi-endpoint consistency and head lag page details monotonicity and quorum reads; the gate here is the pool-level enforcement.
// Freshness gate: only dispatch to endpoints at or above high-water mark
function selectEndpoint(pool, request) {
const hwm = pool.highWaterMark;
const eligible = pool.endpoints.filter(ep =>
ep.state === 'closed' &&
ep.lastHead >= hwm &&
ep.inFlight < ep.maxInFlight
);
if (eligible.length === 0) {
throw new Error('No fresh endpoint available; high-water mark=' + hwm);
}
// Least-outstanding among eligible
return eligible.reduce((a, b) => a.inFlight <= b.inFlight ? a : b);
}Circuit-Breaker States with Blockchain-Specific Ejection
The circuit-breaker state machine—closed, open, half-open—originates from Michael Nygard's 'Release It!' and is implemented in libraries like Opossum. In a blockchain RPC pool, the ejection thresholds must be blockchain-specific: a single semantic failure (stale head) should eject an endpoint immediately, while transport failures may tolerate a few retries. Hysteresis prevents flapping: after ejection, the endpoint stays open for a cooldown period, then enters half-open and is probed with a real cheap read (e.g., eth_blockNumber) rather than a TCP ping. If the probe returns a head at or above the high-water mark, the endpoint returns to closed; otherwise it reopens.
The half-open probe must use a real read because a TCP ping can succeed while the node is still syncing and tens of thousands of blocks behind. The RPC node monitoring, metrics and alerting page covers observability for these states.
- Closed: normal operation; failures increment a counter.
- Open: endpoint ejected; no traffic until cooldown expires.
- Half-open: a single probe request (eth_blockNumber) tests freshness; success closes, failure reopens.
- Ejection thresholds: semantic failure = immediate eject; transport failure = 3 consecutive failures; rate-limit = backoff, not eject.
Health Probes That Cannot Be Fooled
A health probe that only checks TCP connectivity or HTTP 200 is easily fooled by a syncing node. The correct probe compares each endpoint's head against the pool maximum and against the chain's observed block time. If an endpoint's head is more than a few block times behind the pool maximum, it is considered stale and ejected. The chain's block time can be estimated from the difference in block timestamps over a window; an endpoint that is more than, say, three block times behind is likely syncing or stalled.
This probe must run periodically (e.g., every few seconds) and update the high-water mark. The RPC connection reuse and HTTP/2 keep-alive page covers transport efficiency for these probes.
// Health probe: compare head against pool max and block time
async function probeEndpoint(ep, pool) {
try {
const headHex = await ep.call('eth_blockNumber', []);
const head = parseInt(headHex, 16);
ep.lastHead = head;
ep.lastProbe = Date.now();
const poolMax = Math.max(...pool.endpoints.map(e => e.lastHead || 0));
const blockTimeMs = pool.estimatedBlockTimeMs || 12000;
const lagBlocks = poolMax - head;
const lagMs = lagBlocks * blockTimeMs;
if (lagMs > 3 * blockTimeMs) {
ep.state = 'open';
ep.ejectUntil = Date.now() + pool.cooldownMs;
return false;
}
if (ep.state === 'half-open') ep.state = 'closed';
return true;
} catch (err) {
ep.state = 'open';
ep.ejectUntil = Date.now() + pool.cooldownMs;
return false;
}
}Runnable Node.js Dispatch Layer
The following dispatch layer implements an endpoint registry, per-endpoint state, a head high-water mark, an ejection timer, a half-open probe, and a 429-aware wait. It uses fetch for HTTP calls and assumes each endpoint exposes a JSON-RPC 2.0 interface. The dispatcher selects an eligible endpoint via the freshness gate, sends the request, and classifies the response. On 429, it reads Retry-After and waits before retrying the same endpoint; on semantic failure, it ejects the endpoint and retries another. The high-water mark is updated on every successful head sample.
This code is a starting point; production deployments should add metrics, logging, and persistent state. The rate-limit headers and Retry-After handling page covers 429 semantics in detail.
class RpcPool {
constructor(endpoints, opts = {}) {
this.endpoints = endpoints.map(url => ({
url, state: 'closed', lastHead: 0, inFlight: 0,
maxInFlight: opts.maxInFlight || 10,
ejectUntil: 0, failures: 0
}));
this.highWaterMark = 0;
this.cooldownMs = opts.cooldownMs || 30000;
this.estimatedBlockTimeMs = opts.blockTimeMs || 12000;
}
async call(method, params) {
const ep = this.selectEndpoint();
ep.inFlight++;
try {
const res = await fetch(ep.url, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ jsonrpc: '2.0', id: 1, method, params })
});
if (res.status === 429) {
const retryAfter = res.headers.get('Retry-After');
const waitMs = retryAfter ? parseInt(retryAfter, 10) * 1000 : 1000;
await new Promise(r => setTimeout(r, waitMs));
return this.call(method, params);
}
const json = await res.json();
if (json.error) {
ep.failures++;
if (ep.failures >= 3) this.eject(ep);
throw new Error('RPC error: ' + JSON.stringify(json.error));
}
if (method === 'eth_blockNumber') {
const head = parseInt(json.result, 16);
ep.lastHead = head;
this.highWaterMark = Math.max(this.highWaterMark, head);
}
ep.failures = 0;
return json.result;
} catch (err) {
ep.failures++;
if (ep.failures >= 3) this.eject(ep);
throw err;
} finally {
ep.inFlight--;
}
}
selectEndpoint() {
const now = Date.now();
const eligible = this.endpoints.filter(ep =>
ep.state === 'closed' && ep.lastHead >= this.highWaterMark &&
ep.inFlight < ep.maxInFlight && now > ep.ejectUntil
);
if (eligible.length === 0) throw new Error('No fresh endpoint');
return eligible.reduce((a, b) => a.inFlight <= b.inFlight ? a : b);
}
eject(ep) {
ep.state = 'open';
ep.ejectUntil = Date.now() + this.cooldownMs;
}
}Results Table: Measure Against Your Own Providers
Use the following table to record observations from your own providers. Run the dispatch layer against your endpoints for at least 24 hours, sampling eth_blockNumber every 10 seconds. Fill in each column with your measured values. This is a verified-by-the-reader method; no benchmark numbers are provided here because provider behavior varies.
Columns: Provider URL, Observed Head Lag (blocks), Transport Failures (count), Protocol Failures (count), Semantic Failures (count), 429 Responses (count), Average Latency (ms), Ejection Events (count).
- Provider URL: the endpoint under test.
- Observed Head Lag: difference between provider head and pool high-water mark.
- Transport Failures: connection/TLS/DNS errors.
- Protocol Failures: JSON-RPC error objects.
- Semantic Failures: HTTP 200 with head below high-water mark.
- 429 Responses: rate-limit hits.
- Average Latency: mean round-trip time for eth_blockNumber.
- Ejection Events: number of times the circuit breaker opened.
Failure Modes and Troubleshooting
Split heads across providers occur when two providers report different heads and the pool's high-water mark is not updated consistently. This can happen if head sampling is infrequent or if a provider's head jumps ahead due to a reorg. The fix is to sample heads frequently and to use the maximum observed head as the high-water mark, but also to detect reorgs by comparing block hashes. A provider that silently serves a cached response will pass a naive health check; the freshness gate catches it because the cached head will be below the high-water mark. A half-open probe that passes during an outage can happen if the probe uses a cached or stale response; always use a real eth_blockNumber call and compare against the pool maximum. The thundering herd produced by a naive all-endpoints retry can be mitigated by adding jitter to retry delays and by limiting the number of concurrent retries.
The RPC request hedging for tail latency page covers hedging, which is a different technique from failover. The multi-region RPC failover and latency-aware routing page covers region-level failover.
- Split heads: sample heads frequently; detect reorgs via block hashes.
- Cached responses: freshness gate rejects heads below high-water mark.
- Half-open probe passing during outage: use real eth_blockNumber, not TCP ping.
- Thundering herd: add jitter to retries; limit concurrent retries.
Limitations and Tradeoffs
A multi-provider pool adds overhead: extra requests spent on head sampling, the cost of pinning reads to a single endpoint, and the complexity of maintaining per-endpoint state. Head sampling consumes quota and adds latency to the dispatch path. Pinning reads to the write endpoint can reduce load balancing effectiveness and may increase latency if that endpoint is slow. A pool cannot fix an intent problem: if the application logic requires a specific block tag (e.g., finalized), the pool must respect that tag and not substitute latest. The RPC pricing page can help estimate costs for additional requests.
The pool also cannot guarantee consistency across providers if the chain itself is reorging; it can only ensure that the pool's view is monotonic. For applications requiring strong consistency, consider using a single provider with a dedicated node, as described in the API service page.
- Extra requests: head sampling consumes quota.
- Pinning cost: reduced load balancing, potential latency increase.
- Intent mismatch: pool must respect block tags, not override them.
- Reorgs: pool ensures monotonic view, not cross-provider consistency.
Ethereum
Solana
Polygon
Base
Arbitrum
BNB Chain
Polkadot
Optimism
