Multi-Endpoint RPC Consistency: Head Lag and Conflicting Reads

Why the same query returns different data from different endpoints

A JSON-RPC endpoint is not a shared database. It is one node (or one load-balanced group of nodes) with its own peer set, its own sync state, and its own local view of the chain head. At any instant endpoint A may be at height N while endpoint B is at N-2, because block gossip, sync mode, and replica health differ per node. A latest read against A and a latest read against B therefore answer two different questions, even though the request bytes are identical.

This is the root cause of the two most common multi-endpoint bug reports: 'my app shows inconsistent balances' and 'the indexer saw a block the API did not'. Neither is a provider outage. They are the expected consequence of asking a moving target for a value. The multi-region RPC failover routing article flags this hazard around the chain head; this article teaches the reconciliation procedure that resolves it.

The important mental shift is that consistency is a property of your read pattern, not of the endpoint. If you never pin a height, no provider can make your reads agree, because agreement was never requested.

• Endpoint A at height N and endpoint B at N-2 are both healthy; they are simply at different points in the same chain.
• A latest read is a request for 'whatever this node currently believes is the tip', which is node-local and time-varying.
• A by-number or by-hash read is a request for a specific, immutable object, which is endpoint-independent once that block exists everywhere.

Block tags change the question, not just the freshness

The Ethereum JSON-RPC block parameter accepts latest, safe, finalized, earliest, pending, and explicit block numbers or hashes. These are not freshness dials on the same answer; they select different objects with different guarantees. The ethereum.org JSON-RPC API documentation defines the parameter set, and the execution APIs specification is the authoritative reference for method behaviour.

latest is the node's local head: fast, but non-deterministic across endpoints. safe is the supermajority-voted head introduced after the Merge, typically around two epochs behind the tip. finalized is the irreversible head, further behind still. pending is the node's local view of not-yet-included transactions, which is the least portable of all because it depends on that node's mempool. earliest is the genesis block.

By-number and by-hash reads are deterministic. Once a block exists on every endpoint you query, eth_getBlockByNumber at that height returns the same block, and eth_getBalance at that height returns the same balance. This is the only read shape that guarantees cross-endpoint agreement, which is why pinning is the core discipline.

• latest — local head, fast, endpoint-specific, can move backwards on a lagging node.
• safe — supermajority-voted head, documented as roughly two epochs behind on Ethereum post-Merge; per-network behaviour varies.
• finalized — irreversible head, longer delay, strongest guarantee.
• pending — local mempool view, not reproducible across endpoints.
• By-number / by-hash — deterministic and endpoint-independent once the block is present everywhere.

Head lag is normal behaviour, not downtime

A node that is two blocks behind is not failing a health check. It is still accepting connections, still answering eth_blockNumber, and still returning valid data for every height it has. Liveness and freshness are different properties, and most monitoring conflates them. The detecting an RPC node behind the chain tip article covers single-node tip-lag detection; the multi-endpoint case adds a second dimension, because you must decide which of several heads to trust.

Common causes of head lag include gossip propagation delay, a node still catching up after a restart, a node that pruned state and is re-fetching, and a lagging replica sitting behind a load balancer. In a failover pool, a request can land on any of these, so two consecutive calls from the same client can hit two different heads.

The practical consequence: a liveness probe that checks HTTP 200 and a non-null eth_blockNumber will pass on a node that is materially behind. Freshness must be measured separately, and measured per endpoint.

• Liveness: the endpoint responds. Freshness: the endpoint's head is close to the network head.
• A lagging replica behind a load balancer silently mixes stale and fresh answers.
• Failover that routes on liveness alone can route to a stale node.

The reconciliation discipline: pin, enforce monotonicity, compare like with like

Three rules make multi-endpoint reads reproducible. First, pin a height for correctness-critical reads: resolve latest once, capture the block number and hash, then query every source at that exact height or hash. Second, require monotonicity in your own pipeline: never accept a head that goes backwards, and treat a backwards head as a lagging endpoint rather than a reorg unless a hash at the same height changed. Third, read the same entity from the same pinned commitment when comparing sources.

The monotonicity rule deserves emphasis because it is the cheapest reorg detector you have. If endpoint A reports height 1000 and later reports 998, that is almost always A lagging or A having been restarted, not the chain reorganising. A real reorg shows up as the same height with a different hash. Distinguishing these two cases prevents a large class of false reorg alarms.

Pinning also makes caching and idempotency tractable. A read keyed by (method, params, blockHash) is safe to cache and safe to retry, because the underlying object cannot change. A read keyed by latest is neither.

• Resolve the head once, then pin every correctness-critical read to that height or hash.
• Reject backwards heads; only treat a same-height hash change as a reorg.
• Compare sources at the same pinned commitment, never latest against latest.
• Key caches and retries on the pinned hash, not on the tag.

Read-your-writes: a transaction visible on one endpoint but not another

When you broadcast a transaction to endpoint A, it enters A's mempool. Endpoint B has a different mempool and may not see the transaction for some time, if at all. A subsequent eth_getTransactionByHash against B can return null even though the transaction is perfectly valid and already known to A. This is the read-your-writes hazard in a multi-endpoint pool.

The correct pattern is to follow the transaction by hash on the endpoint that accepted it, or to poll all endpoints until a quorum sees it, before declaring success. Declaring success on a single endpoint's acceptance is a common source of 'the transaction disappeared' reports, because the next request may be routed to a different endpoint.

For indexers and reconciliation jobs, the same principle applies at block granularity. The block-by-block indexer reconciliation approach of walking a canonical height sequence and comparing hashes is the durable way to detect divergence between an indexer and an API.

• Broadcast and follow-up reads should target the same endpoint, or use a quorum.
• A null eth_getTransactionByHash on one endpoint is not proof the transaction failed.
• Quorum visibility before declaring success removes the read-your-writes race.

Measuring per-endpoint head lag reproducibly

To measure head lag without guessing, sample eth_blockNumber (or getSlot on Solana-style endpoints) against several endpoints on a timer, record the timestamp and the returned height, and compute the delta distribution per endpoint relative to the maximum observed height at each sample. Also record the block hash at a fixed height across endpoints to detect divergence, not just delay.

Run the sampler long enough to capture normal variation, and record the environment: network, endpoint URLs, sample interval, and total samples. Do not compare numbers from different runs or different networks. The table below is a template to fill with your own measurements; the values are yours to produce, not published benchmarks.

For Solana-style endpoints, the equivalent freshness signal is the slot height and the slot hash; the same delta-distribution method applies. See the Solana network page for endpoint context.

// head-lag sampler: run against several endpoints on a timer
// node sampler.js
const ENDPOINTS = [
  'https://endpoint-a.example/rpc',
  'https://endpoint-b.example/rpc',
  'https://endpoint-c.example/rpc'
];
const INTERVAL_MS = 2000;
const SAMPLES = 60;

async function rpc(url, method, params = []) {
  const res = await fetch(url, {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ jsonrpc: '2.0', id: 1, method, params })
  });
  const json = await res.json();
  if (json.error) throw new Error(json.error.message);
  return json.result;
}

async function sampleOnce() {
  const t = Date.now();
  const heights = await Promise.all(
    ENDPOINTS.map(async (url) => {
      try {
        const hex = await rpc(url, 'eth_blockNumber');
        return { url, height: parseInt(hex, 16), ok: true };
      } catch (e) {
        return { url, height: null, ok: false, error: String(e) };
      }
    })
  );
  const max = Math.max(...heights.filter(h => h.ok).map(h => h.height));
  return { t, heights, max };
}

(async () => {
  const rows = [];
  for (let i = 0; i < SAMPLES; i++) {
    rows.push(await sampleOnce());
    await new Promise(r => setTimeout(r, INTERVAL_MS));
  }
  for (const ep of ENDPOINTS) {
    const deltas = rows
      .map(r => r.heights.find(h => h.url === ep))
      .filter(h => h && h.ok)
      .map(h => r => 0); // placeholder, replaced below
    const d = rows
      .map(r => {
        const h = r.heights.find(x => x.url === ep);
        return h && h.ok ? r.max - h.height : null;
      })
      .filter(x => x !== null);
    d.sort((a, b) => a - b);
    const p50 = d[Math.floor(d.length * 0.5)] ?? null;
    const p95 = d[Math.floor(d.length * 0.95)] ?? null;
    console.log(ep, 'samples=', d.length, 'p50=', p50, 'p95=', p95, 'max=', d[d.length - 1]);
  }
})();

Results table: fill this with your own endpoint measurements

Use the sampler output to populate a table like the one below. Record one row per endpoint, and keep the run parameters fixed so rows are comparable. The point is not to produce a universal number but to characterise your own pool: which endpoints are consistently behind, and by how much.

Alongside the height delta, record hash agreement at a fixed height. Two endpoints can report the same height with different hashes only during a reorg window; outside that window, a same-height hash mismatch indicates a problem worth investigating.

• Endpoint URL — the exact URL sampled.
• Samples — number of successful samples.
• p50 delta — median blocks behind the max observed height.
• p95 delta — 95th percentile blocks behind.
• Max delta — worst observed lag.
• Hash agreement at fixed height — matches / mismatches across endpoints.

Decision table: two endpoints disagree, which do you trust?

When two endpoints return different data for the same logical query, classify the disagreement before reacting. The classification determines the correct action, and most disagreements are lag, not reorg.

Same height and same hash means the endpoints agree; the earlier difference was almost certainly a timing artefact between two latest reads. Same height and different hash means a reorg window: wait for finalized before acting on either. Different heights means lag: trust the higher head only if it is a valid descendant of the lower one, which you can verify by walking parent hashes.

• Same height, same hash — agree; re-read at the pinned height to confirm.
• Same height, different hash — reorg; wait for finality before committing.
• Different height, higher is a descendant — lag; the higher head is the fresher view.
• Different height, higher is not a descendant — investigate; this is not simple lag.

Troubleshooting: symptoms and the read pattern that fixes them

Most multi-endpoint consistency complaints map to a small set of read-pattern mistakes. The fix is usually to change what you ask for, not which provider you use.

If balances flicker between two values, you are reading latest across endpoints; pin a height. If an indexer sees a block the API does not, the API is behind; compare at a pinned height and treat the higher head as fresher. If a transaction appears to vanish, you are reading it on a different endpoint than the one that accepted it; follow by hash on the accepting endpoint or use a quorum. If a head goes backwards, treat it as a lagging endpoint unless the hash at the same height changed.

• Flickering values — pin the height and re-read.
• Indexer ahead of API — expected lag; reconcile at a pinned height.
• Transaction 'disappeared' — read-your-writes across endpoints; follow by hash or quorum.
• Head moved backwards — lagging endpoint, not a reorg, unless same-height hash changed.
• Failover routed to a stale node — add freshness to your routing signal, not just liveness.

Limitations and tradeoffs: freshness, latency, and the cost of quorum

Pinning a height trades freshness for determinism. A pinned read is reproducible and cacheable, but it is by definition not the tip. For correctness-critical reads that is the right trade; for UI freshness it may not be. Choose per read, not per application.

safe and finalized introduce latency by design. On Ethereum post-Merge, safe is documented as roughly two epochs behind the head and finalized further still; per-network behaviour varies, so treat these as documented / varies by provider and network rather than fixed constants. A quorum read costs more requests than a single read, but it is the only way to make a multi-endpoint read trustworthy when you cannot pin.

None of this removes the need for monitoring. Freshness must be measured continuously, and routing should consider it. The RPC node monitoring and failover article covers the operational side; the multi-chain RPC endpoints guide covers endpoint selection across networks.

• Pinned reads: deterministic, cacheable, not the tip.
• safe / finalized: stronger guarantees, added latency, per-network variation.
• Quorum reads: higher request cost, the only trustworthy multi-endpoint read when pinning is impossible.
• Freshness must be monitored and fed into routing, not assumed from liveness.

Next steps: apply the discipline to your own pool

Start by running the sampler against your actual endpoints and filling the results table. That gives you a baseline for how much lag your pool exhibits and which endpoints are consistently behind. Then change your read pattern: pin heights for correctness-critical reads, enforce monotonicity in your pipeline, and follow transactions on the endpoint that accepted them.

If you are evaluating providers, compare them on freshness behaviour and consistency guarantees, not just on liveness. The API service and RPC pricing pages describe the service surface, and the OnFinality Learn hub collects the related reliability articles. For a broader endpoint-selection view, see the multi-chain RPC endpoints guide.

The goal is not to eliminate head lag, which is normal, but to make your reads reproducible in spite of it. Pin, enforce monotonicity, and compare like with like.

• Measure your own pool before changing anything.
• Pin heights for correctness-critical reads.
• Enforce monotonicity and classify disagreements before reacting.
• Follow transactions on the accepting endpoint or use a quorum.