RPC WebSocket Reconnect Without Data Loss: Gap Detection and Backfill

Why a Reconnected Subscription Silently Drops Events

A WebSocket subscription is a cursor-less firehose. Per the Ethereum JSON-RPC PubSub specification, eth_subscribe with a logs parameter returns a subscription ID and then pushes notifications as new blocks are mined. The server does not replay events that occurred while your socket was down, and the close frame gives you no count of lost messages. A naive reconnect therefore produces a stream with a hole in it.

This is materially different from a request/response RPC call, where a failure is visible as an error. A subscription failure is invisible: your handler simply stops receiving, then resumes at some later block. If you are indexing transfers, liquidations, or governance events, a log stream with a hole is worse than no stream, because downstream consumers assume completeness.

The disconnect causes themselves are covered in Why WebSocket RPC disconnects and how to fix it; this article assumes you already reconnect and focuses on the harder problem: proving you did not miss anything, and deterministically recovering the gap.

• Subscriptions push new events only; there is no server-side replay buffer.
• A close frame carries no sequence number or lost-message count.
• Reconnect without reconciliation = silent gap.
• Correctness-critical consumers must treat the subscription as a hint, not a ledger.

The Cursor Mental Model: Block Number as a Monotonic Checkpoint

Treat every notification's block number as a monotonic checkpoint. Persist the highest block number you have fully processed as lastCursor. For reorg-safe designs, also persist the block hash so you can detect a chain reorganisation that invalidates your cursor, following the block-identity conventions described in EIP-1898 and the block-number semantics of EIP-234.

On reconnect, the rule is: do not trust the live stream until you have backfilled [lastCursor + 1 .. latest]. The backfill is a bounded eth_getLogs query over that range. Only after the backfill completes and you have advanced lastCursor to the backfill high-water mark should you resume consuming live notifications.

This turns an unreliable push channel into a reliable pull-plus-push pipeline. The subscription gives you low latency; the backfill gives you completeness. Neither alone is sufficient.

• Persist lastCursor (block number) and optionally lastHash (block hash).
• Backfill range is [lastCursor + 1 .. latest], bounded by latest.
• Advance the cursor only after the backfill is durably processed.
• Reorg safety: if lastHash no longer matches the chain, rewind the cursor.

The Ordering Hazard: Buffer Live Events Until Backfill Completes

A subtle failure mode: after you re-subscribe, the live stream may start delivering blocks that are ahead of your backfill high-water mark. If you process live events immediately, you can process block 100 (live) before block 95 (backfill), producing out-of-order state and double-counting.

The rule is to buffer live events until the backfill completes. Collect incoming notifications in a queue, run the backfill, then merge the queue with the backfill results, dedupe, and process in block order. Only after the queue is drained and the cursor is advanced do you switch to direct live processing.

This buffering window is short (seconds), but it is the difference between a correct indexer and one that occasionally double-counts a transfer. The same discipline applies whether you are on Ethereum or any other chain.

• Live stream can start ahead of the backfill high-water mark.
• Buffer live events in a queue during backfill.
• Merge, dedupe, sort by block number, then process.
• Switch to direct live processing only after the queue drains.

Dedupe by (Tx Hash, Log Index) or (Block Number, Log Index)

Backfill and live delivery can overlap. A log that arrived live at block 100 may also appear in your eth_getLogs backfill if the backfill range extended to block 100. Without dedupe, you process it twice.

The canonical dedupe key for Ethereum logs is (transactionHash, logIndex). For chains without a log index, use (blockNumber, eventIndex) or a deterministic event ID. Maintain a short-lived dedupe set (or a unique constraint in your database) covering at least the backfill window plus a safety margin.

Dedupe is not optional. It is the mechanism that makes the overlap between push and pull safe. If you skip it, your backfill introduces the very double-counting it was meant to prevent.

• Ethereum: dedupe key (transactionHash, logIndex).
• Generic: (blockNumber, eventIndex) or a deterministic event ID.
• Keep the dedupe set at least as long as the backfill window.
• A database unique constraint is the most robust dedupe.

Runnable TypeScript: Subscribe, Detect Reconnect, Backfill, Dedupe, Resume

The following example uses ethers v6 and a persistent cursor. It subscribes to logs, persists lastBlock, detects reconnect, runs a bounded getLogs backfill, dedupes, and only then resumes live processing. Replace the endpoint with your provider's WebSocket URL; provider-specific behaviour such as idle timeouts and max backfill ranges is documented / varies by provider.

Note the buffer array: live events arriving during backfill are queued, not processed. The seen set dedupes by (transactionHash, logIndex). The backfill function is bounded by latest and should be rate-limit-aware.

import { ethers } from "ethers";

const WS_URL = process.env.WS_URL!;
const ADDRESS = process.env.ADDRESS!; // contract to watch
const MAX_RANGE = 2000; // bounded backfill window

let lastBlock = Number(process.env.LAST_BLOCK ?? 0);
let backfilling = false;
const buffer: ethers.Log[] = [];
const seen = new Set<string>();

function key(l: ethers.Log) {
  return `${l.transactionHash}:${l.index}`;
}

async function processLog(l: ethers.Log) {
  const k = key(l);
  if (seen.has(k)) return;
  seen.add(k);
  // TODO: persist to your store
  console.log("processed", k, "block", l.blockNumber);
  if (l.blockNumber > lastBlock) lastBlock = l.blockNumber;
}

async function backfill(provider: ethers.Provider) {
  backfilling = true;
  const latest = await provider.getBlockNumber();
  let from = lastBlock + 1;
  while (from <= latest) {
    const to = Math.min(from + MAX_RANGE - 1, latest);
    const logs = await provider.getLogs({ address: ADDRESS, fromBlock: from, toBlock: to });
    logs.sort((a, b) => a.blockNumber - b.blockNumber || a.index - b.index);
    for (const l of logs) await processLog(l);
    from = to + 1;
  }
  // drain buffered live events
  buffer.sort((a, b) => a.blockNumber - b.blockNumber || a.index - b.index);
  for (const l of buffer) await processLog(l);
  buffer.length = 0;
  backfilling = false;
}

async function main() {
  const provider = new ethers.WebSocketProvider(WS_URL);
  provider.on("error", () => {});
  provider.websocket.on("close", async () => {
    console.warn("socket closed; reconnecting");
    await backfill(provider);
  });
  provider.on({ address: ADDRESS }, async (l: ethers.Log) => {
    if (backfilling) buffer.push(l);
    else await processLog(l);
  });
  await backfill(provider); // initial catch-up
}

main().catch(console.error);

Verification Loop: Kill the Socket, Inject an Event, Assert Exactly-Once Recovery

You cannot trust a gap-recovery design you have not tested. Build a verification loop that deliberately kills the socket, injects a known event during the outage, and asserts the event is recovered exactly once. This is the only way to prove your backfill and dedupe logic actually work.

The loop: (1) start the subscriber and record lastBlock; (2) force-close the WebSocket; (3) while disconnected, send a transaction that emits a known event; (4) allow reconnect and backfill to run; (5) assert the event appears exactly once in your store and that lastBlock advanced past it.

Run this loop against your own endpoint and record the results in a table. Do not rely on vendor-published latency or reliability numbers; measure your own.

• Force-close the socket programmatically (e.g. provider.websocket.close()).
• Inject a known event during the outage window.
• Assert exactly-once recovery and cursor advancement.
• Repeat across idle timeouts, provider deploys, and load-balancer resets.

Results Table: Measure Gap Recovery Against Your Own Endpoint

Use the following table to record your own measurements. Fill it in with results from your endpoint and your workload. Provider-specific numbers are documented / varies by provider, so your own measurements are the only reliable guide.

Run each scenario at least ten times and record the worst case, not the average. The gap window is what matters: if your backfill range exceeds the provider's max eth_getLogs range, you must chunk it.

• Scenario | Disconnect cause | Gap (blocks) | Backfill time (ms) | Events recovered | Duplicates | Pass/Fail
• Idle timeout | No traffic for N minutes | | | | |
• Provider deploy | Server-side restart | | | | |
• Load-balancer reset | Connection dropped | | | | |
• Forced close | Client-side kill | | | | |
• Reorg | Chain reorganisation | | | | |

Generalising the Pattern: Solana, Sui, and Substrate

The same discipline applies across chains with different primitives. On Solana, use a slot-based cursor: subscribe to a program or account, persist the last processed slot, and on reconnect backfill with getSignaturesForAddress and getTransaction. On Sui, use a checkpoint cursor and suix_queryEvents to backfill the gap. On Substrate, use a block-number cursor and system_events at each block.

The primitives differ, but the mental model is identical: monotonic cursor, bounded backfill, dedupe, buffer live events until backfill completes. If you understand the Ethereum case, you understand all of them.

For endpoint selection and failover across these chains, see the RPC endpoints guide and RPC node monitoring, metrics and failover.

• Solana: slot cursor + getSignaturesForAddress backfill.
• Sui: checkpoint cursor + suix_queryEvents backfill.
• Substrate: block-number cursor + system_events backfill.
• Same discipline, different primitives.

Bounding the Backfill: Rate Limits, Backoff, and the Gap Window

The gap window matters. Idle timeouts, provider deploys, and load-balancer resets can produce gaps ranging from seconds to minutes. Your backfill range must be bounded and rate-limit-aware, or you will hit provider limits and fail to recover.

Chunk your eth_getLogs calls (e.g. 2000 blocks per call) and apply exponential backoff on rate-limit errors. The backoff interacts with your reconnect logic: if you reconnect too aggressively, you may reconnect into a rate-limited state and fail again. See RPC timeout errors: causes and fixes for backoff patterns.

If your gap exceeds the provider's max backfill range, you must chunk. If it exceeds your retention window, you must fall back to a full re-sync. Know your limits before you need them.

• Chunk backfill calls to respect provider max ranges.
• Apply exponential backoff on rate-limit errors.
• Do not reconnect aggressively into a rate-limited state.
• Know your retention window; fall back to full re-sync if exceeded.

WebSocket-with-Backfill vs Plain Polling: Choosing for Correctness

Plain polling (eth_getLogs on a timer) is simpler and inherently gap-free if you persist the cursor, but it adds latency and can be more expensive at high frequency. WebSocket-with-backfill gives you low latency plus completeness, at the cost of more complex code.

Choose WebSocket-with-backfill when latency matters and you can implement dedupe and buffering correctly. Choose plain polling when correctness is paramount and latency is tolerable, or when your provider's WebSocket reliability is uncertain. The comparison is covered in eth_subscribe logs vs polling filters.

A hybrid is often best: WebSocket for low-latency hints, plus a periodic polling reconciliation as a safety net. This catches any gap your reconnect logic missed.

• Polling: simpler, gap-free with cursor, higher latency.
• WebSocket-with-backfill: low latency, more complex.
• Hybrid: WebSocket hints + periodic polling reconciliation.
• Choose based on latency tolerance and correctness requirements.

Failure Modes and Troubleshooting

The following table lists common failure modes and their fixes. Use it when your gap recovery is not working as expected.

If you see duplicates, your dedupe key is wrong or your dedupe set is too short. If you see missing events, your backfill range is wrong or your cursor advanced prematurely. If you see out-of-order processing, you are not buffering live events during backfill.

• Duplicates | Dedupe key wrong or set too short | Use (txHash, logIndex), extend set.
• Missing events | Backfill range wrong or cursor advanced early | Verify [lastCursor+1 .. latest], advance after backfill.
• Out-of-order | Live events not buffered | Buffer during backfill, sort by block.
• Rate-limited | Backfill too aggressive | Chunk calls, exponential backoff.
• Reorg corruption | No block hash check | Persist lastHash, rewind on mismatch.
• Silent gap | No backfill at all | Implement cursor + backfill.

Limitations, Tradeoffs, and Next Steps

This pattern has limitations. It assumes your provider supports eth_getLogs over the gap range; some providers cap the range or the number of results. It assumes your cursor is durable; if your process crashes between processing and persisting, you may reprocess or skip. It assumes no deep reorg beyond your retention window; deeper reorgs require a full re-sync.

The tradeoff is complexity: you are building a small reconciliation engine, not just a subscriber. For correctness-critical consumers, that complexity is justified. For low-stakes dashboards, plain polling may be sufficient.

Next steps: implement the verification loop against your own endpoint, fill in the results table, and review RPC pricing and the API service to understand cost implications. For a broader overview, see the OnFinality Learn hub.

Update conditions: revisit this design when your provider changes max backfill ranges, when you add a new chain, or when you observe a reorg deeper than your retention window.

• Provider max backfill range may force chunking.
• Cursor durability requires transactional persistence.
• Deep reorgs beyond retention require full re-sync.
• Revisit on provider changes, new chains, or deep reorgs.