Scanning BNB Smart Chain Logs at Scale: eth_getLogs Range Limits and Safe Pagination

Why eth_getLogs Is Not a Full-History Query

The eth_getLogs method takes fromBlock, toBlock, address[], and topics[][], and returns matching logs. Per the Ethereum execution-apis specification, the node must scan every block in the requested range and deserialise any matching logs. That means a wide range costs O(blocks) work and memory, not O(results).

BNB Smart Chain has a very large block count because it produces blocks every few seconds over years, and it carries high log volume from ERC-20 activity. A naive fromBlock: 0, toBlock: 'latest' query is therefore both expensive and likely to be rejected. Most providers return an explicit error such as 'block range too large' or a limit-exceeded message, and historically a specific limit around a few thousand blocks has been documented in the bnb-chain/bsc GitHub issue #113. Some endpoints silently truncate instead of erroring, which is worse because it produces a patchy scan that looks correct.

• Wide ranges cost O(blocks) work and memory, not O(results).
• Provider caps are documented / varies by provider; never assume a universal number.
• Silent truncation is the most dangerous failure mode because it looks like success.

How Range Caps Manifest: Error, Timeout, or Truncation

When a range is too wide, the endpoint may return a JSON-RPC error with a message like 'block range too large' or 'limit exceeded'. It may also time out at the HTTP layer, returning a 504 or a connection reset. A third possibility is silent truncation: the node returns a subset of logs without an error, so your indexer records an incomplete history.

Because these behaviours vary by provider and by endpoint tier, you must treat the cap as an empirical property of the endpoint you are using. The BNB Smart Chain RPC reliability and timeouts page covers how timeouts and rate limits interact with retries, and the BNB Smart Chain RPC endpoints (RPC Assistant) page lists endpoints you can test against.

• Explicit error: 'block range too large' or limit exceeded.
• Timeout: HTTP 504 or connection reset.
• Silent truncation: fewer logs than expected with no error.

The Correct Strategy: Fixed-Window Pagination with a Persisted Cursor

Page by block range using a fixed window, for example 500–2000 blocks, tuned to your endpoint. Set fromBlock = lastScanned + 1 and toBlock = min(lastScanned + window, latest). After a successful page, advance lastScanned to toBlock and persist it. On restart, resume from the persisted cursor so you never re-scan or skip blocks.

Persist the cursor in durable storage, not in memory. If you process logs before persisting the cursor, you may re-process on restart; if you persist before processing, you may skip. The safe pattern is to write logs and cursor in the same transaction, or to make log writes idempotent via composite-key dedupe.

• Window size is a tunable parameter, not a constant.
• Advance only after a fully successful page.
• Persist the cursor durably and make writes idempotent.

Discovering the Real Cap for Your Endpoint Empirically

Do not trust a blog post's number. Binary-search the window size that succeeds without error or timeout on your endpoint. Start with a small window that works, double it until it fails, then narrow between the last success and the first failure. Record the result in a config file and re-test when you change providers or tiers.

When you hit a 429 or a timeout, back off with exponential backoff and jitter. A simple schedule is 250 ms, 500 ms, 1 s, 2 s, 4 s, capped at 30 s, with random jitter of up to 250 ms. This avoids thundering-herd retries and is consistent with the guidance in the BNB Smart Chain RPC reliability and timeouts page.

• Binary-search the window: double until failure, then narrow.
• Record the discovered cap per endpoint and per tier.
• Back off on 429/timeout with exponential backoff plus jitter.

Narrowing the Query: Address and Topics

Always narrow by address (the token contract) and topics (the Transfer event signature and indexed from/to). The Transfer event signature is 0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef. Filtering by topic reduces the data the node must deserialise and return, which lowers the chance of a timeout and reduces bandwidth.

The chain-neutral mechanics of topics and filter semantics are covered in Filtering event logs with eth_getLogs and topics. Use that page for the filter grammar; use this page for the BNB-specific scan strategy.

• Filter by token contract address.
• Filter by Transfer event signature and indexed from/to.
• Narrower filters reduce timeout risk and bandwidth.

Ordering, Dedupe, and Reorg Repair

Sort results by (blockNumber, logIndex) and dedupe on the composite (transactionHash, logIndex). A reorg can re-emit or move logs, so the same logical transfer may appear with a different block number or log index. Composite-key dedupe prevents double-counting while allowing legitimate re-emissions to replace stale entries.

Re-scan a confirmation tail on each pass, for example the last N blocks, to repair reorged logs. Never treat a log from a tip block as final. The size of N depends on your risk tolerance and the chain's reorg depth; a common starting point is 12–64 blocks, but you should measure it for your use case.

• Sort by (blockNumber, logIndex).
• Dedupe on (transactionHash, logIndex).
• Re-scan a confirmation tail each pass; never finalise tip logs.

Runnable Node.js: Paged Transfer Scan with Backoff and Cursor

The following script pages a token's Transfer logs over a large range with a configurable window, exponential backoff on 429/timeout, a persisted cursor, and composite-key dedupe. It prints progress and a results table you can fill in for your endpoint. Replace the RPC URL and token address with your own.

Run it with Node.js 18+ (global fetch). The cursor is stored in a JSON file for simplicity; in production use a database transaction.

// scan_transfers.js
// Usage: node scan_transfers.js
// Requires Node.js 18+ (global fetch)

const fs = require('fs');

const RPC_URL = process.env.RPC_URL || 'https://your-bnb-endpoint';
const TOKEN = process.env.TOKEN || '0x...';
const WINDOW = Number(process.env.WINDOW || 1000);
const CONFIRMATION_TAIL = Number(process.env.CONFIRMATION_TAIL || 32);
const CURSOR_FILE = './cursor.json';
const TRANSFER_TOPIC = '0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef';

function loadCursor() {
  if (fs.existsSync(CURSOR_FILE)) return JSON.parse(fs.readFileSync(CURSOR_FILE));
  return { lastScanned: 0, seen: {} };
}

function saveCursor(c) {
  fs.writeFileSync(CURSOR_FILE, JSON.stringify(c));
}

async function rpc(method, params, attempt = 0) {
  const res = await fetch(RPC_URL, {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ jsonrpc: '2.0', id: 1, method, params })
  });
  if (res.status === 429 || res.status === 504) {
    const delay = Math.min(30000, 250 * 2 ** attempt) + Math.random() * 250;
    console.warn(`retry ${attempt + 1} after ${Math.round(delay)}ms (status ${res.status})`);
    await new Promise(r => setTimeout(r, delay));
    return rpc(method, params, attempt + 1);
  }
  const json = await res.json();
  if (json.error) {
    const msg = JSON.stringify(json.error);
    if (/range|limit|too large/i.test(msg) && attempt < 6) {
      const delay = Math.min(30000, 250 * 2 ** attempt) + Math.random() * 250;
      console.warn(`range error, backing off ${Math.round(delay)}ms`);
      await new Promise(r => setTimeout(r, delay));
      return rpc(method, params, attempt + 1);
    }
    throw new Error(msg);
  }
  return json.result;
}

async function latestBlock() {
  const hex = await rpc('eth_blockNumber', []);
  return parseInt(hex, 16);
}

async function getLogs(from, to) {
  return rpc('eth_getLogs', [{
    fromBlock: '0x' + from.toString(16),
    toBlock: '0x' + to.toString(16),
    address: TOKEN,
    topics: [TRANSFER_TOPIC]
  }]);
}

async function main() {
  const cursor = loadCursor();
  const latest = await latestBlock();
  let scanned = 0, found = 0, retries = 0;
  let from = cursor.lastScanned + 1;
  while (from <= latest) {
    const to = Math.min(from + WINDOW - 1, latest);
    let logs;
    try {
      logs = await getLogs(from, to);
    } catch (e) {
      console.error(`page ${from}-${to} failed: ${e.message}`);
      break;
    }
    for (const log of logs) {
      const key = `${log.transactionHash}:${log.logIndex}`;
      if (!cursor.seen[key]) {
        cursor.seen[key] = true;
        found++;
      }
    }
    cursor.lastScanned = to;
    saveCursor(cursor);
    scanned += to - from + 1;
    console.log(`scanned ${from}-${to} | logs ${logs.length} | total ${found}`);
    from = to + 1;
  }
  // Re-scan confirmation tail to repair reorgs
  const tailFrom = Math.max(0, cursor.lastScanned - CONFIRMATION_TAIL + 1);
  const tailLogs = await getLogs(tailFrom, cursor.lastScanned);
  console.log(`tail re-scan ${tailFrom}-${cursor.lastScanned} | logs ${tailLogs.length}`);
  console.log('\nResults Table (fill in for your endpoint):');
  console.log('| blocks scanned | logs found | window size | retries |');
  console.log(`| ${scanned} | ${found} | ${WINDOW} | ${retries} |`);
}

main().catch(e => { console.error(e); process.exit(1); });

Results Table: Measure Against Your Own Endpoint

Use the table below to record what your endpoint actually does. Run the script with different WINDOW values and note where errors or timeouts begin. This is the only reliable way to know your cap.

Record the endpoint URL, the window size, whether the page succeeded, the number of logs returned, and the retry count. Repeat for at least three window sizes to find the boundary.

• | Endpoint | Window | Success? | Logs | Retries |
• |----------|--------|----------|------|---------|
• | https://... | 500 | yes | ... | 0 |
• | https://... | 1000 | yes | ... | 0 |
• | https://... | 2000 | no (range too large) | 0 | 2 |

When eth_getLogs Paging Is the Wrong Tool

For continuous realtime use subscriptions over WebSocket or a provider's indexed API. The BNB Smart Chain RPC endpoints (RPC Assistant) page and the API service page describe options. For enormous full-history scans, consider an archive endpoint plus a range-based or third-party indexer.

Archive access is required for old logs on a pruned full node. The BNB Smart Chain historical RPC and archive data page explains archive requirements. If you are scanning a different chain, the Querying Solana historical data over RPC page shows a comparable approach.

• Realtime: use subscriptions or an indexed API.
• Full history: use an archive endpoint or a range-based indexer.
• Pruned nodes cannot serve old logs.

Common Failures and a Troubleshooting Checklist

The most common failures are 'range too large', request timeout, 429 rate limit, empty pages that look correct but are truncation, duplicate or missing logs after a reorg, and querying a pruned full node for old logs. Each has a distinct fix.

Work through the checklist below before changing your code. Most issues are endpoint or window-size problems, not logic bugs.

• Range too large: reduce window size and re-test.
• Timeout: reduce window, add backoff, or use a dedicated endpoint.
• 429: back off with jitter and lower concurrency.
• Empty page: verify against a known block with a known transfer.
• Duplicates/missing after reorg: re-scan confirmation tail and dedupe.
• Old logs missing: switch to an archive endpoint.

Limitations, Tradeoffs, and Next Steps

Paging by block range is simple and portable, but it is slower than an indexed API and requires you to manage a cursor and reorg repair. Window size is a tradeoff between request count and failure risk. Dedupe storage grows with history size, so plan for pruning or a database index.

Next, review the BNB Smart Chain RPC reliability and timeouts page for retry patterns, the BNB Smart Chain historical RPC and archive data page for archive access, and the OnFinality Learn hub for related guides. For endpoint options and pricing, see RPC pricing and the BNB Smart Chain network page.

• Paging is portable but slower than indexed APIs.
• Window size trades request count against failure risk.
• Dedupe storage grows; plan pruning or indexing.