Building a Block-by-Block EVM Indexer: Joining Blocks, Transactions, and Receipts

What a block processor must assemble from the JSON-RPC surface

A block-by-block EVM indexer is not a log scraper; it is a reconciliation pipeline that produces one internally consistent row set per block. The standard JSON-RPC surface gives you three related views: the block with its transactions, the receipts for those transactions, and the logs embedded in each receipt. The Ethereum execution-apis JSON-RPC specification defines eth_getBlockByNumber, eth_getBlockReceipts, eth_getTransactionReceipt, and eth_getLogs as the authoritative methods for these views, and ethereum.org's JSON-RPC API page documents the same surface for application developers.

The job is to join those views without skipping, duplicating, or mis-ordering a block. That means fetching a block with eth_getBlockByNumber(blockParameter, true) so transaction objects come inside the block, then fetching the matching receipts for the SAME block parameter. If your provider supports it, eth_getBlockReceipts returns all receipts for a block in one call; otherwise fall back to per-transaction eth_getTransactionReceipt. The join key is the transaction index, with transactionHash as a cross-check, never a timestamp or an assumed sort. For the canonical method definitions, see the Ethereum execution-apis JSON-RPC specification and the Ethereum JSON-RPC API reference.

• Block view: eth_getBlockByNumber(blockParameter, true) returns transactions as objects, not hashes.
• Receipt view: eth_getBlockReceipts(blockParameter) returns receipts in transaction-index order; eth_getTransactionReceipt is the per-transaction fallback.
• Log view: logs inside a receipt belong to that transaction, and their logIndex is block-global.
• Stable ordering: sort rows by (blockNumber, transactionIndex, logIndex).

Why the block parameter is a correctness decision, not a style choice

Indexing 'latest' is a moving target: the block you fetch can change between the block call and the receipt call, and a reorg can rewrite it while you are writing rows. A processor should walk a fixed numeric range, persist the last fully-committed block number in the same transaction as its rows, and treat confirmations or finality depth as an explicit configuration input rather than an assumption. This is the difference between a pipeline that can prove it never skipped a block and one that merely hopes it did not.

The block parameter also interacts with provider behavior. Archive requirements, rate limits, and caps on wide eth_getLogs scans are documented / varies by provider, so your configuration should expose them as inputs. If you need to detect a node that is behind the chain tip before you start a range, see Detecting an RPC node behind the chain tip. For endpoint selection and provider tradeoffs, the Ethereum RPC node guide is a useful reference.

The join contract: receipts.length, transactionHash, and block-global logIndex

The join function is where most indexers silently break. It must assert that receipts.length === block.transactions.length and that every receipt.transactionHash matches the corresponding block transaction hash. If either assertion fails, the block is not safe to commit; the processor should retry the fetch or halt the range rather than write partial rows. This is the mechanism that prevents a lagging node from pairing a block from one view with receipts from another.

Receipts are returned in transaction-index order, so the index is the join key. Logs inside a receipt belong to that transaction, but their logIndex is block-global, which means sorting logs per-receipt is wrong. Ordering rows by (blockNumber, transactionIndex, logIndex) is what makes the output stable across restarts and re-fetches. For the bulk-receipt method itself, see Fetching every receipt in a block with eth_getBlockReceipts; for log filtering mechanics, see Filtering event logs with eth_getLogs and topics.

• Assert receipts.length === block.transactions.length before any write.
• Assert receipt.transactionHash === block.transactions[i].hash for every i.
• Use transactionIndex as the join key; use transactionHash as the cross-check.
• Sort output by (blockNumber, transactionIndex, logIndex), not by receipt-local log order.

A runnable Node.js processor loop with atomic cursor and continuity check

The loop below fetches a block and its receipts, joins them with assertions, persists rows and the cursor in one transaction, and checks parentHash continuity. It uses a generic JSON-RPC client and a generic SQL transaction; adapt the driver to your database. The key properties are: the fetch step returns {block, receipts, rows}; the join step asserts counts and hashes; the persist step writes rows and the cursor atomically; and the continuity check triggers a bounded re-index when parentHash does not match the previous block hash.

Run this against a fixed numeric range, not 'latest'. The cursor is the last fully-committed block number, and it is written in the same transaction as the rows, so a crash cannot leave the cursor ahead of the data. If the continuity check fails, re-index a bounded window (for example, the last N blocks) rather than the whole chain.

// Node.js 18+ (ESM). Generic JSON-RPC + SQL transaction. Adapt driver to your DB.
import { JsonRpcProvider } from 'ethers'; // or any JSON-RPC client

const provider = new JsonRpcProvider(process.env.RPC_URL);
const CONFIRMATIONS = Number(process.env.CONFIRMATIONS ?? 12);
const REORG_WINDOW = Number(process.env.REORG_WINDOW ?? 64);

async function fetchBlockAndReceipts(blockNumber) {
  const block = await provider.send('eth_getBlockByNumber', [
    '0x' + blockNumber.toString(16), true
  ]);
  if (!block) throw new Error(`missing block ${blockNumber}`);

  let receipts;
  try {
    receipts = await provider.send('eth_getBlockReceipts', [
      '0x' + blockNumber.toString(16)
    ]);
  } catch (e) {
    // Fallback: per-transaction receipts when bulk method is unavailable.
    receipts = await Promise.all(
      block.transactions.map((tx) =>
        provider.send('eth_getTransactionReceipt', [tx.hash])
      )
    );
  }
  return { block, receipts };
}

function joinBlock(block, receipts) {
  if (receipts.length !== block.transactions.length) {
    throw new Error(
      `receipt count mismatch: ${receipts.length} vs ${block.transactions.length}`
    );
  }
  const rows = [];
  for (let i = 0; i < block.transactions.length; i++) {
    const tx = block.transactions[i];
    const rc = receipts[i];
    if (rc.transactionHash.toLowerCase() !== tx.hash.toLowerCase()) {
      throw new Error(`hash mismatch at index ${i}`);
    }
    rows.push({
      blockNumber: parseInt(block.number, 16),
      transactionIndex: i,
      transactionHash: tx.hash,
      from: tx.from,
      to: tx.to,
      status: rc.status,
      gasUsed: rc.gasUsed,
      logs: rc.logs.map((log) => ({
        logIndex: parseInt(log.logIndex, 16),
        address: log.address,
        topics: log.topics,
        data: log.data
      }))
    });
  }
  // Stable ordering: block-global logIndex, then transactionIndex.
  rows.sort((a, b) => a.transactionIndex - b.transactionIndex);
  for (const row of rows) row.logs.sort((a, b) => a.logIndex - b.logIndex);
  return rows;
}

async function persistBlock(db, block, rows, cursor) {
  await db.query('BEGIN');
  try {
    for (const row of rows) {
      await db.query(
        'INSERT INTO tx_rows (block_number, tx_index, tx_hash, payload) VALUES ($1,$2,$3,$4) ON CONFLICT DO NOTHING',
        [row.blockNumber, row.transactionIndex, row.transactionHash, row]
      );
    }
    await db.query(
      'INSERT INTO cursor (id, last_block) VALUES (1,$1) ON CONFLICT (id) DO UPDATE SET last_block = EXCLUDED.last_block',
      [cursor]
    );
    await db.query('COMMIT');
  } catch (e) {
    await db.query('ROLLBACK');
    throw e;
  }
}

async function runRange(db, fromBlock, toBlock) {
  let cursor = fromBlock - 1;
  let prevHash = null;
  for (let n = fromBlock; n <= toBlock; n++) {
    const { block, receipts } = await fetchBlockAndReceipts(n);
    if (prevHash && block.parentHash.toLowerCase() !== prevHash.toLowerCase()) {
      // Bounded re-index: step back and re-process the reorg window.
      const rewind = Math.max(fromBlock, n - REORG_WINDOW);
      console.warn(`reorg detected at ${n}; rewinding to ${rewind}`);
      n = rewind - 1;
      prevHash = null;
      continue;
    }
    const rows = joinBlock(block, receipts);
    await persistBlock(db, block, rows, n);
    cursor = n;
    prevHash = block.hash;
  }
  return cursor;
}

// Metrics assertion: expected blocks for an interval must match committed blocks.
function assertInterval(expected, committed) {
  if (expected !== committed) {
    throw new Error(`interval mismatch: expected ${expected}, committed ${committed}`);
  }
}

Reorg detection and bounded re-indexing with blockHash and parentHash

blockHash and parentHash are what let a processor detect a reorg and re-index the affected range. When you fetch block N, its parentHash should equal the hash of block N-1 that you already committed. If it does not, the chain has reorganized and your committed rows for the affected range may be stale. The correct response is a bounded re-index: step back a configured window, re-fetch those blocks, and overwrite or version the affected rows.

Reorg writes that mutate already-committed rows without a version or a re-index marker are a common failure. Either version rows by blockHash or mark them with a re-index flag so downstream consumers can distinguish canonical from orphaned data. The window size is a configuration input, not a constant; it should reflect the chain's observed reorg depth and your confirmations setting.

• Compare block.parentHash to the previously committed block hash on every iteration.
• On mismatch, rewind a bounded window and re-process; do not continue forward.
• Version rows by blockHash or mark re-indexed rows so consumers can filter orphans.
• Keep the reorg window and confirmations as explicit configuration inputs.

Common failures and how each one shows up in the pipeline

Short reads silently drop a block when a request fails mid-range and the loop advances anyway. The symptom is a gap in committed block numbers with no error. Duplicate rows on restart happen when the cursor was written before the data; the symptom is repeated (blockNumber, transactionIndex) keys. Wrong ordering happens when logs were sorted per-receipt instead of block-globally; the symptom is logIndex values that are not monotonic across the block.

Missing receipts happen when the block was fetched from a lagging node while receipts came from another; the symptom is a receipt count mismatch or a transactionHash mismatch. Reorg writes that mutate already-committed rows without a version or a re-index marker show up as downstream consumers seeing transactions that later disappear. Rate-limit or timeout failures on wide eth_getLogs scans show up as intermittent 429 or timeout errors; the mitigation pattern is to narrow the range, add backoff, and use a provider with documented limits. For BNB range limits and reliability patterns, see the OnFinality Learn hub and the reliability pages linked there.

• Short read: gap in committed block numbers, no error raised.
• Duplicate rows: cursor written before data; repeated primary keys on restart.
• Wrong ordering: logIndex not monotonic across the block.
• Missing receipts: receipt count or transactionHash mismatch.
• Reorg writes: committed rows mutate without a version or re-index marker.
• Rate limits: intermittent 429 or timeout on wide eth_getLogs scans.

Results table: measure your own endpoint's block, transaction, and receipt counts

Do not trust a provider's headline numbers; measure your own endpoint against a fixed block range. Fill in the table below with counts you observe for a range you control, then compare committed rows to expected counts. This is the verified-by-the-reader method: the numbers are yours, not ours. If your endpoint returns fewer receipts than transactions for any block, stop and investigate before committing.

Use the same range for every row so the comparison is meaningful. If you change providers or regions, re-run the table; provider caps, archive requirements, and rate limits are documented / varies by provider.

• Block range: [start, end] — fixed numeric range, not 'latest'.
• Expected blocks: end - start + 1.
• Observed blocks committed: count from your cursor table.
• Expected transactions: sum of block.transactions.length across the range.
• Observed receipts: sum of receipts.length across the range.
• Mismatch count: blocks where receipts.length !== transactions.length.
• Reorg events: count of parentHash continuity failures.
• Rate-limit errors: count of 429 or timeout responses.

Troubleshooting checklist for a block-by-block indexer

Work through this checklist when your pipeline reports a mismatch or a gap. Each item maps to a specific failure mode and a specific fix. Keep the checklist in your runbook so an on-call engineer can follow it without re-deriving the mechanism.

If you need to simulate a call against historical state while debugging, eth_call with state overrides is a useful companion technique. For endpoint selection and pricing tradeoffs, see RPC pricing and the API service pages.

• Verify the block parameter is a fixed number, not 'latest'.
• Assert receipts.length === block.transactions.length before writing.
• Assert every receipt.transactionHash matches the block transaction hash.
• Confirm the cursor is written in the same transaction as the rows.
• Check parentHash continuity against the previously committed block hash.
• Confirm logs are sorted by block-global logIndex, not per-receipt.
• Confirm the block and receipts came from the same node and the same block parameter.
• Check for 429 or timeout errors on wide eth_getLogs scans and narrow the range.
• Re-run the results table after any provider or region change.

Limitations, assumptions, and tradeoffs

This design assumes a JSON-RPC endpoint that returns consistent block and receipt views for the same block parameter. It does not assume eth_getBlockReceipts is available; the fallback to per-transaction eth_getTransactionReceipt is part of the contract. It assumes your database supports atomic transactions; if it does not, you need an equivalent idempotent write pattern with a version column.

The tradeoffs are real. Fetching receipts per transaction is slower and more request-heavy than the bulk method. A larger reorg window costs more re-indexing work but reduces the chance of serving stale rows. Confirmations add latency but reduce reorg exposure. Provider caps, archive requirements, and rate limits are documented / varies by provider, so your configuration must expose them as inputs rather than hard-code them.

• Assumes consistent block and receipt views for the same block parameter.
• Assumes atomic database transactions or an equivalent idempotent write pattern.
• Per-transaction receipt fallback is slower and more request-heavy than bulk receipts.
• Larger reorg windows cost more re-indexing work but reduce stale-row risk.
• Confirmations add latency but reduce reorg exposure.
• Provider caps, archive requirements, and rate limits are documented / varies by provider.

Next steps: from a working loop to a production indexer

Once the loop is correct for a fixed range, the next steps are operational: add metrics for committed blocks per interval, expose the cursor and reorg window as configuration, and run the results table against every endpoint you plan to use. For network-specific endpoints, see Ethereum on OnFinality. For pricing and service scope, see RPC pricing and the API service.

If you are choosing an endpoint, the Ethereum RPC node guide covers provider tradeoffs. If you need to detect a node that is behind the chain tip before starting a range, see Detecting an RPC node behind the chain tip. For the bulk-receipt method and log filtering mechanics, see Fetching every receipt in a block with eth_getBlockReceipts and Filtering event logs with eth_getLogs and topics.

• Add metrics for committed blocks per interval and raise on mismatch.
• Expose cursor, confirmations, and reorg window as configuration.
• Run the results table against every endpoint you plan to use.
• Re-run the table after any provider or region change.
• Keep the troubleshooting checklist in your runbook.