Solana getProgramAccounts: Building a Resumable Account-Set Indexer

The indexer contract: snapshot plus stream

A full getProgramAccounts scan establishes a snapshot of a program's account set at a recorded slot. An incremental stream keeps that snapshot current. The contract is that these two never disagree silently: every write carries a version, and every restart can prove which stream range must be replayed.

The scan's request and response contract is authoritative in the Solana JSON-RPC documentation for getProgramAccounts, including the RpcResponse context slot, the dataSize/memcmp filter array, dataSlice, and the withContext envelope. The streaming surface is documented in the Solana JSON-RPC websocket documentation for accountSubscribe and logsSubscribe.

This article deliberately does not re-teach filter semantics, memcmp offsets, dataSlice as a bandwidth control, or the resumable key-order scan. Those primitives belong to the prerequisite page linked above and again in next steps.

• Snapshot: a full scan whose pages are keyed on a stable account key and stamped with a slot.
• Stream: accountSubscribe for per-account changes, starting at the current slot.
• Reconciliation: a pass that distinguishes closed accounts from accounts merely absent from a page.

The context slot as the snapshot's identity

getProgramAccounts returns an RpcResponse whose context.slot is the slot the page was read at. A correct snapshot records the minimum context slot across its pages, not the maximum, and not the wall-clock time. A resumable scan that pages over an advancing chain spans a slot range, so the earliest slot is the only value that guarantees no change between that slot and the snapshot's completion is missed.

Recording the maximum would silently skip changes that landed between the first page and the last. Recording wall-clock time is worse: it has no defined relationship to chain state and cannot be replayed against getBlocks.

• Record min(context.slot) across all pages as the snapshot slot.
• Persist the snapshot slot alongside the cursor so a restart knows where the stream must resume.
• Never substitute block height or wall-clock time for the context slot.

Why a slot cursor is not a block-height cursor

Slots can be skipped, so a cursor cannot be an integer increment. The cursor must be a slot range walked with getBlocks, which returns the produced blocks in a range. getBlockHeight is a different quantity and must not be used as a cursor, as documented in the Solana JSON-RPC documentation for getSlot, getBlockHeight and getBlocks.

A slot with no produced block cannot be a cursor position. If your cursor is an integer increment, a skipped slot will either stall the indexer or cause it to skip real state. Walking the range with getBlocks makes skipped slots explicit and lets the gap-fill logic treat them correctly.

• Cursor = a slot range, not a single integer.
• Use getBlocks to enumerate produced blocks in the range.
• Treat skipped slots as gaps to be detected, not as errors to be ignored.

Building the snapshot with a persisted cursor

Enumerate pages, key each page on a stable account key, and persist (account key, slot) pairs. Use the recorded slot to decide which incremental stream must be replayed from where on restart. The scan itself is the prerequisite page's resumable key-order scan; here we only add the slot stamping and persistence.

The code below shows the snapshot loop with context slot capture and a cursor write. It assumes the pagination helper from the sibling article and focuses on the indexer lifecycle.

const { Connection, PublicKey } = require('@solana/web3.js');

async function snapshot(connection, programId, store) {
  let cursor = await store.getCursor();
  let minSlot = null;
  let page = 0;

  while (true) {
    const res = await connection.getProgramAccounts(new PublicKey(programId), {
      withContext: true,
      filters: [],
      dataSlice: { offset: 0, length: 0 },
      // pagination via key-order scan is handled by the sibling helper
      ...(cursor ? { before: cursor } : {})
    });

    const slot = res.context.slot;
    if (minSlot === null || slot < minSlot) minSlot = slot;

    for (const { pubkey, account } of res.value) {
      await store.upsert(pubkey.toBase58(), { slot, data: account.data });
    }

    if (res.value.length === 0) break;
    cursor = res.value[res.value.length - 1].pubkey.toBase58();
    await store.setCursor(cursor);
    page += 1;
  }

  await store.setSnapshotSlot(minSlot);
  return { minSlot, page };
}

module.exports = { snapshot };

Keeping the index current with accountSubscribe

accountSubscribe delivers per-account changes, but a subscription starts at the current slot and therefore cannot backfill between the snapshot slot and the subscription slot. That gap must be closed by a bounded re-scan. Bound it by recording the subscription slot and re-scanning the range [snapshotSlot, subscriptionSlot] using getBlocks to walk produced blocks.

The bounded re-scan is not a second full scan; it is a targeted replay of the gap. Its size is the difference between two slots, which is measurable and can be capped. If the gap exceeds your cap, fall back to a fresh snapshot rather than letting the gap grow unbounded.

• Record the subscription slot when the websocket confirms.
• Re-scan [snapshotSlot, subscriptionSlot] with getBlocks to walk produced blocks.
• Cap the gap; if it exceeds the cap, take a fresh snapshot.

Deletion detection by ownership and liveness

A closed account disappears from the scan, so a naive 'replace my table with what I just scanned' indexer silently deletes live data, and a naive merge indexer keeps deleted accounts forever. The correct approach reconciles by ownership and liveness: an account that the scan no longer returns and that getAccountInfo reports as non-existent has been closed.

The account model and what closing an account means for its lamports and owner are documented in the Solana documentation on account ownership and the account model. Use that to distinguish a closed account from an account that was merely not returned because a page errored.

• Closed: scan no longer returns it and getAccountInfo reports non-existent.
• Not returned: the page errored or was truncated; do not delete.
• Reconcile by ownership and liveness, not by scan membership.

Idempotent writes keyed on account key with slot versioning

The same account arrives from both the snapshot and the stream, so the write path must be an upsert keyed by account key with the slot as the version. This also makes replays harmless: a replayed stream message with an older slot is ignored, and a newer slot overwrites.

The code below shows the upsert and the deletion reconciliation pass. It is intentionally small so it can be dropped into an existing store.

async function upsert(store, accountKey, { slot, data }) {
  const existing = await store.get(accountKey);
  if (existing && existing.slot >= slot) return; // stale replay
  await store.put(accountKey, { slot, data });
}

async function reconcileDeletions(connection, store, programId) {
  const known = await store.allKeys();
  for (const key of known) {
    const info = await connection.getAccountInfo(new PublicKey(key));
    if (info === null) {
      await store.delete(key);
    } else if (!info.owner.equals(new PublicKey(programId))) {
      await store.delete(key); // ownership changed
    }
  }
}

module.exports = { upsert, reconcileDeletions };

A runnable Node.js indexer with gap-fill and reconciliation

The full loop combines the snapshot, the persisted cursor, the accountSubscribe stream, the bounded gap-fill re-scan, and the deletion reconciliation pass. Run it against your own endpoint and fill in the results table in the next section.

The stream handler writes through the same upsert, so snapshot and stream converge on one versioned store. The reconciliation pass runs on a schedule and after every restart.

const { Connection, PublicKey } = require('@solana/web3.js');
const { snapshot } = require('./snapshot');
const { upsert, reconcileDeletions } = require('./store');

async function run(connection, programId, store) {
  const { minSlot } = await snapshot(connection, programId, store);

  const subId = connection.onProgramAccountChange(
    new PublicKey(programId),
    async (keyedAccountInfo, context) => {
      await upsert(store, keyedAccountInfo.accountId.toBase58(), {
        slot: context.slot,
        data: keyedAccountInfo.accountInfo.data
      });
    },
    'confirmed'
  );

  const subscriptionSlot = await connection.getSlot('confirmed');
  const gap = await connection.getBlocks(minSlot, subscriptionSlot);
  for (const blockSlot of gap) {
    // bounded re-scan of the gap range
    await snapshotRange(connection, programId, store, blockSlot);
  }

  setInterval(() => reconcileDeletions(connection, store, programId), 60_000);
  return subId;
}

module.exports = { run };

Results table to fill against your own endpoint

Measure against your own endpoint and record the values below. Do not rely on published numbers; the point is to characterize your endpoint's behavior under your workload.

Run the indexer for a fixed window, then fill in each row. The snapshot slot span is the difference between the maximum and minimum context slots across pages. The gap-fill range is the difference between the subscription slot and the snapshot slot.

• Snapshot slot span: max(context.slot) - min(context.slot) across pages.
• Page count: number of getProgramAccounts pages in the snapshot.
• Stream lag: subscription slot minus the slot of the last streamed write.
• Gap-fill range: subscription slot minus snapshot slot.
• Reconciled deletions: number of accounts removed by the reconciliation pass.

Failure modes and troubleshooting

A scan too large for one endpoint's response cap will either error or truncate. An endpoint that truncates the scan without erroring is the most dangerous case, because the snapshot looks complete but is not. Detect it by comparing page counts against a known-good baseline and by checking that the last page is empty.

Pages on a moving chain are expected; the minimum context slot handles them. Subscription drops must be detected and the gap-fill re-run. Rate-limit pressure from a scan meeting a stream on the same key is covered in Solana RPC rate limits and 429 errors.

• Response cap: reduce page size or use dataSlice to shrink payloads.
• Silent truncation: verify the last page is empty and page counts match baseline.
• Subscription drops: re-run the bounded gap-fill from the last recorded slot.
• Rate limits: stagger the scan and the stream, and back off on 429.

Limitations, tradeoffs, and cost model

This design is eventually consistent by construction: the snapshot and the stream converge, but there is always a window where the index lags the chain. It is not a replacement for a purpose-built indexer on very large program account sets, where a full scan is prohibitively expensive and a dedicated ingestion pipeline is warranted.

The cost model is dominated by the full scan and the reconciliation pass. The stream is comparatively cheap. Use RPC pricing to estimate, and consider the API service for managed access. For network context, see Solana.

• Eventually consistent: acceptable for most read paths, not for strict consistency.
• Not a replacement for purpose-built indexers on very large account sets.
• Cost dominated by full scan and reconciliation; stream is cheap.

Next steps and related reading

Start with the prerequisite Solana getProgramAccounts: dataSlice, filters and safe pagination, then extend to Solana getSignaturesForAddress pagination and Solana getBlocks, skipped slots and indexer gap detection.

For account and rent reads, see Reading Solana accounts: rent and token accounts. For endpoint behavior, see the Solana RPC API guide (RPC Assistant) and the OnFinality Learn hub.

• Prerequisite: filter semantics and resumable key-order scans.
• Related: signature pagination, skipped-slot gap detection, account/rent reads.
• Operational: rate limits, RPC pricing, API service, Solana network page.