Iterating Substrate Storage with state_getKeysPaged over RPC

Method Signature and Pagination Semantics

The state_getKeysPaged method accepts four parameters: prefix (hex string), count (integer), startKey (hex string, optional), and at (block hash, optional). It returns an array of up to count hex-encoded storage keys that begin with prefix, starting the page at startKey (exclusive) and evaluated at block at. If startKey is omitted, iteration begins at the first key matching the prefix. The method is documented in the polkadot.js Substrate JSON-RPC reference and is part of the Substrate JSON-RPC API.

Pagination is driven by feeding the last returned key back in as startKey on the next call. You continue until the result is shorter than count or empty. This is a cursor-free pattern: the node does not maintain server-side state between calls, so you must manage the iteration loop yourself. Because the method is prefix-scoped, it only returns keys that share the exact byte prefix you provide. This makes it ideal for iterating a single storage map, but also means an incorrect prefix will either return nothing or spill into unrelated storage items.

The at parameter pins the query to a specific block. If omitted, the node evaluates at the latest block, which can change between calls. For a consistent iteration across multiple pages, you should always pass the same block hash. This is a documented behavior: the state read is only consistent at the block you pass. For more on block-specific reads, see Polkadot RPC extrinsics and events at a block.

• prefix: hex-encoded storage key prefix (e.g., twox128 hash of pallet and item).
• count: maximum number of keys to return per page.
• startKey: hex-encoded key to start after (exclusive); omit for first page.
• at: block hash to evaluate at; omit for latest (not recommended for iteration).

Constructing the Correct Storage Key Prefix

A Substrate storage key is built from the twox128 hashes of the pallet name and the storage item name, concatenated. For storage maps, the map key is then SCALE-encoded and appended. The prefix you pass to state_getKeysPaged must be the exact bytes that precede the map key. For a map like System.Account, the prefix is twox128('System') ++ twox128('Account'). This is 32 bytes (16 + 16). Passing a shorter prefix (e.g., only the pallet hash) will iterate all storage items in that pallet, which may be unintended.

The Substrate storage documentation explains that twox128 is a non-cryptographic hash used for its speed and low collision probability. The pallet and item names are ASCII strings. You can compute these hashes using @polkadot/util-crypto or any twox128 implementation. The resulting prefix is a hex string without the 0x prefix when passed to the RPC method? Actually, the RPC expects a hex string with 0x prefix. Always verify the prefix length: for a map, it should be 32 bytes (64 hex characters plus 0x).

If you accidentally use a broader prefix, you may retrieve keys from other storage items in the same pallet. For example, using only twox128('System') would return keys for Account, Events, BlockHash, and others. This is rarely desired. To scope to a single map, always concatenate both hashes. You can decode the runtime metadata to confirm the exact storage item names and their prefixes; see Substrate state_getMetadata and runtime versions.

const { xxhashAsHex } = require('@polkadot/util-crypto');

function storageMapPrefix(pallet, item) {
  const palletHash = xxhashAsHex(pallet, 128);
  const itemHash = xxhashAsHex(item, 128);
  return palletHash + itemHash.slice(2); // remove 0x from second hash
}

const prefix = storageMapPrefix('System', 'Account');
console.log('Prefix:', prefix); // 0x... (32 bytes)

Driving Pagination Without Gaps or Loops

To iterate a full map, start with startKey omitted (or null). Call state_getKeysPaged(prefix, count, startKey, at). If the returned array length equals count, set startKey to the last key in the array and repeat. If the length is less than count, you have reached the end. If the array is empty, the map is empty or the prefix is wrong. This loop is safe because startKey is exclusive: the node returns keys strictly after the given key in its internal iteration order.

Crucially, you must not assume that keys are returned in lexicographic order. The Substrate storage trie is a Merkle Patricia Trie, and iteration order is determined by the trie's structure, which is not guaranteed to be sorted by the raw key bytes. A client-side cursor that assumes monotonic ordering (e.g., comparing keys with > or <) will be buggy. Instead, always use the last returned key as the next startKey, regardless of its value. This is the documented pattern in the polkadot.js reference.

If you need to resume iteration later, you can store the last key and the block hash. However, if the chain has advanced, the state at that block may no longer be available on pruned nodes. For long iterations, consider pinning to a recent finalized block and completing the iteration within the node's pruning window. For more on state queries, see Polkadot state_queryStorageAt storage changes.

• Start with startKey omitted or null.
• Loop while returned length === count.
• Set startKey to the last key of the previous page.
• Stop when length < count or empty.
• Never sort or compare keys client-side; rely on node order.

Decoding Returned Keys and Reading Values

Each returned key is a hex string. To decode the map key, you must strip the known prefix (the 32-byte pallet+item hash) and then SCALE-decode the remaining bytes according to the map's key type. The key type is defined in the runtime metadata. For example, System.Account uses AccountId32 as the key, which is 32 bytes. After stripping the prefix, you have the SCALE-encoded key. You can use @polkadot/types to create a type from metadata and decode it.

Once you have the decoded keys, you can read the corresponding values. You can call state_getStorage for each key individually, but that is inefficient for large maps. Instead, use state_queryStorageAt with an array of keys and a block hash. This method returns the values for all keys in a single round trip, which is much faster and reduces the number of RPC calls. The method is documented in the polkadot.js reference.

When batching, be mindful of the maximum request size. Some providers limit the number of keys per state_queryStorageAt call. A common practice is to batch in chunks of 100–500 keys. You can also use state_getStorage for individual reads if you only need a few values. For a deeper dive into batch reads, see Polkadot state_queryStorageAt storage changes.

const { ApiPromise, WsProvider } = require('@polkadot/api');
const { xxhashAsHex } = require('@polkadot/util-crypto');

async function iterateMap(pallet, item, pageSize = 100) {
  const api = await ApiPromise.create({ provider: new WsProvider('wss://rpc.polkadot.io') });
  const prefix = xxhashAsHex(pallet, 128) + xxhashAsHex(item, 128).slice(2);
  const at = (await api.rpc.chain.getHeader()).hash;
  let startKey = null;
  let allKeys = [];
  let page;
  do {
    page = await api.rpc.state.getKeysPaged(prefix, pageSize, startKey, at);
    allKeys.push(...page);
    if (page.length > 0) startKey = page[page.length - 1];
  } while (page.length === pageSize);
  console.log(`Total keys: ${allKeys.length}`);
  // Batch read values
  const values = await api.rpc.state.queryStorageAt(allKeys, at);
  console.log(`Values fetched: ${values.length}`);
  await api.disconnect();
}

iterateMap('System', 'Account').catch(console.error);

Storage Kind and Node Version Differences

The state_getKeysPaged method historically accepted a storageKind parameter (e.g., 'value' for the main trie, 'child' for child tries). In modern Substrate nodes, this parameter is often deprecated or replaced by separate methods for child storage. The exact behavior is documented / varies by provider and node version. You should consult your node's RPC documentation or the polkadot.js reference for the version you are targeting.

If you are working with child tries (e.g., contract storage), you may need to use state_getChildKeysPaged or similar. The prefix semantics remain the same, but the trie is different. Always verify the method signature against your node's metadata. For runtime metadata decoding, see Substrate state_getMetadata and runtime versions.

When in doubt, test with a small page size and a known map. If the method returns an error about unknown parameter, your node may not support storageKind. In that case, omit it and use the default (value trie).

Scoping Iteration to a Single Map vs. Broader Prefixes

The prefix you pass determines the scope. A full twox128 pallet+item prefix scopes to exactly one storage map. A pallet-only prefix (twox128 of pallet name) scopes to all storage items in that pallet. A shorter prefix (e.g., first 16 bytes) may spill into other pallets if the hash collides, though twox128 collisions are extremely unlikely. The safest approach is to always use the full 32-byte prefix for a specific map.

If you intentionally want to iterate all storage in a pallet, you can use the pallet hash as prefix. However, be aware that the returned keys will include different storage items, and you will need to decode each key's item name from the prefix to know which map it belongs to. This is rarely necessary and can be error-prone.

To verify your prefix, you can call state_getKeysPaged with a small count and inspect the returned keys. They should all start with your prefix. If they don't, your prefix is wrong. You can also use state_getMetadata to list all storage items and their prefixes.

• Full map prefix: twox128(pallet) ++ twox128(item) — 32 bytes.
• Pallet-wide prefix: twox128(pallet) — 16 bytes.
• Always verify returned keys start with your prefix.
• Use metadata to confirm exact item names.

Reproducible Method to Detect Incomplete Iteration

To ensure your iteration is complete, you need an independent way to verify the total number of keys. One method is to compare the paged key count against a trusted total from another source, such as a block explorer or a descendant count if the node exposes one. For example, System.Account has a known number of accounts that can be cross-checked with a block explorer. If your count is significantly lower, your iteration may have stopped early due to a bug or a rate limit.

Another method is to run the iteration twice with different page sizes (e.g., 100 and 500) and compare the resulting key sets. If they differ, your iteration logic is flawed. You can also compute a checksum of all keys (e.g., SHA-256 of the concatenated sorted keys) and compare across runs. This is a reproducible way to detect gaps or duplicates.

For a more rigorous check, you can use the state_getKeysPaged method with a very large count (e.g., 10000) on a small map and compare the result with your paginated result. If they match, your pagination is correct. Note that large count values may be rejected by some providers, so use with caution.

• Compare paged count against a trusted total (e.g., block explorer).
• Run with different page sizes and compare key sets.
• Compute a checksum of all keys and compare across runs.
• Use a single large page as a reference for small maps.

Results Table: Measuring Against Your Own Endpoint

Because latency, throughput, and rate limits vary by provider, you should measure the performance of state_getKeysPaged against your own endpoint. The following table provides a template for recording your measurements. Fill it in with your own results. Do not rely on generic benchmarks; your network conditions and provider limits matter.

To collect data, run the Node.js example above with different page sizes and record the time taken for each full iteration. Use a stopwatch or console.time. Also record the number of RPC calls made (which equals the number of pages). If you encounter HTTP 429 errors, note the page size and the time at which the error occurred. This will help you tune your page size and backoff strategy.

For a production system, consider using a dedicated API service or RPC pricing plan that matches your expected request volume. OnFinality's Polkadot network endpoints support state_getKeysPaged and can be used for testing. Always respect rate limits and implement exponential backoff.

• Page size: number of keys per request.
• Total keys: total number of keys in the map.
• Number of pages: total RPC calls.
• Total time: wall-clock time for full iteration.
• Errors: any 429 or timeout errors encountered.

Limitations and Tradeoffs of state_getKeysPaged

The startKey exclusivity and ordering semantics are documented / varies by client. While the general pattern is consistent, some node implementations may have subtle differences in how they handle the exclusive start or the iteration order. Always test against your target node. The polkadot.js reference is the authoritative source for the JavaScript API, but the underlying RPC behavior is defined by the Substrate node.

Iterating a large map with many small pages is slow. Each page is a separate round trip, so a rate-limited endpoint will return HTTP 429 if you exceed its limits. This is a fundamental tradeoff: larger page sizes reduce the number of round trips but increase the response size and memory usage. You must find a balance that works for your provider and network.

The state read is only consistent at the block you pass. If you omit at or use different blocks across pages, you may mix states from different blocks, leading to an inconsistent view. Always pin at to a single block hash for the entire iteration. Additionally, an unstable prefix or a runtime upgrade that renames a pallet or storage item will invalidate your prefix entirely. You must re-derive the prefix from the new metadata after an upgrade. For more on runtime versions, see Substrate state_getMetadata and runtime versions.

• Ordering not guaranteed; do not assume lexicographic order.
• Many small pages = many round trips = rate limit risk.
• State consistency requires pinning at to one block.
• Runtime upgrades can change prefixes; re-derive from metadata.
• Large page sizes may hit response size limits.

Troubleshooting Common Iteration Failures

If you receive an empty array on the first call, check your prefix. It may be incorrect or the map may be empty. Use state_getMetadata to verify the pallet and item names. If you receive an error about invalid parameters, ensure your hex strings are properly formatted with 0x prefix and that count is a positive integer.

If your iteration stops early, check if you are comparing the page length to count correctly. A common bug is to stop when the page is empty, but if the map size is an exact multiple of count, the last page will be full and the next call will return empty. You should stop when the page length is less than count, not when it is zero. Also, ensure you are updating startKey to the last key of the previous page.

If you encounter HTTP 429 errors, reduce your page size or add a delay between requests. Implement exponential backoff. If you are using a shared endpoint, consider upgrading to a dedicated plan. For more on RPC best practices, see the Polkadot RPC guide (RPC Assistant).

• Empty first page: check prefix and map existence.
• Early stop: ensure loop condition is length === count.
• 429 errors: reduce page size, add backoff, or upgrade plan.
• Inconsistent results: pin at to a single block hash.
• Prefix invalid after upgrade: re-derive from new metadata.

Next Steps: Integrating Iteration into Your Application

Now that you understand the mechanics, you can integrate state_getKeysPaged into your application. Start by writing a small script to iterate a known map, such as System.Account, and verify the count against a block explorer. Then, adapt the pattern to your specific storage map. Use the OnFinality Learn hub for more guides on Substrate RPC methods.

For production use, consider caching the results and updating incrementally. Since the state is only consistent at a block, you can iterate at each new finalized block and diff the changes. This is more efficient than re-iterating the entire map every time. You can also use state_queryStorageAt to fetch values for changed keys. For more on storage changes, see Polkadot state_queryStorageAt storage changes.

Finally, always monitor your RPC usage and error rates. If you are building a high-throughput application, consider using a dedicated API service or reviewing RPC pricing to ensure you have enough capacity. OnFinality provides reliable Polkadot network endpoints that support these methods.

• Test with a known map and verify count.
• Cache results and update incrementally at new blocks.
• Monitor RPC usage and error rates.
• Use dedicated endpoints for high throughput.
• Refer to the Polkadot RPC guide (RPC Assistant) for more.