Polkadot system_health and system_syncState: Node Health over RPC

The Substrate health surface and its three authoritative methods

Polkadot exposes a native health surface over JSON-RPC that predates and is independent of any EVM compatibility layer. The Substrate JSON-RPC specification defines three methods that matter here: system_health, system_syncState and system_peers. Each is authoritative for a different fact, and none of them alone is a complete health verdict.

system_health is authoritative for peer count and sync intent. It returns peers (the number of connected peers), isSyncing (a boolean) and shouldHavePeers (a boolean). system_syncState is authoritative for block progress: startingBlock, currentBlock and highestBlock. system_peers is authoritative for per-peer detail, including the peer id, role, best hash and best number, which lets you distinguish 'connected to nobody' from 'connected to peers that are themselves behind'.

Before trusting any of those fields, read the sanity preamble: system_name, system_version and system_chain. These tell you what software and which chain you are actually talking to. A probe that assumes Polkadot but lands on a testnet or a differently configured node will produce a verdict about the wrong system. The typed accessors for these calls are documented in the Polkadot-JS API documentation, which is the reference used by the example client below.

• system_health: peers, isSyncing, shouldHavePeers — peer count and sync intent.
• system_syncState: startingBlock, currentBlock, highestBlock — block progress.
• system_peers: per-peer role, best hash and best number — peer quality.
• system_name / system_version / system_chain — identity sanity preamble.

Raw JSON-RPC calls for the same health fields

The typed client is convenient, but the underlying transport is plain JSON-RPC 2.0, so the same fields can be read with a single curl request per method. This is useful when you want to confirm what the library is actually sending, when you are debugging a provider that rewrites responses, or when you are writing a probe in a language without a Substrate client.

The snippet below issues the three health calls plus chain_getHeader against a local node. Each request is a standard JSON-RPC envelope with a method name and an empty params array; the response carries the result object described in the Substrate JSON-RPC specification. Run the calls separately so a failure in one method does not hide the others, and compare the returned currentBlock against the header number to derive the same lag delta the Node.js probe computes.

# system_health — peers, isSyncing, shouldHavePeers
curl -s -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"system_health","params":[]}' \
  http://127.0.0.1:9933

# system_syncState — startingBlock, currentBlock, highestBlock
curl -s -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"system_syncState","params":[]}' \
  http://127.0.0.1:9933

# system_peers — per-peer role, best hash and best number
curl -s -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"system_peers","params":[]}' \
  http://127.0.0.1:9933

# chain_getHeader — local reference head for the lag delta
curl -s -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"chain_getHeader","params":[]}' \
  http://127.0.0.1:9933

Why isSyncing=false is not liveness

The most common production mistake is treating system_health.isSyncing=false as proof that a node is healthy. It is not. isSyncing reflects whether the node believes it is actively catching up; a node that has stalled — for example because it lost its peers, hit a disk problem, or is wedged on a bad block — can report isSyncing=false while its currentBlock stops advancing. The flag describes intent, not progress.

The reliable liveness signal is a lag delta: compare system_syncState.currentBlock against a reference head. You can obtain the reference head locally with chain_getHeader (the header carries the block number), or externally from a second, independently operated endpoint. The difference between the reference head and currentBlock is your lag. A node whose lag is stable and small is keeping up; a node whose lag grows monotonically is falling behind even if isSyncing is false.

Peer count closes the remaining gap. A node can be fully synced and still be partitioned, serving stale reads because it is no longer receiving new blocks. Reading system_health.peers catches that case: a synced node with zero peers is not a healthy read target. This is the same class of problem described in Detecting RPC node head lag and stale responses, but expressed through Substrate fields rather than EVM block numbers.

• isSyncing=false means 'not currently catching up', not 'advancing'.
• Lag delta = reference head − system_syncState.currentBlock.
• A growing lag delta indicates a node falling behind.
• peers=0 on a synced node indicates a partition, not health.

Substrate-specific caveats: shouldHavePeers, stale highestBlock and node modes

shouldHavePeers has precise semantics that are easy to misread. It is false only for genuinely peerless roles, such as a light client, or a node configured in a way that does not participate in the peer-to-peer network. For validators and parachain collators it is true, because those roles are expected to maintain peers. If you see shouldHavePeers=false on a node you expected to be a full node, treat it as a configuration signal, not a transient error.

highestBlock from system_syncState can be stale. It represents the node's view of the network's best block, which is derived from its peers; if the node is partitioned or its peers are behind, highestBlock understates the true network head. That is exactly why the lag delta should be computed against an independent reference head rather than against highestBlock alone. Using highestBlock as both the target and the current value hides the very lag you are trying to detect.

Node mode changes your read guarantees. A full node prunes historical state, so reads of old state may fail or be unavailable; an archive node retains historical state and can serve those reads. The Polkadot documentation on node operation and synchronization modes is authoritative for what 'syncing', 'full' and 'archive' mean. Your health probe should record the mode it expects, because a full node that is perfectly healthy will still fail historical state queries that an archive node would serve.

• shouldHavePeers=false: light client or peerless configuration.
• shouldHavePeers=true: validators and parachain collators.
• highestBlock is peer-derived and can understate the true head.
• Full nodes prune state; archive nodes retain it — different read guarantees.

A runnable Node.js probe returning {healthy, reason}

The example below uses @polkadot/api, whose typed system.* accessors are documented in the Polkadot-JS API documentation. It reads the identity preamble, calls system.health(), system.syncState() and system.peers(), then reads a local reference head with chain.getHeader(). It computes the lag delta and returns a structured verdict. The JSON-RPC 2.0 Specification governs the request/response envelope that the library emits underneath.

The probe is deliberately conservative: it fails closed on missing fields, because a provider-managed endpoint may hide or omit some system fields. Treat a missing field as 'unknown', not as 'healthy'. Run it against your own endpoint and record the output in the results table in the next section.

// probe.mjs — run with: node probe.mjs wss://your-endpoint
import { ApiPromise, WsProvider } from '@polkadot/api';

const endpoint = process.argv[2];
if (!endpoint) { console.error('usage: node probe.mjs <ws-endpoint>'); process.exit(2); }

const MAX_LAG = 5;        // blocks of tolerance before flagging lag
const MIN_PEERS = 1;      // a synced node with 0 peers is partitioned

async function probe(endpoint) {
  const api = await ApiPromise.create({ provider: new WsProvider(endpoint) });
  try {
    // 1. Identity sanity preamble
    const [name, version, chain] = await Promise.all([
      api.rpc.system.name(),
      api.rpc.system.version(),
      api.rpc.system.chain(),
    ]);

    // 2. Native health surface
    const health = await api.rpc.system.health();
    const sync = await api.rpc.system.syncState();
    const peers = await api.rpc.system.peers();

    // 3. Local reference head via chain_getHeader
    const header = await api.rpc.chain.getHeader();
    const referenceHead = header.number.toNumber();
    const currentBlock = sync.currentBlock.toNumber();
    const lag = referenceHead - currentBlock;

    const peerCount = health.peers.toNumber();
    const isSyncing = health.isSyncing.valueOf();
    const shouldHavePeers = health.shouldHavePeers.valueOf();

    let healthy = true;
    let reason = 'ok';

    if (lag > MAX_LAG) { healthy = false; reason = `lag=${lag} exceeds ${MAX_LAG}`; }
    else if (peerCount < MIN_PEERS && shouldHavePeers) {
      healthy = false; reason = `peers=${peerCount} but shouldHavePeers=true`;
    } else if (isSyncing && lag > MAX_LAG) {
      healthy = false; reason = 'syncing and behind reference head';
    }

    return {
      healthy, reason,
      identity: { name, version, chain: chain.toString() },
      health: { peers: peerCount, isSyncing, shouldHavePeers },
      sync: {
        startingBlock: sync.startingBlock.toNumber(),
        currentBlock,
        highestBlock: sync.highestBlock.toNumber(),
      },
      referenceHead,
      lag,
      peerSample: peers.slice(0, 3).map((p) => ({
        peerId: p.peerId.toString(),
        role: p.role.toString(),
        bestNumber: p.bestNumber.toNumber(),
      })),
    };
  } finally {
    await api.disconnect();
  }
}

probe(endpoint)
  .then((r) => { console.log(JSON.stringify(r, null, 2)); process.exit(r.healthy ? 0 : 1); })
  .catch((e) => { console.error('probe failed:', e.message); process.exit(3); });

Results table to fill against your own endpoint

Run the probe above against your own endpoint at several times of day and record the values below. The goal is not a single reading but a baseline: you want to know what normal lag and peer count look like for your endpoint so that an alert threshold is meaningful. Do not copy thresholds from another provider; measure your own.

Fill one row per run. If a field is missing, write 'unknown' rather than guessing, and note it — a provider-managed endpoint that hides system fields changes what you can assert.

• Timestamp | Endpoint | chain | version | peers | isSyncing | shouldHavePeers | currentBlock | highestBlock | referenceHead | lag | healthy | reason
• Example row: 2026-09-20T00:00Z | wss://… | Polkadot | <version> | <n> | false | true | <n> | <n> | <n> | <n> | true | ok
• Repeat at least three times across a 24-hour window to see lag variance.
• Record the node mode (full or archive) alongside the endpoint.

Composing the composite verdict

A single method cannot produce a trustworthy verdict, so compose them in a fixed order. First, confirm identity with system_name, system_version and system_chain. Second, read system_syncState and compute lag against a reference head. Third, read system_health.peers and shouldHavePeers to detect partition. Fourth, sample system_peers to see whether your peers are themselves near the head.

The decision logic is small: unhealthy if lag exceeds your measured tolerance; unhealthy if shouldHavePeers is true and peers is zero; unhealthy if isSyncing is true and lag is growing across consecutive probes. Everything else is healthy. This mirrors the chain-agnostic approach in RPC node monitoring: metrics, alerts and failover, but the fields are Substrate-native rather than Prometheus counters.

• Identity → syncState lag → health peers → peers sample.
• Fail on lag over tolerance, on partition, or on growing lag while syncing.
• Keep the verdict as {healthy, reason} so callers can log a cause.

Gating architecture: interval probe, health endpoint and failover

In production, run the probe on an interval and expose the verdict on an internal health endpoint. Your application should read that endpoint before routing traffic, and drain the endpoint when the verdict flips to unhealthy. This is the same gating pattern used for EVM endpoints, but the trigger fields are Substrate's lag and peer count rather than EVM block height.

Wire failover so that a drain moves traffic to a second endpoint, then re-probe the drained endpoint before returning it to the pool. Because a node can recover its peer count quickly but still be behind, require two consecutive healthy probes before re-admitting an endpoint. The latency characteristics of the endpoints you are choosing between are covered separately in Polkadot RPC latency: measuring and optimizing.

• Interval probe writes {healthy, reason} to an internal health endpoint.
• Application reads health before routing; drain on unhealthy.
• Failover to a second endpoint; require two healthy probes to re-admit.
• Log the reason string so incidents are diagnosable.

Failure modes and troubleshooting checklist

When the probe reports unhealthy, work through the cause rather than restarting blindly. A lag that grows while peers is healthy usually means the node is CPU- or disk-bound. A lag that grows while peers is zero means a network partition. A node that reports isSyncing=true for a long time is genuinely catching up, which is expected after downtime but not during steady state.

If the probe itself fails to connect, that is a transport problem, not a health verdict — see Polkadot RPC timeout errors for that class of failure. If reads of finalized state behave oddly, check Polkadot finality: GRANDPA justifications and finalized head, because finality and head progress are related but distinct signals.

• Growing lag + healthy peers: suspect resource saturation on the node.
• Growing lag + zero peers: suspect network partition.
• Persistent isSyncing=true: node is catching up after downtime.
• Probe connection failure: transport issue, not a health verdict.
• Missing system fields: provider-managed endpoint hides them; mark unknown.

Limitations and tradeoffs

The native health surface is not free of tradeoffs. Probing costs a round trip per method, so a four-call probe is four requests; batch or reduce frequency if you are probing many endpoints. Some nodes require the RPC server to be exposed externally (the rpc-external style flag) before these methods are reachable at all, which is a deployment decision with security implications.

Provider-managed endpoints may hide or omit some system fields, which means your probe must treat missing data as unknown rather than healthy. Finally, the health surface tells you about the node, not about the correctness of the data it returns; a node can be healthy and still be on a fork you do not want. For chain-level guarantees, combine this probe with finality checks.

• Each method is a round trip; batch or reduce probe frequency.
• External RPC exposure may be required for these methods to be reachable.
• Provider-managed endpoints can hide fields — treat missing as unknown.
• Health is not correctness; pair with finality checks.

Next steps and related reading

Start by running the probe against a Polkadot endpoint and filling the results table for your own environment. If you need an endpoint to test against, see the Polkadot network page and the Polkadot RPC guide for connection details. For pricing and service options, review RPC pricing and the API service.

To go deeper, read the chain-agnostic monitoring patterns in RPC node monitoring: metrics, alerts and failover, the EVM-framed lag discussion in Detecting RPC node head lag and stale responses, and the latency measurement method in Polkadot RPC latency: measuring and optimizing. The OnFinality Learn hub collects the rest of the infrastructure series.

• Run the probe, fill the table, set thresholds from your own baseline.
• Connect via the Polkadot network page.
• Compare with RPC node monitoring.
• Review RPC pricing and the API service.