RPC Node Monitoring: Metrics, Alerts, and Failover

What RPC Node Monitoring Actually Means

RPC node monitoring is the practice of continuously measuring the health, performance, and availability of blockchain JSON-RPC endpoints and the underlying nodes. It answers three questions: Is the endpoint reachable? Is it responding correctly? Is it fast enough for your application?

Monitoring is not just about uptime. A node can be up but stuck behind the chain tip, returning stale data, or rate-limiting your requests. Effective monitoring tracks both the node's internal state (sync status, peer count) and the external service quality (latency, error rate).

This guide focuses on the practical side: what metrics to collect, how to collect them with Prometheus and Grafana, how to set up health checks and alerts, and how to use monitoring to drive failover decisions. It applies to any EVM chain (Ethereum, Polygon, BSC) and Solana, with slight differences in node-specific metrics.

• Key monitoring categories: availability, performance, correctness, and capacity.
• Two layers: node-level metrics (via exporters) and endpoint-level metrics (via synthetic probes).
• Monitoring enables proactive incident response and informs failover logic.

Key Metrics Every RPC Node Should Track

The metrics you collect depend on your chain and your role (node operator vs. application developer). At minimum, you need the following categories.

Availability: Is the endpoint reachable? This is usually a simple TCP or HTTP check. For JSON-RPC, you can send a lightweight method like eth_blockNumber (EVM) or getHealth (Solana) and expect a valid response within a timeout.

Latency: Measure the time to complete a request, typically reported as percentiles: p50, p95, p99. High p99 latency indicates slow responses that can cause timeouts in your application. Use a tool like curl -w to measure from your location, but for continuous monitoring, use a synthetic probe or a Prometheus exporter that records request durations.

Error rate: The percentage of requests that return an error. For JSON-RPC, errors include HTTP 5xx, JSON-RPC error codes (e.g., -32000 server error), and timeouts. A sudden spike in errors often indicates node issues or network problems.

429 rate: The rate of HTTP 429 Too Many Requests responses. This is a sign of rate limiting, either from your provider or from your own node's connection limits. High 429 rates can cause application failures if not handled.

Sync status: For a full node, the node must be synced to the chain tip. For EVM, you can compare the latest block number from the node with a known reference (e.g., a public explorer). For Solana, use getHealth and getSlot to check if the node is behind.

Resource usage: CPU, memory, disk I/O, and network bandwidth. These are critical for node operators to plan capacity and detect issues like disk filling up.

• For EVM: eth_blockNumber, eth_syncing, net_peerCount.
• For Solana: getHealth, getSlot, getClusterNodes.
• Always track both success and failure responses, and record latency percentiles.

How to Collect Metrics with Prometheus

Prometheus is the de facto standard for scraping and storing time-series metrics. To monitor your RPC node, you need an exporter that exposes metrics in Prometheus format. There are several approaches:

Use a node exporter: Many blockchain nodes expose Prometheus metrics natively. For example, Geth (Ethereum) has a --metrics flag that enables an HTTP endpoint. Solana validators expose metrics via solana-metrics (InfluxDB) but you can use a Prometheus exporter like solana-exporter.

Use a generic JSON-RPC exporter: Tools like json-rpc-exporter or prometheus-json-exporter can scrape any JSON-RPC endpoint and convert responses to metrics. This is useful for monitoring the endpoint from the outside, without touching the node.

Write a custom exporter: For full control, you can write a small script that queries the node and exposes metrics. This is common for tracking application-specific metrics like transaction success rate.

Once you have an exporter, configure Prometheus to scrape it at a regular interval (e.g., 15 seconds). Here's a minimal prometheus.yml example:

scrape_configs:
  - job_name: 'rpc-node'
    static_configs:
      - targets: ['localhost:9090']  # exporter address
    metrics_path: /metrics
    scrape_interval: 15s

Setting Up Grafana Dashboards for RPC Health

Grafana visualizes Prometheus data and lets you build dashboards that show the health of your RPC endpoints at a glance. You can create panels for each metric: latency percentiles, error rate, 429 rate, sync status, and resource usage.

A good RPC dashboard includes:

Latency graph: A time series of p50, p95, and p99 latency. Use a heatmap to see the distribution over time.

Error rate panel: A graph showing the percentage of requests that returned errors. Color-code thresholds (e.g., >1% red).

429 rate panel: A graph of 429 responses per second. This helps you detect rate limiting issues.

Sync status panel: For EVM, show eth_syncing result (false if synced). For Solana, show the difference between the node's slot and the cluster's max slot.

Resource usage panels: CPU, memory, disk, and network I/O from the node exporter.

You can also set up alert rules in Grafana to notify you via Slack, email, or PagerDuty when a metric crosses a threshold. For example, alert if p99 latency > 2s for 5 minutes, or if error rate > 5% for 1 minute.

• Use Grafana's alerting to send notifications to your team.
• Create separate dashboards for different chains or environments.
• Share dashboards with your team using Grafana's public dashboards feature.

Health Checks and Synthetic Probes

Health checks are simple, frequent requests that verify an endpoint is alive and responding correctly. They are the foundation of uptime monitoring and failover. You can implement health checks in two ways:

Active health checks: Your monitoring system sends a request to the endpoint at regular intervals (e.g., every 30 seconds) and expects a valid response. For EVM, use eth_blockNumber; for Solana, use getHealth. If the request fails or times out, mark the endpoint as unhealthy.

Passive health checks: Your application monitors the responses it receives during normal operation. If it sees a high error rate or repeated timeouts, it can mark the endpoint as unhealthy. This is useful for failover because it reflects real user experience.

Synthetic probes go a step further by simulating realistic user requests, such as sending a transaction or querying a specific contract. This catches issues that simple health checks miss, like a node that is up but returning incorrect data.

For example, you can use a tool like curl to perform a health check from a cron job:

curl -s -o /dev/null -w "%{http_code} %{time_total}\n" \
  -X POST https://your-rpc-endpoint \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

Alerting on Anomalies: What to Watch For

Alerting is the bridge between monitoring and action. You need to define thresholds that indicate a problem, and then notify the right people. Common alerting rules include:

Endpoint down: The health check fails for more than 1 minute. This triggers immediate investigation.

High latency: p99 latency exceeds 2 seconds for 5 minutes. This may indicate node overload or network issues.

Error rate spike: Error rate exceeds 5% for 1 minute. This could be a node crash, a network partition, or a bug in your application.

429 rate increase: 429 responses exceed a threshold, indicating you are being rate-limited. This may require you to reduce request volume or switch to a different endpoint.

Sync lag: The node is behind the chain tip by more than 10 blocks (EVM) or 100 slots (Solana). This is critical for applications that need up-to-date data.

Resource exhaustion: Disk usage > 90%, memory > 90%, or CPU > 80% for 10 minutes. This helps you plan capacity before the node fails.

When setting thresholds, consider your application's tolerance. A DeFi trading bot may need p99 < 500ms, while a block explorer can tolerate 2s. Use historical data to set realistic baselines.

• Use Prometheus alerting rules or Grafana alerts.
• Include a runbook link in the alert notification.
• Test your alerts by intentionally causing a failure (e.g., stopping the node).

Building High Availability with Failover

Monitoring is only useful if you act on it. High availability (HA) for RPC endpoints means your application can continue to function even if one endpoint fails. The core idea is to have multiple endpoints and a failover mechanism that routes traffic to healthy ones.

Multiple endpoints: Use at least two independent RPC endpoints, ideally from different providers or self-hosted nodes in different regions. This reduces the risk of a single point of failure.

Health-based routing: Your application or a load balancer should periodically check the health of each endpoint and only route traffic to healthy ones. This can be done with a simple script or a service like HAProxy or Envoy.

Automatic failover: When an endpoint becomes unhealthy, the system should automatically switch to the next available endpoint. This requires a health check that runs frequently (e.g., every 10 seconds) and a routing layer that reacts quickly.

Load balancing: Distribute requests across multiple endpoints to avoid overloading any single one. This also improves latency by using the closest endpoint.

Retry logic: In your application, implement retries with exponential backoff when a request fails. This handles transient errors without requiring immediate failover.

For example, you can use a simple Python script to check health and update a DNS record or a load balancer configuration. Or you can use a service like OnFinality's RPC Assistant, which provides managed endpoints with built-in failover and monitoring. See our how to choose an RPC provider for details.

• Always have at least two endpoints from different providers.
• Use health checks to drive failover, not just for alerting.
• Test failover regularly by simulating an endpoint failure.

Common Pitfalls and How to Avoid Them

Even with monitoring in place, there are common mistakes that can undermine your efforts:

Monitoring only from one location: If you monitor from a single region, you may miss issues that affect users in other regions. Use multiple monitoring locations or a service like UptimeRobot.

Ignoring 429 responses: Rate limiting is a common cause of RPC failures. If you see 429s, you need to either reduce your request rate or get a higher quota from your provider.

Not monitoring sync status: A node that is behind the chain tip will return stale data, which can be worse than downtime. Always monitor sync status.

Alert fatigue: Setting thresholds too low can cause too many alerts, leading to alert fatigue. Use severity levels and only alert on actionable issues.

Not testing failover: If you have failover in place but never test it, it may not work when you need it. Regularly simulate failures to ensure your system behaves as expected.

Forgetting about security: Monitoring endpoints can expose sensitive information. Ensure your monitoring dashboards are protected and that you don't log sensitive request data.

• Use multiple monitoring locations for global coverage.
• Set up separate alerts for different severity levels.
• Document your failover procedures and run drills.

Next Steps: From Monitoring to Managed Solutions

Once you have a monitoring and failover setup, you can consider whether to build and maintain your own infrastructure or use a managed RPC provider. Managed providers like OnFinality offer built-in monitoring, high availability, and failover, saving you the operational overhead.

If you choose to self-host, you'll need to handle node upgrades, security patches, and scaling. This can be time-consuming but gives you full control. If you prefer to focus on your application, a managed provider can be a better choice.

To learn more about choosing between self-hosted and managed RPC, see our guide on how to choose an RPC provider. For specific chain details, check our Solana RPC endpoints and Ethereum network page.

If you're experiencing latency issues, our article on reducing RPC latency can help. For timeout errors, see how to fix RPC timeout errors. And if you're comparing public vs. dedicated endpoints, read public RPC endpoints vs. dedicated.

Finally, consider using OnFinality's API service for a fully managed solution with built-in monitoring and failover. Our pricing is transparent, and you can start with a free tier.

• Evaluate the cost of self-hosting vs. managed services.
• Leverage existing tools like Prometheus and Grafana for self-hosted setups.
• Explore OnFinality's managed RPC for production workloads.