Sui Indexer: How to Index Sui Data Without Running Your Own Pipeline

A Sui indexer is the component that turns Sui's raw on-chain output into something your application can query quickly: balances, object histories, event streams, and aggregate metrics. If you are building a wallet, an explorer, a DeFi dashboard, or a backend that needs more than a single object lookup, you will eventually need one.

The catch is that indexing is a data-engineering problem, not just an RPC problem. You need a reliable source of Sui data, a way to process it without falling behind, and a storage layer that answers queries fast. This article walks through how that pipeline is built, where it usually breaks, and how to decide whether to run it yourself or lean on managed infrastructure.

What a Sui indexer actually does

Sui produces data in a few distinct shapes, and a good indexer understands the difference:

• Checkpoints — ordered batches of transactions. Checkpoints are the backbone of any Sui indexer because they give you a deterministic, gap-free sequence to process.
• Transactions and effects — the inputs and the resulting state changes (object mutations, created and deleted objects, balance changes).
• Events — structured messages emitted by Move packages. Events are the easiest hook for application-specific indexing because you can filter by package and event type.
• Objects — the current state of owned and shared objects. Object state is what most read paths actually need.

A typical Sui indexer reads checkpoints in order, extracts the transactions and events it cares about, transforms them into a schema that suits your queries, and writes the result to a database. The query layer (REST, GraphQL, or SQL) then serves your application.

The important design choice is whether you index everything or only what you need. Full-history indexing is expensive and slow to backfill. Targeted indexing — a specific package, a set of event types, or a subset of objects — is usually what production apps actually need.

Build vs buy: where the real cost sits

Before writing any indexing code, decide who runs the pipeline. The tradeoff is rarely about the query layer; it is about the data source and the operational load.

Most teams land in the middle: they run their own transformation and storage but rely on a managed provider for the Sui data source. That is where a reliable RPC and dedicated node setup matters, because a lagging or rate-limited source will stall the whole pipeline regardless of how good your processing code is.

Choosing your Sui data source

Your indexer is only as fresh as the endpoint feeding it. There are three common access patterns on Sui, and they suit different stages of a pipeline.

JSON-RPC for object and transaction reads. Good for targeted lookups, transaction submission, and lightweight polling. It is the simplest integration and works well for low-volume or request-driven indexing.

gRPC streaming for continuous ingestion. When you need to follow checkpoints or subscribe to a stream of updates, a streaming interface is far more efficient than polling. This is the pattern most production indexers converge on, because it keeps you close to the chain head without hammering a request/response endpoint.

Bulk historical access for backfill. Backfilling months of history over a request/response endpoint is slow. If your provider offers bulk or archive access, use it for the initial load and reserve streaming for staying current.

A practical split looks like this: backfill from an archive-capable source, then switch to streaming for the live tail, and use JSON-RPC for ad-hoc reads your application makes at request time.

A minimal ingestion loop

The shape of a Sui indexer is usually a checkpoint cursor plus a transform step. The example below uses a JSON-RPC call to fetch a checkpoint and advance a cursor. Treat it as a sketch, not a complete indexer — the point is the control flow.

curl -s https://sui.api.onfinality.io/public \
  -H 'Content-Type: application/json' \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "sui_getCheckpoint",
    "params": ["1000000"]
  }'

In a real pipeline you would wrap this in a loop that persists the last processed checkpoint, handles gaps, and writes transformed rows in batches. Two details matter more than the request itself:

1. Persist the cursor after the write, not before. If you advance the cursor and then crash before writing, you silently skip data.
2. Make the transform idempotent. Reprocessing a checkpoint should overwrite the same rows, not duplicate them. A unique key on (checkpoint, tx_digest, event_index) is the usual fix.

For continuous ingestion, replace the polling loop with a streaming subscription and keep the same cursor and idempotency rules.

Production readiness checklist

Once the pipeline runs locally, the questions shift to reliability. Work through these before you point real traffic at it:

• Lag monitoring. Track the gap between the chain head and your last processed checkpoint. Alert on sustained lag, not on a single spike.
• Gap and reorg handling. Decide how you detect a missed checkpoint and how you re-fetch it. Sui's checkpoint model is ordered, which makes gap detection tractable — use it.
• Backfill strategy. Know how long a full backfill takes and whether you can run it in parallel with live ingestion.
• Endpoint failover. A single endpoint is a single point of failure. Have a fallback and a way to detect a degraded source.
• Storage growth. Event and transaction tables grow fast. Plan retention and partitioning before the database becomes the bottleneck.
• Query latency budget. Separate write-optimized and read-optimized paths if your application queries need sub-second responses.

If any of these are unowned, that is the gap to close first.

Where RPC infrastructure fits

The indexer is your code; the data source is infrastructure. Keeping those separate is what lets you scale one without rewriting the other.

A managed Sui RPC endpoint handles the node side — sync, upgrades, and availability — so your pipeline only has to deal with data. For higher-volume ingestion, a dedicated Sui node gives you isolated capacity instead of sharing a public pool, which matters when your ingestion loop is constant rather than bursty. You can review the Sui network page for supported transports and connection details, and compare RPC pricing if you are sizing for sustained throughput.

The practical rule: if your indexer runs continuously and a stalled source means stale data for users, treat the data source as production infrastructure, not a free endpoint.

Common failure modes

Most Sui indexer incidents fall into a small set of patterns. Recognizing them shortens debugging considerably.

None of these are exotic. They are the normal costs of running a data pipeline, and they are all easier to handle when the data source is stable.

Key Takeaways

• A Sui indexer reads checkpoints, transactions, events, and objects, then reshapes them into a queryable schema.
• Targeted indexing (specific packages or event types) is usually cheaper and faster than full-history indexing.
• Streaming is the right pattern for the live tail; bulk or archive access is the right pattern for backfill.
• Persist your cursor after the write and make transforms idempotent — these two rules prevent most data-integrity bugs.
• The data source is production infrastructure. A managed Sui RPC or dedicated node removes node operations from your critical path.
• Monitor lag, plan retention, and always have a failover endpoint.

Frequently Asked Questions

Do I need a separate indexer if I already have an RPC endpoint?

Usually yes, for anything beyond single-object lookups. RPC answers point queries well, but it does not maintain the aggregates, histories, or filtered event views that dashboards and analytics need. The indexer builds and stores those; the RPC is where it gets its data.

Should I index every checkpoint or only what I need?

Start with what you need. Filtering by package and event type keeps storage and backfill time manageable. You can always widen the scope later, but shrinking a full-history index after the fact is painful.

How do I keep a Sui indexer from falling behind?

Use a streaming interface for the live tail instead of polling, monitor the gap between chain head and your cursor, and make sure your data source has enough capacity for sustained ingestion. If lag grows steadily rather than spiking, the source is usually the constraint.

Can I run an indexer against a public endpoint?

For development and low-volume testing, yes. For continuous production ingestion, a public endpoint is usually the wrong tool because shared capacity and rate limits will eventually throttle you. A dedicated node gives your pipeline predictable capacity.

What is the difference between an indexer and an RPC provider?

An RPC provider runs nodes and exposes chain access. An indexer is application logic that consumes that access and produces a queryable dataset. They are complementary: the provider supplies data, the indexer shapes and stores it.

Next steps

If you are starting a Sui indexer, begin with a narrow scope — one package or a small set of event types — and get the cursor and idempotency rules right before scaling. Then decide whether your data source can sustain continuous ingestion. If it cannot, look at a managed Sui RPC or a dedicated node so your pipeline is not competing for shared capacity. You can review supported RPC networks to see where Sui sits alongside the rest of the infrastructure OnFinality runs, and start from the Sui network page when you are ready to connect.