[sui-indexer] Issue with starting sui-indexer

[alexandrurosca]

Hello,

I'm trying to start the sui-indexer but I got some issues, everything went well until checkpoint 6231247 but now is stuck with the following error:

ERROR commit_checkpoints{first=6231247 last=6231346}: sui_indexer::store: Error with persisting data into DB: PostgresError(DatabaseError(UnableToSendCommand, "error encoding message to server: value too large to transmit")), retrying...

Another option would be to get a snapshot for the indexer but I cannot find it anywhere:

sui-indexer --database-url <DATABASE_URL> restore ...

Does anyone one where can I get a snapshot for the indexer, because going from block 1 takes very long time.

Info:

start cmd:

sui-indexer --pool-size 50  --db-url <DATABASE_URL> indexer --rpc-client-url http://localhost:9000 --remote-store-url https://checkpoints.mainnet.sui.io --checkpoint-download-queue-size 500 --epochs-to-keep 10

sui-indexer binary was build on:

commit 4b0e0aceee7c3a8382b6798b8db084a3925aafbf (HEAD -> main, tag: sui_v1.35.0_1727196238_ci, tag: sui_v1.35.0_1727190285_rel_notes, origin/main, origin/HEAD)

[gegaowp]

Hi @alexandrurosca, thanks for the report!

get a snapshot for the indexer but I cannot find it anywhere:

this is still work in progress and has not been officially rolled out, stay tuned and it will be out in one of the coming several releases and yes that will significantly reduce the backfill time.

sui_indexer::store: Error with persisting data into DB

to unblock, can you try set env var PG_COMMIT_OBJECTS_PARALLEL_CHUNK_SIZE to 200? to narrow down the issue, can you share all error messages from pg_indexer_store file?

[gegaowp]

Did not hear back from the reporter since last week, assuming the issue is no longer relevant and closing it for now -- feel free to re-open if needed.

[qkniep]

I had the same issue, on a version of the main branch as of today. At some checkpoint around 6.23M, the indexer stopped making progress with the same error message as above. I tried changing the above environment variable and tuning some PostgreSQL parameters, both of which did not help. Finally, I found the environment variable CHECKPOINT_COMMIT_BATCH_SIZE in the code. Reducing that from 100 to 10 did unblock the indexer up to checkpoint 6379122, at which point it got stuck again. But when setting CHECKPOINT_COMMIT_BATCH_SIZE, PG_COMMIT_OBJECTS_PARALLEL_CHUNK_SIZE and PG_COMMIT_PARALLEL_CHUNK_SIZE all to 1, the indexer was able to progress past that point.

[gegaowp]

Hey @qkniep yes I was able to reproduce and the fix is wip.

[gegaowp]

the fix pr #19754 is now merged to main