services/horizon: Use encoding buffer in state verifier

[2opremio]

Profiling shows there are a lot of allocations due to buffer resizing.

[tamirms]

I think the encoding buffer could also be used by the checkpoint change reader which calls MarshalBinaryCompress() in two places:

go/ingest/checkpoint_change_reader.go

Line 370 in 3e1e280

keyBytes, e := key.MarshalBinaryCompress()

go/ingest/checkpoint_change_reader.go

Line 424 in 3e1e280

keyBytes, e := key.MarshalBinaryCompress()

There is also the XdrStream struct in the historyarchive package. We use that to extract xdr ledger entries out of the history archive snapshots. There are lots of decoding operations happening in XdrStream:

go/historyarchive/xdrstream.go

Line 178 in a613f85

readi, err := xdr.Unmarshal(&x.buf, in)

Would it be possible to use a decoding buffer there? I think that would speed up state verification and the buildState step in the ingestion state machine.

[2opremio]

Yep, I was already working on migrating the compress functions. I will look into the XdrStream next. Thanks.

…

On Sat, Nov 13, 2021, 13:10 tamirms ***@***.***> wrote: ***@***.**** approved this pull request. I think the encoding buffer could also be used by the checkpoint change reader which calls MarshalBinaryCompress() in two places: https://github.com/stellar/go/blob/3e1e280c1108ab08747c6d17604bd48d74d6b757/ingest/checkpoint_change_reader.go#L370 https://github.com/stellar/go/blob/3e1e280c1108ab08747c6d17604bd48d74d6b757/ingest/checkpoint_change_reader.go#L424 There is also the XdrStream struct in the historyarchive package. We use that to extract xdr ledger entries out of the history archive snapshots. There are lots of decoding operations happening in XdrStream: https://github.com/stellar/go/blob/a613f85a5654a7d9a6efd01678dd19af72acd5a1/historyarchive/xdrstream.go#L178 Would it be possible to use a decoding buffer there? I think that would speed up state verification and the buildState step in the ingestion state machine. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4069 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASA4JF3POTYWZ42GOK62TDULZISJANCNFSM5H6LIKPA> .

[2opremio]

I think the encoding buffer could also be used by the checkpoint change reader which calls MarshalBinaryCompress() in two places:

Done at #4071

There is also the XdrStream struct in the historyarchive package. We use that to extract xdr ledger entries out of the history archive snapshots. There are lots of decoding operations happening in XdrStream:

I don't think it's worth optimizing this until stellar/xdrgen#71 is merged

[2opremio]

There is also the XdrStream struct in the historyarchive package. We use that to extract xdr ledger entries out of the history archive snapshots. There are lots of decoding operations happening in XdrStream:

Addressed at #4075