Investigate query duration of backend

[2opremio]

After the transactions where moved from in-memory data structures to the backend, the average query duration has considerably increased.

This makes sense, but it has increased to 100ms, which is not the end of the world, but isn't great either.

See https://grafana.stellar-ops.com/d/ut9rEquVz/soroban-rpc?orgId=1&var-Datasource=Prometheus-kube001-prd&var-namespace=soroban-rpc-pubnet-prd

[2opremio]

It seems to be caused by getHealth so, my bet is on GetLedgerRange(), which, according to #217 (comment) , it only takes microseconds.

So, something's up with either the benchmark of the environment.

[mollykarcher]

so, my bet is on GetLedgerRange()

Isn't this called by other endpoints as well though? Why would we not see similar degradations on other endpoints if this was the culprit?

Semi-related; I guess this is a good thing rather than an issue, but do we understand why the performance of getEvents got a lot better due to this release?

[Shaptic]

From my perspective there are a handful of possible approaches here:

1. caching the results of GetLedgerRange with eviction on ledger ingestion, which we've discussed
2. trying a different query that might end up being faster
3. adding a row to the table to store earliest ledger in addition to latest ledger instead of removing the table entirely? this is effectively the same as option (1) but in the DB instead of in memory (related: db: Remove latestledger key from meta table #233)

Maybe there are others.

Also, since it's only getHealth, it could also be related to Captive Core since it still uses the /info endpoint, right? Maybe #198 could fix it.

[2opremio]

1. caching the results of GetLedgerRange with eviction on ledger ingestion, which we've discussed

This is planned as per #233

2. trying a different query that might end up being faster

I think the query is pretty fast already but, with catching, we should be able to only execute it once (we can book-keep it in memory after that)

Also, since it's only getHealth, it could also be related to Captive Core since it still uses the /info endpoint, right? Maybe #198 could fix it.

Ah, you are right. I forgot #198 is not it yet, that may be it!

[psheth9]

oh wait getHealth does not use /info endpoint right? so it should not be related to the improvement of #198 but regardless PR is ready to go and we should merge it ASAP

[jacekn]

https://stellarfoundation.slack.com/archives/C02B04RMK/p1724068773238179

[jacekn]

The in-memory data structures will be removed in the future.
For the time being @2opremio gained some performance by changing page size so we're good.
I'll close this since there isn't much value in optimising something that will disappear in the future