services/horizon: Enable BucketListDB by default in captive core

[SirTyson]

Addresses #4710.

This PR exposes BucketListDB related configuration parameters in the captive-core toml file and enables BucketListDB by default in captive-core when using --captive-core-use-db. When enabled, BucketListDB reduces captive-core disk usage by around 50% and significantly reduces startup and catchup time.

This update exposes the captive core flags EXPERIMENTAL_BUCKETLIST_DB, EXPERIMENTAL_BUCKETLIST_DB_INDEX_PAGE_SIZE_EXPONENT, and EXPERIMENTAL_BUCKETLIST_DB_INDEX_CUTOFF. These flags control the memory usage of BucketListDB. See the stellar-core docs for more information on configuring these flags.

By default, captive-core will enable BucketListDB with a page size of 4 KB. This is different than the validator's default page size of 16 KB. While SDF's validators have NVME storage, the Horizon SKU uses EBS, requiring a different page size. If a Horizon operator is using storage medium other than AWS EBS, the page size parameter should be tuned accordingly.

With the default BucketListDB configuration, captive-core will use an additional ~2 GB of memory. However, this is partially offset by the memory savings from reducing the size of the SQL database. Should this memory overhead be too high, memory usage can be configured using the exposed configuration parameters.

This update will not automatically trigger a state rebuild. However, if EXPERIMENTAL_BUCKETLIST_DB is set to false in the captive-core toml, a state rebuild will be triggered.

[sreuland]

@SirTyson , thanks for incorporating theses changes, was checking out that one CI test failure in ubuntu-20.04, 1.18.6, 9.6.5, captive-core-remote-storage, 19, which seems to be failing on TestClaimableBalanceCreationOperationsAndEffects, maybe merge latest from master back in here, and let's see if that clears up?

[mollykarcher]

Very excited to see the difference this ends up making, thanks for testing this out! Would love it if you could share the results (+process) of the benchmarking/calibration you're doing once that's complete.

[SirTyson]

Very excited to see the difference this ends up making, thanks for testing this out! Would love it if you could share the results (+process) of the benchmarking/calibration you're doing once that's complete.

What would be the best format for something like this? It's a bit of an involved process where you have to build a special version of core and use some some low-level bench marking tools, but I could put a process doc together with some results if it's helpful.

[sreuland]

looks good, approved, but before merging, let's get a review from at least one other horizon team member to confirm this approach with core runtime version checking is acceptable approach:

#4733 (comment)

thanks!

[SirTyson]

Looks like all the comments have been addressed and the regex stuff has been reviewed, should be good to merge?

[sreuland]

Looks like all the comments have been addressed and the regex stuff has been reviewed, should be good to merge?

@SirTyson , yes, I resolved those older comments, hopefully the repo allows you to proceed on merge button, if not lmk can click it, thanks again, great work!