* ticker ingest assets optimizations

* Update services/ticker/internal/actions_asset.go

Co-authored-by: Leigh McCulloch <[email protected]>

* Update services/ticker/internal/scraper/asset_scraper.go

Co-authored-by: Leigh McCulloch <[email protected]>

* Update services/ticker/internal/scraper/main.go

Co-authored-by: Leigh McCulloch <[email protected]>

* remove commented code

* remove pointer passing of FinalAsset

Co-authored-by: Leigh McCulloch <[email protected]>

…r#4220)

This change ensures that all existing operations are reingested
and that all future operations are covered(by going through
xdr.OperationTypeToStringMap)

…llar#4092)

…ed (stellar#4228)

Release DB connection to the pool as soon as it's no longer needed in
`/paths/*`.

Horizon starts `REPEATABLE READ` transaction in `StateMiddleware` to ensure that
queries return consistent (at the same ledger) data when sending multiple
queries. However, the tx is held in `/paths/*` for entire request time, even
though the path finding algorithm is not using a DB.

…de aggregations (stellar#4178)

* Exclude trades with >10% rounding_slippage from trade aggregations

* Fixing tests

* Add db.NullRat for null bigrats in the db

* Add integration tests for rounding slippage filtering in trade aggs

* Add --rounding-slippage-filter flag

* Fix typo

* Review feedback

* note div-by-zero

* Track the base and counter reserves on each trade so we can sanity-check ingest results

* Review feedback

* re-use orderbook package to calculate rounding slippage

* Clean up test a bit

* Missed a few test updates

* Clarify terminology around slippage to fix calculations

* Add an extra column we might need in order to calculate rounding slippage later

* Disable roundingSlippage calc for /paths

* OPTIMIZE ALLOCATIONS :robot-face:

* Workaround for stellar#4203

* Add changelog entry

* RoundingSlippage should be in bips not megabips

* remove redundant if clauses

* Fixing test

* Fix flag description

* remove trade reserves tracking for debugging

* Add some comments to clarify slippage calculations

Co-authored-by: Paul Bellamy <[email protected]>

…ellar#3727)

Merge Horizon 2.15.0 back to Master

…#4251)

This commit revers LedgerHashStore removed from Horizon's CaptiveCoreConfig in:
b7fde58. This change makes it impossible to PrepareRange using Captive-Core if
there is any other ingestion instance running because CaptiveCoreBackend tries
to load the hash of the ledger from archives when LedgerHashStore is not set.
But ledgers are published every 5 minutes so they contain the hash of the latest
ledger only for a brief amount of time (the moment when checkpoint is
published).

…#4250)

…tellar#4253)


Co-authored-by: Leigh McCulloch <[email protected]>

Merge horizon v2.15.1 release branch back to master

…ected (stellar#4266)

* Reset the module and build cache in master/protected

Before, we:

1. Didn't use the build cache in the master branch. This caused the cache
   to not be updated. This caused PRs to rerun all the tests.

2. Didn't reset the go module cache. This would eventually cause the cache
   to accumulate orphan dependencies when we remove them.

* Update .github/actions/setup-go/action.yml

Co-authored-by: Leigh McCulloch <[email protected]>

* Prioritize the master cache over caches from other branches

Co-authored-by: Leigh McCulloch <[email protected]>

* More go-setup github action fixes

* Address review feedback

Add a readme for the xdr package describing how to generate xdr.

I frequently forget and have to figure it out each time. This might help someone else too.

The examples use docker so that it is a quick copy/paste for most people. We use docker frequently so most of us have docker. There's a good chance that less of us have ruby setup in a good state locally.

I also added the ref to the Rakefile because I also forget how to specify a different source location for downloading XDR and if the ref was always specified it would be easier to just edit it.

Cache TOMLs to during scraping to some degree. Caching is local to each routine that is scraping and not shared across those routines, so it is possible to still get duplicate requests. Cache usage is logged.

The asset scraper may end up repeatedly scraping the same issuer's TOML many times if the issuer has issued many assets. This is not great since any scraper should attempt to respect the resources of hosts as much as possible.

The cache is in-memory and per routine because the cache is only required temporarily, and to introduce an external cache, such as redis, would be overkill.

Cache usage is logged so that it can be inspected and understood.

Use log fields in logs in the asset ingestion job of Ticker and include TOML url in logs.

To make filtering and analyzing logs easier.

Change the maximum size of the toml cache used during asset ingestion in Ticker to a single toml, and sort the assets by their TOML URLs when loading.

If we sort assets by their TOML URLs we only need to store one toml in the cache, the last one. This reduces the amount of memory required.

…stellar#4279)

* Add Protocol 19 XDR: raw and generated
* Update codebase to use new XDR for predicates/timebounds
* Add stringification method for Preconditions
* Add encode/decode helper for CAP-40 signed payloads
* Add Horizon DB migration: signers can be longer now
* Disable generating the V3 extension on accounts

)

Signed-off-by: Satyam Zode <[email protected]>

Co-authored-by: Satyam Zode <[email protected]>

stellar#4296)

Fixes stellar#4290

It's the first time I get into this code and the naming has made it
hard to understand. So, I decided to make the naming more
understandable.

horizon: Merge master into horizon-protocol-19 branch

…lar#4302)

…ditions (stellar#4301)

… that they are implemented. (stellar#4304)

Adds support for new CAP-21 preconditions with a simple, but breaking API
change:

TransactionParams.Timebounds -> TransactionParams.Preconditions.TimeBounds

You can now pass a lot more preconditions to a Transaction (including
timebounds), and these are managed by the new Preconditions object. All of the
XDR abstractions are hidden away behind this object.

…4297)

* Add cap-21 transaction preconditions

* Replace xdr.TransactionEnvelope.TimeBounds -> .Preconditions function

* Rename Signers -> ExtraSigners for more consistent naming

* Store the new precondition fields in the DB

* Render the new precondition fields in the resourceadapter

* ./gogenerate.sh

* use a pointer so omitempty works

* more consistent naming

* Keep Timebounds helper

* PR feedback

* tidy transaction precondition helpers

* typo fixes

* remove dead param

* use bigint so we have enough space for a uint32

* use text instead of varchar, to avoid future migrations

* Updating scenario sqls

* need to load the preconditions from the db

* Temporary passing data for v2 preconditions transaction insertion test

* Update xdr/transaction_envelope_test.go

Co-authored-by: George <[email protected]>

* Update services/horizon/internal/db2/history/transaction_ledger_bounds.go

Co-authored-by: George <[email protected]>

* PR feedback

Co-authored-by: George <[email protected]>

…s`. (stellar#4307)

* Timebounds -> TimeBounds, Ledgerbounds -> LedgerBounds
* Duration -> int64, SignerKey -> string

* Update XDR to latest from sisuresh/stellar-core#cap21-40-2
* Update codebase to make cb predicates use Int64s again

…r#4310)

Add a per second rate limit for path finding requests backed by https://pkg.go.dev/golang.org/x/time/rate

…query (stellar#4248)

* Add cursor and limit parameters to the ClaimableBalanceRequest
* Close stellar#4247

horizon: Merge master into horizon-protocol-19 branch

…#4312)



Co-authored-by: Shawn Reuland <[email protected]>

Use the GitHub actions Go workflow badge (liking to the master push Go workflow page)

…itions

Added additional round trip verification by re-querying preconditions on tx after submission, per pr feedback

Port Release 2.16.0 notes back to master

Co-authored-by: Leigh McCulloch <[email protected]>

services/horizon/integration: Added more integration tests for coverage of preconditions on transactions

…merge_2.16.1

Co-authored-by: Bartek Nowotarski <[email protected]>

This reverts commit c83df6a.

…years (stellar#4337)

* Merge Protocol 19 migrations into a single file
* Actually generate V3 account extensions
* Change accounts.sequence_time to be bigint over timestamp:

This is necessary because PostgreSQL puts "reasonable" limitations
on how dates can be represented. Because the state verifier test
will generate random int64s for the V3 account extension field
introduced in CAP-21, this will blow up on inserting into the DB
with errors like:

    pq: date/time field value out of range: "120911623444-02-01 21:35:20Z"

because PostgreSQL only allows years up to 5874897
(https://stackoverflow.com/a/36446977).

Technically, since this field is not user-controlled and comes
directly from Stellar Core, we would not see this in practice.

However, it's a more durable fix to stop the extra layer of
"interpretation" of this value and just treat it like it is:
a 64-bit integer that happens to represent a UNIX timestamp.

…nt` to string (stellar#4339)

* `minSeqAge` can be unsigned in the XDR
* Update codebase to conform to new data type: null.Int -> null.String
* Fix time-based tests to be tz-agnostic + work w/ uint64

* all: Add Protocol 19 XDR and update StrKey to support Signed Payloads (stellar#4279)
* services/horizon: Support new account fields for protocol-19. (stellar#4294)
* xdr, keypair: Add helpers to create CAP-40 decorated signatures (stellar#4302)
* services/horizon: Update txsub queue to account for new CAP-21 preconditions (stellar#4301)
* txnbuild: Add support for new CAP-21 preconditions. (stellar#4303)

Adds support for new CAP-21 preconditions with a simple, but breaking API
change:

TransactionParams.Timebounds -> TransactionParams.Preconditions.TimeBounds

You can now pass a lot more preconditions to a Transaction (including
timebounds), and these are managed by the new Preconditions object. All of the
XDR abstractions are hidden away behind this object.

* services/horizon: Support new CAP-21 transaction conditions (stellar#4297)
* txnbuild: Complete rename, avoid using XDR types in `TransactionParams`. (stellar#4307)
* all: Update Protocol 19 XDR to the latest (stellar#4308)
* horizon: Set up protocol 19 integration tests infrastructure (stellar#4312)
* horizon: Test new protocol 19 account fields (stellar#4322)
* horizon: Add integration test for extra signers using tx preconditions (stellar#4305)
* horizon: Add transaction submission test for Protocol 19 (stellar#4327)
* services/horizon: Use `bigint` over `timestamp` to accommodate large years (stellar#4337)

This is necessary because PostgreSQL puts "reasonable" limitations
on how dates can be represented. Because the state verifier test
will generate random int64s for the V3 account extension field
introduced in CAP-21, this will blow up on inserting into the DB
with errors like:

    pq: date/time field value out of range: "120911623444-02-01 21:35:20Z"

because PostgreSQL only allows years up to 5874897
(https://stackoverflow.com/a/36446977).

Technically, since this field is not user-controlled and comes
directly from Stellar Core, we would not see this in practice.

However, it's a more durable fix to stop the extra layer of
"interpretation" of this value and just treat it like it is:
a 64-bit integer that happens to represent a UNIX timestamp.

* services/horizon: Change `min_account_sequence_age` column from `bigint` to string (stellar#4339)
* Add changelog details and prep v10.0.0 SDK release

Co-authored-by: Alfonso Acosta <[email protected]>
Co-authored-by: erika-sdf <[email protected]>
Co-authored-by: Paul Bellamy <[email protected]>
Co-authored-by: Shawn Reuland <[email protected]>
Co-authored-by: shawn <[email protected]>
Co-authored-by: George <[email protected]>

…tellar#4345)

…tellar#4347)

…deref (stellar#4349)

…rade boundary. (stellar#4354)

…tellar#4357)

…llar#4356).

…or (stellar#4360)

* Omit max_ledger when it's 0
* Drops minSeqAge from response when it's 0:

Because we use a string for the field (to work around languages
that don't natively support uint64 values) in the JSON response,
but the field is a non-pointer in the XDR, it will always be set
and will be 0 by default.

Since it's a duration, 0 is effectively "not set", and so it
should be omitted from the resulting precondition.

Caveat: if someone explicitly sets it to 0 when building their tx,
this is equivalent to not setting it, so they may be confused
about its omission, but this is working as intended.

* Use UNIX timestamps instead of RFC3339 strings for timebounds. (stellar#4361)

…stellar#4366)

* remove gemfile and rakefile to just use docker for the xdrgen

* Fetch xdr/*.x from github

* add make xdr-clean and xdr-update

* Update Makefile

Co-authored-by: Leigh McCulloch <[email protected]>

* move xdr go lint frontmatter to xdrgen

* Rebuild with new xdrgen frontmatter

* lock the xdrgen commit we use

Co-authored-by: Leigh McCulloch <[email protected]>

stellar#4368)

…ished (stellar#4185)