exp/orderbook: Expand path payment algorithm to search liquidity pools.

[Shaptic]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've updated the relevant CHANGELOG (here for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

What

Allows bidirectional path-finding through both liquidity pools and offers.

Major changes:

• Support for simulating exchanges w/ a liquidity pool: see makeTrade and its corresponding calculation functions
• Modify edgeSet to represent all trade opportunities: both offers (as before) and now liquidity pools.
• Modify searchState interface to process offers and LPs ("venues") properly:
  • venues(asset) retrieves all pools and LPs for a given asset
  • chooseVenue(a, b) determines whether the path should choose offers (a) or pools (b), since the choice depends on if we're going forwards (FindFixedPath) or backwards (FindPath) through the graph's edges.
  • consumePool(pool, asset, amount) evaluates an exchange with a pool, which is also direction-dependent
• the new processVenues() is a catch-all helper to evaluate pools and offers (in that order) fairly
• Modify the OrderBookGraph to handle batch add-update-removal of liquidity pools

Why

We need to enable users to find paths through both the existing DEX and through the newly-introduced liquidity pools.
See #3836 for more.

Known limitations

I'd like to incorporate Jon's tests to evaluate the actual liquidity pool exchange math, and I'd also like to ensure that the test cases sufficiently cover the new functionality I've introduced here.

[tamirms]

to be consistent with the other operations we should either encapsulate the liquidity pool operations into functions on the orderbook graph, or, we should inline orderbook.add() and orderbook.remove() into orderBookBatchedUpdates .apply(). I think it would be better to encapsulate the liquidity pool operations into separate functions

[Shaptic]

Yep, agreed. Now that these are stable (:sweat_smile:) it makes sense to abstract them like for offers.

[tamirms]

I think we can do a slight refactoring to make the interface easier to understand. Instead of having chooseVenue() , consumeOffers(), and consumePool() we could replace it with a single function:

trade(currentAsset xdr.Asset, currentAssetAmount xdr.Int64, venues Venues) (xdr.Asset, xdr.Int64, error)

The trade() function will essentially do the logic to select the best venue and then execute the trade

[Shaptic]

Nice idea, I'll give it a whirl.

[Shaptic]

So after trying this out, I realized why I broke it up into parts like this. If you look at the processVenues helper (which is what DFS uses):

https://github.com/stellar/go/blob/1c7e42b7e389c8b1b1b3b1a997d4f0e08deb5f44/exp/orderbook/dfs.go#L432-L479

The only points of divergence are at state.consumePool, state.consumeOffers, and state.chooseVenue, all necessary because buys and sells interact with the pool/offers differently and the results need to be evaluated differently (< vs. >).

If we combine these into a single state.trade(...), then we have ~45 lines of duplicated code across the sellingGraphSearchState and buyingGraphSearchState, with divergences exactly as they are above, except I guess they wouldn't need to be attached to the searchState.

I agree that this can be improved, but I'm not sure this suggestion would lead to cleaner code overall. A cleaner interface, maybe, but uglier interface implementations. I did do some interface cleanup in the latest push and dropped the awkward chooseVenue() method. Now it's just consumePool and consumeOffers, which I think is pretty straightforward to understand.

[Shaptic]

See cd60906 for the delta: I essentially extended consumeOffers to take a "current best" parameter that makes its return values cleaner to process (and opens up an opportunity for a processing optimization, but this is TBD).

[paulbellamy]

Sorry for coming in swinging here, but maybe fresh eyes are useful. Maybe a more object-oriented approach here would make the code easier to maintain and understand?

If we think of each venue as a graph edge, the new situation AMMs introduce is that we have multiple edges/venues between each asset pair. But you could see that being extended in future with "Concentrated Liquidity Pools" or somesuch else, like:

So, we could model the different venue types as an interface, which would let us do something like:

  // All venues implement the Venue interface
  type Venue interface {
    // Calculate how much you would receive if you sent the inputs.
    Send(
      asset xdr.Asset,
      amount xdr.Int64,
    ) (xdr.Asset, xdr.Int64, error)

    // Calculate how much you would have to send in order to receive the expected.
    Receive(
      expectedAsset xdr.Asset,
      expectedAmount xdr.Int64,
    ) (xdr.Asset, xdr.Int64, error)

    // Utility, maybe if we need
    OtherAsset(asset xdr.Asset) xdr.Asset
  }

  // Implement Venue for liquidity pools
  type LiquidityPoolVenue xdr.LiquidityPoolEntry
  func (v *LiquidityPoolVenue) OtherAsset(asset xdr.Asset) xdr.Asset
  func (v *LiquidityPoolVenue) Send(asset xdr.Asset, amount xdr.Int64) (xdr.Asset, xdr.Int64, error)
  func (v *LiquidityPoolVenue) Receive(expectedAsset xdr.Asset, expectedAmount xdr.Int64) (xdr.Asset, xdr.Int64, error)

  // Implement Venue for orderbooks
  type OrderbookVenue []xdr.OfferEntry
  func (v *OrderbookVenue) OtherAsset(asset xdr.Asset) xdr.Asset
  func (v *OrderbookVenue) Send(asset xdr.Asset, amount xdr.Int64) (xdr.Asset, xdr.Int64, error)
  func (v *OrderbookVenue) Receive(expectedAsset xdr.Asset, expectedAmount xdr.Int64) (xdr.Asset, xdr.Int64, error)

  // Implement Venue for arrays, choosing the best path. To simplify graph traversal.
  type Venues []Venue
  func (v *Venues) OtherAsset(asset xdr.Asset) xdr.Asset
  func (v *Venues) Send(asset xdr.Asset, amount xdr.Int64) (xdr.Asset, xdr.Int64, error)
  func (v *Venues) Receive(expectedAsset xdr.Asset, expectedAmount xdr.Int64) (xdr.Asset, xdr.Int64, error)

  // then we can model each edge as an array of venues
  type edgeSet map[string]Venues

Just an idea, but maybe it's a big change, or has too much performance overhead?

[tamirms]

@Shaptic the code looks good to me. I just left a couple of suggestions about some minor issues. if you can fix the tests and resolve any outsanding todos / fixmes then I think we can merge this