Fix and improve path finding #719
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closed
nikhilsaraf
reviewed
Oct 18, 2018
| @@ -237,7 +237,7 @@ func (base *Base) GetAccountID(name string) (result xdr.AccountId) { | |||
| // conventions | |||
| func (base *Base) GetAmount(name string) (result xdr.Int64) { | |||
| var err error | |||
| result, err = amount.Parse(base.GetString("destination_amount")) | |||
There was a problem hiding this comment.
why did this change -- was the name param never used correctly earlier?
There was a problem hiding this comment.
Yes, this was working correctly only when name="destination_amount".
nikhilsaraf
approved these changes
Oct 18, 2018
bartekn
force-pushed
the
fix-path-finding
branch
from
cb317fb to
f20bbfc
Compare
|
Thanks @nikhilsaraf! I will merge it now and it will be autodeployed to https://horizon-dev-pubnet.stellar.org/ so more people can easily test it. In the end we can revert this PR if there's a need but I believe it should work fine and give better results. |
bartekn
added a commit
that referenced
this pull request
Oct 19, 2018
This was referenced Oct 19, 2018
bartekn
added a commit
that referenced
this pull request
Oct 25, 2018
This commit fixes several bugs and adds improvements making path finding
results better, returning them in a shorter time.
The most severe bug was found in `pathNode.Cost` method. `pathNode` is a
linked list node with `Tail` field pointing to the next asset on the
path. So the front of the list is the currently extended source asset
and the tail is always the destination asset. The previous code was
wrongly consuming offers in the wrong order. Fixing this bug required to
walk through the current path from the tail (the destination asset) to
front (the current source).
I added a new test explaining why the old code was wrong. Consider the
path: `AAA -> BBB -> CCC`. The core of the problem is that the previous
code was using `DestinationAmount` as a starting value for `AAA -> BBB`
trade, in other words: "give me amount of `AAA` needed to buy
`DestinationAmount` of `BBB`". It doesn't make any sense. What you
should do first is: "give me amount of `BBB` needed to buy
`DestinationAmount` of `CCC`". So for the destination amount equal `10`
and the following offers (`scc` notation):
```
offer :trader, {for:["AAA", :gateway], sell:["BBB", :gateway]}, 1, 11
offer :trader, {for:["BBB", :gateway], sell:["CCC", :gateway]}, 10, 0.1
```
The old algorithm would not find a path as it is not possible to buy
`1 BBB` with `10 AAA` and `1 BBB` is needed to buy `10 CCC`.
Fixing this introduced a huge optimization opportunity. Basically the
cost of trades between current source asset and the destination asset
does not change after prepending the path with a new asset. So we can
cache the results instead of checking every order book on the path every
time we add a new asset in the front of the path.
Another improvement was to remove `A -> B -> A` subpaths from the paths
as they will always be a bad deal. This also removed all `destination
asset -> A -> destination asset` paths.
Finally, added `MaxPathLength` that is configurable via CLI switch or
environment variable and defines the maximum length of the path that
`/paths` will return, with a default value of `4` (`source asset -> A ->
B -> destination asset`). The default value of 4 comes as a result of
observation of paths found between many different assets: most of them
have the maximum length of 4 and if the path length was higher it was a
bad deal. The `MaxPathLength` allowed another optimization in
`search.extendSearch` method. Basically if `p.Depth == s.MaxLength-1`
and connected asset is not a target we don't extend such path as it will
not be possible to extend the path any longer.
After implementing all the optimizations average search time for my
queries dropped from 10 seconds to 2 seconds giving a better results.
Finally, a single search is run in `REPEATABLE READ, READ ONLY` DB
transaction to have a stable view of the offers table.
bartekn
added a commit
that referenced
this pull request
Oct 25, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes several bugs and adds improvements making path finding results better, returning them in a shorter time.
The most severe bug was found in
pathNode.Costmethod.pathNodeis a linked list node withTailfield pointing to the next asset on the path. So the front of the list is the currently extended source asset and the tail is always the destination asset. The previous code was wrongly consuming offers in the wrong order. Fixing this bug required to walk through the current path from the tail (the destination asset) to front (the current source).I added a new test explaining why the old code was wrong. Consider the path:
AAA -> BBB -> CCC. The core of the problem is that the previous code was usingDestinationAmountas a starting value forAAA -> BBBtrade, in other words: "give me amount ofAAAneeded to buyDestinationAmountofBBB". It doesn't make any sense. What you should do first is: "give me amount ofBBBneeded to buyDestinationAmountofCCC". So for the destination amount equal10and the following offers (sccnotation):The old algorithm would not find a path as it is not possible to buy
1 BBBwith10 AAAand1 BBBis needed to buy10 CCC.Fixing this introduced a huge optimization opportunity. Basically the cost of trades between current source asset and the destination asset does not change after prepending the path with a new asset. So we can cache the results instead of checking every order book on the path every time we add a new asset in the front of the path.
Another improvement was to remove
A -> B -> Asubpaths from the paths as they will always be a bad deal. This also removed alldestination asset -> A -> destination assetpaths.Finally, added
MaxPathLengththat is configurable via CLI switch or environment variable and defines the maximum length of the path that/pathswill return, with a default value of4(source asset -> A -> B -> destination asset). The default value of 4 comes as a result of observation of paths found between many different assets: most of them have the maximum length of 4 and if the path length was higher it was a bad deal. TheMaxPathLengthallowed another optimization insearch.extendSearchmethod. Basically ifp.Depth == s.MaxLength-1and connected asset is not a target we don't extend such path as it will not be possible to extend the path any longer.After implementing all the optimizations average search time for my queries dropped from 10 seconds to 2 seconds giving a better results.
Finally, a single search is run in
REPEATABLE READ, READ ONLYDB transaction to have a stable view of the offers table.