Cheaper and more accurate fill tracking (trade tracking)

[nikhilsaraf]

Alternative title "Better integrity of trade data on non-SDEX exchanges in the context of open offers caused by a race condition". The race condition only occurs on non-DEX exchanges but the solution will be cheaper on SDEX so went with the above title.

Describe the bug

The bot can end up in a situation where it has an offer consumed and cannot see the corresponding trade and therefore makes incorrect decisions.

This is a race condition because it purely occurs only when the below (or similar) sequence occurs:

1. Bot places offer
2. Fill tracker checks for trades, finds nothing (correct) -- the fill tracker runs every X milliseconds. This is of course limited by the rate limit on the trading exchange.
3. Trade happens, which removes the open offer
4. We load offers and see no open offers.

Bug: the trade is not registered with the fill tracker and the offer also does not show up

Expected behavior

I think the correct behavior in this situation is for the bot to do one of the two options:

1. show the open offer and not register the trade (because it has not occurred at some cutoff point)
2. not show the open offer and have the trade registered (because it has already occurred at some cutoff point)

Frequency

The frequency is: Unpredictable (but with a high enough frequency)

Steps To Reproduce

See above

Possible Solution

There are many issues with this:

1. On centralized exchanges, there is no atomic way by which we can do both: load offers and trades
   a. this is not an issue with SDEX because this can be done atomically
2. We have currently implemented fill tracking as a separate thread and cannot exactly predict when it will run the fill trackers; this needs to be made more predictable

This bug can be fixed by:

• introduce a new config entry in trader.cfg: SYNCHRONIZE_STATE_LOAD_ENABLE and SYNCHRONIZE_STATE_LOAD_MAX_RETRIES
  • fetch trades immediately after loading offers somewhere here.
  • add checks to ensure atomicity of (offers, balances, trades): load offers and balances, load trades, reload offers and balances -- if both results of loading offers (and balances) are the same then it means nothing has changed during this window, otherwise reload trades and then reload offers (and balances) and recheck. If this fails SYNCHRONIZE_STATE_LOAD_MAX_RETRIES times then crash bot.
• update documentation so that it is clear that you do not need to set FILL_TRACKER_SLEEP_MILLIS to fetch trades and can use SYNCHRONIZE_STATE_LOAD_ENABLE instead, which is a cheaper and more consistent alternative
  • there is a valid use case to still want to track fills every X seconds in addition to using the new config (such as the mirror strategy). Strategies such as the balanced strategy could switch over to using this new field though.
• remove this line in sample_trader.cfg since fill tracking is supported in non-SDEX exchanges
• testing

Your Environment

cli version: master:nightly-33ab2afe-17-g8f927d83
gui version: v1.0.0-rc1
git branch: master
git hash: 8f927d8
build date: 20200723T104331Z
env: dev
GOOS: linux
GOARCH: amd64

Context

I am unable to reliably run a trading strategy that depends on having consistent trade data because of this.

relates to #306 and can be a possible solution.
Note that this is different from #450 but will provide an additional check on balances