Gemini Inconsistent OrderBook

[traviscollins]

I have a long running Websocket running for Gemini subscribed to the BTC/USD and ETH/USD channels. Both channels are apparently floating to inconsistency in the local order book with the Gemini side orderbook.

Over long periods of time, I've seen the highest price BID order book entry become a stale entry that remains for long periods of time. Restarting the JVM (causing a reconnect to the web socket) immediately fixes the issue.

Has anyone else observed this behavior?

[traviscollins]

I believe I've nailed down this issue. Caused by the use of BigDecimal (a non-thread-safe class) as the key in the GeminiOrderBook price level map for asks/bids internally. The same BigDecimal instance is used as the value in the level - and can therefore be accessed by outside classes. This allows outside methods to access the BigDecimal object, which appears to cause possibly differing hashCodes (when two threads are accessing the same BigDecimal at once).

See GeminiOrderBook.java line 44.

Will test for a few days to ensure the fix is correct, and then submit a PR.

[Dclusin-og]

Isn't the order book only ever supposed to be touched on the event loop thread? Where's the other thread coming from? I'm curious as I'm building against this.

[traviscollins]

Multi-threaded application that depends on this library uses the orderbook in another thread.

[Dclusin-og]

You would probably want to make a deep copy of the order book each time you hand it off to another thread. The whole architecture of this library is designed with a single thread use case in mind, hence the data structures it uses are not thread safe. You've only solved one minor issue, lots more will pop up since you're using data structures that are not intended to be shared amongst threads without external syncrhonization.

One example: In the code for OrderBook it uses an ArrayList to store the limit orders for each side. If you use this across multiple threads and then try to iterate over it in your thread and the library comes and updates the rate book, your thread will receive a ConcurrentModificationException (per the ArrayList iterator java docs).

Copying the book will solve concurrency issues at the cost of application performance.

[traviscollins]

All good points. We’re doing some pretty heavy multi-thread access to the orderbooks. Haven't run into the concurrent modification issue, because the implementations we’re using so far all create a new list of LimitOrder for each observable event. Problem we’re running into is that we have multiple threads accessing the order books after they are created - so we’d need a copy per target thread.

…

On Feb 26, 2018, at 9:38 PM, Dclusin-og ***@***.***> wrote: You would probably want to make a deep copy of the order book each time you hand it off to another thread. The whole architecture of this library is designed with a single thread use case in mind, hence the data structures it uses are not thread safe. You've only solved one minor issue, lots more will pop up since you're using data structures that are not thread safe across multiple threads. One example: In the code for OrderBook it uses an ArrayList to store the limit orders for each side. If you use this across multiple threads and then try to iterate over it in your thread and the library comes and updates the rate book, your thread will receive a ConcurrentModificationException (per the ArrayList iterator java docs). The simplest way to solve this in your application would be a deep copy of the book each time you hand it off to your processor thread. This will adversely effect your performance. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[dozd]

Thanks for the fix!