streaming preview: initial baseline

[DaveSkender]

Description

A preview of streaming use cases for quotes, .Use, EMA, and SMA only, for prototyping of initial scenarios.

• handle live quotes and provide them to other subscribers
• enable Use, EMA, and SMA indicator streaming
• live test case for WebSocket in /tests/observe/ in v3 branch

This is the first in a series of preview releases that implement streaming use cases. We expect possibly 5-10 preview pre-release versions before we make an official v3 package. They are all experimental and will be quite volatile with breaking changes and conventions.

# initialize quotes provider,
# can be extended to integrate with your
# quote stream or WebSocket source
QuoteProvider provider = new();

# subscribe individual indicators as observers
EmaObserver ema = provider.GetEma(20);
SmaObserver sma = provider.GetSma(13);
[..]

# these results are automatically updated
IEnumerable<EmaResult> emaResults = ema.Results;
IEnumerable<SmaResult> smaResults = sma.Results;

# when quotes are added in a separate process
provider.Add(quote);

# and also works with `.Use` chain
EmaObserver emaAlt = provider
    .Use(CandlePart.HL2)
    .GetEma(20);

# stop streaming
# by unsubscribing each observer
ema.Unsubscribe();

// stop providing quotes to everyone
// will unsubscribe all gracefully
provider.EndTransmission();

# feedback appreciated

Known limitations (cannot do)

• We do not handle out-of-order arrivals of quotes from streaming feeds. Quotes are updated, but subscribers ignored them.
• Indicators of indicators not supported yet, but we do support .Use chains
• Incomplete documentation

Warning
if you hookup to a trades stream, you will get many different quotes for a single candle period. Only an aggregate of individual trade information can represent a period candle. Use caution when selecting a provider or streaming API endpoint.

What we'll do next

• see v3.0.0 preview | Streaming indicators and quotes #1018

[codebeaulieu]

This would be super handy for use with web sockets. Reactive Extensions internals would be a major plus... possibly consider Dynamic Data as well, which is what I'm currently using.

[DaveSkender]

This would be super handy for use with web sockets. Reactive Extensions internals would be a major plus... possibly consider Dynamic Data as well, which is what I'm currently using.

For sure. I'm thinking of WebSocket specifically for this one.

Thanks for the reference on Dynamic Data -- it might be easier than simply using the native IObservable alone, something I will explore further. I'll have to weigh my appetite for third-party packages; I like the idea of continuing without any.

homework completed

• enable init empty (handle insufficient quotes case, calc when ready)
• experiment with LinkedList, also possibly for SMA lookback, for example --> not using, slow performance
• review Reactive Extension and Dynamic Data --> both of these have some nice advanced Rx things, but seem like overkill or likely performance-limiting overhead for the library (may still be good for library users). The out-of-the-box IObservable and general Rx design patterns seem to work fine for what I'm planning, no need for extra package dependencies IMO.

[DaveSkender]

I've come back to this and am currently working through the complexities ... like dealing with out-of-sequence late quote arrivals through chains of dependencies 😮. @codebeaulieu and @danbopes, I'd love some early feedback on the public interface UX depicted in the description above if you have any.

[martonb]

Just my humble opinion: you started to solve too many issues at once?
Personally, i would just have started with creating indicators that don't need to recalculate the full sequence every time a new quote comes in. Only later on create the extra code needed to connect to web sockets, or leave people to make their own hook-up connections, maybe by implementing an interface provided by your API?

Then, for my use case, I would be able to:

• request historic quotes from an arbitrary exchange, using whatever method (eg: REST request), from this moment, backwards N + warmup periods count quotes
• create the indicator, initialized with these quotes
• now I already have a valid indicator value calculated
• when a new quote notification is received from the exchange (via pure TCP/IP (protobuff) or WebSocket (JSON/XML), etc) your API user can process that and pass it to the indicator
• the indicator might delete the oldest quote from the sequence, then progressively calculate the new value
• the user can use the new calculated indicator value

Edit: I just found the "STREAM INITIALIZATION" marked functions in Ema.Api.cs and yes, those look exactly like what I meant, thanks! I assume they are internal, as not production ready?

[DaveSkender]

Just my humble opinion: you started to solve too many issues at once? Personally, I would just have started with creating indicators that don't need to recalculate the full sequence every time a new quote comes in.

This is some good general advice. Thank you. My goal here is to do a single tracer the whole way through with one chainable indicator before widely implementing across all other indicators. Once I get the basic concept working well, I'll think about how to apply some useful increments in smaller forms.

And you're right, I probably could have done the quote stream first or last; but found it hard to test the flow without at least one indicator and chain case. In isolation, the quote stream currently has these dimensions:

# initialize quotes provider,
# can be extended to integrate with your
# quote stream or WebSocket source
QuoteProvider provider = new();

# subscribe individual indicators as observers at any time

# you can add batch quotes
provider.Add(quotes);

# or incremental quotes at any time
provider.Add(quote);

# to pass quotes to subscribers

// stop providing quotes to everyone
// will unsubscribe all gracefully
provider.EndTransmission();

[DaveSkender]

I assume they are internal, as not production ready?

That’s right. It’s only in the feature branch right now. I’ll release a preview package soon, for people to try it out.