Add WriteBack Message Store Wrapper For Large-Scale Traffic

[ChoiJunsik]

Add WriteBack Message Store Wrapper For Large-Scale Traffic

I've encountered a issue where large-scale traffic leads to a reduced simultaneous throughput due to the Fix Session Lock mechanism. This problem becomes particularly acute when using JdbcStore.

In our current setup with a Spring-based web server, we're experiencing a bottleneck, with the system only able to process up to 200 Transactions Per Second (TPS).

The core of the problem seems to lie in the store.set() function, which acts as a critical section and is significantly affecting performance.

To mitigate this, I'm considering the implementation of a write-back pattern for the store.set() function.

Proposed Solution:

Apply a write-back pattern to the store.set() function to alleviate the bottleneck and enhance throughput.
I'm reaching out to seek advice on the best practices for implementing this solution or to hear alternative suggestions that may resolve the throughput issue more effectively.

Any insights or guidance would be greatly appreciated.

This is my personal POC branch below!

ChoiJunsik#1

[chrjohn]

In general I have no objections against this but one should keep in mind that it could happen that your FIX session could be in an unrecoverable (i.e. needs manual intervention) state.
Example:

1. you send 1000 messages but did only persist up to seqnum 500
2. your application crashes
3. you try to re-login with seqnum 501 and counterparty logs you out because they expect seqnum 1001

Which kind of data are you transmitting via FIX? 200TPS is maybe no good fit for the plain FIX protocol, you should rather use something like SBE. But only IMHO, do what you want. ;)

Did you look at MemoryStore or tried to disable persisting of messages?

There were related discussions which could be of interest here:
#745
#449
#427
#415

[ChoiJunsik]

Thanks for Opinion @chrjohn !!

I have worried my service might not be able to handle 20,000 TPS, but I'm thinking Redis could be the key for both durability and speed.

Is there any talk of integrating Redis features directly into QuickFIX/J ?
I'd be really excited to help out with the development.

[chrjohn]

So what data are you actually transmitting? Market data? And you need it to be persisted?

[ChoiJunsik]

it's stock trading data !

[chrjohn]

So market data? Actually it doesn't make sense to persist it then because probably no-one will ask for that much data to be resent.
If I misunderstood then it probably makes sense if you explained your use case.