A PoC to retrieve real-time Crypto Exchange data, as a first step for an ordering bot! The code belongs to this YouTube sequence.
A docker compose script is provided. Start the application with:
docker compose --profile full up -d
With scaling:
docker compose --profile full up --scale frontend=2 --scale backend=2 --scale analysis=2 -d
The profiles can be used in every call --profile=full for example, or by setting the environment variable
COMPOSE_PROFILES.
For example in Window/Powershell: $Env:COMPOSE_PROFILES="full".
...
| System | link |
|---|---|
| frontend | http://localhost:8081/ |
| Kafka console | http://localhost:8181/ |
Using testcontainers can be slow during testing, because the used containers are removed after the test. The article Reuse Containers With Testcontainers for Fast Integration Tests tells how to make integration testing faster.
The following queries show some useful information:
Number of records:
db.message.count()
Date values :
db.message.aggregate(
[
{
$group:{
_id: {
exchange: "$exchange"
},
"min date": {
$min : "$created"
},
"max date" : {
$max : "$created"
},
"#messages" : {
$sum:1
}
}
}
]
)
Message sizes:
db.message.aggregate(
[
{
$project: {
currencyPair: 1,
exchange: 1,
length: {
$strLenCP: "$message"
}
}
},
{
$group:{
_id: {
exchange: "$exchange",
currencypair:"$currencyPair"
},
"avg message" : {
$avg : "$length"
},
"max message" : {
$max : "$length"
},
"min message" : {
$min : "$length"
},
"#messages" : {
$sum:1
},
"volume" : {
$sum : "$length"
}
}
}
]
)
You can build the application locally via a maven command:
| target | meaning |
|---|---|
| mvn clean | clean environment |
| mvn install | build self executable JAR files |
| mvn install -P docker | build docker images |
| mvn versions:display-property-updates | check if latest versions of libraries are used. |
To use the created local docker image, replace ghcr.io/buildingsoftwareblocks/withlocalin
the used docker-compose.yml file.
To make sure that you have the latest images, use: docker compose --profile=full pull.
The profiles can be used in every call --profile=full for example, or by setting the environment variable
COMPOSE_PROFILES.
For example in Window/Powershell: $Env:COMPOSE_PROFILES="full".
- 1 Exchange, 1 Currency pair connected
- multiple services connected via Kafka
- build pipeline
- store events in database
- multiple exchanges / multiple Currency pairs
- order book analysis module
- simple arbitrage
- scalable backend module
- scalable frontend module
- scalable docker frontend
- scalable analysis module
- better logging
- triangular Arbitrage
- better frontend GUI
- binary messages in Kafka
- more robust error handling
- run on Kubernetes
- ElasticSearch to version 8 (SB3 supported version)