Skip to content

Latest commit

 

History

History
177 lines (138 loc) · 5.53 KB

README.md

File metadata and controls

177 lines (138 loc) · 5.53 KB
Raw

Develop Main

Exchange Data PoC

A PoC to retrieve real-time Crypto Exchange data, as a first step for an ordering bot! The code belongs to this YouTube sequence.

System Overview

System Overview

Run Application

Docker Compose

A docker compose script is provided. Start the application with:

docker compose --profile full up -d

With scaling:

docker compose --profile full up --scale frontend=2 --scale backend=2 --scale analysis=2 -d

The profiles can be used in every call --profile=full for example, or by setting the environment variable COMPOSE_PROFILES.

For example in Window/Powershell: $Env:COMPOSE_PROFILES="full".

Kubernetes

...

Links

System link
frontend http://localhost:8081/
Kafka console http://localhost:8181/

Test Performance

Using testcontainers can be slow during testing, because the used containers are removed after the test. The article Reuse Containers With Testcontainers for Fast Integration Tests tells how to make integration testing faster.

MongoDB queries

The following queries show some useful information:

Number of records:

db.message.count()

Date values :

db.message.aggregate(
    [
        {
            $group:{
                _id: {
                    exchange: "$exchange"
                },
                "min date": {
                    $min : "$created"
                },
                "max date" : {
                    $max : "$created"
                },
                "#messages" : {
                    $sum:1
                }
            }
        }
    ]
    )

Message sizes:

db.message.aggregate(
    [
        {
            $project: {
                currencyPair: 1,
                exchange: 1,
                length: {
                    $strLenCP: "$message"
                }
            }
        },
        {
            $group:{
                _id: {
                    exchange: "$exchange",
                    currencypair:"$currencyPair"
                },
                "avg message" : {
                    $avg : "$length"
                },
                "max message" : {
                    $max : "$length"
                },
                "min message" : {
                    $min : "$length"
                },
                "#messages" : {
                    $sum:1
                },
                "volume" : {
                    $sum : "$length"
                }
            }
        }
    ]
    )

Build Application

You can build the application locally via a maven command:

target meaning
mvn clean clean environment
mvn install build self executable JAR files
mvn install -P docker build docker images
mvn versions:display-property-updates check if latest versions of libraries are used.

To use the created local docker image, replace ghcr.io/buildingsoftwareblocks/withlocalin the used docker-compose.yml file.

To make sure that you have the latest images, use: docker compose --profile=full pull.

The profiles can be used in every call --profile=full for example, or by setting the environment variable COMPOSE_PROFILES.

For example in Window/Powershell: $Env:COMPOSE_PROFILES="full".

TODO List

  • 1 Exchange, 1 Currency pair connected
  • multiple services connected via Kafka
  • build pipeline
  • store events in database
  • multiple exchanges / multiple Currency pairs
  • order book analysis module
  • simple arbitrage
  • scalable backend module
  • scalable frontend module
  • scalable docker frontend
  • scalable analysis module
  • better logging
  • triangular Arbitrage
  • better frontend GUI
  • binary messages in Kafka
  • more robust error handling
  • run on Kubernetes
  • ElasticSearch to version 8 (SB3 supported version)

Related

Background Information