Skip to content

Latest commit

 

History

History
59 lines (39 loc) · 1.54 KB

README.md

File metadata and controls

59 lines (39 loc) · 1.54 KB

Using Redpanda to Track Cryptocurrency Price Trends

This repo polls Coinbase API for various cryptocurrency prices and uses Redpanda to calculate moving average prices and stores the data in a timeseries database, QuestDB, for further analysis.

Prerequisites

  • Docker (min of 4GB memory)
  • Python 3.7+

Note: the Kafka Connect image is compiled for AMD64 architecture. While Docker may utilize Rosetta 2 to run AMD64 images on ARM64 architectures (e.g. Mac M1), it may have degraded performance.

Structure

  • docker-compose: holds docker-compose file to start Redpanda, QuestDB, and JSON file to initialize Kafka Connect
  • docker: Dockerfile to build Kafka Connect image (if you wish to build locally or extend)
  • Python files:
    • config.py: specify polling frequency, cryptocurrency
    • getData.py: polls Coinbase API and publishes records to Kafka
    • kafkaHelper.py: wrapper around kafka-python lib
    • movingAverage.py: calculates a moving average per cryptocurrency

Quickstart

Redpanda/QuestDB Setup

Start up the Redpanda/QuestDB stack:

cd docker-compose
docker-compose up -d

Wait until all the components are healthy (look at Kafka Connect container logs).

Post postgres-sink connector schemas to Kafka Connect:

docker-compose/add_connectors.sh

Python Setup

Install the necessary packages:

pip install -r requirements.txt

Run the script to poll Coinbase API:

python getData.py

(Optional): calculate moving averages

python movingAverage.py