CoinCap ETL Project

This is a simple data pipeline extracting cryptocurrency data from CoinCap API utilising tools such as Docker, cron, AWS EC2, PostgreSQL, and Metabase.

It is based on a project by Joseph Machado. I've made various amendments, including using a different endpoint and extraction script, updating containers to allow for data persistence, and documenting some design decisions.

Motivation

I wanted to create a simple ETL which would help develop my skills and knowledge further. I also have a passing interest in crypto, and the CoinCap API was a good place to explore this.

Architecture

1. Extract data using CoinCap API
2. Load data into PostgreSQL
3. Connect with Google Data Studio dashboard
4. Orchestrate using cron
5. Run with Docker
6. Data visualisation with Metabase

Output

The Pipeline

1. Setup ~ step-by-step instructions to setup pipeline.

2. Design ~ pipeline description and design decisions.

3. Improvements ~ listing of improvements I plan to make.