Tech Stack

The scope of this project was limited due to the GCP free trial constraints.

Pre-requisites

1. Set environment variables for the following:
   1. WEATHER_API - Visual Crossing API Key
   2. GCP_GCS_BUCKET - The name of your GCP GCS Bucket, can be found via GCP Console
   3. GCP_PROJECT_ID - GCP Project ID, can be found via GCP Console

Setup

Airflow Setup with Docker, through official guidelines

Execution

1. Build the image (only first-time, or when there's any change in the Dockerfile, takes ~15 mins for the first-time):

   docker-compose build

   then

   docker-compose up

2. In another terminal, run docker-compose ps to see which containers are up & running (there should be 7, matching with the services in your docker-compose file).

3. Login to Airflow web UI on localhost:8080 with default creds: airflow/airflow

4. Run your DAG on the Web Console.

5. When finished or to shut down the container:

   docker-compose down

   To stop and delete containers, delete volumes with database data, and download images, run:

   docker-compose down --volumes --rmi all
   

   or

   docker-compose down --volumes --remove-orphans
   

DAG Structure