Skip to content

Latest commit

 

History

History
72 lines (47 loc) · 2.08 KB

README.md

File metadata and controls

72 lines (47 loc) · 2.08 KB
Raw

Dockerized Apache Druid cluster

This project demostrates how you can setup a Dockerized example/development Apache Druid cluster.

The cluster is being composed of the following components:

  • S3 Compatible Object Storage MinIO for Deep storage

  • PostgreSQL for metadata storage

  • Zookeeper for internal service discovery, coordination, and leader election

  • Apache Druid platform:

    • Middle Manager to handle the ingestion of data into the cluster
    • Historical to handle the storage and querying on “historical” data
    • Broker to receive queries from external clients
    • Coordinator to assign segments to Historical nodes
    • Overlord to assign ingestion tasks to Middle Managers and to coordinate segment publishing
    • Router provides a unified API gateway in front of Brokers, Overlords and Coordinators

Instructions to build Druid image

make image

or by using docker-compose

docker-compose build

You can also specify the version of Druid to build, for example:

make DRUID_VERSION=0.14.1-incubating image

or by using docker-compose

docker-compose build --build-arg ARG_DRUID_VERSION=0.14.1-incubating

Run the cluster

docker-compose up

or to run in the backgroumd:

docker-compose up -d

After a while the Druid console should be available in http://localhost:8888

Load example data

For example data we are using a subset of the NYC Taxi & Limousine Commission - Trip Record Data, specifically from months 2015-01 to 2015-03.

cd dataset
./03-load_to_druid.sh

Please note that you can download data for different months and adjust the sample size by adjusting the parameters of ./dataset/01-download.sh and ./dataset/02-create_sample_tripdata.sh.

The schema of the dataset and the indexing task is being defined in ./dataset/yellow_tripdata-index.json

...enjoy :)