ELK Stack of the Human Intracranial EEG Platform

This project builds on Elastic stack (ELK) on Docker, a project which provides a facilitated way of running the Elastic / Logstash / Kibana stack with TLS using Docker and Docker Compose. It is distributed under the MIT license.

It differs, however, in the following:

• Certificate files for Kibana are now trusted, being generated by Let's Encrypt using certbot tool.
• A .env.template is provided instead of the original .env. This guarantees that we do not overwrite it if we pull new changes from github.
• The use of environment variables have been reviewed in order to reduce the process of adapting the repo for installation and deployment, to only the edition of the main .env.
• • A tls/instances.yml.template is provided instead of the original tls/instances.yml file, which is now automatically generated by reading the URL (Hostname) and IP address of the stack from new variables of the .env.
• Creation of a Makefile with commands to support the generation of the certificates, and the installation, deployment, and re-deployment of the stack.
• This new README has been created. The original README has been kept and is still available here.

Installation Guidelines

First ELK Stack Deployment

If you are starting from scratch, please make sure the following dependencies are installed:

• Docker / Docker compose: Software containerization engine
• certbot: Tool to generate Let's Encrypt Trusted SSL certificates
• make: Tool to ease the installation and deployment

This should be achieved on Ubuntu will the following commands:

sudo apt-get update
# Setup the docker debian repository
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
# Install Docker tools
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# Launch docker at reboot
sudo systemctl enable docker
# Add user to docker and take change into account
sudo adduser $USER docker
su - $(USER)
# Install make and certbot
sudo apt-get install make certbot

Then, please follow these steps:

1. Clone the repo:

   git clone https://github.com/HIP-infrastructure/docker-elk.git

2. Fetch and checkout the tls-8-6 branch, if necessary:

   git fetch
   git checkout –track origin/tls-8-6

3. Copy .env.template as .env and edit or use an existing one

4. Make the full deployment from scratch:

   make deploy-with-certbot

   First, this will make sure the stack is down and the elasticsearch volume is removed. Then, this will start by calling certbot, where you will be prompted to enter different source of information to generate Let's encrypt SSL certificate files used for kibana. Then, this will generate the rest of the SSL certificates with the elasticsearch certutil tool, and replace the ones of kibana by the ones generated by certbot. Finally, when all certificates are ready, the stack is deployed in detached mode (e.g. docker compose up -d).

   You can see the logs of the different containers of the stacks by running:

   docker logs docker-elk-setup-1
   docker logs docker-elk-elasticsearch-1
   docker logs docker-elk-kibana-1

   To see the whole status of the ELK stack:

   docker-compose ps

ELK Stack Redeployment

As soon as the ELK stack has been deployed a first, the typical command to redeploy it is:

make redeploy

This command will first make sure that all containers of the stacks are down, but will keep the existing elasticsearch volume to make the data persistent. Then, it will skip certbot but will regenerate the rest of the SSL certificates with the elasticsearch certutil tool and the symlinks for kibana. Finally, when all certificates are ready, the stack is deployed in detached mode (e.g. docker compose up -d).

Alternatively, if you wish to reset your elasticsearch instance, you could use:

make deploy

This command will proceed to the same workflow provided by make redeploy command, except that it will remove the elastic volume. This acts as a reset, which loose all existing indices and indexed datasets. In addition, setup with creation of users / roles is run again.

Wish to launch certbot?

This can be achieved by the following command:

make certbot

Want to see the list of available make commands?

Command:

make

Output:

stack-down-except-volume       Make the stack down (except volume)
stack-down-all                 Make the stack down and removing volume
generate_tls_instances         Fill hostname and IP with values of environment variables
cert                           Generare initial certificates with "docker-compose up tls"
certbot                        Generare CA trusted certificates for ELK_HOSTNAME"
stack-up                       Deploy ELK stack
clean-cert-all                 Remove whole directory with certificates
clean-kibana-cert              Clean kibana SSL certificate and key files
certbot-kibana-symlinks        Make symlinks to certificates generated by certbot for kibana.
deploy-with-certbot            Full deployment from scratch. It calls certbot to generate a CA trusted SSL certificate.
deploy                         Deployment of ELK stack with deletion of volume. This acts as resetting elasticsearch.
redeploy                       Typical redeployment of ELK stack where we want to keep the volume to make elasticsearch data persistent.
help                           List available tasks on this project