diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 819c5416dd..be4a45ab63 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -74,7 +74,7 @@ The following section explains various suggestions and procedures to note during It's key to get familiarized with the style guide and mechanics of Thanos, especially if your contribution touches more than one component of the Thanos distributed system. We recommend: -* Reading the [getting started docs](docs/getting-started.md) and working through them. +* Reading the [getting started docs](docs/getting-started.md) and working through them, or alternatively working through the [Thanos tutorial](https://killercoda.com/thanos). * Familiarizing yourself with our [coding style guidelines.](docs/contributing/coding-style-guide.md). * Familiarizing yourself with the [Makefile](Makefile) commands, for example `format`, `build`, `proto`, `docker` and `test`. `make help` will print most of available commands with relevant details. * Spin up a prebuilt dev environment using Gitpod.io [![Gitpod Ready-to-Code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/thanos-io/thanos) diff --git a/docs/getting-started.md b/docs/getting-started.md index 2f6c64a4f3..38dda21708 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -79,6 +79,7 @@ If you want to add yourself to this list, let us know! ## Deploying Thanos +* [WIP] Detailed, free, in-browser interactive tutorial [as Killercoda Thanos Course](https://killercoda.com/thanos/) * [Quick Tutorial](quick-tutorial.md) on Thanos website. ## Operating diff --git a/docs/quick-tutorial.md b/docs/quick-tutorial.md index fcb9a7f0b2..e77caf0fc3 100644 --- a/docs/quick-tutorial.md +++ b/docs/quick-tutorial.md @@ -1,5 +1,9 @@ # Quick Tutorial +Feel free to check the free, in-browser interactive tutorial [as Killercoda Thanos Course](https://killercoda.com/thanos). We will be progressively updating our Katacoda Course with more scenarios. + +On top of this feel free to go through our tutorial presented here: + ## Prometheus Thanos is based on Prometheus. With Thanos you use more or less Prometheus features depending on the deployment model, however Prometheus always stays as integral foundation for *collecting metrics* and alerting using local data. diff --git a/docs/release-process.md b/docs/release-process.md index 98a6bd3d95..84415b0f11 100644 --- a/docs/release-process.md +++ b/docs/release-process.md @@ -104,7 +104,7 @@ Feel free to mimic following PR: https://github.com/thanos-io/thanos/pull/3861 6. *(Applies only to minor, non-`rc` release)* Update tutorials: - 1. Update the Thanos version used in the [tutorials](../tutorials) manifests. + 1. Update the Thanos version used in the [tutorials](https://github.com/thanos-io/tutorials) manifests to use the latest version. 2. In case of any breaking changes or necessary updates adjust the manifests so the tutorial stays up to date. 3. Update the [scripts/quickstart.sh](https://github.com/thanos-io/thanos/blob/main/scripts/quickstart.sh) script if needed. diff --git a/tutorials/katacoda/README.md b/tutorials/katacoda/README.md deleted file mode 100644 index 5b113487af..0000000000 --- a/tutorials/katacoda/README.md +++ /dev/null @@ -1,7 +0,0 @@ -# Katacoda tutorials - -Definitions of our courses for Katacoda. - -## Development - -We are currently in process of moving to an alternative interactive tutorial platform, since Katacoda has been decomissioned. Follow [this issue](https://github.com/thanos-io/thanos/issues/5385) to learn more. diff --git a/tutorials/katacoda/thanos-pathway.json b/tutorials/katacoda/thanos-pathway.json deleted file mode 100644 index 5f78305e33..0000000000 --- a/tutorials/katacoda/thanos-pathway.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "title": "Learn Thanos", - "description": "Introduction, Tips and Advanced Tutorials for Thanos: the CNCF, Global, Scalable System for Prometheus Metrics with cheap, Long-term Storage Capabilities. Course version: v0.4", - "icon": "fa-thanos", - "courses": [ - { - "course_id": "1-globalview", - "title": "Intro: Global View and seamless HA for Prometheus", - "description": "Learn how to easily transform Prometheus into centralized, highly available monitoring using Thanos." - }, - { - "course_id": "2-lts", - "title": "Intro: Downsampling and unlimited metric retention for Prometheus", - "description": "Learn how to extend your metric retention in a cheap and easy way with Thanos." - }, - { - "course_id": "3-receiver", - "title": "Intermediate: Streaming metrics from remote source with Thanos Receive", - "description": "Learn how to ingest and then query metrics data from egress-only sources with Thanos Receive." - }, - { - "course_id": "4-receiver-agent", - "title": "Bonus: Using Prometheus Agent for streaming metrics to Thanos Receive", - "description": "Learn how to use Prometheus in agent mode to efficiently stream metrics out of cluster." - }, - { - "course_id": "6-query-caching", - "title": "Advanced: Querying with low tail-latency and low cost - Query caching with Thanos", - "description": "Learn how to introduce query caching using Query Frontend with Thanos." - }, - { - "course_id": "7-multi-tenancy", - "title": "Advanced: Achieving Multi-Tenancy with Thanos", - "description": "Extend your Prometheus setup for Multiple Teams use with Thanos." - }, - { - "course_id": "x-playground", - "title": "Playground: All-in, Interactive Thanos un-guided Demo", - "description": "Demo playground that spins all the components from previous courser together." - }, - { - "course_id": "x-more-to-come", - "title": "", - "description": "Anything you want to like to see? Or you want to help? Let us know!" - } - ] -} diff --git a/tutorials/katacoda/thanos/1-globalview/courseBase.sh b/tutorials/katacoda/thanos/1-globalview/courseBase.sh deleted file mode 100644 index 90675cb804..0000000000 --- a/tutorials/katacoda/thanos/1-globalview/courseBase.sh +++ /dev/null @@ -1,4 +0,0 @@ -#!/usr/bin/env bash - -docker pull quay.io/prometheus/prometheus:v2.16.0 -docker pull quay.io/thanos/thanos:v0.26.0 diff --git a/tutorials/katacoda/thanos/1-globalview/finish.md b/tutorials/katacoda/thanos/1-globalview/finish.md deleted file mode 100644 index f2384742e8..0000000000 --- a/tutorials/katacoda/thanos/1-globalview/finish.md +++ /dev/null @@ -1,16 +0,0 @@ -# Summary - -Congratulations! πŸŽ‰πŸŽ‰πŸŽ‰ -You completed our very first Thanos tutorial. Let's summarize what we learned: - -* The most basic installation of Thanos with Sidecars and Querier allows global view for Prometheus queries. -* Querier operates on `StoreAPI` gRPC API. It does not know if it's Prometheus, OpenTSDB, another Querier or any other storage, as long as API is implemented. -* With Thanos you can (and it's recommended to do so!) run multi-replica Prometheus servers. Thanos Querier `--query.replica-label` flag controls this behaviour. -* Sidecar allows to dynamically reload configuration for Prometheus and recording & alerting rules in Prometheus. - -See next courses for other tutorials about different deployment models and more advanced features of Thanos! - -### Feedback - -Do you see any bug, typo in the tutorial or you have some feedback for us? -Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io diff --git a/tutorials/katacoda/thanos/1-globalview/index.json b/tutorials/katacoda/thanos/1-globalview/index.json deleted file mode 100644 index 993a891c3e..0000000000 --- a/tutorials/katacoda/thanos/1-globalview/index.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "title": "Intro: Global View and seamless HA for Prometheus", - "description": "Learn how to easily transform Prometheus into centralized, highly available monitoring using Thanos.", - "difficulty": "Beginner", - "time": "10-15 Minutes", - "details": { - "steps": [ - { - "title": "Initial Prometheus Setup", - "text": "step1.md", - "verify": "step1-verify.sh", - "answer": "step1-answer.md" - }, - { - "title": "Thanos Sidecars", - "text": "step2.md", - "verify": "step2-verify.sh" - }, - { - "title": "Thanos Querier", - "text": "step3.md", - "verify": "step3-verify.sh" - } - ], - "intro": { - "text": "intro.md", - "courseData": "courseBase.sh", - "credits": "https://thanos.io" - }, - "finish": { - "text": "finish.md", - "credits": "test" - } - }, - "files": [ - "prometheus0_eu1.yml", - "prometheus0_us1.yml", - "prometheus1_us1.yml" - ], - "environment": { - "uilayout": "editor-terminal", - "uisettings": "yaml", - "showdashboard": true, - "dashboards": [ - {"name": "Prometheus 0 EU1", "port": 9090}, - {"name": "Prometheus 0 US1", "port": 9091}, - {"name": "Prometheus 1 US1", "port": 9092}, - {"name": "Thanos Query", "port": 29090} - ] - }, - "backend": { - "imageid": "docker-direct" - } -} diff --git a/tutorials/katacoda/thanos/1-globalview/intro.md b/tutorials/katacoda/thanos/1-globalview/intro.md deleted file mode 100644 index fb74fb7838..0000000000 --- a/tutorials/katacoda/thanos/1-globalview/intro.md +++ /dev/null @@ -1,27 +0,0 @@ -# Welcome to the Thanos introduction! - -[Thanos](https://thanos.io) is a set of components that can be composed into a highly available metric system with unlimited storage capacity. -It can be added seamlessly on top of existing Prometheus deployments. - -Thanos provides a global query view, data backup, and historical data access as its core features. -All three features can be run independently of each other. This allows you to have a subset of Thanos features ready for -immediate benefit or testing, while also making it flexible for gradual adoption in more complex environments. - -Thanos will work in cloud native environments like Kubernetes as well as more traditional ones. However, this course uses docker -containers which will allow us to use pre-built docker images available [here](https://quay.io/repository/thanos/thanos) - -This tutorial will take us from transforming vanilla Prometheus to basic Thanos deployment enabling: - -* Reliable querying multiple Prometheus instances from single [Prometheus API endpoint](https://prometheus.io/docs/prometheus/latest/querying/api/#expression-queries). -* Seamless handling of Highly Available Prometheus (multiple replicas) - -Let's jump in! πŸ€“ - -### Feedback - -Do you see any bug, typo in the tutorial or you have some feedback for us? -Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io - -### Contributed by: - -* Bartek [@bwplotka](https://bwplotka.dev/) \ No newline at end of file diff --git a/tutorials/katacoda/thanos/1-globalview/step1-answer.md b/tutorials/katacoda/thanos/1-globalview/step1-answer.md deleted file mode 100644 index 50c7f5bb2e..0000000000 --- a/tutorials/katacoda/thanos/1-globalview/step1-answer.md +++ /dev/null @@ -1,18 +0,0 @@ -## Answer - -**How many series (metrics) we collect overall on all Prometheus instances we have?** - -How to get this information? As you probably guess it's not straightforward. The current step would be: - -* Query Prometheus-0 EU1 for `prometheus_tsdb_head_series` -* Query Prometheus-0 US1 or Prometheus-1 US1 for `prometheus_tsdb_head_series` -Both holds the same data (number of series for each replica) so we just need to choose available one. -* Sum both results manually. - -As you can see this is not very convenient for both human as well as automation on top of metrics (e.g Alerting). - -The feature we are missing here is called **Global View** and it might be necessary once you scale out Prometheus to multiple instances. - -Great! We have now running 3 Prometheus instances. - -In the next steps we will learn how we can install Thanos on top of our initial Prometheus setup to solve problems shown in the challenge. diff --git a/tutorials/katacoda/thanos/1-globalview/step1-verify.sh b/tutorials/katacoda/thanos/1-globalview/step1-verify.sh deleted file mode 100644 index 58a40c3d36..0000000000 --- a/tutorials/katacoda/thanos/1-globalview/step1-verify.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/usr/bin/env bash - -curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:9092/metrics >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/1-globalview/step1.md b/tutorials/katacoda/thanos/1-globalview/step1.md deleted file mode 100644 index 1f99d997d6..0000000000 --- a/tutorials/katacoda/thanos/1-globalview/step1.md +++ /dev/null @@ -1,178 +0,0 @@ -# Step 1 - Start initial Prometheus servers - -Thanos is meant to scale and extend vanilla Prometheus. This means that you can gradually, without disruption, deploy Thanos on top of your existing Prometheus setup. - -Let's start our tutorial by spinning up three Prometheus servers. Why three? -The real advantage of Thanos is when you need to scale out Prometheus from a single replica. Some reason for scale-out might be: - -* Adding functional sharding because of metrics high cardinality -* Need for high availability of Prometheus e.g: Rolling upgrades -* Aggregating queries from multiple clusters - -For this course, let's imagine the following situation: - -![initial-case](https://docs.google.com/drawings/d/e/2PACX-1vQ5n5dAJSJPRXWA9INOViJJy9Ci6TUwlCrDv7_TtV9vE41rFOpg26V3jQv9gf1NQjVWSFyauG5XgzOW/pub?w=1061&h=604) - -1. We have one Prometheus server in some `eu1` cluster. -2. We have 2 replica Prometheus servers in some `us1` cluster that scrapes the same targets. - -Let's start this initial Prometheus setup for now. - -## Prometheus Configuration Files - -Now, we will prepare configuration files for all Prometheus instances. - -Click `Copy To Editor` for each config to propagate the configs to each file. - -First, for the EU Prometheus server that scrapes itself: - -
-global:
-  scrape_interval: 15s
-  evaluation_interval: 15s
-  external_labels:
-    cluster: eu1
-    replica: 0
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9090']
-
- -For the second cluster we set two replicas: - -
-global:
-  scrape_interval: 15s
-  evaluation_interval: 15s
-  external_labels:
-    cluster: us1
-    replica: 0
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9091','127.0.0.1:9092']
-
- -
-global:
-  scrape_interval: 15s
-  evaluation_interval: 15s
-  external_labels:
-    cluster: us1
-    replica: 1
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9091','127.0.0.1:9092']
-
- -**NOTE** : Every Prometheus instance must have a globally unique set of identifying labels. These labels are important as they represent certain "stream" of data (e.g in the form of TSDB blocks). Within those exact external labels, the compactions and downsampling are performed, Querier filters its store APIs, further sharding option, deduplication, and potential multi-tenancy capabilities are available. Those are not easy to edit retroactively, so it's important to provide a compatible set of external labels as in order to for Thanos to aggregate data across all the available instances. - -## Starting Prometheus Instances - -Let's now start three containers representing our three different Prometheus instances. - -Please note the extra flags we're passing to Prometheus: - -* `--web.enable-admin-api` allows Thanos Sidecar to get metadata from Prometheus like `external labels`. -* `--web.enable-lifecycle` allows Thanos Sidecar to reload Prometheus configuration and rule files if used. - -Execute following commands: - -### Prepare "persistent volumes" - -``` -mkdir -p prometheus0_eu1_data prometheus0_us1_data prometheus1_us1_data -```{{execute}} - -### Deploying "EU1" - -``` -docker run -d --net=host --rm \ - -v $(pwd)/prometheus0_eu1.yml:/etc/prometheus/prometheus.yml \ - -v $(pwd)/prometheus0_eu1_data:/prometheus \ - -u root \ - --name prometheus-0-eu1 \ - quay.io/prometheus/prometheus:v2.14.0 \ - --config.file=/etc/prometheus/prometheus.yml \ - --storage.tsdb.path=/prometheus \ - --web.listen-address=:9090 \ - --web.external-url=https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com \ - --web.enable-lifecycle \ - --web.enable-admin-api && echo "Prometheus EU1 started!" -```{{execute}} - -NOTE: We are using the latest Prometheus image so we can take profit from the latest remote read protocol. - -### Deploying "US1" - -``` -docker run -d --net=host --rm \ - -v $(pwd)/prometheus0_us1.yml:/etc/prometheus/prometheus.yml \ - -v $(pwd)/prometheus0_us1_data:/prometheus \ - -u root \ - --name prometheus-0-us1 \ - quay.io/prometheus/prometheus:v2.14.0 \ - --config.file=/etc/prometheus/prometheus.yml \ - --storage.tsdb.path=/prometheus \ - --web.listen-address=:9091 \ - --web.external-url=https://[[HOST_SUBDOMAIN]]-9091-[[KATACODA_HOST]].environments.katacoda.com \ - --web.enable-lifecycle \ - --web.enable-admin-api && echo "Prometheus 0 US1 started!" -```{{execute}} - -and - -``` -docker run -d --net=host --rm \ - -v $(pwd)/prometheus1_us1.yml:/etc/prometheus/prometheus.yml \ - -v $(pwd)/prometheus1_us1_data:/prometheus \ - -u root \ - --name prometheus-1-us1 \ - quay.io/prometheus/prometheus:v2.14.0 \ - --config.file=/etc/prometheus/prometheus.yml \ - --storage.tsdb.path=/prometheus \ - --web.listen-address=:9092 \ - --web.external-url=https://[[HOST_SUBDOMAIN]]-9092-[[KATACODA_HOST]].environments.katacoda.com \ - --web.enable-lifecycle \ - --web.enable-admin-api && echo "Prometheus 1 US1 started!" -```{{execute}} - -## Setup Verification - -Once started you should be able to reach all of those Prometheus instances: - -* [Prometheus-0 EU1](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/) -* [Prometheus-1 US1](https://[[HOST_SUBDOMAIN]]-9091-[[KATACODA_HOST]].environments.katacoda.com/) -* [Prometheus-2 US1](https://[[HOST_SUBDOMAIN]]-9092-[[KATACODA_HOST]].environments.katacoda.com/) - -## Additional info - -Why would one need multiple Prometheus instances? - -* High Availability (multiple replicas) -* Scaling ingestion: Functional Sharding -* Multi cluster/environment architecture - -## Problem statement: Global view challenge - -Let's try to play with this setup a bit. You are free to query any metrics, however, let's try to fetch some certain information from -our multi-cluster setup: **How many series (metrics) we collect overall on all Prometheus instances we have?** - -Tip: Look for `prometheus_tsdb_head_series` metric. - -πŸ•΅οΈβ€β™‚οΈ - -Try to get this information from the current setup! - -To see the answer to this question click SHOW SOLUTION below. - -## Next - -Great! We have now running 3 Prometheus instances. - -In the next steps, we will learn how we can install Thanos on top of our initial Prometheus setup to solve problems shown in the challenge. diff --git a/tutorials/katacoda/thanos/1-globalview/step2-verify.sh b/tutorials/katacoda/thanos/1-globalview/step2-verify.sh deleted file mode 100644 index 5ebb30ba3f..0000000000 --- a/tutorials/katacoda/thanos/1-globalview/step2-verify.sh +++ /dev/null @@ -1,11 +0,0 @@ -#!/usr/bin/env bash - -curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:9092/metrics >/dev/null || exit 1 - -curl -s 127.0.0.1:19090/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:19091/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:19092/metrics >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/1-globalview/step2.md b/tutorials/katacoda/thanos/1-globalview/step2.md deleted file mode 100644 index ecf7fe8575..0000000000 --- a/tutorials/katacoda/thanos/1-globalview/step2.md +++ /dev/null @@ -1,163 +0,0 @@ -# Step 2 - Installing Thanos sidecar - -Let's take the setup from the previous step and seamlessly install Thanos to add Global View with HA handling feature. - -## Thanos Components - -Thanos is a single Go binary capable to run in different modes. Each mode represents a different -component and can be invoked in a single command. - -Let's take a look at all the Thanos commands: - -``` -docker run --rm quay.io/thanos/thanos:v0.26.0 --help -```{{execute}} - -You should see multiple commands that solves different purposes. - -In this step we will focus on `thanos sidecar`: - -``` - sidecar [] - sidecar for Prometheus server -``` - -## Sidecar - -Sidecar as the name suggests should be deployed together with Prometheus. Sidecar has multiple features: - -* It exposes Prometheus metrics as a common Thanos [StoreAPI](https://thanos.io/tip/thanos/integrations.md/#storeapi). StoreAPI -is a generic gRPC API allowing Thanos components to fetch metrics from various systems and backends. -* It is essentially in further long term storage options described in [next]() courses. -* It is capable to watch for configuration and Prometheus rules (alerting or recording) and notify Prometheus for dynamic reloads: - * optionally substitute with environment variables - * optionally decompress if gzipp-ed - -You can read more about sidecar [here](https://thanos.io/tip/components/sidecar.md/) - -## Installation - -To allow Thanos to efficiently query Prometheus data, let's install sidecar to each Prometheus instances we deployed in the previous step as shown below: - -![sidecar install](https://docs.google.com/drawings/d/e/2PACX-1vRHlEJd9OVH80hczxkqZKYDVXxwugX55VWKtLJhS6R7D3BbmkBW9qGyyD4JyLbAe9CK9EzvurWTagTR/pub?w=1058&h=330) - -For this setup the only configuration required for sidecar is the Prometheus API URL and access to the configuration file. -Former will allow us to access Prometheus metrics, the latter will allow sidecar to reload Prometheus configuration in runtime. - -Click snippets to add sidecars to each Prometheus instance. - -### Adding sidecar to "EU1" Prometheus - -``` -docker run -d --net=host --rm \ - -v $(pwd)/prometheus0_eu1.yml:/etc/prometheus/prometheus.yml \ - --name prometheus-0-sidecar-eu1 \ - -u root \ - quay.io/thanos/thanos:v0.26.0 \ - sidecar \ - --http-address 0.0.0.0:19090 \ - --grpc-address 0.0.0.0:19190 \ - --reloader.config-file /etc/prometheus/prometheus.yml \ - --prometheus.url http://127.0.0.1:9090 && echo "Started sidecar for Prometheus 0 EU1" -```{{execute}} - -### Adding sidecars to each replica of Prometheus in "US1" - -``` -docker run -d --net=host --rm \ - -v $(pwd)/prometheus0_us1.yml:/etc/prometheus/prometheus.yml \ - --name prometheus-0-sidecar-us1 \ - -u root \ - quay.io/thanos/thanos:v0.26.0 \ - sidecar \ - --http-address 0.0.0.0:19091 \ - --grpc-address 0.0.0.0:19191 \ - --reloader.config-file /etc/prometheus/prometheus.yml \ - --prometheus.url http://127.0.0.1:9091 && echo "Started sidecar for Prometheus 0 US1" -```{{execute}} - -``` -docker run -d --net=host --rm \ - -v $(pwd)/prometheus1_us1.yml:/etc/prometheus/prometheus.yml \ - --name prometheus-1-sidecar-us1 \ - -u root \ - quay.io/thanos/thanos:v0.26.0 \ - sidecar \ - --http-address 0.0.0.0:19092 \ - --grpc-address 0.0.0.0:19192 \ - --reloader.config-file /etc/prometheus/prometheus.yml \ - --prometheus.url http://127.0.0.1:9092 && echo "Started sidecar for Prometheus 1 US1" -```{{execute}} - -## Verification - -Now, to check if sidecars are running well, let's modify Prometheus scrape configuration to include our added sidecars. - -As always click `Copy To Editor` for each config to propagate the configs to each file. - -Note that only thanks to the sidecar, all those changes will be immediately reloaded and updated in Prometheus! - -
-global:
-  scrape_interval: 15s
-  evaluation_interval: 15s
-  external_labels:
-    cluster: eu1
-    replica: 0
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9090']
-  - job_name: 'sidecar'
-    static_configs:
-      - targets: ['127.0.0.1:19090']
-
- -
-global:
-  scrape_interval: 15s
-  evaluation_interval: 15s
-  external_labels:
-    cluster: us1
-    replica: 0
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9091','127.0.0.1:9092']
-  - job_name: 'sidecar'
-    static_configs:
-      - targets: ['127.0.0.1:19091','127.0.0.1:19092']
-
- -
-global:
-  scrape_interval: 15s
-  evaluation_interval: 15s
-  external_labels:
-    cluster: us1
-    replica: 1
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9091','127.0.0.1:9092']
-  - job_name: 'sidecar'
-    static_configs:
-      - targets: ['127.0.0.1:19091','127.0.0.1:19092']
-
- -Now you should see new, updated configuration on each Prometheus. For example here in [Prometheus 0 EU1 /config](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/config). -In the same time [`up`](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/graph?g0.expr=up&g0.tab=1) should show `job=sidecar` metrics. - -Since now Prometheus has access to sidecar metrics we can query for [`thanos_sidecar_prometheus_up`](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/graph?g0.expr=thanos_sidecar_prometheus_up&g0.tab=1) -to check if sidecar has access to Prometheus. - -## Next - -Great! Now you should have setup deployed as in the presented image: - -![sidecar install](https://docs.google.com/drawings/d/e/2PACX-1vRHlEJd9OVH80hczxkqZKYDVXxwugX55VWKtLJhS6R7D3BbmkBW9qGyyD4JyLbAe9CK9EzvurWTagTR/pub?w=1058&h=330) - -In the next step, we will add a final component allowing us to fetch Prometheus metrics from a single endpoint. diff --git a/tutorials/katacoda/thanos/1-globalview/step3-verify.sh b/tutorials/katacoda/thanos/1-globalview/step3-verify.sh deleted file mode 100644 index 6eb40ec5c5..0000000000 --- a/tutorials/katacoda/thanos/1-globalview/step3-verify.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/usr/bin/env bash - -curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:9092/metrics >/dev/null || exit 1 - -curl -s 127.0.0.1:19090/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:19091/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:19092/metrics >/dev/null || exit 1 - -curl -s 127.0.0.1:29090/metrics >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/1-globalview/step3.md b/tutorials/katacoda/thanos/1-globalview/step3.md deleted file mode 100644 index 3221c23080..0000000000 --- a/tutorials/katacoda/thanos/1-globalview/step3.md +++ /dev/null @@ -1,124 +0,0 @@ -# Step 3 - Adding Thanos Querier - -Thanks to the previous step we have three running Prometheus instances with a sidecar each. In this step we will install -Thanos Querier which will use sidecars and allow querying all metrics from the single place as presented below: - -![with querier](https://docs.google.com/drawings/d/e/2PACX-1vSB9gN82px0lxk9vN6wNw3eXr8Z0EVROW3xubsq7tgjbx_nXsoZ02ElzvxeDevyjGPWv-f9Gie0NeNz/pub?w=926&h=539) - -But before that, let's take a closer look at what the Querier component does: - -## Querier - -The Querier component (also called "Query") is essentially a vanilla PromQL Prometheus engine that fetches the data from any service -that implements Thanos [StoreAPI](https://thanos.io/tip/thanos/integrations.md/#storeapi). This means that Querier exposes the Prometheus HTTP v1 API to query the data in a common PromQL language. -This allows compatibility with Grafana or other consumers of Prometheus' API. - -Additionally, Querier is capable of deduplicating StoreAPIs that are in the same HA group. We will see how it -looks in practice later on. - -You can read more about Thanos Querier [here](https://thanos.io/tip/components/query.md/) - -## Deploying Thanos Querier - -Let's now start the Query component. As you remember [Thanos sidecar](https://thanos.io/tip/components/query.md/) exposes `StoreAPI` -so we will make sure we point the Querier to the gRPC endpoints of all our three sidecars: - -Click the snippet below to start the Querier. - -``` -docker run -d --net=host --rm \ - --name querier \ - quay.io/thanos/thanos:v0.26.0 \ - query \ - --http-address 0.0.0.0:29090 \ - --query.replica-label replica \ - --store 127.0.0.1:19190 \ - --store 127.0.0.1:19191 \ - --store 127.0.0.1:19192 && echo "Started Thanos Querier" -```{{execute}} - -## Setup verification - -Thanos Querier exposes very similar UI to the Prometheus, but on top of many `StoreAPIs you wish to connect to. - -To check if the Querier works as intended let's look on [Querier UI `Store` page](https://[[HOST_SUBDOMAIN]]-29090-[[KATACODA_HOST]].environments.katacoda.com/stores). - -This should list all our three sidecars, including their external labels. - -## Global view - Not challenging anymore? - -Now, let's get back to our challenge from step 1, so finding the answer to **How many series (metrics) we collect overall on all Prometheus instances we have?** - -With the querier this is now super simple. - -It's just enough to query Querier for `sum(prometheus_tsdb_head_series)` - -You should see the single value representing the number of series scraped in both clusters in the current mode. - -If we query `prometheus_tsdb_head_series` we will see that we have complete info about all three Prometheus instances: - -``` -prometheus_tsdb_head_series{cluster="eu1",instance="127.0.0.1:9090",job="prometheus"} -prometheus_tsdb_head_series{cluster="us1",instance="127.0.0.1:9091",job="prometheus"} -prometheus_tsdb_head_series{cluster="us1",instance="127.0.0.1:9092",job="prometheus"} -``` - -## Handling of Highly Available Prometheus - -Now, as you remember we configured Prometheus 0 US1 and Prometheus 1 US1 to scrape the same things. We also connect Querier -to both, so how Querier knows what is an HA group? - -Try to query the same query as before: `prometheus_tsdb_head_series` - -Now turn off deduplication (`deduplication` button on Querier UI) and hit `Execute` again. Now you should see 5 results: - -``` -prometheus_tsdb_head_series{cluster="eu1",instance="127.0.0.1:9090",job="prometheus",replica="0"} -prometheus_tsdb_head_series{cluster="us1",instance="127.0.0.1:9091",job="prometheus",replica="0"} -prometheus_tsdb_head_series{cluster="us1",instance="127.0.0.1:9091",job="prometheus",replica="1"} -prometheus_tsdb_head_series{cluster="us1",instance="127.0.0.1:9092",job="prometheus",replica="0"} -prometheus_tsdb_head_series{cluster="us1",instance="127.0.0.1:9092",job="prometheus",replica="1"} -``` - -So how Thanos Querier knows how to deduplicate correctly? - -If we would look again into Querier configuration we can see that we also set `query.replica-label` flag. -This is exactly the label Querier will try to deduplicate by for HA groups. This means that any metric with exactly -the same labels *except replica label* will be assumed as the metric from the same HA group, and deduplicated accordingly. - -If we would open `prometheus1_us1.yml` config file in the editor or if you go to Prometheus 1 US1 [/config](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/config). -you should see our external labels in `external_labels` YAML option: - -```yaml - external_labels: - cluster: us1 - replica: 1 -``` - -Now if we compare to `prometheus0_us1.yaml`: - -```yaml - external_labels: - cluster: us1 - replica: 0 -``` - -We can see that since those two replicas scrape the same targets, any metric will be produced twice. -Once by `replica=1, cluster=us1` Prometheus and once by `replica=0, cluster=us1` Prometheus. If we configure Querier to -deduplicate by `replica` we can transparently handle this High Available pair of Prometheus instances to the user. - -## Production deployment - -Normally Querier runs in some central global location (e.g next to Grafana) with remote access to all Prometheus-es (e.g via ingress, proxies vpn or peering) - -You can also stack (federate) Queriers on top of other Queries, as Query expose `StoreAPI` as well! - -More information about those advanced topics can be found in the next courses that will be added soon. - -## Next - -Awesome! Feel free to play around with the following setup: - -![with querier](https://docs.google.com/drawings/d/e/2PACX-1vSB9gN82px0lxk9vN6wNw3eXr8Z0EVROW3xubsq7tgjbx_nXsoZ02ElzvxeDevyjGPWv-f9Gie0NeNz/pub?w=926&h=539) - -Once done hit `Continue` for summary. diff --git a/tutorials/katacoda/thanos/2-lts/courseBase.sh b/tutorials/katacoda/thanos/2-lts/courseBase.sh deleted file mode 100644 index 3f919c9bc9..0000000000 --- a/tutorials/katacoda/thanos/2-lts/courseBase.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/usr/bin/env bash - -docker pull minio/minio:RELEASE.2019-01-31T00-31-19Z -docker pull quay.io/prometheus/prometheus:v2.20.0 -docker pull quay.io/thanos/thanos:v0.26.0 -docker pull quay.io/thanos/thanosbench:v0.2.0-rc.1 - -mkdir /root/editor diff --git a/tutorials/katacoda/thanos/2-lts/finish.md b/tutorials/katacoda/thanos/2-lts/finish.md deleted file mode 100644 index eee4b96ffc..0000000000 --- a/tutorials/katacoda/thanos/2-lts/finish.md +++ /dev/null @@ -1,17 +0,0 @@ -# Summary - -Congratulations! πŸŽ‰πŸŽ‰πŸŽ‰ -You completed our second Thanos tutorial. Let's summarize what we learned: - -* To preserve the data beyond Prometheus regular retention time, we used an object storage system for backing up our historical data. -* The Thanos Store component acts as a data retrieval proxy for data inside our object storage. -* With Sidecar uploading metric blocks to the object store as soon as it is written to disk, it keeps the β€œscraper” (Prometheus with Thanos Sidecar), lightweight. This simplifies maintenance, cost, and system design. -* Thanos Compactor improved query efficiency and also reduced the required storage size. - -See next courses for other tutorials about different deployment models and more advanced features of Thanos! - -### Feedback - -Do you see any bug, typo in the tutorial or you have some feedback for us? - -let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io \ No newline at end of file diff --git a/tutorials/katacoda/thanos/2-lts/index.json b/tutorials/katacoda/thanos/2-lts/index.json deleted file mode 100644 index 5a665c6358..0000000000 --- a/tutorials/katacoda/thanos/2-lts/index.json +++ /dev/null @@ -1,55 +0,0 @@ -{ - "title": "Intro: Downsampling and unlimited metric retention for Prometheus", - "description": "Learn how to extend your metric retention in a cheap and easy way with Thanos.", - "difficulty": "Moderate", - "details": { - "steps": [ - { - "title": "Configuring Initial Prometheus Server", - "text": "step1.md", - "verify": "step1-verify.sh" - }, - { - "title": "Thanos Sidecars", - "text": "step2.md", - "verify": "step2-verify.sh" - }, - { - "title": "Thanos Store Gateway", - "text": "step3.md", - "answer": "step3-answer.md" - }, - { - "title": "Thanos Compactor", - "text": "step4.md" - } - ], - "intro": { - "text": "intro.md", - "courseData": "courseBase.sh", - "credits": "https://thanos.io" - }, - "finish": { - "text": "finish.md", - "credits": "test" - } - }, - "files": [ - "prometheus0_eu1.yml", - "bucket_storage.yaml" - ], - "environment": { - "uilayout": "editor-terminal", - "uisettings": "yaml", - "uieditorpath": "/root/editor", - "showdashboard": true, - "dashboards": [ - {"name": "Prometheus 0 EU1", "port": 9090}, - {"name": "Minio", "port": 9000}, - {"name": "Thanos Query", "port": 9091} - ] - }, - "backend": { - "imageid": "docker-direct" - } -} diff --git a/tutorials/katacoda/thanos/2-lts/intro.md b/tutorials/katacoda/thanos/2-lts/intro.md deleted file mode 100644 index eeb8e58273..0000000000 --- a/tutorials/katacoda/thanos/2-lts/intro.md +++ /dev/null @@ -1,30 +0,0 @@ -# Intro: Downsampling and unlimited metric retention for Prometheus - -They say that [Thanos](thanos.io) is a set of components that can be composed into a highly available metric system with **unlimited storage capacity** -and that it can be added **seamlessly** on top of existing Prometheus deployments. πŸ€”πŸ€” - -In this course you can experience all of this yourself. - -In this tutorial, you will learn about: - -* How to start uploading your Prometheus data seamlessly to cheap object storage thanks to Thanos sidecar. -* How to further query your data in object storage thanks to Thanos Store Gateway. -* How to query both fresh and older data in easy way through Thanos Querier. - -All of this allows you to keep your metrics in cheap and reliable object storage, allowing virtually unlimited metric retention for Prometheus. - -> NOTE: This course uses docker containers with pre-built Thanos, Prometheus, and Minio Docker images available publicly. -> However, a similar scenario will work with any other deployment method like Kubernetes or systemd, etc. - -### Prerequisites - -Please complete first intro course about GlobalView before jumping into this one! πŸ€— - -### Feedback - -Do you see any bug, typo in the tutorial or you have some feedback for us? -Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io - -### Contributed by: - -* Sonia Singla [@soniasingla](http://github.com/soniasingla) \ No newline at end of file diff --git a/tutorials/katacoda/thanos/2-lts/query.png b/tutorials/katacoda/thanos/2-lts/query.png deleted file mode 100644 index 5561402330..0000000000 Binary files a/tutorials/katacoda/thanos/2-lts/query.png and /dev/null differ diff --git a/tutorials/katacoda/thanos/2-lts/step1-verify.sh b/tutorials/katacoda/thanos/2-lts/step1-verify.sh deleted file mode 100644 index 7e9a0cd6c2..0000000000 --- a/tutorials/katacoda/thanos/2-lts/step1-verify.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/usr/bin/env bash - -curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:19090/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/2-lts/step1.md b/tutorials/katacoda/thanos/2-lts/step1.md deleted file mode 100644 index 3040b4db27..0000000000 --- a/tutorials/katacoda/thanos/2-lts/step1.md +++ /dev/null @@ -1,149 +0,0 @@ -# Step 1 - Initial Prometheus Setup - -In this tutorial, we will mimic the usual state with a Prometheus server running for... a year!. -We will use it to seamlessly backup all old data in the object storage and configure Prometheus for continuous backup mode, which -will allow us to cost-effectively achieve unlimited retention for Prometheus. - -Last but not the least, we will go through setting all up for querying and automated maintenance (e.g compactions, retention and downsampling). - -In order to showcase all of this, let's start with a single cluster setup from the previous course. Let's start this initial Prometheus setup, ready? - -## Generate Artificial Metrics for 1 year - -Actually, before starting Prometheus, let's generate some **artificial data**. You most likely want to learn about Thanos fast, -so you probably don't have months to wait for this tutorial until Prometheus collects the month of metrics, do you? (: - -We will use our handy [thanosbench](https://github.com/thanos-io/thanosbench) project to do so! Let's generate Prometheus -data (in form of TSDB blocks) with just 5 series (gauges) that spans from a year ago until now (-6h)! - -Execute the following command (should take few seconds): - -``` -mkdir -p /root/prom-eu1 && docker run -i quay.io/thanos/thanosbench:v0.2.0-rc.1 block plan -p continuous-365d-tiny --labels 'cluster="eu1"' --max-time=6h | docker run -v /root/prom-eu1:/prom-eu1 -i quay.io/thanos/thanosbench:v0.2.0-rc.1 block gen --output.dir prom-eu1 -```{{execute}} - -On successful block creation you should see following log lines: - -``` -level=info ts=2020-10-20T18:28:42.625041939Z caller=block.go:87 msg="all blocks done" count=13 -level=info ts=2020-10-20T18:28:42.625100758Z caller=main.go:118 msg=exiting cmd="block gen" -``` - -Run the below command to see dozens of generated TSDB blocks: - -``` -ls -lR /root/prom-eu1 -```{{execute}} - -## Prometheus Configuration File - -Here, we will prepare configuration files for the Prometheus instance that will run with our pre-generated data. -It will also scrape our components we will use in this tutorial. - -Click `Copy To Editor` for config to propagate the configs to file. - -
-global:
-  scrape_interval: 5s
-  external_labels:
-    cluster: eu1
-    replica: 0
-    tenant: team-eu # Not needed, but a good practice if you want to grow this to multi-tenant system some day.
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9090']
-  - job_name: 'sidecar'
-    static_configs:
-      - targets: ['127.0.0.1:19090']
-  - job_name: 'minio'
-    metrics_path: /minio/prometheus/metrics
-    static_configs:
-      - targets: ['127.0.0.1:9000']
-  - job_name: 'querier'
-    static_configs:
-      - targets: ['127.0.0.1:9091']
-  - job_name: 'store_gateway'
-    static_configs:
-      - targets: ['127.0.0.1:19091']
-
- -## Starting Prometheus Instance - -Let's now start the container representing Prometheus instance. - -Note `-v /root/prom-eu1:/prometheus \` and `--storage.tsdb.path=/prometheus` that allows us to place our generated data -in Prometheus data directory. - -Let's deploy Prometheus now. Note that we disabled local Prometheus compactions `storage.tsdb.max-block-duration` and `min` flags. -Currently, this is important for the basic object storage backup scenario to avoid conflicts between the bucket and local compactions. -Read more [here](https://thanos.io/tip/components/sidecar.md/#sidecar). - -We also extend Prometheus retention: `--storage.tsdb.retention.time=1000d`. This is because Prometheus by default removes all data older -than 2 weeks. And we have a year (: - -### Deploying "EU1" - -``` -docker run -d --net=host --rm \ - -v /root/editor/prometheus0_eu1.yml:/etc/prometheus/prometheus.yml \ - -v /root/prom-eu1:/prometheus \ - -u root \ - --name prometheus-0-eu1 \ - quay.io/prometheus/prometheus:v2.20.0 \ - --config.file=/etc/prometheus/prometheus.yml \ - --storage.tsdb.retention.time=1000d \ - --storage.tsdb.path=/prometheus \ - --storage.tsdb.max-block-duration=2h \ - --storage.tsdb.min-block-duration=2h \ - --web.listen-address=:9090 \ - --web.external-url=https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com \ - --web.enable-lifecycle \ - --web.enable-admin-api -```{{execute}} - -## Setup Verification - -Once started you should be able to reach the Prometheus instance here and query.. 1 year of data! - -* [Prometheus-0 EU1](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/graph?g0.range_input=1y&g0.expr=continuous_app_metric0&g0.tab=0) - -## Thanos Sidecar & Querier - -Similar to previous course, let's setup global view querying with sidecar: - -``` -docker run -d --net=host --rm \ - --name prometheus-0-eu1-sidecar \ - -u root \ - quay.io/thanos/thanos:v0.26.0 \ - sidecar \ - --http-address 0.0.0.0:19090 \ - --grpc-address 0.0.0.0:19190 \ - --prometheus.url http://127.0.0.1:9090 -```{{execute}} - -And Querier. As you remember [Thanos sidecar](https://thanos.io/tip/components/query.md/) exposes `StoreAPI` -so we will make sure we point the Querier to the gRPC endpoints of the sidecar: - -``` -docker run -d --net=host --rm \ - --name querier \ - quay.io/thanos/thanos:v0.26.0 \ - query \ - --http-address 0.0.0.0:9091 \ - --query.replica-label replica \ - --store 127.0.0.1:19190 -```{{execute}} - -## Setup verification - -Similar to previous course let's check if the Querier works as intended. Let's look on -[Querier UI `Store` page](https://[[HOST_SUBDOMAIN]]-9091-[[KATACODA_HOST]].environments.katacoda.com/stores). - -This should list the sidecar, including the external labels. - -On graph you should also see our 5 series for 1y time, thanks to Prometheus and sidecar StoreAPI: [Graph](https://[[HOST_SUBDOMAIN]]-9091-[[KATACODA_HOST]].environments.katacoda.com/graph?g0.range_input=1y&g0.max_source_resolution=0s&g0.expr=continuous_app_metric0&g0.tab=0). - -Click `Continue` to see how we can move this data to much cheaper and easier to operate object storage. diff --git a/tutorials/katacoda/thanos/2-lts/step2-verify.sh b/tutorials/katacoda/thanos/2-lts/step2-verify.sh deleted file mode 100644 index 68bdd69733..0000000000 --- a/tutorials/katacoda/thanos/2-lts/step2-verify.sh +++ /dev/null @@ -1,9 +0,0 @@ -#!/usr/bin/env bash - -curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:19090/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1 - -curl -s 127.0.0.1:19090/metrics >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/2-lts/step2.md b/tutorials/katacoda/thanos/2-lts/step2.md deleted file mode 100644 index 52f8e067fd..0000000000 --- a/tutorials/katacoda/thanos/2-lts/step2.md +++ /dev/null @@ -1,97 +0,0 @@ -# Step 2 - Object Storage Continuous Backup - -Maintaining one year of data within your Prometheus is doable, but not easy. It's tricky to -resize, backup or maintain this data long term. On top of that Prometheus does not do any replication, -so any unavailability of Prometheus results in query unavailability. - -This is where Thanos comes to play. With a single configuration change we can allow Thanos Sidecar to continuously upload blocks of metrics -that are periodically persisted to disk by the Prometheus. - -> NOTE: Prometheus when scraping data, initially aggregates all samples in memory and WAL (on-disk write-head-log). Only after 2-3h it "compacts" -> the data into disk in form of 2h TSDB block. This is why we need to still query Prometheus for latest data, but overall with this change -> we can keep Prometheus retention to minimum. It's recommended to keep Prometheus retention in this case at least 6 hours long, to have safe buffer -> for a potential event of network partition. - -## Starting Object Storage: Minio - -Let's start simple S3-compatible Minio engine that keeps data in local disk: - -``` -mkdir /root/minio && \ -docker run -d --rm --name minio \ - -v /root/minio:/data \ - -p 9000:9000 -e "MINIO_ACCESS_KEY=minio" -e "MINIO_SECRET_KEY=melovethanos" \ - minio/minio:RELEASE.2019-01-31T00-31-19Z \ - server /data -```{{execute}} - -Create `thanos` bucket: - -``` -mkdir /root/minio/thanos -```{{execute}} - -## Verification - -To check if the Minio is working as intended, let's [open Minio server UI](https://[[HOST_SUBDOMAIN]]-9000-[[KATACODA_HOST]].environments.katacoda.com/minio/) - -Enter the credentials as mentioned below: - -**Access Key** = `minio` -**Secret Key** = `melovethanos` - -## Sidecar block backup - -All Thanos components that use object storage uses the same `objstore.config` flag with the same "little" bucket config format. - -Click `Copy To Editor` for config to propagate the configs to the file `bucket_storage.yaml`: - -
-type: S3
-config:
-  bucket: "thanos"
-  endpoint: "127.0.0.1:9000"
-  insecure: true
-  signature_version2: true
-  access_key: "minio"
-  secret_key: "melovethanos"
-
- -Let's restart sidecar with updated configuration in backup mode. - -``` -docker stop prometheus-0-eu1-sidecar -```{{execute}} - -[Thanos sidecar](https://thanos.io/tip/components/sidecar.md/) allows to backup all the blocks that Prometheus persists to -the disk. In order to accomplish this we need to make sure that: - -* Sidecar has direct access to the Prometheus data directory (in our case host's /root/prom-eu1 dir) (`--tsdb.path` flag) -* Bucket configuration is specified `--objstore.config-file` -* `--shipper.upload-compacted` has to be set if you want to upload already compacted blocks when sidecar starts. Use this only -when you want to upload blocks never seen before on new Prometheus introduced to Thanos system. - -Let's run sidecar: - -``` -docker run -d --net=host --rm \ - -v /root/editor/bucket_storage.yaml:/etc/thanos/minio-bucket.yaml \ - -v /root/prom-eu1:/prometheus \ - --name prometheus-0-eu1-sidecar \ - -u root \ - quay.io/thanos/thanos:v0.26.0 \ - sidecar \ - --tsdb.path /prometheus \ - --objstore.config-file /etc/thanos/minio-bucket.yaml \ - --shipper.upload-compacted \ - --http-address 0.0.0.0:19090 \ - --grpc-address 0.0.0.0:19190 \ - --prometheus.url http://127.0.0.1:9090 -```{{execute}} - -## Verification - -We can check whether the data is uploaded into `thanos` bucket by visitng [Minio](https://[[HOST_SUBDOMAIN]]-9000-[[KATACODA_HOST]].environments.katacoda.com/minio/). -It will take couple of seconds to synchronize all blocks. - -Once all blocks appear in the minio `thanos` bucket, we are sure our data is backed up. Awesome! πŸ’ͺ diff --git a/tutorials/katacoda/thanos/2-lts/step3-answer.md b/tutorials/katacoda/thanos/2-lts/step3-answer.md deleted file mode 100644 index 40ab7c2f64..0000000000 --- a/tutorials/katacoda/thanos/2-lts/step3-answer.md +++ /dev/null @@ -1,19 +0,0 @@ -## Answer - -**In an HA Prometheus setup with Thanos sidecars, would there be issues with multiple sidecars attempting to upload the same data blocks to object storage?** - -This is handled by having unique **external labels** for all Prometheus, sidecar instances and HA replicas. To indicate that all replicas are storing same targets, they differ only in one label. - -For an instance, consider the situation below: - -``` -First: -"cluster": "prod1" -"replica": "0" - -Second: -"cluster":"prod1" -"replica": "1" -``` - -There is no problem with storing them since the label sets are **unique**. \ No newline at end of file diff --git a/tutorials/katacoda/thanos/2-lts/step3-verify.sh b/tutorials/katacoda/thanos/2-lts/step3-verify.sh deleted file mode 100644 index 1bdfd136b7..0000000000 --- a/tutorials/katacoda/thanos/2-lts/step3-verify.sh +++ /dev/null @@ -1,11 +0,0 @@ -#!/usr/bin/env bash - -curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:19090/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1 - -curl -s 127.0.0.1:19090/metrics >/dev/null || exit 1 - -curl -s 127.0.0.1:19091/metrics >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/2-lts/step3.md b/tutorials/katacoda/thanos/2-lts/step3.md deleted file mode 100644 index 7700c676c2..0000000000 --- a/tutorials/katacoda/thanos/2-lts/step3.md +++ /dev/null @@ -1,98 +0,0 @@ -# Step 3 - Fetching metrics from Bucket - -In this step, we will learn about Thanos Store Gateway and how to deploy it. - -## Thanos Components - -Let's take a look at all the Thanos commands: - -```docker run --rm quay.io/thanos/thanos:v0.26.0 --help```{{execute}} - -You should see multiple commands that solve different purposes, block storage based long-term storage for Prometheus. - -In this step we will focus on thanos `store gateway`: - -``` - store [] - Store node giving access to blocks in a bucket provider -``` - -## Store Gateway: - -* This component implements the Store API on top of historical data in an object storage bucket. It acts primarily as an API gateway and therefore does not need -significant amounts of local disk space. -* It keeps a small amount of information about all remote blocks on the local disk and keeps it in sync with the bucket. -This data is generally safe to delete across restarts at the cost of increased startup times. - -You can read more about [Store](https://thanos.io/tip/components/store.md/) here. - -### Deploying store for "EU1" Prometheus data - -``` -docker run -d --net=host --rm \ - -v /root/editor/bucket_storage.yaml:/etc/thanos/minio-bucket.yaml \ - --name store-gateway \ - quay.io/thanos/thanos:v0.26.0 \ - store \ - --objstore.config-file /etc/thanos/minio-bucket.yaml \ - --http-address 0.0.0.0:19091 \ - --grpc-address 0.0.0.0:19191 -```{{execute}} - -## How to query Thanos store data? - -In this step, we will see how we can query Thanos store data which has access to historical data from the `thanos` bucket, and play with this setup a bit. - -Currently querier does not know about store yet. Let's change it by adding Store Gateway gRPC address `--store 127.0.0.1:19191` to Querier: - -``` -docker stop querier && \ -docker run -d --net=host --rm \ - --name querier \ - quay.io/thanos/thanos:v0.26.0 \ - query \ - --http-address 0.0.0.0:9091 \ - --query.replica-label replica \ - --store 127.0.0.1:19190 \ - --store 127.0.0.1:19191 -```{{execute}} - -Click on the Querier UI `Graph` page and try querying data for a year or two by inserting metrics `continuous_app_metric0` ([Query UI](https://[[HOST_SUBDOMAIN]]-9091-[[KATACODA_HOST]].environments.katacoda.com/graph?g0.range_input=1y&g0.max_source_resolution=0s&g0.expr=continuous_app_metric0&g0.tab=0)). Make sure `deduplication` is selected and you will be able to discover all the data fetched by Thanos store. - -![](https://github.com/thanos-io/thanos/raw/main/tutorials/katacoda/thanos/2-lts/query.png) - -Also, you can check all the active endpoints located by thanos-store by clicking on [Stores](https://[[HOST_SUBDOMAIN]]-9091-[[KATACODA_HOST]].environments.katacoda.com/stores). - -### What we know so far? - -We've added Thanos Query, a component that can query a Prometheus instance and Thanos Store at the same time, -which gives transparent access to the archived blocks and real-time metrics. The vanilla PromQL Prometheus engine used inside Query deduces -what time series and for what time ranges we need to fetch the data for. - -Also, StoreAPIs propagate external labels and the time range they have data for, so we can do basic filtering on this. -However, if you don't specify any of these in the query (only "up" series) the querier concurrently asks all the StoreAPI servers. - -The potential duplication of data between Prometheus+sidecar results and store Gateway will be transparently handled and invisible in results. - -### Checking what StoreAPIs are involved in query - -Another interesting question here is how to ensure if we query the data from bucket only? - -We can check this by visitng the [New UI](https://[[HOST_SUBDOMAIN]]-9091-[[KATACODA_HOST]].environments.katacoda.com/new/graph?g0.expr=&g0.tab=0&g0.stacked=0&g0.range_input=1h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=[]), inserting `continuous_app_metric0` metrics again with 1 year time range of graph, -and click on `Enable Store Filtering`. - -This allows us to filter stores and helps in debugging from where we are querying the data exactly. - -Let's chose only `127.0.0.1:19191`, so store gateway. This query will only hit that store to retrieve data, so we are sure that store works. - -## Question Time? πŸ€” - -In an HA Prometheus setup with Thanos sidecars, would there be issues with multiple sidecars attempting to upload the same data blocks to object storage? - -Think over this πŸ˜‰ - -To see the answer to this question click `SHOW SOLUTION` below. - -## Next - -Voila! In the next step, we will talk about Thanos Compactor, its retention capabilities, and how it improves query efficiency and reduce the required storage size. diff --git a/tutorials/katacoda/thanos/2-lts/step4-verify.sh b/tutorials/katacoda/thanos/2-lts/step4-verify.sh deleted file mode 100644 index b980b137de..0000000000 --- a/tutorials/katacoda/thanos/2-lts/step4-verify.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/usr/bin/env bash - -curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:19090/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1 - -curl -s 127.0.0.1:19090/metrics >/dev/null || exit 1 - -curl -s 127.0.0.1:19091/metrics >/dev/null || exit 1 - -curl -s 127.0.0.1:19095/metrics >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/2-lts/step4.md b/tutorials/katacoda/thanos/2-lts/step4.md deleted file mode 100644 index 82fbd324b8..0000000000 --- a/tutorials/katacoda/thanos/2-lts/step4.md +++ /dev/null @@ -1,65 +0,0 @@ -# Step 4 - Thanos Compactor - -In this step, we will install Thanos Compactor which applies the compaction, retention, deletion and downsampling operations -on the TSDB block data object storage. - -Before, moving forward, let's take a closer look at what the `Compactor` component does: - -> Note: Thanos Compactor is currently designed to be run as a singleton. Unavailability (hours) is not problematic as it does not serve any live requests. -## Compactor - -The `Compactor` is an essential component that operates on a single object storage bucket to compact, downsample, apply retention, to the TSDB blocks held inside, -thus, making queries on historical data more efficient. It creates aggregates of old metrics (based upon the rules). - -It is also responsible for downsampling of data, performing 5m downsampling after 40 hours, and 1h downsampling after 10 days. - -If you want to know more about Thanos Compactor, jump [here](https://thanos.io/tip/components/compact.md/). - -> Note: Thanos Compactor is mandatory if you use object storage otherwise Thanos Store Gateway will be too slow without using a compactor. - -## Deploying Thanos Compactor - -Click below snippet to start the Compactor. - -``` -docker run -d --net=host --rm \ - -v /root/editor/bucket_storage.yaml:/etc/thanos/minio-bucket.yaml \ - --name thanos-compact \ - quay.io/thanos/thanos:v0.26.0 \ - compact \ - --wait --wait-interval 30s \ - --consistency-delay 0s \ - --objstore.config-file /etc/thanos/minio-bucket.yaml \ - --http-address 0.0.0.0:19095 -```{{execute}} - -The flag `wait` is used to make sure all compactions have been processed while `--wait-interval` is kept in 30s to perform all the compactions and downsampling very quickly. Also, this only works when when `--wait` flag is specified. Another flag `--consistency-delay` is basically used for buckets which are not consistent strongly. It is the minimum age of non-compacted blocks before they are being processed. Here, we kept the delay at 0s assuming the bucket is consistent. - -## Setup Verification - -To check if compactor works fine, we can look at the [Bucket View](https://[[HOST_SUBDOMAIN]]-19095-[[KATACODA_HOST]].environments.katacoda.com/new/loaded). - -Now, if we click on the blocks, they will provide us all the metadata (Series, Samples, Resolution, Chunks, and many more things). - -## Compaction and Downsampling - -When we query large historical data there are millions of samples that we need to go through which makes the queries slower and slower as we retrieve year's worth of data. - -Thus, Thanos uses the technique called downsampling (a process of reducing the sampling rate of the signal) to keep the queries responsive, -and no special configuration is required to perform this process. - -The Compactor applies compaction to the bucket data and also completes the downsampling for historical data. - -To expierience this, click on the [Querier](https://[[HOST_SUBDOMAIN]]-9091-[[KATACODA_HOST]].environments.katacoda.com/new/graph?g0.expr=&g0.tab=0&g0.stacked=0&g0.range_input=1h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=[]) and insert metrics `continuous_app_metric0` with 1 year time range of graph, and also, click on `Enable Store Filtering`. - -Let's try querying `Max 5m downsampling` data, it uses 5m resolution and it will be faster than the raw data. Also, Downsampling is built on top of data, and never done on **young** data. - -## Unlimited Retention - Not Challenging anymore? - -Having a long time metric retention for Prometheus was always involving lots of complexity, disk space, and manual work. With Thanos, you can make Prometheus almost stateless, while having most of the data in durable and cheap object storage. - -## Next - -Awesome work! Feel free to play with the setup πŸ€— - -Once done, hit `Continue` for summary. diff --git a/tutorials/katacoda/thanos/3-receiver/assets/receive-cluster-result.png b/tutorials/katacoda/thanos/3-receiver/assets/receive-cluster-result.png deleted file mode 100644 index a6134b9e28..0000000000 Binary files a/tutorials/katacoda/thanos/3-receiver/assets/receive-cluster-result.png and /dev/null differ diff --git a/tutorials/katacoda/thanos/3-receiver/assets/receive-empty-query-result.png b/tutorials/katacoda/thanos/3-receiver/assets/receive-empty-query-result.png deleted file mode 100644 index e54aeb2c6d..0000000000 Binary files a/tutorials/katacoda/thanos/3-receiver/assets/receive-empty-query-result.png and /dev/null differ diff --git a/tutorials/katacoda/thanos/3-receiver/courseBase.sh b/tutorials/katacoda/thanos/3-receiver/courseBase.sh deleted file mode 100644 index c6225c55e6..0000000000 --- a/tutorials/katacoda/thanos/3-receiver/courseBase.sh +++ /dev/null @@ -1,6 +0,0 @@ -#!/usr/bin/env bash - -docker pull quay.io/prometheus/prometheus:v2.27.0 -docker pull quay.io/thanos/thanos:v0.21.0 - -mkdir /root/editor diff --git a/tutorials/katacoda/thanos/3-receiver/finish.md b/tutorials/katacoda/thanos/3-receiver/finish.md deleted file mode 100644 index 88ec7238fa..0000000000 --- a/tutorials/katacoda/thanos/3-receiver/finish.md +++ /dev/null @@ -1,23 +0,0 @@ -# Summary - -Congratulations! πŸŽ‰πŸŽ‰πŸŽ‰ -You completed this `Thanos Receive` tutorial. Let's summarize what we learned: - -* `Thanos Receive` is a component that implements the `Prometheus Remote Write` protocol. -* Prometheus can be configured to remote-write its metric data in real-time to another server that implements the `Remote Write` protocol. -* Thanos Receive allows us to ingest the data from sources which have limited accessibility, or that has no querying / storing capabilities and they have to forward data somewhere else (e.g Prometheus in Agent mode explained in next tutorial). - -See next courses for other tutorials about different deployment models and more advanced features of Thanos! - -## Further Reading - -To understand more about `Thanos Receive` - check out the following resources: -* [Thanos Receive Documentation](https://thanos.io/tip/components/receive.md/) -* [Thanos Receive Design Document](https://thanos.io/tip/proposals/201812_thanos-remote-receive.md/) -* [Pros/Cons of allowing remote write in Prometheus](https://docs.google.com/document/d/1H47v7WfyKkSLMrR8_iku6u9VB73WrVzBHb2SB6dL9_g/edit#heading=h.2v27snv0lsur) - -### Feedback - -Do you see any bug, typo in the tutorial or you have some feedback for us? - -let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io \ No newline at end of file diff --git a/tutorials/katacoda/thanos/3-receiver/index.json b/tutorials/katacoda/thanos/3-receiver/index.json deleted file mode 100644 index e9a2608b1e..0000000000 --- a/tutorials/katacoda/thanos/3-receiver/index.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "title": "Intermediate: Streaming metrics from remote source with Thanos Receive", - "description": "Learn how to ingest and then query metrics data from egress-only sources with Thanos Receive.", - "difficulty": "Moderate", - "details": { - "steps": [ - { - "title": "Problem Statement & Setup", - "text": "step1.md", - "verify": "step1-verify.sh" - }, - { - "title": "Setup Thanos Receive", - "text": "step2.md", - "verify": "step2-verify.sh" - }, - { - "title": "Configure Prometheus Remote Write", - "text": "step3.md", - "verify": "step3-verify.sh" - }, - { - "title": "Verify Setup", - "text": "step4.md", - "verify": "step4-verify.sh" - } - ], - "intro": { - "text": "intro.md", - "courseData": "courseBase.sh", - "credits": "https://thanos.io" - }, - "finish": { - "text": "finish.md", - "credits": "test" - } - }, - "files": [ - "prometheus-batcave.yaml", - "prometheus-batcomputer.yaml" - ], - "environment": { - "uilayout": "editor-terminal", - "uisettings": "yaml", - "uieditorpath": "/root/editor", - "showdashboard": true, - "dashboards": [ - {"name": "Prometheus Batcave", "port": 9090}, - {"name": "Prometheus Batcomputer", "port": 9001}, - {"name": "Thanos Query", "port": 39090} - ] - }, - "backend": { - "imageid": "docker-direct" - } -} diff --git a/tutorials/katacoda/thanos/3-receiver/intro.md b/tutorials/katacoda/thanos/3-receiver/intro.md deleted file mode 100644 index 81cdc76fac..0000000000 --- a/tutorials/katacoda/thanos/3-receiver/intro.md +++ /dev/null @@ -1,26 +0,0 @@ -# Intermediate: Streaming metrics from remote source with Thanos Receive - -The [Thanos](thanos.io) project defines a set of components that can be composed together into a highly available metric system with **unlimited storage capacity** that **seamlessly** integrates into your existing Prometheus deployments. - -In this course you get first-hand experience building and deploying this infrastructure yourself. - -In this tutorial, you will learn: - -* How to ingest metrics data from Prometheus instances without ingress traffic and need to store data locally for longer time. -* How to setup a Thanos Querier to access this data. -* How Thanos Receive is different from Thanos Sidecar, and when is the right time to use each of them. - -> NOTE: This course uses docker containers with pre-built Thanos, Prometheus, and Minio Docker images available publicly. - -### Prerequisites - -Please complete tutorial #1 first: [Global View and seamless HA for Prometheus](https://www.katacoda.com/thanos/courses/thanos/1-globalview) πŸ€— - -### Feedback - -Do you see any bug, typo in the tutorial or you have some feedback for us? -Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io - -### Contributed by: - -* Ian Billett [@bill3tt](http://github.com/bill3tt) diff --git a/tutorials/katacoda/thanos/3-receiver/step1-verify.sh b/tutorials/katacoda/thanos/3-receiver/step1-verify.sh deleted file mode 100644 index 3c23268f34..0000000000 --- a/tutorials/katacoda/thanos/3-receiver/step1-verify.sh +++ /dev/null @@ -1,6 +0,0 @@ -#!/usr/bin/env bash - -curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/3-receiver/step1.md b/tutorials/katacoda/thanos/3-receiver/step1.md deleted file mode 100644 index 8e0b6a0300..0000000000 --- a/tutorials/katacoda/thanos/3-receiver/step1.md +++ /dev/null @@ -1,95 +0,0 @@ -# Problem Statement & Setup - -## Problem Statement - -Let's imagine that you run a company called `Wayne Enterprises`. This company runs two clusters: `Batcave` & `Batcomputer`. Each of these sites runs an instance of Prometheus that collects metrics data from applications and services running there. - -However, these sites are special. For security reasons, they do not expose public endpoints to the Prometheus instances running there, and so cannot be accessed directly from other parts of your infrastructure. - -As the person responsible for implementing monitoring these sites, you have two requirements to meet: - -1. Implement a global view of this data. `Wayne Enterprises` needs to know what is happening in all parts of the company - including secret ones! -1. Global view must be queryable in near real-time. We can't afford any delay in monitoring these locations! - -Firstly, let us setup two Prometheus instances... - -## Setup - -### Batcave - -Let's use a very simple configuration file, that tells prometheus to scrape its own metrics page every 5 seconds. - -
-global:
-  scrape_interval: 5s
-  external_labels:
-    cluster: batcave
-    replica: 0
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9090']
-
- -Run the prometheus instance: - -``` -docker run -d --net=host --rm \ - -v /root/editor/prometheus-batcave.yaml:/etc/prometheus/prometheus.yaml \ - -v /root/prometheus-batcave-data:/prometheus \ - -u root \ - --name prometheus-batcave \ - quay.io/prometheus/prometheus:v2.27.0 \ - --config.file=/etc/prometheus/prometheus.yaml \ - --storage.tsdb.path=/prometheus \ - --web.listen-address=:9090 \ - --web.external-url=https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com \ - --web.enable-lifecycle -```{{execute}} - -Verify that `prometheus-batcave` is running by navigating to the [Batcave Prometheus UI](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com). - -
- Why do we enable the web lifecycle flag? - - By specifying `--web.enable-lifecycle`, we tell Prometheus to expose the `/-/reload` HTTP endpoint. - - This lets us tell Prometheus to dynamically reload its configuration, which will be useful later in this tutorial. -
- - -### Batcomputer - -Almost exactly the same configuration as above, except we run the Prometheus instance on port `9091`. - -
-global:
-  scrape_interval: 5s
-  external_labels:
-    cluster: batcomputer
-    replica: 0
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9091']
-
- -``` -docker run -d --net=host --rm \ - -v /root/editor/prometheus-batcomputer.yaml:/etc/prometheus/prometheus.yaml \ - -v /root/prometheus-batcomputer:/prometheus \ - -u root \ - --name prometheus-batcomputer \ - quay.io/prometheus/prometheus:v2.27.0 \ - --config.file=/etc/prometheus/prometheus.yaml \ - --storage.tsdb.path=/prometheus \ - --web.listen-address=:9091 \ - --web.external-url=https://2886795291-9091-elsy05.environments.katacoda.com \ - --web.enable-lifecycle -```{{execute}} - -Verify that `prometheus-batcomputer` is running by navigating to the [Batcomputer Prometheus UI](https://[[HOST_SUBDOMAIN]]-9091-[[KATACODA_HOST]].environments.katacoda.com). - -With these Prometheus instances configured and running, we can now start to architect our global view of all of `Wayne Enterprises`. diff --git a/tutorials/katacoda/thanos/3-receiver/step2-verify.sh b/tutorials/katacoda/thanos/3-receiver/step2-verify.sh deleted file mode 100644 index be32a11ac9..0000000000 --- a/tutorials/katacoda/thanos/3-receiver/step2-verify.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env bash - -# prometheus-batcave -curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1 -# prometheus-batcomputer -curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1 -# receive -curl -s 127.0.0.1:10909/metrics >/dev/null || exit 1 -# query -curl -s 127.0.0.1:39090/metrics >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/3-receiver/step2.md b/tutorials/katacoda/thanos/3-receiver/step2.md deleted file mode 100644 index c22b0cd42d..0000000000 --- a/tutorials/katacoda/thanos/3-receiver/step2.md +++ /dev/null @@ -1,104 +0,0 @@ -# Setup Thanos Receive - -With `prometheus-batcave` & `prometheus-batcomputer` now running, we need to think about how we satisfy our two requirements: -1. Implement a global view of this data. -1. Global view must be queryable in near real-time. - -How are we going to do this? - -## Thanos Sidecar - -After completing [Tutorial #1: Global View](https://www.katacoda.com/thanos/courses/thanos/1-globalview), you may think of running the following architecture: - -* Run Thanos Sidecar next to each of the Prometheus instances. -* Configure Thanos Sidecar to upload Prometheus data to an object storage (S3, GCS, Minio). -* Run Thanos Store connected to the data stored in object storage. -* Run Thanos Query to pull data from Thanos Store. - -However! This setup **does not** satisfy our requirements above. - -
- Can you think why? - -Since we cannot access the `Thanos Sidecar` directly - we cannot query metrics data in real-time. - -`Thanos Sidecar` only uploads `blocks` of metrics data that have been written to disk, which happens every 2 hours in Prometheus. -
-This means that the Global View would be at least 2 hours out of date, and does not satisfy requirement #2. -
- - -## Thanos Receive - -Enter [Thanos Receive](https://thanos.io/tip/components/receive.md/). - -`Thanos Receive` is a component that implements the [Prometheus Remote Write API](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write). This means that it will accept metrics data that is sent to it by other Prometheus instances (or any other process that implements Remote Write API). - -Prometheus can be configured to `Remote Write`. This means that Prometheus will send all of its metrics data to a remote endpoint as they are being ingested - useful for our requirements! - -In its simplest form, when `Thanos Receive` receives data from Prometheus, it stores it locally and exposes a `Store API` server so this data can be queried by `Thanos Query`. - -`Thanos Receive` has more features that will be covered in future tutorials, but let's keep things simple for now. - -## Run Thanos Receive - -The first component we will run in our new architecture is `Thanos Receive`: - -``` -docker run -d --rm \ - -v $(pwd)/receive-data:/receive/data \ - --net=host \ - --name receive \ - quay.io/thanos/thanos:v0.21.0 \ - receive \ - --tsdb.path "/receive/data" \ - --grpc-address 127.0.0.1:10907 \ - --http-address 127.0.0.1:10909 \ - --label "receive_replica=\"0\"" \ - --label "receive_cluster=\"wayne-enterprises\"" \ - --remote-write.address 127.0.0.1:10908 -```{{execute}} - -This starts Thanos Receive that listens on `http://127.0.0.1:10908/api/v1/receive' endpoint for Remote Write and on `127.0.0.1:10907` for Thanos StoreAPI. - -Let's talk about some important parameters: -* `--label` - `Thanos Receive` requires at least one label to be set. These are called 'external labels' and are used to uniquely identify this instance of `Thanos Receive`. -* `--remote-write.address` - This is the address that `Thanos Receive` is listening on for Prometheus' remote write requests. - -Let's verify that this is running correctly. Since `Thanos Receive` does not expose a UI, we can check it is up by retrieving its metrics page. - -``` -curl http://127.0.0.1:10909/metrics -```{{execute}} - -## Run Thanos Query - -Next, let us run a `Thanos Query` instance: - -``` -docker run -d --rm \ - --net=host \ - --name query \ - quay.io/thanos/thanos:v0.21.0 \ - query \ - --http-address "0.0.0.0:39090" \ - --store "127.0.0.1:10907" -```{{execute}} - -`Thanos Receive` exposed its gRPC endpoint at `127.0.0.1:10907`, so we need to tell `Thanos Query` to use this endpoint with the `--store` flag. - -Verify that `Thanos Query` is working and configured correctly by looking at the 'stores' tab [here](https://[[HOST_SUBDOMAIN]]-39090-[[KATACODA_HOST]].environments.katacoda.com/stores). - -Now we are done right? Try querying for some data... - -![alt text](./assets/receive-empty-query-result.png) - -
- Uh-oh! Do you know why are we seeing 'Empty Query Result' responses? - -We have correctly configured `Thanos Receive` & `Thanos Query`, but we have not yet configured Prometheus to write to remote write its data to the right place. - -
- - -Hit continue and we will fix this setup! \ No newline at end of file diff --git a/tutorials/katacoda/thanos/3-receiver/step3-verify.sh b/tutorials/katacoda/thanos/3-receiver/step3-verify.sh deleted file mode 100644 index be32a11ac9..0000000000 --- a/tutorials/katacoda/thanos/3-receiver/step3-verify.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env bash - -# prometheus-batcave -curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1 -# prometheus-batcomputer -curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1 -# receive -curl -s 127.0.0.1:10909/metrics >/dev/null || exit 1 -# query -curl -s 127.0.0.1:39090/metrics >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/3-receiver/step3.md b/tutorials/katacoda/thanos/3-receiver/step3.md deleted file mode 100644 index ae3611e3a5..0000000000 --- a/tutorials/katacoda/thanos/3-receiver/step3.md +++ /dev/null @@ -1,54 +0,0 @@ -# Configure Prometheus Remote Write - -Our problem in the last step was that we have not yet configured Prometheus to `remote_write` to our `Thanos Receive` instance. - -We need to tell `prometheus-batcave` & `prometheus-batcomputer` where to write their data to. - -## Update Configuration - -The docs for this configuration option can be found [here](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write). - -
-global:
-  scrape_interval: 5s
-  external_labels:
-    cluster: batcave
-    replica: 0
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9090']
-remote_write:
-- url: 'http://127.0.0.1:10908/api/v1/receive'
-
- -
-global:
-  scrape_interval: 5s
-  external_labels:
-    cluster: batcomputer
-    replica: 0
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9091']
-remote_write:
-- url: 'http://127.0.0.1:10908/api/v1/receive'
-
- -## Reload Configuration - -Since we supplied the `--web.enable-lifecycle` flag in our Prometheus instances, we can dynamically reload the configuration by `curl`-ing the `/-/reload` endpoints. - -``` -curl -X POST http://127.0.0.1:9090/-/reload -curl -X POST http://127.0.0.1:9091/-/reload -```{{execute}} - -Verify this has taken affect by checking the `/config` page on our Prometheus instances: -* `prometheus-batcave` [config page](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/config) -* `prometheus-batcomputer` [config page](https://[[HOST_SUBDOMAIN]]-9091-[[KATACODA_HOST]].environments.katacoda.com/config) - -In both cases you should see the `remote_write` options in the configuration. \ No newline at end of file diff --git a/tutorials/katacoda/thanos/3-receiver/step4-verify.sh b/tutorials/katacoda/thanos/3-receiver/step4-verify.sh deleted file mode 100644 index be32a11ac9..0000000000 --- a/tutorials/katacoda/thanos/3-receiver/step4-verify.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env bash - -# prometheus-batcave -curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1 -# prometheus-batcomputer -curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1 -# receive -curl -s 127.0.0.1:10909/metrics >/dev/null || exit 1 -# query -curl -s 127.0.0.1:39090/metrics >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/3-receiver/step4.md b/tutorials/katacoda/thanos/3-receiver/step4.md deleted file mode 100644 index 77dc730b51..0000000000 --- a/tutorials/katacoda/thanos/3-receiver/step4.md +++ /dev/null @@ -1,21 +0,0 @@ -# Verify Setup - -At this point, we have: -* Two prometheus instances configured to `remote_write`. -* `Thanos Receive` component ingesting data from Prometheus -* `Thanos Query` component configured to query `Thanos Receive`'s Store API. - -The final task on our list is to verify that data is flowing correctly. - -Stop and think how you could do this before opening the answer below πŸ‘‡ - -
- How are we going to check that the components are wired up correctly? - -Let's make sure that we can query data from each of our Prometheus instances from our `Thanos Query` instance. - -Navigate to the [Thanos Query UI](https://[[HOST_SUBDOMAIN]]-39090-[[KATACODA_HOST]].environments.katacoda.com), and query for a metric like `up` - inspect the output and you should see `batcave` and `batcomputer` in the `cluster` label. - -![alt-text](./assets/receive-cluster-result.png) - -
\ No newline at end of file diff --git a/tutorials/katacoda/thanos/4-receiver-agent/assets/batmobile.jpg b/tutorials/katacoda/thanos/4-receiver-agent/assets/batmobile.jpg deleted file mode 100644 index 4b3cae1c4f..0000000000 Binary files a/tutorials/katacoda/thanos/4-receiver-agent/assets/batmobile.jpg and /dev/null differ diff --git a/tutorials/katacoda/thanos/4-receiver-agent/assets/expected.png b/tutorials/katacoda/thanos/4-receiver-agent/assets/expected.png deleted file mode 100644 index 7a60f90667..0000000000 Binary files a/tutorials/katacoda/thanos/4-receiver-agent/assets/expected.png and /dev/null differ diff --git a/tutorials/katacoda/thanos/4-receiver-agent/courseBase.sh b/tutorials/katacoda/thanos/4-receiver-agent/courseBase.sh deleted file mode 100644 index 3726fe82e3..0000000000 --- a/tutorials/katacoda/thanos/4-receiver-agent/courseBase.sh +++ /dev/null @@ -1,6 +0,0 @@ -#!/usr/bin/env bash - -docker pull quay.io/prometheus/prometheus:v2.32.0-beta.0 -docker pull quay.io/thanos/thanos:v0.21.0 - -mkdir /root/editor diff --git a/tutorials/katacoda/thanos/4-receiver-agent/finish.md b/tutorials/katacoda/thanos/4-receiver-agent/finish.md deleted file mode 100644 index 7c0f12261f..0000000000 --- a/tutorials/katacoda/thanos/4-receiver-agent/finish.md +++ /dev/null @@ -1,25 +0,0 @@ -# Summary - -Congratulations! πŸŽ‰πŸŽ‰πŸŽ‰ - -`Batmobile` is now well-monitored! Batman says thank you! - -![batman](./assets/batmobile.jpg) - -You completed this Prometheus Agent tutorial. Let's summarize what we learned: - -* Thanos Receive is a component that implements the `Prometheus Remote Write` protocol. -* Prometheus Agent can be deployed to remote write its metric data in real-time to another server that implements the Remote Write protocol. -* Prometheus Agent uses a _fraction_ of the resources as normal Prometheus, by not enabling features like: local metrics storage and local query APIs. - -See next courses for other tutorials about different deployment models and more advanced features of Thanos! - -## Further Reading - -* [Prometheus Blog Post](https://prometheus.io/blog/2021/11/16/agent/) - -### Feedback - -Do you see any bug, typo in the tutorial or you have some feedback for us? - -let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io \ No newline at end of file diff --git a/tutorials/katacoda/thanos/4-receiver-agent/index.json b/tutorials/katacoda/thanos/4-receiver-agent/index.json deleted file mode 100644 index 3c9e763143..0000000000 --- a/tutorials/katacoda/thanos/4-receiver-agent/index.json +++ /dev/null @@ -1,51 +0,0 @@ -{ - "title": "Bonus: Using Prometheus Agent for streaming metrics to Thanos Receive", - "description": "Learn how to use Prometheus in agent mode to efficiently stream metrics out of cluster.", - "difficulty": "Moderate", - "details": { - "steps": [ - { - "title": "Problem Statement & Setup", - "text": "step1.md", - "verify": "step1-verify.sh" - }, - { - "title": "Setup Prometheus Agents", - "text": "step2.md", - "verify": "step2-verify.sh" - }, - { - "title": "Verify Setup", - "text": "step3.md", - "verify": "step3-verify.sh" - } - ], - "intro": { - "text": "intro.md", - "courseData": "courseBase.sh", - "credits": "https://thanos.io" - }, - "finish": { - "text": "finish.md", - "credits": "test" - } - }, - "files": [ - "prom-agent-batmobile.yaml", - "prom-agent-batcopter.yaml" - ], - "environment": { - "uilayout": "editor-terminal", - "uisettings": "yaml", - "uieditorpath": "/root/editor", - "showdashboard": true, - "dashboards": [ - {"name": "Prometheus Agent Batmobile", "port": 9090}, - {"name": "Prometheus Agent Batcopter", "port": 9091}, - {"name": "Thanos Query", "port": 39090} - ] - }, - "backend": { - "imageid": "docker-direct" - } -} diff --git a/tutorials/katacoda/thanos/4-receiver-agent/intro.md b/tutorials/katacoda/thanos/4-receiver-agent/intro.md deleted file mode 100644 index 51dc893b8c..0000000000 --- a/tutorials/katacoda/thanos/4-receiver-agent/intro.md +++ /dev/null @@ -1,30 +0,0 @@ -# Intermediate: (Bonus) Using Prometheus Agent for pure forward-only metric streaming with Thanos Receive - -The [Thanos](thanos.io) project defines a set of components that can be composed together into a highly available metric system with **unlimited storage capacity** that **seamlessly** integrates into your existing Prometheus deployments. - -But [Prometheus](https://prometheus.io/) project is far from slowing down the development. Together with the community, it constantly evolves to bring value to different network and cluster topologies that we see. Thanos brings distributed and cloud storage and querying to the table, built on the core Prometheus code and design. It allowed Prometheus to focus on critical collection and single cluster monitoring functionalities. - -As we learned in the previous tutorial, certain situations require us to collect (pull) data from applications and stream them out of the cluster as soon as possible. `Thanos Receive` allows doing that by ingesting metrics using the Remote Write protocol that the sender can implement. Typically we recommended using Prometheus with short retention and blocked read and query API as a "lightweight" sender. - -In November 2021, however, we, the Prometheus community, introduced a brand new Prometheus mode called "Agent mode". The implementation itself was already battle tested on https://github.com/grafana/agent, where it was available and authored by [Robert Fratto](https://github.com/rfratto) since 2020. - -The agent mode is optimized for efficient metric scraping and forwarding (i.e. immediate metric removal once it's securely delivered to a remote location). Since this is incredibly helpful for the Thanos community, we wanted to give you first-hand experience deploying Prometheus Agent together with Thanos Receive in this course. - -In this tutorial, you will learn: - -* How to reduce Prometheus based client-side metric collection to a minimum, using the new Prometheus "Agent mode" with `Thanos Receiver`, explained in the previous tutorial. - -> NOTE: This course uses docker containers with pre-built Thanos, Prometheus, and Minio Docker images available publicly. - -### Prerequisites - -Please complete tutorial #3 first: [Intermediate: Streaming metrics from remote source with Thanos Receive](https://www.katacoda.com/thanos/courses/thanos/3-receiver) πŸ€— - -### Feedback - -Do you see any bug, typo in the tutorial, or do you have some feedback for us? -Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io - -### Contributed by: - -* Bartek Plotka [@bwplotka](http://bwplotka.dev) diff --git a/tutorials/katacoda/thanos/4-receiver-agent/step1-verify.sh b/tutorials/katacoda/thanos/4-receiver-agent/step1-verify.sh deleted file mode 100644 index 5143a53c61..0000000000 --- a/tutorials/katacoda/thanos/4-receiver-agent/step1-verify.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/usr/bin/env bash - -# receive -curl -s 127.0.0.1:10909/metrics >/dev/null || exit 1 -# query -curl -s 127.0.0.1:39090/metrics >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/4-receiver-agent/step1.md b/tutorials/katacoda/thanos/4-receiver-agent/step1.md deleted file mode 100644 index 465f2974f8..0000000000 --- a/tutorials/katacoda/thanos/4-receiver-agent/step1.md +++ /dev/null @@ -1,56 +0,0 @@ -## Problem Statement - -Let's get back to our example from [Tutorial 3](https://www.katacoda.com/thanos/courses/thanos/3-receiver). Imagine you run a company called `Wayne Enterprises`. In tutorial 3, we established monitoring of two special clusters: `Batcave` & `Batcomputer`. These are special because they do not expose public endpoints to the Prometheus instances running there for security reasons, so we used the Remote Write protocol to stream all metrics to Thanos Receive in our centralized space. - -Let's imagine we want to expand our `Wayne Enterprises` by adding metrics collection to applications running on smaller devices inside more mobile Batman tools: `Batmobile` and `Batcopter`. - -Each of these vehicles has a smaller computer that runs applications from which we want to scrape Prometheus-like metrics using OpenMetrics format. - -As the person responsible for implementing monitoring in these environments, you have three requirements to meet: - -1. Implement a global view of this data. `Wayne Enterprises` needs to know what is happening in all company parts - including secret ones! -2. `Batmobile` and `Batcopter` can be out of network for some duration of time. You don't want to lose precious data. -3. `Batmobile` and `Batcopter` environments are very **resource constrained**, so you want an efficient solution that avoids extra computations and storage. - -Firstly, let us set up Thanos as we explained in the previous tutorial. - -## Setup Central Platform - -As you might remember from the previous tutorial, in the simplest form for streaming use cases, we need to deploy Thanos Receive and Thanos Querier. - -Let's run `Thanos Receive`: - -``` -docker run -d --rm \ - -v $(pwd)/receive-data:/receive/data \ - --net=host \ - --name receive \ - quay.io/thanos/thanos:v0.21.0 \ - receive \ - --tsdb.path "/receive/data" \ - --grpc-address 127.0.0.1:10907 \ - --http-address 127.0.0.1:10909 \ - --label "receive_replica=\"0\"" \ - --label "receive_cluster=\"wayne-enterprises\"" \ - --remote-write.address 127.0.0.1:10908 -```{{execute}} - -This starts Thanos Receive that listens on `http://127.0.0.1:10908/api/v1/receive` endpoint for Remote Write and on `127.0.0.1:10907` for Thanos StoreAPI. - -Next, let us run a `Thanos Query` instance connected to Thanos Receive: - -``` -docker run -d --rm \ ---net=host \ ---name query \ -quay.io/thanos/thanos:v0.21.0 \ -query \ ---http-address "0.0.0.0:39090" \ ---store "127.0.0.1:10907" -```{{execute}} - -Verify that `Thanos Query` is working and configured correctly by looking at the 'stores' tab [here](https://[[HOST_SUBDOMAIN]]-39090-[[KATACODA_HOST]].environments.katacoda.com/stores) (try refreshing the Thanos UI if it does not show up straight away). - -We should see Receive store on this page and, as expected, no metric data since we did not connect any Remote Write sender yet. - -With our "central" platform that is ready to ingest metrics, we can now start to architect our collection pipeline that will stream all metrics to our `Wayne Enterprises`. diff --git a/tutorials/katacoda/thanos/4-receiver-agent/step2-verify.sh b/tutorials/katacoda/thanos/4-receiver-agent/step2-verify.sh deleted file mode 100644 index be32a11ac9..0000000000 --- a/tutorials/katacoda/thanos/4-receiver-agent/step2-verify.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env bash - -# prometheus-batcave -curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1 -# prometheus-batcomputer -curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1 -# receive -curl -s 127.0.0.1:10909/metrics >/dev/null || exit 1 -# query -curl -s 127.0.0.1:39090/metrics >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/4-receiver-agent/step2.md b/tutorials/katacoda/thanos/4-receiver-agent/step2.md deleted file mode 100644 index 2ba7b9b8ae..0000000000 --- a/tutorials/katacoda/thanos/4-receiver-agent/step2.md +++ /dev/null @@ -1,104 +0,0 @@ - -# Setup `batmobile` and `batcopter` lightweight metric collection - -With `Wayne Enterprise` platform ready to ingest Remote Write data, we need to think about how we satisfy our three requirements: - -1. Implement a global view of this data. `Wayne Enterprises` needs to know what is happening in all company parts - including secret ones! -2. `Batmobile` and `Batcopter` can be out of network for some duration of time. You don't want to lose precious data. -3. `Batmobile` and `Batcopter` do not have large compute power, so you want an efficient solution that avoids extra computations and storage. - -How are we going to do this? - -Previously, the recommended approach was to deploy Prometheus to our edge environment which scrapes required applications, then remote writes all metrics to Thanos Receive. This will work and satisfy 1st and 2nd requirements, however Prometheus does some things that we don't need in this specific scenario: - -* We don't want to alert locally or use recording rules from our edge places. -* We don't want to query data locally, and we want to store it for an as short duration as possible. - -Prometheus was designed as a stateful time-series database, and it adds certain mechanics which are not desired for full forward mode. For example: - -* Prometheus builds additional memory structures for easy querying from memory. -* Prometheus does not remove data when it is safely sent via remote write. It waits for at least two hours and only after the TSDB block is persisted to the disk, it may or may not be removed, depending on retention configuration. - -This is where Agent mode comes in handy! It is a native Prometheus mode built into the Prometheus binary. If you add the `--agent` flag when running Prometheus, it will run a dedicated, specially streamlined database, optimized for forwarding purposes, yet able to persist scraped data in the event of a crash, restart or network disconnection. - -Let's try to deploy it to fulfil `batmobile` and `batcopter` monitoring requirements. - -## Deploy `Prometheus Agent` on `batmobile` - -Let's use a very simple configuration file that tells prometheus agent to scrape its own metrics page every 5 seconds and forwards it's to our running `Thanos Receive`. - -
-global:
-  scrape_interval: 5s
-  external_labels:
-    cluster: batmobile
-    replica: 0
-
-scrape_configs:
-  - job_name: 'prometheus-agent'
-    static_configs:
-      - targets: ['127.0.0.1:9090']
-
-remote_write:
-- url: 'http://127.0.0.1:10908/api/v1/receive'
-
- -Run the prometheus in agent mode: - -``` -docker run -d --net=host --rm \ --v /root/editor/prom-agent-batmobile.yaml:/etc/prometheus/prometheus.yaml \ --v /root/prom-batmobile-data:/prometheus \ --u root \ ---name prom-agent-batmobile \ -quay.io/prometheus/prometheus:v2.32.0-beta.0 \ ---enable-feature=agent \ ---config.file=/etc/prometheus/prometheus.yaml \ ---storage.agent.path=/prometheus \ ---web.listen-address=:9090 -```{{execute}} - -This runs Prometheus Agent, which will scrape itself and forward all to Thanos Receive. It also exposes UI with pages that relate to scraping, service discovery, configuration and build information. - -Verify that `prom-agent-batmobile` is running by navigating to the [Batmobile Prometheus Agent UI](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/targets). - -You should see one target: Prometheus Agent on `batmobile` itself. - -## Deploy `Prometheus Agent` on `batcopter` - -Similarly, we can configure and deploy the second agent: - -
-global:
-  scrape_interval: 5s
-  external_labels:
-    cluster: batcopter
-    replica: 0
-
-scrape_configs:
-  - job_name: 'prometheus-agent'
-    static_configs:
-      - targets: ['127.0.0.1:9091']
-
-remote_write:
-- url: 'http://127.0.0.1:10908/api/v1/receive'
-
- -``` -docker run -d --net=host --rm \ --v /root/editor/prom-agent-batcopter.yaml:/etc/prometheus/prometheus.yaml \ --v /root/prom-batcopter-data:/prometheus \ --u root \ ---name prom-agent-batcopter \ -quay.io/prometheus/prometheus:v2.32.0-beta.0 \ ---enable-feature=agent \ ---config.file=/etc/prometheus/prometheus.yaml \ ---storage.agent.path=/prometheus \ ---web.listen-address=:9091 -```{{execute}} - -Verify that `prom-agent-batcopter` is running by navigating to the [Batcopter Prometheus Agent UI](https://[[HOST_SUBDOMAIN]]-9091-[[KATACODA_HOST]].environments.katacoda.com/targets). - -You should see one target: Prometheus Agent on `batcopter` itself. - -Now, let's navigate to the last step to verify our `Wayne Enterprises` setup! diff --git a/tutorials/katacoda/thanos/4-receiver-agent/step3-verify.sh b/tutorials/katacoda/thanos/4-receiver-agent/step3-verify.sh deleted file mode 100644 index be32a11ac9..0000000000 --- a/tutorials/katacoda/thanos/4-receiver-agent/step3-verify.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env bash - -# prometheus-batcave -curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1 -# prometheus-batcomputer -curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1 -# receive -curl -s 127.0.0.1:10909/metrics >/dev/null || exit 1 -# query -curl -s 127.0.0.1:39090/metrics >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/4-receiver-agent/step3.md b/tutorials/katacoda/thanos/4-receiver-agent/step3.md deleted file mode 100644 index 4f0be5a42f..0000000000 --- a/tutorials/katacoda/thanos/4-receiver-agent/step3.md +++ /dev/null @@ -1,25 +0,0 @@ -# Verify Setup - -At this point, we have: - -* Two Prometheus instances configured to `remote_write` and running in `agent` mode. -* `Thanos Receive` component ingesting data from Prometheus -* `Thanos Query` component configured to query `Thanos Receive`'s Store API. - -The final task on our list is to verify that data is flowing correctly. - -Stop and think how you could do this before opening the answer below πŸ‘‡ - -
- How are we going to check that the components are wired up correctly? - - -Let's make sure that we can query data from each of our Prometheus instances from our `Thanos Query` instance. - -Navigate to the [Thanos Query UI](https://[[HOST_SUBDOMAIN]]-39090-[[KATACODA_HOST]].environments.katacoda.com), and query for a metric like `up` or `go_goroutines` - inspect the output and you should see `batmobile` and `batcopter` in the `cluster` label. - -`go_goroutines` should look something like on image below: - -![expected](./assets/expected.png) - -
\ No newline at end of file diff --git a/tutorials/katacoda/thanos/6-query-caching/courseBase.sh b/tutorials/katacoda/thanos/6-query-caching/courseBase.sh deleted file mode 100644 index 9d25c74c17..0000000000 --- a/tutorials/katacoda/thanos/6-query-caching/courseBase.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/usr/bin/env bash - -docker pull quay.io/prometheus/prometheus:v2.22.2 -docker pull quay.io/thanos/thanos:v0.26.0 -docker pull yannrobert/docker-nginx diff --git a/tutorials/katacoda/thanos/6-query-caching/finish.md b/tutorials/katacoda/thanos/6-query-caching/finish.md deleted file mode 100644 index cf66be277b..0000000000 --- a/tutorials/katacoda/thanos/6-query-caching/finish.md +++ /dev/null @@ -1,12 +0,0 @@ -# Summary - -Congratulations! πŸŽ‰πŸŽ‰πŸŽ‰ - -You completed our Query caching tutorial with Thanos course! -Go ahead and try it on your own stack. -To check out more advance patterns please have a look at [kube-thanos](https://github.com/thanos-io/kube-thanos)! - -### Feedback - -Do you see any bug, typo in the tutorial or you have some feedback for us? -Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io diff --git a/tutorials/katacoda/thanos/6-query-caching/index.json b/tutorials/katacoda/thanos/6-query-caching/index.json deleted file mode 100644 index 936da712a1..0000000000 --- a/tutorials/katacoda/thanos/6-query-caching/index.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "title": "Advanced: Querying with low tail-latency and low cost - Query caching with Thanos", - "description": "Learn how to introduce query caching using Query Frontend with Thanos", - "difficulty": "Advanced", - "details": { - "steps": [ - { - "title": "Initial Prometheus and Thanos Setup", - "text": "step1.md" - }, - { - "title": "Thanos Query Frontend", - "text": "step2.md" - } - ], - "intro": { - "text": "intro.md", - "courseData": "courseBase.sh" - }, - "finish": { - "text": "finish.md", - "credits": "https://thanos.io" - } - }, - "files": [ - "prometheus0.yml", - "prometheus1.yml", - "prometheus2.yml", - "frontend.yml", - "nginx.conf" - ], - "environment": { - "uilayout": "editor-terminal", - "uisettings": "yaml", - "showdashboard": true, - "dashboards": [ - { "name": "Prometheus", "port": 9090 }, - { "name": "Thanos Query", "port": 10902 }, - { "name": "Thanos Query Frontend", "port": 20902 } - ] - }, - "backend": { - "imageid": "docker-direct" - } -} diff --git a/tutorials/katacoda/thanos/6-query-caching/intro.md b/tutorials/katacoda/thanos/6-query-caching/intro.md deleted file mode 100644 index bc1ada3e23..0000000000 --- a/tutorials/katacoda/thanos/6-query-caching/intro.md +++ /dev/null @@ -1,23 +0,0 @@ -# Advanced: Querying with low tail-latency and low cost - Query caching with Thanos - -Welcome to the tutorial where you learn how to introduce query caching using Query Frontend with Thanos. - -In this tutorial you will learn: - -* Setup a simple observability stack using Prometheus -* Deploy and configure Thanos Query Frontend in front of your Queriers so that we can cache query responses - -We aim to achieve: - -* Low cost query execution -* Low latency query execution -* Si reduce cost of infrastructure - -### Feedback - -Do you see any bug, typo in the tutorial or you have some feedback for us? -Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io - -### Contributed by: - -* Kemal [@kakkoyun](https://kakkoyun.me/) diff --git a/tutorials/katacoda/thanos/6-query-caching/step1.md b/tutorials/katacoda/thanos/6-query-caching/step1.md deleted file mode 100644 index 84eb04f9df..0000000000 --- a/tutorials/katacoda/thanos/6-query-caching/step1.md +++ /dev/null @@ -1,147 +0,0 @@ - -## Simple observability stack with Prometheus and Thanos - -> NOTE: Click `Copy To Editor` for each config to propagate the configs to each file. - -Let's imagine we have to deliver centralized metrics platform to multiple teams. For each team we will have a dedicated Prometheus. These could be in the same environment or in different environments (data centers, regions, clusters etc). - -And then we will try to provide low cost, fast global overview. Let's see how we achieve that with Thanos. - -## Let's lay the ground work - -Let's quickly deploy Prometheuses with sidecars and Querier. - -Execute following commands to setup Prometheus: - -### Prepare "persistent volumes" for Prometheuses - -``` -mkdir -p prometheus_data -```{{execute}} - -### Deploy Prometheus - -Let's deploy a couple of Prometheus instances and let them scrape themselves, so we can produce some metrics. - -### Prepare configuration - -Click `Copy To Editor` for each config to propagate the configs to each file. - -First, Prometheus server that scrapes itself: - -
-global:
-  scrape_interval: 5s
-  external_labels:
-    cluster: eu0
-    replica: 0
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9090']
-
- -
-global:
-  scrape_interval: 5s
-  external_labels:
-    cluster: eu1
-    replica: 0
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9091']
-
- -
-global:
-  scrape_interval: 5s
-  external_labels:
-    cluster: eu2
-    replica: 0
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9092']
-
- -### Deploy Prometheus - -``` -for i in $(seq 0 2); do -docker run -d --net=host --rm \ - -v $(pwd)/prometheus"${i}".yml:/etc/prometheus/prometheus.yml \ - -v $(pwd)/prometheus_data:/prometheus"${i}" \ - -u root \ - --name prometheus"${i}" \ - quay.io/prometheus/prometheus:v2.22.2 \ - --config.file=/etc/prometheus/prometheus.yml \ - --storage.tsdb.path=/prometheus \ - --web.listen-address=:909"${i}" \ - --web.external-url=https://[[HOST_SUBDOMAIN]]-909"${i}"-[[KATACODA_HOST]].environments.katacoda.com \ - --web.enable-lifecycle \ - --web.enable-admin-api && echo "Prometheus ${i} started!" -done -```{{execute}} - -#### Verify - -Let's check if all of Prometheuses are running! - -``` -docker ps -```{{execute}} - -### Inject Thanos Sidecars - -``` -for i in $(seq 0 2); do -docker run -d --net=host --rm \ - -v $(pwd)/prometheus"${i}".yml:/etc/prometheus/prometheus.yml \ - --name prometheus-sidecar"${i}" \ - -u root \ - quay.io/thanos/thanos:v0.26.0 \ - sidecar \ - --http-address=0.0.0.0:1909"${i}" \ - --grpc-address=0.0.0.0:1919"${i}" \ - --reloader.config-file=/etc/prometheus/prometheus.yml \ - --prometheus.url=http://127.0.0.1:909"${i}" && echo "Started Thanos Sidecar for Prometheus ${i}!" -done -```{{execute}} - -#### Verify - -Let's check if all of Thanos Sidecars are running! - -``` -docker ps -```{{execute}} - -## Prepare Thanos Global View - -And now, let's deploy Thanos Querier to have a global overview on our services. - -### Deploy Querier - -``` -docker run -d --net=host --rm \ - --name querier \ - quay.io/thanos/thanos:v0.26.0 \ - query \ - --http-address 0.0.0.0:10912 \ - --grpc-address 0.0.0.0:10901 \ - --query.replica-label replica \ - --store 127.0.0.1:19190 \ - --store 127.0.0.1:19191 \ - --store 127.0.0.1:19192 && echo "Started Thanos Querier!" -```{{execute}} - -### Setup Verification - -Once started you should be able to reach the Querier and Prometheus. - -* [Prometheus](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/) -* [Querier](https://[[HOST_SUBDOMAIN]]-10912-[[KATACODA_HOST]].environments.katacoda.com/) diff --git a/tutorials/katacoda/thanos/6-query-caching/step2.md b/tutorials/katacoda/thanos/6-query-caching/step2.md deleted file mode 100644 index e9c99b2aaf..0000000000 --- a/tutorials/katacoda/thanos/6-query-caching/step2.md +++ /dev/null @@ -1,95 +0,0 @@ - - -> NOTE: Click `Copy To Editor` for each config to propagate the configs to each file. - -What if we can have one single point of entry in front of Queries instead of separate Queriers? And by doing so slice and dice our queries depend on the time and distribute them between queriers to balance the load? Moreover, why not cache these responses so that next time someone asks for the same time range we can just serve it from memory. Wouldn't it be faster? - -Yes, we can do all these using Thanos Query Frontend. Let's see how we can do it. - -### First let's deploy a nginx proxy to simulate latency - -We are running this tutorial on a single machine in this setup, as a result it's really hard to observe latencies that you would normally experience in a real-life setups. In order to simulate a real-life latency we are going to put a proxy in front of our Thanos Querier. - -For that let's setup a nginx instance: - -
-server {
- listen 10902;
- server_name proxy;
- location / {
-  echo_exec @default;
- }
- location ^~ /api/v1/query_range {
-     echo_sleep 1;
-     echo_exec @default;
- }
- location @default {
-     proxy_pass http://127.0.0.1:10912;
- }
-}
-
- -``` -docker run -d --net=host --rm \ - -v $(pwd)/nginx.conf:/etc/nginx/conf.d/default.conf \ - --name nginx \ - yannrobert/docker-nginx && echo "Started Querier Proxy!" -```{{execute}} - -### Verify - -Let's check if it's running! - -``` -docker ps -```{{execute}} - -## Deploy Thanos Query Frontend - -First, let's create necessary cache configuration for Frontend: - -
-type: IN-MEMORY
-config:
-  max_size: "0"
-  max_size_items: 2048
-  validity: "6h"
-
- -And deploy Query Frontend: - -``` -docker run -d --net=host --rm \ - -v $(pwd)/frontend.yml:/etc/thanos/frontend.yml \ - --name query-frontend \ - quay.io/thanos/thanos:v0.26.0 \ - query-frontend \ - --http-address 0.0.0.0:20902 \ - --query-frontend.compress-responses \ - --query-frontend.downstream-url=http://127.0.0.1:10902 \ - --query-frontend.log-queries-longer-than=5s \ - --query-range.split-interval=1m \ - --query-range.response-cache-max-freshness=1m \ - --query-range.max-retries-per-request=5 \ - --query-range.response-cache-config-file=/etc/thanos/frontend.yml \ - --cache-compression-type="snappy" && echo "Started Thanos Query Frontend!" -```{{execute}} - -### Setup Verification - -Once started you should be able to reach the Querier, Query Frontend and Prometheus. - -* [Prometheus](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/) -* [Querier](https://[[HOST_SUBDOMAIN]]-10902-[[KATACODA_HOST]].environments.katacoda.com/) -* [Query Frontend](https://[[HOST_SUBDOMAIN]]-20902-[[KATACODA_HOST]].environments.katacoda.com/) - -Now, go and execute a query on [Querier](https://[[HOST_SUBDOMAIN]]-10902-[[KATACODA_HOST]].environments.katacoda.com/) and observe the latency. -And then go and execute the same query on [Query Frontend](https://[[HOST_SUBDOMAIN]]-20902-[[KATACODA_HOST]].environments.katacoda.com/). -For the first execution you will observe that the query execution takes longer than the query on Querier. -That's because we have an nginx proxy between Query Frontend and Querier. - -Now if you execute the same query again on Query Frontend for the same time frame using time selector in graph section in the UI (time is always shifting). -See that it's much faster? -It's taking much less time because we are just serving the response from the cached results. - -Good! You've done it! diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/assets/no-isolation.png b/tutorials/katacoda/thanos/7-multi-tenancy/assets/no-isolation.png deleted file mode 100644 index a3774c7793..0000000000 Binary files a/tutorials/katacoda/thanos/7-multi-tenancy/assets/no-isolation.png and /dev/null differ diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/assets/no-read-tenancy-reuse.png b/tutorials/katacoda/thanos/7-multi-tenancy/assets/no-read-tenancy-reuse.png deleted file mode 100644 index 8deadbb710..0000000000 Binary files a/tutorials/katacoda/thanos/7-multi-tenancy/assets/no-read-tenancy-reuse.png and /dev/null differ diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/assets/no-read-tenancy-tomato.png b/tutorials/katacoda/thanos/7-multi-tenancy/assets/no-read-tenancy-tomato.png deleted file mode 100644 index 4c0f05896a..0000000000 Binary files a/tutorials/katacoda/thanos/7-multi-tenancy/assets/no-read-tenancy-tomato.png and /dev/null differ diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/assets/no-read-tenancy.png b/tutorials/katacoda/thanos/7-multi-tenancy/assets/no-read-tenancy.png deleted file mode 100644 index 8639dd671c..0000000000 Binary files a/tutorials/katacoda/thanos/7-multi-tenancy/assets/no-read-tenancy.png and /dev/null differ diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/assets/read-soft-tenancy.png b/tutorials/katacoda/thanos/7-multi-tenancy/assets/read-soft-tenancy.png deleted file mode 100644 index 3334c81b3f..0000000000 Binary files a/tutorials/katacoda/thanos/7-multi-tenancy/assets/read-soft-tenancy.png and /dev/null differ diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh b/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh deleted file mode 100644 index 8260602ffe..0000000000 --- a/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/usr/bin/env bash - -docker pull quay.io/prometheus/prometheus:v2.20.0 -docker pull quay.io/thanos/thanos:v0.26.0 -docker pull quay.io/thanos/prom-label-proxy:v0.3.0-rc.0-ext1 -docker pull caddy:2.2.1 - -mkdir /root/editor diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/finish.md b/tutorials/katacoda/thanos/7-multi-tenancy/finish.md deleted file mode 100644 index 70c2dc8a85..0000000000 --- a/tutorials/katacoda/thanos/7-multi-tenancy/finish.md +++ /dev/null @@ -1,10 +0,0 @@ -# Summary - -Congratulations! πŸŽ‰πŸŽ‰πŸŽ‰ - -You completed our Multi-tenancy with Thanos course! - -### Feedback - -Do you see any bug, typo in the tutorial or you have some feedback for us? -Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/index.json b/tutorials/katacoda/thanos/7-multi-tenancy/index.json deleted file mode 100644 index be7ea37717..0000000000 --- a/tutorials/katacoda/thanos/7-multi-tenancy/index.json +++ /dev/null @@ -1,48 +0,0 @@ -{ - "title": "Advanced: Achieving Multi-Tenancy with Thanos", - "description": "Extend your Prometheus setup for Multiple Teams use with Thanos", - "difficulty": "Advanced", - "details": { - "steps": [ - { - "title": "So You Have Two Teams?", - "text": "step1.md" - }, - { - "title": "Thanos Query Multi Tenancy", - "text": "step2.md" - } - ], - "intro": { - "text": "intro.md", - "courseData": "courseBase.sh" - }, - "finish": { - "text": "finish.md", - "credits": "https://thanos.io" - } - }, - "files": [ - "prometheus0_fruit.yml", - "prometheus0_veggie.yml", - "prometheus1_veggie.yml", - "Caddyfile" - ], - "environment": { - "uilayout": "editor-terminal", - "uisettings": "yaml", - "showdashboard": true, - "uieditorpath": "/root/editor", - "dashboards": [ - {"name": "Prometheus 0 Fruit", "port": 9090}, - {"name": "Prometheus 0 Veggie", "port": 9091}, - {"name": "Prometheus 1 Veggie", "port": 9091}, - {"name": "Thanos Query", "port": 29090}, - {"name": "Thanos Query for Fruit Team", "port": 39091}, - {"name": "Thanos Query for Veggie Team", "port": 39092} - ] - }, - "backend": { - "imageid": "docker-direct" - } -} diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/intro.md b/tutorials/katacoda/thanos/7-multi-tenancy/intro.md deleted file mode 100644 index b1cb5a6ac5..0000000000 --- a/tutorials/katacoda/thanos/7-multi-tenancy/intro.md +++ /dev/null @@ -1,23 +0,0 @@ -# Advanced: Achieving Multi-Tenancy with Thanos - -Welcome to the tutorial where you learn how to manage fruits πŸ‡ and vegetables πŸ₯¦ reliably and efficiently. - -On a serious note, we will look into how to use Thanos for multi-tenant use cases. Centralized, long-term -metrics capabilities are very addictive, so it's very often the case that multiple teams want to use Thanos without -impacting each other or accessing each other data. - -Thanos was built with this in mind and it in this tutorial you will learn: - -* Query Read **Hard-Tenancy**: How to setup it and what are the tradeoffs -* How to support πŸ… (seriously!) and reduce cost of infrastructure. -* Query Read **Soft-Tenancy**: How to setup it and what are the tradeoffs (feat: [prom-label-proxy](https://github.com/prometheus-community/prom-label-proxy)) - -### Feedback - -Do you see any bug, typo in the tutorial or you have some feedback for us? -Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io - -### Contributed by: - -* Kemal [@kakkoyun](https://kakkoyun.me/) -* Bartek [@bwplotka](https://bwplotka.dev/) diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/step1-verify.sh b/tutorials/katacoda/thanos/7-multi-tenancy/step1-verify.sh deleted file mode 100644 index cee0ed0a9b..0000000000 --- a/tutorials/katacoda/thanos/7-multi-tenancy/step1-verify.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/usr/bin/env bash - -curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:9092/metrics >/dev/null || exit 1 - -curl -s 127.0.0.1:29091/metrics >/dev/null || exit 1 -curl -s 127.0.0.1:29092/metrics >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/step1.md b/tutorials/katacoda/thanos/7-multi-tenancy/step1.md deleted file mode 100644 index 7a0c6d253f..0000000000 --- a/tutorials/katacoda/thanos/7-multi-tenancy/step1.md +++ /dev/null @@ -1,230 +0,0 @@ -## Scenario - -> NOTE: Click `Copy To Editor` for each config to propagate the configs to each file. - -Let's imagine we have to deliver centralized metrics platform to two teams **Team Fruit** and **Team Veggie**. -We don't want each team to see each other data or even know about their existence. Let's see how we achieve that with Thanos. - -## Starting Fruit and Veggie Prometheus and Thanos Global View - -Fruits and Veggies want to allow more Prometheus and replicas at some point so they want to have Thanos upfront. Let's quickly -deploy Prometheuses with sidecars and Querier. - -### Configure Prometheus-es - -First, Prometheus server for **Team Fruit** that scrapes itself: - -
-global:
-  scrape_interval: 5s
-  external_labels:
-    cluster: eu1
-    replica: 0
-    tenant: team-fruit
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9090']
-
- -For the **Team Veggie** we set second instance with two replicas (Veggies care for high availability - everyone should eat vegetables every day after all!): - -
-global:
-  scrape_interval: 5s
-  external_labels:
-    cluster: eu1
-    replica: 0
-    tenant: team-veggie
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-     - targets: ['127.0.0.1:9091','127.0.0.1:9092']
-
- -
-global:
-  scrape_interval: 5s
-  external_labels:
-    cluster: eu1
-    replica: 1
-    tenant: team-veggie
-
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['127.0.0.1:9091','127.0.0.1:9092']
-
- -### Prepare "persistent volumes" - -Execute following commands: - -``` -mkdir -p prometheus0_fruit_data prometheus0_veggie_data prometheus1_veggie_data -```{{execute}} - -### Deploying Team Fruit Prometheus with sidecar - -``` -docker run -d --net=host --rm \ - -v $(pwd)/editor/prometheus0_fruit.yml:/etc/prometheus/prometheus.yml \ - -v $(pwd)/prometheus0_fruit_data:/prometheus \ - -u root \ - --name prometheus-0-fruit \ - quay.io/prometheus/prometheus:v2.20.0 \ - --config.file=/etc/prometheus/prometheus.yml \ - --storage.tsdb.path=/prometheus \ - --web.listen-address=:9090 \ - --web.external-url=https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com \ - --web.enable-lifecycle \ - --web.enable-admin-api && echo "Prometheus for Fruit Team started!" -```{{execute}} - -``` -docker run -d --net=host --rm \ - -v $(pwd)/editor/prometheus0_fruit.yml:/etc/prometheus/prometheus.yml \ - --name prometheus-0-sidecar-fruit \ - -u root \ - quay.io/thanos/thanos:v0.26.0 \ - sidecar \ - --http-address 0.0.0.0:19090 \ - --grpc-address 0.0.0.0:19190 \ - --reloader.config-file /etc/prometheus/prometheus.yml \ - --prometheus.url http://127.0.0.1:9090 && echo "Started sidecar for Fruit Prometheus" -```{{execute}} - -### Same for Team Veggie, but with 2-replica Prometheus: - -First: - -``` -docker run -d --net=host --rm \ - -v $(pwd)/editor/prometheus0_veggie.yml:/etc/prometheus/prometheus.yml \ - -v $(pwd)/prometheus0_veggie_data:/prometheus \ - -u root \ - --name prometheus-0-veggie \ - quay.io/prometheus/prometheus:v2.20.0 \ - --config.file=/etc/prometheus/prometheus.yml \ - --storage.tsdb.path=/prometheus \ - --web.listen-address=:9091 \ - --web.external-url=https://[[HOST_SUBDOMAIN]]-9091-[[KATACODA_HOST]].environments.katacoda.com \ - --web.enable-lifecycle \ - --web.enable-admin-api && echo "Prometheus for Veggie Team started!" -```{{execute}} - -``` -docker run -d --net=host --rm \ - -v $(pwd)/editor/prometheus0_veggie.yml:/etc/prometheus/prometheus.yml \ - --name prometheus-0-sidecar-veggie \ - -u root \ - quay.io/thanos/thanos:v0.26.0 \ - sidecar \ - --http-address 0.0.0.0:19091 \ - --grpc-address 0.0.0.0:19191 \ - --reloader.config-file /etc/prometheus/prometheus.yml \ - --prometheus.url http://127.0.0.1:9091 && echo "Started sidecar for Veggie Prometheus" -```{{execute}} - -Second: - -``` -docker run -d --net=host --rm \ - -v $(pwd)/editor/prometheus1_veggie.yml:/etc/prometheus/prometheus.yml \ - -v $(pwd)/prometheus1_veggie_data:/prometheus \ - -u root \ - --name prometheus-1-veggie \ - quay.io/prometheus/prometheus:v2.20.0 \ - --config.file=/etc/prometheus/prometheus.yml \ - --storage.tsdb.path=/prometheus \ - --web.listen-address=:9092 \ - --web.external-url=https://[[HOST_SUBDOMAIN]]-9092-[[KATACODA_HOST]].environments.katacoda.com \ - --web.enable-lifecycle \ - --web.enable-admin-api && echo "Prometheus for Veggie Team started!" -```{{execute}} - -Second: - -``` -docker run -d --net=host --rm \ - -v $(pwd)/editor/prometheus1_veggie.yml:/etc/prometheus/prometheus.yml \ - --name prometheus-01-sidecar-veggie \ - -u root \ - quay.io/thanos/thanos:v0.26.0 \ - sidecar \ - --http-address 0.0.0.0:19092 \ - --grpc-address 0.0.0.0:19192 \ - --reloader.config-file /etc/prometheus/prometheus.yml \ - --prometheus.url http://127.0.0.1:9092 && echo "Started sidecar for Veggie Prometheus" -```{{execute}} - -### Querier - -Now the naive approach to ensure querying isolation (we can't afford veggies to look on fruits data!) would be to setup separate -isolated Queriers for each team, so let's start with that. - -Fruit: - -``` -docker run -d --net=host --rm \ - --name querier-fruit \ - quay.io/thanos/thanos:v0.26.0 \ - query \ - --http-address 0.0.0.0:29091 \ - --grpc-address 0.0.0.0:29191 \ - --query.replica-label replica \ - --store 127.0.0.1:19190 && echo "Started Thanos Fruit Querier" -```{{execute}} - -Veggie: - -``` -docker run -d --net=host --rm \ - --name querier-veggie \ - quay.io/thanos/thanos:v0.26.0 \ - query \ - --http-address 0.0.0.0:29092 \ - --grpc-address 0.0.0.0:29192 \ - --query.replica-label replica \ - --store 127.0.0.1:19191 \ - --store 127.0.0.1:19192 && echo "Started Thanos Veggie Querier" -```{{execute}} - -### Setup Verification - -At the end we should see this case: - -![diagram](./assets/no-read-tenancy.png) - -This setup can be called "No or Hard Tenancy" where we are setting up separate components (technically disconnected two systems) for -each of tenants. - -Once started you should be able to reach both Queriers - each exposing either Fruit's or Veggies's data: - -* [Fruit Query](https://[[HOST_SUBDOMAIN]]-29091-[[KATACODA_HOST]].environments.katacoda.com/) -* [Veggies Query](https://[[HOST_SUBDOMAIN]]-29092-[[KATACODA_HOST]].environments.katacoda.com/) - -## Problem statement 1: Tomato problem. - -Let's try to play with this setup a bit. You are free to query any metrics. Data isolation is there, each team has its own -endpoint. - -However, let's try to imagine a common case in such setups: **What if you are.. a Tomato?** Surprisingly Tomato [is both -fruit and vegetable](https://www.sciencealert.com/here-s-why-a-tomato-is-actually-both-a-fruit-and-vegetable), so it -should be able not only to access metrics from both Veggie and Fruit teams, but also run PromQL across them. - -![diagram](./assets/no-read-tenancy-tomato.png) - -We call this "Tomato" problem a **Cross-tenant or Admin View** and it's a common case when teams/tenants are changing, users have access to multiple -tenants data etc. This can be solved with another layer of global view: we know how to solve this problem from previous courses, we could -start another Querier on top of our Team's Queries and open another endpoint just for Tomatoes. But here comes another problem... - -## Problem statement 2: Exclusive Infra for Each Tenant is very Expensive and does not scale. - -![diagram](./assets/no-read-tenancy-reuse.png) - -## Don't worry: Thanos was built with multi-tenancy in mind! - -See next step to learn more. diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/step2-verify.sh b/tutorials/katacoda/thanos/7-multi-tenancy/step2-verify.sh deleted file mode 100644 index 597aa3c14f..0000000000 --- a/tutorials/katacoda/thanos/7-multi-tenancy/step2-verify.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/usr/bin/env bash - -curl -s 127.0.0.1:29090/metrics >/dev/null || exit 1 - -curl -s 127.0.0.1:39091 >/dev/null || exit 1 -curl -s 127.0.0.1:39092 >/dev/null || exit 1 - -echo '"done"' diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/step2.md b/tutorials/katacoda/thanos/7-multi-tenancy/step2.md deleted file mode 100644 index 9f3e9e7295..0000000000 --- a/tutorials/katacoda/thanos/7-multi-tenancy/step2.md +++ /dev/null @@ -1,122 +0,0 @@ -## Reuse More! - -What if we can have one set of Queries instead of separate set for each tenant? Why not reusing exactly same component? - -Let's stop fruit and veggies queriers and run single one spanning all the tenant's Prometheus data: - -``` -docker stop querier-fruit && docker stop querier-veggie -```{{execute}} - -``` -docker run -d --net=host --rm \ - --name querier-multi \ - quay.io/thanos/thanos:v0.26.0 \ - query \ - --http-address 0.0.0.0:29090 \ - --grpc-address 0.0.0.0:29190 \ - --query.replica-label replica \ - --store 127.0.0.1:19190 \ - --store 127.0.0.1:19191 \ - --store 127.0.0.1:19192 && echo "Started Thanos Querier with access to both Veggie's and Fruit's data" -```{{execute}} - -Within short time we should be able to see "Tomato" view [when we open Querier UI](https://[[HOST_SUBDOMAIN]]-29090-[[KATACODA_HOST]].environments.katacoda.com/) - -## Tenant Query Isolation - -Undoubtedly, the main problem with this setup is that by default **every tenant will see each other data**, similar to what you have in Prometheus, -if single Prometheus scrapes data from multiple teams. - -![diagram](./assets/no-isolation.png) - -Both Prometheus and Thanos [follow UNIX philosopy](https://github.com/thanos-io/thanos#thanos-philosophy). **One of the principles is to ensure -each component is doing one thing and do it well**. Thanos Querier does not perform any authentication or authorization. -This is because you probably already have consistent auth mechanism in your organization. So why not composing that with flexible -flat label pairs identifying the data blocks and each individual series for data isolation? - -### Meet [prom-label-proxy](https://github.com/prometheus-community/prom-label-proxy) - -[prom-label-proxy](https://github.com/prometheus-community/prom-label-proxy) allows read tenancy for all the -resources that Prometheus and Thanos currently exposes, by enforcing certain `tenant` label to be used in all APIs retrieving the data. -This proxy works natively for Prometheus, but since Thanos uses same HTTP APIs on top, it will work for us as well. - -So why not we start something like this in front of our "Tomato" Querier? - -``` -docker run -d --net=host --rm \ - --name prom-label-proxy \ - quay.io/prometheuscommunity/prom-label-proxy:v0.3.0 \ - -label tenant \ - -upstream http://127.0.0.1:29090 \ - -insecure-listen-address 0.0.0.0:39090 \ - -enable-label-apis && echo "Started prom-label-proxy" -```{{execute}} - -### Laveraging prom-label-proxy - -All requests now have to have extra URL parameter `tenant=` with the value being tenant to limit scope with. - -Our running proxy does not do any authN or authZ for us - so let's setup some basic flow. To make it simple, -let's deploy [Caddy](https://caddyserver.com/) server (kind of fancy nginx but written in Go) that will expose two ports: -`39091` that will redirect to prom-label-proxy with `tenant=team-fruit` and `39092` for `tenant=team-veggie` injection. - -Let's create Caddy config file: - -
-{
-    admin off
-}
-
-:39091  {
-    rewrite * ?{query}&tenant=team-fruit
-    reverse_proxy 127.0.0.1:39090
-}
-
-:39092 {
-    rewrite * ?{query}&tenant=team-veggie
-    reverse_proxy 127.0.0.1:39090
-}
-
- -And start our caddy server using that config: - -``` -docker run -d --net=host --rm \ - --name caddy \ - -v $PWD/editor/Caddyfile:/etc/caddy/Caddyfile \ - caddy:2.2.1 && echo "Started Caddy Server" -```{{execute}} - -### Verification - -At the end we should have setup as on following diagram: - -![diagram](./assets/read-soft-tenancy.png) - -Let's check if our read isolation works: - -#### Team Fruit - -Firstly for `Team Fruit`, let's make a query for some data: -``` -curl -g 'http://127.0.0.1:39091/api/v1/query?query=up' -```{{execute}} -Inspecting the output we should only see metrics with `"tenant":"team-fruit"`. - -#### Team Veggie - -Secondly for `Team Veggie`, let's make the same query to the other port: -``` -curl -g 'http://127.0.0.1:39092/api/v1/query?query=up' -```{{execute}} - Inspecting the output we should only see metrics with `"tenant":"team-veggie"`. - -Feel free to play around, you will see that we can only see Fruit or Veggie data depends where we go! - -## Next steps - -Similarly for Write and Storage you can deploy Thanos Hard or Soft tenant components as you wish. Follow up our docs and KubeCon -talks to learn more! πŸ€— - -When you ready "click" next to finish this course. diff --git a/tutorials/katacoda/thanos/x-more-to-come/courseBase.sh b/tutorials/katacoda/thanos/x-more-to-come/courseBase.sh deleted file mode 100644 index f1f641af19..0000000000 --- a/tutorials/katacoda/thanos/x-more-to-come/courseBase.sh +++ /dev/null @@ -1 +0,0 @@ -#!/usr/bin/env bash diff --git a/tutorials/katacoda/thanos/x-more-to-come/index.json b/tutorials/katacoda/thanos/x-more-to-come/index.json deleted file mode 100644 index 68c9d53f3a..0000000000 --- a/tutorials/katacoda/thanos/x-more-to-come/index.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "title": "", - "description": "Anything you want to like to see? Or you want to help? Let us know!", - "details": { - "steps": [ - ], - "intro": { - "text": "intro.md", - "courseData": "courseBase.sh" - } - }, - "environment": { - "uilayout": "editor-terminal", - "uisettings": "yaml" - }, - "backend": { - "imageid": "docker-direct" - } -} \ No newline at end of file diff --git a/tutorials/katacoda/thanos/x-more-to-come/intro.md b/tutorials/katacoda/thanos/x-more-to-come/intro.md deleted file mode 100644 index 68b6e43d0b..0000000000 --- a/tutorials/katacoda/thanos/x-more-to-come/intro.md +++ /dev/null @@ -1,7 +0,0 @@ -# Work in Progress - -🚧 More tutorials are in progress. 🚧 - -Do you want to see this soon or you want to help us? - -Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io \ No newline at end of file diff --git a/tutorials/katacoda/thanos/x-playground/courseBase.sh b/tutorials/katacoda/thanos/x-playground/courseBase.sh deleted file mode 100644 index cd719a4d16..0000000000 --- a/tutorials/katacoda/thanos/x-playground/courseBase.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/usr/bin/env bash - -docker pull quay.io/prometheus/prometheus:v2.20.0 -docker pull quay.io/thanos/thanos:v0.26.0 -docker pull quay.io/thanos/thanosbench:v0.2.0-rc.1 -docker pull minio/minio:RELEASE.2019-01-31T00-31-19Z - -mkdir /root/editor diff --git a/tutorials/katacoda/thanos/x-playground/finish.md b/tutorials/katacoda/thanos/x-playground/finish.md deleted file mode 100644 index b138775e79..0000000000 --- a/tutorials/katacoda/thanos/x-playground/finish.md +++ /dev/null @@ -1,25 +0,0 @@ -# Summary - -Uff done. - - - -### Cleanup if you ran locally - -``` -docker stop prom-eu1-0 -docker stop prom-eu1-1 -docker stop prom-us1-0 -docker stop prom-eu1-0-sidecar -docker stop prom-eu1-1-sidecar -docker stop prom-us1-0-sidecar -docker stop querier -docker stop minio -docker stop store-gateway -docker stop compactor -``` - -### Feedback - -Do you see any bug, typo in the tutorial or you have some feedback for us? -Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io diff --git a/tutorials/katacoda/thanos/x-playground/index.json b/tutorials/katacoda/thanos/x-playground/index.json deleted file mode 100644 index 31d2c43edd..0000000000 --- a/tutorials/katacoda/thanos/x-playground/index.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "title": "Playground: All-in, Interactive Thanos un-guided Demo", - "description": "\uD83E\uDD14\uD83E\uDD14", - "difficulty": "YOLO", - "details": { - "steps": [ - { - "title": "Yolo Demo Resources", - "text": "step1.md" - }, - { - "title": "Yolo Demo Resources", - "text": "step2.md" - } - ], - "intro": { - "text": "intro.md", - "courseData": "courseBase.sh" - } - }, - "environment": { - "uilayout": "terminal", - "uisettings": "yaml", - "showdashboard": true, - "uieditorpath": "/root/editor" - }, - "backend": { - "imageid": "docker-direct" - } -} \ No newline at end of file diff --git a/tutorials/katacoda/thanos/x-playground/intro.md b/tutorials/katacoda/thanos/x-playground/intro.md deleted file mode 100644 index 4d96387528..0000000000 --- a/tutorials/katacoda/thanos/x-playground/intro.md +++ /dev/null @@ -1,18 +0,0 @@ -# Demo Resources - -Let's go! - -This playground was created for `@rawkode` live demo made by `@bwplotka` and David McKay. - -You can watch all here: https://www.youtube.com/watch?v=j4TAGO019HU&feature=emb_title&ab_channel=DavidMcKay - -Otherwise, have fun with those resources! - -### Feedback - -Do you see any bug, typo in the tutorial or you have some feedback for us? -Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io - -### Contributed by: - -* Bartek [@bwplotka](https://bwplotka.dev/) \ No newline at end of file diff --git a/tutorials/katacoda/thanos/x-playground/step1.md b/tutorials/katacoda/thanos/x-playground/step1.md deleted file mode 100644 index ba0c196f62..0000000000 --- a/tutorials/katacoda/thanos/x-playground/step1.md +++ /dev/null @@ -1,236 +0,0 @@ -## Global View + HA: Querier with 1y of data. - -``` -cd /root/editor && CURR_DIR=$(pwd) -```{{execute}} - -### Generating data: - -* Plan for 1y worth of metrics data: - -``` -docker run -it --rm quay.io/thanos/thanosbench:v0.2.0-rc.1 block plan -p continuous-365d-tiny --max-time=6h > ${CURR_DIR}/block-spec.yaml -```{{execute}} - -* Plan and generate data for `cluster="eu1", replica="0"` Prometheus: - -``` -mkdir ${CURR_DIR}/prom-eu1-replica0 && docker run -it --rm quay.io/thanos/thanosbench:v0.2.0-rc.1 block plan -p continuous-365d-tiny --labels 'cluster="eu1"' --max-time=6h | \ - docker run -v ${CURR_DIR}/prom-eu1-replica0:/out -i quay.io/thanos/thanosbench:v0.2.0-rc.1 block gen --output.dir /out -```{{execute}} - -* Plan and generate data for `cluster="eu1", replica="1"` Prometheus: - -``` -mkdir ${CURR_DIR}/prom-eu1-replica1 && docker run -it --rm quay.io/thanos/thanosbench:v0.2.0-rc.1 block plan -p continuous-365d-tiny --labels 'cluster="eu1"' --max-time=6h | \ - docker run -v ${CURR_DIR}/prom-eu1-replica1:/out -i quay.io/thanos/thanosbench:v0.2.0-rc.1 block gen --output.dir /out -```{{execute}} - -* Plan and generate data for `cluster="us1", replica="0"` Prometheus: - -``` -mkdir ${CURR_DIR}/prom-us1-replica0 && docker run -it --rm quay.io/thanos/thanosbench:v0.2.0-rc.1 block plan -p continuous-365d-tiny --labels 'cluster="us1"' --max-time=6h | \ - docker run -v ${CURR_DIR}/prom-us1-replica0:/out -i quay.io/thanos/thanosbench:v0.2.0-rc.1 block gen --output.dir /out -```{{execute}} - -### Create Promethues configs to use: - -* `cluster="eu1", replica="0"` Prometheus: - -``` -cat < ${CURR_DIR}/prom-eu1-replica0-config.yaml -global: - scrape_interval: 5s - external_labels: - cluster: eu1 - replica: 0 - tenant: team-eu # Not needed, but a good practice if you want to grow this to multi-tenant system some day. - -scrape_configs: - - job_name: 'prometheus' - static_configs: - - targets: ['127.0.0.1:9091','127.0.0.1:9092'] -EOF -```{{execute}} - -* `cluster="eu1", replica="1"` Prometheus: - -``` -cat < ${CURR_DIR}/prom-eu1-replica1-config.yaml -global: - scrape_interval: 5s - external_labels: - cluster: eu1 - replica: 1 - tenant: team-eu # Not needed, but a good practice if you want to grow this to multi-tenant system some day. - -scrape_configs: - - job_name: 'prometheus' - static_configs: - - targets: ['127.0.0.1:9091','127.0.0.1:9092'] -EOF -```{{execute}} - -* `cluster="us1", replica="0"` Prometheus: - -``` -cat < ${CURR_DIR}/prom-us1-replica0-config.yaml -global: - scrape_interval: 5s - external_labels: - cluster: us1 - replica: 0 - tenant: team-us - -scrape_configs: - - job_name: 'prometheus' - static_configs: - - targets: ['127.0.0.1:9093'] -EOF -```{{execute}} - -### Ports for Prometheus-es/Prometheis/Prometheus - -``` -PROM_EU1_0_PORT=9091 && \ -PROM_EU1_1_PORT=9092 && \ -PROM_US1_0_PORT=9093 -PROM_EU1_0_EXT_ADDRESS=https://[[HOST_SUBDOMAIN]]-${PROM_EU1_0_PORT}-[[KATACODA_HOST]].environments.katacoda.com && \ -PROM_EU1_1_EXT_ADDRESS=https://[[HOST_SUBDOMAIN]]-${PROM_EU1_1_PORT}-[[KATACODA_HOST]].environments.katacoda.com && \ -PROM_US1_0_EXT_ADDRESS=https://[[HOST_SUBDOMAIN]]-${PROM_US1_0_PORT}-[[KATACODA_HOST]].environments.katacoda.com -```{{execute}} - -### Deploying Prometheus-es/Prometheus/Prometheus instances - -``` -docker run -it --rm quay.io/prometheus/prometheus:v2.20.0 --help -```{{execute}} - -* `cluster="eu1", replica="0"` Prometheus: - -``` -docker run -d --net=host --rm \ - -v ${CURR_DIR}/prom-eu1-replica0-config.yaml:/etc/prometheus/prometheus.yml \ - -v ${CURR_DIR}/prom-eu1-replica0:/prometheus \ - -u root \ - --name prom-eu1-0 \ - quay.io/prometheus/prometheus:v2.20.0 \ - --config.file=/etc/prometheus/prometheus.yml \ - --storage.tsdb.path=/prometheus \ - --storage.tsdb.retention.time=1000d \ - --storage.tsdb.max-block-duration=2h \ - --storage.tsdb.min-block-duration=2h \ - --web.listen-address=:${PROM_EU1_0_PORT} \ - --web.external-url=${PROM_EU1_0_EXT_ADDRESS} \ - --web.enable-lifecycle \ - --web.enable-admin-api -```{{execute}} - -* `cluster="eu1", replica="1"` Prometheus: - -``` -docker run -d --net=host --rm \ - -v ${CURR_DIR}/prom-eu1-replica1-config.yaml:/etc/prometheus/prometheus.yml \ - -v ${CURR_DIR}/prom-eu1-replica1:/prometheus \ - -u root \ - --name prom-eu1-1 \ - quay.io/prometheus/prometheus:v2.20.0 \ - --config.file=/etc/prometheus/prometheus.yml \ - --storage.tsdb.path=/prometheus \ - --storage.tsdb.retention.time=1000d \ - --storage.tsdb.max-block-duration=2h \ - --storage.tsdb.min-block-duration=2h \ - --web.listen-address=:${PROM_EU1_1_PORT} \ - --web.external-url=${PROM_EU1_1_EXT_ADDRESS} \ - --web.enable-lifecycle \ - --web.enable-admin-api -```{{execute}} - -* `cluster="us1", replica="0"` Prometheus: - -``` -docker run -d --net=host --rm \ - -v ${CURR_DIR}/prom-us1-replica0-config.yaml:/etc/prometheus/prometheus.yml \ - -v ${CURR_DIR}/prom-us1-replica0:/prometheus \ - -u root \ - --name prom-us1-0 \ - quay.io/prometheus/prometheus:v2.20.0 \ - --config.file=/etc/prometheus/prometheus.yml \ - --storage.tsdb.path=/prometheus \ - --storage.tsdb.retention.time=1000d \ - --storage.tsdb.max-block-duration=2h \ - --storage.tsdb.min-block-duration=2h \ - --web.listen-address=:${PROM_US1_0_PORT} \ - --web.external-url=${PROM_US1_0_EXT_ADDRESS} \ - --web.enable-lifecycle \ - --web.enable-admin-api -```{{execute}} - -### Step1: Sidecar - -``` -docker run -it --rm quay.io/thanos/thanos:v0.26.0 --help -```{{execute}} - - -* `cluster="eu1", replica="0"` sidecar: - -``` -docker run -d --net=host --rm \ - -v ${CURR_DIR}/prom-eu1-replica0-config.yaml:/etc/prometheus/prometheus.yml \ - --name prom-eu1-0-sidecar \ - -u root \ - quay.io/thanos/thanos:v0.26.0 \ - sidecar \ - --http-address 0.0.0.0:19091 \ - --grpc-address 0.0.0.0:19191 \ - --reloader.config-file /etc/prometheus/prometheus.yml \ - --prometheus.url "http://127.0.0.1:${PROM_EU1_0_PORT}" -```{{execute}} - -* `cluster="eu1", replica="1"` sidecar: - -``` -docker run -d --net=host --rm \ - -v ${CURR_DIR}/prom-eu1-replica1-config.yaml:/etc/prometheus/prometheus.yml \ - --name prom-eu1-1-sidecar \ - -u root \ - quay.io/thanos/thanos:v0.26.0 \ - sidecar \ - --http-address 0.0.0.0:19092 \ - --grpc-address 0.0.0.0:19192 \ - --reloader.config-file /etc/prometheus/prometheus.yml \ - --prometheus.url "http://127.0.0.1:${PROM_EU1_1_PORT}" -```{{execute}} - -* `cluster="us1", replica="0"` sidecar: - -``` -docker run -d --net=host --rm \ - -v ${CURR_DIR}/prom-us1-replica0-config.yaml:/etc/prometheus/prometheus.yml \ - --name prom-us1-0-sidecar \ - -u root \ - quay.io/thanos/thanos:v0.26.0 \ - sidecar \ - --http-address 0.0.0.0:19093 \ - --grpc-address 0.0.0.0:19193 \ - --reloader.config-file /etc/prometheus/prometheus.yml \ - --prometheus.url "http://127.0.0.1:${PROM_US1_0_PORT}" -```{{execute}} - -### Step2: Global View + HA: Querier - -``` -docker run -d --net=host --rm \ - --name querier \ - quay.io/thanos/thanos:v0.26.0 \ - query \ - --http-address 0.0.0.0:9090 \ - --grpc-address 0.0.0.0:19190 \ - --query.replica-label replica \ - --store 127.0.0.1:19191 \ - --store 127.0.0.1:19192 \ - --store 127.0.0.1:19193 -```{{execute}} - -Visit https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com to see Thanos UI. diff --git a/tutorials/katacoda/thanos/x-playground/step2.md b/tutorials/katacoda/thanos/x-playground/step2.md deleted file mode 100644 index 7351226e83..0000000000 --- a/tutorials/katacoda/thanos/x-playground/step2.md +++ /dev/null @@ -1,183 +0,0 @@ -## Long term retention! - -### Step 3: Let's start object storage first. - -In this step, we will configure the object store and change sidecar to upload to the -object-store. - -## Running Minio - -``` -mkdir ${CURR_DIR}/minio && \ -docker run -d --rm --name minio \ - -v ${CURR_DIR}/minio:/data \ - -p 9000:9000 -e "MINIO_ACCESS_KEY=rawkode" -e "MINIO_SECRET_KEY=rawkodeloveobs" \ - minio/minio:RELEASE.2019-01-31T00-31-19Z \ - server /data -```{{execute}} - -Create `thanos` bucket: - -``` -mkdir ${CURR_DIR}/minio/thanos -```{{execute}} - - -To check if the Minio is working as intended, let's [open Minio server UI](https://[[HOST_SUBDOMAIN]]-9000-[[KATACODA_HOST]].environments.katacoda.com/minio/) - -Enter the credentials as mentioned below: - -**Access Key** = `rawkode` -**Secret Key** = `rawkodeloveobs` - -### Step 4: Configure sidecar to upload blocks. - -``` -cat < ${CURR_DIR}/minio-bucket.yaml -type: S3 -config: - bucket: "thanos" - endpoint: "127.0.0.1:9000" - insecure: true - signature_version2: true - access_key: "rawkode" - secret_key: "rawkodeloveobs" -EOF -```{{execute}} - -Before moving forward, we need to roll new sidecar configuration with new bucket config. - -``` -docker stop prom-eu1-0-sidecar -docker stop prom-eu1-1-sidecar -docker stop prom-us1-0-sidecar -```{{execute}} - -Now, execute the following command : - - -* `cluster="eu1", replica="0"` sidecar: - -``` -docker run -d --net=host --rm \ - -v ${CURR_DIR}/prom-eu1-replica0-config.yaml:/etc/prometheus/prometheus.yml \ - -v ${CURR_DIR}/minio-bucket.yaml:/etc/thanos/minio-bucket.yaml \ - -v ${CURR_DIR}/prom-eu1-replica0:/prometheus \ - --name prom-eu1-0-sidecar \ - -u root \ - quay.io/thanos/thanos:v0.26.0 \ - sidecar \ - --tsdb.path /prometheus \ - --objstore.config-file /etc/thanos/minio-bucket.yaml \ - --shipper.upload-compacted \ - --http-address 0.0.0.0:19091 \ - --grpc-address 0.0.0.0:19191 \ - --reloader.config-file /etc/prometheus/prometheus.yml \ - --prometheus.url "http://127.0.0.1:${PROM_EU1_0_PORT}" -```{{execute}} - -* `cluster="eu1", replica="1"` sidecar: - -``` -docker run -d --net=host --rm \ - -v ${CURR_DIR}/prom-eu1-replica1-config.yaml:/etc/prometheus/prometheus.yml \ - -v ${CURR_DIR}/minio-bucket.yaml:/etc/thanos/minio-bucket.yaml \ - -v ${CURR_DIR}/prom-eu1-replica1:/prometheus \ - --name prom-eu1-1-sidecar \ - -u root \ - quay.io/thanos/thanos:v0.26.0 \ - sidecar \ - --tsdb.path /prometheus \ - --objstore.config-file /etc/thanos/minio-bucket.yaml \ - --shipper.upload-compacted \ - --http-address 0.0.0.0:19092 \ - --grpc-address 0.0.0.0:19192 \ - --reloader.config-file /etc/prometheus/prometheus.yml \ - --prometheus.url "http://127.0.0.1:${PROM_EU1_1_PORT}" -```{{execute}} - -* `cluster="us1", replica="0"` sidecar: - -``` -docker run -d --net=host --rm \ - -v ${CURR_DIR}/prom-us1-replica0-config.yaml:/etc/prometheus/prometheus.yml \ - -v ${CURR_DIR}/minio-bucket.yaml:/etc/thanos/minio-bucket.yaml \ - -v ${CURR_DIR}/prom-us1-replica0:/prometheus \ - --name prom-us1-0-sidecar \ - -u root \ - quay.io/thanos/thanos:v0.26.0 \ - sidecar \ - --tsdb.path /prometheus \ - --objstore.config-file /etc/thanos/minio-bucket.yaml \ - --shipper.upload-compacted \ - --http-address 0.0.0.0:19093 \ - --grpc-address 0.0.0.0:19193 \ - --reloader.config-file /etc/prometheus/prometheus.yml \ - --prometheus.url "http://127.0.0.1:${PROM_US1_0_PORT}" -```{{execute}} - -We can check whether the data is uploaded into `thanos` bucket by visiting [Minio](https://[[HOST_SUBDOMAIN]]-9000-[[KATACODA_HOST]].environments.katacoda.com/minio/) (or `localhost:9000`) It will take a minute to synchronize all blocks. Note that sidecar by default uploads only "non compacted by Prometheus" blocks. - -See [this](https://thanos.io/tip/components/sidecar.md/#upload-compacted-blocks) to read more about uploading old data already touched by Prometheus. - -Once we see all ~40 blocks appear in the minio, we are sure our data is backed up. Awesome! - -### Mhm, how to query those data now? - -Let's run Store Gateway server: - -``` -docker run -d --net=host --rm \ - -v ${CURR_DIR}/minio-bucket.yaml:/etc/thanos/minio-bucket.yaml \ - --name store-gateway \ - quay.io/thanos/thanos:v0.26.0 \ - store \ - --objstore.config-file /etc/thanos/minio-bucket.yaml \ - --http-address 0.0.0.0:19094 \ - --grpc-address 0.0.0.0:19194 -```{{execute}} - -### Let's point query to new StoreAPI! - -``` -docker stop querier && \ -docker run -d --net=host --rm \ - --name querier \ - quay.io/thanos/thanos:v0.26.0 \ - query \ - --http-address 0.0.0.0:9090 \ - --grpc-address 0.0.0.0:19190 \ - --query.replica-label replica \ - --store 127.0.0.1:19191 \ - --store 127.0.0.1:19192 \ - --store 127.0.0.1:19193 \ - --store 127.0.0.1:19194 -```{{execute}} - -Visit https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com to see Thanos UI. - -### Long term maintenance, retention, dedup and downsampling: - -``` -docker run -d --net=host --rm \ - -v ${CURR_DIR}/minio-bucket.yaml:/etc/thanos/minio-bucket.yaml \ - --name compactor \ - quay.io/thanos/thanos:v0.26.0 \ - compact \ - --wait --wait-interval 30s \ - --consistency-delay 0s \ - --objstore.config-file /etc/thanos/minio-bucket.yaml \ - --http-address 0.0.0.0:19095 -```{{execute}} - -Visit https://[[HOST_SUBDOMAIN]]-19095-[[KATACODA_HOST]].environments.katacoda.com/new/loaded to see Compactor Web UI. - -### Data should be immediately downsampled as well for smooth experience! - -Visit https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com to see Thanos UI and query for 1 year. - -Check 5m downsampling vs raw data. - -## Feel free to play around, more components will be added in future (: - - diff --git a/tutorials/killercoda/README.md b/tutorials/killercoda/README.md new file mode 100644 index 0000000000..caacae6d30 --- /dev/null +++ b/tutorials/killercoda/README.md @@ -0,0 +1,21 @@ +# Killercoda tutorials + +Our interactive, in-browser tutorials are available under https://killercoda.com/thanos. + +## Development + +Process of adding / editing tutorials: + +1. Sign up to [Killercoda website](https://killercoda.com). +2. Link your fork of Thanos to your account, see [this](https://killercoda.com/creators/get-started). +3. Add / edit tutorials. +4. Push to `main` branch of your fork. + * You should see the updated preview of the tutorials on https://killercoda.com/. +5. Make a PR to Thanos [tutorials repo](https://github.com/thanos-io/tutorials) with your changes. +6. Once PR is merged the official profile https://killercoda.com/thanos will be updated. + +Find more docs [here](https://killercoda.com/creators) + +Thanos repo is hooked to [this](https://killercoda.com/thanos) Katacoda account. + +Tutorials scenario definitions can be found under our Thanos [tutorials repo](https://github.com/thanos-io/tutorials).