Receive: Tenants' external labels proposal #5720

haanhvu · 2022-09-25T19:15:57Z

Signed-off-by: haanhvu [email protected]

Proposal for #5434

haanhvu · 2022-09-25T19:17:02Z

@fpetkovski @saswatamcode This probably needs more details. Let's discuss this tomorrow.

docs/proposals-accepted/202209-receive-tenant-external-labels.md

saswatamcode

Thanks for awesome work!

Some comments!

docs/proposals-accepted/202209-receive-tenant-external-labels.md

haanhvu · 2022-10-01T07:44:38Z

@fpetkovski I'll push something here at the end of the week. So don't review this yet if you don't have much time.

haanhvu · 2022-10-07T16:23:04Z

@fpetkovski @saswatamcode @yeya24 I made some changes. Please help review.

docs/proposals-accepted/202209-receive-tenant-external-labels.md

fpetkovski

I like this proposal, simple and straight to the point. We need to polish out some typos, but other than that looks quite good already

docs/proposals-accepted/202209-receive-tenant-external-labels.md

saswatamcode

Thanks! Great work! 🙂

Some small nits and typos from above.

docs/proposals-accepted/202209-receive-tenant-external-labels.md

yeya24

I am wondering if the following dynamic external label is part of the scope or not.
User can specify the external label names they want to extract as external labels to blocks. Upon receiving series from remote write we automatically group new blocks based on the external label value.

This sounds kind of hard but would be a super useful feature not only in Thanos but also in Cortex.

docs/proposals-accepted/202209-receive-tenant-external-labels.md

fpetkovski · 2022-11-14T08:14:02Z

We can start with statically defined labels first. Once that is in place, it would be interesting to explore extracting external labels from timeseries data.

yeya24 · 2022-11-20T20:02:24Z

Hey @haanhvu, could you please address the comments and we can merge the proposal?

haanhvu · 2022-11-20T20:07:05Z

Hey @haanhvu, could you please address the comments and we can merge the proposal?

@yeya24 Yes. I'll also add your idea as a follow-up^^

yeya24 · 2022-12-19T06:16:05Z

Ping @haanhvu. It is better to resolve the comments and merge the proposal before merging the implementation pr.

haanhvu · 2022-12-19T08:58:41Z

@yeya24 Yeah sorry for the delay I'm taking some days off. I'll update this proposal this week don't worry^^

Signed-off-by: haanhvu <[email protected]>

haanhvu · 2022-12-22T16:23:41Z

@fpetkovski @saswatamcode @yeya24 I updated the proposal. Besides the suggested wording changes, I also made some other changes based on what we've learned and discussed in the past months.

Please help review!

Signed-off-by: haanhvu <[email protected]>

yeya24

Thanks @haanhvu. This looks good to me. Anyone wants to take another review before I merge it?

fpetkovski

Looks good to me too 🚀

We can fix docs on main.

yeya24

Thanks. Let's fix doc on main.

* Receive external labels proposal Signed-off-by: haanhvu <[email protected]> * Restructure and edit proposal's content Signed-off-by: haanhvu <[email protected]> * Update proposal Signed-off-by: haanhvu <[email protected]> * Fix doc error Signed-off-by: haanhvu <[email protected]> Signed-off-by: haanhvu <[email protected]>

* Update Thanos engine to latest version (thanos-io#6069) This commit updates the Thanos PromQL engine to the latest version. Signed-off-by: Filip Petkovski <[email protected]> Signed-off-by: Filip Petkovski <[email protected]> * Receive: Tenants' external labels proposal (thanos-io#5720) * Receive external labels proposal Signed-off-by: haanhvu <[email protected]> * Restructure and edit proposal's content Signed-off-by: haanhvu <[email protected]> * Update proposal Signed-off-by: haanhvu <[email protected]> * Fix doc error Signed-off-by: haanhvu <[email protected]> Signed-off-by: haanhvu <[email protected]> * fixing doc CI (thanos-io#6072) Signed-off-by: Ben Ye <[email protected]> Signed-off-by: Ben Ye <[email protected]> * Fix stores filtering resets on reload (thanos-io#6063) * Fix stores filtering resets on reload `g0.store_matches` parameter appears in the url but doesn't applies in the frontend. Looks like it has been done on purpose and by removing a small piece of code fixes this issue. variable named `debugMode` is used for the store filtering checkbox which is an unappropriate name. Using `enableStoreFiltering` variable to represent the state of checkbox. Signed-off-by: Pradyumna Krishna <[email protected]> * Regenerate bindata.go Signed-off-by: Pradyumna Krishna <[email protected]> Signed-off-by: Pradyumna Krishna <[email protected]> * Store: Make initial sync more robust Added re-try mechanism for store inital sync, where if the initial sync fails, it tries to do the initial sync again for given timeout duration. Signed-off-by: Kartik-Garg <[email protected]> * Recover from panics in Series calls (thanos-io#6077) * Recover from panics in Series calls This commit adds panic recovery for Series calls in all Store servers. Signed-off-by: Filip Petkovski <[email protected]> * Apply error suggestion Signed-off-by: Filip Petkovski <[email protected]> --------- Signed-off-by: Filip Petkovski <[email protected]> * query: reuse our own gate (thanos-io#6079) Do not call promgate directly but rather use our own wrapper that does everything we want - duration histogram, current in-flight calls, total calls. Signed-off-by: Giedrius Statkevičius <[email protected]> * Store: Support disable cache index header file. (thanos-io#5773) * Store: Support disable cache index header file. Signed-off-by: wanjunlei <[email protected]> * Store: add a seprate flag to disable caching index header file Signed-off-by: wanjunlei <[email protected]> * Tools: add cleanup API for bucket web Signed-off-by: wanjunlei <[email protected]> * resolve conversation Signed-off-by: wanjunlei <[email protected]> * resolve confilcts Signed-off-by: wanjunlei <[email protected]> * change the flag to `--cache-index-header` Signed-off-by: wanjunlei <[email protected]> * Wrap mem writer in file writer Signed-off-by: wanjunlei <[email protected]> * update CHANGELOG Signed-off-by: wanjunlei <[email protected]> * update CHANGELOG Signed-off-by: wanjunlei <[email protected]> * fix bug Signed-off-by: wanjunlei <[email protected]> --------- Signed-off-by: wanjunlei <[email protected]> Co-authored-by: wanjunlei <[email protected]> * CVE: Fix Receiver malicious tenant (thanos-io#5969) If running as root or with enough privileges, receiver can create a directory outside of the configured TenantHeader. This commit fixes it up by sanitizing the user input and explicity not allowing such behavior. Signed-off-by: Daniel Mellado <[email protected]> * Add adopter Grupo MasMovil (thanos-io#6084) Signed-off-by: Pablo Moncada Isla <[email protected]> * fix typo (thanos-io#6087) Signed-off-by: cyip <[email protected]> Co-authored-by: cyip <[email protected]> * optimize selector to string (thanos-io#6076) Signed-off-by: Kama Huang <[email protected]> * Fix: Failure to close BlockSeriesClient cause store-gateway deadlock (thanos-io#6086) * Fix: Failure to close BlockSeriesClient cause store-gateway deadlock Signed-off-by: Alan Protasio <[email protected]> * Adding tests Signed-off-by: Alan Protasio <[email protected]> * reverting the change on get series Signed-off-by: Alan Protasio <[email protected]> * fix lint Signed-off-by: Alan Protasio <[email protected]> --------- Signed-off-by: Alan Protasio <[email protected]> * Cut 0.30.2 (thanos-io#6081) * tracing: fixed panic because of nil sampler (thanos-io#6066) * fixed panic because of nil sampler Signed-off-by: Vasiliy Rumyantsev <[email protected]> * added CHANGELOG entry Signed-off-by: Vasiliy Rumyantsev <[email protected]> Signed-off-by: Vasiliy Rumyantsev <[email protected]> * bump version to 0.30.2 Signed-off-by: Ben Ye <[email protected]> * Updates busybox SHA (thanos-io#6046) Signed-off-by: GitHub <[email protected]> Signed-off-by: GitHub <[email protected]> Co-authored-by: yeya24 <[email protected]> * Use `e2edb.NewMinio` to disable SSE-S3 in e2e tests (thanos-io#6055) * Use e2edb.NewMinio to disable SSE Signed-off-by: Saswata Mukherjee <[email protected]> * Use temp fork for TLS Signed-off-by: Saswata Mukherjee <[email protected]> * Fix broken rules api fanout test Signed-off-by: Saswata Mukherjee <[email protected]> * Fix broken query compatibility test Signed-off-by: Saswata Mukherjee <[email protected]> * Remove fork Signed-off-by: Saswata Mukherjee <[email protected]> Signed-off-by: Saswata Mukherjee <[email protected]> --------- Signed-off-by: Vasiliy Rumyantsev <[email protected]> Signed-off-by: Ben Ye <[email protected]> Signed-off-by: GitHub <[email protected]> Signed-off-by: Saswata Mukherjee <[email protected]> Co-authored-by: Vasiliy Rumyantsev <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: yeya24 <[email protected]> Co-authored-by: Saswata Mukherjee <[email protected]> * cherry pick store gateway fix to release 0.30 (thanos-io#6089) * Fix: Failure to close BlockSeriesClient cause store-gateway deadlock (thanos-io#6086) * Fix: Failure to close BlockSeriesClient cause store-gateway deadlock Signed-off-by: Alan Protasio <[email protected]> * Adding tests Signed-off-by: Alan Protasio <[email protected]> * reverting the change on get series Signed-off-by: Alan Protasio <[email protected]> * fix lint Signed-off-by: Alan Protasio <[email protected]> --------- Signed-off-by: Alan Protasio <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Alan Protasio <[email protected]> Signed-off-by: Ben Ye <[email protected]> Co-authored-by: Alan Protasio <[email protected]> * fix changelog entries Signed-off-by: Ben Ye <[email protected]> * docs: improving the description for tsdb.retention on the receiver Signed-off-by: Victor Fernandes <[email protected]> * Receiver: Use `intern` package when reallocating label strings (thanos-io#5926) * Cleanup go mod Signed-off-by: Matej Gera <[email protected]> * Use string interning for labels realloc method Signed-off-by: Matej Gera <[email protected]> * Enhance label realloc benchmarks Signed-off-by: Matej Gera <[email protected]> * Make interning optional; put behind hiddend flag Signed-off-by: Matej Gera <[email protected]> * Update CHANGELOG Signed-off-by: Matej Gera <[email protected]> * Address feedback - Fix wrong condition - Adjust benchmarks Signed-off-by: Matej Gera <[email protected]> --------- Signed-off-by: Matej Gera <[email protected]> Signed-off-by: Matej Gera <[email protected]> Signed-off-by: Matej Gera <[email protected]> * Updaing README with drawing fixes and minor wording clarification (thanos-io#6078) * New drawing and wording for Thanos other deployment models Signed-off-by: Jonah Kowall <[email protected]> * New drawing and wording for Thanos other deployment models Signed-off-by: Jonah Kowall <[email protected]> * Added comments to README.md and updated the quick-tutorial.md with the same diagram updates and text to match Signed-off-by: Jonah Kowall <[email protected]> * Ran make docs Signed-off-by: Jonah Kowall <[email protected]> --------- Signed-off-by: Jonah Kowall <[email protected]> * Compact: Remove spam of replica label removed log (thanos-io#6088) * Remove spam of replica label removed log Signed-off-by: Douglas Camata <[email protected]> * Reduce amount of removed replica label instead of removing it Signed-off-by: Douglas Camata <[email protected]> * Reformat code Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Douglas Camata <[email protected]> * Store: Don't error when no stores are matched (thanos-io#6082) It's normal and not an error if a query does not match due to no downstream stores. This is common when querying with external labels and tiered query servers. This bug was introduced in thanos-io#5296 Fixes: thanos-io#5862 Signed-off-by: SuperQ <[email protected]> * docs: Fix must have Ruler alerts definition (thanos-io#6058) * Fix must have Ruler alerts definition ThanosRuler missing rule intervals metric used the wrong comparator sign, confusing users trying to create the rule. Signed-off-by: Maxim Muzafarov <[email protected]> * Update docs/components/rule.md Co-authored-by: Saswata Mukherjee <[email protected]> Signed-off-by: Maxim Muzafarov <[email protected]> --------- Signed-off-by: Maxim Muzafarov <[email protected]> Co-authored-by: Saswata Mukherjee <[email protected]> * Fix conflicts Signed-off-by: haanhvu <[email protected]> * Specify overwriting behavior in flag and add validation Signed-off-by: haanhvu <[email protected]> * Add log and doc Signed-off-by: haanhvu <[email protected]> * Mixins(Rule): Fix query for long rule evaluations (thanos-io#6103) * mixin(Rule): Fix query for long rule evaluations Signed-off-by: Douglas Camata <[email protected]> * Update changelog Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Filip Petkovski <[email protected]> Signed-off-by: haanhvu <[email protected]> Signed-off-by: Ben Ye <[email protected]> Signed-off-by: Pradyumna Krishna <[email protected]> Signed-off-by: Kartik-Garg <[email protected]> Signed-off-by: Giedrius Statkevičius <[email protected]> Signed-off-by: wanjunlei <[email protected]> Signed-off-by: Daniel Mellado <[email protected]> Signed-off-by: Pablo Moncada Isla <[email protected]> Signed-off-by: cyip <[email protected]> Signed-off-by: Kama Huang <[email protected]> Signed-off-by: Alan Protasio <[email protected]> Signed-off-by: Vasiliy Rumyantsev <[email protected]> Signed-off-by: GitHub <[email protected]> Signed-off-by: Saswata Mukherjee <[email protected]> Signed-off-by: Victor Fernandes <[email protected]> Signed-off-by: Matej Gera <[email protected]> Signed-off-by: Matej Gera <[email protected]> Signed-off-by: Matej Gera <[email protected]> Signed-off-by: Jonah Kowall <[email protected]> Signed-off-by: Douglas Camata <[email protected]> Signed-off-by: SuperQ <[email protected]> Signed-off-by: Maxim Muzafarov <[email protected]> Signed-off-by: Sebastian Rabenhorst <[email protected]> Co-authored-by: Filip Petkovski <[email protected]> Co-authored-by: Ha Anh Vu <[email protected]> Co-authored-by: Ben Ye <[email protected]> Co-authored-by: Pradyumna Krishna <[email protected]> Co-authored-by: Kartik-Garg <[email protected]> Co-authored-by: Giedrius Statkevičius <[email protected]> Co-authored-by: wanjunlei <[email protected]> Co-authored-by: wanjunlei <[email protected]> Co-authored-by: Daniel Mellado <[email protected]> Co-authored-by: Pablo Moncada <[email protected]> Co-authored-by: Chantel Yip <[email protected]> Co-authored-by: cyip <[email protected]> Co-authored-by: Kama Huang <[email protected]> Co-authored-by: Alan Protasio <[email protected]> Co-authored-by: Vasiliy Rumyantsev <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: yeya24 <[email protected]> Co-authored-by: Saswata Mukherjee <[email protected]> Co-authored-by: Victor Fernandes <[email protected]> Co-authored-by: Matej Gera <[email protected]> Co-authored-by: Jonah Kowall <[email protected]> Co-authored-by: Douglas Camata <[email protected]> Co-authored-by: Ben Kochie <[email protected]> Co-authored-by: Maxim Muzafarov <[email protected]> Co-authored-by: haanhvu <[email protected]>

* Receive external labels proposal Signed-off-by: haanhvu <[email protected]> * Restructure and edit proposal's content Signed-off-by: haanhvu <[email protected]> * Update proposal Signed-off-by: haanhvu <[email protected]> * Fix doc error Signed-off-by: haanhvu <[email protected]> Signed-off-by: haanhvu <[email protected]>

pull-request-size bot added the size/M label Sep 25, 2022

yeya24 reviewed Sep 25, 2022

View reviewed changes

docs/proposals-accepted/202209-receive-tenant-external-labels.md Outdated Show resolved Hide resolved

saswatamcode reviewed Sep 26, 2022

View reviewed changes

haanhvu force-pushed the proposal branch from fded92b to 9badd9b Compare October 7, 2022 16:20

haanhvu mentioned this pull request Oct 10, 2022

Receive: Allow specifying tenant-specific external labels in RouterIngestor #5777

Merged

2 tasks

yeya24 reviewed Oct 10, 2022

View reviewed changes

docs/proposals-accepted/202209-receive-tenant-external-labels.md Outdated Show resolved Hide resolved

fpetkovski reviewed Oct 11, 2022

View reviewed changes

saswatamcode reviewed Oct 11, 2022

View reviewed changes

docs/proposals-accepted/202209-receive-tenant-external-labels.md Outdated Show resolved Hide resolved

docs/proposals-accepted/202209-receive-tenant-external-labels.md Outdated Show resolved Hide resolved

haanhvu commented Oct 19, 2022

View reviewed changes

docs/proposals-accepted/202209-receive-tenant-external-labels.md Outdated Show resolved Hide resolved

yeya24 reviewed Nov 4, 2022

View reviewed changes

docs/proposals-accepted/202209-receive-tenant-external-labels.md Outdated Show resolved Hide resolved

haanhvu added 3 commits December 22, 2022 23:18

Receive external labels proposal

40ed4b5

Signed-off-by: haanhvu <[email protected]>

Restructure and edit proposal's content

08cc3a2

Signed-off-by: haanhvu <[email protected]>

Update proposal

54e6af0

Signed-off-by: haanhvu <[email protected]>

haanhvu force-pushed the proposal branch from f5ec96c to 54e6af0 Compare December 22, 2022 16:18

haanhvu changed the title ~~[WIP] Receive external labels proposal~~ Receive: Tenants' external labels proposal Dec 22, 2022

Fix doc error

f1b1909

Signed-off-by: haanhvu <[email protected]>

yeya24 approved these changes Jan 15, 2023

View reviewed changes

matej-g approved these changes Jan 16, 2023

View reviewed changes

fpetkovski approved these changes Jan 16, 2023

View reviewed changes

yeya24 approved these changes Jan 24, 2023

View reviewed changes

yeya24 merged commit 5553244 into thanos-io:main Jan 24, 2023

yeya24 mentioned this pull request Jan 24, 2023

Fixing doc CI #6072

Merged

2 tasks

haanhvu mentioned this pull request May 25, 2023

Receive: Allow specifying tenant-specific external labels #5434

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Receive: Tenants' external labels proposal #5720

Receive: Tenants' external labels proposal #5720

haanhvu commented Sep 25, 2022 •

edited

Loading

haanhvu commented Sep 25, 2022

saswatamcode left a comment

haanhvu commented Oct 1, 2022

haanhvu commented Oct 7, 2022

fpetkovski left a comment

saswatamcode left a comment

yeya24 left a comment

fpetkovski commented Nov 14, 2022

yeya24 commented Nov 20, 2022

haanhvu commented Nov 20, 2022

yeya24 commented Dec 19, 2022

haanhvu commented Dec 19, 2022 •

edited

Loading

haanhvu commented Dec 22, 2022

yeya24 left a comment

fpetkovski left a comment

yeya24 left a comment

Receive: Tenants' external labels proposal #5720

Receive: Tenants' external labels proposal #5720

Conversation

haanhvu commented Sep 25, 2022 • edited Loading

haanhvu commented Sep 25, 2022

saswatamcode left a comment

Choose a reason for hiding this comment

haanhvu commented Oct 1, 2022

haanhvu commented Oct 7, 2022

fpetkovski left a comment

Choose a reason for hiding this comment

saswatamcode left a comment

Choose a reason for hiding this comment

yeya24 left a comment

Choose a reason for hiding this comment

fpetkovski commented Nov 14, 2022

yeya24 commented Nov 20, 2022

haanhvu commented Nov 20, 2022

yeya24 commented Dec 19, 2022

haanhvu commented Dec 19, 2022 • edited Loading

haanhvu commented Dec 22, 2022

yeya24 left a comment

Choose a reason for hiding this comment

fpetkovski left a comment

Choose a reason for hiding this comment

yeya24 left a comment

Choose a reason for hiding this comment

haanhvu commented Sep 25, 2022 •

edited

Loading

haanhvu commented Dec 19, 2022 •

edited

Loading