[Compact] duplicated blocks with different id but same data

[jnyi]

Thanos Compactor contains a lot of duplicated blocks, they are different TSDB blocks with same data but different ULID, we only have 1 compactor to do the compaction work. One thing might be related to this is we break compactor to two statefulset, 1 relabel drop replica labels and relabel containsreplica` labels.

Thanos, Prometheus and Golang version used:

goversion: go1.21.7
version: 0.34.0

Object Storage Provider: S3

What happened: duplicated blocks

What you expected to happen: only 1 block per time range

How to reproduce it (as minimally and precisely as possible): not sure

Full logs to relevant components:

Anything else we need to know:

[baelish]

Actually our issue is probably this one: #7488 so probably ignore below.

We're having a similar problem, we're running several compactors but separated correctly by either label or time. We're having an issue where a single compactor (same pod running for 14days) has downsampled the same block twice (for many blocks) The local meta-syncer has meta data for both blocks and reports 0 differences apart from the ULID.

Example, first time downsampling:

ts=2024-08-08T13:43:42.573776301Z caller=downsample.go:391 level=info msg=\"downsampled block\" from=01J2ZDQWQMTDTNHXM8KE5YQ4AG to=01J4QXNTY7JCHD11YNMYT47XWG duration=11h4m24.846847992s duration_ms=39864846

Second time:

ts=2024-08-18T18:25:32.175982135Z caller=downsample.go:391 level=info msg=\"downsampled block\" from=01J2ZDQWQMTDTNHXM8KE5YQ4AG to=01J5JR3B1A5NCFRKVNC7A6EZVX duration=5h44m12.730842255s duration_ms=20652730

Pod uptime:

thanos-compactor-cu2-lf1-0                1/1     Running   0               14d

Meta files on pod:

/var/thanos/compactor/meta-syncer $ ls 01J2ZDQWQMTDTNHXM8KE5YQ4AG 01J4QXNTY7JCHD11YNMYT47XWG 01J5JR3B1A5NCFRKVNC7A6EZVX
01J2ZDQWQMTDTNHXM8KE5YQ4AG:
meta.json

01J4QXNTY7JCHD11YNMYT47XWG:
meta.json

01J5JR3B1A5NCFRKVNC7A6EZVX:
meta.json
/var/thanos/compactor/meta-syncer $ diff 01J4QXNTY7JCHD11YNMYT47XWG/meta.json 01J5JR3B1A5NCFRKVNC7A6EZVX/m
eta.json
--- 01J4QXNTY7JCHD11YNMYT47XWG/meta.json
+++ 01J5JR3B1A5NCFRKVNC7A6EZVX/meta.json
@@ -1,5 +1,5 @@
 {
-	"ulid": "01J4QXNTY7JCHD11YNMYT47XWG",
+	"ulid": "01J5JR3B1A5NCFRKVNC7A6EZVX",
 	"minTime": 1651104000000,
 	"maxTime": 1652313600000,
 	"stats": {
/var/thanos/compactor/meta-syncer $

Version running:

/var/thanos/compactor/meta-syncer $ thanos --version
thanos, version 0.34.1 (branch: HEAD, revision: 4cf1559998bf6d8db3f9ca0fde2a00d217d4e23e)
  build user:       root@61db75277a55
  build date:       20240219-17:13:48
  go version:       go1.21.7
  platform:         linux/amd64
  tags:             netgo
/var/thanos/compactor/meta-syncer $

I'd upgrade to something more current but as this bug is currently open and the issue may disappear when I restart the pods I'll wait for advice.