Compactor is unable to properly compact blocks

[shybbko]

Describe the bug
I'm running two config-identical Cortex clusters, let's say: prod & nonprod.
Nonprod looks fine.
In prod it seems that compactor is unable to properly compact blocks and this leads to prod having ~11 000 blocks, while nonprod ~600 (the volume of data alone is not 20x bigger on prod, so this is unexpected).
Having 11k blocks causes problems with store gateway pods which tend to take a lot of time to load blocks and, until all blocks loaded, the cluster does not work great.
Why am I assuming that compacting does not work for prod? It seems that upon successful compaction there should be some entries such as compacted blocks and marking compacted block for deletion etc. There are none, only endless entries like the ones above. Also I'm running various dashboards, ie. https://github.com/monitoring-mixins/website/blob/master/assets/cortex/dashboards/cortex-compactor-resources.json which shows literally no compacted blocks for prod (and some for nonprod). Sharing my logs below.

I am aware that there are at least several issues that could be causing compactor not to work #4453 or #3569, but I'd very much welcome any hints that could allow me to unblock compacting as the current volume of blocks makes the cluster prone to not working properly (which is not great for production usage, obviously).

Expected behavior
Compacting works, the amount of blocks in prod in not ~20x the amount of blocks in nonprod (more like ~3-4 times at best).

Environment:
K8s in GKE v1.21, deployed by the official cortex Helm chart v1.4.0

Storage Engine
Blocks

Additional Context
Two sets of logs are here:

• Initial debug level logs (nontruncated)
• Extra logs - disabled debug and didn't include the usual noise for more clarity

[shybbko]

Also tried using thanos tools (https://github.com/thanos-io/thanos/blob/main/docs/components/tools.md) in hopes that maybe I'll be able to use the repair option, but seems that it's unable to see the blocks (even though I believe the config to be a proper one, as no errors are presented)

level=debug ts=2022-03-18T17:33:42.835491406Z caller=main.go:66 msg="maxprocs: Leaving GOMAXPROCS=[16]: CPU quota undefined"
level=info ts=2022-03-18T17:33:42.835776032Z caller=factory.go:49 msg="loading bucket configuration"
level=debug ts=2022-03-18T17:33:42.838233901Z caller=fetcher.go:319 component=block.BaseFetcher msg="fetching meta data" concurrency=32
level=info ts=2022-03-18T17:33:42.996412892Z caller=fetcher.go:470 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=158.208834ms duration_ms=158 cached=0 returned=0 partial=0
level=info ts=2022-03-18T17:33:42.996497217Z caller=tools_bucket.go:458 msg="ls done" objects=0
level=info ts=2022-03-18T17:33:43.014347816Z caller=main.go:161 msg=exiting
level=debug ts=2022-03-18T17:33:43.01442981Z caller=main.go:66 msg="maxprocs: No GOMAXPROCS change to reset%!(EXTRA []interface {}=[])"

Here's the config I used:

apiVersion: v1
kind: Pod
metadata:
  name: thanos-tools
spec:
  serviceAccountName: xxx-service-account
  containers:
    - name: thanos-tools
      image: quay.io/thanos/thanos:v0.25.1
      command: ["/bin/sh", "-c"]
      args:
        - >-
          thanos tools bucket ls
          --log.level=debug
          --objstore.config-file=/config/bucket.yml
      volumeMounts:
      - name: config-file-volume
        mountPath: /config
  volumes:
    - name: config-file-volume
      configMap:
        name: bucket-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: bucket-config
data:
  bucket.yml: |
    type: GCS
    config: 
      bucket: "xxx-blocks-storage"

[shybbko]

So, while I haven't been able to fix my problem (hopefully - yet), I've been investigating and came to some conclusions:

thanos tools are unlikely to work, as Cortex has made some tiny changes to Thanos format which is the likely reason thanos tools are unable to see Cortex GCS blocks: https://github.com/cortexproject/cortex/blob/94b0137e7133652f32a6ca641510b8c4c861791e/docs/blocks-storage/migrate-storage-from-thanos-and-prometheus.md
Also repairing out of order blocks seems not to work great for Thanos anyway: thanos-io/thanos#3442

While I do not know the reason out of order blocks appeared in Cortex blocks (whatever happens, Cortex is designed not to ingest anything that is out of order), I am leaning to assuming that this might be related to a Prometheus race condition issue: #4573 that Cortex inherited.

Some time ago Thanos introduced a feature to allow skipping out of order blocks upon compaction thanos-io/thanos#4469, however this change hasn't made to Cortex yet #4453.
And part of the reason Cortex being notoriously out of date with Thanos code seems to be here: https://github.com/cortexproject/cortex/pull/4509/files

The PR to introduce Thanos out of order blocks skipping feature #4453 was raised late August 2021 (and has stayed open ever since). There was an another PR in October 2021 #4505 that got merged and introduced that feature, however hardcoded it to be disabled https://github.com/cortexproject/cortex/blob/release-1.11/pkg/compactor/compactor.go#L660. I am assuming this might be because the code is only partially ready for this feature to be enabled, but since was unlucky in finding this on my own or getting responses in the CNCF Slack https://cloud-native.slack.com/archives/CCYDASBLP/p1648214149043299, I've created a separate issue to hopefully get answers: #4692

I've been also comparing log sequence for a proper compaction and the one that fails, so here we go. The proper one:

"start of compactions"
"compaction available and planned; downloading blocks..."
"not downloading again because a provided path matches this one..."
"downloaded and verified blocks; compacting blocks"
"Found overlapping blocks during compaction"
"compact blocks"
"compacted blocks"
(then uploading & marking for deletion)

The "out of order error" one:

"start of compactions"
"compaction available and planned; downloading blocks..."
"not downloading again because a provided path matches this one..."
"start of compactions"
"compaction available and planned; downloading blocks..."
"not downloading again because a provided path matches this one..."
"start of compactions"
"compaction available and planned; downloading blocks..."
"not downloading again because a provided path matches this one..."
"msg="failed to compact user blocks" user=fake err="compaction: group 0@5679675083797525161: blocks with out-of-order chunks are dropped from compaction:  data/compact/0@5679675083797525161/01FT72DJY9WFAAMR2HSDNC0GBA: 2/132124 series have an average of 1.000 out-of-order chunks: 2.000 of these are exact duplicates (in terms of data and time range)"

I don't know the reason it seems to loop three times before failing.

Anyway, in my case the bad block seems to store about 50 minutes of metrics. If it comes to that, I think removing it would be OK, however as of now I'm trying to meddle with Cortex code to force it to go through all blocks in GCS and verify them for any errors that would prevent compaction. No point in deleting one block if there are 50 more that are bad / out of order. Fingers crossed.

[shybbko]

I don't know the reason it seems to loop three times before failing.

It's because of that:

  # How many times to retry a failed compaction within a single compaction run.
  # CLI flag: -compactor.compaction-retries
  [compaction_retries: <int> | default = 3]

[shybbko]

Closed by #4707

Kudos to @alanprot and @alvinlin123 <3