Repair data in (S3) store (help wanted)

[AtzeDeVries]

thanos version: v0.6.0
prom version: v2.7.2

When we try to query data which should come from store. Store logs this

level=warn ts=2019-07-31T03:36:35.898134375Z caller=bucket.go:296 msg="loading block failed" id=01DH31H6SFKNRKGCSDX7R0B2ZA err="new bucket block: load meta: download meta.json: get file: The specified key does not exist."
level=warn ts=2019-08-02T03:36:35.922795939Z caller=bucket.go:296 msg="loading block failed" id=01DH86AV77R05TD7JZH7RKH00V err="new bucket block: load meta: download meta.json: get file: The specified key does not exist."
level=warn ts=2019-08-05T15:00:35.897179924Z caller=bucket.go:296 msg="loading block failed" id=01DHH4PNFFW680ZGCFJBKYJHXF err="new bucket block: load meta: download meta.json: get file: The specified key does not exist."
level=warn ts=2019-08-06T03:36:35.936521626Z caller=bucket.go:296 msg="loading block failed" id=01DHJFXFH3CWAM8ZR40VR1EB4M err="new bucket block: load meta: download meta.json: get file: The specified key does not exist."
level=warn ts=2019-08-06T09:00:35.942445126Z caller=bucket.go:296 msg="loading block failed" id=01DHK2G6NCTW9DWCJ7ZT4QM3MA err="new bucket block: load meta: download meta.json: get file: The specified key does not exist."

thanos compact is run with

compact --data-dir=/data/thanos-compact --objstore.config-file=<redacted> --retention.resolution-raw=7d --retention.resolution-5m=14d --retention.resolution-1h=400d --block-sync-concurrency=1 --log.level=info --wait

I can verify that these keys do not exist, but are listed under /debug/metas/<id>.json

When i try to run thanos bucket verify --repair
docker run -v $(pwd):/bucket-configs --rm improbable/thanos:v0.6.0 bucket verify --objstore.config-file=/bucket-configs/bucket-config.yml --objstore-backup.config-file=/bucket-configs/bucket-backup-config.yml --log.level=debug --repair

the output is

level=info ts=2019-08-07T10:44:00.311368667Z caller=main.go:154 msg="Tracing will be disabled"
level=info ts=2019-08-07T10:44:00.311456286Z caller=factory.go:39 msg="loading bucket configuration"
level=info ts=2019-08-07T10:44:00.332025484Z caller=factory.go:39 msg="loading bucket configuration"
level=warn ts=2019-08-07T10:44:00.332923663Z caller=verify.go:49 msg="GLOBAL COMPACTOR SHOULD __NOT__ BE RUNNING ON THE SAME BUCKET" issues=2 repair=true
level=info ts=2019-08-07T10:44:00.333058479Z caller=index_issue.go:29 msg="started verifying issue" with-repair=true issue=index_issue
level=info ts=2019-08-07T10:45:01.223633978Z caller=index_issue.go:130 msg="verified issue" with-repair=true issue=index_issue
level=info ts=2019-08-07T10:45:01.223682376Z caller=overlapped_blocks.go:26 msg="started verifying issue" with-repair=true issue=overlapped_blocks
level=info ts=2019-08-07T10:45:01.376925611Z caller=verify.go:68 msg="verify completed" issues=2 repair=true
level=info ts=2019-08-07T10:45:01.377041038Z caller=main.go:202 msg=exiting

but nothing seems to happen. There is also no data written to the backup bucket

How can we repair our data in S3?

[bwplotka]

Duplicate of #1335

We are working on it (: