Downsampling is not working but compaction seem to be fine

[rnachire]

Hi,

we are running Thanos with google bucket as remote storage. Below are the configuration settings on the current setup:(min and max compaction : 2h)

• --log.level=debug
  • --objstore.config-file=/var/bucket.yml
  • --retention.resolution-raw=30d
  • --retention.resolution-5m=365d
  • --retention.resolution-1h=1h
  • --sync-delay=30m
  • --data-dir=/var/thanos/store
  • --wait

In the compactor logs, we have seen compaction as well as downsampling is happening. But when we look at store and remote storage ULID(meta.json) file we don't anything got downsampled. Attached the logs for the same.

evel=info ts=2019-01-24T18:41:35.293282997Z caller=compact.go:816 msg="start sync of metas"
level=info ts=2019-01-24T18:41:35.832007529Z caller=compact.go:822 msg="start of GC"
level=info ts=2019-01-24T18:41:35.84597813Z caller=compact.go:207 msg="compaction iterations done"
level=info ts=2019-01-24T18:41:35.846158613Z caller=compact.go:214 msg="start first pass of downsampling"
level=info ts=2019-01-24T18:41:36.453676668Z caller=compact.go:220 msg="start second pass of downsampling"
level=info ts=2019-01-24T18:41:36.788466012Z caller=compact.go:225 msg="downsampling iterations done"
level=info ts=2019-01-24T18:41:36.788610995Z caller=retention.go:17 msg="start optional retention"
level=info ts=2019-01-24T18:41:37.175464351Z caller=retention.go:46 msg="optional retention apply done"
level=info ts=2019-01-24T18:46:35.290352093Z caller=compact.go:816 msg="start sync of metas"
level=debug ts=2019-01-24T18:46:35.649414618Z caller=compact.go:174 msg="download meta" block=01D20JNFWMN5A98DCK3C48HH1E
level=debug ts=2019-01-24T18:46:35.916952831Z caller=compact.go:192 msg="block is too fresh for now" block=01D20JNFWMN5A98DCK3C48HH1E
level=info ts=2019-01-24T18:46:35.91726417Z caller=compact.go:822 msg="start of GC"
level=info ts=2019-01-24T18:46:35.925582759Z caller=compact.go:207 msg="compaction iterations done"
level=info ts=2019-01-24T18:46:35.925696324Z caller=compact.go:214 msg="start first pass of downsampling"
level=info ts=2019-01-24T18:46:36.450720059Z caller=compact.go:220 msg="start second pass of downsampling"
level=info ts=2019-01-24T18:46:36.895809384Z caller=compact.go:225 msg="downsampling iterations done"
level=info ts=2019-01-24T18:46:36.895909826Z caller=retention.go:17 msg="start optional retention"
level=info ts=2019-01-24T18:46:37.354046696Z caller=retention.go:46 msg="optional retention apply done"
level=info ts=2019-01-24T18:51:35.299952556Z caller=compact.go:816 msg="start sync of metas"
level=debug ts=2019-01-24T18:51:36.17085564Z caller=compact.go:174 msg="download meta" block=01D20JNFWMN5A98DCK3C48HH1E
level=debug ts=2019-01-24T18:51:36.437762023Z caller=compact.go:192 msg="block is too fresh for now" block=01D20JNFWMN5A98DCK3C48HH1E
level=info ts=2019-01-24T18:51:36.437867922Z caller=compact.go:822 msg="start of GC"

Can you help us find the root cause for the same and let me know if you need any other config options or logs to confirm the issue?

[bwplotka]

Hey, Thanks for report! 👋

We are aware that this part related to downsampling might be confusing. Created ticket with some answers you might want and to track this further: #813 Let's move the discussion there.

[songenjie]

#813 is about delete data,but this issues is downsampling not work

[FUSAKLA]

Hi, does the issue still persist?
As noted in the issue Bartek linked above. The dowsampling happens only after 40h in the object storage for the 5m data so till then you won't see any downsampled blocks. The conpactor logs the downsampling phase even when it does not find any data to downsample.

If it still does not work please try running command thanos bucket inspect and paste it here so we can see all the blocks in the storage
https://thanos.io/components/bucket.md/#inspect

[bwplotka]

I would recommend new freshly merged web page even: #1248