Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: vertical sharding has issue with absent_over_time #5203

Closed
yeya24 opened this issue Mar 9, 2023 · 1 comment · Fixed by #5221
Closed

bug: vertical sharding has issue with absent_over_time #5203

yeya24 opened this issue Mar 9, 2023 · 1 comment · Fixed by #5221

Comments

@yeya24
Copy link
Contributor

yeya24 commented Mar 9, 2023

Describe the bug

    query_fuzz_test.go:169: case 0 results mismatch.
        instant query: quantile without (__name__) (--pi(), absent_over_time({series="2"}[1h:1m]))
        res1: 
        res2: {} => +Inf @[1678391416.275]

While testing vertical sharding with PromQLsmith, I found out that we will hit this inconsistency result between sharding and non sharding Cortex.

The query itself can be simplified to sum by (xxx) (absent_over_time({series="2"}[1h:1m])) and we can still reproduce this issue.
The series matching {series="2"} exist in both Cortex instances.

If vertical sharding enabled, if the shard doesn't contains series matching series=2, absent_over_time will return 1 so causing inconsistency.

@yeya24 yeya24 changed the title bug: vertical sharding has issue with absent_over_time bug: vertical sharding has issue with absent_over_time Mar 9, 2023
@yeya24
Copy link
Contributor Author

yeya24 commented Mar 9, 2023

Fixing this on Thanos thanos-io/thanos#6201.

@justinjung04 justinjung04 mentioned this issue Mar 20, 2023
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant