Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: vertical sharding aggregation with absent displays shard by label #5204

Closed
yeya24 opened this issue Mar 9, 2023 · 3 comments · Fixed by #5221
Closed

bug: vertical sharding aggregation with absent displays shard by label #5204

yeya24 opened this issue Mar 9, 2023 · 3 comments · Fixed by #5221

Comments

@yeya24
Copy link
Contributor

yeya24 commented Mar 9, 2023

Describe the bug

    query_fuzz_test.go:169: case 0 results mismatch.
        instant query: count_values without (job) ("value", absent({series="7"}))
        res1: {series="7", value="1"} => 1 @[1678393035.187]
        res2: {__CORTEX_SHARD_BY__="CAEQAiIDam9i", series="7", value="1"} => 1 @[1678393035.187]
        {__CORTEX_SHARD_BY__="EAIiA2pvYg==", series="7", value="1"} => 1 @[1678393035.187]

Got this bug when running the PromQLsmith test.

Ran a query count_values without (job) ("value", absent({series="7"})). Here {series="7"} couldn't match anything so absent() is expected to return a series.

res1 is from non vertical sharding Cortex and it gives expected result. res2 runs vertical sharding Cortex and it gave me bad results and additional __CORTEX_SHARD_BY__ label.

@yeya24 yeya24 changed the title bug: vertical sharding count_values gets hidden labels bug: vertical sharding with count_values Mar 9, 2023
@yeya24 yeya24 changed the title bug: vertical sharding with count_values bug: vertical sharding aggregation with absent displays shard by label Mar 9, 2023
@yeya24
Copy link
Contributor Author

yeya24 commented Mar 9, 2023

Seems the count_values aggregation doesn't matter here.

    query_fuzz_test.go:168: case 17 results mismatch.
        instant query: max without (series) (absent({__name__="test_series"}))
        res1: 
        res2: {__CORTEX_SHARD_BY__="CAEQAiIGc2VyaWVz"} => 1 @[1678393609.301]

Got another case with max and absent that displays the additional shard label.

@yeya24
Copy link
Contributor Author

yeya24 commented Mar 9, 2023

Probably we should just disable absent and absent_over_time for vertical sharding and then this can be resolved.

@yeya24
Copy link
Contributor Author

yeya24 commented Mar 9, 2023

Should be solved by thanos-io/thanos#6201

@justinjung04 justinjung04 mentioned this issue Mar 20, 2023
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant