Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TSDB] Ingester Index Sampling #6852

Merged
merged 15 commits into from
Aug 9, 2022
Merged

Conversation

owen-d
Copy link
Member

@owen-d owen-d commented Aug 8, 2022

This PR does two things:

  1. Fixes a proto package path incompatibility. Older versions (prior to [TSDB] Indexgateway Protobuf refactor  #6841) used the indexgatewaypb package. While [TSDB] Indexgateway Protobuf refactor  #6841 ensured the payloads were compatible, the grpc service description generated didn't match the historical indexgatewaypb.IndexGateway, which caused errors when trying to query the old index gateways with new queriers. This PR moves the IndexGateway service into it's own proto package with the name indexgatewaypb and generates code in the logproto golang package, which fixes errors during rollouts as the service descriptions now match.
  2. Wires up the ingester & ingester-querier to support querying TSDB for index statistics so we can more effectively shard data still inside the ingesters which haven't made it to storage yet.

@owen-d owen-d requested a review from a team as a code owner August 8, 2022 19:45
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
@grafanabot
Copy link
Collaborator

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

-           ingester	-1%
+        distributor	0%
-            querier	-1.1%
+ querier/queryrange	0%
+               iter	0%
-            storage	-1.3%
+           chunkenc	0%
+              logql	0%
+               loki	0.6%

Copy link
Collaborator

@slim-bean slim-bean left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@owen-d owen-d merged commit 57cec8e into grafana:main Aug 9, 2022
lxwzy pushed a commit to lxwzy/loki that referenced this pull request Nov 7, 2022
* queryrangebase definitions protos & aliases
Signed-off-by: Owen Diehl <[email protected]>

* indexgatewaypb to logproto
Signed-off-by: Owen Diehl <[email protected]>

* updates queryrange.proto for new definitions file
Signed-off-by: Owen Diehl <[email protected]>

* s/indexgatewaypb/logproto/g
Signed-off-by: Owen Diehl <[email protected]>

* renames to indexseries for clobbering reasons
Signed-off-by: Owen Diehl <[email protected]>

* more indexgatewaypb removal compatibility work
Signed-off-by: Owen Diehl <[email protected]>

* proto fmt
Signed-off-by: Owen Diehl <[email protected]>

* rebuild imports
Signed-off-by: Owen Diehl <[email protected]>

* goimports -s
Signed-off-by: Owen Diehl <[email protected]>

* query ingesters for index stats
Signed-off-by: Owen Diehl <[email protected]>

* compatibility during rollouts for idx-gw+ingester-querier post proto refactor
Signed-off-by: Owen Diehl <[email protected]>

* proto compatibility with old indexgateway pkg path
Signed-off-by: Owen Diehl <[email protected]>

* test compat
Signed-off-by: Owen Diehl <[email protected]>

* gofmt
Signed-off-by: Owen Diehl <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants