Add metadata indexes to help with segment allocation. #6348
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Segment allocation queries can take a long time (10s of seconds) when you have a lot of segments. Adding these indexes helps greatly.
|
LGTM！ |
QiuMM
reviewed
Sep 20, 2018
| getQuoteString() | ||
| ), | ||
| StringUtils.format( | ||
| "CREATE INDEX idx_%1$s_datasource_sequence ON %1$s(dataSource, sequence_name)", |
There was a problem hiding this comment.
@gianm sorry, I did not review these code carefully. There are no sequence_name of segmentTable, pendingSegmentsTable does...
There was a problem hiding this comment.
Ouch, you're right. Thanks for the catch- I will fix it. I'm surprised the tests didn't catch it - I thought they ran these create table commands. I will look into that too.
There was a problem hiding this comment.
Raised #6356 to fix this. The PR description also explains why the tests didn't catch this.
QiuMM
reviewed
Sep 20, 2018
| @@ -215,6 +215,11 @@ public void createPendingSegmentsTable(final String tableName) | |||
| + " UNIQUE (sequence_name_prev_id_sha1)\n" | |||
| + ")", | |||
| tableName, getPayloadType(), getQuoteString() | |||
| ), | |||
| StringUtils.format( | |||
| "CREATE INDEX idx_%1$s_datasource_used_end ON %1$s(dataSource, used, %2$send%2$s)", | |||
There was a problem hiding this comment.
No used field of pendingSegmentsTable . You placed these code in wrong place. Just exchange their location will be good.
gianm
added a commit
to gianm/druid
that referenced
this pull request
Sep 20, 2018
The indexes introduced in apache#6348 were on the wrong table. The tests did not catch them due to retries on the create table steps (the first try created the table but not the bogus indexes; the second try noticed that the table already existed and did nothing). This patch doesn't fix the issue with the tests, since the best way to do that would be to do the table and index creation in a transaction; but, this is not supported by all of our supported database engines.
fjy
pushed a commit
that referenced
this pull request
Sep 26, 2018
The indexes introduced in #6348 were on the wrong table. The tests did not catch them due to retries on the create table steps (the first try created the table but not the bogus indexes; the second try noticed that the table already existed and did nothing). This patch doesn't fix the issue with the tests, since the best way to do that would be to do the table and index creation in a transaction; but, this is not supported by all of our supported database engines.
gianm
added a commit
to implydata/druid-public
that referenced
this pull request
Nov 16, 2018
Segment allocation queries can take a long time (10s of seconds) when you have a lot of segments. Adding these indexes helps greatly.
gianm
added a commit
to implydata/druid-public
that referenced
this pull request
Nov 16, 2018
The indexes introduced in apache#6348 were on the wrong table. The tests did not catch them due to retries on the create table steps (the first try created the table but not the bogus indexes; the second try noticed that the table already existed and did nothing). This patch doesn't fix the issue with the tests, since the best way to do that would be to do the table and index creation in a transaction; but, this is not supported by all of our supported database engines.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Segment allocation queries can take a long time (10s of seconds) when
you have a lot of segments. Adding these indexes helps greatly.