HBASE-28215: region reopen procedure batching/throttling #5534
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
...rver/src/main/java/org/apache/hadoop/hbase/master/procedure/ReopenTableRegionsProcedure.java
Outdated
Show resolved
Hide resolved
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
rmdmattingly
force-pushed
the
HBASE-28215
branch
from
84ef2ec to
2f30fec
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
I believe that test failure |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
...rver/src/main/java/org/apache/hadoop/hbase/master/procedure/ReopenTableRegionsProcedure.java
Show resolved
Hide resolved
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Apache9
reviewed
Nov 23, 2023
...rver/src/main/java/org/apache/hadoop/hbase/master/procedure/ReopenTableRegionsProcedure.java
Outdated
Show resolved
Hide resolved
...rver/src/main/java/org/apache/hadoop/hbase/master/procedure/ReopenTableRegionsProcedure.java
Show resolved
Hide resolved
Comment on lines
173
to
192
| // we need to create a set of region names because the HRegionLocation hashcode is only | ||
| // based | ||
| // on the server name | ||
| Set<byte[]> currentRegionBatchNames = currentRegionBatch.stream() | ||
| .map(r -> r.getRegion().getRegionName()).collect(Collectors.toSet()); | ||
| currentRegionBatch = regions.stream() | ||
| .filter(r -> currentRegionBatchNames.contains(r.getRegion().getRegionName())) | ||
| .collect(Collectors.toList()); | ||
| if (currentRegionBatch.isEmpty()) { | ||
| if (regions.isEmpty()) { | ||
| return Flow.NO_MORE_STATE; | ||
| } else { | ||
| setNextState(ReopenTableRegionsState.REOPEN_TABLE_REGIONS_REOPEN_REGIONS); | ||
| if (reopenBatchBackoffMillis > 0) { | ||
| backoff(reopenBatchBackoffMillis); | ||
| } | ||
| return Flow.HAS_MORE_STATE; | ||
| } | ||
| } | ||
| if (regions.stream().anyMatch(loc -> canSchedule(env, loc))) { | ||
| if (currentRegionBatch.stream().anyMatch(loc -> canSchedule(env, loc))) { |
There was a problem hiding this comment.
I think the changes in this method are sort of confusing. I think the division of labor should be:
REOPEN_TABLE_REGIONS_REOPEN_REGIONS -- breaks the set of regions into a smaller batch, and reopens that batch. The code you have for that looks ok
REOPEN_TABLE_REGIONS_CONFIRM_REOPENED -- checks if any regions are still needing reopen. If so, go back to REOPEN_REGIONS. This is how it used to work.
If we totally reverted the changes in REOPEN_TABLE_REGIONS_CONFIRM_REOPENED, I think we'd have successfully broken the full region list into batches and the flow would work, just without backoff. So in this method we really just need to add the backoff.
The backoff as you have it below looks ok at first glance. This is just a long winded way of saying we should drop the highlighted code :)
There was a problem hiding this comment.
I think this complexity is necessary in order to differentiate between are there more regions to reopen vs are regions still reopening in the current batch, the former being a state change and fixed backoff, and the latter being a retry and exponentially increasing backoff
There was a problem hiding this comment.
Let's look back at the original impl of CONFIRM_REOPEN. It only does the backoff if there are still regions to reopen and none of them are schedulable. Meaning we back off in order to wait for regions to be schedulable.
Now let's look at your new code. I see what you are saying. You are basically doing:
- If currentRegionBatch are all reopened, exit or process the next batch
- Otherwise, can any of the remaining be scheduled? If not, exponential backoff. If so, process next batch.
The key thing there is that "process next batch". You'd think that would be "re-schedule existing batch". But as is, when you go to process next batch, the first thing REOPEN_REGIONS does is create a new currentRegionBatch. So lets say for the first batch of 50, only 10 reopen. With the current design, you'd expect it to try again to reopen those remaining 40. But it will actually reopen another 50 (40 of which are likely to have been in the last batch).
TBH I don't think there's a huge problem with that approach. But given that's how it works, we can simplify this code as I described before -- we don't really care about currentRegionBatch here, just that if any regions are left to reopen and schedulable.
If we wanted to do what I think you are intending to do, we probably need to complicate things a bit further. Maybe we update REOPEN_REGIONS to only create currentRegionBatch if it's currently null. Then in CONFIRM_REOPEN, null it out after confirming that they've been reopened. That way, in the above example when we kickoff REOPEN_REGIONS again we'll only open the remaining 40.
Does that make sense? Between these 2 options I might opt for the simpler one where we keep REOPEN_REGIONS as is and remove most of the diff from CONFIRM_REOPEN.
There was a problem hiding this comment.
On second thought, after a discussion on https://issues.apache.org/jira/browse/HBASE-25549, i wonder if we should take the safer approach where require each batch to finish before scheduling more. This feature could serve a dual purpose of progressive rollout and rate limiting.
We could also trivially update this code to do an actual progressive deploy -- first batch size 1, then 2, then 4, etc up to the current batch size config you added. At that point it stays at that max concurrency until completion.
Thoughts on adding that? I think we just need one more field for currentBatchSize which we increment after each batch up to reopenBatchSize. It might make sense to persist this new field in the proto.
There was a problem hiding this comment.
Now let's look at your new code. I see what you are saying. You are basically doing:
If currentRegionBatch are all reopened, exit or process the next batch
Otherwise, can any of the remaining be scheduled? If not, exponential backoff. If so, process next batch.
I think that's missing a few distinct outcomes of CONFIRM_REOPENED, and that's where this complexity comes into play. Here are the possibilities:
- The current batch has reopened successfully and there are no more regions to produce a next batch, so finish
- The current batch has reopened successfully and there are more regions to produce a next batch, so set the next state to
REOPEN_REGIONSand suspend. This will process the next batch - The current batch has not reopened successfully, and some of the current batch can still be scheduled. In this case return to
REOPEN_REGIONSto avoid an infinite loop due to a bypassed proc or something. This could technically reprocess the next batch, or redefine what the "current batch" looks like, but I think that's ok given the exceptional nature of this case - The current batch has not reopened successfully, and none of the current batch can still be scheduled (because they have already been scheduled). In this case we're just waiting on TRSPs to complete, so retain current state of
CONFIRM_REOPENEDand retry w/ increasing backoff
If I just revert this diff as described then this procedure will just get stuck in infinite loops (that I think could theoretically be exited if TRSP finishes quickly enough?) and all unit tests fail
We could also trivially update this code to do an actual progressive deploy -- first batch size 1, then 2, then 4, etc up to the current batch size config you added. At that point it stays at that max concurrency until completion.
I like this idea, and I think it would be trivial to add in the current state of this PR. We'd just need to store a maxReopenBatchSize and increase the reopenBatchSize to Math.min(2*reopenBatchSize, maxReopenBatchSize) each time we enter state 2 above
There was a problem hiding this comment.
Ok lets table the missing outcomes/simplification discussion in that case, since we'll want to keep the current logic for the progressive reopen. We'll also want to do as i said, where in CONFIRM we update currentRegionBatch to filter out the reopened regions, and then in REOPEN we should only create a new currentRegionBatch if the existing isEmpty. That way if there's some problematic region in the batch, we won't move on to the next batch until its been reopened. Right now I think we'd just move on and create a new batch including the problematic region + more?
Once we have that in there, I need to take another look at the logic here. I think it could be a bit clearer, but don't want to make any suggestions until we have the complete impl in place.
In terms of progressive, it'd be nice to start with 1 region but otherwise agree with your Math.min idea.
There was a problem hiding this comment.
Sounds good, I just pushed a change which does the following:
- each reopen proc will start with a batch size of 1
- the batch size will double after each confirmed batch (until we hit our max)
- our tests confirm the intended batching behavior, and that each region is only opened once
| retryCounter = null; | ||
| setNextState(ReopenTableRegionsState.REOPEN_TABLE_REGIONS_REOPEN_REGIONS); | ||
| if (reopenBatchBackoffMillis > 0) { | ||
| backoff(reopenBatchBackoffMillis); |
There was a problem hiding this comment.
This is a little confusing to read. Specifically, the way to backoff is by throwing ProcedureSuspendedException but below you are returning Flow.HAS_MORE_STATE. It would be nice to indicate that the return only happens if we don't back-off. One way to do that would be to add a comment below. But it might be better to rename backoff to setBackoffState() and throw a ProcedureSuspendedException here. You're already throwing it below (which is redundant given backoff throws it now)
There was a problem hiding this comment.
One other thing to note -- since we do skipPersistence(), we won't persist the setNextState change above. I think this should be fine. The implication is that in the case of a crash/restart while the procedure is suspended, we might backoff twice which is not the end of the world.
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Comment on lines
195
to
199
| Set<byte[]> currentRegionBatchNames = currentRegionBatch.stream() | ||
| .map(r -> r.getRegion().getRegionName()).collect(Collectors.toSet()); | ||
| currentRegionBatch = regions.stream() | ||
| .filter(r -> currentRegionBatchNames.contains(r.getRegion().getRegionName())) | ||
| .collect(Collectors.toList()); |
There was a problem hiding this comment.
Are you sure we need to do this? What if we did something like this:
case REOPEN_TABLE_REGIONS_CONFIRM_REOPENED:
// update region lists based on what's been reopened
regions = filterReopened(regions);
currentRegionBatch = filterReopened(currentRegionBatch)
// existing batch didn't fully reopen, so try to resolve that first.
// since this is a retry, don't do the batch backoff
if (!currentRegionBatch.isEmpty()) {
return reopenIfSchedulable(currentRegionBatch, false);
}
if (regions.isEmpty()) {
return Flow.NO_MORE_STATE;
}
// no batching or batch is finished, schedule more from main regions
return reopenIfSchedulable(regions, true);
default:
throw new UnsupportedOperationException("unhandled state=" + state);
}
...
private List<HRegionLocation> filterReopened(List<HRegionLocation> regionsToCheck) {
return regionsToCheck.stream().map(env.getAssignmentManager().getRegionStates()::checkReopened)
.filter(l -> l != null).collect(Collectors.toList());
}
private void reopenIfSchedulable(List<HRegionLocations> regionsToReopen, boolean shouldBatchBackoff) throws ProcedureSuspendedException {
if (regionsToReopen.stream().anyMatch(loc -> canSchedule(env, loc))) {
retryCounter = null;
setNextState(ReopenTableRegionsState.REOPEN_TABLE_REGIONS_REOPEN_REGIONS);
if (shouldBatchBackoff && reopenBatchBackoffMillis > 0) {
reopenBatchSize = Math.min(reopenBatchSizeMax, 2 * reopenBatchSize);
setBackoffStateAndSuspend(reopenBatchBackoffMillis);
} else {
return Flow.HAS_MORE_STATE;
}
}
// We can not schedule TRSP for all the regions need to reopen, wait for a while and retry
// again.
if (retryCounter == null) {
retryCounter = ProcedureUtil.createRetryCounter(env.getMasterConfiguration());
}
long backoffMillis = retryCounter.getBackoffTimeAndIncrementAttempts();
LOG.info(
"There are still {} region(s) which need to be reopened for table {}. {} are in "
+ "OPENING state, suspend {}secs and try again later",
regions.size(), tableName, regionsToCheck.size(), backoffMillis / 1000);
setBackoffStateAndSuspend(backoffMillis);
}It feels cleaner to read and understand what's happening. As mentioned in another comment, I'd recommend updating REOPEN_REGIONS to:
// if we didn't finish reopening the last batch yet, let's keep trying until we do.
// at that point, the batch will be empty and we can generate a new batch
if (currentRegionBatch.isEmpty()) {
currentRegionBatch = regions.stream().limit(reopenBatchSize).collect(Collectors.toList());
}There was a problem hiding this comment.
Agreed all around about the feedback here, just pushed a simplification that follows this advice almost exactly. The tests are all still passing of course
|
🎊 +1 overall
This message was automatically generated. |
rmdmattingly
force-pushed
the
HBASE-28215
branch
from
f7c146f to
afa02f9
Compare
| if (regionsToReopen.stream().anyMatch(loc -> canSchedule(env, loc))) { | ||
| retryCounter = null; | ||
| setNextState(ReopenTableRegionsState.REOPEN_TABLE_REGIONS_REOPEN_REGIONS); | ||
| reopenBatchSize = Math.min(reopenBatchSizeMax, 2 * reopenBatchSize); |
There was a problem hiding this comment.
i think we should only increase the batch size in the below if block
There was a problem hiding this comment.
That would mean that the default case of reopenBatchBackoffMillis = 0 is stuck processing single region batches indefinitely
There was a problem hiding this comment.
oh wait, i dont think we want progressive reopen enabled by default right away. i was actually thinking that this progressive batching would only apply if one of the new configs is enabled. i missed that above.
we might want to change our default values so that we can tell if a user actually changed the configs, and only enable progressive if so.
There was a problem hiding this comment.
oh interesting, ok that makes sense
Comment on lines
57
to
62
| public static final String REOPEN_BATCH_BACKOFF_MILLIS_KEY = | ||
| "hbase.table.regions.reopen.batch.backoff.ms"; | ||
| public static final long REOPEN_BATCH_BACKOFF_MILLIS_DEFAULT = 0L; | ||
| public static final String REOPEN_BATCH_SIZE_MAX_KEY = | ||
| "hbase.table.regions.reopen.batch.size.max"; | ||
| public static final int REOPEN_BATCH_SIZE_MAX_DEFAULT = Integer.MAX_VALUE; |
There was a problem hiding this comment.
nitpick: we might want to have these names be hbase.reopen.table.regions.progressive.batch.size.max and backoff.ms. It's more of a progressive reopen system now, and i reordered the table.regions.reopen to align more with the procedure name.
bbeaudreault
reviewed
Dec 1, 2023
...rver/src/main/java/org/apache/hadoop/hbase/master/procedure/ReopenTableRegionsProcedure.java
Outdated
Show resolved
Hide resolved
bbeaudreault
approved these changes
Dec 1, 2023
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
No rush, but looks like test failures are related and new compile/javac warnings |
|
The javac warning is legit, will get to that asap. The test failure was for the previous commit I believe |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
rmdmattingly
force-pushed
the
HBASE-28215
branch
from
eb51c02 to
8eec7ce
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
rmdmattingly
force-pushed
the
HBASE-28215
branch
from
8eec7ce to
0d010a0
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
bbeaudreault
approved these changes
Dec 4, 2023
bbeaudreault
pushed a commit
that referenced
this pull request
Dec 4, 2023
Signed-off-by: Bryan Beaudreault <[email protected]>
bbeaudreault
pushed a commit
that referenced
this pull request
Dec 4, 2023
Signed-off-by: Bryan Beaudreault <[email protected]>
bbeaudreault
pushed a commit
that referenced
this pull request
Dec 4, 2023
Signed-off-by: Bryan Beaudreault <[email protected]>
bbeaudreault
pushed a commit
to HubSpot/hbase
that referenced
this pull request
Dec 4, 2023
…tling (apache#5534) Signed-off-by: Bryan Beaudreault <[email protected]>
kadirozde
pushed a commit
to kadirozde/hbase
that referenced
this pull request
Jan 5, 2024
Signed-off-by: Bryan Beaudreault <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/HBASE-28215
The mass reopening of regions caused by a table descriptor modification can be quite disruptive. For latency/error sensitive workloads, like our user facing traffic, we need to be very careful about when we modify table descriptors, and it can be virtually impossible to do it painlessly for busy tables.
This PR introduces two new configurations:
@bbeaudreault @hgromer @eab148 @bozzkar