Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-22923 Consider minVersionToMoveSysTables while moving region and creating regionPlan (ADDENDUM) #3455

Merged
merged 1 commit into from
Jul 3, 2021

Conversation

virajjasani
Copy link
Contributor

No description provided.

Verified

This commit was signed with the committer’s verified signature.
virajjasani Viraj Jasani
…d creating regionPlan (ADDENDUM)
@virajjasani
Copy link
Contributor Author

FYI @d-c-manning
Tested region_mover and balancer with mixed mode of servers, works as expected, meaning no workflow considers moving system tables to higher version servers. Please review.
Thanks

@virajjasani virajjasani requested review from bharathv and apurtell July 2, 2021 19:25
@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 27s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 3m 55s master passed
+1 💚 compile 3m 10s master passed
+1 💚 checkstyle 1m 4s master passed
+1 💚 spotbugs 2m 5s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 37s the patch passed
+1 💚 compile 3m 10s the patch passed
+1 💚 javac 3m 10s the patch passed
+1 💚 checkstyle 1m 1s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 18m 15s Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚 spotbugs 2m 13s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 15s The patch does not generate ASF License warnings.
46m 56s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #3455
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname Linux 5f25bc424dba 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 5ef5e64
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 96 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@d-c-manning d-c-manning left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @virajjasani !

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 31s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 4m 41s master passed
+1 💚 compile 1m 16s master passed
+1 💚 shadedjars 8m 20s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 43s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 20s the patch passed
+1 💚 compile 1m 14s the patch passed
+1 💚 javac 1m 14s the patch passed
+1 💚 shadedjars 8m 28s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 50s the patch passed
_ Other Tests _
+1 💚 unit 143m 33s hbase-server in the patch passed.
175m 58s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #3455
Optional Tests javac javadoc unit shadedjars compile
uname Linux 927414ddb83f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 5ef5e64
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/testReport/
Max. process+thread count 3105 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 31s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 4m 4s master passed
+1 💚 compile 1m 2s master passed
+1 💚 shadedjars 8m 5s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 39s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 46s the patch passed
+1 💚 compile 0m 59s the patch passed
+1 💚 javac 0m 59s the patch passed
+1 💚 shadedjars 8m 4s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 37s the patch passed
_ Other Tests _
-1 ❌ unit 153m 49s hbase-server in the patch failed.
183m 50s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #3455
Optional Tests javac javadoc unit shadedjars compile
uname Linux 52d015dfa3c5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 5ef5e64
Default Java AdoptOpenJDK-1.8.0_282-b08
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/testReport/
Max. process+thread count 3358 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9
Copy link
Contributor

Apache9 commented Jul 3, 2021

So what's the problem here?

@virajjasani
Copy link
Contributor Author

virajjasani commented Jul 3, 2021

So what's the problem here?

With earlier patch of HBASE-22923, we only covered case of "AssignmentManager taking care of excluding lower version servers for system table assignment" for the thread that keeps track of version differences in the cluster and soon as it finds new version, it tries to move system tables there, so this is auto-migration of system tables use-case. But we missed the case of graceful stop and balancer's plans generation. Graceful stop prepares move commands based on RSGroup but if we don't change the logic of "excluding servers with lower version for system table assignment", system table regions will be in hanging state when we execute graceful stop command.
With the current patch, we have covered all cases that master uses to coordinate region movement, regardless of whether it is auto-migration of regions by AM or whether we execute graceful stop manually.

@virajjasani virajjasani merged commit 1883889 into apache:master Jul 3, 2021
@virajjasani virajjasani deleted the HBASE-22923-master-add branch July 3, 2021 17:11
virajjasani added a commit that referenced this pull request Jul 3, 2021

Verified

This commit was signed with the committer’s verified signature.
virajjasani Viraj Jasani
…d creating regionPlan (ADDENDUM) (#3455)

Signed-off-by: David Manning <david.manning@salesforce.com>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
virajjasani added a commit that referenced this pull request Jul 3, 2021

Verified

This commit was signed with the committer’s verified signature.
virajjasani Viraj Jasani
…d creating regionPlan (ADDENDUM) (#3455)

Signed-off-by: David Manning <david.manning@salesforce.com>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
virajjasani added a commit that referenced this pull request Jul 3, 2021

Verified

This commit was signed with the committer’s verified signature.
virajjasani Viraj Jasani
…d creating regionPlan (ADDENDUM) (#3455)

Signed-off-by: David Manning <david.manning@salesforce.com>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
virajjasani added a commit that referenced this pull request Jul 3, 2021

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
…d creating regionPlan (ADDENDUM) (#3456) (#3455)

Signed-off-by: David Manning <david.manning@salesforce.com>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants