HBASE-23078 BaseLoadBalancer should consider region replicas when ran… #663

infraio · 2019-09-26T12:21:06Z

…domAssignment and roundRobinAssignment

saintstack

Nice cleanup.

How you think this changes the behavior of the balancer?

saintstack · 2019-09-26T16:19:58Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java

+        TableName tableName = regionInfo.getTable();
+        if (services != null &&
+            services.getTableDescriptors().get(tableName).getRegionReplication() > 1) {
+          hasRegionReplica = true;


Apache-HBase · 2019-09-26T17:27:56Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
💙	reexec	1m 12s	Docker mode activated.
		_ Prechecks _
💚	dupname	0m 0s	No case conflicting files found.
💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
💚	@author	0m 0s	The patch does not contain any @author tags.
💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ master Compile Tests _
💚	mvninstall	5m 56s	master passed
💚	compile	0m 55s	master passed
💚	checkstyle	1m 33s	master passed
💚	shadedjars	5m 9s	branch has no errors when building our shaded downstream artifacts.
💚	javadoc	0m 38s	master passed
💙	spotbugs	4m 30s	Used deprecated FindBugs config; considering switching to SpotBugs.
💚	findbugs	4m 27s	master passed
		_ Patch Compile Tests _
💚	mvninstall	5m 31s	the patch passed
💚	compile	0m 59s	the patch passed
💚	javac	0m 59s	the patch passed
💔	checkstyle	1m 28s	hbase-server: The patch generated 2 new + 61 unchanged - 0 fixed = 63 total (was 61)
💚	whitespace	0m 0s	The patch has no whitespace issues.
💚	shadedjars	5m 3s	patch has no errors when building our shaded downstream artifacts.
💚	hadoopcheck	17m 25s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚	javadoc	0m 37s	the patch passed
💚	findbugs	4m 40s	the patch passed
		_ Other Tests _
💔	unit	234m 32s	hbase-server in the patch failed.
💚	asflicense	0m 30s	The patch does not generate ASF License warnings.
		297m 34s

Reason	Tests
Failed junit tests	hadoop.hbase.master.TestRegionPlacement2
	hadoop.hbase.master.balancer.TestBaseLoadBalancer

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/1/artifact/out/Dockerfile
GITHUB PR	#663
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux 0460cea06f1d 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-663/out/precommit/personality/provided.sh
git revision	master / `f0dddd1`
Default Java	1.8.0_181
checkstyle	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/1/artifact/out/diff-checkstyle-hbase-server.txt
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/1/artifact/out/patch-unit-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/1/testReport/
Max. process+thread count	4963 (vs. ulimit of 10000)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/1/console
versions	git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by	Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

ramkrish86 · 2019-09-27T05:24:51Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java

+      for (RegionInfo regionInfo : regions) {
+        TableName tableName = regionInfo.getTable();
+        if (services != null &&
+            services.getTableDescriptors().get(tableName).getRegionReplication() > 1) {


I think the descriptors are anyway cached right? If so good.

Updated in new patch. Will only getTableDescriptors once now.

ramkrish86 · 2019-09-27T05:26:14Z

Looks good to me.

Apache-HBase · 2019-09-27T07:31:41Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
💙	reexec	2m 28s	Docker mode activated.
		_ Prechecks _
💚	dupname	0m 0s	No case conflicting files found.
💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
💚	@author	0m 0s	The patch does not contain any @author tags.
💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ master Compile Tests _
💚	mvninstall	8m 3s	master passed
💚	compile	1m 14s	master passed
💚	checkstyle	1m 40s	master passed
💚	shadedjars	6m 15s	branch has no errors when building our shaded downstream artifacts.
💚	javadoc	0m 52s	master passed
💙	spotbugs	5m 22s	Used deprecated FindBugs config; considering switching to SpotBugs.
💚	findbugs	5m 20s	master passed
		_ Patch Compile Tests _
💚	mvninstall	7m 27s	the patch passed
💚	compile	1m 18s	the patch passed
💚	javac	1m 18s	the patch passed
💚	checkstyle	1m 38s	the patch passed
💚	whitespace	0m 0s	The patch has no whitespace issues.
💚	shadedjars	6m 16s	patch has no errors when building our shaded downstream artifacts.
💚	hadoopcheck	23m 2s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚	javadoc	0m 48s	the patch passed
💚	findbugs	6m 27s	the patch passed
		_ Other Tests _
💔	unit	240m 23s	hbase-server in the patch failed.
💚	asflicense	0m 26s	The patch does not generate ASF License warnings.
		322m 18s

Reason	Tests
Failed junit tests	hadoop.hbase.master.TestRegionPlacement2
	hadoop.hbase.master.TestMasterShutdown
	hadoop.hbase.master.balancer.TestBaseLoadBalancer

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/2/artifact/out/Dockerfile
GITHUB PR	#663
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux b9e768eaa24c 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-663/out/precommit/personality/provided.sh
git revision	master / `3250a80`
Default Java	1.8.0_181
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/2/artifact/out/patch-unit-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/2/testReport/
Max. process+thread count	5097 (vs. ulimit of 10000)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/2/console
versions	git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by	Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2019-09-27T14:44:07Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
💙	reexec	0m 35s	Docker mode activated.
		_ Prechecks _
💚	dupname	0m 0s	No case conflicting files found.
💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
💚	@author	0m 0s	The patch does not contain any @author tags.
💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ master Compile Tests _
💚	mvninstall	5m 26s	master passed
💚	compile	0m 56s	master passed
💚	checkstyle	1m 21s	master passed
💚	shadedjars	4m 31s	branch has no errors when building our shaded downstream artifacts.
💚	javadoc	0m 38s	master passed
💙	spotbugs	4m 1s	Used deprecated FindBugs config; considering switching to SpotBugs.
💚	findbugs	3m 59s	master passed
		_ Patch Compile Tests _
💚	mvninstall	4m 55s	the patch passed
💚	compile	0m 56s	the patch passed
💚	javac	0m 56s	the patch passed
💚	checkstyle	1m 19s	the patch passed
💚	whitespace	0m 0s	The patch has no whitespace issues.
💚	shadedjars	4m 32s	patch has no errors when building our shaded downstream artifacts.
💚	hadoopcheck	15m 30s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚	javadoc	0m 35s	the patch passed
💚	findbugs	4m 10s	the patch passed
		_ Other Tests _
💚	unit	157m 12s	hbase-server in the patch passed.
💚	asflicense	0m 35s	The patch does not generate ASF License warnings.
		213m 34s

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/3/artifact/out/Dockerfile
GITHUB PR	#663
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux 44d0e592d884 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-663/out/precommit/personality/provided.sh
git revision	master / `3250a80`
Default Java	1.8.0_181
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/3/testReport/
Max. process+thread count	4855 (vs. ulimit of 10000)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/3/console
versions	git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by	Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

infraio · 2019-09-28T13:52:45Z

Add the region replica UT back.

Apache-HBase · 2019-09-28T19:11:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
💙	reexec	2m 10s	Docker mode activated.
		_ Prechecks _
💚	dupname	0m 0s	No case conflicting files found.
💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
💚	@author	0m 0s	The patch does not contain any @author tags.
💚	test4tests	0m 0s	The patch appears to include 3 new or modified test files.
		_ master Compile Tests _
💚	mvninstall	7m 43s	master passed
💚	compile	1m 2s	master passed
💚	checkstyle	1m 35s	master passed
💚	shadedjars	5m 21s	branch has no errors when building our shaded downstream artifacts.
💚	javadoc	0m 41s	master passed
💙	spotbugs	4m 40s	Used deprecated FindBugs config; considering switching to SpotBugs.
💚	findbugs	4m 37s	master passed
		_ Patch Compile Tests _
💚	mvninstall	5m 39s	the patch passed
💚	compile	1m 4s	the patch passed
💚	javac	1m 4s	the patch passed
💔	checkstyle	1m 34s	hbase-server: The patch generated 1 new + 71 unchanged - 0 fixed = 72 total (was 71)
💚	whitespace	0m 0s	The patch has no whitespace issues.
💚	shadedjars	5m 17s	patch has no errors when building our shaded downstream artifacts.
💚	hadoopcheck	18m 20s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚	javadoc	0m 34s	the patch passed
💚	findbugs	4m 32s	the patch passed
		_ Other Tests _
💔	unit	244m 16s	hbase-server in the patch failed.
💚	asflicense	0m 28s	The patch does not generate ASF License warnings.
		312m 1s

Reason	Tests
Failed junit tests	hadoop.hbase.master.TestMasterShutdown
	hadoop.hbase.client.TestAsyncRegionAdminApi2

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/4/artifact/out/Dockerfile
GITHUB PR	#663
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux ce172c12f057 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-663/out/precommit/personality/provided.sh
git revision	master / `ef79b40`
Default Java	1.8.0_181
checkstyle	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/4/artifact/out/diff-checkstyle-hbase-server.txt
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/4/artifact/out/patch-unit-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/4/testReport/
Max. process+thread count	5062 (vs. ulimit of 10000)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/4/console
versions	git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by	Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

…domAssignment and roundRobinAssignment

Apache-HBase · 2019-09-29T05:41:16Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
💙	reexec	1m 15s	Docker mode activated.
		_ Prechecks _
💚	dupname	0m 0s	No case conflicting files found.
💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
💚	@author	0m 0s	The patch does not contain any @author tags.
💚	test4tests	0m 0s	The patch appears to include 3 new or modified test files.
		_ master Compile Tests _
💚	mvninstall	5m 59s	master passed
💚	compile	0m 57s	master passed
💚	checkstyle	1m 32s	master passed
💚	shadedjars	5m 5s	branch has no errors when building our shaded downstream artifacts.
💚	javadoc	0m 40s	master passed
💙	spotbugs	4m 25s	Used deprecated FindBugs config; considering switching to SpotBugs.
💚	findbugs	4m 23s	master passed
		_ Patch Compile Tests _
💚	mvninstall	5m 30s	the patch passed
💚	compile	0m 56s	the patch passed
💚	javac	0m 56s	the patch passed
💚	checkstyle	1m 28s	the patch passed
💚	whitespace	0m 0s	The patch has no whitespace issues.
💚	shadedjars	5m 3s	patch has no errors when building our shaded downstream artifacts.
💚	hadoopcheck	17m 9s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚	javadoc	0m 35s	the patch passed
💚	findbugs	4m 32s	the patch passed
		_ Other Tests _
💔	unit	254m 8s	hbase-server in the patch failed.
💚	asflicense	0m 26s	The patch does not generate ASF License warnings.
		316m 28s

Reason	Tests
Failed junit tests	hadoop.hbase.master.assignment.TestRegionBypass
	hadoop.hbase.util.TestFromClientSide3WoUnsafe
	hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure
	hadoop.hbase.master.TestSplitWALManager
	hadoop.hbase.master.assignment.TestMergeTableRegionsProcedure

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/5/artifact/out/Dockerfile
GITHUB PR	#663
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux 672d38183af9 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-663/out/precommit/personality/provided.sh
git revision	master / `8363681`
Default Java	1.8.0_181
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/5/artifact/out/patch-unit-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/5/testReport/
Max. process+thread count	5006 (vs. ulimit of 10000)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-663/5/console
versions	git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by	Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

infraio · 2019-09-29T09:35:39Z

The failed ut not related and passed.

…domAssignment and roundRobinAssignment (#663) Signed-off-by: stack <[email protected]>

…domAssignment and roundRobinAssignment (apache#663) Signed-off-by: stack <[email protected]>

…domAssignment and roundRobinAssignment (apache#663) Signed-off-by: stack <[email protected]> (cherry picked from commit c50e3f3) Change-Id: I596ee594082021941f21ceb1b8150fc85ccb513b

saintstack approved these changes Sep 26, 2019

View reviewed changes

infraio force-pushed the HBASE-23078 branch from e8de3bd to f7ee213 Compare September 27, 2019 02:00

ramkrish86 reviewed Sep 27, 2019

View reviewed changes

infraio force-pushed the HBASE-23078 branch from f7ee213 to ae95608 Compare September 27, 2019 11:03

infraio force-pushed the HBASE-23078 branch from ae95608 to 74514a6 Compare September 28, 2019 13:49

HBASE-23078 BaseLoadBalancer should consider region replicas when ran…

9cd844b

…domAssignment and roundRobinAssignment

infraio force-pushed the HBASE-23078 branch from 74514a6 to 9cd844b Compare September 29, 2019 00:21

infraio merged commit 1170f28 into apache:master Sep 29, 2019

infraio deleted the HBASE-23078 branch September 29, 2019 09:36

asfgit pushed a commit that referenced this pull request Sep 29, 2019

HBASE-23078 BaseLoadBalancer should consider region replicas when ran…

9f703fc

…domAssignment and roundRobinAssignment (#663) Signed-off-by: stack <[email protected]>

asfgit pushed a commit that referenced this pull request Sep 29, 2019

HBASE-23078 BaseLoadBalancer should consider region replicas when ran…

c50e3f3

…domAssignment and roundRobinAssignment (#663) Signed-off-by: stack <[email protected]>

infraio added a commit to infraio/hbase that referenced this pull request Aug 17, 2020

HBASE-23078 BaseLoadBalancer should consider region replicas when ran…

2dafc24

…domAssignment and roundRobinAssignment (apache#663) Signed-off-by: stack <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-23078 BaseLoadBalancer should consider region replicas when ran… #663

HBASE-23078 BaseLoadBalancer should consider region replicas when ran… #663

infraio commented Sep 26, 2019

saintstack left a comment

saintstack Sep 26, 2019

Apache-HBase commented Sep 26, 2019

ramkrish86 Sep 27, 2019

infraio Sep 28, 2019

ramkrish86 commented Sep 27, 2019

Apache-HBase commented Sep 27, 2019

Apache-HBase commented Sep 27, 2019

infraio commented Sep 28, 2019

Apache-HBase commented Sep 28, 2019

Apache-HBase commented Sep 29, 2019

infraio commented Sep 29, 2019

HBASE-23078 BaseLoadBalancer should consider region replicas when ran… #663

HBASE-23078 BaseLoadBalancer should consider region replicas when ran… #663

Conversation

infraio commented Sep 26, 2019

saintstack left a comment

Choose a reason for hiding this comment

saintstack Sep 26, 2019

Choose a reason for hiding this comment

Apache-HBase commented Sep 26, 2019

ramkrish86 Sep 27, 2019

Choose a reason for hiding this comment

infraio Sep 28, 2019

Choose a reason for hiding this comment

ramkrish86 commented Sep 27, 2019

Apache-HBase commented Sep 27, 2019

Apache-HBase commented Sep 27, 2019

infraio commented Sep 28, 2019

Apache-HBase commented Sep 28, 2019

Apache-HBase commented Sep 29, 2019

infraio commented Sep 29, 2019