HBASE-26556 IT and Chaos Monkey improvements #3932

wchevreuil · 2021-12-10T00:19:09Z

No description provided.

Change-Id: I8e26c27c2f2f5600406c25de6fceb598c980f1d5

Apache-HBase · 2021-12-10T00:50:10Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 27s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 44s	master passed
+1 💚	compile	0m 30s	master passed
+1 💚	shadedjars	8m 20s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 21s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 29s	the patch passed
+1 💚	compile	0m 32s	the patch passed
-0 ⚠️	javac	0m 32s	hbase-it generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedjars	8m 12s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 18s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 56s	hbase-it in the patch passed.
		30m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3932
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 84c14a394c1a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b5cf3cd`
Default Java	AdoptOpenJDK-11.0.10+9
javac	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/1/artifact/yetus-jdk11-hadoop3-check/output/diff-compile-javac-hbase-it.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/1/testReport/
Max. process+thread count	698 (vs. ulimit of 30000)
modules	C: hbase-it U: hbase-it
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-10T00:53:17Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 56s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 13s	master passed
+1 💚	compile	0m 28s	master passed
+1 💚	shadedjars	8m 33s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 20s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 0s	the patch passed
+1 💚	compile	0m 29s	the patch passed
-0 ⚠️	javac	0m 29s	hbase-it generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedjars	8m 39s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 17s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 59s	hbase-it in the patch passed.
		33m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3932
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c4b07340dcd5 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b5cf3cd`
Default Java	AdoptOpenJDK-1.8.0_282-b08
javac	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/1/artifact/yetus-jdk8-hadoop3-check/output/diff-compile-javac-hbase-it.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/1/testReport/
Max. process+thread count	689 (vs. ulimit of 30000)
modules	C: hbase-it U: hbase-it
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-10T01:10:33Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 24s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	5m 13s	master passed
+1 💚	compile	0m 49s	master passed
+1 💚	checkstyle	0m 22s	master passed
+1 💚	spotbugs	0m 46s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 43s	the patch passed
+1 💚	compile	0m 50s	the patch passed
-0 ⚠️	javac	0m 50s	hbase-it generated 2 new + 101 unchanged - 2 fixed = 103 total (was 103)
-0 ⚠️	checkstyle	0m 18s	hbase-it: The patch generated 3 new + 19 unchanged - 24 fixed = 22 total (was 43)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	24m 47s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	0m 51s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		50m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3932
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 67f2b61e44f1 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b5cf3cd`
Default Java	AdoptOpenJDK-1.8.0_282-b08
javac	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-it.txt
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-it.txt
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-it U: hbase-it
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

taklwu

LGTM, just one nit comment for javac and checkstyle

hbase-it/src/test/java/org/apache/hadoop/hbase/HBaseClusterManager.java

taklwu · 2021-12-10T05:30:31Z

...java/org/apache/hadoop/hbase/chaos/factories/ConfigurableSlowDeterministicMonkeyFactory.java

+  @Override
+  protected Action[] getHeavyWeightedActions() {
+    String actions = this.properties.getProperty(HEAVY_ACTIONS);
+    if( actions==null||actions.isEmpty() ){


[nit] javac and checkstyle seems failing, please see if you can fix it.

joshelser

Spelling errors to fix.

Could you also consider adding some documentation as to how one would use the new policy? https://hbase.apache.org/book.html#maven.build.commands.integration.tests.destructive Ok to me if that is done separately

joshelser · 2021-12-10T16:16:27Z

...it/src/test/java/org/apache/hadoop/hbase/chaos/factories/SlowDeterministicMonkeyFactory.java

@@ -72,53 +72,65 @@
  private long rollingBatchSuspendRSSleepTime;
  private float rollingBatchSuspendtRSRatio;

+  protected Action[] getLightWeigthedActions(){


Suggested change

protected Action[] getLightWeigthedActions(){

protected Action[] getLightWeightedActions(){

...it/src/test/java/org/apache/hadoop/hbase/chaos/factories/SlowDeterministicMonkeyFactory.java

...java/org/apache/hadoop/hbase/chaos/factories/ConfigurableSlowDeterministicMonkeyFactory.java

Change-Id: If75dc3fa225cf9359e92bac0579bf46cbc3cf2ca

Apache-HBase · 2021-12-14T15:53:42Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 29s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 59s	master passed
+1 💚	compile	0m 33s	master passed
+1 💚	shadedjars	8m 48s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 22s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 30s	the patch passed
+1 💚	compile	0m 32s	the patch passed
-0 ⚠️	javac	0m 32s	hbase-it generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedjars	8m 24s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 18s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 57s	hbase-it in the patch passed.
		31m 12s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3932
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux bd6f1c16bab5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `13f3b17`
Default Java	AdoptOpenJDK-11.0.10+9
javac	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/2/artifact/yetus-jdk11-hadoop3-check/output/diff-compile-javac-hbase-it.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/2/testReport/
Max. process+thread count	677 (vs. ulimit of 30000)
modules	C: hbase-it U: hbase-it
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-14T15:54:27Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 16s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 34s	master passed
+1 💚	compile	0m 27s	master passed
+1 💚	shadedjars	9m 4s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 18s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 18s	the patch passed
+1 💚	compile	0m 28s	the patch passed
-0 ⚠️	javac	0m 28s	hbase-it generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedjars	9m 5s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 15s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 57s	hbase-it in the patch passed.
		31m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3932
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 9dafd1d08aab 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `13f3b17`
Default Java	AdoptOpenJDK-1.8.0_282-b08
javac	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/2/artifact/yetus-jdk8-hadoop3-check/output/diff-compile-javac-hbase-it.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/2/testReport/
Max. process+thread count	638 (vs. ulimit of 30000)
modules	C: hbase-it U: hbase-it
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-14T16:02:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 11s	master passed
+1 💚	compile	0m 44s	master passed
+1 💚	checkstyle	0m 21s	master passed
+1 💚	spotbugs	0m 37s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 51s	the patch passed
+1 💚	compile	0m 42s	the patch passed
-0 ⚠️	javac	0m 42s	hbase-it generated 2 new + 101 unchanged - 2 fixed = 103 total (was 103)
+1 💚	checkstyle	0m 17s	hbase-it: The patch generated 0 new + 19 unchanged - 24 fixed = 19 total (was 43)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	19m 22s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	0m 45s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		39m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3932
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 49c990609756 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `13f3b17`
Default Java	AdoptOpenJDK-1.8.0_282-b08
javac	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/2/artifact/yetus-general-check/output/diff-compile-javac-hbase-it.txt
Max. process+thread count	96 (vs. ulimit of 30000)
modules	C: hbase-it U: hbase-it
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3932/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

joshelser

Re-reviewed after last update. Looks good to me. We can update docs with how to use in follow-on.

wchevreuil · 2021-12-14T21:20:46Z

Re-reviewed after last update. Looks good to me. We can update docs with how to use in follow-on.

Thanks @joshelser . Yeah, was just waiting the qa run before replying to you, I will submit an update to docs in another PR.

Signed-off-by: Josh Elser <[email protected]> Reviewed-by: Tak Lon (Stephen) Wu <[email protected]> (cherry picked from commit a36d41a)

Signed-off-by: Josh Elser <[email protected]> Reviewed-by: Tak Lon (Stephen) Wu <[email protected]> (cherry picked from commit a36d41a) (cherry picked from commit 0866975) Change-Id: Ie8d9f995268233ec6a140860bf4a61300d656fba

HBASE-26556 IT and Chaos Monkey improvements

845cd02

Change-Id: I8e26c27c2f2f5600406c25de6fceb598c980f1d5

taklwu reviewed Dec 10, 2021

View reviewed changes

joshelser approved these changes Dec 10, 2021

View reviewed changes

addressing review comments

2ddf62f

Change-Id: If75dc3fa225cf9359e92bac0579bf46cbc3cf2ca

joshelser approved these changes Dec 14, 2021

View reviewed changes

wchevreuil merged commit a36d41a into apache:master Dec 14, 2021

wchevreuil added a commit that referenced this pull request Dec 14, 2021

HBASE-26556 IT and Chaos Monkey improvements (#3932)

5defd8c

Signed-off-by: Josh Elser <[email protected]> Reviewed-by: Tak Lon (Stephen) Wu <[email protected]> (cherry picked from commit a36d41a)

wchevreuil added a commit that referenced this pull request Dec 14, 2021

HBASE-26556 IT and Chaos Monkey improvements (#3932)

55fdff5

Signed-off-by: Josh Elser <[email protected]> Reviewed-by: Tak Lon (Stephen) Wu <[email protected]> (cherry picked from commit a36d41a)

wchevreuil added a commit that referenced this pull request Dec 14, 2021

HBASE-26556 IT and Chaos Monkey improvements (#3932)

0866975

Signed-off-by: Josh Elser <[email protected]> Reviewed-by: Tak Lon (Stephen) Wu <[email protected]> (cherry picked from commit a36d41a)

wchevreuil added a commit that referenced this pull request Dec 14, 2021

HBASE-26556 IT and Chaos Monkey improvements (#3932)

dc1f546

Signed-off-by: Josh Elser <[email protected]> Reviewed-by: Tak Lon (Stephen) Wu <[email protected]> (cherry picked from commit a36d41a)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-26556 IT and Chaos Monkey improvements #3932

HBASE-26556 IT and Chaos Monkey improvements #3932

wchevreuil commented Dec 10, 2021

Apache-HBase commented Dec 10, 2021

Apache-HBase commented Dec 10, 2021

Apache-HBase commented Dec 10, 2021

taklwu left a comment

taklwu Dec 10, 2021

wchevreuil Dec 14, 2021

joshelser left a comment

joshelser Dec 10, 2021

wchevreuil Dec 14, 2021

Apache-HBase commented Dec 14, 2021

Apache-HBase commented Dec 14, 2021

Apache-HBase commented Dec 14, 2021

joshelser left a comment

wchevreuil commented Dec 14, 2021

	protected Action[] getLightWeigthedActions(){
	protected Action[] getLightWeightedActions(){

HBASE-26556 IT and Chaos Monkey improvements #3932

HBASE-26556 IT and Chaos Monkey improvements #3932

Conversation

wchevreuil commented Dec 10, 2021

Apache-HBase commented Dec 10, 2021

Apache-HBase commented Dec 10, 2021

Apache-HBase commented Dec 10, 2021

taklwu left a comment

Choose a reason for hiding this comment

taklwu Dec 10, 2021

Choose a reason for hiding this comment

wchevreuil Dec 14, 2021

Choose a reason for hiding this comment

joshelser left a comment

Choose a reason for hiding this comment

joshelser Dec 10, 2021

Choose a reason for hiding this comment

wchevreuil Dec 14, 2021

Choose a reason for hiding this comment

Apache-HBase commented Dec 14, 2021

Apache-HBase commented Dec 14, 2021

Apache-HBase commented Dec 14, 2021

joshelser left a comment

Choose a reason for hiding this comment

wchevreuil commented Dec 14, 2021