HBASE-27061 two phase bulkload is broken when SFT is in use. #4465

ss77892 · 2022-05-26T04:03:37Z

o reproduce the bug I've used hbase with hbase root dir pointing to s3 bucket and WAL dir pointing to HDFS. 2 phase bulkload was used:
Phase1 (Completed successfully):
hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=',' -Dimporttsv.bulk.output=output -Dimporttsv.columns=HBASE_ROW_KEY,name,department employees /tmp/test.csv

Phase2 (Failed with IllegalArgumentException on the RS side):
hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles output employees

With the provided changes Phase2 was completed successfully.

Apache-HBase · 2022-05-26T04:31:58Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 28s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 45s	master passed
+1 💚	compile	0m 48s	master passed
+1 💚	shadedjars	3m 42s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 29s	master passed
		_ Patch Compile Tests _
-1 ❌	mvninstall	1m 24s	root in the patch failed.
-1 ❌	compile	0m 29s	hbase-server in the patch failed.
-0 ⚠️	javac	0m 29s	hbase-server in the patch failed.
-1 ❌	shadedjars	2m 44s	patch has 16 errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	the patch passed
		_ Other Tests _
-1 ❌	unit	0m 29s	hbase-server in the patch failed.
		16m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4465
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 490ed89d2197 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `286f6c5`
Default Java	AdoptOpenJDK-11.0.10+9
mvninstall	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt
compile	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt
javac	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt
shadedjars	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/testReport/
Max. process+thread count	70 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-05-26T04:32:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 44s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 33s	master passed
+1 💚	compile	2m 10s	master passed
+1 💚	checkstyle	0m 31s	master passed
+1 💚	spotless	0m 46s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 12s	master passed
		_ Patch Compile Tests _
-1 ❌	mvninstall	0m 59s	root in the patch failed.
-1 ❌	compile	0m 25s	hbase-server in the patch failed.
-0 ⚠️	javac	0m 25s	hbase-server in the patch failed.
+1 💚	checkstyle	0m 26s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
-1 ❌	hadoopcheck	1m 3s	The patch causes 16 errors with Hadoop v3.1.2.
-1 ❌	hadoopcheck	2m 18s	The patch causes 16 errors with Hadoop v3.2.2.
-1 ❌	hadoopcheck	3m 30s	The patch causes 16 errors with Hadoop v3.3.1.
+1 💚	spotless	0m 34s	patch has no errors when running spotless:check.
-1 ❌	spotbugs	0m 20s	hbase-server in the patch failed.
		_ Other Tests _
+1 💚	asflicense	0m 8s	The patch does not generate ASF License warnings.
		16m 25s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4465
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux f2e3be0c3b5c 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `286f6c5`
Default Java	AdoptOpenJDK-1.8.0_282-b08
mvninstall	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/patch-mvninstall-root.txt
compile	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/patch-compile-hbase-server.txt
javac	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/patch-compile-hbase-server.txt
hadoopcheck	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/patch-javac-3.1.2.txt
hadoopcheck	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/patch-javac-3.2.2.txt
hadoopcheck	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/patch-javac-3.3.1.txt
spotbugs	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/patch-spotbugs-hbase-server.txt
Max. process+thread count	60 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-05-26T04:33:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 3s	Docker mode activated.
-0 ⚠️	yetus	0m 5s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 11s	master passed
+1 💚	compile	0m 42s	master passed
+1 💚	shadedjars	4m 25s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 34s	master passed
		_ Patch Compile Tests _
-1 ❌	mvninstall	1m 21s	root in the patch failed.
-1 ❌	compile	0m 29s	hbase-server in the patch failed.
-0 ⚠️	javac	0m 29s	hbase-server in the patch failed.
-1 ❌	shadedjars	2m 54s	patch has 16 errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 21s	the patch passed
		_ Other Tests _
-1 ❌	unit	0m 22s	hbase-server in the patch failed.
		17m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4465
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 8c1cbcbd5a7a 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `286f6c5`
Default Java	AdoptOpenJDK-1.8.0_282-b08
mvninstall	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt
compile	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt
javac	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt
shadedjars	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/testReport/
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-05-26T05:27:04Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 44s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 4s	master passed
+1 💚	compile	2m 9s	master passed
+1 💚	checkstyle	0m 26s	master passed
+1 💚	spotless	0m 37s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 9s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 7s	the patch passed
+1 💚	compile	2m 7s	the patch passed
+1 💚	javac	2m 7s	the patch passed
+1 💚	checkstyle	0m 27s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	11m 24s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotless	0m 37s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 16s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 7s	The patch does not generate ASF License warnings.
		30m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4465
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux c551550358e1 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `286f6c5`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-05-26T08:30:39Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 43s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 11s	master passed
+1 💚	compile	0m 34s	master passed
+1 💚	shadedjars	3m 51s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 21s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 10s	the patch passed
+1 💚	compile	0m 35s	the patch passed
+1 💚	javac	0m 35s	the patch passed
+1 💚	shadedjars	3m 56s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 21s	the patch passed
		_ Other Tests _
+1 💚	unit	196m 26s	hbase-server in the patch passed.
		213m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4465
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 5d338ca60ce8 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `286f6c5`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/testReport/
Max. process+thread count	2636 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-05-26T08:36:19Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 11s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 34s	master passed
+1 💚	compile	0m 46s	master passed
+1 💚	shadedjars	3m 41s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 32s	the patch passed
+1 💚	compile	0m 46s	the patch passed
+1 💚	javac	0m 46s	the patch passed
+1 💚	shadedjars	3m 40s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 27s	the patch passed
		_ Other Tests _
+1 💚	unit	201m 35s	hbase-server in the patch passed.
		219m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4465
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 8cd380abd9e9 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `286f6c5`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/testReport/
Max. process+thread count	2460 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Signed-off-by: Wellington Chevreuil <[email protected]>

…4465) Signed-off-by: Wellington Chevreuil <[email protected]>

…pache#4465)" This reverts commit ae494c1.

…4465) Signed-off-by: Wellington Chevreuil <[email protected]> Change-Id: I21f336eed6a597ff84c7014db32d5d052681b652

HBASE-27061 two phase bulkload is broken when SFT is in use.

98cc234

ss77892 force-pushed the HBASE-27061-master branch from 7a4836b to 98cc234 Compare May 26, 2022 04:49

wchevreuil approved these changes May 26, 2022

View reviewed changes

wchevreuil merged commit fdf054d into apache:master May 26, 2022

wchevreuil pushed a commit that referenced this pull request May 26, 2022

HBASE-27061 two phase bulkload is broken when SFT is in use. (#4465)

50dd961

Signed-off-by: Wellington Chevreuil <[email protected]>

wchevreuil pushed a commit that referenced this pull request May 26, 2022

HBASE-27061 two phase bulkload is broken when SFT is in use. (#4465)

c9fc396

Signed-off-by: Wellington Chevreuil <[email protected]>

wenwj0 pushed a commit to wenwj0/hbase that referenced this pull request Jun 14, 2022

HBASE-27061 two phase bulkload is broken when SFT is in use. (apache#…

ae494c1

…4465) Signed-off-by: Wellington Chevreuil <[email protected]>

wenwj0 added a commit to wenwj0/hbase that referenced this pull request Jun 14, 2022

Revert "HBASE-27061 two phase bulkload is broken when SFT is in use. (a…

11967b8

…pache#4465)" This reverts commit ae494c1.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-27061 two phase bulkload is broken when SFT is in use. #4465

HBASE-27061 two phase bulkload is broken when SFT is in use. #4465

ss77892 commented May 26, 2022 •

edited

Loading

Apache-HBase commented May 26, 2022

Apache-HBase commented May 26, 2022

Apache-HBase commented May 26, 2022

Apache-HBase commented May 26, 2022

Apache-HBase commented May 26, 2022

Apache-HBase commented May 26, 2022

HBASE-27061 two phase bulkload is broken when SFT is in use. #4465

HBASE-27061 two phase bulkload is broken when SFT is in use. #4465

Conversation

ss77892 commented May 26, 2022 • edited Loading

Apache-HBase commented May 26, 2022

Apache-HBase commented May 26, 2022

Apache-HBase commented May 26, 2022

Apache-HBase commented May 26, 2022

Apache-HBase commented May 26, 2022

Apache-HBase commented May 26, 2022

ss77892 commented May 26, 2022 •

edited

Loading