HBASE-26646 WALPlayer should obtain token from filesystem #4003

mymeiyi · 2022-01-05T09:04:08Z

No description provided.

Apache-HBase · 2022-01-05T09:45:50Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 3s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 20s	master passed
+1 💚	compile	0m 51s	master passed
+1 💚	checkstyle	0m 19s	master passed
+1 💚	spotbugs	0m 47s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 50s	the patch passed
+1 💚	compile	0m 49s	the patch passed
+1 💚	javac	0m 49s	the patch passed
+1 💚	checkstyle	0m 17s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	19m 16s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	0m 56s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 13s	The patch does not generate ASF License warnings.
		40m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4003
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 2cdd01e3cb60 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `ce2bbaf`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	96 (vs. ulimit of 30000)
modules	C: hbase-mapreduce U: hbase-mapreduce
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-01-05T09:52:18Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 28s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	5m 8s	master passed
+1 💚	compile	0m 30s	master passed
+1 💚	shadedjars	9m 42s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 51s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	6m 7s	the patch passed
+1 💚	compile	0m 30s	the patch passed
+1 💚	javac	0m 30s	the patch passed
+1 💚	shadedjars	9m 6s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
+1 💚	unit	11m 52s	hbase-mapreduce in the patch passed.
		47m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4003
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 1845943de09a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `ce2bbaf`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/1/testReport/
Max. process+thread count	4651 (vs. ulimit of 30000)
modules	C: hbase-mapreduce U: hbase-mapreduce
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-01-05T09:56:40Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 17s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	5m 27s	master passed
+1 💚	compile	0m 28s	master passed
+1 💚	shadedjars	9m 29s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 20s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 23s	the patch passed
+1 💚	compile	0m 30s	the patch passed
+1 💚	javac	0m 30s	the patch passed
+1 💚	shadedjars	9m 25s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 19s	the patch passed
		_ Other Tests _
+1 💚	unit	18m 36s	hbase-mapreduce in the patch passed.
		51m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4003
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 2a8a0875500d 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `ce2bbaf`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/1/testReport/
Max. process+thread count	3641 (vs. ulimit of 30000)
modules	C: hbase-mapreduce U: hbase-mapreduce
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

shahrs87 · 2022-01-05T13:38:40Z

@mymeiyi Can you add a test case with secure cluster setup ?

shahrs87 · 2022-01-05T13:50:09Z

hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java

@@ -344,6 +345,9 @@ public Job createSubmittableJob(String[] args) throws IOException {
    try {
      TableMapReduceUtil.addDependencyJarsForClasses(job.getConfiguration(),
        Class.forName(codecCls));
+      // get delegation token for the filesystem
+      TokenCache.obtainTokensForNamenodes(job.getCredentials(), new Path[] { new Path(inputDirs) },


@mymeiyi
We do get credentials from namenode here
I don't know much about WALPlayer internals. Does this MR job access 2 different namenodes ?

I think JobSubmitter get a token from hdfs for yarn cluster, here we also need a token from hdfs for hbase cluster

Apache9 · 2022-01-06T12:32:44Z

I think this should be done in WALInputFormat.

Looking at the code of FileInputFormat, TokenCache.obtainTokensForNamenodes will be called in listStatus method, which will be called in getSplits method. So I think for WALInputFormat, we should also call TokenCache.obtainTokensForNamenodes in getSplits method?

Thanks.

Apache9 · 2022-01-07T09:15:06Z

So could this solve your problem?

mymeiyi · 2022-01-07T09:24:43Z

So could this solve your problem?

Yes, it works in our cluster.

Apache9 · 2022-01-07T09:34:30Z

So could this solve your problem?

Yes, it works in our cluster.

OK, good.

Apache-HBase · 2022-01-07T09:52:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	4m 16s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 55s	master passed
+1 💚	compile	0m 28s	master passed
+1 💚	shadedjars	9m 18s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 22s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 29s	the patch passed
+1 💚	compile	0m 28s	the patch passed
+1 💚	javac	0m 28s	the patch passed
+1 💚	shadedjars	9m 27s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 20s	the patch passed
		_ Other Tests _
+1 💚	unit	12m 21s	hbase-mapreduce in the patch passed.
		47m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4003
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 065ebdd1b6ed 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `5e263da`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/2/testReport/
Max. process+thread count	4165 (vs. ulimit of 30000)
modules	C: hbase-mapreduce U: hbase-mapreduce
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-01-07T09:57:38Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	7m 17s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 50s	master passed
+1 💚	compile	0m 55s	master passed
+1 💚	checkstyle	0m 22s	master passed
+1 💚	spotbugs	0m 53s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 26s	the patch passed
+1 💚	compile	0m 49s	the patch passed
+1 💚	javac	0m 49s	the patch passed
+1 💚	checkstyle	0m 19s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	22m 17s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	1m 0s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 13s	The patch does not generate ASF License warnings.
		52m 30s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4003
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux de46e0db2b9d 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `5e263da`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-mapreduce U: hbase-mapreduce
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-01-07T09:59:02Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	6m 46s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	5m 21s	master passed
+1 💚	compile	0m 32s	master passed
+1 💚	shadedjars	9m 8s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 13s	the patch passed
+1 💚	compile	0m 33s	the patch passed
+1 💚	javac	0m 33s	the patch passed
+1 💚	shadedjars	9m 5s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 21s	the patch passed
		_ Other Tests _
+1 💚	unit	15m 14s	hbase-mapreduce in the patch passed.
		53m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4003
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b9201dd61eba 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `5e263da`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/2/testReport/
Max. process+thread count	2787 (vs. ulimit of 30000)
modules	C: hbase-mapreduce U: hbase-mapreduce
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4003/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Signed-off-by: Duo Zhang <[email protected]>

Signed-off-by: Duo Zhang <[email protected]> (cherry picked from commit b68c4aa) Change-Id: Id879fda366d2063fa97c73c41d394e436b8aad35

shahrs87 reviewed Jan 5, 2022

View reviewed changes

HBASE-26646 WALPlayer should obtain token from filesystem

50e23e0

mymeiyi force-pushed the HBASE-26646 branch from ff3d7c9 to 50e23e0 Compare January 7, 2022 09:04

Apache9 approved these changes Jan 7, 2022

View reviewed changes

mymeiyi merged commit f1e263a into apache:master Jan 7, 2022

mymeiyi added a commit that referenced this pull request Jan 7, 2022

HBASE-26646 WALPlayer should obtain token from filesystem (#4003)

d25fa46

Signed-off-by: Duo Zhang <[email protected]>

mymeiyi added a commit that referenced this pull request Jan 7, 2022

HBASE-26646 WALPlayer should obtain token from filesystem (#4003)

b68c4aa

Signed-off-by: Duo Zhang <[email protected]>

mymeiyi deleted the HBASE-26646 branch July 12, 2022 04:07

jarrydlee pushed a commit to HubSpot/hbase that referenced this pull request May 4, 2023

HBASE-26646 WALPlayer should obtain token from filesystem (apache#4003)

7c384ac

Signed-off-by: Duo Zhang <[email protected]>

bbeaudreault pushed a commit to HubSpot/hbase that referenced this pull request May 4, 2023

HBASE-26646 WALPlayer should obtain token from filesystem (apache#4003)

ada7c28

Signed-off-by: Duo Zhang <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-26646 WALPlayer should obtain token from filesystem #4003

HBASE-26646 WALPlayer should obtain token from filesystem #4003

mymeiyi commented Jan 5, 2022

Apache-HBase commented Jan 5, 2022

Apache-HBase commented Jan 5, 2022

Apache-HBase commented Jan 5, 2022

shahrs87 commented Jan 5, 2022 •

edited

Loading

shahrs87 Jan 5, 2022

mymeiyi Jan 6, 2022

Apache9 commented Jan 6, 2022

Apache9 commented Jan 7, 2022

mymeiyi commented Jan 7, 2022

Apache9 commented Jan 7, 2022

Apache-HBase commented Jan 7, 2022

Apache-HBase commented Jan 7, 2022

Apache-HBase commented Jan 7, 2022

HBASE-26646 WALPlayer should obtain token from filesystem #4003

HBASE-26646 WALPlayer should obtain token from filesystem #4003

Conversation

mymeiyi commented Jan 5, 2022

Apache-HBase commented Jan 5, 2022

Apache-HBase commented Jan 5, 2022

Apache-HBase commented Jan 5, 2022

shahrs87 commented Jan 5, 2022 • edited Loading

shahrs87 Jan 5, 2022

Choose a reason for hiding this comment

mymeiyi Jan 6, 2022

Choose a reason for hiding this comment

Apache9 commented Jan 6, 2022

Apache9 commented Jan 7, 2022

mymeiyi commented Jan 7, 2022

Apache9 commented Jan 7, 2022

Apache-HBase commented Jan 7, 2022

Apache-HBase commented Jan 7, 2022

Apache-HBase commented Jan 7, 2022

shahrs87 commented Jan 5, 2022 •

edited

Loading