HBASE-28836 Parallize the file archival to improve the split times #6483

mnpoonia · 2024-11-20T09:59:08Z

Jira: HBASE-28836

mnpoonia · 2024-11-20T10:01:05Z

@d-c-manning @virajjasani @Umeshkumar9414 @Apache9 @apurtell FYI

hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java

Umeshkumar9414

LGTM

d-c-manning · 2024-11-22T22:00:05Z

hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java

+        boolean fileCleaned = fileFutureEntry.getValue().get();
+        if (!fileCleaned) {
+          LOG.warn("Couldn't archive %s into backup directory: %s"
+            .formatted(fileFutureEntry.getKey(), baseArchiveDir));


I don't know all the jdk requirements, but looks like .formatted was added only in JDK 15. If this is intended for backport, there may be earlier JDK versions to support.

Thanks for pointing to this. Yes i was using jdk 17 so missed this.

… object store (s3)

Apache-HBase · 2024-11-25T06:14:09Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 29s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	3m 18s		master passed
+1 💚	compile	3m 10s		master passed
+1 💚	checkstyle	0m 37s		master passed
+1 💚	spotbugs	1m 35s		master passed
+1 💚	spotless	0m 45s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 0s		the patch passed
+1 💚	compile	3m 14s		the patch passed
+1 💚	javac	3m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 36s		the patch passed
+1 💚	spotbugs	1m 38s		the patch passed
+1 💚	hadoopcheck	11m 37s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 43s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 11s		The patch does not generate ASF License warnings.
		38m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6483
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux c2fb7253f3e4 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `925ff4a`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	84 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-11-25T06:20:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
-0 ⚠️	yetus	0m 2s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	3m 26s		master passed
+1 💚	compile	0m 53s		master passed
+1 💚	javadoc	0m 27s		master passed
+1 💚	shadedjars	5m 36s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 54s		the patch passed
+1 💚	compile	0m 58s		the patch passed
+1 💚	javac	0m 58s		the patch passed
+1 💚	javadoc	0m 28s		the patch passed
+1 💚	shadedjars	5m 50s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
-1 ❌	unit	21m 39s	/patch-unit-hbase-server.txt	hbase-server in the patch failed.
		44m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6483
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 045cbd2cb87c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `925ff4a`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/testReport/
Max. process+thread count	2201 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

virajjasani

It looks like until this change, only MasterFileSystem#checkTempDir and DeleteTableProcedure#deleteFromFs were using parallel archival using thread pool of size defined by hbase.hfilearchiver.thread.pool.max.
With this change, we will use parallel archival for all use cases.

virajjasani · 2024-12-03T06:12:22Z

@Apache9 @apurtell any reviews from your side before we merge it?
Keeping this open for at least 24 hr.

virajjasani

+1

virajjasani · 2024-12-08T13:37:07Z

@mnpoonia could you also create branch-2 backport PR so that after clean build results, the changes can be ported to all active branches?

mnpoonia · 2024-12-09T16:24:12Z

@virajjasani Sure. It can be applied to all active branches.

virajjasani · 2024-12-09T16:30:55Z

Sounds good, let me verify on branch-2 and then will merge it today.

…6483) Signed-off-by: Viraj Jasani <[email protected]> Signed-off-by: David Manning <[email protected]> Signed-off-by: Umesh Kumar <[email protected]>

mnpoonia · 2024-12-10T04:15:49Z

Thank you @virajjasani

virajjasani · 2024-12-10T18:47:48Z

Thank you @mnpoonia for this nice improvement!

…pache#6483) Signed-off-by: Viraj Jasani <[email protected]> Signed-off-by: David Manning <[email protected]> Signed-off-by: Umesh Kumar <[email protected]>

mnpoonia mentioned this pull request Nov 20, 2024

HBASE-28836 Parallelize the file archival to improve the split times #6243

Closed

This comment has been minimized.

Sign in to view

Umeshkumar9414 reviewed Nov 20, 2024

View reviewed changes

hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java Show resolved Hide resolved

mnpoonia force-pushed the HBASE-28836 branch from d3703bb to f86aa9b Compare November 21, 2024 05:11

This comment has been minimized.

Sign in to view

Umeshkumar9414 approved these changes Nov 21, 2024

View reviewed changes

virajjasani self-requested a review November 21, 2024 17:07

d-c-manning reviewed Nov 22, 2024

View reviewed changes

HBASE-28836 Parallize the file archival to improve the split times in…

925ff4a

… object store (s3)

mnpoonia force-pushed the HBASE-28836 branch from f86aa9b to 925ff4a Compare November 25, 2024 05:31

d-c-manning approved these changes Nov 25, 2024

View reviewed changes

virajjasani reviewed Dec 3, 2024

View reviewed changes

virajjasani requested review from apurtell and Apache9 December 3, 2024 06:11

virajjasani approved these changes Dec 3, 2024

View reviewed changes

virajjasani merged commit 262c5bb into apache:master Dec 9, 2024
1 check failed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-28836 Parallize the file archival to improve the split times #6483

HBASE-28836 Parallize the file archival to improve the split times #6483

mnpoonia commented Nov 20, 2024 •

edited by virajjasani

Loading

mnpoonia commented Nov 20, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Umeshkumar9414 left a comment

d-c-manning Nov 22, 2024

mnpoonia Nov 25, 2024

Apache-HBase commented Nov 25, 2024

Apache-HBase commented Nov 25, 2024

virajjasani left a comment

virajjasani commented Dec 3, 2024 •

edited

Loading

virajjasani left a comment

virajjasani commented Dec 8, 2024

mnpoonia commented Dec 9, 2024

virajjasani commented Dec 9, 2024

mnpoonia commented Dec 10, 2024

virajjasani commented Dec 10, 2024

HBASE-28836 Parallize the file archival to improve the split times #6483

HBASE-28836 Parallize the file archival to improve the split times #6483

Conversation

mnpoonia commented Nov 20, 2024 • edited by virajjasani Loading

mnpoonia commented Nov 20, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Umeshkumar9414 left a comment

Choose a reason for hiding this comment

d-c-manning Nov 22, 2024

Choose a reason for hiding this comment

mnpoonia Nov 25, 2024

Choose a reason for hiding this comment

Apache-HBase commented Nov 25, 2024

Apache-HBase commented Nov 25, 2024

virajjasani left a comment

Choose a reason for hiding this comment

virajjasani commented Dec 3, 2024 • edited Loading

virajjasani left a comment

Choose a reason for hiding this comment

virajjasani commented Dec 8, 2024

mnpoonia commented Dec 9, 2024

virajjasani commented Dec 9, 2024

mnpoonia commented Dec 10, 2024

virajjasani commented Dec 10, 2024

mnpoonia commented Nov 20, 2024 •

edited by virajjasani

Loading

virajjasani commented Dec 3, 2024 •

edited

Loading