HBASE-28702 TestBackupMerge fails 100% of times on flaky dashboard #6078

2005hithlj · 2024-07-14T08:34:28Z

https://issues.apache.org/jira/browse/HBASE-28702

Apache-HBase · 2024-07-14T09:18:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 27s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	3m 7s		master passed
+1 💚	compile	0m 28s		master passed
+1 💚	checkstyle	0m 9s		master passed
+1 💚	spotbugs	0m 30s		master passed
+1 💚	spotless	0m 42s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 49s		the patch passed
+1 💚	compile	0m 27s		the patch passed
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 9s		the patch passed
+1 💚	spotbugs	0m 33s		the patch passed
+1 💚	hadoopcheck	10m 49s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 40s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 9s		The patch does not generate ASF License warnings.
		27m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6078/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6078
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux c5b422fcf41d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `e7f7d05`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	83 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6078/1/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-07-14T09:21:44Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 27s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	3m 21s		master passed
+1 💚	compile	0m 18s		master passed
+1 💚	javadoc	0m 15s		master passed
+1 💚	shadedjars	5m 53s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 10s		the patch passed
+1 💚	compile	0m 19s		the patch passed
+1 💚	javac	0m 19s		the patch passed
+1 💚	javadoc	0m 14s		the patch passed
+1 💚	shadedjars	5m 47s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	10m 12s		hbase-backup in the patch passed.
		30m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6078/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6078
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 7906e8be2036 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `e7f7d05`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6078/1/testReport/
Max. process+thread count	3821 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6078/1/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2024-07-14T15:28:13Z

This PR changes what we want to test here...

@bbeaudreault I think this is a very critical problem which should be considered in the backup&restore code. After HBASE-28625 gets in, if we copy snapshot between different file systems without specifying the ignore checksum flag, the copy will fail because data checksum highly depend on the file sytem implementation so the data checksum between different file systems can not be compared...

We should have a way to deal with this, at least we should give user the ability to specify this flag if they want to backup to a different file system like S3...

bbeaudreault · 2024-07-14T17:07:39Z

I'm not sure I follow @Apache9. Are you saying this PR should fix production code instead of test code?

Apache9 · 2024-07-15T02:25:04Z

I'm not sure I follow @Apache9. Are you saying this PR should fix production code instead of test code?

Yes, exactly. The test is for testing copying between different file systems, but the fix here just make it copy to the same file system...

2005hithlj · 2024-07-15T07:44:48Z

@Apache9 @bbeaudreault I get it, this PR is not just about fixing the test code, should fix the production code completely. And I will try a new commit to address this.

Apache9 · 2024-07-16T13:54:33Z

For me I think the current PR can work at least, @bbeaudreault do you think this is an acceptable way to fix the checksum mismatch problem?

Thanks.

2005hithlj · 2024-07-16T13:58:07Z

@Apache9 @bbeaudreault I'm not very familiar with the hbase-backup module. Please help review it ? Thanks.

Apache-HBase · 2024-07-16T14:33:55Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 27s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	3m 4s		master passed
+1 💚	compile	0m 28s		master passed
+1 💚	checkstyle	0m 9s		master passed
+1 💚	spotbugs	0m 28s		master passed
+1 💚	spotless	0m 43s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 51s		the patch passed
+1 💚	compile	0m 27s		the patch passed
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 9s		the patch passed
+1 💚	spotbugs	0m 34s		the patch passed
+1 💚	hadoopcheck	10m 54s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 42s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 10s		The patch does not generate ASF License warnings.
		27m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6078/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6078
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 07c7cf6b1f94 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `82abb6d`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	82 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6078/2/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-07-16T14:37:56Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 13s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	3m 5s		master passed
+1 💚	compile	0m 21s		master passed
+1 💚	javadoc	0m 16s		master passed
+1 💚	shadedjars	5m 24s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 57s		the patch passed
+1 💚	compile	0m 21s		the patch passed
+1 💚	javac	0m 21s		the patch passed
+1 💚	javadoc	0m 15s		the patch passed
+1 💚	shadedjars	6m 35s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	11m 19s		hbase-backup in the patch passed.
		31m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6078/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6078
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux d1659dba61b3 5.4.0-182-generic #202-Ubuntu SMP Fri Apr 26 12:29:36 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `82abb6d`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6078/2/testReport/
Max. process+thread count	3154 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6078/2/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

bbeaudreault

I suppose the addition of the flag can work.

Should we improve the comments and/or error message to make it clear that checksum may not match across filesystems? Otherwise user may not know how to handle the failure.

2005hithlj · 2024-07-17T04:25:28Z

@bbeaudreault Thanks for your review. I have revised it. Could you take a look? Thanks.

Apache-HBase · 2024-07-17T05:09:03Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 28s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 10s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 3s		master passed
+1 💚	compile	1m 3s		master passed
+1 💚	checkstyle	0m 21s		master passed
+1 💚	spotbugs	0m 58s		master passed
+1 💚	spotless	0m 45s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 51s		the patch passed
+1 💚	compile	1m 6s		the patch passed
+1 💚	javac	1m 6s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 12s	/results-checkstyle-hbase-mapreduce.txt	hbase-mapreduce: The patch generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9)
+1 💚	spotbugs	1m 10s		the patch passed
+1 💚	hadoopcheck	10m 54s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 42s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 16s		The patch does not generate ASF License warnings.
		31m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6078/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6078
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 06a0fb7879e0 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `ed5fbe8`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	84 (vs. ulimit of 30000)
modules	C: hbase-mapreduce hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6078/3/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-07-17T05:28:05Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 26s		Docker mode activated.
-0 ⚠️	yetus	0m 2s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 10s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 8s		master passed
+1 💚	compile	0m 38s		master passed
+1 💚	javadoc	0m 28s		master passed
+1 💚	shadedjars	5m 42s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 58s		the patch passed
+1 💚	compile	0m 38s		the patch passed
+1 💚	javac	0m 38s		the patch passed
+1 💚	javadoc	0m 26s		the patch passed
+1 💚	shadedjars	5m 38s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	18m 15s		hbase-mapreduce in the patch passed.
+1 💚	unit	10m 16s		hbase-backup in the patch passed.
		50m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6078/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6078
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 449043ca37ad 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `ed5fbe8`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6078/3/testReport/
Max. process+thread count	3858 (vs. ulimit of 30000)
modules	C: hbase-mapreduce hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6078/3/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

2005hithlj · 2024-07-19T05:55:07Z

ping @bbeaudreault @Apache9

bbeaudreault

LGTM Thanks for fixing

…6078) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Bryan Beaudreault <[email protected]>

…pache#6078) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Bryan Beaudreault <[email protected]>

2005hithlj requested a review from Apache9 July 14, 2024 08:34

2005hithlj force-pushed the HBASE-28702 branch from e7f7d05 to 82abb6d Compare July 16, 2024 13:51

bbeaudreault reviewed Jul 16, 2024

View reviewed changes

HBASE-28702 TestBackupMerge fails 100% of times on flaky dashboard

ed5fbe8

2005hithlj force-pushed the HBASE-28702 branch from 82abb6d to ed5fbe8 Compare July 17, 2024 04:21

bbeaudreault approved these changes Jul 19, 2024

View reviewed changes

Apache9 approved these changes Jul 19, 2024

View reviewed changes

2005hithlj merged commit 171ecf2 into apache:master Jul 19, 2024
1 check passed

2005hithlj added a commit that referenced this pull request Jul 20, 2024

HBASE-28702 TestBackupMerge fails 100% of times on flaky dashboard (#…

10e7105

…6078) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Bryan Beaudreault <[email protected]>

2005hithlj added a commit to 2005hithlj/hbase that referenced this pull request Jul 20, 2024

HBASE-28702 TestBackupMerge fails 100% of times on flaky dashboard (a…

b18c3d4

…pache#6078) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Bryan Beaudreault <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-28702 TestBackupMerge fails 100% of times on flaky dashboard #6078

HBASE-28702 TestBackupMerge fails 100% of times on flaky dashboard #6078

2005hithlj commented Jul 14, 2024

Apache-HBase commented Jul 14, 2024

Apache-HBase commented Jul 14, 2024

Apache9 commented Jul 14, 2024

bbeaudreault commented Jul 14, 2024

Apache9 commented Jul 15, 2024

2005hithlj commented Jul 15, 2024

Apache9 commented Jul 16, 2024

2005hithlj commented Jul 16, 2024

Apache-HBase commented Jul 16, 2024

Apache-HBase commented Jul 16, 2024

bbeaudreault left a comment

2005hithlj commented Jul 17, 2024

Apache-HBase commented Jul 17, 2024

Apache-HBase commented Jul 17, 2024

2005hithlj commented Jul 19, 2024

bbeaudreault left a comment

HBASE-28702 TestBackupMerge fails 100% of times on flaky dashboard #6078

HBASE-28702 TestBackupMerge fails 100% of times on flaky dashboard #6078

Conversation

2005hithlj commented Jul 14, 2024

Apache-HBase commented Jul 14, 2024

Apache-HBase commented Jul 14, 2024

Apache9 commented Jul 14, 2024

bbeaudreault commented Jul 14, 2024

Apache9 commented Jul 15, 2024

2005hithlj commented Jul 15, 2024

Apache9 commented Jul 16, 2024

2005hithlj commented Jul 16, 2024

Apache-HBase commented Jul 16, 2024

Apache-HBase commented Jul 16, 2024

bbeaudreault left a comment

Choose a reason for hiding this comment

2005hithlj commented Jul 17, 2024

Apache-HBase commented Jul 17, 2024

Apache-HBase commented Jul 17, 2024

2005hithlj commented Jul 19, 2024

bbeaudreault left a comment

Choose a reason for hiding this comment