HBASE-28042 Snapshot corruptions due to non-atomic rename within same filesystem #5369

virajjasani · 2023-08-23T21:50:09Z

Co-authored-by: Ujjawal <[email protected]>

… filesystem

virajjasani · 2023-08-23T21:50:55Z

Apache-HBase · 2023-08-23T22:31:52Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 52s	master passed
+1 💚	compile	2m 28s	master passed
+1 💚	checkstyle	0m 38s	master passed
+1 💚	spotless	0m 42s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 29s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 34s	the patch passed
+1 💚	compile	2m 26s	the patch passed
+1 💚	javac	2m 26s	the patch passed
+1 💚	checkstyle	0m 35s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	8m 59s	Patch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 38s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		31m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5369
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 6dcba3f06eef 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `89ca7f4`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	87 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-08-24T01:56:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 25s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 5s	master passed
+1 💚	compile	0m 45s	master passed
+1 💚	shadedjars	5m 9s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 44s	the patch passed
+1 💚	compile	0m 45s	the patch passed
+1 💚	javac	0m 45s	the patch passed
+1 💚	shadedjars	5m 6s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
-1 ❌	unit	212m 55s	hbase-server in the patch failed.
		235m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5369
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 2f468fe5350a 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `89ca7f4`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/1/testReport/
Max. process+thread count	4571 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-08-24T02:06:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 33s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 24s	master passed
+1 💚	compile	0m 40s	master passed
+1 💚	shadedjars	4m 34s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 20s	the patch passed
+1 💚	compile	0m 39s	the patch passed
+1 💚	javac	0m 39s	the patch passed
+1 💚	shadedjars	4m 31s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
-1 ❌	unit	225m 28s	hbase-server in the patch failed.
		246m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5369
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 52c4c82818ee 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `89ca7f4`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/1/testReport/
Max. process+thread count	3963 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-08-24T02:39:12Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 35s	master passed
+1 💚	compile	2m 24s	master passed
+1 💚	checkstyle	0m 35s	master passed
+1 💚	spotless	0m 40s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 26s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 40s	the patch passed
+1 💚	compile	2m 29s	the patch passed
+1 💚	javac	2m 29s	the patch passed
+1 💚	checkstyle	0m 34s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	9m 3s	Patch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 39s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		31m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5369
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux e05bdca1ef0a 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `89ca7f4`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	77 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/2/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

mnpoonia

Thanks @ujjawal4046 for looking into this.
+1 non binding.

virajjasani · 2023-08-24T05:35:09Z

TestSnapshotScannerHDFSAclController needs some work

virajjasani · 2023-08-24T05:49:56Z

@mymeiyi could you please take a look at TestSnapshotScannerHDFSAclController? it seems that after atomic rename of snapshot dirs, global user is unable to access snapshot, in case global permission is set before table and snapshot creations.

For instance, for test testGrantGlobal1, this patch would allow global user to access snapshot:

diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
index d79e3f3081..a5a194f797 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
@@ -158,6 +158,7 @@ public class TestSnapshotScannerHDFSAclController {
 
     TestHDFSAclHelper.createTableAndPut(TEST_UTIL, table);
     snapshotAndWait(snapshot1, table);
+    snapshotAndWait(snapshot2, table);
     // grant G(R)
     SecureTestUtil.grantGlobal(TEST_UTIL, grantUserName, READ);
     TestHDFSAclHelper.canUserScanSnapshot(TEST_UTIL, grantUser, snapshot1, 6);
@@ -174,8 +175,6 @@ public class TestSnapshotScannerHDFSAclController {
     // grant G(R)
     SecureTestUtil.grantGlobal(TEST_UTIL, grantUserName, READ);
     TestHDFSAclHelper.canUserScanSnapshot(TEST_UTIL, grantUser, snapshot1, 6);
-    // take a snapshot and ACLs are inherited automatically
-    snapshotAndWait(snapshot2, table);
     TestHDFSAclHelper.canUserScanSnapshot(TEST_UTIL, grantUser, snapshot2, 6);
     assertTrue(hasUserGlobalHdfsAcl(aclTable, grantUserName));
     deleteTable(table);

virajjasani · 2023-08-24T05:57:13Z

Similarly, on testModifyTable1, only global user is not able to scan the snapshot whereas table and namespace users are able to (though this appears only when all tests from TestSnapshotScannerHDFSAclController are run, not when testModifyTable1 is run individually)

Apache-HBase · 2023-08-24T06:49:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 30s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 4s	master passed
+1 💚	compile	2m 35s	master passed
+1 💚	checkstyle	0m 37s	master passed
+1 💚	spotless	0m 44s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 31s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 45s	the patch passed
+1 💚	compile	2m 30s	the patch passed
+1 💚	javac	2m 30s	the patch passed
+1 💚	checkstyle	0m 31s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	10m 14s	Patch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚	spotless	0m 40s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 37s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		33m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5369
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux d85cea13c4dc 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `89ca7f4`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/3/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-08-24T10:09:13Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 25s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 45s	master passed
+1 💚	compile	0m 47s	master passed
+1 💚	shadedjars	5m 6s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 22s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 40s	the patch passed
+1 💚	compile	0m 45s	the patch passed
+1 💚	javac	0m 45s	the patch passed
+1 💚	shadedjars	5m 6s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 22s	the patch passed
		_ Other Tests _
-1 ❌	unit	211m 10s	hbase-server in the patch failed.
		233m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5369
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 5be105cb0400 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `89ca7f4`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/3/testReport/
Max. process+thread count	4709 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/3/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-08-24T10:38:57Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 34s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 22s	master passed
+1 💚	compile	0m 39s	master passed
+1 💚	shadedjars	4m 35s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 18s	the patch passed
+1 💚	compile	0m 40s	the patch passed
+1 💚	javac	0m 40s	the patch passed
+1 💚	shadedjars	4m 32s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
-1 ❌	unit	242m 23s	hbase-server in the patch failed.
		263m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5369
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b9a185efe683 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `89ca7f4`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/3/testReport/
Max. process+thread count	3976 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/3/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

wchevreuil · 2023-08-24T13:13:02Z

.../test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java

@@ -158,6 +158,7 @@ public void testGrantGlobal1() throws Exception {

    TestHDFSAclHelper.createTableAndPut(TEST_UTIL, table);
    snapshotAndWait(snapshot1, table);
+    snapshotAndWait(snapshot2, table);


Does this test really need this change, or is this part of the PR by mistake?

it actually does, i am still trying to understand why the order of granting permissions to global user matter in the order. e.g. before table creation, granting global user permission allows user to scan snapshot but after table creation if we grant it, it doesn't work.
perhaps some bug in coproc SnapshotScannerHDFSAclController

wchevreuil · 2023-08-24T13:24:53Z

hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDescriptionUtils.java

+    // check scheme, e.g. file, hdfs
+    if (
+      workingURI.getScheme() == null
+        && (rootURI.getScheme() != null && !rootURI.getScheme().equalsIgnoreCase("file"))


Why we need the extra check for the "file" URI? Wouldn't it be enough to return "true" here if workingURI scheme is null and rootURI is not?

with "file", we could have ignored local fs, but you are right that we don't need such explicit check, let me make this change, thanks

virajjasani · 2023-08-24T20:36:18Z

the ordering issue with TestSnapshotScannerHDFSAclController that now i encounter after this change, seems to be reflecting other hidden issue, at least core changes are not supposed to impact snapshot functionality. we now perform atomic rename.

wchevreuil

LGTM, +1.

Just as an extra note, "fs.rename" may not always be atomic (think of some object store file systems), and the ultimate solution for this problem would be something similar to what SFT did for the store files rename, but that could be worked on a different jira.

Apache-HBase · 2023-08-24T20:58:39Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 34s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 13s	master passed
+1 💚	compile	0m 39s	master passed
+1 💚	shadedjars	4m 32s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 17s	the patch passed
+1 💚	compile	0m 39s	the patch passed
+1 💚	javac	0m 39s	the patch passed
+1 💚	shadedjars	4m 32s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
-1 ❌	unit	10m 8s	hbase-server in the patch failed.
		28m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5369
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c313297f7ac0 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `89ca7f4`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/5/testReport/
Max. process+thread count	1377 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/5/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-08-24T21:03:30Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 43s	master passed
+1 💚	compile	2m 35s	master passed
+1 💚	checkstyle	0m 33s	master passed
+1 💚	spotless	0m 41s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 42s	the patch passed
+1 💚	compile	2m 31s	the patch passed
+1 💚	javac	2m 31s	the patch passed
+1 💚	checkstyle	0m 34s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	10m 10s	Patch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 37s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 8s	The patch does not generate ASF License warnings.
		32m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5369
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 36e1b589bf1c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `89ca7f4`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/5/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

virajjasani · 2023-08-24T21:04:29Z

the ultimate solution for this problem would be something similar to what SFT did for the store files rename, but that could be worked on a different jira.

agree to using SFT 👍

Apache-HBase · 2023-08-24T21:41:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 27s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 42s	master passed
+1 💚	compile	2m 29s	master passed
+1 💚	checkstyle	0m 33s	master passed
+1 💚	spotless	0m 40s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 27s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 44s	the patch passed
+1 💚	compile	2m 31s	the patch passed
+1 💚	javac	2m 31s	the patch passed
+1 💚	checkstyle	0m 34s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	10m 7s	Patch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 35s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 7s	The patch does not generate ASF License warnings.
		32m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/6/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5369
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 3bd8709c3175 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `89ca7f4`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/6/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-08-25T01:16:07Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 33s	master passed
+1 💚	compile	0m 49s	master passed
+1 💚	shadedjars	4m 37s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 34s	the patch passed
+1 💚	compile	0m 47s	the patch passed
+1 💚	javac	0m 47s	the patch passed
+1 💚	shadedjars	4m 42s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	225m 23s	hbase-server in the patch passed.
		247m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5369
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 780a0502d7e8 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `89ca7f4`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/6/testReport/
Max. process+thread count	4460 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/6/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-08-25T01:18:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 15s	master passed
+1 💚	compile	0m 39s	master passed
+1 💚	shadedjars	4m 32s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 16s	the patch passed
+1 💚	compile	0m 40s	the patch passed
+1 💚	javac	0m 40s	the patch passed
+1 💚	shadedjars	4m 35s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
+1 💚	unit	229m 6s	hbase-server in the patch passed.
		250m 12s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5369
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 6ee3534119e2 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `89ca7f4`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/6/testReport/
Max. process+thread count	3959 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/6/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-08-25T01:50:02Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 32s	master passed
+1 💚	compile	2m 27s	master passed
+1 💚	checkstyle	0m 35s	master passed
+1 💚	spotless	0m 41s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 29s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 34s	the patch passed
+1 💚	compile	2m 26s	the patch passed
+1 💚	javac	2m 26s	the patch passed
+1 💚	checkstyle	0m 35s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	8m 54s	Patch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 37s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 8s	The patch does not generate ASF License warnings.
		30m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/7/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5369
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 5c0dbf565f7c 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `89ca7f4`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/7/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-08-25T05:23:47Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 40s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 27s	master passed
+1 💚	compile	0m 41s	master passed
+1 💚	shadedjars	4m 35s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 18s	the patch passed
+1 💚	compile	0m 40s	the patch passed
+1 💚	javac	0m 40s	the patch passed
+1 💚	shadedjars	4m 30s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	223m 47s	hbase-server in the patch passed.
		244m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5369
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 5e2c000b3594 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `89ca7f4`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/7/testReport/
Max. process+thread count	3960 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/7/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-08-25T05:28:59Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 41s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 37s	master passed
+1 💚	compile	0m 48s	master passed
+1 💚	shadedjars	4m 41s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 39s	the patch passed
+1 💚	compile	0m 47s	the patch passed
+1 💚	javac	0m 47s	the patch passed
+1 💚	shadedjars	4m 39s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	227m 34s	hbase-server in the patch passed.
		249m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5369
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 9885615f1cce 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `89ca7f4`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/7/testReport/
Max. process+thread count	4461 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5369/7/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

virajjasani · 2023-08-28T02:27:37Z

Thanks for the reviews @wchevreuil @mnpoonia @ujjawal4046 @Abhey !!

… filesystem (#5369) Co-authored-by: Ujjawal <[email protected]> Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Abhey Rana <[email protected]> Signed-off-by: Ujjawal <[email protected]> Signed-off-by: Aman Poonia <[email protected]>

… filesystem (#5372) (#5369) Co-Authored-By: Ujjawal <[email protected]> Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Abhey Rana <[email protected]> Signed-off-by: Ujjawal <[email protected]> Signed-off-by: Aman Poonia <[email protected]>

… filesystem (#5372) (#5369) Co-Authored-By: ujjawal4046 <[email protected]> Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Abhey Rana <[email protected]> Signed-off-by: Ujjawal <[email protected]> Signed-off-by: Aman Poonia <[email protected]>

… filesystem (apache#5369) Co-authored-by: Ujjawal <[email protected]> Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Abhey Rana <[email protected]> Signed-off-by: Ujjawal <[email protected]> Signed-off-by: Aman Poonia <[email protected]>

… filesystem (apache#5372) (apache#5369) Co-Authored-By: ujjawal4046 <[email protected]> Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Abhey Rana <[email protected]> Signed-off-by: Ujjawal <[email protected]> Signed-off-by: Aman Poonia <[email protected]> (cherry picked from commit 61250ad) Change-Id: I8f93a9b89df3c1195d427ac1b67001708e716b49

HBASE-28042 Snapshot corruptions due to non-atomic rename within same…

2898bc3

… filesystem

virajjasani requested a review from Apache9 August 23, 2023 21:54

virajjasani requested review from taklwu and huaxiangsun August 23, 2023 22:48

Abhey approved these changes Aug 24, 2023

View reviewed changes

ujjawal4046 approved these changes Aug 24, 2023

View reviewed changes

mnpoonia approved these changes Aug 24, 2023

View reviewed changes

virajjasani requested a review from mymeiyi August 24, 2023 05:44

test addendum

8ca3b0b

wchevreuil reviewed Aug 24, 2023

View reviewed changes

virajjasani added 2 commits August 24, 2023 11:43

addendum

880ffd7

addendum - test order changes for read/write perm grant

0603f20

wchevreuil approved these changes Aug 24, 2023

View reviewed changes

addendum

ce1b14e

virajjasani mentioned this pull request Aug 24, 2023

HBASE-28042 Snapshot corruptions due to non-atomic rename within same filesystem #5372

Merged

virajjasani merged commit 5527dd9 into apache:master Aug 28, 2023

virajjasani mentioned this pull request Sep 26, 2023

HBASE-28081 Snapshot working dir does not retain ACLs after snapshot commit phase #5437

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-28042 Snapshot corruptions due to non-atomic rename within same filesystem #5369

HBASE-28042 Snapshot corruptions due to non-atomic rename within same filesystem #5369

virajjasani commented Aug 23, 2023 •

edited

Loading

virajjasani commented Aug 23, 2023

Apache-HBase commented Aug 23, 2023

Apache-HBase commented Aug 24, 2023

Apache-HBase commented Aug 24, 2023

Apache-HBase commented Aug 24, 2023

mnpoonia left a comment

virajjasani commented Aug 24, 2023

virajjasani commented Aug 24, 2023

virajjasani commented Aug 24, 2023 •

edited

Loading

Apache-HBase commented Aug 24, 2023

Apache-HBase commented Aug 24, 2023

Apache-HBase commented Aug 24, 2023

wchevreuil Aug 24, 2023

virajjasani Aug 24, 2023

wchevreuil Aug 24, 2023

wchevreuil Aug 24, 2023

virajjasani Aug 24, 2023

virajjasani commented Aug 24, 2023

wchevreuil left a comment

Apache-HBase commented Aug 24, 2023

Apache-HBase commented Aug 24, 2023

virajjasani commented Aug 24, 2023

Apache-HBase commented Aug 24, 2023

Apache-HBase commented Aug 25, 2023

Apache-HBase commented Aug 25, 2023

Apache-HBase commented Aug 25, 2023

Apache-HBase commented Aug 25, 2023

Apache-HBase commented Aug 25, 2023

virajjasani commented Aug 28, 2023

HBASE-28042 Snapshot corruptions due to non-atomic rename within same filesystem #5369

HBASE-28042 Snapshot corruptions due to non-atomic rename within same filesystem #5369

Conversation

virajjasani commented Aug 23, 2023 • edited Loading

virajjasani commented Aug 23, 2023

Apache-HBase commented Aug 23, 2023

Apache-HBase commented Aug 24, 2023

Apache-HBase commented Aug 24, 2023

Apache-HBase commented Aug 24, 2023

mnpoonia left a comment

Choose a reason for hiding this comment

virajjasani commented Aug 24, 2023

virajjasani commented Aug 24, 2023

virajjasani commented Aug 24, 2023 • edited Loading

Apache-HBase commented Aug 24, 2023

Apache-HBase commented Aug 24, 2023

Apache-HBase commented Aug 24, 2023

wchevreuil Aug 24, 2023

Choose a reason for hiding this comment

virajjasani Aug 24, 2023

Choose a reason for hiding this comment

wchevreuil Aug 24, 2023

Choose a reason for hiding this comment

wchevreuil Aug 24, 2023

Choose a reason for hiding this comment

virajjasani Aug 24, 2023

Choose a reason for hiding this comment

virajjasani commented Aug 24, 2023

wchevreuil left a comment

Choose a reason for hiding this comment

Apache-HBase commented Aug 24, 2023

Apache-HBase commented Aug 24, 2023

virajjasani commented Aug 24, 2023

Apache-HBase commented Aug 24, 2023

Apache-HBase commented Aug 25, 2023

Apache-HBase commented Aug 25, 2023

Apache-HBase commented Aug 25, 2023

Apache-HBase commented Aug 25, 2023

Apache-HBase commented Aug 25, 2023

virajjasani commented Aug 28, 2023

virajjasani commented Aug 23, 2023 •

edited

Loading

virajjasani commented Aug 24, 2023 •

edited

Loading