[ADDENDUM] HBASE-28458 BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached (#5777) #5791

wchevreuil · 2024-04-02T18:55:55Z

While backporting original PR to branch-2, I still got some failures for TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning. I believe I was using the wrong offset in BucketCache.notifyFileCachingCompleted, so if the eviction happens while in the loop of notifyFileCachingCompleted, we fail to acknowledge the eviction and still consider the file fully cached.

…orrectly consider a file fully cached (apache#5777) Change-Id: I681581b91000cf1f84253ad66f7195776793b364

Apache-HBase · 2024-04-02T19:42:00Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 44s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 58s	master passed
+1 💚	compile	3m 23s	master passed
+1 💚	checkstyle	0m 51s	master passed
+1 💚	spotless	0m 53s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 56s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 37s	the patch passed
+1 💚	compile	3m 16s	the patch passed
+1 💚	javac	3m 16s	the patch passed
+1 💚	checkstyle	0m 40s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	6m 28s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 55s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 12s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		36m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5791
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux cb08b37a27d0 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `90639d7`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-04-02T22:57:56Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 27s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 4s	master passed
+1 💚	compile	0m 52s	master passed
+1 💚	shadedjars	5m 29s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 50s	the patch passed
+1 💚	compile	0m 52s	the patch passed
+1 💚	javac	0m 52s	the patch passed
+1 💚	shadedjars	5m 28s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
-1 ❌	unit	207m 30s	hbase-server in the patch failed.
		232m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#5791
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 6f90fd44c565 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `90639d7`
Default Java	Eclipse Adoptium-17.0.10+7
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/1/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/1/testReport/
Max. process+thread count	4480 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-04-03T01:01:41Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 47s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 10s	master passed
+1 💚	compile	0m 51s	master passed
+1 💚	shadedjars	6m 19s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 27s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 26s	the patch passed
+1 💚	compile	0m 56s	the patch passed
+1 💚	javac	0m 56s	the patch passed
+1 💚	shadedjars	6m 12s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	the patch passed
		_ Other Tests _
-1 ❌	unit	327m 57s	hbase-server in the patch failed.
		355m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5791
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 1ab86dd474b4 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `90639d7`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/1/testReport/
Max. process+thread count	4644 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-04-03T01:09:57Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 14s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 44s	master passed
+1 💚	compile	0m 57s	master passed
+1 💚	shadedjars	8m 13s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 50s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 14s	the patch passed
+1 💚	compile	1m 33s	the patch passed
+1 💚	javac	1m 33s	the patch passed
+1 💚	shadedjars	8m 33s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 45s	the patch passed
		_ Other Tests _
-1 ❌	unit	329m 24s	hbase-server in the patch failed.
		364m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5791
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 860d06905acc 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `90639d7`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/1/testReport/
Max. process+thread count	4590 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Change-Id: Ifcd7fe5595cbf0d54c597c5ec72b504cc9ebbfae

Apache-HBase · 2024-04-03T15:34:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 5s	master passed
+1 💚	compile	2m 28s	master passed
+1 💚	checkstyle	0m 37s	master passed
+1 💚	spotless	0m 45s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 33s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 49s	the patch passed
+1 💚	compile	2m 27s	the patch passed
+1 💚	javac	2m 27s	the patch passed
+1 💚	checkstyle	0m 32s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	4m 56s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 42s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 37s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		28m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5791
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux e7c2d6cbd788 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `28c1e3b`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	82 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/2/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-04-03T18:56:06Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 29s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 4s	master passed
+1 💚	compile	0m 50s	master passed
+1 💚	shadedjars	5m 24s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 45s	the patch passed
+1 💚	compile	0m 51s	the patch passed
+1 💚	javac	0m 51s	the patch passed
+1 💚	shadedjars	5m 24s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	206m 18s	hbase-server in the patch passed.
		230m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#5791
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 4211bf8d8283 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `28c1e3b`
Default Java	Eclipse Adoptium-17.0.10+7
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/2/testReport/
Max. process+thread count	5206 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-04-03T19:16:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 14s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 1s	master passed
+1 💚	compile	0m 50s	master passed
+1 💚	shadedjars	5m 15s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 28s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 51s	the patch passed
+1 💚	compile	0m 51s	the patch passed
+1 💚	javac	0m 51s	the patch passed
+1 💚	shadedjars	5m 16s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	the patch passed
		_ Other Tests _
+1 💚	unit	227m 3s	hbase-server in the patch passed.
		250m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5791
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b900ccbbbd53 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `28c1e3b`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/2/testReport/
Max. process+thread count	4718 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-04-03T19:24:05Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 37s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 41s	master passed
+1 💚	compile	0m 42s	master passed
+1 💚	shadedjars	5m 10s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 27s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 28s	the patch passed
+1 💚	compile	0m 40s	the patch passed
+1 💚	javac	0m 40s	the patch passed
+1 💚	shadedjars	5m 9s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	235m 33s	hbase-server in the patch passed.
		258m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5791
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 406b3a25c01c 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `28c1e3b`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/2/testReport/
Max. process+thread count	5396 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5791/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…orrectly consider a file fully cached (#5777) (#5791) Signed-off-by: Peter Somogyi <[email protected]>

…orrectly consider a file fully cached (#5777) (#5791) Signed-off-by: Peter Somogyi <[email protected]> (cherry picked from commit d7566ab)

…hingCompleted may incorrectly consider a file fully cached (apache#5777) (apache#5791) Signed-off-by: Peter Somogyi <[email protected]> (cherry picked from commit d7566ab) Change-Id: I27599d2c82cf2dea074bba8caa1e6ba5b14b764a

[ADDENDUM] HBASE-28458 BucketCache.notifyFileCachingCompleted may inc…

ad123d3

…orrectly consider a file fully cached (apache#5777) Change-Id: I681581b91000cf1f84253ad66f7195776793b364

wchevreuil requested review from bbeaudreault, Apache9 and petersomogyi April 2, 2024 18:56

anpother UT fix

d5ab023

Change-Id: Ifcd7fe5595cbf0d54c597c5ec72b504cc9ebbfae

petersomogyi approved these changes Apr 4, 2024

View reviewed changes

wchevreuil merged commit aea7e7c into apache:master Apr 5, 2024
1 check passed

wchevreuil added a commit that referenced this pull request Apr 5, 2024

[ADDENDUM] HBASE-28458 BucketCache.notifyFileCachingCompleted may inc…

d7566ab

…orrectly consider a file fully cached (#5777) (#5791) Signed-off-by: Peter Somogyi <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ADDENDUM] HBASE-28458 BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached (#5777) #5791

[ADDENDUM] HBASE-28458 BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached (#5777) #5791

wchevreuil commented Apr 2, 2024

Apache-HBase commented Apr 2, 2024

Apache-HBase commented Apr 2, 2024

Apache-HBase commented Apr 3, 2024

Apache-HBase commented Apr 3, 2024

Apache-HBase commented Apr 3, 2024

Apache-HBase commented Apr 3, 2024

Apache-HBase commented Apr 3, 2024

Apache-HBase commented Apr 3, 2024

[ADDENDUM] HBASE-28458 BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached (#5777) #5791

[ADDENDUM] HBASE-28458 BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached (#5777) #5791

Conversation

wchevreuil commented Apr 2, 2024

Apache-HBase commented Apr 2, 2024

Apache-HBase commented Apr 2, 2024

Apache-HBase commented Apr 3, 2024

Apache-HBase commented Apr 3, 2024

Apache-HBase commented Apr 3, 2024

Apache-HBase commented Apr 3, 2024

Apache-HBase commented Apr 3, 2024

Apache-HBase commented Apr 3, 2024