HBASE-26488 Memory leak when MemStore retry flushing #3899

comnetwork · 2021-11-30T11:39:43Z

No description provided.

Apache-HBase · 2021-11-30T12:42:45Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 28s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 48s	master passed
+1 💚	compile	3m 34s	master passed
+1 💚	checkstyle	1m 11s	master passed
+1 💚	spotbugs	2m 17s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 21s	the patch passed
+1 💚	compile	3m 39s	the patch passed
+1 💚	javac	3m 39s	the patch passed
+1 💚	checkstyle	1m 11s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	23m 55s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	2m 54s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		62m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3899
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 14cbd9039b0f 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `f108997`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-11-30T15:49:06Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 28s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	5m 43s	master passed
+1 💚	compile	1m 29s	master passed
+1 💚	shadedjars	9m 27s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 51s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 25s	the patch passed
+1 💚	compile	1m 23s	the patch passed
+1 💚	javac	1m 23s	the patch passed
+1 💚	shadedjars	9m 18s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 45s	the patch passed
		_ Other Tests _
+1 💚	unit	208m 39s	hbase-server in the patch passed.
		248m 25s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3899
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux dcb220ad7e83 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `f108997`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/1/testReport/
Max. process+thread count	2983 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-11-30T15:56:19Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 22s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 54s	master passed
+1 💚	compile	1m 11s	master passed
+1 💚	shadedjars	9m 13s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 40s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 25s	the patch passed
+1 💚	compile	1m 10s	the patch passed
+1 💚	javac	1m 10s	the patch passed
+1 💚	shadedjars	9m 13s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 37s	the patch passed
		_ Other Tests _
+1 💚	unit	220m 56s	hbase-server in the patch passed.
		255m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3899
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 61d9b0ad15b0 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `f108997`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/1/testReport/
Max. process+thread count	3725 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-01T03:50:08Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 4s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 17s	master passed
+1 💚	compile	3m 28s	master passed
+1 💚	checkstyle	1m 10s	master passed
+1 💚	spotbugs	2m 16s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 14s	the patch passed
+1 💚	compile	3m 20s	the patch passed
+1 💚	javac	3m 20s	the patch passed
+1 💚	checkstyle	1m 9s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	21m 43s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	2m 24s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		54m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3899
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux eea5d0a66e04 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `358c4dc`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-01T05:49:12Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	7m 6s	master passed
+1 💚	compile	1m 12s	master passed
+1 💚	shadedjars	8m 17s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 43s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 35s	the patch passed
+1 💚	compile	1m 15s	the patch passed
+1 💚	javac	1m 15s	the patch passed
+1 💚	shadedjars	8m 15s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 40s	the patch passed
		_ Other Tests _
+1 💚	unit	138m 23s	hbase-server in the patch passed.
		173m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3899
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 547a69cb6fe1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `358c4dc`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/2/testReport/
Max. process+thread count	4068 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-01T07:01:06Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 1s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 35s	master passed
+1 💚	compile	1m 5s	master passed
+1 💚	shadedjars	9m 4s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 16s	the patch passed
+1 💚	compile	1m 4s	the patch passed
+1 💚	javac	1m 4s	the patch passed
+1 💚	shadedjars	9m 4s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 36s	the patch passed
		_ Other Tests _
+1 💚	unit	211m 37s	hbase-server in the patch passed.
		245m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3899
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux a5bb2eccc9f7 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `358c4dc`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/2/testReport/
Max. process+thread count	3432 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2021-12-04T15:11:16Z

I think the intention here is that, for a memstore snapshot, it will never change, so we do not need to create a scanner for it every time, just reuse the same one.

So here, I prefer we introduce a DelegateKeyValueScanner, which will delegate all the methods to the segment scanner, except the close method. And once the snapshot itself is closed, we will call the close method of the actual segment scanners.

WDYT?

Thanks.

comnetwork · 2021-12-05T12:14:13Z

@Apache9 , thank you very much for review. In my opinion seems that introducing a DelegateKeyValueScanner could not solve the problem:

It seems a reference count problem , the SnapshotSegmentScanner seems independent of whether snapshot segment itself is closed or not, even if snapshot segment is closed, if there is SnapshotSegmentScanner or SegmentScanner is using，we could not release the MemStoreLAB of the snapshot segment.
Mostly，I find that MemStoreSnapshot.close is not always invoked under every situation, eg. flushing success or failed does not invoke MemStoreSnapshot.close in fact, so we could not depend on MemStoreSnapshot.close to close the SnapshotSegmentScanner and I think we could remove the MemStoreSnapshot.close in the future to simplify the logic.
and I think introducing another DelegateKeyValueScanner for wrapping SnapshotSegmentScanner would make the code more complex, because SnapshotSegmentScanner is only used for MemStoreSnapshot here, so seems that we create another additional wrapper for this SnapshotSegmentScanner would cause the lazy close logic overlaying reference count logic which is hard to understand and error-prone, just use SnapshotSegmentScanner for reference count logic is clear and simple.
The overhead of creating a new SnapshotSegmentScanner every time when MemStoreSnapshot.getScanners is called is low, just as the behavior of DefaultMemStore.getScanners and Segment.getScanners.The key in the SnapshotSegmentScanner ctor is to increase the reference count of the MemStoreLAB of the snapshot segment.

@Apache9 , that is all of my thought, hoping for your feedback.

Apache9 · 2021-12-13T16:14:41Z

I do not think you fully got my point...

My point is, the design of the MemStoreSnapshot is mainly for storing the snapshot id and the snapshot scanner, if we do not want to store the scanner in it, I think we could just remove this class, just call MemStore.getSnapshot every time when we want to use it...

And for the problem 2, it is a problem, if it is a Closeable, then we should always close it. I checked the code, seems the only place where we miss the snapshot.close is when committing a flush, we should call close before calling updateStorefiles.

And for problem 3, I do not see any difficulties to understand the logic? The design is that, the segment will not be changed any more, so we always use the same scanners to scan it. in order to not close it before we clear the snapshot, we should try to use DelegatingKeyValueScanner to implement an empty close method, and then close it in the MemStoreSnapshot.close method. This is an optimization, it will have more logics but I do not think it is too hard to understand.

And for problem 4, as said above, this is an optimization, maybe it can not impact the performance a lot, but it does not introduce too much complexity too, so for me I think it is worth a try.

There is a DelegatingKeyValueScanner implementation under the src/test, I think we can just move it src/main and use it.

Thanks.

comnetwork · 2021-12-14T03:47:06Z

@Apache9 ，thank you very much for detailed reply, yes, I understand your point, maybe I did not explain clearly. My opinion is that whether we could make the code more simpler? Whether we could make the MemStoreSnapshot just a Context Object which just as branch-1 does, not a Closeable object(we could remove Closeable from MemStoreSnapshot)？By creating a new SnapshotSegmentScanner every time when MemStoreSnapshot.getScanners is called we could avoid forgetful MemStoreSnapshot.close and could also avoid using a DelegatingKeyValueScanner to delay close.

And for problem 3, I did not mean DelegatingKeyValueScanner itself is hard to understand, I mean that on the one hand we use DelegatingKeyValueScanner to delay close SnapshotSegmentScanner and we should make sure MemStoreSnapshot.close is invoked to trace SnapshotSegmentScanner.close, on the other hand SnapshotSegmentScanner itself also has reference count, so these factors overlay make it more indirect when we want to locate or track some memory problem about the segment , and this indirect could avoid.

btw. And for the problem 2，seems that if the flush failed，because SnapshotSegmentScanner.close is called, seems that MemStoreSnapshot.close is also not called.

@Apache9 ,in short, my opinion is just to make the code somewhat more simpler，we could remove Closeable from MemStoreSnapshot and avoid using aDelegatingKeyValueScanner to delay close, hoping for your further feedback.

Apache9 · 2021-12-15T15:23:52Z

Fine, then let's just remove the close method of MemStoreSnapshot since it is useless now? And also change the javadoc to reflect the new design and implementation.

comnetwork · 2021-12-16T04:08:35Z

@Apache9 , thank you very much for feedback, removing the close method of MemStoreSnapshot is added in the PR.

Apache-HBase · 2021-12-16T05:09:36Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 29s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	5m 8s	master passed
+1 💚	compile	3m 58s	master passed
+1 💚	checkstyle	1m 21s	master passed
+1 💚	spotbugs	2m 48s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 52s	the patch passed
+1 💚	compile	3m 57s	the patch passed
+1 💚	javac	3m 57s	the patch passed
-0 ⚠️	checkstyle	1m 25s	hbase-server: The patch generated 1 new + 31 unchanged - 1 fixed = 32 total (was 32)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	24m 31s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	2m 53s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 17s	The patch does not generate ASF License warnings.
		62m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3899
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 047a399f1f1c 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c93e457`
Default Java	AdoptOpenJDK-1.8.0_282-b08
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/3/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-16T06:57:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 40s	master passed
+1 💚	compile	1m 11s	master passed
+1 💚	shadedjars	8m 27s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 41s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 31s	the patch passed
+1 💚	compile	1m 14s	the patch passed
+1 💚	javac	1m 14s	the patch passed
+1 💚	shadedjars	8m 13s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 41s	the patch passed
		_ Other Tests _
-1 ❌	unit	138m 19s	hbase-server in the patch failed.
		170m 30s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3899
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 9f097532eee5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c93e457`
Default Java	AdoptOpenJDK-11.0.10+9
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/3/testReport/
Max. process+thread count	4298 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/3/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-16T08:18:32Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 17s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 36s	master passed
+1 💚	compile	1m 3s	master passed
+1 💚	shadedjars	9m 7s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 39s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 22s	the patch passed
+1 💚	compile	1m 3s	the patch passed
+1 💚	javac	1m 3s	the patch passed
+1 💚	shadedjars	9m 14s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	the patch passed
		_ Other Tests _
-1 ❌	unit	215m 51s	hbase-server in the patch failed.
		251m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3899
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 47ad3bb66f14 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c93e457`
Default Java	AdoptOpenJDK-1.8.0_282-b08
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/3/testReport/
Max. process+thread count	3545 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/3/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-16T09:20:25Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 12s	master passed
+1 💚	compile	3m 20s	master passed
+1 💚	checkstyle	1m 9s	master passed
+1 💚	spotbugs	2m 15s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 9s	the patch passed
+1 💚	compile	3m 19s	the patch passed
+1 💚	javac	3m 19s	the patch passed
+1 💚	checkstyle	1m 9s	hbase-server: The patch generated 0 new + 31 unchanged - 1 fixed = 31 total (was 32)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	24m 48s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	2m 28s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		57m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3899
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 4e87d40c4886 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c93e457`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/4/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-16T11:16:45Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	4m 51s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 51s	master passed
+1 💚	compile	1m 15s	master passed
+1 💚	shadedjars	8m 37s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 45s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 33s	the patch passed
+1 💚	compile	1m 14s	the patch passed
+1 💚	javac	1m 14s	the patch passed
+1 💚	shadedjars	8m 16s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 41s	the patch passed
		_ Other Tests _
-1 ❌	unit	137m 0s	hbase-server in the patch failed.
		174m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3899
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 410ab8d41904 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c93e457`
Default Java	AdoptOpenJDK-11.0.10+9
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/4/testReport/
Max. process+thread count	4003 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/4/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-16T12:35:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 46s	Docker mode activated.
-0 ⚠️	yetus	0m 5s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 42s	master passed
+1 💚	compile	1m 7s	master passed
+1 💚	shadedjars	9m 24s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 44s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 29s	the patch passed
+1 💚	compile	1m 7s	the patch passed
+1 💚	javac	1m 7s	the patch passed
+1 💚	shadedjars	9m 24s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 39s	the patch passed
		_ Other Tests _
-1 ❌	unit	217m 19s	hbase-server in the patch failed.
		252m 45s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3899
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c9b611abac37 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c93e457`
Default Java	AdoptOpenJDK-1.8.0_282-b08
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/4/testReport/
Max. process+thread count	3126 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/4/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-17T14:25:12Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 28s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 40s	master passed
+1 💚	compile	3m 34s	master passed
+1 💚	checkstyle	1m 17s	master passed
+1 💚	spotbugs	2m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 21s	the patch passed
+1 💚	compile	3m 37s	the patch passed
+1 💚	javac	3m 37s	the patch passed
+1 💚	checkstyle	1m 13s	hbase-server: The patch generated 0 new + 31 unchanged - 1 fixed = 31 total (was 32)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	22m 9s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	2m 33s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		58m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3899
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux e1325fb1a6f1 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `e598f2c`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/5/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-17T17:58:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 0s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	5m 53s	master passed
+1 💚	compile	1m 40s	master passed
+1 💚	shadedjars	11m 15s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 51s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 39s	the patch passed
+1 💚	compile	1m 39s	the patch passed
+1 💚	javac	1m 39s	the patch passed
+1 💚	shadedjars	9m 54s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 51s	the patch passed
		_ Other Tests _
+1 💚	unit	230m 12s	hbase-server in the patch passed.
		270m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3899
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 73e10eab774a 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `e598f2c`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/5/testReport/
Max. process+thread count	2803 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/5/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-17T18:03:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 5s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 59s	master passed
+1 💚	compile	1m 15s	master passed
+1 💚	shadedjars	9m 52s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 45s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 47s	the patch passed
+1 💚	compile	1m 14s	the patch passed
+1 💚	javac	1m 14s	the patch passed
+1 💚	shadedjars	10m 54s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 43s	the patch passed
		_ Other Tests _
+1 💚	unit	238m 57s	hbase-server in the patch passed.
		276m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3899
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c9900345f95b 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `e598f2c`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/5/testReport/
Max. process+thread count	3874 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/5/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Signed-off-by: Duo Zhang <[email protected]>

Signed-off-by: Duo Zhang <[email protected]> (cherry picked from commit e20b2f3) Change-Id: Iffe855b34b3e667321c6a3550e5d59036ba7a8ca

HBASE-26488 Memory leak when MemStore retry flushing

aef4dfd

Merge remote-tracking branch 'upstream/master' into flushretry

7f10819

remove MemStoreSnapshot.close

e9d4560

comnetwork added 2 commits December 16, 2021 15:38

fix checkstyle

7caaf66

fix checkstyle

715cfc8

Apache9 approved these changes Dec 16, 2021

View reviewed changes

Apache9 merged commit da616c0 into apache:master Dec 18, 2021

Apache9 pushed a commit that referenced this pull request Dec 18, 2021

HBASE-26488 Memory leak when MemStore retry flushing (#3899)

e20b2f3

Signed-off-by: Duo Zhang <[email protected]>

Apache9 pushed a commit that referenced this pull request Dec 18, 2021

HBASE-26488 Memory leak when MemStore retry flushing (#3899)

482ef69

Signed-off-by: Duo Zhang <[email protected]>

Apache9 pushed a commit that referenced this pull request Dec 18, 2021

HBASE-26488 Memory leak when MemStore retry flushing (#3899)

2d5d5b2

Signed-off-by: Duo Zhang <[email protected]>

guangxuCheng pushed a commit to guangxuCheng/hbase that referenced this pull request Mar 28, 2022

HBASE-26488 Memory leak when MemStore retry flushing (apache#3899)

f116558

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-26488 Memory leak when MemStore retry flushing #3899

HBASE-26488 Memory leak when MemStore retry flushing #3899

comnetwork commented Nov 30, 2021

Apache-HBase commented Nov 30, 2021

Apache-HBase commented Nov 30, 2021

Apache-HBase commented Nov 30, 2021

Apache-HBase commented Dec 1, 2021

Apache-HBase commented Dec 1, 2021

Apache-HBase commented Dec 1, 2021

Apache9 commented Dec 4, 2021

comnetwork commented Dec 5, 2021 •

edited

Loading

Apache9 commented Dec 13, 2021

comnetwork commented Dec 14, 2021 •

edited

Loading

Apache9 commented Dec 15, 2021

comnetwork commented Dec 16, 2021

Apache-HBase commented Dec 16, 2021

Apache-HBase commented Dec 16, 2021

Apache-HBase commented Dec 16, 2021

Apache-HBase commented Dec 16, 2021

Apache-HBase commented Dec 16, 2021

Apache-HBase commented Dec 16, 2021

Apache-HBase commented Dec 17, 2021

Apache-HBase commented Dec 17, 2021

Apache-HBase commented Dec 17, 2021

HBASE-26488 Memory leak when MemStore retry flushing #3899

HBASE-26488 Memory leak when MemStore retry flushing #3899

Conversation

comnetwork commented Nov 30, 2021

Apache-HBase commented Nov 30, 2021

Apache-HBase commented Nov 30, 2021

Apache-HBase commented Nov 30, 2021

Apache-HBase commented Dec 1, 2021

Apache-HBase commented Dec 1, 2021

Apache-HBase commented Dec 1, 2021

Apache9 commented Dec 4, 2021

comnetwork commented Dec 5, 2021 • edited Loading

Apache9 commented Dec 13, 2021

comnetwork commented Dec 14, 2021 • edited Loading

Apache9 commented Dec 15, 2021

comnetwork commented Dec 16, 2021

Apache-HBase commented Dec 16, 2021

Apache-HBase commented Dec 16, 2021

Apache-HBase commented Dec 16, 2021

Apache-HBase commented Dec 16, 2021

Apache-HBase commented Dec 16, 2021

Apache-HBase commented Dec 16, 2021

Apache-HBase commented Dec 17, 2021

Apache-HBase commented Dec 17, 2021

Apache-HBase commented Dec 17, 2021

comnetwork commented Dec 5, 2021 •

edited

Loading

comnetwork commented Dec 14, 2021 •

edited

Loading