Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-25507 Leak of ESTABLISHED sockets when compaction encountered "… #2882

Merged
merged 1 commit into from
Feb 9, 2021
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
HBASE-25507 Leak of ESTABLISHED sockets when compaction encountered "…
…java.io.IOException: Invalid HFile block magic"
sunhelly committed Jan 18, 2021
commit 30b91113ba17aeb7d31b6b54bdb288f5f2e2d038
Original file line number Diff line number Diff line change
@@ -321,10 +321,10 @@ protected final List<Path> compact(final CompactionRequestImpl request,
dropCache = this.dropCacheMinor;
}

List<StoreFileScanner> scanners =
createFileScanners(request.getFiles(), smallestReadPoint, dropCache);
InternalScanner scanner = null;
boolean finished = false;
List<StoreFileScanner> scanners =
createFileScanners(request.getFiles(), smallestReadPoint, dropCache);
try {
/* Include deletes, unless we are doing a major compaction */
ScanType scanType = scannerFactory.getScanType(request);
@@ -345,7 +345,18 @@ protected final List<Path> compact(final CompactionRequestImpl request,
+ store.getRegionInfo().getRegionNameAsString() + " because it was interrupted.");
}
} finally {
Closeables.close(scanner, true);
// createScanner may fail when seeking hfiles encounter Exception, e.g. even only one hfile
// reader encounters java.io.IOException: Invalid HFile block magic:
// \x00\x00\x00\x00\x00\x00\x00\x00
// and then scanner will be null, but scanners for all the hfiles should be closed,
// or else we will find leak of ESTABLISHED sockets.
if (scanner == null) {
for (StoreFileScanner sfs : scanners) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is nice find.

sfs.close();
}
} else {
Closeables.close(scanner, true);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First, nice debugging. Good find.

Looking at this, I wonder if other InternalScanners can fail in similar way and leak? I see the InternalScanner is implemented in a few places... in KVHeap, and RegionScanner. Could these have the same problem?

Would it be more clean if the createFileScanners were inside a try/finally? Or will scanners act strangely if closed twice... once by internalscanner on the happy path, and then again in the outer finally?

Otherwise, nice work. Above are nits.... just stuff to consider.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @saintstack . I check the codes of KVHeap and RegionScanner, they seem OK. Thanks your suggests, I also think createFileScanners should be inside the try/finally. If we only close the scanners when InnerScanner is null, may be we can avoid to close the scanners twice. What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me @sunhelly . Does that mean update to the PR? Thanks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @saintstack , the PR has been updated.
Function createFileScanners closes the store file scanners it created when encounters any exception, so it is out the try/finally. When the InternalScanner which contains all the store file scanners is not null, all the scanners will be processed when it closes. So I added a check for InternalScanner if it is null, if so, close the store file scanners manually in finally.

}
if (!finished && writer != null) {
abortWriter(writer);
}