Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-22867 The ForkJoinPool in CleanerChore will spawn thousands of threads in our cluster with thousands table #513

Merged
merged 1 commit into from
Aug 22, 2019

Conversation

openinx
Copy link
Member

@openinx openinx commented Aug 20, 2019

…threads in our cluster with thousands table

@openinx
Copy link
Member Author

openinx commented Aug 20, 2019

PR an initial patch, need more consideration..

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 141 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 474 master passed
+1 compile 76 master passed
+1 checkstyle 104 master passed
+1 shadedjars 382 branch has no errors when building our shaded downstream artifacts.
+1 javadoc 46 master passed
0 spotbugs 308 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 306 master passed
_ Patch Compile Tests _
-1 mvninstall 194 root in the patch failed.
-1 compile 20 hbase-server in the patch failed.
-1 javac 20 hbase-server in the patch failed.
-1 checkstyle 89 hbase-server: The patch generated 2 new + 3 unchanged - 0 fixed = 5 total (was 3)
+1 whitespace 0 The patch has no whitespace issues.
-1 shadedjars 248 patch has 10 errors when building our shaded downstream artifacts.
-1 hadoopcheck 132 The patch causes 10 errors with Hadoop v2.8.5.
-1 hadoopcheck 271 The patch causes 10 errors with Hadoop v2.9.2.
-1 hadoopcheck 414 The patch causes 10 errors with Hadoop v3.1.2.
+1 javadoc 45 the patch passed
-1 findbugs 21 hbase-server in the patch failed.
_ Other Tests _
-1 unit 24 hbase-server in the patch failed.
+1 asflicense 15 The patch does not generate ASF License warnings.
2699
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/1/artifact/out/Dockerfile
GITHUB PR #513
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 71ec5656a698 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-513/out/precommit/personality/provided.sh
git revision master / a59f7d4
Default Java 1.8.0_181
mvninstall https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/1/artifact/out/patch-mvninstall-root.txt
compile https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/1/artifact/out/patch-compile-hbase-server.txt
javac https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/1/artifact/out/patch-compile-hbase-server.txt
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/1/artifact/out/diff-checkstyle-hbase-server.txt
shadedjars https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/1/artifact/out/patch-shadedjars.txt
hadoopcheck https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/1/artifact/out/patch-javac-2.8.5.txt
hadoopcheck https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/1/artifact/out/patch-javac-2.9.2.txt
hadoopcheck https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/1/artifact/out/patch-javac-3.1.2.txt
findbugs https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/1/artifact/out/patch-findbugs-hbase-server.txt
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/1/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/1/testReport/
Max. process+thread count 86 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/1/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

task.fork();
}
allSubdirsDeleted = deleteAction(() -> getCleanResult(tasks), "subdirs");
subDirs.forEach(dir -> futures.add(pool.execute(new CleanerTask(dir, false))));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think submitting the task to the same pool is not safe here. In ForkJoinPool, it is safe to call join as in the join method, the current thread will try to execute tasks if there are any, so in general, we can always make progress and finally finish all the tasks. But here, the current thread will be blocked on the future.get, so it is possible that all the threads in the pool are blocked on a future.get and cause dead lock here.

@openinx
Copy link
Member Author

openinx commented Aug 20, 2019

Ping @Apache9 , I've rewriten the cleaner , please take a look again.

@openinx
Copy link
Member Author

openinx commented Aug 20, 2019

Seems still some problem, the UT are easy to hang now, Let me check.

@openinx
Copy link
Member Author

openinx commented Aug 20, 2019

OK, I got the problem, forget to complete the future somewhere.. it's easy to fix .

@openinx
Copy link
Member Author

openinx commented Aug 20, 2019

Fixed all the UT in cleanerChore, should be good now.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 128 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 462 master passed
+1 compile 77 master passed
+1 checkstyle 99 master passed
+1 shadedjars 384 branch has no errors when building our shaded downstream artifacts.
+1 javadoc 53 master passed
0 spotbugs 339 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 336 master passed
_ Patch Compile Tests _
+1 mvninstall 426 the patch passed
+1 compile 74 the patch passed
+1 javac 74 the patch passed
+1 checkstyle 99 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 378 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 1143 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 javadoc 35 the patch passed
+1 findbugs 267 the patch passed
_ Other Tests _
-1 unit 14097 hbase-server in the patch failed.
+1 asflicense 25 The patch does not generate ASF License warnings.
18509
Reason Tests
Failed junit tests hadoop.hbase.client.TestMobRestoreSnapshotFromClientClone
hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient
hadoop.hbase.client.TestRestoreSnapshotFromClientClone
hadoop.hbase.client.TestCloneSnapshotFromClientCloneLinksAfterDelete
hadoop.hbase.mob.compactions.TestMobCompactor
hadoop.hbase.client.TestMobCloneSnapshotFromClientCloneLinksAfterDelete
hadoop.hbase.master.cleaner.TestHFileLinkCleaner
hadoop.hbase.security.access.TestSnapshotScannerHDFSAclController
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/2/artifact/out/Dockerfile
GITHUB PR #513
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux ad090ea2fa42 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-513/out/precommit/personality/provided.sh
git revision master / a59f7d4
Default Java 1.8.0_181
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/2/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/2/testReport/
Max. process+thread count 4788 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/2/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 260 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 398 master passed
+1 compile 62 master passed
+1 checkstyle 95 master passed
+1 shadedjars 302 branch has no errors when building our shaded downstream artifacts.
+1 javadoc 39 master passed
0 spotbugs 264 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 261 master passed
_ Patch Compile Tests _
+1 mvninstall 335 the patch passed
+1 compile 66 the patch passed
+1 javac 66 the patch passed
-1 checkstyle 125 hbase-server: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 337 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 1224 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 javadoc 36 the patch passed
+1 findbugs 553 the patch passed
_ Other Tests _
-1 unit 2844 hbase-server in the patch failed.
+1 asflicense 14 The patch does not generate ASF License warnings.
7411
Reason Tests
Failed junit tests hadoop.hbase.master.cleaner.TestCleanerChore
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/3/artifact/out/Dockerfile
GITHUB PR #513
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 6831506eb8b3 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-513/out/precommit/personality/provided.sh
git revision master / a59f7d4
Default Java 1.8.0_181
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/3/artifact/out/diff-checkstyle-hbase-server.txt
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/3/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/3/testReport/
Max. process+thread count 670 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/3/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 66 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 352 master passed
+1 compile 53 master passed
+1 checkstyle 78 master passed
+1 shadedjars 282 branch has no errors when building our shaded downstream artifacts.
+1 javadoc 38 master passed
0 spotbugs 257 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 255 master passed
_ Patch Compile Tests _
+1 mvninstall 308 the patch passed
+1 compile 55 the patch passed
+1 javac 55 the patch passed
-1 checkstyle 78 hbase-server: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 305 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 1054 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 javadoc 36 the patch passed
+1 findbugs 260 the patch passed
_ Other Tests _
-1 unit 17630 hbase-server in the patch failed.
+1 asflicense 39 The patch does not generate ASF License warnings.
21333
Reason Tests
Failed junit tests hadoop.hbase.client.TestSnapshotTemporaryDirectory
hadoop.hbase.master.procedure.TestSCPWithReplicasWithoutZKCoordinated
hadoop.hbase.client.TestFromClientSideWithCoprocessor
hadoop.hbase.client.TestFromClientSide3
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/4/artifact/out/Dockerfile
GITHUB PR #513
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux dbdfabc60310 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-513/out/precommit/personality/provided.sh
git revision master / a59f7d4
Default Java 1.8.0_181
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/4/artifact/out/diff-checkstyle-hbase-server.txt
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/4/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/4/testReport/
Max. process+thread count 5276 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/4/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 78 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 449 master passed
+1 compile 72 master passed
+1 checkstyle 97 master passed
+1 shadedjars 363 branch has no errors when building our shaded downstream artifacts.
+1 javadoc 51 master passed
0 spotbugs 325 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 323 master passed
_ Patch Compile Tests _
+1 mvninstall 412 the patch passed
+1 compile 74 the patch passed
+1 javac 74 the patch passed
-1 checkstyle 99 hbase-server: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 378 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 1371 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 javadoc 49 the patch passed
+1 findbugs 332 the patch passed
_ Other Tests _
-1 unit 14754 hbase-server in the patch failed.
+1 asflicense 31 The patch does not generate ASF License warnings.
19480
Reason Tests
Failed junit tests hadoop.hbase.security.access.TestSnapshotScannerHDFSAclController
hadoop.hbase.master.TestMasterShutdown
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/5/artifact/out/Dockerfile
GITHUB PR #513
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux b6694b07fdeb 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-513/out/precommit/personality/provided.sh
git revision master / ab28f9d
Default Java 1.8.0_181
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/5/artifact/out/diff-checkstyle-hbase-server.txt
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/5/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/5/testReport/
Max. process+thread count 4886 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/5/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@Apache9 Apache9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general I think this is a possible way to solve the problem. Using CompletableFuture to get the final result, and in the actions which will be executed in thread pool, just return without wating for the result of the CompletableFuture. But seems we still use the ForkJoinPool? This may still lead to too many threads?

// Step.4: Once all sub-files & sub-directories are deleted, then can try to delete the
// current directory asynchronously.
CompletableFuture.allOf(futures.toArray(new CompletableFuture[futures.size()]))
.whenComplete((voidObj, e) -> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better use FutureUtils.addListener here. Although we do not run error-prone checks now in pre commit, this will lead to a FutureReturnValueIgnored warning, and it is a problem. If the code in whenComplete throw an exception, the current implementation will hide it and you can only see the program hangs, using FutureUtils.addListener can print out the exception stacktrace at least.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, let me update the patch.

@openinx
Copy link
Member Author

openinx commented Aug 21, 2019

But seems we still use the ForkJoinPool? This may still lead to too many threads?

@Apache9
There should have no place where we use the ForkJoinPool, because all of the work are done in the ThreadPoolExectuor now. Or which place you mean we will use the ForkJoinPool you mean ?

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 35 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 316 master passed
+1 compile 50 master passed
+1 checkstyle 71 master passed
+1 shadedjars 263 branch has no errors when building our shaded downstream artifacts.
+1 javadoc 34 master passed
0 spotbugs 245 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 242 master passed
_ Patch Compile Tests _
+1 mvninstall 288 the patch passed
+1 compile 50 the patch passed
+1 javac 50 the patch passed
-1 checkstyle 70 hbase-server: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 259 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 887 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 javadoc 33 the patch passed
+1 findbugs 251 the patch passed
_ Other Tests _
+1 unit 8191 hbase-server in the patch passed.
+1 asflicense 22 The patch does not generate ASF License warnings.
11424
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/6/artifact/out/Dockerfile
GITHUB PR #513
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 1482d0ceaea3 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-513/out/precommit/personality/provided.sh
git revision master / 66ad42c
Default Java 1.8.0_181
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/6/artifact/out/diff-checkstyle-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/6/testReport/
Max. process+thread count 4675 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/6/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 86 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 360 master passed
+1 compile 59 master passed
+1 checkstyle 77 master passed
+1 shadedjars 298 branch has no errors when building our shaded downstream artifacts.
+1 javadoc 38 master passed
0 spotbugs 289 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 287 master passed
_ Patch Compile Tests _
+1 mvninstall 314 the patch passed
+1 compile 58 the patch passed
+1 javac 58 the patch passed
+1 checkstyle 76 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 287 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 1038 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 javadoc 38 the patch passed
+1 findbugs 266 the patch passed
_ Other Tests _
-1 unit 18121 hbase-server in the patch failed.
+1 asflicense 35 The patch does not generate ASF License warnings.
21861
Reason Tests
Failed junit tests hadoop.hbase.client.TestFromClientSide3
hadoop.hbase.master.TestMasterShutdown
hadoop.hbase.client.TestFromClientSideWithCoprocessor
hadoop.hbase.client.TestSnapshotTemporaryDirectory
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/7/artifact/out/Dockerfile
GITHUB PR #513
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 2b82d052a020 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-513/out/precommit/personality/provided.sh
git revision master / 66ad42c
Default Java 1.8.0_181
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/7/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/7/testReport/
Max. process+thread count 5005 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/7/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@openinx
Copy link
Member Author

openinx commented Aug 21, 2019

Ping @Apache9 @Reidddddd any other concerns ? I've made this working in our offline cluster (which encountered this problem before), it seems work pretty good now.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
0 reexec 41 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 334 master passed
+1 compile 54 master passed
+1 checkstyle 79 master passed
+1 shadedjars 296 branch has no errors when building our shaded downstream artifacts.
+1 javadoc 38 master passed
0 spotbugs 278 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 276 master passed
_ Patch Compile Tests _
+1 mvninstall 327 the patch passed
+1 compile 54 the patch passed
+1 javac 54 the patch passed
+1 checkstyle 76 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 273 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 937 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 javadoc 33 the patch passed
+1 findbugs 250 the patch passed
_ Other Tests _
+1 unit 8496 hbase-server in the patch passed.
+1 asflicense 26 The patch does not generate ASF License warnings.
11972
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/9/artifact/out/Dockerfile
GITHUB PR #513
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux f2678a717a71 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-513/out/precommit/personality/provided.sh
git revision master / 1b50404
Default Java 1.8.0_181
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/9/testReport/
Max. process+thread count 4939 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/9/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@@ -213,9 +213,14 @@ private void preRunCleaner() {

public Boolean runCleaner() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be boolean here? We will not return null in the future.get?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah , it can be.

} catch (Exception ie) {
// Must handle the inner exception here, otherwise the result may get stuck if one
// sub-directory get some failure.
result.completeExceptionally(ie);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in the past we will return false instead of throwing the exception out?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Em, let me check whether it will impact the behavior ..

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've checked the logic, should have no difference here. Both the exeption from sub-directory & boolean flag will skip the deletion of current directory, should be OK.

synchronized void execute(ForkJoinTask<?> task) {
pool.execute(task);
synchronized void runAsync(Runnable runnable) {
CompletableFuture.runAsync(runnable, pool);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Emmm, what is this used for? The method will return a CompletableFuture but we just ignore it? Then what is the difference comparing to just call pool.execute?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine, can just use the pool#execute now. plan to use the CompletableFuture of runAsync to get the final result befoe.

LOG.info("Update chore's pool size from {} to {}", pool.getParallelism(), size);
pool = new ForkJoinPool(size);
LOG.info("Update chore's pool size from {} to {}", pool.getPoolSize(), size);
pool = initializePool(size);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For ThreadPoolExecutor, there is a setCorePoolSize method so we do not need to recreate it. I think we can make it as a final member.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, let me address this in the new patch, good suggestion.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
0 reexec 37 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 408 master passed
+1 compile 59 master passed
+1 checkstyle 80 master passed
+1 shadedjars 290 branch has no errors when building our shaded downstream artifacts.
+1 javadoc 38 master passed
0 spotbugs 265 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 256 master passed
_ Patch Compile Tests _
+1 mvninstall 298 the patch passed
+1 compile 52 the patch passed
+1 javac 52 the patch passed
+1 checkstyle 71 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 270 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 923 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 javadoc 31 the patch passed
+1 findbugs 241 the patch passed
_ Other Tests _
+1 unit 8778 hbase-server in the patch passed.
+1 asflicense 24 The patch does not generate ASF License warnings.
12223
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/10/artifact/out/Dockerfile
GITHUB PR #513
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 2169c270100e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-513/out/precommit/personality/provided.sh
git revision master / b8857ec
Default Java 1.8.0_181
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/10/testReport/
Max. process+thread count 4667 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/10/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
0 reexec 36 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 344 master passed
+1 compile 56 master passed
+1 checkstyle 78 master passed
+1 shadedjars 294 branch has no errors when building our shaded downstream artifacts.
+1 javadoc 36 master passed
0 spotbugs 274 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 273 master passed
_ Patch Compile Tests _
+1 mvninstall 338 the patch passed
+1 compile 58 the patch passed
+1 javac 58 the patch passed
+1 checkstyle 77 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 289 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 1005 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 javadoc 34 the patch passed
+1 findbugs 256 the patch passed
_ Other Tests _
+1 unit 8624 hbase-server in the patch passed.
+1 asflicense 25 The patch does not generate ASF License warnings.
12216
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/11/artifact/out/Dockerfile
GITHUB PR #513
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 55969b1d3def 4.4.0-157-generic #185-Ubuntu SMP Tue Jul 23 09:17:01 UTC 2019 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-513/out/precommit/personality/provided.sh
git revision master / b8857ec
Default Java 1.8.0_181
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/11/testReport/
Max. process+thread count 4650 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/11/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@openinx openinx force-pushed the master branch 2 times, most recently from 1483af2 to a3b3826 Compare August 22, 2019 03:14
Copy link
Contributor

@Apache9 Apache9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No big problems, only a few small questions. Will give a +1 after the QA is fine.

LOG.info("Cleaner pool size is {}", size);
cleanerLatch = 0;
}

private static ThreadPoolExecutor initializePool(int size) {
return new ThreadPoolExecutor(size, size, 500, TimeUnit.SECONDS, new LinkedBlockingQueue<>(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually it will be 1 minute? Why 500 seconds? And I think we can call allowCoreThreadTimeOut(true) here to release the threads if not needed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, did not consider much about the keepAliveTime (can make it 1min if neccessary). the allowCoreThreadTimeOut sounds good.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 98 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 371 master passed
+1 compile 64 master passed
+1 checkstyle 92 master passed
+1 shadedjars 306 branch has no errors when building our shaded downstream artifacts.
+1 javadoc 41 master passed
0 spotbugs 317 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 314 master passed
_ Patch Compile Tests _
+1 mvninstall 340 the patch passed
+1 compile 62 the patch passed
+1 javac 62 the patch passed
+1 checkstyle 89 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 309 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 1057 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 javadoc 38 the patch passed
+1 findbugs 312 the patch passed
_ Other Tests _
-1 unit 16431 hbase-server in the patch failed.
+1 asflicense 28 The patch does not generate ASF License warnings.
20379
Reason Tests
Failed junit tests hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint
hadoop.hbase.client.TestFromClientSideWithCoprocessor
hadoop.hbase.replication.TestSyncReplicationStandbyKillMaster
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/14/artifact/out/Dockerfile
GITHUB PR #513
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 6357eb7d6148 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-513/out/precommit/personality/provided.sh
git revision master / 84ee378
Default Java 1.8.0_181
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/14/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/14/testReport/
Max. process+thread count 4715 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/14/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@openinx
Copy link
Member Author

openinx commented Aug 22, 2019

The hbase QA seems is unrelated , but will try the HBase QA again.

Copy link
Contributor

@Apache9 Apache9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1.

}
});
} catch (Exception e) {
LOG.warn("Failed to traverse and delete the path: {}", dir, e);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOG.warn here may be very chatty. Debug or Trace is up to your choice.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, here can be DEBUG because we will put the exception in result, and will get it in runCleaner method. Let me fix this.

Copy link
Contributor

@Reidddddd Reidddddd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One trivial comment about Logging, LTGM overall.

Copy link
Contributor

@Reidddddd Reidddddd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One trivial comment about Logging, LTGM overall.

@openinx openinx merged commit 4268774 into apache:master Aug 22, 2019
asfgit pushed a commit that referenced this pull request Aug 22, 2019
…threads in our cluster with thousands table (#513)

Signed-off-by: Duo Zhang <[email protected]>
Signed-off-by: Reid Chan <[email protected]>
asfgit pushed a commit that referenced this pull request Aug 22, 2019
…threads in our cluster with thousands table (#513)

Signed-off-by: Duo Zhang <[email protected]>
Signed-off-by: Reid Chan <[email protected]>
asfgit pushed a commit that referenced this pull request Aug 22, 2019
…threads in our cluster with thousands table (#513)

Signed-off-by: Duo Zhang <[email protected]>
Signed-off-by: Reid Chan <[email protected]>
@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 96 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 459 master passed
+1 compile 66 master passed
+1 checkstyle 81 master passed
+1 shadedjars 299 branch has no errors when building our shaded downstream artifacts.
+1 javadoc 39 master passed
0 spotbugs 282 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 278 master passed
_ Patch Compile Tests _
+1 mvninstall 379 the patch passed
+1 compile 61 the patch passed
+1 javac 61 the patch passed
+1 checkstyle 79 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 297 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 1098 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 javadoc 37 the patch passed
+1 findbugs 271 the patch passed
_ Other Tests _
-1 unit 17582 hbase-server in the patch failed.
+1 asflicense 38 The patch does not generate ASF License warnings.
21572
Reason Tests
Failed junit tests hadoop.hbase.client.TestRestoreSnapshotFromClientClone
hadoop.hbase.client.TestFromClientSideWithCoprocessor
hadoop.hbase.security.access.TestSnapshotScannerHDFSAclController
hadoop.hbase.client.TestAsyncTableAdminApi
hadoop.hbase.client.TestRestoreSnapshotFromClientSimple
hadoop.hbase.client.TestFromClientSide3
hadoop.hbase.client.TestSnapshotTemporaryDirectory
hadoop.hbase.client.TestCloneSnapshotFromClientNormal
hadoop.hbase.client.TestSplitOrMergeStatus
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/15/artifact/out/Dockerfile
GITHUB PR #513
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux b72bb7da1591 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-513/out/precommit/personality/provided.sh
git revision master / b329bf0
Default Java 1.8.0_181
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/15/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/15/testReport/
Max. process+thread count 5000 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-513/15/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

symat pushed a commit to symat/hbase that referenced this pull request Feb 17, 2021
…threads in our cluster with thousands table (apache#513)

Signed-off-by: Duo Zhang <[email protected]>
Signed-off-by: Reid Chan <[email protected]>
(cherry picked from commit 0690864)

Change-Id: Ic4fcf6c8c5d585da2f65e881bee500149a3c1ace
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants