Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-26217 Clean shared state pollution to avoid flaky tests. #3615

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

LALAYANG
Copy link

@LALAYANG LALAYANG commented Aug 21, 2021

What is the purpose of this PR

  • This PR cleans the state polluted by org.apache.hadoop.hbase.monitoring.TestTaskMonitor.testDoNotPurgeRPCTask.

  • It may be better to clean state pollutions so that some other tests won't fail in the future due to the shared state pollution.

Reproduce the test failure

  • Run the test twice in the same JVM.

Expected result:

  • The tests should run successfully when multiple tests that use this state are run in the same JVM.

Actual result:

  • We get the failure:
    [ERROR] Failures: [ERROR] testDoNotPurgeRPCTask:144 RPC Tasks have been purged! expected:<10> but was:<20>

Why the test fails

  • Each time this test runs,10 RPCtasks are created and added to the ArrayList rpcTasks, but rpcTasks is not completely purged when the test ends. So next time the test starts, the remaining RPC tasks lead to an assertion failure.

Fix

Please let me know which fix you think is better. Thanks!

Issue link: https://issues.apache.org/jira/browse/HBASE-26217

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 2m 32s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 4m 1s master passed
+1 💚 compile 3m 12s master passed
+1 💚 checkstyle 1m 2s master passed
+1 💚 spotbugs 2m 4s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 39s the patch passed
+1 💚 compile 3m 7s the patch passed
+1 💚 javac 3m 7s the patch passed
+1 💚 checkstyle 1m 0s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 18m 36s Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚 spotbugs 2m 15s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 15s The patch does not generate ASF License warnings.
49m 19s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #3615
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname Linux 07d0d630593a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 6bb5701
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 95 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 30s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 4m 58s master passed
+1 💚 compile 1m 25s master passed
+1 💚 shadedjars 9m 6s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 49s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 37s the patch passed
+1 💚 compile 1m 21s the patch passed
+1 💚 javac 1m 21s the patch passed
+1 💚 shadedjars 8m 59s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 43s the patch passed
_ Other Tests _
+1 💚 unit 143m 1s hbase-server in the patch passed.
177m 14s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #3615
Optional Tests javac javadoc unit shadedjars compile
uname Linux acf0638325ed 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9 23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 6bb5701
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/testReport/
Max. process+thread count 3673 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 32s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 4m 34s master passed
+1 💚 compile 1m 10s master passed
+1 💚 shadedjars 8m 39s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 39s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 0s the patch passed
+1 💚 compile 1m 3s the patch passed
+1 💚 javac 1m 3s the patch passed
+1 💚 shadedjars 8m 20s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 37s the patch passed
_ Other Tests _
+1 💚 unit 149m 8s hbase-server in the patch passed.
180m 41s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #3615
Optional Tests javac javadoc unit shadedjars compile
uname Linux b8a2098a94d3 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 6bb5701
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/testReport/
Max. process+thread count 3458 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@LALAYANG LALAYANG changed the title Clean shared state pollution to avoid flaky tests. HBASE-26217 Clean shared state pollution to avoid flaky tests. Aug 22, 2021
@Apache9
Copy link
Contributor

Apache9 commented Aug 22, 2021

Thanks for the PR. It is always good that we have developers want to clean up tests code.

But what I do not understand is, we always create a new TaskMonitor in every test method, so how could it mess up later tests run? Looking at the code in TaskMonitor, we do not have static field in this class?

Thanks.

@LALAYANG
Copy link
Author

LALAYANG commented Nov 9, 2021

In test org.apache.hadoop.hbase.monitoring.TestTaskMonitor.testDoNotPurgeRPCTask, TaskMonitor is not created each time, so the pollution is not cleaned.

There are 2 kinds of fixes:

  • Clean all tasks when the test ends.
  • Another solution is to create a new TaskMonitor each time to avoid pollution similar to the other tests (e.g., testTaskMonitorBasics, testTasksGetAbortedOnLeak) in TestTaskMonitor. Create a new TaskMonitor each time to avoid flaky tests. Create a new TaskMonitor each time to avoid flaky tests. LALAYANG/hbase#1
    Please let me know which fix you think is better. Thanks!

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 0s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 4m 35s master passed
+1 💚 compile 3m 30s master passed
+1 💚 checkstyle 1m 11s master passed
+1 💚 spotbugs 2m 17s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 17s the patch passed
+1 💚 compile 3m 23s the patch passed
+1 💚 javac 3m 23s the patch passed
+1 💚 checkstyle 1m 9s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 21m 46s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚 spotbugs 2m 25s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 14s The patch does not generate ASF License warnings.
54m 22s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #3615
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname Linux a27fff01d96e 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / f6348d4
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 86 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 42s Docker mode activated.
-0 ⚠️ yetus 0m 4s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 4m 46s master passed
+1 💚 compile 1m 16s master passed
+1 💚 shadedjars 8m 33s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 45s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 41s the patch passed
+1 💚 compile 1m 18s the patch passed
+1 💚 javac 1m 18s the patch passed
+1 💚 shadedjars 8m 26s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 42s the patch passed
_ Other Tests _
-1 ❌ unit 164m 48s hbase-server in the patch failed.
198m 58s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #3615
Optional Tests javac javadoc unit shadedjars compile
uname Linux 82cf4cc56328 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15 13:41:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / f6348d4
Default Java AdoptOpenJDK-11.0.10+9
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/testReport/
Max. process+thread count 3673 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 26s Docker mode activated.
-0 ⚠️ yetus 0m 4s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 4m 59s master passed
+1 💚 compile 1m 10s master passed
+1 💚 shadedjars 9m 15s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 41s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 26s the patch passed
+1 💚 compile 1m 8s the patch passed
+1 💚 javac 1m 8s the patch passed
+1 💚 shadedjars 9m 16s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 38s the patch passed
_ Other Tests _
-1 ❌ unit 223m 35s hbase-server in the patch failed.
258m 27s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #3615
Optional Tests javac javadoc unit shadedjars compile
uname Linux 100be35682ca 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / f6348d4
Default Java AdoptOpenJDK-1.8.0_282-b08
unit https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/testReport/
Max. process+thread count 3790 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants