Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-23210 Backport HBASE-15519 (Add per-user metrics) to branch-1 #755

Merged
merged 1 commit into from
Oct 28, 2019

Conversation

apurtell
Copy link
Contributor

HBASE-15519 Add per-user metrics with lossy counting

Introducing property hbase.regionserver.user.metrics.enabled(Default:true)
to disable user metrics in case it accounts for any performance issues

@apurtell
Copy link
Contributor Author

New unit tests pass

@apurtell
Copy link
Contributor Author

@joshelser I couldn't ask @ankitsinghal for a review for some reason, but i see you recently reviewed similar

@apurtell apurtell force-pushed the HBASE-23210-branch-1 branch from b770177 to b31f442 Compare October 24, 2019 22:47
Copy link
Member

@joshelser joshelser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two minor things, but LGTM.

class SweepRunnable implements Runnable {
@Override public void run() {
if (LOG.isTraceEnabled()) {
LOG.trace("Starting sweep of lossyCounting-" + name);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, could use the {} markers and drop the isTraceEnabled()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This came in from branch-2 change. I agree but try not to second guess the original contributor on backports.

Copy link
Member

@joshelser joshelser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two minor things, but LGTM.

try {
sweep();
} catch (Exception exception) {
LOG.debug("Error while sweeping of lossyCounting-{}", name, exception);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(damnit, lost the previous comment)

Make sure you're getting the right debug method here. The debug(String, Object, Exception) not debug(String, Object, Object). Hit this the other day.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixing

@apurtell apurtell force-pushed the HBASE-23210-branch-1 branch from b31f442 to ba2d5ff Compare October 24, 2019 23:09
@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
💙 reexec 2m 29s Docker mode activated.
_ Prechecks _
💚 dupname 0m 0s No case conflicting files found.
💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
💚 @author 0m 0s The patch does not contain any @author tags.
💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ branch-1 Compile Tests _
💙 mvndep 1m 27s Maven dependency ordering for branch
💚 mvninstall 8m 35s branch-1 passed
💚 compile 1m 27s branch-1 passed with JDK v1.8.0_232
💚 compile 1m 32s branch-1 passed with JDK v1.7.0_242
💚 checkstyle 2m 25s branch-1 passed
💚 shadedjars 4m 23s branch has no errors when building our shaded downstream artifacts.
💚 javadoc 1m 14s branch-1 passed with JDK v1.8.0_232
💚 javadoc 1m 32s branch-1 passed with JDK v1.7.0_242
💙 spotbugs 3m 20s Used deprecated FindBugs config; considering switching to SpotBugs.
💚 findbugs 4m 43s branch-1 passed
_ Patch Compile Tests _
💙 mvndep 0m 20s Maven dependency ordering for patch
💚 mvninstall 2m 38s the patch passed
💚 compile 1m 31s the patch passed with JDK v1.8.0_232
💚 javac 1m 31s the patch passed
💚 compile 1m 42s the patch passed with JDK v1.7.0_242
💚 javac 1m 42s the patch passed
💔 checkstyle 1m 58s hbase-server: The patch generated 6 new + 4 unchanged - 0 fixed = 10 total (was 4)
💚 whitespace 0m 0s The patch has no whitespace issues.
💚 shadedjars 4m 14s patch has no errors when building our shaded downstream artifacts.
💚 hadoopcheck 6m 38s Patch does not cause any errors with Hadoop 2.8.5 2.9.2.
💚 javadoc 1m 17s the patch passed with JDK v1.8.0_232
💚 javadoc 1m 31s the patch passed with JDK v1.7.0_242
💚 findbugs 5m 19s the patch passed
_ Other Tests _
💚 unit 0m 32s hbase-hadoop-compat in the patch passed.
💚 unit 0m 44s hbase-hadoop2-compat in the patch passed.
💔 unit 180m 41s hbase-server in the patch failed.
💚 asflicense 1m 7s The patch does not generate ASF License warnings.
246m 18s
Reason Tests
Failed junit tests hadoop.hbase.master.TestWarmupRegion
hadoop.hbase.master.TestMasterBalanceThrottling
Subsystem Report/Notes
Docker Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-755/1/artifact/out/Dockerfile
GITHUB PR #755
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux a8218eedfc70 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-755/out/precommit/personality/provided.sh
git revision branch-1 / 41f6713
Default Java 1.7.0_242
Multi-JDK versions /usr/lib/jvm/zulu-8-amd64:1.8.0_232 /usr/lib/jvm/zulu-7-amd64:1.7.0_242
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-755/1/artifact/out/diff-checkstyle-hbase-server.txt
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-755/1/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-755/1/testReport/
Max. process+thread count 3624 (vs. ulimit of 10000)
modules C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: .
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-755/1/console
versions git=1.9.1 maven=3.0.5 findbugs=3.0.1
Powered by Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

@apurtell
Copy link
Contributor Author

Back soon to fix checkstyle issues

@apurtell apurtell force-pushed the HBASE-23210-branch-1 branch from ba2d5ff to dc96c24 Compare October 25, 2019 04:25
@apurtell
Copy link
Contributor Author

Fixed some checkstyle nits, will merge tomorrow

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
💙 reexec 1m 25s Docker mode activated.
_ Prechecks _
💚 dupname 0m 0s No case conflicting files found.
💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
💚 @author 0m 0s The patch does not contain any @author tags.
💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ branch-1 Compile Tests _
💙 mvndep 1m 25s Maven dependency ordering for branch
💚 mvninstall 7m 32s branch-1 passed
💚 compile 1m 13s branch-1 passed with JDK v1.8.0_232
💚 compile 1m 20s branch-1 passed with JDK v1.7.0_242
💚 checkstyle 2m 12s branch-1 passed
💚 shadedjars 3m 16s branch has no errors when building our shaded downstream artifacts.
💚 javadoc 1m 1s branch-1 passed with JDK v1.8.0_232
💚 javadoc 1m 14s branch-1 passed with JDK v1.7.0_242
💙 spotbugs 3m 0s Used deprecated FindBugs config; considering switching to SpotBugs.
💚 findbugs 4m 9s branch-1 passed
_ Patch Compile Tests _
💙 mvndep 0m 15s Maven dependency ordering for patch
💚 mvninstall 2m 9s the patch passed
💚 compile 1m 8s the patch passed with JDK v1.8.0_232
💚 javac 1m 8s the patch passed
💚 compile 1m 21s the patch passed with JDK v1.7.0_242
💚 javac 1m 21s the patch passed
💔 checkstyle 1m 44s hbase-server: The patch generated 6 new + 4 unchanged - 0 fixed = 10 total (was 4)
💚 whitespace 0m 0s The patch has no whitespace issues.
💚 shadedjars 3m 8s patch has no errors when building our shaded downstream artifacts.
💚 hadoopcheck 5m 39s Patch does not cause any errors with Hadoop 2.8.5 2.9.2.
💚 javadoc 0m 58s the patch passed with JDK v1.8.0_232
💚 javadoc 1m 19s the patch passed with JDK v1.7.0_242
💚 findbugs 4m 41s the patch passed
_ Other Tests _
💚 unit 0m 27s hbase-hadoop-compat in the patch passed.
💚 unit 0m 36s hbase-hadoop2-compat in the patch passed.
💔 unit 164m 24s hbase-server in the patch failed.
💚 asflicense 1m 11s The patch does not generate ASF License warnings.
218m 57s
Reason Tests
Failed junit tests hadoop.hbase.master.TestMasterBalanceThrottling
hadoop.hbase.master.cleaner.TestReplicationZKNodeCleaner
Subsystem Report/Notes
Docker Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-755/2/artifact/out/Dockerfile
GITHUB PR #755
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 72d01cb3876c 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-755/out/precommit/personality/provided.sh
git revision branch-1 / 41f6713
Default Java 1.7.0_242
Multi-JDK versions /usr/lib/jvm/zulu-8-amd64:1.8.0_232 /usr/lib/jvm/zulu-7-amd64:1.7.0_242
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-755/2/artifact/out/diff-checkstyle-hbase-server.txt
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-755/2/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-755/2/testReport/
Max. process+thread count 3655 (vs. ulimit of 10000)
modules C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: .
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-755/2/console
versions git=1.9.1 maven=3.0.5 findbugs=3.0.1
Powered by Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
💙 reexec 2m 15s Docker mode activated.
_ Prechecks _
💚 dupname 0m 0s No case conflicting files found.
💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
💚 @author 0m 0s The patch does not contain any @author tags.
💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ branch-1 Compile Tests _
💙 mvndep 1m 23s Maven dependency ordering for branch
💚 mvninstall 7m 57s branch-1 passed
💚 compile 1m 32s branch-1 passed with JDK v1.8.0_232
💚 compile 1m 39s branch-1 passed with JDK v1.7.0_242
💚 checkstyle 2m 25s branch-1 passed
💚 shadedjars 4m 10s branch has no errors when building our shaded downstream artifacts.
💚 javadoc 1m 15s branch-1 passed with JDK v1.8.0_232
💚 javadoc 1m 31s branch-1 passed with JDK v1.7.0_242
💙 spotbugs 3m 20s Used deprecated FindBugs config; considering switching to SpotBugs.
💚 findbugs 4m 42s branch-1 passed
_ Patch Compile Tests _
💙 mvndep 0m 18s Maven dependency ordering for patch
💚 mvninstall 2m 34s the patch passed
💚 compile 1m 31s the patch passed with JDK v1.8.0_232
💚 javac 1m 31s the patch passed
💚 compile 1m 42s the patch passed with JDK v1.7.0_242
💚 javac 1m 42s the patch passed
💔 checkstyle 2m 7s hbase-server: The patch generated 2 new + 4 unchanged - 0 fixed = 6 total (was 4)
💚 whitespace 0m 0s The patch has no whitespace issues.
💚 shadedjars 3m 49s patch has no errors when building our shaded downstream artifacts.
💚 hadoopcheck 6m 20s Patch does not cause any errors with Hadoop 2.8.5 2.9.2.
💚 javadoc 1m 9s the patch passed with JDK v1.8.0_232
💚 javadoc 1m 30s the patch passed with JDK v1.7.0_242
💚 findbugs 5m 32s the patch passed
_ Other Tests _
💚 unit 0m 31s hbase-hadoop-compat in the patch passed.
💚 unit 0m 41s hbase-hadoop2-compat in the patch passed.
💚 unit 144m 16s hbase-server in the patch passed.
💚 asflicense 0m 57s The patch does not generate ASF License warnings.
207m 44s
Subsystem Report/Notes
Docker Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-755/3/artifact/out/Dockerfile
GITHUB PR #755
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 103ee95c33b3 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-755/out/precommit/personality/provided.sh
git revision branch-1 / 41f6713
Default Java 1.7.0_242
Multi-JDK versions /usr/lib/jvm/zulu-8-amd64:1.8.0_232 /usr/lib/jvm/zulu-7-amd64:1.7.0_242
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-755/3/artifact/out/diff-checkstyle-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-755/3/testReport/
Max. process+thread count 4510 (vs. ulimit of 10000)
modules C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: .
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-755/3/console
versions git=1.9.1 maven=3.0.5 findbugs=3.0.1
Powered by Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

HBASE-15519 Add per-user metrics with lossy counting

Introducing property hbase.regionserver.user.metrics.enabled(Default:true)
to disable user metrics in case it accounts for any performance issues
@apurtell apurtell force-pushed the HBASE-23210-branch-1 branch from dc96c24 to 5127c2e Compare October 28, 2019 16:03
@apurtell
Copy link
Contributor Author

Fixed the other checkstyle indent nit

@apurtell apurtell merged commit d16cbfe into apache:branch-1 Oct 28, 2019
@apurtell apurtell deleted the HBASE-23210-branch-1 branch October 28, 2019 16:05
this.bucketSize = (long) Math.ceil(1 / errorRate);
this.currentTerm = 1;
this.totalDataCount = 0;
this.data = new ConcurrentHashMap<>();
this.listener = listener;
calculateCurrentTerm();
executor = Executors.newSingleThreadExecutor();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this causes the RegionServer process to hang upon shutdown since we're not using daemon threads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants