Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cpu/cpuset cgroup and procfs data gathering #11763

Merged
merged 12 commits into from
Oct 7, 2021

Conversation

arunramani
Copy link
Contributor

@arunramani arunramani commented Oct 1, 2021

Description

Add different ways to collect CPU availability information. This PR gets information from cpu/cpuset cgroups and from /proc/cpuinfo. Also adds 2 more cgroup monitors to emit this info.

The motivation here is to improve the way we figure out available processors since Java's availableProcessors implementation doesn't account for fractional quotas, effective CPU sets and shares. This becomes especially important in containerized setups like Kubernetes. Eventually we will extend RuntimeInfo to be smarter and when we move to Java 11, we can swap to theMetrics class.

Key changed/added classes in this PR
  • Added Cpu and CpuSet to java.util.metrics.cgroups
  • Added ProcFsUtil for procfs info
  • Added CgroupCpuMonitor and CgroupCpuSetMonitor

This PR has:

  • been self-reviewed.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.

Copy link
Contributor

@abhishekagarwal87 abhishekagarwal87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@clintropolis clintropolis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm after CI 👍

we should probably add docs to https://github.com/apache/druid/blob/master/docs/configuration/index.md#enabling-metrics and maybe to https://github.com/apache/druid/blob/master/docs/operations/metrics.md though the latter does not have the other cgroups metrics. im ok if docs are added in a follow-up PR

@abhishekagarwal87
Copy link
Contributor

@arunramani should be good to go once we eliminate the possibility of division by zero error.

@arunramani
Copy link
Contributor Author

@arunramani should be good to go once we eliminate the possibility of division by zero error.

It should just return Infinity without exceptions but I've set it to 0 since I have no idea how monitors react to infinity values.

@suneet-s suneet-s closed this Oct 6, 2021
@suneet-s suneet-s reopened this Oct 6, 2021
@clintropolis clintropolis merged commit 1578913 into apache:master Oct 7, 2021
@abhishekagarwal87 abhishekagarwal87 added this to the 0.23.0 milestone May 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants