Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use PidfdChildWatcher by default when available #87951

Merged
merged 1 commit into from
Feb 12, 2023

Conversation

bdraco
Copy link
Member

@bdraco bdraco commented Feb 12, 2023

Proposed change

Users tend to be fond of command line sensors even though they tend to be extremely inefficient (its compounded by posix_spawn not being detected on alpine which is addressed in #87958). We can make that a bit better by using PidfdChildWatcher when available since we avoid creating a separate thread per child process.

https://docs.python.org/3/library/asyncio-policy.html#asyncio.PidfdChildWatcher

PidfdChildWatcher is a “Goldilocks” child watcher implementation. It doesn’t require signals or threads, doesn’t interfere with any processes launched outside the event loop, and scales linearly with the number of subprocesses launched by the event loop. The main disadvantage is that pidfds are specific to Linux, and only work on recent (5.3+) kernels.

This is a backport from cpython 3.12 python/cpython#98024. Note that there are some additional fixes in cpython 3.12 in python/cpython#94184 when there is no event loop running in the main thread but this is not a problem we have.

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to documentation pull request:

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • The code has been formatted using Black (black --fast homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
  • Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

This is a backport from cpython 3.12

https://docs.python.org/3/library/asyncio-policy.html
> PidfdChildWatcher is a “Goldilocks” child watcher implementation. It doesn’t require signals or threads, doesn’t interfere with any processes launched outside the event loop, and scales linearly with the number of subprocesses launched by the event loop. The main disadvantage is that pidfds are specific to Linux, and only work on recent (5.3+) kernels.

python/cpython#98024
There are some additional fixes in cpython 3.12
in python/cpython#94184
when there is no event loop running in the main thread
but this is not a problem we have
@bdraco bdraco marked this pull request as ready for review February 12, 2023 20:12
@bdraco bdraco requested a review from a team as a code owner February 12, 2023 20:12
@@ -49,13 +51,46 @@ class RuntimeConfig:
open_ui: bool = False


def can_use_pidfd() -> bool:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure there is much value in adding a tests for this since its a line for line backport from 3.12

Copy link
Member

@balloob balloob left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@balloob balloob merged commit d1e1734 into dev Feb 12, 2023
@balloob balloob deleted the prepare_py3.12_child_watcher branch February 12, 2023 20:41
@github-actions github-actions bot locked and limited conversation to collaborators Feb 13, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants