Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter to possible package paths before trying to resolve a module. #18038

Merged
merged 6 commits into from
Oct 25, 2024

Conversation

falsedrow
Copy link
Contributor

With a long sys.path (it's got 300 entries), this removes 94% of stat syscalls from running mypy. With all the filesystem caching, that's only a small time savings, though it will depend on your filesystem. Local benchmarks showed a 20% time savings but they're pretty noisy from all the I/O.

With a long sys.path (it's got 300 entries), this removes 94% of stat syscalls
from running mypy. With all the filesystem caching, that's only a small time savings,
though it will depend on your filesystem.

This comment has been minimized.

Copy link
Collaborator

@hauntsaninja hauntsaninja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome stuff! I pushed two commits to try and get tests passing, I'll also run this through the benchmarking setup that I had going :-)

This comment has been minimized.

This comment has been minimized.

@hauntsaninja
Copy link
Collaborator

hauntsaninja commented Oct 25, 2024

Okay, I benchmarked ac79825 as the following on mypy -c 'import torch' (see #17948 for description of the environments):

  • 1.09x faster in long environment
  • 1.06x faster in openai environment
  • 1.42x faster in long environment (incremental)
  • 1.22x faster in openai environment (incremental)

I'll check the extra normpath is slowing us down, if so should be very easy to save (update: looks like this might be a few 100ms, so a couple more percent)

Doing this also uncovered a minor regression from #15347 , which I shall try to fix... (update: I think best done in a new PR)

This comment has been minimized.

Copy link
Contributor

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

@hauntsaninja hauntsaninja merged commit 715b768 into python:master Oct 25, 2024
19 checks passed
hauntsaninja added a commit that referenced this pull request Oct 26, 2024
Follow up to #18038

This maybe should have been two PRs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants