Use a tuple for processed parameter to facilitate future caching

[correctmost]

Type of Changes

	Type
✓	🔨 Refactoring

Description

I would like to cache the results of find_module soon. To make that possible, the function parameters will need to be hashable. My plan is to split up the changes to reduce the risk of regressions and to make them easier to review.

This is a follow-up to #2509 and #2516.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.98%. Comparing base (d8dbc46) to head (f4d934f).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2529   +/-   ##
=======================================
  Coverage   92.98%   92.98%           
=======================================
  Files          93       93           
  Lines       11049    11049           
=======================================
  Hits        10274    10274           
  Misses        775      775           

Flag	Coverage Δ
linux	92.85% <ø> (ø)
pypy	92.98% <ø> (ø)
windows	92.96% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
astroid/interpreter/_import/spec.py	97.52% <ø> (ø)

[Pierre-Sassoulas]

'Processed' is changing at every loop, I suppose it's never going to be the same twice ? Is it really a good candidate for caching ? (Something is appended to it right after the newly added cast to tuple)

[Pierre-Sassoulas]

Here

[correctmost]

'Processed' is changing at every loop, I suppose it's never going to be the same twice ? Is it really a good candidate for caching ? (Something is appended to it right after the newly added cast to tuple)

When I lint yt-dlp, I see repeated invocations even when processed is non-empty.

Here's one example with an empty tuple and one with a non-empty tuple:

Invocations	Method	Arguments
7895	ZipFinder.find_module	modname: 'yt_dlp' module_parts: ('yt_dlp',) processed: () submodule_path: ('yt-dlp/yt_dlp/extractor',)
948	ImportlibFinder.find_module	modname: 'InfoExtractor' module_parts: ('yt_dlp', 'extractor', 'common', 'InfoExtractor') processed: ('yt_dlp', 'extractor', 'common') submodule_path: ['yt-dlp/yt_dlp/extractor/common.py']

[Pierre-Sassoulas]

Thank you for digging into this, You're doing an amazing work with all the performance improvements and detailed benchmark results. I wonder why we're calling find_module so much with identical values in the first place. But I have too much on my plate to investigate right now.

[correctmost]

Thank you :). Some of the repetition can be explained by duplicate imports across files:

$ git grep "from .common import InfoExtractor" | wc -l
941

Caching will help in the short term, but a different strategy for visiting import nodes may be needed for maximum speed.

[Pierre-Sassoulas]

Yes definitely. My unverified intuition here is that if we make import 'canonical' and create a mapping between the canonical import and the result of find module we could avoid searching for module and only translate an import to its canonical form instead. Caching find_module directly is close enough.