bpo-45020: Identify which frozen modules are actually aliases. #28655
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ericsnowcurrently
force-pushed
the
frozen-modules-aliases
branch
from
8a954d9 to
38b3893
Compare
ericsnowcurrently
added
the
🔨 test-with-buildbots
|
🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit 38b38938b5110d8038710ed4244750dce00eed3d 🤖 If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again. |
bedevere-bot
removed
the
🔨 test-with-buildbots
ericsnowcurrently
force-pushed
the
frozen-modules-aliases
branch
from
38b3893 to
071ab26
Compare
ericsnowcurrently
changed the title
ericsnowcurrently
force-pushed
the
frozen-modules-aliases
branch
3 times, most recently
from
48ee937 to
ccadbde
Compare
ericsnowcurrently
force-pushed
the
frozen-modules-aliases
branch
from
ccadbde to
1e40678
Compare
ericsnowcurrently
requested review from
brettcannon,
encukou,
ncoghlan and
warsaw
as code owners
ericsnowcurrently
force-pushed
the
frozen-modules-aliases
branch
from
1e40678 to
7d23290
Compare
ericsnowcurrently
added a commit
that referenced
this pull request
Oct 14, 2021
Currently frozen modules do not have __file__ set. In their spec, origin is set to "frozen" and they are marked as not having a location. (Similarly, for frozen packages __path__ is set to an empty list.) However, for frozen stdlib modules we are able to extrapolate __file__ as long as we can determine the stdlib directory at runtime. (We now do so since gh-28586.) Having __file__ set is helpful for a number of reasons. Likewise, having a non-empty __path__ means we can import submodules of a frozen package from the filesystem (e.g. we could partially freeze the encodings module). This change sets __file__ (and adds to __path__) for frozen stdlib modules. It uses sys._stdlibdir (from gh-28586) and the frozen module alias information (from gh-28655). All that work is done in FrozenImporter (in Lib/importlib/_bootstrap.py). Also, if a frozen module is imported before importlib is bootstrapped (during interpreter initialization) then we fix up that module and its spec during the importlib bootstrapping step (i.e. imporlib._bootstrap._setup()) to match what gets set by FrozenImporter, including setting the file info (if the stdlib dir is known). To facilitate this, modules imported using PyImport_ImportFrozenModule() have __origname__ set using the frozen module alias info. __origname__ is popped off during importlib bootstrap. (To be clear, even with this change the new code to set __file__ during fixups in imporlib._bootstrap._setup() doesn't actually get triggered yet. This is because sys._stdlibdir hasn't been set yet in interpreter initialization at the point importlib is bootstrapped. However, we do fix up such modules at that point to otherwise match the result of importing through FrozenImporter, just not the __file__ and __path__ parts. Doing so will require changes in the order in which things happen during interpreter initialization. That can be addressed separately. Once it is, the file-related fixup code from this PR will kick in.) Here are things this change does not do: * set __file__ for non-stdlib modules (no way of knowing the parent dir) * set __file__ if the stdlib dir is not known (nor assume the expense of finding it) * relatedly, set __file__ if the stdlib is in a zip file * verify that the filename set to __file__ actually exists (too expensive) * update __path__ for frozen packages that alias a non-package (since there is no package dir) Other things this change skips, but we may do later: * set __file__ on modules imported using PyImport_ImportFrozenModule() * set co_filename when we unmarshal the frozen code object while importing the module (e.g. in FrozenImporter.exec_module()) -- this would allow tracebacks to show source lines * implement FrozenImporter.get_filename() and FrozenImporter.get_source() https://bugs.python.org/issue21736
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the list of generated frozen modules at the top of
Tools/scripts/freeze_modules.py, you will find that some of the modules have a different name than the module (or .py file) that is actually frozen. Let's call each case an "alias". Aliases do not come into play until we get to the (generated) list of modules inPython/frozen.c. (The tool for freezing modules,Programs/_freeze_module, is only concerned with the source file, not the module it will be used for.)Knowledge of which frozen modules are aliases (and the identity of the original module) normally isn't important. However, this information is valuable when we go to set
__file__on frozen stdlib modules. This change updatesTools/scripts/freeze_modules.pyto map aliases to the original module name (orNoneif not a stdlib module) inPython/frozen.c. We also add a helper function inPython/import.cto look up a frozen module's alias and add the result of that function to the frozen info returned fromfind_frozen().https://bugs.python.org/issue45020