Freeze all modules imported during startup. #89183
Comments
|
Currently we freeze the 3 main import-related modules into the python binary (along with one test module). This allows us to bootstrap the import machinery from Python modules. It also means we get better performance importing those modules. If we freeze modules that are likely to be used during execution then we get even better startup times. I'll be putting up a PR that does so, freezing all the modules that are imported during startup. This could also be done for any stdlib modules that are commonly imported. (also see bpo-45019 and faster-cpython/ideas#82) |
ericsnowcurrently
added
build
ericsnowcurrently
added
type-feature
|
FYI, with my branch I'm getting a 15% improvement to startup for "./python -c pass". |
|
I'm aware of two potentially problematic consequences to this change:
For the former, I'm not sure there's a way around it. We may consider the inconvenience worth it in order to get the performance benefits. For the latter, the obvious solution is to introduce a startup hook (e.g. on the CLI) like we've talked about doing for years. (I wasn't able to find previous discussions on that topic after a quick search.) |
|
Not sure whether you are aware, but the PyRun project I'm maintaining https://www.egenix.com/products/python/PyRun/ Startup is indeed better, but not as much as you might think. The big time consumer is turning marshal'ed code objects back |
That's something Guido has been exploring. :) See: faster-cpython/ideas#84 (and others) |
I'm not sure why I said "obvious". Sorry about that. |
|
I noticed nedbat un-nosied himself. Probably he didn't realize you were |
I've explored a couple of different approaches here (see the issue Eric linked to and a few adjacent ones) and this is a tricky issue. Marshal seems to be pretty darn efficient as a storage format, because it's very compact compared to the Python objects it creates. My final (?) proposal is creating static data structures embedded in the code that just *are* Python objects. Unfortunately on Windows the C compiler balks at this -- the C++ compiler handles it just fine, but it's not clear that we are able to statically link C++ object files into Python without depending on a lot of other C++ infrastructure. (In GCC and Clang this is apparently a language extension.) |
|
@guido, @mark Shannon, do you recall the other issue where folks objected to that other patch, due to local changes to source files not being reflected? Also, one thought that comes to mind is that we could ignore the frozen modules when in a dev environment (and opt in to using the frozen modules when an environment variable). |
|
I don't recall, but... You can't modify any builtin modules. Freezing modules effectively makes them builtin from a user's perspective. There are plenty of modules that can't be modified: >>> sys.builtin_module_names
('_abc', '_ast', '_codecs', '_collections', '_functools', '_imp', '_io', '_locale', '_operator', '_signal', '_sre', '_stat', '_string', '_symtable', '_thread', '_tokenize', '_tracemalloc', '_warnings', '_weakref', 'atexit', 'builtins', 'errno', 'faulthandler', 'gc', 'itertools', 'marshal', 'posix', 'pwd', 'sys', 'time', 'xxsubtype')I don't see why adding a few more modules to that list would be a problem. Was the objection to feezing *all* modules, not just the core ones? |
|
We should ask Neil S. for the issue where Larry introduced this. That might But if I had to guess, it’s confusing that you can see *Python* source that I know that occasionally a debug session I add a print statement to a
|
|
Neil, do you recall the story here? |
|
The plot thickens. By searching my extensive GMail archives for Jeethu Rao I found an email from Sept. 14 to python-dev by Larry Hastings titled "Store startup modules as C structures for 20%+ startup speed improvement?" It references an issue and a PR: Here's a link to the python-dev thread: There's a lot of discussion there. I'll try to dig through it. |
|
Adding Larry in case he remembers more color. (Larry: the key question here is whether some version of this (like the one I've been working on, or a simpler one that Eric has prepared) is viable, given that any time someone works on one of the frozen or deep-frozen stdlib modules, they will have to run make (with the default target) to rebuild the Python binary with the deep-frozen files. (Honestly if I were working on any of those modules, I'd just comment out some lines from Eric's freeze_modules.py script and do one rebuild until I was totally satisfied with my work. Either way it's a suboptimal experience for people contributing to those modules. But we stand to gain a ~20% startup time improvement.) PS. The top comment links to Eric's work. |
|
Since nobody's said so in so many words (so far in this thread anyway): the prototype from Jeethu Rao in 2018 was a different technology than what Eric is doing. The "Programs/_freeze_importlib.c" Eric's playing with essentially inlines a .pyc file as C static data. The Jeethu Rao approach is more advanced: instead of serializing the objects, it stores the objects from the .pyc file as pre-initialized C static objects. So it saves the un-marshalling step, and therefore should be faster. To import the module you still need to execute the module body code object though--that seems unavoidable. The python-dev thread covers nearly everything I remember about this. The one thing I guess I never mentioned is that building and working with the prototype was frightful; it had both Python code and C code, and it was fragile and hard to get working. My hunch at the time was that it shouldn't be so fragile; it should be possible to write the converter in Python: read in .pyc file, generate .c file. It might have to make assumptions about the internal structure of the CPython objects it instantiates as C static data, but since we'd ship the tool with CPython this should be only a minor maintenance issue. In experimenting with the prototype, I observed that simply calling stat() to ensure the frozen .py file hadn't changed on disk lost us about half the performance win from this approach. I'm not much of a systems programmer, but I wonder if there are (system-proprietary?) library calls one could make to get the stat info for all files in a single directory all at once that might be faster overall. (Of course, caching this information at startup might make for a crappy experience for people who edit Lib/*.py files while the interpreter is running.) One more observation about the prototype: it doesn't know how to deal with any mutable types. marshal.c can deal with list, dict, and set. Does this matter? ISTM the tree of objects under a code object will never have a reference to one of these mutable objects, so it's probably already fine. Not sure what else I can tell you. It gave us a measurable improvement in startup time, but it seemed fragile, and it was annoying to work with/on, so after hacking on it for a week (at the 2018 core dev sprint in Redmond WA) I put it aside and moved on to other projects. |
|
There should be a boolean flag that enables/disables cached copies of .py files from Lib/. You should be able to turn it off with either an environment variable or a command-line option, and when it's off it skips all the internal cached stuff and uses the normal .py / .pyc machinery. With that in place, it'd be great to pre-cache all the .py files automatically read in at startup. As for changes to the build process: the most analogous thing we have is probably Argument Clinic. For what it's worth, Clinic hasn't been very well integrated into the CPython build process. There's a pseudotarget that runs it for you in the Makefile, but it's only ever run manually, and I'm not sure there's *any* build automation for Windows developers. AFAIK it hasn't really been a problem. But then I'm not sure this is a very good analogy--the workflow for making Clinic changes is very different from people hacking on Lib/*.py. It might be sensible to add a mechanism that checks whether or not the pre-cached modules are current. Store a hash for each cached module and check that they all match. This could then be part of the release process, run from a GitHub hook, etc. |
Yes, I know. We're discussing two separate ideas -- Eric's approach, which is doing the same we're doing for importlib for more stdlib modules; and "my" approach, dubbed "deep-freeze", which is similar to Jeethu's approach (details in faster-cpython/ideas#84). What the two approaches have in common is that they require rebuilding the python binary whenever you edit any of the changed modules. I heard somewhere (I'm sorry, I honestly don't recall who said it first, possibly Eric himself) that Jeethu's approach was rejected because of that. FWIW in my attempts to time this, it looks like the perf benefits of Eric's approach are close to those of deep-freezing. And deep-freezing causes much more bloat of the source code and of the resulting binary. (At runtime the binary size is made up by matching heap savings, but to some people binary size is important too.)
Deep-freezing doesn't seem frightful to work with, to me at least. :-) Maybe the foundational work by Eric (e.g. generating sections of Makefile.pre.in) has helped. I don't understand entirely why Jeethu's prototype had part written in C. I never ran it so I don't know what the generated code looked like, but I have a feeling that for objects that don't reference other objects, it would generate a byte array containing the exact contents of the object structure (which it would get from constructing the object in memory and copying the bytes) which was then put together with the object header (containing the refcount and type) and cast to (PyObject *). In contrast, for deep-freeze I just reverse engineered what the structures look like and wrote a Python script to generate C code for an initialized instance of those structures. You can look at some examples here: https://github.com/gvanrossum/cpython/blob/codegen/Python/codegen__collections_abc.c . It's verbose but the C compiler handles it just fine (C compilers have evolved to handle *very* large generated programs).
I think the only solution here was hinted at in the python-dev thread from 2018: have a command-line flag to turn it on or off (e.g. -X deepfreeze=1/0) and have a policy for what the default for that flag should be (e.g. on by default in production builds, off by default in developer builds -- anything that doesn't use --enable-optimizations).
Correct, marshal supports things that you will never see in a code object. Perhaps the reason is that when marshal was invented, it wasn't so clear that code objects should be immutable -- that realization came later, when Greg Stein proposed making them ROM-able. That didn't work out, but the notion that code objects should be strictly mutable (to the python user, at least) was born and is now ingrained.
I'm not so quick to give up. I do believe I have seen similar startup time improvements. But Eric's version (i.e. this issue) is nearly as good, and the binary bloat is much less -- marshal is way more compact than in-memory objects. (Second message)
Yeah.
*All* the .py files? I think the binary bloat cause by deep-freezing the entire stdlib would be excessive. In fact, Eric's approach freezes everything in the encodings package, which turns out to be a lot of files and a lot of code (lots of simple data tables expressed in code), and I found that for basic startup time, it's best not to deep-freeze the encodings module except for __init__.py, aliases.py and utf_8.py.
I think we've got reasonably good automation for both Eric's approach and the deep-freeze approach -- all you need to do is run "make" when you've edited one of the (deep-)frozen modules.
I think the automation that Eric developed is already good enough. (He even generates Windows project files.) See #27980 . |
My dim recollection was that Jeethu's approach wasn't explicitly rejected, more that the community was more "conflicted" than "strongly interested", so I lost interest, and nobody else followed up.
My theory: it's easier to serialize C objects from C. It's maybe even slightly helpful? But it made building a pain. And yeah it just doesn't seem necessary. The code generator will be tied to the C representation no matter how you do it, so you might as well write it in a nice high-level language.
You can see an example of Jeethu's serialized objects here: Yours is generally more readable because you're using the new named structure initializers syntax. Though Jeethu's code is using some symbolic constants (e.g. PyUnicode_1BYTE_KIND) where you're just printing the actual value.
I did say "all the .py files automatically read in at startup". In current trunk, there are 32 modules in sys.module at startup (when run non-interactively), and by my count 13 of those are written in Python. If we go with Eric's approach, that means we'd turn those .pyc files into static data. My quick experiment suggests that'd be less than 300k. On my 64-bit Linux system, a default build of current trunk (configure && make -j) yields a 23mb python executable, and a 44mb libpython3.11.a. If I build without -g, they are 4.3mb and 7mb respectively. So this speedup would add another 2.5% to the size of a stripped build. If even that 300k was a concern, the marshal approach would also permit us to compile all the deep-frozen modules into a separate shared library and unload it after we're done. I don't know what the runtime impact of "deep-freeze" is, but it seems like it'd be pretty minimal. You're essentially storing these objects in C static data instead of the heap, which should be about the same. Maybe it'd mean the code objects for the module bodies would stick around longer than they otherwise would? But that doesn't seem like it'd add that much overhead. It's interesting to think about applying these techniques to the entire standard library, but as you suggest that would probably be wasteful. On the other hand: if we made a viable tool that could consume some arbitrary set of .py files and produce a C file, and said C file could then be compiled into a shared library, end users could enjoy this speedup over the subset of the standard library their program used, and perhaps even their own source tree(s). |
|
[Larry]
I took Larry's PR and did a fair amount of cleanup on it to make the build less I didn't make any attempt to work on the serializer, other than to make it work https://github.com/nascheme/cpython/tree/static_frozen It was good enough to pass nearly(?) all tests and I did some profiling. It helped reduce startup time quite a bit. |
|
On 28.08.2021 06:06, Guido van Rossum wrote:
Eric's approach, as I understand it, is pretty much what PyRun does. For Python 3.8 (I haven't ported it to more recent Python versions yet), -rwxr-xr-x 1 lemburg lemburg 15M May 19 15:26 pyrun3.8 There's no bloat, since you don't need the .py/.pyc files for the stdlib That said, some things don't work with such an approach, e.g. For PyRun I have patched some of those packages to include the Whether this is a good approach for Python in general is a different |
|
When I investigated freezing the standard library for PyOxidizer, I ran into a rash of problems. The frozen importer doesn't behave like PathFinder. It doesn't (didn't?) set some common module-level attributes that are documented by the importer "specification" to be set and this failed a handful of tests and lead to runtime issues or breakage in 3rd party packages (such as random packages looking for a __file__ on a common stdlib module). Also, when I last looked at the CPython source, the frozen importer performed a linear scan of its indexed C array performing strcmp() on each entry until it found what it was looking for. So adding hundreds of modules could result in sufficient overhead and justify using a more efficient lookup algorithm. (PyOxidizer uses Rust's HashMap to index modules by name.) I fully support more aggressive usage of frozen modules in the standard library to speed up interpreter startup. However, if you want to ship this as enabled by default, from my experience with PyOxidizer, I highly recommend:
|
|
Oh, PyOxidizer also ran into more general issues with the frozen importer in that it broke various importlib APIs. e.g. because the frozen importer only supports bytecode, you can't use .__loader__.get_source() to obtain the source of a module. This makes tracebacks more opaque and breaks legitimate API consumers relying on these importlib interfaces. The fundamental limitations with the frozen importer are why I implemented my own meta path importer (implemented in pure Rust), which is more fully featured, like the PathFinder importer that most people rely on today. That importer is available on PyPI (https://pypi.org/project/oxidized-importer/) and has its own API to facilitate PyOxidizer-like functionality (https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer.html) if anyone wants to experiment with it. |
softsolsolutions
mannequin
added
3.9
|
@santhu_reddy12, why did you assign this to the Parser category? IMO this issue is clearly in the Build category. (We haven't met, I assume you have triage permissions?) |
|
And it's most definitely 3.11, not 3.9. (Did you mean to change a different issue?) |
|
Whoa. os.path is not always an alias for posixpath, is it? |
|
Nope. On Windows, os.path is "ntpath". |
|
On Tue, Oct 5, 2021 at 11:31 AM Guido van Rossum <[email protected]> wrote:
Steve brought this to my attention a couple weeks ago. Bottom line: |
|
could changes related to this be the cause of https://bugs.python.org/issue45506 ? out of tree builds in main usually cannot pass key tests today. they often hang or blow up with strange exceptions. |
|
Is #73126 only for UNIX? I built on Windows with default options (PCbuild\build.bat) and it looks like the frozen modules are used by default even though I am running in the source directory. (I put a printf() call in unmarshal_frozen_code().) I also put a printf() in is_dev_env() and found that it returns 0 on this check: /* If dirname() is the same for both then it is a dev build. */
if (len != _Py_find_basename(stdlib)) {
return 0;
}I assume that's because the binary (in my case at least) is at PCbuild\amd64\python.exe which is not the same as my current directory (which is the repo root). |
|
On Mon, Oct 18, 2021 at 7:14 PM Guido van Rossum <[email protected]> wrote:
It wasn't meant to be. :(
I'll look into this. |
|
On Mon, Oct 18, 2021 at 7:14 PM Guido van Rossum <[email protected]> wrote:
FYI, I opened https://bugs.python.org/issue45651 for sorting this out. |
|
I consider this done. There is some lingering follow-up work, for which I've created a number of issues:
|
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: