Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_build_tables: Invalidate cache before importing generated modules #494

Merged
merged 1 commit into from
Feb 26, 2023

Conversation

mgorny
Copy link
Contributor

@mgorny mgorny commented Feb 25, 2023

Make sure to invalidate finder caches before trying to import generated modules. This is necessary according to the Python documentation: https://docs.python.org/3/library/importlib.html#importlib.invalidate_caches

This fixes a hard-to-reproduce bug that Python would be unable to find just-generated lextab.py if mtime of the current directory did not change from the moment the script was started. This could e.g. be the case if one has second-precision timestamps and removes the generated file just before starting the build, e.g.:

$ rm pycparser/lextab.py; python -m build -nw

It could also be reproduced easier by doing something like:

$ cd pycparser
$ touch .; python -B _build_tables.py
Traceback (most recent call last):
  File "/var/tmp/pycparser/pycparser/_build_tables.py", line 38, in <module>
    import lextab
ModuleNotFoundError: No module named 'lextab'

This is because the first command (rm or touch) updates the mtime of the directory to the current time. If the script is run fast enough, it manages to scan the directory and then write the new lextab.py within the same second. As a result, mtime of the directory after writing the new file is the same as when the script was started, finder does not invalidate the cache and assumes that lextab.py does not exist since it did not exist when the directory was scanned earlier.

This potentially fixes #493.

It was originally reported on https://bugs.gentoo.org/701878. Thanks to Gary E. Miller for patience in reproducing the problem and proxy-debugging it for me, as well as testing the final patch before submission.

Make sure to invalidate finder caches before trying to import generated
modules.  This is necessary according to the Python documentation:
https://docs.python.org/3/library/importlib.html#importlib.invalidate_caches

This fixes a hard-to-reproduce bug that Python would be unable to find
just-generated `lextab.py` if mtime of the current directory did not
change from the moment the script was started.  This could
e.g. be the case if one has second-precision timestamps and removes
the generated file just before starting the build, e.g.:

    $ rm pycparser/lextab.py; python -m build -nw

It could also be reproduced easier by doing something like:

    $ cd pycparser
    $ touch .; python -B _build_tables.py
    Traceback (most recent call last):
      File "/var/tmp/pycparser/pycparser/_build_tables.py", line 38, in <module>
        import lextab
    ModuleNotFoundError: No module named 'lextab'

This is because the first command (`rm` or `touch`) updates the mtime
of the directory to the current time.  If the script is run fast enough,
it manages to scan the directory and then write the new `lextab.py`
within the same second.  As a result, mtime of the directory after
writing the new file is the same as when the script was started, finder
does not invalidate the cache and assumes that `lextab.py` does not
exist since it did not exist when the directory was scanned earlier.

This potentially fixes eliben#493.

It was originally reported on https://bugs.gentoo.org/701878.
Thanks to Gary E. Miller for patience in reproducing the problem
and proxy-debugging it for me, as well as testing the final patch before
submission.
Copy link
Owner

@eliben eliben left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@eliben eliben merged commit 35a279e into eliben:master Feb 26, 2023
@mgorny mgorny deleted the inval-caches branch February 26, 2023 04:52
@mgorny
Copy link
Contributor Author

mgorny commented Feb 26, 2023

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ModuleNotFoundError: No module named 'lextab'
2 participants