Add more import APIs #210
Merged
Add more import APIs #210
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
indygreg
force-pushed
the
more-import-apis
branch
from
8af819c
to
6d07392
Compare
markbt
requested changes
Feb 24, 2020
python3-sys/src/import.rs
Outdated
|
|
||
| #[cfg(not(Py_3_7))] | ||
| pub fn _PyImport_FindBuiltin(name: *const c_char) -> *mut PyObject; | ||
| #[cfg(Py_3_6)] |
There was a problem hiding this comment.
This should be Py_3_7, too.
There was a problem hiding this comment.
Good catch. I force pushed a fix.
I need some of these APIs to support in-memory extension module importing in PyOxidizer. While I was adding the ones I needed, I figured I would include some other missing symbols from import.h as well.
indygreg
force-pushed
the
more-import-apis
branch
from
6d07392
to
8d2f857
Compare
indygreg
added a commit
to indygreg/PyOxidizer
that referenced
this pull request
Feb 28, 2020
There exists a dark arts mechanism for loading Windows PE files from memory. A mechanism to facilitate this is implemented in the MemoryModule library at https://github.com/fancycode/MemoryModule. I have published a `memory-module-sys` Rust crate to expose bindings to this library. This enables Rust to load DLLs from memory. Previously in PyOxidizer, we taught the embedded Python resources data structure to define the contents of a shared library extension module to be imported from memory. This commit combines the two efforts and enables the `pyembed` crate to import Python extension modules which reside in memory. Getting this working took a fair amount of effort. There were a handful of attempts that did not pan out. Some of the failed attempts appeared to work. But they were subtly broken due to e.g. the `LazyLoader` importer assuming that the `sys.modules()` entry wouldn't be modified. In the end, the final implementation emulates CPython's extension module loading mechanism as closely as possible. This was the only way I was able to preserve compatibility with `LazyLoader` (just implementing `exec_module()` without `create_module()` appears impossible - at least without writing our own lazy module implementation). While this commit produces working results, it is far from feature complete. We still do not handle library dependencies properly. We will likely need to teach the embedded resources data structure about the existence of shared library resources and dependencies from extension modules so that shared libraries can be imported from memory when an extension module is imported. Because this commit utilizes some CPython APIs outside the paved road of CPython APIs, we had to contribute support for these symbols to python3-sys (dgrunwald/rust-cpython#210). This is why we now depend on a specific Git commit of python3-sys and the cpython crates. This means we can't release pyembed to crates.io until a new version of these crates is published... We're likely a ways off from a new release, as I don't want to solidify the new embedded resources format until it has more features. So hopefully this isn't a problem...
indygreg
added a commit
to indygreg/PyOxidizer
that referenced
this pull request
Feb 28, 2020
There exists a dark arts mechanism for loading Windows PE files from memory. A mechanism to facilitate this is implemented in the MemoryModule library at https://github.com/fancycode/MemoryModule. I have published a `memory-module-sys` Rust crate to expose bindings to this library. This enables Rust to load DLLs from memory. Previously in PyOxidizer, we taught the embedded Python resources data structure to define the contents of a shared library extension module to be imported from memory. This commit combines the two efforts and enables the `pyembed` crate to import Python extension modules which reside in memory. Getting this working took a fair amount of effort. There were a handful of attempts that did not pan out. Some of the failed attempts appeared to work. But they were subtly broken due to e.g. the `LazyLoader` importer assuming that the `sys.modules()` entry wouldn't be modified. In the end, the final implementation emulates CPython's extension module loading mechanism as closely as possible. This was the only way I was able to preserve compatibility with `LazyLoader` (just implementing `exec_module()` without `create_module()` appears impossible - at least without writing our own lazy module implementation). While this commit produces working results, it is far from feature complete. We still do not handle library dependencies properly. We will likely need to teach the embedded resources data structure about the existence of shared library resources and dependencies from extension modules so that shared libraries can be imported from memory when an extension module is imported. Because this commit utilizes some CPython APIs outside the paved road of CPython APIs, we had to contribute support for these symbols to python3-sys (dgrunwald/rust-cpython#210). This is why we now depend on a specific Git commit of python3-sys and the cpython crates. This means we can't release pyembed to crates.io until a new version of these crates is published... We're likely a ways off from a new release, as I don't want to solidify the new embedded resources format until it has more features. So hopefully this isn't a problem...
indygreg
added a commit
to indygreg/PyOxidizer
that referenced
this pull request
Feb 28, 2020
There exists a dark arts mechanism for loading Windows PE files from memory. A mechanism to facilitate this is implemented in the MemoryModule library at https://github.com/fancycode/MemoryModule. I have published a `memory-module-sys` Rust crate to expose bindings to this library. This enables Rust to load DLLs from memory. Previously in PyOxidizer, we taught the embedded Python resources data structure to define the contents of a shared library extension module to be imported from memory. This commit combines the two efforts and enables the `pyembed` crate to import Python extension modules which reside in memory. Getting this working took a fair amount of effort. There were a handful of attempts that did not pan out. Some of the failed attempts appeared to work. But they were subtly broken due to e.g. the `LazyLoader` importer assuming that the `sys.modules()` entry wouldn't be modified. In the end, the final implementation emulates CPython's extension module loading mechanism as closely as possible. This was the only way I was able to preserve compatibility with `LazyLoader` (just implementing `exec_module()` without `create_module()` appears impossible - at least without writing our own lazy module implementation). While this commit produces working results, it is far from feature complete. We still do not handle library dependencies properly. We will likely need to teach the embedded resources data structure about the existence of shared library resources and dependencies from extension modules so that shared libraries can be imported from memory when an extension module is imported. Because this commit utilizes some CPython APIs outside the paved road of CPython APIs, we had to contribute support for these symbols to python3-sys (dgrunwald/rust-cpython#210). This is why we now depend on a specific Git commit of python3-sys and the cpython crates. This means we can't release pyembed to crates.io until a new version of these crates is published... We're likely a ways off from a new release, as I don't want to solidify the new embedded resources format until it has more features. So hopefully this isn't a problem...
indygreg
added a commit
to indygreg/PyOxidizer
that referenced
this pull request
Feb 28, 2020
There exists a dark arts mechanism for loading Windows PE files from memory. A mechanism to facilitate this is implemented in the MemoryModule library at https://github.com/fancycode/MemoryModule. I have published a `memory-module-sys` Rust crate to expose bindings to this library. This enables Rust to load DLLs from memory. Previously in PyOxidizer, we taught the embedded Python resources data structure to define the contents of a shared library extension module to be imported from memory. This commit combines the two efforts and enables the `pyembed` crate to import Python extension modules which reside in memory. Getting this working took a fair amount of effort. There were a handful of attempts that did not pan out. Some of the failed attempts appeared to work. But they were subtly broken due to e.g. the `LazyLoader` importer assuming that the `sys.modules()` entry wouldn't be modified. In the end, the final implementation emulates CPython's extension module loading mechanism as closely as possible. This was the only way I was able to preserve compatibility with `LazyLoader` (just implementing `exec_module()` without `create_module()` appears impossible - at least without writing our own lazy module implementation). While this commit produces working results, it is far from feature complete. We still do not handle library dependencies properly. We will likely need to teach the embedded resources data structure about the existence of shared library resources and dependencies from extension modules so that shared libraries can be imported from memory when an extension module is imported. Because this commit utilizes some CPython APIs outside the paved road of CPython APIs, we had to contribute support for these symbols to python3-sys (dgrunwald/rust-cpython#210). This is why we now depend on a specific Git commit of python3-sys and the cpython crates. This means we can't release pyembed to crates.io until a new version of these crates is published... We're likely a ways off from a new release, as I don't want to solidify the new embedded resources format until it has more features. So hopefully this isn't a problem...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I need some of these APIs to support in-memory extension module
importing in PyOxidizer. While I was adding the ones I needed, I
figured I would include some other missing symbols from
import.h as well.