diff --git a/pyodide/cryptography-in-pyodide.md b/pyodide/cryptography-in-pyodide.md new file mode 100644 index 0000000000..af769b22eb --- /dev/null +++ b/pyodide/cryptography-in-pyodide.md @@ -0,0 +1,106 @@ +# Cryptography in Pyodide + +Today I was evaluating if the Python [cryptography](https://cryptography.io/) package was a sensible depedency for one of my projects. + +It's clearly the most responsible package to use for implementing encryption in Python, but I was nervous about adding it as a dependency to a project that could work without using it at all (with some design changes). + +My key concern: I want this project to work running in Pyodide in WebAssembly in the future, and was nervous that `cryptography`, being written partially in Rust, wouldn't work in that environment. + +It turns out it works just fine! + +I tested it in the Pyodide REPL at https://pyodide.org/en/stable/console.html + +![Welcome to the Pyodide terminal emulator 🐍 Python 3.11.3 (main, Sep 25 2023 20:45:01) on WebAssembly/Emscripten Type 'help', 'copyright', 'credits' or 'license' for more information. >>> import micropip >>> await micropip.install('cryptography') >>> from cryptography.fernet import Fernet >>> key = Fernet.generate_key() >>> cipher_suite = Fernet(key) >>> encrypted_text = cipher_suite.encrypt(b'Secret message') >>> encrypted_text b'gAAAAABlY7AZ-OuJAEv1J4KufF8vpreyehPeejdPqluXwD0G_mfFg-Zl1AvEza6F2DXb WMEkIcKc4B0Hb0qJ457pje5FhO5Vyw==' >>> decrypted_text = cipher_suite.decrypt(encrypted_text) >>> decrypted_text b'Secret message' - then on the right the DevTools Network panel shows that this worked by loading a 4.27MB file called cryptography-39.0.2-cp311-cp311-emscripten_3_1_45_wasm32.whl from the Pyodide CDN package. +](https://github.com/simonw/til/assets/9599/94440423-af87-4c78-b413-35a31dd0a894) + +The following import code worked without errors: +```python +import micropip +await micropip.install("cryptography") +from cryptography.fernet import Fernet +``` +Then I tested it like this: +```python +key = Fernet.generate_key() +cipher_suite = Fernet(key) +encrypted_text = cipher_suite.encrypt(b"Secret message") +print(encrypted_text) +# b'gAAAAABlY7AZ-OuJAEv1J4KufF8vpreyehPeejdPqluXwD0G_mfFg-Zl1AvEza6F2DXbWMEkIcKc4B0Hb0qJ457pje5FhO5Vyw==' +decrypted_text = cipher_suite.decrypt(encrypted_text) +print(decrypted_text) +# b'Secret message' +``` + +From sniffing around in the browser DevTools network panel, it turns out Pyodide provides its own packaged version of the Cryptography package in a file called `cryptography-39.0.2-cp311-cp311-emscripten_3_1_45_wasm32.whl`. + +I found the source of this custom package in the Pyodide repository, in the [packages/cryptography](https://github.com/pyodide/pyodide/tree/main/packages/cryptography) folder. + +## More packages + +That [packages/](https://github.com/pyodide/pyodide/tree/main/packages) folder has a whole bunch of other useful modules that have been custom packaged to work with Pyodide. A few that caught my eye: + +- `Jinja2` +- `Pillow` +- `Pygments` +- `biopython` +- `fastparquet` +- `ffmpg` +- `gdal` +- `geopandas` +- `geos` +- `html5lib` +- `jsonschema` +- `libheif` +- `libwebp` +- `lxml` +- `msgpack` +- `shapely` +- `sqlalchemy` +- `xgboost` +- `sqlite3` + +Each of these comes with a `meta.yaml` file that defines how it should be compiled, plus a test Python module to verify it and optionally a set of patches to apply before compilation. + +Here's what [packages/sqlite3/meta.yaml](https://github.com/pyodide/pyodide/blob/main/packages/sqlite3/meta.yaml) looks like: + +```yaml +package: + name: sqlite3 + version: 1.0.0 # Nonsense + tag: + - always + top-level: + - sqlite3 + - _sqlite3 +source: + sha256: $(PYTHON_ARCHIVE_SHA256) + url: $(PYTHON_ARCHIVE_URL) +build: + type: cpython_module + script: | + export FILES=( + "Modules/_sqlite/blob.c" + "Modules/_sqlite/connection.c" + "Modules/_sqlite/cursor.c" + "Modules/_sqlite/microprotocols.c" + "Modules/_sqlite/module.c" + "Modules/_sqlite/prepare_protocol.c" + "Modules/_sqlite/row.c" + "Modules/_sqlite/statement.c" + "Modules/_sqlite/util.c" + ) + + embuilder build sqlite3 --pic + + for file in "${FILES[@]}"; do + emcc $STDLIB_MODULE_CFLAGS -c "${file}" -o "${file/.c/.o}" \ + -sUSE_SQLITE3 -DMODULENAME=sqlite + done + + OBJECT_FILES=$(find Modules/_sqlite/ -name "*.o") + emcc $OBJECT_FILES -o $DISTDIR/_sqlite3.so $SIDE_MODULE_LDFLAGS \ + -sUSE_SQLITE3 -lsqlite3 + + cd Lib && tar --exclude=test -cf - sqlite3 | tar -C $DISTDIR -xf - +``` +One thing I don't understand is why some pure-Python packages such as Click also [get this treatment](https://github.com/pyodide/pyodide/blob/main/packages/click/meta.yaml) - since those should work if installed directly from PyPI using `micropip.install()`.