feat: add Pyodide support

[hoodmane]

Resolves #1454. Also close #1002.

TODO:

• Automate Python installation in certain CI environments?

Working:

• multidict -- (requires this fix to their config)
• numpy -- builds wheel but hangs 99% of the way through test suite
• scikitlearn
• scikit-image -- works on v0.19.3, doesn't work on v0.20 because of Meson
• mypy -- NonPlatformWheelError
• ultrajson -- one test failure due to fork
• ril -- rust compilation errors
• pydantic -- one test failure due to asyncio.run

[hoodmane]

@henryiii could you approve the github actions workflows?

[hoodmane]

@henryiii is the windows_x86 failure a flake?

failed to download pip version 20.0.2, pip download exit code 1
The conflict is caused by:
    The user requested pip==20.0.2
    The user requested (constraint) pip==20.0.2
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
WARNING: Skipping page https://pypi.org/simple/pip/ because the GET request got Content-Type: .The only supported Content-Type is text/html
ERROR: Cannot install pip==20.0.2 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

[rth]

Thanks a lot for doing this work! It's certainly more work that I anticipated (without knowing much about cibuildwheel) :)

[henryiii]

@henryiii is the windows_x86 failure a flake?

Yes. Some networking issue that is hard to track down.

[henryiii]

BTW, would you like me to rebase for you? I don't mind.

[hoodmane]

I already merged main locally.

[hoodmane]

Okay I think this is working now. Would appreciate a review @henryiii, @joerick

[hoodmane]

Presumably we need documentation somewhere for the fact that Python 311 needs to be installed.

[joerick]

Splendid work on this so far @hoodmane ! Quality looks really good. Comments above.

[henryiii]

We talked about the Python requirement at PyCon a bit - I think it's best to require 3.11 instead of adding a lot of time & bandwidth for a docker image. Since pyodide is a bit behind official CPython, all CI's should have a suitable host Python available. If PyBI's land, then we might be able to make this simpler. :)

I think the name is a bit in the air. Technically, it's emscripten + version + flags. But pyodide selects a specific set, and the hope is it won't update between Python versions. So 3.12 would be a new version of emscripten + flag updates, etc. But I think that's going to be incorrect at least once (I think there's a planned 3.11 update). The hope is by the time wheels are cleared for PyPI, it will be stable. There's also the possibility that CPython will move emscripten to tier 2 (not soon, sadly, looks like only WASI is going to tier 2), in which case CPython could take over the definition of the emscripten version + flags for whatever version it started on.

This will need docs - it should be clear this is experimental and you can't upload the wheels to PyPI yet. Setting it up is a bit more involved than just adding it to a matrix, etc. I can help with docs later (traveling at conference this week).

Personally and quite selfishly, I really want the cmake bug fixed before it ships, I expect scikit-build/scikit-build-core users will want to make pyodide wheels once this ships in cibuildwheel, and can't do that due to a variable being stripped from CMake's environment by emscripten and pyodide isn't auto-fixing out-of-tree wheels. I can help fix this early next week.

[hoodmane]

Thanks for the detailed review @joerick!

[hoodmane]

pyodide isn't auto-fixing out-of-tree wheels

You're talking about replace_so_abi_tags()? I think we can release a Pyodide 0.23.3 that calls that for out of tree wheels.

@ryanking13 do you see any reason not to call replace_so_abi_tags() when building out of tree? This probably fits into your project of making the out of tree builds more the same as the in-tree ones.

Of course it would be great if we could also fix the issue with Emscripten, but it seems potentially easier to change Pyodide.
emscripten-core/emscripten#19301

[ryanking13]

@ryanking13 do you see any reason not to call replace_so_abi_tags() when building out of tree? This probably fits into your project of making the out of tree builds more the same as the in-tree ones.

Sounds good. We should try to make out of tree and in tree builds identical.

[henryiii]

You're talking about replace_so_abi_tags()?

There are three fixes, any one of them would be enough:

1. Run replace_so_abi_tags() for out-of-tree builds. It's a bit of a hack, but it's a minimum to get things working.
2. Replace emcmake with pyodide's own implementation. This has to be pretty easy, it's a really simple script. This would allow scikit-build-core to get the extension right to begin with, and would support FindPython properly I think.
3. Fix emscripten to not strip this variable when using emcmake. That's a harder fix, and would require an upstream change.

I'd really like both 1 and 2 if possible. :)

[hoodmane]

Okay I think all the integration tests are passing and I think I addressed the most recent round of review comments. So this is waiting on:

• review round 2?
• fix the so_abi problems

@henryiii I'll make a PR to run replace_so_abi_tags for out of tree builds, you can work on an emcmake replacement. Then we can release pyodide 0.23.3 and update cibuildwheel to use it. Then we can merge this (assuming that I address any further review comments).

[hoodmane]

I guess it's failing because of the color / formatting in pyodide build -h.

[hoodmane]

Hmm test passes locally...

[hoodmane]

Hopefully this works? I have no idea why it passes locally though??

[hoodmane]

Seems that it fixed it indeed =)

[mayeut]

it seems too have been style/color as you pointed out.
I don't know why it passes locally either...

[henryiii]

Maybe something is checking for the GitHub actions var and assuming it is in a CI run that supports color?

[henryiii]

IMO, this is ready to merge, we can do followup patches if we find any improvements. It looks like the two outstanding issues might both be on the pyodide side, so we might not want to make a release until 0.26.1.

Unless anyone has any reservations, will merge tomorrow.

[henryiii]

Edit: I guess the manual one was on Linux,

Yes, scikit-hep/boost-histogram#938 (linux) makes it through the "pytest" command and dies on the other bug. But we were seeing this issue in cibuildwheel's tests, even on Linux.

[henryiii]

Thanks everyone!