Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-23221: [C++] webassembly / pyodide / emscripten build support #37696

Closed
wants to merge 12 commits into from

Conversation

joemarshall
Copy link
Contributor

@joemarshall joemarshall commented Sep 13, 2023

This PR adds a cmake build preset for emscripten / pyodide, and updates the setup for pyarrow so that it is possible to build a webassembly version of libarrow, and a version of pyarrow which works on pyodide.

Things that changed:

  1. Cmake preset ninja-release-emscripten-python
  2. Platform/toolchain override for cmake on emscripten, to set various build parameters
  3. Some fixes to cmake files - e.g. pyarrow can build with static libarrow now, and third party dependencies pass the toolchain file through.
  4. pyarrow knows about ARROW_ENABLE_THREADING and doesn't use threads if they are not enabled in libarrow.
  5. pyarrow sets defaults for inclusion of submodules based on their inclusion in the arrow build. e.g. pyarrow.parquet is built only if ARROW_PARQUET is set. This makes it possible to build in situations where you don't have access to set the build environment variables (e.g. in cross compiling situations like pyodide). It also seems to me to make sense - I can't see why you'd want to build pyarrow without parquet if you've built arrow with it, and the inverse is obviously not possible.

@github-actions
Copy link

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

@joemarshall joemarshall changed the title webassembly / pyodide / emscripten build support GH-23221: webassembly / pyodide / emscripten build support Sep 14, 2023
@github-actions
Copy link

⚠️ GitHub issue #23221 has been automatically assigned in GitHub to PR creator.

@joemarshall
Copy link
Contributor Author

Also fixes #34996 once this is in the pyodide package tree.

@joemarshall joemarshall marked this pull request as draft September 15, 2023 11:23
@joemarshall joemarshall marked this pull request as ready for review September 15, 2023 12:36
@joemarshall
Copy link
Contributor Author

@kou I'm not sure who is the right person to review this - it's a smidgen of python / python packaging changes and cmake changes.

@kou kou changed the title GH-23221: webassembly / pyodide / emscripten build support GH-23221: [C++] webassembly / pyodide / emscripten build support Sep 15, 2023
@kou
Copy link
Member

kou commented Sep 15, 2023

Can we split this pull request to the C++ part and then the Python part for easy to review?

@joemarshall
Copy link
Contributor Author

joemarshall commented Sep 21, 2023

@kou - I've split out the cmake changes into #37821.

Python changes in #37822

@joemarshall
Copy link
Contributor Author

Closing this as all changes are in the other PRs

kou added a commit that referenced this pull request Apr 5, 2024
Split from #37696 

This is just the cmake changes to enable building on emscripten.

Changes are:
1) Support for target system "emscripten"
2) Cmake preset for building libarrow ` ninja-release-python-emscripten` (same as `ninja-release-python`, but with emscripten support)
3) Override file for cmake on emscripten, to set various build parameters that need setting to make it build there.
4) Changes in pyarrow cmake so it works if you are building libarrow as shared library, and also an option to enable the cmake file there to just dump the current arrow configuration, which is useful for cross-compile builds.

* Closes: #23221

Lead-authored-by: Joe Marshall <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
tolleybot pushed a commit to tmct/arrow that referenced this pull request May 2, 2024
…e#37821)

Split from apache#37696 

This is just the cmake changes to enable building on emscripten.

Changes are:
1) Support for target system "emscripten"
2) Cmake preset for building libarrow ` ninja-release-python-emscripten` (same as `ninja-release-python`, but with emscripten support)
3) Override file for cmake on emscripten, to set various build parameters that need setting to make it build there.
4) Changes in pyarrow cmake so it works if you are building libarrow as shared library, and also an option to enable the cmake file there to just dump the current arrow configuration, which is useful for cross-compile builds.

* Closes: apache#23221

Lead-authored-by: Joe Marshall <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
tolleybot pushed a commit to tmct/arrow that referenced this pull request May 4, 2024
…e#37821)

Split from apache#37696 

This is just the cmake changes to enable building on emscripten.

Changes are:
1) Support for target system "emscripten"
2) Cmake preset for building libarrow ` ninja-release-python-emscripten` (same as `ninja-release-python`, but with emscripten support)
3) Override file for cmake on emscripten, to set various build parameters that need setting to make it build there.
4) Changes in pyarrow cmake so it works if you are building libarrow as shared library, and also an option to enable the cmake file there to just dump the current arrow configuration, which is useful for cross-compile builds.

* Closes: apache#23221

Lead-authored-by: Joe Marshall <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
rok pushed a commit to tmct/arrow that referenced this pull request May 8, 2024
…e#37821)

Split from apache#37696 

This is just the cmake changes to enable building on emscripten.

Changes are:
1) Support for target system "emscripten"
2) Cmake preset for building libarrow ` ninja-release-python-emscripten` (same as `ninja-release-python`, but with emscripten support)
3) Override file for cmake on emscripten, to set various build parameters that need setting to make it build there.
4) Changes in pyarrow cmake so it works if you are building libarrow as shared library, and also an option to enable the cmake file there to just dump the current arrow configuration, which is useful for cross-compile builds.

* Closes: apache#23221

Lead-authored-by: Joe Marshall <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
rok pushed a commit to tmct/arrow that referenced this pull request May 8, 2024
…e#37821)

Split from apache#37696 

This is just the cmake changes to enable building on emscripten.

Changes are:
1) Support for target system "emscripten"
2) Cmake preset for building libarrow ` ninja-release-python-emscripten` (same as `ninja-release-python`, but with emscripten support)
3) Override file for cmake on emscripten, to set various build parameters that need setting to make it build there.
4) Changes in pyarrow cmake so it works if you are building libarrow as shared library, and also an option to enable the cmake file there to just dump the current arrow configuration, which is useful for cross-compile builds.

* Closes: apache#23221

Lead-authored-by: Joe Marshall <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
vibhatha pushed a commit to vibhatha/arrow that referenced this pull request May 25, 2024
…e#37821)

Split from apache#37696 

This is just the cmake changes to enable building on emscripten.

Changes are:
1) Support for target system "emscripten"
2) Cmake preset for building libarrow ` ninja-release-python-emscripten` (same as `ninja-release-python`, but with emscripten support)
3) Override file for cmake on emscripten, to set various build parameters that need setting to make it build there.
4) Changes in pyarrow cmake so it works if you are building libarrow as shared library, and also an option to enable the cmake file there to just dump the current arrow configuration, which is useful for cross-compile builds.

* Closes: apache#23221

Lead-authored-by: Joe Marshall <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
kou added a commit that referenced this pull request Jul 5, 2024
pyarrow knows about ARROW_ENABLE_THREADING and doesn't use threads if they are not enabled in libarrow.

Split from #37696 

* GitHub Issue: #41910

Lead-authored-by: Joe Marshall <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Raúl Cumplido <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
raulcd added a commit that referenced this pull request Jul 8, 2024
pyarrow knows about ARROW_ENABLE_THREADING and doesn't use threads if they are not enabled in libarrow.

Split from #37696 

* GitHub Issue: #41910

Lead-authored-by: Joe Marshall <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Raúl Cumplido <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
zanmato1984 pushed a commit to zanmato1984/arrow that referenced this pull request Jul 9, 2024
pyarrow knows about ARROW_ENABLE_THREADING and doesn't use threads if they are not enabled in libarrow.

Split from apache#37696 

* GitHub Issue: apache#41910

Lead-authored-by: Joe Marshall <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Raúl Cumplido <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[C++] Add/document configuration to build with WebAssembly
2 participants