GH-23221: [C++] Add support for building with Emscripten #37821

joemarshall · 2023-09-21T14:42:36Z

Split from #37696

This is just the cmake changes to enable building on emscripten.

Changes are:

Support for target system "emscripten"
Cmake preset for building libarrow ninja-release-python-emscripten (same as ninja-release-python, but with emscripten support)
Override file for cmake on emscripten, to set various build parameters that need setting to make it build there.
Changes in pyarrow cmake so it works if you are building libarrow as shared library, and also an option to enable the cmake file there to just dump the current arrow configuration, which is useful for cross-compile builds.

Closes: [C++] Add/document configuration to build with WebAssembly #23221

github-actions · 2023-09-21T14:43:03Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

joemarshall · 2023-09-21T14:53:19Z

@kou here are the cmake changes for building libarrow in emscripten.

kou

(Sorry. My review isn't completed yet. I'll continue later.)

cpp/CMakePresets.json

kou

(My first review is completed.)

cpp/cmake_modules/Emscripten/Platform/EmscriptenOverrides.cmake

docs/source/developers/cpp/emscripten.rst

cpp/CMakePresets.json

Co-authored-by: Joris Van den Bossche <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]>

…changes

kou · 2024-04-04T14:12:56Z

@github-actions crossbow submit test-ubuntu-22.04-cpp-emscripten

github-actions · 2024-04-04T14:15:18Z

Revision: d3bc3b4

Submitted crossbow builds: ursacomputing/crossbow @ actions-b7535ddb15

Task	Status
test-ubuntu-22.04-cpp-emscripten

kou · 2024-04-04T15:33:16Z

@github-actions crossbow submit -g cpp

github-actions · 2024-04-04T15:35:58Z

Revision: d3bc3b4

Submitted crossbow builds: ursacomputing/crossbow @ actions-fd564c7b19

Task	Status
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-14

kou

+1

bitsondatadev · 2024-04-05T03:38:20Z

Thanks for your reviews @kou and thanks @joemarshall for all the hard work!!

joemarshall · 2024-04-05T06:36:39Z

Brilliant, thanks everyone

conbench-apache-arrow · 2024-04-05T10:16:51Z

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 1a1d2c8.

There was 1 benchmark result indicating a performance regression:

Commit Run on ursa-i9-9960x at 2024-04-05 03:34:48Z
- dataset-selectivity (Python) with dataset=nyctaxi_multi_ipc_s3, selectivity=100%

The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.

raulcd · 2024-04-10T09:37:44Z

@github-actions crossbow submit test-build-vcpkg-win

raulcd · 2024-04-10T09:39:58Z

I am trying to understand if this PR was the one causing the test-build-vcpkg-win job failure.

github-actions · 2024-04-10T09:40:05Z

Revision: 1307910

Submitted crossbow builds: ursacomputing/crossbow @ actions-5c4dbcd949

Task	Status
test-build-vcpkg-win

pitrou · 2024-04-11T12:06:36Z

ci/docker/ubuntu-22.04-cpp.dockerfile

    apt-get clean && \
    rm -rf /var/lib/apt/lists*

+# install emscripten using EMSDK
+ARG emscripten_version="3.1.45"
+RUN cd ~ && git clone https://github.com/emscripten-core/emsdk.git && \


Hmm, two things:

it would have been better for resource usage to make a separate dockerfile for this, perhaps

could this be a script to avoid having tons of hand-coded bash code in dockerfiles?

pitrou · 2024-04-11T12:07:08Z

ci/docker/ubuntu-22.04-cpp.dockerfile

@@ -151,6 +162,9 @@ RUN if [ "${gcc_version}" = "" ]; then \
      update-alternatives --set c++ /usr/bin/g++; \
    fi

+# make sure zlib is cached in the EMSDK folder
+RUN source ~/emsdk/emsdk_env.sh && embuilder --pic build zlib


Any reason why this isn't together with the install step above?

…41178) ### Rationale for this change #37821 changed to use `add_test()` usage from old style to new style: 1a1d2c8?diff=unified&w=1#diff-1ce47eec54afaee769086e1a720c5ed65bc347cd8fc60a233de67fd895dda329L763-R764 MSVC generators multi-config generators. With old style, all tests are run without specifying `--build-config` explicitly. With new style, we need to specify `--build-config` explicitly. See also: https://cmake.org/cmake/help/latest/command/add_test.html ### What changes are included in this PR? Specify `--build-config` explicitly. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #41169 Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…ndows (apache#41178) ### Rationale for this change apache#37821 changed to use `add_test()` usage from old style to new style: apache@1a1d2c8?diff=unified&w=1#diff-1ce47eec54afaee769086e1a720c5ed65bc347cd8fc60a233de67fd895dda329L763-R764 MSVC generators multi-config generators. With old style, all tests are run without specifying `--build-config` explicitly. With new style, we need to specify `--build-config` explicitly. See also: https://cmake.org/cmake/help/latest/command/add_test.html ### What changes are included in this PR? Specify `--build-config` explicitly. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: apache#41169 Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…41178) ### Rationale for this change #37821 changed to use `add_test()` usage from old style to new style: 1a1d2c8?diff=unified&w=1#diff-1ce47eec54afaee769086e1a720c5ed65bc347cd8fc60a233de67fd895dda329L763-R764 MSVC generators multi-config generators. With old style, all tests are run without specifying `--build-config` explicitly. With new style, we need to specify `--build-config` explicitly. See also: https://cmake.org/cmake/help/latest/command/add_test.html ### What changes are included in this PR? Specify `--build-config` explicitly. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #41169 Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…e#37821) Split from apache#37696 This is just the cmake changes to enable building on emscripten. Changes are: 1) Support for target system "emscripten" 2) Cmake preset for building libarrow ` ninja-release-python-emscripten` (same as `ninja-release-python`, but with emscripten support) 3) Override file for cmake on emscripten, to set various build parameters that need setting to make it build there. 4) Changes in pyarrow cmake so it works if you are building libarrow as shared library, and also an option to enable the cmake file there to just dump the current arrow configuration, which is useful for cross-compile builds. * Closes: apache#23221 Lead-authored-by: Joe Marshall <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Co-authored-by: Joris Van den Bossche <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…ndows (apache#41178) ### Rationale for this change apache#37821 changed to use `add_test()` usage from old style to new style: apache@1a1d2c8?diff=unified&w=1#diff-1ce47eec54afaee769086e1a720c5ed65bc347cd8fc60a233de67fd895dda329L763-R764 MSVC generators multi-config generators. With old style, all tests are run without specifying `--build-config` explicitly. With new style, we need to specify `--build-config` explicitly. See also: https://cmake.org/cmake/help/latest/command/add_test.html ### What changes are included in this PR? Specify `--build-config` explicitly. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: apache#41169 Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…e#37821) Split from apache#37696 This is just the cmake changes to enable building on emscripten. Changes are: 1) Support for target system "emscripten" 2) Cmake preset for building libarrow ` ninja-release-python-emscripten` (same as `ninja-release-python`, but with emscripten support) 3) Override file for cmake on emscripten, to set various build parameters that need setting to make it build there. 4) Changes in pyarrow cmake so it works if you are building libarrow as shared library, and also an option to enable the cmake file there to just dump the current arrow configuration, which is useful for cross-compile builds. * Closes: apache#23221 Lead-authored-by: Joe Marshall <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Co-authored-by: Joris Van den Bossche <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…ndows (apache#41178) ### Rationale for this change apache#37821 changed to use `add_test()` usage from old style to new style: apache@1a1d2c8?diff=unified&w=1#diff-1ce47eec54afaee769086e1a720c5ed65bc347cd8fc60a233de67fd895dda329L763-R764 MSVC generators multi-config generators. With old style, all tests are run without specifying `--build-config` explicitly. With new style, we need to specify `--build-config` explicitly. See also: https://cmake.org/cmake/help/latest/command/add_test.html ### What changes are included in this PR? Specify `--build-config` explicitly. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: apache#41169 Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

cmake changes for emscripten

5503984

github-actions bot added Component: C++ Component: Python Component: Documentation awaiting review Awaiting review labels Sep 21, 2023

joemarshall mentioned this pull request Sep 21, 2023

GH-23221: [C++] webassembly / pyodide / emscripten build support #37696

Closed

reverted cmakelists from python

331aaa3

github-actions bot removed the Component: Python label Sep 21, 2023

kou changed the title ~~GH23221 [C++] cmake changes for emscripten build support~~ GH-23221: [C++] Add suppor for building with Emscripten Sep 22, 2023

kou reviewed Sep 23, 2023

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Sep 23, 2023

kou reviewed Sep 25, 2023

View reviewed changes

jorisvandenbossche reviewed Sep 25, 2023

View reviewed changes

docs/source/developers/cpp/emscripten.rst Outdated Show resolved Hide resolved

jorisvandenbossche reviewed Sep 25, 2023

View reviewed changes

cpp/CMakePresets.json Outdated Show resolved Hide resolved

jorisvandenbossche changed the title ~~GH-23221: [C++] Add suppor for building with Emscripten~~ GH-23221: [C++] Add support for building with Emscripten Sep 25, 2023

documentation tidying after review

3e4156e

Co-authored-by: Joris Van den Bossche <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]>

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 25, 2023

joemarshall added 5 commits September 25, 2023 20:42

preset fixes

80b21d1

updates to simplify emscripten build process

908abf9

Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…

6effae5

…changes

build fixes

31ca7f9

better comments

c26d68c

github-actions bot added Component: Python awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 25, 2023

cmake fixes for python emscripten build to work

f5ec444

Remove trailing spaces

1307910

kou approved these changes Apr 5, 2024

View reviewed changes

kou merged commit 1a1d2c8 into apache:main Apr 5, 2024
58 of 62 checks passed

kou removed the awaiting change review Awaiting change review label Apr 5, 2024

github-actions bot added the awaiting merge Awaiting merge label Apr 5, 2024

joemarshall mentioned this pull request Apr 5, 2024

pyarrow pyodide/pyodide#2933

Closed

bitsondatadev mentioned this pull request Apr 10, 2024

Add Daft Package pyodide/pyodide#4685

Open

pitrou reviewed Apr 11, 2024

View reviewed changes

kou mentioned this pull request Apr 12, 2024

GH-41169: [CI][Release] Specify --build-config explicitly on Windows #41178

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-23221: [C++] Add support for building with Emscripten #37821

GH-23221: [C++] Add support for building with Emscripten #37821

joemarshall commented Sep 21, 2023 •

edited

Loading

github-actions bot commented Sep 21, 2023

joemarshall commented Sep 21, 2023

kou left a comment

kou left a comment

kou commented Apr 4, 2024

github-actions bot commented Apr 4, 2024

kou commented Apr 4, 2024

github-actions bot commented Apr 4, 2024

kou left a comment

bitsondatadev commented Apr 5, 2024

joemarshall commented Apr 5, 2024

conbench-apache-arrow bot commented Apr 5, 2024

raulcd commented Apr 10, 2024

raulcd commented Apr 10, 2024

github-actions bot commented Apr 10, 2024

pitrou Apr 11, 2024

pitrou Apr 11, 2024

GH-23221: [C++] Add support for building with Emscripten #37821

GH-23221: [C++] Add support for building with Emscripten #37821

Conversation

joemarshall commented Sep 21, 2023 • edited Loading

github-actions bot commented Sep 21, 2023

joemarshall commented Sep 21, 2023

kou left a comment

Choose a reason for hiding this comment

kou left a comment

Choose a reason for hiding this comment

kou commented Apr 4, 2024

github-actions bot commented Apr 4, 2024

kou commented Apr 4, 2024

github-actions bot commented Apr 4, 2024

kou left a comment

Choose a reason for hiding this comment

bitsondatadev commented Apr 5, 2024

joemarshall commented Apr 5, 2024

conbench-apache-arrow bot commented Apr 5, 2024

raulcd commented Apr 10, 2024

raulcd commented Apr 10, 2024

github-actions bot commented Apr 10, 2024

pitrou Apr 11, 2024

Choose a reason for hiding this comment

pitrou Apr 11, 2024

Choose a reason for hiding this comment

joemarshall commented Sep 21, 2023 •

edited

Loading