-
-
Notifications
You must be signed in to change notification settings - Fork 583
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Javascript Web WASM Fix #1247
Javascript Web WASM Fix #1247
Conversation
Thanks! So, wait... You actually have GDExtensions working in a web build!? My understanding was that we were still blocked by an Emscripten bug. I'll try to test this myself when I have a chance. |
I just tried using your PR with the Summator example - everything seems to have compiled and built fine (using Emscripten 3.1.45). I tested this with web export templates built from Godot Am I missing a step? Your hosted demo seems to be working for me. |
I had built with the version 4.1.1-stable version of Godot, haven't tested with master branch yet. You need to enable dlink as part of the web template. I'll test with the latest master branch and see if I run into any error. Is this the Summator repo you were using? - https://github.com/paddy-exe/GDExtensionSummator |
Yep, there's a different error about loadDynamicLibrary not being supported without doing that, and my testing got past that point.
Yes |
Added some changes to get build working with web export for demo - https://github.com/nicholas-maltbie/GDExtensionSummator/tree/nickmaltbie/javascript-wasm-fix Hey @dsnopek , I was looking into that bug from the latest master and I was able to repo the error, however it is working for build version 4.2-dev5 (I included some notes below). Current build with no modifications # checkout default commit
cd godot-cpp
git fetch origin
git checkout godot-4.1-stable
cd ..
# build godot-cpp
scons --directory godot-cpp
scons platform=javascript --directory godot-cpp
scons
scons platform=javascript
# export project using template
# I have godot-v4.1.1-stable installed in my cli
godot -v -e --headless --path game --quit
mkdir -p game/builds/web-gl
godot --headless --path game --export-debug Web
# Host local server for test
npx local-web-server --cors.embedder-policy "require-corp" --cors.opener-policy "same-origin" --directory .\game\builds\web-gl Attempt failed with v4.1.1-stable and no modifications to godot-cpp
When I added the patch, it seems to work fine: # Add patch I wrote to v4.1.1-stable
cd godot-cpp
git clean -xdf
# checkout default commit
git fetch origin
git checkout godot-4.1-stable
# Add remote for my patch for v4.1.1-stable
git remote add nickmaltbie git@github.com:nicholas-maltbie/godot-cpp.git
git fetch nickmaltbie
git cherry-pick 8cc7abd45f9a8821b8ac8f6454842043228544c7
cd ..
# then build binaries
scons platform=windows --directory godot-cpp
scons platform=windows
scons platform=javascript --directory godot-cpp
scons platform=javascript
# export project using template
godot -v -e --headless --path game --quit
mkdir -p game/builds/web-gl
godot --headless --path game --export-debug Web
# Host local server for test
npx local-web-server --cors.embedder-policy "require-corp" --cors.opener-policy "same-origin" --directory .\game\builds\web-gl Works just fine Building via godot v4.2-dev (most recent development release according to download archives for v4.2 - https://godotengine.org/download/archive/). # Reset any files
git clean -xdf
# Setup godot-cpp files for most recent fix
cd godot-cpp
git clean -xdf
git checkout nickmaltbie/javascript-wasm-fix
cd ..
# Rebuild godot-cpp
scons --directory godot-cpp
scons platform=javascript --directory godot-cpp
scons
scons platform=javascript
# build project with godot_4_2_dev
$godot_4_2_dev = "C:\Users\nickd\OneDrive\Programs\Godot\Godot_v4.2-dev5_win64.exe\Godot_v4.2-dev5_win64.exe"
&$godot_4_2_dev -v -e --headless --path game --quit
mkdir -p game/builds/web-gl
&$godot_4_2_dev --headless --path game --export-debug Web
# Host local server for test
npx local-web-server --cors.embedder-policy "require-corp" --cors.opener-policy "same-origin" --directory .\game\builds\web-gl Also works just fine Build godot from master branch # From the godot repo
scons target=editor
scons platform=web dlink_enabled=yes target=template_debug
# From the GDExtensionSummator repo
# Reset any files
git clean -xdf
# Setup godot-cpp files for most recent fix
cd godot-cpp
git clean -xdf
git checkout nickmaltbie/javascript-wasm-fix
cd ..
# Rebuild godot-cpp
scons --directory godot-cpp
scons platform=javascript --directory godot-cpp
scons
scons platform=javascript
# build project with latest master build
$godot_latest = "D:\Projects\godot\bin\godot.windows.editor.x86_64.exe"
&$godot_latest -v -e --headless --path game --quit
mkdir -p game/builds/web-gl
&$godot_latest --headless --path game --export-debug Web
# Host local server for test
npx local-web-server --cors.embedder-policy "require-corp" --cors.opener-policy "same-origin" --directory .\game\builds\web-gl Was able to repro the error
Windows build works just fine however
|
Looking at diff between version 4.2-dev5 and latest master of godot repo Some changes in platform/web/detect.py # Embree is heavy and requires too much memory (GH-70621).
("module_raycast_enabled", False), # Get version info for checks below.
cc_version = get_compiler_version(env)
cc_semver = (int(cc_version["major"]), int(cc_version["minor"]), int(cc_version["patch"]))
if env["lto"] != "none":
# Workaround https://github.com/emscripten-core/emscripten/issues/19781.
if cc_semver >= (3, 1, 42) and cc_semver < (3, 1, 46):
env.Append(LINKFLAGS=["-Wl,-u,scalbnf"]) I'm not sure exactly what these changes mean, however, we are in that EM version range (at 3.1.45) so it could be affected by that change. Looking up the issue, it seems like this is related to some import errors and excluding libraries that were causing problems - emscripten-core/emscripten#19781 This might be the cause of the error, however I'm not sure. It will take some more time to investigate but it seems like it works with build v4.2-dev5 but not with the latest master. @dsnopek are you familiar with this part of the godot codebase? |
Ok, so I gave it another shot this morning after some rest.
I was able to confirm my suspicion, I rebuilt godot master branch with EM version 3.1.14 and it works just fine I selected 3.1.14 as it's the lowest version that supports dlink and web. https://github.com/godotengine/godot/blob/c12d63556b5c1da03a00dd4c45c40e60bd8d68c2/platform/web/detect.py#L213-L216 if env["dlink_enabled"]:
if cc_semver < (3, 1, 14):
print("GDExtension support requires emscripten >= 3.1.14, detected: %s.%s.%s" % cc_semver)
sys.exit(255) Then install emsdk v3.1.14 # install emsdk v3.1.14
cd 'D:\Projects\emsdk\'
.\emsdk.bat install 3.1.14
.\emsdk.bat activate 3.1.14 Then we can rebuild the target for godot with emcc v3.1.14 emcc --version
Once we have the right version, we can rebuild the export tempalte. # in the godot repo, rebuild the web_dlink_debug.zip
scons platform=web dlink_enabled=yes target=template_debug
# I then had to copy it to the export templates folder and rename it to web_dlink_debug.zip
# for windows, it's here - %APPDATA%\Godot\export_templates\4.2.dev
# From the GDExtensionSummator project, we can rebuild
# Reset any files
git clean -xdf
# Setup godot-cpp files for most recent fix
cd godot-cpp
git clean -xdf
git checkout nickmaltbie/javascript-wasm-fix
cd ..
# Rebuild godot-cpp
scons --directory godot-cpp
scons platform=javascript --directory godot-cpp
scons
scons platform=javascript
# build project with latest master build
$godot_latest = "D:\Projects\godot\bin\godot.windows.editor.x86_64.exe"
&$godot_latest -v -e --headless --path game --quit
mkdir -p game/builds/web-gl
&$godot_latest --headless --path game --export-debug Web
# Host local server for test
npx local-web-server --cors.embedder-policy "require-corp" --cors.opener-policy "same-origin" --directory .\game\builds\web-gl |
Given this information that it works with emscripten version 3.1.14, I think the simplest option to handle this issue going forward would be to make a PR to main godot repo to build export templates with EM version 3.1.14 when building the dlink export since we know that 3.1.14 is stable while there are known issues with emscripten between versions 3.1.42 and 3.1.45 that cause the dlink build to fail. Although this is my first time contributing to the godot projects, is there any protocol that should be followed when making these kins of fixes? |
I was trying to build for web and I successfully accomplished it with your changes. However, I'm running into the following error: Linux. It seems that the final runtime is not building properly since it's missing that function. Also, if I print that rtenv Object, a bunch of errors show up. Which suggests there are many env variables missing. Any pointers as to why this is happening? |
I got rid of this by checking "Extension Support" in export settings. See if that helps! |
@Virus-Axel at least there's progress! [UPDATE] Which I believe is where everyone is at this point in time. I'll dive into this later to see if there's any solution. |
Hey @Virus-Axel, I was able to get it working, I posted notes and followed the same steps, happy you were able to get it working as well.
Also, @KoltPenny, I believe those messages are warnings, it seems like the engine.js for web builds with 4.x as this error occurs even when GDExtensions is disabled. Reading up on the wiki for exporting for web there seems to be some parts still in progress. https://docs.godotengine.org/en/stable/tutorials/export/exporting_for_web.html#audio |
Hi @nicholas-maltbie! I did get a gdextension running on web. However, it seems like it does not work on Firefox on linux. I tried your demo too but it does not work either. Works great on Chrome and chromium. What happens on firefox: it just keeps loading and no error response. I tried firefox 117.0.1 and 118.0.1 (latest ATM). Does it work on firefox-windows? |
Last time I checked (emcc 3.1.45), threads + dynamic linking was broken.
@Virus-Axel what version of emscripten did you use to make the build? which godot branch/commit did you build? Which flags did you use? |
I just rebuilt godot with dlink support and the latest emcc version (3.1.46), and it seems the error is indeed still there.
|
@Faless I haven't had a chance to retest and verify, but it appears that some folks have reported success with Emscripten 3.1.14 and Godot 4.1.1 (with the godot-cpp for 4.1). Apparently, it doesn't work with either (a) newer Emscripten and (b) Godot master. |
@Faless I am using emcc version 3.1.18 in a fedora container. I don't know anything about the dynamic linking, I have a static library this is linked into my .wasm module. Like this:
Flags when compiling: |
@Virus-Axel sorry, I meant the godot compilation flags, not the godot-cpp ones. @dsnopek I will try Godot 3.1.14 and godot 4.1, but I'm pretty sure I did try that at some point and resulted in the same crash as above (IIRC 3.1.14 was the version we originally used in 4.x branch). |
To clarify, if you use a regular godot build, i.e. not built with the |
Godot's
|
Well, it seems that the official build of 4.1.1 actually works (and does not spit the |
Yes actually (FF 118.0.1 on linux), but I had to clear my cache first (which is weird, as we should have a no-cache option for local dev) |
Well, never mind, while the godot build with gdextension support works on firefox, loading the test extension in godot-cpp does not, stalling during load with no error for me (high CPU usage, dev tools unresponsive) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice job, and thanks a lot everybody for pointing out that this works on Chrom* with official 4.1.1 production builds (requires LTO to reduce the number of exported/impoerted symbols).
Could you apply this patch?
diff --git a/tools/javascript.py b/tools/javascript.py
index 3bab7de..e8f6af1 100644
--- a/tools/javascript.py
+++ b/tools/javascript.py
@@ -25,11 +25,8 @@ def generate(env):
env["ARCOM"] = "${TEMPFILE(ARCOM_POSIX)}"
# Thread support (via SharedArrayBuffer).
- env.Append(CPPDEFINES=["PTHREAD_NO_RENAME"])
env.Append(CCFLAGS=["-s", "USE_PTHREADS=1"])
env.Append(LINKFLAGS=["-s", "USE_PTHREADS=1"])
- env.Append(LINKFLAGS=["-s", "PTHREAD_POOL_SIZE=8"])
- env.Append(LINKFLAGS=["-s", "WASM_MEM_MAX=2048MB"])
# All intermediate files are just LLVM bitcode.
env["OBJPREFIX"] = ""
@@ -44,9 +41,4 @@ def generate(env):
env.Replace(SHLINKFLAGS="$LINKFLAGS")
env.Replace(SHLINKFLAGS="$LINKFLAGS")
- if env["target"] in ["editor", "template_debug"]:
- env.Append(CCFLAGS=["-O0", "-g"])
- elif env["target"] == "template_release":
- env.Append(CCFLAGS=["-O3"])
-
env.Append(CPPDEFINES=["WEB_ENABLED", "UNIX_ENABLED"])
The optimization flags are now handled by tools/targets.py
and are no longer supposed to stay in tools/javascript.py
.
The WASM_MEM_MAX
, and PTHREAD_POOL_SIZE
flags should only be relevant for the main module.
The PTHREAD_NO_RENAME
is used in internal Godot threading code (do we want to still set it in godot-cpp for consistency? CC @dsnopek )
The tool and platform should also probably be renamed |
Ok same for me. Should it be solved extensions are buildable for web too, wow. Let's get to it. |
For reference, I've opened godotengine/godot#82633 which should fix 4.2 builds (and builds without LTO in general) |
Hey, trying it out with the latest firefox on windows and it doesn't seem to be working either. but chromium based browsers are working fine.
Thanks for the suggestion @Faless, it seemed odd to include them in the file, but I wanted to make the smallest change to have the fix :) I'll apply the fix and test locally. |
Added changes to tools/javascript.py to add PFlags to fix SharedArrayBuffer memory error. Corrected some small errors in tools/javascript.py to support new target names. Also updated ci to include validation for web build.
8cc7abd
to
2b4bcbb
Compare
Thanks for the help and review @Faless. If you need any other updates let me know. Happy to help working on renaming platform to web in future as well as a follow up once we merge this change. |
Just to verify: does Firefox work for you when exporting a non-gdextension project to web? Context: I just started experimenting with Web exports for my own project on Godot 4.1 a couple days ago. Even with pure gdscript, I found Firefox would hang. Chrome would also hang if the ANGLE backend was set to OpenGL but worked fine if the ANGLE backend was set to Metal. Safari just worked. I'm hesitant to post this because I don't want to distract from the main conversation on this PR. That said, from what I've seen while testing, it's possible any remaining firefox issues are actually unrelated to gdextension or any of the changes in this PR. It seemed like the current GA Godot 4.1 web exports might fail on firefox as-is. 🤷♂️ I had planned to open an issue on the main godot repo, but wanted to test some more first. |
Wow, you are correct. I am unable to have any godot export run on firefox. Thank you for the information. If you create the issue could you post it here? :) |
Hello, https://github.com/emscripten-core/emscripten/blob/main/src/settings.js#L2162 My personal opinion is that it could be unwise to perform the switch immediately if compatibility with older emsdk versions is needed; this could just be something to keep in mind in the future (if the older option is ever removed). |
@bobziuchkovski well, it does work in my tests, you can try this link (Godot 4.1.1 export): https://fales.itch.io/godot-4-1-test-regular (confirmed to work on FF 118.0.1 on linux)
@PgBiel Given we currently build with |
For the record, I've started doing some work on update the build containers for Godot 4.2 (if I can finish early enough to validate the changes during the beta phase): godotengine/build-containers#128 So we might want to ensure Godot 4.2 works well with latest Emscripten both in godotengine/godot and here. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks so much for figuring this out!
Like @Faless wrote, we should rename the tool and the platform to web
, but I think this is fine for now, we can (and should!) further improve web builds in follow-up PRs.
Hey @Faless and @dsnopek , thanks for reviewing the PR, I see you added the tag cherrypick:4.1 to the PR. Should I create a follow up PR to merge this change (2b4bcbb) into branch for 4.1 as well, or is that something handled by the maintainers of the repo? |
This is something the maintainers (mainly, me :-)) do in batches. I've just cherry-picked this one for 4.1 in PR #1261 |
The current
tools/javascript.py
has some errors that don't work with godot 4.x, this seems to be due to the SharedArrayBuffer memory not being enabled properly when compiled for wasm.This can be fixed by adding the PThread flags.
https://github.com/godotengine/godot/blob/59139df16e7a10c3b9176f697d23b557af46601e/platform/web/detect.py#L195-L200
Additionally, I added tests via the
.github\workflows\ci.yml
as a new job: 🌐 Web (wasm32)I have verified and tested the changes with a sample project that uses GDExtensions - https://github.com/nicholas-maltbie/godot_debug_draw_3d/ (it uses the changes as a path to the godot-cpp repo). You can checkout the demo hosted here - https://nickmaltbie.com/godot_debug_draw_3d/
If you have any comments or suggestions for the change let me know.
This should help address issue #1103 as I was running into that error before adding the fix.