WIP: wasi/wasm support #2738

skoppe · 2019-08-17T12:04:21Z

This pull request contains everything I need (minus 1 tiny specific ldc druntime change) to compile to WebAssembly without betterC.

This pull request does not contain a port of druntime to WebAssembly. It only contains bindings to WASI, and only as much as I needed to compile.

It does not enable any of druntime's features (e.g. no new, no class, no AA, no appender, etc.), except that it lifts the ctfe restrictions that come with betterC.

Still, lifting the ctfe restrictions that come with betterC is a major step forward if one attempts to do meta-programming with WebAssembly.

Also, this pull request could be considered a starting point to incrementally support more and more of druntime in WebAssembly/WASI.

I do not intend to merge this pull request at this time, but mostly to get feedback.

The one controversial thing about this pull request is the fact that the WASI bindings are versioned as WebAssembly. I have done this for several reasons:

targeting WASI is a simple as compiling for WebAssembly and linking with the WASI's libc. (See example for C).
by not using any of the libc functions you are effectively targeting the web. (if you compile with ldc --mtriple=wasm32-unknown-unknown-wasm and not call any of druntime functions, nothing gets linked in. No rt_init(), nothing.)

In time more WebAssembly runtimes will emerge (e.g. intel's wasm-micro-runtime), which would necessitate splitting the bindings into WASI and others.

dlang-bot · 2019-08-17T12:04:23Z

Thanks for your pull request and interest in making D better, @skoppe! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.
Please verify that your PR follows this checklist:

My PR is fully covered with tests (you can see the coverage diff by visiting the details link of the codecov check)
My PR is as minimal as possible (smaller, focused PRs are easier to review than big ones)
I have provided a detailed rationale explaining my changes
New or modified functions have Ddoc comments (with Params: and Returns:)

Please see CONTRIBUTING.md for more information.

If you have addressed all reviews or aren't sure how to proceed, don't hesitate to ping us with a simple comment.

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub fetch digger
dub run digger -- build "master + druntime#2738"

JinShil · 2019-08-17T12:18:13Z

src/core/stdc/stdio.d

+        ///
+        void rewind(FILE* stream);
+        ///
+        pure void clearerr(FILE* stream);


I think placing pure at the end of the function declaration is more standard idiomatic-D.

JinShil · 2019-08-17T12:19:39Z

src/core/stdc/fenv.d

@@ -783,11 +790,20 @@ else
            FE_TOWARDZERO   = 0x1, ///
        }
    }
+    else version (WebAssembly) // TODO: needs to be WASI


I think you can add the WASI identifier using this PR (dlang/dmd#7202) as an example.

JinShil · 2019-08-17T12:24:02Z

src/core/stdc/errno.d

@@ -2037,6 +2037,142 @@ else version (Haiku)
    enum B_NO_TRANSLATOR                 = (B_TRANSLATION_ERROR_BASE + 1);
    enum B_ILLEGAL_DATA                  = (B_TRANSLATION_ERROR_BASE + 2);
 }
+else version(WebAssembly) {


curly braces on new line

JinShil · 2019-08-17T12:31:29Z

targeting WASI is a simple as compiling for WebAssembly and linking with the WASI's libc.

Is WASI's libc even necessary? Can druntime be ported to WASI without using any libc implementations?

CyberShadow · 2019-08-17T12:48:35Z

How is WASM + a libc different from emscripten, which can also compile to WASM and has its own libc?

The biggest problem with targeting JS/WASM is the garbage collector. Any plans on how to tackle that?

skoppe · 2019-08-17T14:01:00Z

How is WASM + a libc different from emscripten, which can also compile to WASM and has its own libc?

WASI is the official system interface for WebAssembly. Its scope is beyond the browser. See for instance lucet from fastly, which allows you to run code on their edge network. Since it is the official interface I expect its adoption to grow.

Emscripten does a lot more. Not only does it implement libc, it is meant to make porting existing codebases to the web require little change. It will transform your SDL calls to WebGL for instance. As such it encompasses a lot more than just WASI. At the cost of being complex, and as far as I known, Emscripten is only targeting the web.

If you want to compile an existing codebase to the web (like autocad did), the easiest route is emscripten, by far. When you write new code it is a whole different matter, and emscripten can be bloated at times, as well as feel a little bit stuck in the past. For instance, emscripten is still busy converting their complex toolchain (llvm ir -> asm.js -> wasm) to use llvm's wasm code generation directly. With WASI I can just use a dub and ldc, or clang for c/c++.

As an example, I have compiled imgui with WASI and wrote a D webgl backend, see skoppe/spasm-imgui. Someone else did this with emscripten as well - schteppe/imgui-wasm - but the resulting binary is 2 times as big (while showcasing less of imgui). Also, I am just using dub and ldc.

CyberShadow · 2019-08-17T14:09:20Z

Emscripten is only targeting the web.

IIRC the classical D hello world program works fine with Dscripten + Node.

skoppe · 2019-08-17T14:09:44Z

The biggest problem with targeting JS/WASM is the garbage collector. Any plans on how to tackle that?

Yeah I agree. That is the biggest hurdle. Right now I just want to get rid of the betterC flags since it rejects a lot during ctfe.

I have a simple GC in spasm right now. It is very experimental, and only meant to be called from JS (since it doesn't scan the stack), but it does work. It also calls the destructors.

My approach is to convert all the druntime hooks dealing with memory allocation to templates. Then use that compile time information to generate typed allocation pools. I use those as well in spasm and the have worked great so far (no typeinfo).

I haven't figured out how to scan the globals as well as the stack precisely. I have to dig deeper into druntime's GC to see how it does that.

CyberShadow · 2019-08-17T14:12:06Z

How can a GC work if it doesn't scan the stack? You can't be sure that any object is unreachable if there is even one word where a pointer could be kept that you can't scan.

skoppe · 2019-08-17T14:19:13Z

Emscripten is only targeting the web.

IIRC the classical D hello world program works fine with Dscripten + Node.

True, I forgot about node.

skoppe · 2019-08-17T14:25:06Z

How can a GC work if it doesn't scan the stack? You can't be sure that any object is unreachable if there is even one word where a pointer could be kept that you can't scan.

When WebAssembly is running in the browser, control is always returned to the main thread, at which point you have no stack left. I am leveraging that to avoid scanning the stack.

Well, it is not that I can't scan the stack, the problem is that I don't yet know how to do it precisely. WebAssembly's memory always starts at 0 so there is a lot more chance of false pointers.

CyberShadow · 2019-08-17T14:27:47Z

When WebAssembly is running in the browser, control is always returned to the main thread, at which point you have no stack left. I am leveraging that to avoid scanning the stack.

I see, but then you can't have WebWorkers (or threads in general).

Well, it is not that I can't scan the stack, the problem is that I don't yet know how to do it precisely.

How would you scan it even non-precisely? AFAIK the same approach that allows tracing GCs to scan JS/WASM stacks implies precise scanning of the stack.

skoppe · 2019-08-17T14:30:13Z

targeting WASI is a simple as compiling for WebAssembly and linking with the WASI's libc.

Is WASI's libc even necessary? Can druntime be ported to WASI without using any libc implementations?

Sure. The only thing the wasi-libc implementation does is forward libc to wasi calls. Well, and fixup any function mismatch.

I expect it to be a lot more work though, but it sounds like an awesome project.

Geod24 · 2022-07-09T22:48:57Z

Druntime have been merged into DMD. Please re-submit your PR to dlang/dmd repository.

skoppe · 2022-07-10T19:56:55Z

Druntime have been merged into DMD. Please re-submit your PR to dlang/dmd repository.

Ohh that is nice. I really need to get this going again...

Thanks for the ping.

skoppe requested review from andralex, Burgos, CyberShadow, ibuclaw, jmdavis, MartinNowak, schveiguy, wilzbach and PetarKirov as code owners August 17, 2019 12:04

JinShil reviewed Aug 17, 2019

View reviewed changes

skoppe mentioned this pull request Aug 18, 2019

Improve user experience with ldc2, incl. cross-compilation dlang/dub#1755

Merged

skoppe force-pushed the dlang-wasi branch from 6b36e3c to f6d0b9e Compare August 22, 2019 17:02

WIP: wasi support

0689d75

skoppe force-pushed the dlang-wasi branch from f6d0b9e to 0689d75 Compare August 22, 2019 17:04

dlang-bot added Needs Rebase needs a `git rebase` performed Needs Work stalled labels May 27, 2021

dkorpel mentioned this pull request Jun 17, 2022

Add WebAssembly definitions to src/core/stdc #3848

Merged

Geod24 closed this Jul 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: wasi/wasm support #2738

WIP: wasi/wasm support #2738

skoppe commented Aug 17, 2019

dlang-bot commented Aug 17, 2019

JinShil Aug 17, 2019

JinShil Aug 17, 2019

JinShil Aug 17, 2019

JinShil commented Aug 17, 2019 •

edited

Loading

CyberShadow commented Aug 17, 2019

skoppe commented Aug 17, 2019

CyberShadow commented Aug 17, 2019

skoppe commented Aug 17, 2019

CyberShadow commented Aug 17, 2019

skoppe commented Aug 17, 2019

skoppe commented Aug 17, 2019

CyberShadow commented Aug 17, 2019

skoppe commented Aug 17, 2019

Geod24 commented Jul 9, 2022

skoppe commented Jul 10, 2022

WIP: wasi/wasm support #2738

WIP: wasi/wasm support #2738

Conversation

skoppe commented Aug 17, 2019

dlang-bot commented Aug 17, 2019

Bugzilla references

Testing this PR locally

JinShil Aug 17, 2019

Choose a reason for hiding this comment

JinShil Aug 17, 2019

Choose a reason for hiding this comment

JinShil Aug 17, 2019

Choose a reason for hiding this comment

JinShil commented Aug 17, 2019 • edited Loading

CyberShadow commented Aug 17, 2019

skoppe commented Aug 17, 2019

CyberShadow commented Aug 17, 2019

skoppe commented Aug 17, 2019

CyberShadow commented Aug 17, 2019

skoppe commented Aug 17, 2019

skoppe commented Aug 17, 2019

CyberShadow commented Aug 17, 2019

skoppe commented Aug 17, 2019

Geod24 commented Jul 9, 2022

skoppe commented Jul 10, 2022

JinShil commented Aug 17, 2019 •

edited

Loading