Support native system access from wasm

[lucacasonato]

This commit adds support for native system access for WASM. This is used
through a new wasmSystemAccess option on the esbuild.initialize
options bag. This options bag can contain either a specifier or
namespace for a Node.js compatible fs and process module. During
setup these are injected into globalThis.fs and globalThis.process,
where Go's WASM runtime uses them to back the regular Go FS APIs.
Finally, if this option is set, we set the hasFS option on
createChannel, and everything just works :)

Example for Deno:

import * as esbuild from "./deno/wasm.js"

await esbuild.initialize({ wasmSystemAccess: {
  fsSpecifier: "node:fs",
  processSpecifier: "node:process"
} })

await esbuild.build({
  format: "esm",
  entryPoints: ["./lib/deno/mod.ts"],
  target: "esnext",
  outfile: "./out.js"
})

Closes #690

[lucacasonato]

The test failure on Windows is related to denoland/deno#18053. Shall I ignore Windows for now?

[lucacasonato]

denoland/deno#18053 is fixed in Deno canary. However this still does not work on Windows, because Go assumes unix-style paths when running within WASM, which is incompatible with Windows backed fs syscalls. This is an upstream limitation in Go's WASM target. As such, I have disabled support on Windows for now and added a doc comment.

@evanw this is now ready for review.

[evanw]

However this still does not work on Windows, because Go assumes unix-style paths when running within WASM, which is incompatible with Windows backed fs syscalls. This is an upstream limitation in Go's WASM target. As such, I have disabled support on Windows for now and added a doc comment.

Yes, this is unfortunate. I have filed an issue with Go about this and they have indicated that they do not plan on fixing this: golang/go#43768. However, I have tried to work around this by forking the Go standard library and making the Unix-vs-Windows behavior a dynamic run-time decision instead of a static compile-time decision:

esbuild/internal/fs/fs_real.go

Lines 67 to 73 in 3b38ce3

	if CheckIfWindows() {
	fp.isWindows = true
	fp.pathSeparator = '\\'
	} else {
	fp.isWindows = false
	fp.pathSeparator = '/'
	}

Specifically the CheckIfWindows() function is a static compile-time decision for native builds (see Unix vs Windows) except for WASM where it happens at run-time. So esbuild is not actually using Go's built-in path manipulation code at all, and Go's limitations regarding this edge case should not apply. AFAIK this has been working fine when running esbuild with node's WebAssembly implementation and node's fs API.

So in theory esbuild should use Windows-style paths on Windows even with WASM and this should work. I wonder why this isn't working with your implementation. Ideally this would work on Windows too, as Windows is a platform that esbuild works hard to support natively. Do you know why esbuild's Windows-style path manipulation isn't happening in your case given that esbuild has built-in support for it? Is CheckIfWindows() not working in your case, or is it something else?

[evanw]

Regarding this PR:

My first impression of this is that the wasmSystemAccess API seems too low-level. I think ideally this would just work without needing to configure things like this. You can already use esbuild's WASM-based API in node with the file system without extra configuration, and it seems like it would be a shame for esbuild's Deno API to be unnecessarily more complicated than esbuild's node API.

Also I don't necessarily want to expose the ability to inject a custom file system like this. That's already an existing feature request (#690) and I'd like to expose that ability more uniformly to all API environments that esbuild supports instead of only to Deno + WASM.

[jakebailey]

Go just landed GOOS=wasip1; is it possible that this could be an easier way to embed things rather than extending the API? May help if you already embed a wasi runtime.

Here's it working with wazero, passing through my env and FS:

[jakebailey]

Took a couple weeks of trying all of the WASI shims + a fixed os.Open bug in Go, but, I got my playground to start using WASI (so long as you limit yourself to CLI args; working on that UX).

Here's an example.

So, to answer my own question, yes, it's very possible!

[nestarz]

thanks @lucacasonato , any plan to have your PR merged or deliver forked wasm binaries for use (for example in Deno Fresh) ?