Unable to use LTO in Bazel.

[PiotrSikora]

For context, I'm working on updating Proxy-Wasm C++ SDK from our custom toolchain to @emsdk (see: proxy-wasm/proxy-wasm-cpp-sdk#132) and I'm seeing some regressions in terms of features.

$ bazel build -c opt //example/...

[...]
Target //example:http_wasm_example.wasm up-to-date:
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.js
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.wasm
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.wasm.map
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.js.mem
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.fetch.js
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.worker.js
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.data
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.js.symbols
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.wasm.debug.wasm
  bazel-bin/example/http_wasm_example.wasm/proxy_wasm_http_wasm_example.html

bazel build -s -c opt --linkopt=-flto //example/...

[...]
Exception: FROZEN_CACHE is set, but cache file is missing: "sysroot/lib/wasm32-emscripten/lto/crt1_reactor.o" 

Same without --no-entry for commands:

Exception: FROZEN_CACHE is set, but cache file is missing: "sysroot/lib/wasm32-emscripten/lto/crt1.o"

[sbc100]

I think the problem is that we don't build all libraries in all configurations when building the emsdk tar archives.

In order to fix this we would need to have bazel create additional libraries, perhaps in a supplemental archive, which are also archived somewhere. I'm not sure what the best way to do that might be? @dschuff @kripken .. should be look into adding pre-built lto versions of all the standard libraries to the emsdk download? How much do we care about marginal increases in download size?

[PiotrSikora]

But that used to work, since previously those missing libraries were built on demand.

However, @emsdk sets FROZEN_CACHE=True, and Bazel stores Emscripten's cache on a read-only mountpoint, which prevents that from happening.

[sbc100]

Yes, the bazel toolchain does not support the "build libraries on demand" feature of emscripten. I think this is generally a good thing.. one does not generally want the contents of the sysroot to change during a build.

[PiotrSikora]

Yeah, I agree it's a good thing, once things are working :)

[kripken]

If there is a strong reason to add more builds by default that might make sense. But the matrix of combinations is constantly increasing...

In general I think the right workflow would be to invoke embuilder.py to generate the necessary libraries before anything else, using something like this for LTO:

embuilder.py build MINIMAL --lto

That is, to fully construct the necessary sysroot first. Could the bazel support do that automatically perhaps? (Does it have the necessary information? Sorry, I'm not familiar with bazel.)

[sbc100]

I think the problem is that bazel pulls the .tar.gz files that we generate.

@PiotrSikora @walkingeyerobot do you know if we can run a kind of "post-download" step on the downloaded archive? That would be a good solution I think.. as long as it was a just a one-off thing that run only on first download.

[PiotrSikora]

You can override the patch_cmds to do whatever post-processing you want, but it's a bit of a hack.

This is what we're currently doing:

http_archive(
    name = "emscripten_toolchain",
    build_file = "@proxy_wasm_cpp_sdk//:emscripten-toolchain.BUILD",
    patch_cmds = [
        "./emsdk install 2.0.7",
        "./emsdk activate --embedded 2.0.7",
    ],
    strip_prefix = "emsdk-2.0.7",
    url = "https://github.com/emscripten-core/emsdk/archive/2.0.7.tar.gz",
    sha256 = "ce7a5c76e8b425aca874cea329fd9ac44b203b777053453b6a37b4496c5ce34f"
)

[sbc100]

In that case I think you should be able to run ./upstream/emscripten/embuilder build SYSTEM --lto to get the libraries you need pre-built.

We should also do this for the un-patched toolchain I think.

[PiotrSikora]

In that case I think you should be able to run ./upstream/emscripten/embuilder build SYSTEM --lto to get the libraries you need pre-built.

This actually doesn't work. While patch_cmds could be used when building @emscripten_toolchain from sources, this is not something that can be easily used with @emsdk, which downloads prebuilt binaries.

I tried to make this work, patching @emsdk to use patched @emscripten_toolchain instead of the prebuilt binaries, but re-building everything and installing npm dependencies twice, and maintaining hacks on top of @emsdk doesn't seem to be worth the trouble.

Is there a chance that we could include LTO system in the official binaries? Looks like I'm not the only one that asked about this (see: #807).

[sbc100]

emsdk still relies on shipping just a subset of libraries pre-built, while leaving some libraries to be built on demand. This allows is to strike a balance between build time and download size and user convenience. Even if we start shipping LTO libraries there are also maybe other configuration for the system libraries: LTO + debug, -fPIC, -fPIC + debug, -fPIC + LTO + debug, ThinLTO, -fPIC + ThinLTO...etc.. I think we want avoid shipping all combinations of all libraries all the time.

I wonder if we can come up with a way to allow bazel to allow built-on-demand libraries? Or maybe to allow users to inject their own overlay of extra pre-built libraries.

Possible solutions:

1. Allow bazel users to inject extra/overlay archives of system libraries
2. Allow bazel builds to build libraries on demand in a temporary location
3. Describe the system libraries themselves as BUILD files and have them build by Bazel just like user code (I'm guess bazel has some kind of clever cache here that makes this efficient, unlike (2))
4. Build a separate tar archive of a hyper-sysroot that includes all the prebuilt libraries (not sure how/when to build and upload this).

Having said all that as a short term solution we could consider adding just one more configuration (i.e. doubling the amount of libraries we include in the tar archive).

[PiotrSikora]

• Describe the system libraries themselves as BUILD files and have them build by Bazel just like user code (I'm guess bazel has some kind of clever cache here that makes this efficient, unlike (2))

I think this would be the perfect solution (even ignoring LTO issue, prebuilt libraries should be avoided in Bazel ecosystem).

Could we get this done? cc @trybka @walkingeyerobot

[trybka]

This is definitely on our TODO list, but basically I was the only person who thought this was useful. Having another use-case is motivating.

[benjamind]

Just ran into this issue myself, would be great to get this going properly.

Is there a workaround for now I could use to get LTO?

[walkingeyerobot]

A workaround here is tricky, but doable. The problem is the system libraries don't exist with the flags relevant to lto. Right now, the bazel build is limited to what prebuilts exist, and lto prebuilts don't exist. If you were to build them yourself outside of bazel (as described here: #971 (comment) ) and then put them where bazel can see them, that should work.

I have been working to fix this. The proper fix is to have bazel build the system libraries on demand. This is tricky, but I'm making progress.

[dschuff]

So, I don't know much about bazel, but with LTO I think it's still possible to link the same system libraries that a non-LTO build would use (i.e. you can link LTO/bitcode object files together with regular object files). So it might just be a matter of juggling the linker search paths, even if you can't build LTO-enabled libraries. Emscripten does use LTO-enabled library builds by default, but I don't think it's actually necessary.

[sbc100]

Ah yeah, it should possible to simply drop -flto from the link flags. This will result your program being LTO, just not the system libraries.

All -flto does at link time is cause LTO versions of system libraries to be linked in.

[wzheng21]

To whoever having a problem with FROZEN_CACHE. In my case I was building with Memory64, then hit this problem. Then I disabled FROZEN_CACHE, then I hit the "read-only blabla" issue with bazel in sandbox mode. Here're two different workarounds that get this to build:

1. disable sandbox. Just build in standalone mode with bazel by specifying --spawn_strategy=local (standalone is deprecated)
2. with sandbox, then specify a writable folder for CACHE purpose by specifying --action_env=EM_CACHE=~/.cache/emscrpten_cache (note that you might need to specify other env var other than EM_CACHE, in my case, I need to specify EM_LLVM_ROOT as well)

In sandbox mode, bazel is allowed to write into ~/.cache folder, but not emscripten's default implementation ~/.emscripten_cache folder

[logankaser]

Does anyone know if it would be possible to change the default for the bazel tool chain from ~/.emscripten_cache to /.cache/emscrpten_cache so this will work for people?
I know how to override it for my code integration of emsdk, but probably many could benefit from this

[wzheng21]

Does anyone know if it would be possible to change the default for the bazel tool chain from ~/.emscripten_cache to /.cache/emscrpten_cache so this will work for people? I know how to override it for my code integration of emsdk, but probably many could benefit from this

I assume you meant for ~/.cache/emscripten_cache

My understanding is that it would require people to patch emscripten and build it from source and in emsdk people need to bypass prebuilt binaries (referred in emscripten_deps macro in emsdk) but rather refer to the built-from-souce version.

[sbc100]

My understanding is that it would require people to patch emscripten and build it from source and in emsdk people need to bypass prebuilt binaries (referred in emscripten_deps macro in emsdk) but rather refer to the built-from-souce version.

I don't think so no. I believe what is being proposed here is to set the cache location via EM_CACHE which does not require any patching of emscripten or the bypassing of any prebuilt binaries.

Setting EM_CACHE it an existing feature of emscripten. The effect of setting this is that that emscripten sysroot (aloung with any libraries required) will be built on first use in this location. You can use embuilder to build out the sysroot ahead of time if you like.

[allsey87]

@sbc100 @walkingeyerobot would it be possible to build the Emscripten ports when declaring the toolchain in a bazel workspace, e.g., something like:

load("@emsdk//:emscripten_deps.bzl", emsdk_emscripten_ports = "emscripten_ports")
emsdk_emscripten_ports("bzip2", "zlib", "sqlite3")

The idea being that if the dependencies of the given ports (and their flags) were already built, then it wouldn't be necessary to build/lock the cache later on, right?

[allsey87]

To whoever having a problem with FROZEN_CACHE. In my case I was building with Memory64, then hit this problem. Then I disabled FROZEN_CACHE, then I hit the "read-only blabla" issue with bazel in sandbox mode. Here're two different workarounds that get this to build:

1. disable sandbox. Just build in standalone mode with bazel by specifying --spawn_strategy=local (standalone is deprecated)
2. with sandbox, then specify a writable folder for CACHE purpose by specifying --action_env=EM_CACHE=~/.cache/emscrpten_cache (note that you might need to specify other env var other than EM_CACHE, in my case, I need to specify EM_LLVM_ROOT as well)

In sandbox mode, bazel is allowed to write into ~/.cache folder, but not emscripten's default implementation ~/.emscripten_cache folder

@wzheng21 for some reason it seems like it was sufficient for me to just set FROZEN_CACHE to False. I am building different dependencies to you (zlib, bzip2, sqlite3), but this doesn't make sense, right?