The State of Path Mapping

[fmeum]

What is path mapping?

With path mapping enabled, Bazel automatically rewrites paths in action command lines with the aim of making them more likely to be disk or remote cache hits.
Specifically, configuration prefixes such as bazel-out/darwin-amd64-fastbuild/bin are replaced with a fixed string such as bazel-out/cfg/bin, so that the result of an action that doesn't depend on all aspects of the configuration encoded in the path (e.g. the OS, architecture and build mode) can be shared between different target platforms and configurations.

Path mapping by itself won't result in cache hits between different execution platforms because toolchains typically differ between platforms (in content, not just paths). One way to address this is to bundle toolchains for multiple execution platforms, see "Universal toolchains". Another is to use cache key scrubbing.

For more information on path mapping, watch the "Towards faster cross-platform builds" talk.

Using path mapping

Bazel has experimental support for path mapping for both native and Starlark rules as detailed below.

General requirements

• At least Bazel 7.2.0rc3 or 8.0.0-pre.20240523.3 are recommended.
• Set --experimental_output_paths=strip.
• Use a disk or remote cache (or remote execution).
• Actions must use a compatible strategy:
  • remote
  • dynamic
  • sandboxed
  • multiplex-worker with --experimental_worker_multiplex_sandboxing
  • worker with --worker_sandboxing and --noworker_multiplex

Java

Java compilation actions as well as header compilation actions support path mapping.

Requirements

Since the default Java toolchain doesn't enable support for multiplex sandboxing yet, non-remote execution with path mapping requires using a custom Java toolchain:

load("@bazel_tools//tools/jdk:default_java_toolchain.bzl", "default_java_toolchain")
default_java_toolchain(
    name = "java_toolchain",
    source_version = "17",
    target_version = "17",
    javac_supports_worker_multiplex_sandboxing = True,
)

rules_java 7.5.0 and higher have support for multiplex sandboxing.

C/C++

C++ compilation actions have experimental support for path mapping.

Requirements

Bazel 7.4.0 or higher is required and path mapping needs to be explicitly enabled for C/C++ actions via --modify_execution_info=CppCompile=+supports-path-mapping,CppModuleMap=+supports-path-mapping.

Go

rules_go optionally uses path mapping for its non-cgo compilation actions, including the stdlib.

Requirements

rules_go from HEAD including this commit.

Custom Starlark rules

Custom Starlark rules can opt into path mapping by adding an entry with the key supports-path-mapping to the execution_requirements dict passed to ctx.actions.run or ctx.actions.run_shell.
They can also be opted in from the command line via --modify_execution_info.

Additionally, command lines need to be constructed using ctx.actions.args and avoiding any of the methods on File that return a path string. Assuming that src is a File representing a file and dir is a File representing a directory ("tree artifact"):

1. instead of args.add(src.path), use args.add(src);
2. instead of args.add(dir.path), use args.add_all([dir], expand_directories = False);
3. instead of args.add(src.dirname), use:

def _dirname(file):
    return file.dirname

def _my_rule_impl(ctx):
    ...
    args.add_all([src], map_each = _dirname)

Generally speaking, all functions that return a path string that may contain a configuration prefix such as bazel-out/darwin-amd64-fastbuild/bin must only be called in map_each callbacks, where they are automatically path mapped by Bazel.

[illicitonion]

FWIW rules_pkl is a starlark rule which supports path mapping, see https://github.com/apple/rules_pkl/blob/bf3de08ea8c7f05329a2d59d9790ec8653d2a28c/pkl/private/pkl.bzl#L232 for an example.

[illicitonion]

When implementing support, we ran into #16430 (comment) - @fmeum I think the discussion there is currently waiting on a reply from you?

[fmeum]

I'll try to get back to it soon, but the main part of the discussion (what's the API going to look like, is Args supported, ...) is still very much open.

[lior10r]

Do you think that c++ support would be available at the next version (7.3.0)?

[fmeum]

I can't promise anything, but I would say that's very likely. The main PR has already been approved and the follow-ups are pretty small.

[fmeum]

Basic support landed in 7.3.0, please see the updated main post

[lior10r]

Thanks! We started to try and working with it, at the moment our remote cache doesn't work. When we would manage to get I working I would let you know how it affected our build time.

[lior10r]

Hi, We started to work on integrating path mapping with our project and I have a question.
I saw that only the CppCompile supports path mapping and other cpp actions like CppArchive doesn't, is this true or I just didn't succeed to enable path mapping to this action? And if it really doesn't support path mapping is it planned to be supported in the future?

[fmeum]

This is correct, but adding support should be pretty simple. I will look into it for 8.1.0.

If this is a bottleneck for you, you could try switching to a linker with support for --start-lib/--end-lib to avoid building static libraries in the first place.

[ThomasCJY]

Is there any plan to support android?

[ThomasCJY]

Yes, I pushed the updates on my test repo https://github.com/ThomasCJY/testApp/tree/jchen-testPathMapping which registered the custom java toolchain and it still failed with the same error

[ThomasCJY]

@fmeum did you get any chance to test it out?

[fmeum]

I tried to get it to work, but I didn't manage to get to this point: Android Studio fails to install build-tools 30.0.2 for me, but when I set build_tools_version = "31.0.0", the Bazel build still fails with references to @android_sdk//build-tools/30.0.2:aapt2 missing, even after bazel clean --expunge. Do you know how I can get them updated?

That said, I think that you are missing a flag (the description of this discussion didn't make it clear that you need it): try adding --noworker_multiplex.

[ThomasCJY]

yeah you are right. Adding --noworker_multiplex fixed the build issue. Any idea why I am still seeing darwin_arm64-xxx directories under bazel-out while no bazel-out/cfg/ is generated?

ls bazel-out/
_tmp
darwin_arm64-opt-exec-ST-d57f47055a04
darwin_arm64-fastbuild
darwin_arm64-fastbuild-android-ST-5b74a929aefd

[fmeum]

This is expected and actually a feature: path mapping only applies at execution time to ensure that downstream actions see all the files in the expected locations, regardless of whether they use path mapping themselves. Try running with -s to see the mapped paths.

[fmeum]

Edit: Path mapping in 7.3.0 is currently broken due to race conditions in the new action deduplication functionality. See #23288.

[fmeum]

This has been hotfixed in 7.3.1 by disabling local execution deduplication. I have a full fix out in #23382.

Performance data obtained with the patch on Bazel itself with a suitable Java toolchain:

# Baseline
INFO: Elapsed time: 102.879s, Critical Path: 24.86s
INFO: 5128 processes: 145 internal, 3076 darwin-sandbox, 1907 worker.

# --experimental_worker_multiplex_sandboxing
INFO: Elapsed time: 115.551s, Critical Path: 29.83s
INFO: 5128 processes: 145 internal, 3076 darwin-sandbox, 1907 worker.

# --experimental_worker_multiplex_sandboxing --experimental_output_paths=strip, without deduplication
INFO: Elapsed time: 95.024s, Critical Path: 27.01s
INFO: 5128 processes: 1438 disk cache hit, 145 internal, 2375 darwin-sandbox, 1170 worker.

# --experimental_worker_multiplex_sandboxing --experimental_output_paths=strip, with deduplication
INFO: Elapsed time: 80.242s, Critical Path: 22.04s
INFO: 5128 processes: 1367 disk cache hit, 145 internal, 2171 darwin-sandbox, 459 deduplicated, 986 worker.

[sputt]

Just started prototyping this and the results so far are amazing. Subtle configuration differences no longer cause duplication of super expensive actions. For me it is generated C files and architectureless data files. Thanks for doing this! I feel like we're just beginning to recognize the impact and will find much more savings.

Deduplicating executions is important for us since local executions of these duplicates often get scheduled around the same time. Another interesting note is the ability to cap disk cache size might be required before we roll this out to workstations.

[fmeum]

Action deduplication will land in 7.4.0 (#23674).

[nikhilkalige]

@fmeum As I am trying to fix my rules to support path mapping, I wanted to see what rules could possibly be fixed to use this. I wanted to see all the cmd line args of a particular target, to help me figure out the rules that need fixing. I tried aquery and in that, I don't see the arguments post mapping, which kind of makes sense. So, do you know of a way to get the cmd line post mapping somehow?

[fmeum]

The only realistic way to see what Bazel actually executes is to run a build with -s. You could set up a --disk_cache and build after a bazel clean, which will still be very fast due to caching but show you all the command lines.

[nikhilkalige]

That's exactly what I have been doing.. Thank you..

[nikhilkalige]

We have a few rules that take directory as an argument. Most locations kind used this

args.add(file.dirname)
# Updated now to
args.add_all([file], map_each = _dirname)

Above suffices in a lot of places, but becomes very quirky when you have complex rules. For example, in protobuf rules, there is a lot of directory references passed to -I.

• Is it possible to have an api to construct a path from starlark?
• File.dirname return a path instead of string, would also be useful

[fmeum]

So far we haven't done this since path mapping may (but currently doesn't) map different files in the same directory to different output roots, so file.dirname returning a path still wouldn't have enough information to path map. When we get to the point that we decide that a static output root (bazel-out/cfg/bin) is sufficient, we could start adding better API for this use case.

@gregestren @aranguyen Are you still interested in exploring content-based path mapping schemes?

[gregestren]

Are you still interested in exploring content-based path mapping schemes?

Was there a doc reviewing this or did we only discuss this in PR reviews / chats?

[fmeum]

I only know what's in https://docs.google.com/document/d/17snvmic26-QdGuwVw55Gl0oOufw9sCVuOAvHqGZJFr4/edit and code comments.

[gregestren]

Am I reading this thread correctly that exploring content-based approaches is an obstacle to this request, since it forks the directories of an action's inputs?

Thanks for reminding me about https://docs.google.com/document/d/17snvmic26-QdGuwVw55Gl0oOufw9sCVuOAvHqGZJFr4/edit. :) A few interesting ideas we could further explore:

• Adding smarter cfg distinguishers to handle multi-configured actions. That's still unsupported, right?
• Applying the mapping to Bazel's output tree instead of just the executor (which the above doc describes). That can better guarantee all consumers of action outputs (IDEs, post-processing shell scripts, ...) have the same view as the executor that ran the action
• If we went all-in making the local output tree a CAS with some mapping API to realize "important" outputs, we could avoid action conflicts by design. In other words, break the pattern that Bazel needs distinct artifact paths over the scope of the entire output tree. Instead paths only need to be unique over the scope of what outputs the build materializes. I'm hand-waving here because ideally this also applies inside the Bazel process itself, but that's a whole new level of complication.

I don't know - what are your current thoughts? I remember you were interested in exploring more sophisticated mapping algorithms. Anything particular still on your mind? As much as I think the above ideas are fascinating, at some point they become major engineering work and I wonder if the current simple model is good enough for most practical use cases.

[keith]

I was testing this out with #23630 applied, I got pretty far but I hit this issue:

ERROR: AsyncRT/BUILD.bazel:138:23: Compiling AsyncRT/lib/Runtime/Globals/Globals.cpp failed: undeclared inclusion(s) in rule '//AsyncRT:RuntimeGlobals.static':
this rule is missing dependency declarations for the following files included by 'AsyncRT/lib/Runtime/Globals/Globals.cpp':
  'bazel-out/cfg/bin/external/gperftools/tcmalloc/include/gperftools/tcmalloc.h'

Where the header here is from a rules_foreign_cc target:

configure_make(
    name = "tcmalloc",
    autogen = True,
    configure_in_place = True,
    configure_options = [
        "--enable-shared=no",
    ],
    copts = [
        "-fPIC",
        "-w",
        "-fexceptions",  # Needed for tests, which ideally we wouldn't build
        "-Wno-register",
    ],
    lib_source = ":all_srcs",
    out_static_libs = ["libtcmalloc_minimal.a"],
    target_compatible_with = select({
        "@platforms//os:linux": [],
        "//conditions:default": ["@platforms//:incompatible"],
    }),
    visibility = ["//visibility:public"],
)

It might be relevant that I access it through an alias (although removing that doesn't help):

alias(
    name = "tcmalloc",
    actual = select({
        "@platforms//os:macos": "@gperftools-macos//:tcmalloc",
        "@platforms//os:linux": "@gperftools//:tcmalloc",
    }),
)

Are you tracking this one?

[fmeum]

I am now: #23681
I have an idea what could be wrong, will try to reproduce it in an integration test.

[jjudd]

I've been implementing path mapping in few rule sets and I think I'm either implementing something incorrectly or I've found an issue with path mapping + multiplex worker sandboxing. I'm seeing errors where the worker doesn't get run because the executable isn't present in the sandbox. IOException while preparing the execution environment of a worker

Both path mapping and multiplex sandboxing work on their own:

• Disabling multiplex workers with --noworker_multiplex gets path mapping working.
• Disabling path mapping with --experimental_output_paths=off gets multiplex sandboxing working.

Here's a minimal repro case as well as the full error (in the README): https://github.com/lucidsoftware/path-mapping-bug-repro You can repro with bazel build example. The worker doesn't do anything, but it doesn't need to in order to repro the bug as we're not getting to the point where the worker actually runs.

[fmeum]

I think that this is happening because your worker binary is a generated file. That's not a case I have tested so far since the Java worker is a precompiled deploy jar. I will look into this.

[jjudd]

Thank you very much. Greatly appreciated.

[fmeum]

I have a fix: #23991

[jadenPete]

I'm attempting to implement path mapping in a ruleset, and I'm not sure if I'm misunderstanding how path mapping works, but I'm not able to get cache hits between what should be identical actions. I think I may have discovered a bug.

Here's a repository that minimally reproduces the issue:
https://github.com/lucidsoftware/path-mapping-action-key-bug-repro

The README.md explains the issue and running example/test should demonstrate it.

Do you know why this is happening and how I can prevent //example:file from being built twice?

[fmeum]

You need to use a disk or remote cache to see any benefits from path mapping. This is because as you discovered, the actions defined by the two rules are still distinct, but the command invocations they result in (which Bazel internally refers to as "spawns") are identical.

You can confirm this by running with -s and observing that the executed command lines are the same, which means that either will be a (disk or remote) cache hit for the other. Since these spawns are very likely to be run concurrently, the deduplication feature added in 7.4.0 will kick in and give you a hit even before any of the spawns have uploaded their result to the cache. On your example, with --disk_cache=some/path, this looks as follows:

Starting local Bazel server and connecting to it...
INFO: Invocation ID: 9785dcd9-9f2f-43c9-b96b-f3e6ddae5851
INFO: Analyzed 4 targets (6 packages loaded, 14 targets configured).
INFO: From StaticFile example/file/output.txt:
Generating static file...
INFO: From StaticFile example/file/output.txt:
Generating static file...
INFO: Found 4 targets...
INFO: Elapsed time: 4.337s, Critical Path: 0.09s
INFO: 5 processes: 1 internal, 3 darwin-sandbox, 1 deduplicated.
INFO: Build completed successfully, 5 total actions

[jadenPete]

I can confirm that makes it work. I think this was a misunderstanding of how Bazel does caching on my part.
Thank you for your help!