macOS sandboxing doubles build time

[twpayne]

Description

Sandboxing on macOS more than doubles the build from scratch execution time of a large-ish (3K source files) Go project.

More information

Profile output with sandboxing enabled:

Phase Summary Information

Total launch phase time	20.0 ms	0.00%
Total init phase time	38.1 ms	0.01%
Total loading phase time	4.931 s	0.75%
Total analysis phase time	1.125 s	0.17%
Total preparation phase time	18.2 ms	0.00%
Total execution phase time	653.128 s	99.07%
Total finish phase time	11.2 ms	0.00%
Total run time	659.272 s	100.00%

Profile output with sandboxing disabled:

Phase Summary Information

Total launch phase time	21.0 ms	0.01%
Total init phase time	40.5 ms	0.01%
Total loading phase time	4.480 s	1.43%
Total analysis phase time	1.128 s	0.36%
Total preparation phase time	14.5 ms	0.00%
Total execution phase time	307.026 s	98.18%
Total finish phase time	10.3 ms	0.00%
Total run time	312.721 s	100.00%

Sandboxing is enabled by default. Disabling sandboxing (--spawn_strategy=local) more than halves the build time.

From observing Activity Monitor during the build, it looks like sandboxing roughly doubles the amount of time spent in system calls during the build, so it looks like setting up and tearing down the sandbox is very expensive on macOS.

Possible fixes and workarounds

• Implement sandboxing on tmpfs for macOS (it might help).
• Wait for Apple to improve sandbox performance on macOS (out of scope).

Environment

• Operating System: macOS 10.12.3
• Bazel version: 0.4.3-homebrew
• Filesystem: Apple HFS, FileVault full disk encryption
• Hardware: MacBookPro11,1 (Retina, 13-inch, Mid 2014), 250GB SSD drive

[twpayne]

The discussion in #1836 is relevant: it includes some Linux-only tricks for speeding up sandboxing on slow filesystems.

[hermione521]

Yes, I have to admit that we know it's slow. We create a lot of hardlinks and symlinks for all the input, and sandbox-exec on Mac is also slow. @philwo said he has something about tmpfs that might also work on Mac, but we will have to wait. I'm sorry :(

[ittaiz]

@philwo is this under work?

[johnynek]

To me, sandboxing is like runtime checking of types. If you have a type safe program already, it is only a waste.

Assuming you have rules that do run correctly when sandboxed, what additional benefit do you get from actually running them that way? I mean, if the rule is already hermetic, sandboxing isn't going to make it more so. To me, sandboxing seems most useful to test that rules are working correctly.

[ittaiz]

interesting perspective. What does this mean for tests? Some tests touch the filesystem as well as bind to ports. Sandboxing should solve this while I'm not sure how the rules themselves can/should handle this?

[johnynek]

To be honest, I hadn't considered the application to running code, such as tests. I assumed binary and test targets can still see the file system as normal, but we don't launch binaries with bazel so I don't really know (we build deploy packages of compiled code).

[ittaiz]

I guess we'll have to wait for some feedback and see since I might have misinterpreted sandboxing to also apply to tests but I don't think so.
Don't you have tests for the code you're building? Or maybe they don't do any I/O so you've never thought about it?

[philwo]

This is under work, but considering that I have quite a number of things "under work", I'm not sure when exactly I can improve the current situation here. :) But it's definitely on my radar and I'd love to get this fixed as soon as possible.

The biggest problem on macOS is that we just don't have many mechanisms for sandboxing available and the existing ones (like sandbox-exec) are poorly documented or not documented at all. It's hard to come up with a fast & correct solution on the platform. (That said, it's probably still easier than on Windows...)

Sandboxing is actually very important for tests as they're often written with the assumption that they can just modify system state and someone will clean up after them - Java unit tests maybe not so much, but heavy weight shell script integration tests are a different story: Creating temporary files with hardcoded names, spawning background processes that they don't kill, opening listening TCP sockets on a fixed port number, ... or even happily "rm -rf"'ing whatever they thought was their temporary working directory, even if that happened to be your $HOME.

We generally recommend to always use sandboxing, especially on Linux. However, I agree that this might just not be practical for some builds, for example when it makes them very slow. I think it's reasonable to e.g. make release builds using sandboxing, but disable it during the edit & refresh workflow, if that increases your productivity. Please use your own judgment.

FWIW, I personally always build and test with sandboxing enabled.

[ittaiz]

@philwo is it possible to run only some things in the sandbox? asking since if the price is on the symlinks maybe one can pay a smaller price for the critical targets. My use-case mainly talks about running ~30% of our tests in the sandbox for filesystem/network isolation.

[hermione521]

It's possible to disable sandboxing for some of your targets with tags = ["local"]

[philwo]

It's also possible to only enable sandboxing for certain kinds of actions (e.g. Java compilation, genrules, tests, ...) by using the --strategy flag.

Example to run only tests with sandboxing and nothing else:
bazel test --spawn_strategy=standalone --strategy=TestRunner=sandboxed

The --spawn_strategy=standalone part disables sandboxing globally.
The --strategy=TestRunner=sandboxed part enables it for tests. (I'm 99% sure that "TestAction" is the right mnemonic here.)

The "TestRunner" string that you see in the flag is the so called mnemonic of the action, which identifies the kind of action. Other examples would be "Javac" or "Closure". We should probably somehow automatically generate a list of all known mnemonics in our documentation... you can currently find out by grepping for the "getMnemonic" method in Bazel's code for native built-in rules or checking the "mnemonic" parameter to ctx.action() in Skylark rules.

[jerrymarino]

Great thread! I'm also running into sandboxing as being a blocking issue for building an iOS app. The context is I'm building few thousand files ( mix of C++, ObjC++, ObjC, and C ) on consumer grade hardware: i.e. a 2013 MacBook Air.

My baseline build is an Xcode debug build and when I try to compile my entire app under bazel, it is significantly slower than Xcode ( i.e. 2.5x slower ). In some dependencies, like Texture, the impact of building with sandboxing is ~10x slower.

I agree with @johnynek

Assuming you have rules that do run correctly when sandboxed, what additional benefit do you get from actually running them that way

I haven't found any major downsides or issues with turning sandboxing off for the local builds I'm doing.

Given that, I'm planning to turn sandboxing off as a sensible default for local builds. I can't justify enabling sandboxing for local builds as a default since it adds a significant slowdown ( assuming the build is already designed to be sandboxed, and turning it off won't impact build stability )

[ulfjack]

@philwo has done some work on MacOS sandboxing performance that may have only landed in the 0.5.0 release; whenever you're posting performance numbers, please be so kind as to include which version of Bazel you used to collect them. At this point, we have no credible evidence that sandboxing performance on MacOS is still a problem with 0.5.0.

We also have no possibility of making sandboxing faster than the underlying implementation; it's up to Apple to improve sandboxing performance. Certainly, if someone can show that we're holding it wrong, by all means do.

At this point, I am not aware of any open action items for Bazel team in this area, so I'm closing this issue.