Allow timeout on server and directory ownership

[ob]

One of the uses of the remote cache is as a proxy on local machines (Bazel waits for all the artifacts to upload before completing the build so a slow network connection can impact build times).

I've been playing with the idea of having my Bazel wrapper automatically spawn a bazel-remote subprocess on a local port and rewire the Bazel command to prefer the local cache server than the remote one.

In order for this to work, there are two features that I think would be needed from bazel-remote:

1. Detect when a bazel-remote process is already serving objects from a directory.
2. Exit after an idle timeout.

I think for 1. the easiest would be to write the PID and port number in a file inside the cache directory. On startup, bazel-remote could check if that PID is still running, if not, it would start on the same port.

I could manage this from the Bazel wrapper, but I think it would be better to add it to bazel-remote so I'm opening an issue to start the discussion.

[ob]

I realized the original title was misleading. I'm not proposing full Unix daemonization. Only the ability to timeout after a period of inactivity and to bail if called on a directory that is owned by another running instance.

[buchgr]

Hi Oscar,

that sounds like a great idea! I am happy to hear that you want to use it like this. In fact, there are early plans on adding functionality for Bazel to be able to launch a remote caching process locally. That is, so that 1) we can support the use you mentioned and 2) we'd be able to move authentication libraries from the Bazel binary into a remote cache binary. This binary could be lazily fetched from the internet and support proxying cache/execution requests to a remote system. This should make maintenance of such libraries easier and reduce Bazel's binary size.

Sounds good?

[ob]

@buchgr sounds perfect! In my case, we have a go-written bazel-wrapper that already handles versioning, fetching, and caching of the Bazel main binary. Our plan is to add Bazel-remote to the wrapper and have it be spawned automatically, on demand.

I'm wondering if bazel is going the route of using Bazel-wrapper like I intend to, maybe it'd be better to add this functionality straight to bazel?

We have an internal design document on how we are planning on doing the integration, I'll see if I can share the details if that is interesting.

[buchgr]

I'm wondering if bazel is going the route of using Bazel-wrapper like I intend to, maybe it'd be better to add this functionality straight to bazel?

I think so but I'd guess that's 9-12 months out at least, because other things are higher priority. However, I'd be happy to work with you to make the necessary changes to bazel-remote so that they integrate well with your Bazel-wrapper. I think this could also give us valuable feedback when ultimately adding this functionality to Bazel.

[buchgr]

@bayareabear implemented the --idle_timeout in 57a1816. The directory ownership is still up for grabs :-)

[mostynb]

@ob: the remaining part of this feels to me like something that might be better handled by launchd (on mac) / systemd (on linux).

I played around a bit with a launchd, and was able to setup a user agent, that runs bazel-remote with an --idle_timeout flag, without requiring root/sudo. Then I was able to run launchctl start bazel-remote and it starts bazel-remote but only if it's not already running. I can share the configuration if it would help.

(Systemd is probably even easier to get up and running, since it has better documentation than launchd.)

[ob]

In our case, it would be harder to rely on launchd or systemd since we are using bazel-remote on-demand and automatically spawned by bazel-wrapper. Configuring launchd/systemd would require the user configuring it or us installing hooks on the system. We also automatically upgrade bazel-remote before launching it.

[mostynb]

It's fairly easy to install the launchd config, it doesn't require root access and could be done by your bazel-wrapper. IIRC it works fine if you upgrade the bazel-remote binary while it's running too, the new bazel-remote binary is used next time it starts.

If you really want to avoid launchd, you could implement this system in your bazel-wrapper rather than in bazel-remote.

[brentleyjones]

@mostynb Can you share the full launchd setup process here? We are in the process of finalizing our local proxy and while we have a solution something launchd related would probably be better.

[mostynb]

https://www.launchd.info was the best reference I found, IIRC. I modified the example there for bazel-remote, and was able to load and start/stop it without sudo.