Keep credentials cached across build commands.

[tjgq]

When using a credential helper, the lifetime of the credential cache is currently tied to an individual command, which causes the helper to be called for every command resulting in poor incremental build latency for builds using a non-trivial helper.

Since the cache must be shared by RemoteModule and BazelBuildServiceModule, I've introduced a new CredentialModule whose sole purpose is to provide access to it.

[tjgq]

cc @Yannic

[Yannic]

Mostly drive-by comment, will take a closer look tomorrow.

I think if we cache credentials accross invocations, we need to make the helper part of the cache key

[tjgq]

Mostly drive-by comment, will take a closer look tomorrow.

I think if we cache credentials accross invocations, we need to make the helper part of the cache key

Could you elaborate on the scenario you're concerned about? It's not immediately obvious to me that changing the helper necessarily invalidates credentials that were obtained before.

[coeuvre]

Probably clear the cache when the command is clean?

[tjgq]

Yeah, I think that's probably expected behavior for bazel clean. It also gives us a reliable way to bust the cache. Added a test and amended the flag description to document this behavior.

[coeuvre]

Not sure whether this is the best way to provide the cache. For now it seems fine as we have limited scope but if we ever want to share the cache for more modules (e.g. for bzlmod), please consider adding a provider to ServerBuilder.

[tjgq]

Acknowledged; I'd prefer to wait until repository fetching is wired up to see how the whole thing looks.

I think in any case we'd still want this module to host the beforeCommand logic (otherwise we'd have to arbitrarily choose one of the many modules that will eventually use credentials, making the scope unclear).

[Yannic]

FWIW, there's https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/runtime/ServerBuilder.java;l=202;bpv=1;bpt=1 which we may be able to re-use for the credential helper.

[Yannic]

Mostly drive-by comment, will take a closer look tomorrow.
I think if we cache credentials accross invocations, we need to make the helper part of the cache key

Could you elaborate on the scenario you're concerned about? It's not immediately obvious to me that changing the helper necessarily invalidates credentials that were obtained before.

Let's say someone does a build with --credential_helper=foo, then the cache would be populated with credentials for example.com from foo. In the next build, they change to --credential_helper=bar (also against example.com). I would expect bar to be always called since we changed the flag, but it won't necessarily because the helper isn't part of the cache key and there are credentials for example.com in the cache. I guess Bazel would run bar if the RPC with the credentials from foo fail, but it seems very much reasonable that Bazel would invalidate all credentials that come from a different helper than what's configured for the current invocation.

[tjgq]

Mostly drive-by comment, will take a closer look tomorrow.
I think if we cache credentials accross invocations, we need to make the helper part of the cache key

Could you elaborate on the scenario you're concerned about? It's not immediately obvious to me that changing the helper necessarily invalidates credentials that were obtained before.

Let's say someone does a build with --credential_helper=foo, then the cache would be populated with credentials for example.com from foo. In the next build, they change to --credential_helper=bar (also against example.com). I would expect bar to be always called since we changed the flag, but it won't necessarily because the helper isn't part of the cache key and there are credentials for example.com in the cache. I guess Bazel would run bar if the RPC with the credentials from foo fail, but it seems very much reasonable that Bazel would invalidate all credentials that come from a different helper than what's configured for the current invocation.

I think it can be argued both ways: if I'm replacing foo with bar but they both call the same external service, I know that the cached credentials are still good.

But assuming we do want to make it part of the key: how far do we want to go? Are you proposing to only take the name of the helper into account, or also the executable it points to, the values of environment variables, etc?

[coeuvre]

I will let you and Yannic decide whether to include helper inside the key. Other part LGTM.

[Yannic]

I'm fine with starting with just the URI as cache key. Shouldn't be too hard to change if a more concrete use case comes up. I also expect that we need to change the key in any case once we tackle scopes for credentials from helpers.

[tjgq]

I'm fine with starting with just the URI as cache key. Shouldn't be too hard to change if a more concrete use case comes up. I also expect that we need to change the key in any case once we tackle scopes for credentials from helpers.

Thanks, I'll merge this as-is then. I agree that we're going to have to think more carefully about it (cache invalidation: one of the two famously hard problems in compsci :))