server,*: the go runtime and vendored deps are sensitive to environment variable

[knz]

See discussion in #66805: at least the Go runtime http package transparently uses the HTTP_PROXY, HTTPS_PROXY, NO_PROXY and REQUEST_TYPE env vars to issue HTTP requests.

This impacts e.g:

• backups to cloud
• sentry reporting
• http network logging (util/log: add http log sink #66196)

We have at least one customer who is requesting crdb to not be influenced by env vars generally, especially not those that have an impact on networking, unless opted in explicitly for specific features (E.g. include a proxy flag in the BACKUP SQL statement)

What to do about this?

Option 1: unset the env vars explicitly when the process starts up.

This option is to unset all env vars except for a few "allowlist" (including the COCKROACH_ env vars, GOMAXPROCS etc and perhaps TERM).

We cannot use an env var denylist: as detailed in #66805 there are dozens of environment variables that may influence the process implicitly. Even if we had an explicit list of variables to clear before the process starts up, we'd need to manually review that list every time we upgrade a dependency, which could introduce new env vars.

Pros: very conservative. Certainty of no impact from env vars anywhere.

Cons: for certain deployment scenarios it's actually beneficial to let some amount of control on the process "from the outside" to the system administrator. Having a blanket ban on pre-set env vars would completely shut out system administrators from what's customizable in the go source code.

In fact we even had customer demand for the HTTP_PROXY env var before (#34067) and this option would break that use case.

Option 2: inform in logs and docs, and recommend env var hygiene in orchestration

In this option, we keep the current behavior and let the process be influenced by env vars.
We would do three things:

1. audit the code for areas where we want to opt out of env var impact for a good reason, for example what we did already in rpc: disable use of http proxies by default #55502
2. produce logs upon startup including all env vars (but maybe not their value) to ensure we have an audit trail in support cases
3. document that env vars may have an impact in our official docs, and make a recommendation to deploy crdb in such a way that only the specific desired env vars are visible to the server processes.

Pros: moves control in the hands of the user DBA/SRE

Cons: unsure if there is one

Jira issue: CRDB-8253

[knz]

@bdarnell suggests:

the usual approach is not "use a proxy for s3 and nothing else", but "use this proxy for everything, and the proxy will disallow anything but s3"

[knz]

@dt suggests:

i would be fine with, in addition to supporting the usual env vars as we do now, allowing a &proxy_uri=... on an s3 URI that is used only if a) outbound network is allowed and b) a proxy is not configured in the env

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!