crypto/x509: add SetFallbackRoots and golang.org/x/crypto/x509roots/fallback package

[breml]

Proposal

Go programs currently rely on the operating system to provide CA root certificate information in some sort of certificate system store. There are situations, where no such up-to-date CA root certificates are available like (Docker) containers built FROM scratch or out of date environments like poorly maintained or no longer updatable systems (e.g. older hardware appliances).

For some environments it is possible for the user of a Go program to add additional CA root certificates via SSL_CERT_FILE or SSL_CERT_DIR, but this is not the case for all supported environments (e.g. Windows) and it is definitively not user friendly.

Therefore, it is desirable for Go programs to have some mechanism to directly embed CA root certificate information into the program itself, so that they don't have to rely on system store to provide CA root certificates that may be absent (or out of date).

I make the following assumptions, which I think are reasonable:

• adding the CA root certificates into a program should be optional, as it increases the size of the program by approximately 250K (uncompressed), and most programs run in properly maintained systems with an up-to-date certificate system store
• it should be possible to embed CA root certificates when building an arbitrary program
• it should be possible for a program to always embed CA root certificates without requiring any special build step
• the program should always prefer data from the system if available, as it is likely to be more up to date than that included in the program
• the program should allow to override the preference and force the usage of the embeded data (controled by an environment variable, e.g. GO_ROOTCERTS_ENABLE)

Given those assumptions, I propose adding a new package crypto/x509/rootcerts. Importing this package (as import _ "crypto/x509/rootcerts") will cause CA root certificates to be embedded in the program. Also, building with the build tag rootcerts will force the package to be imported, and therefore will cause CA root certificates to be embedded in the program. The embedded CA root certificates will be used if and when no certificate system store is available (or the user forces the usage of the embedded data).

Source for the CA Root Certificates

I propose to use the Mozilla Included CA Certificate List, more specifically the PEM of Root Certificates in Mozilla's Root Store with the Websites (TLS/SSL) Trust Bit Enabled as the source for the CA root certificates.

The Mozilla Included CA Certificate List is the source for the CA root certificates embeded in the well known products of the Mozilla Foundation like for example Firefox (web browser) or Thunderbird (email client).

In contrast to most of the other software vendors, Mozilla maintains its Included CA Certificate List publicly and distributes it under an open source license (Mozilla Public License Version 2). This is also the reason why most of the Linux distributions, as well as
other free unix derivates and wide spread tools, use this list of CA root certificates as part of their distribution.

Some examples:

• Debian (and its derivates): ca-certificates
• Red Hat / Fedora / CentOS: ca-certificates / ca-certificates
• Alpine Linux: ca-certificates
• FreeBSD: ca_root_nss
• NetBSD: ca-certificates
• curl: cacert.pem

In summary in my opinion it is safe to say that the Mozilla Included CA Certificate List is well established and widely used.
In fact, if a Go program is run on Linux or an other free Unix derivate, chances are high that the root certificates used by the program are already provided by the Mozilla Included CA Certificate List.

Why include into the Go Standard Library

As the sample implementation (link in Annex below) clearly demostrates, that it is possible to write a 3rd party Go package, which achieves the same goal as the proposed package crypto/x509/rootcerts would. The main difference between a package in the standard library and a 3rd party package is: TRUST.

The root certificates are the top-most certificates in the trust chain and used to ensure the trustworthiness of the certificates signed by them either directly (intermediate certificates) or indirectly (through intermediate certificates). Therefore for a package containing and replacing the root certificates, trust is essential.

The same way, most users of Linux trust the CA root certificates provided by their distribution, it is very likely, that user would trust the CA root certificates provided by a package included in the Go standard library.

Additionally, the possibility to include the CA root certificates during build time, without altering the source code, is not possible with a 3rd party package but only if this package is included into the Go standard library and the build tag is implemented in to Go tool chain.

Update of the CA Root Certificates

The CA Root Certificates included in the standard library are updated with every release of Go (with the current schedule every 6 months). This would work the same way as it currently does for the package time/tzdata. The update frequency of the Included CA Certificate List is roughly every few months (2020: 5 times, 2019: 4 times, according to curl ca extract), which seems to be similar to the update frequency of the time zone data information.

In regards to updating the CA root certificates compiled into a Go binary, the same limitations apply as for the time/tzdata package. The information compiled into a binary is not updated. That being said, for the situations, this package is intended for, it is still an improvement because containers built FROM scratch are also not updated by default and out of date / not updatable systems obviously also do not get updates for the CA root certificates.

Annex

There is a sample implementation of this approach at github.com/breml/rootcerts with some additional reasoning about when to use such a package and what to keep in mind.

This proposal as well as the sample implementation are highly influenced by the proposal #38017 - time/tzdata and the implementation of the package time/tzdata by @ianlancetaylor

cc: @FiloSottile, @katiehockman, @mvdan

[mvdan]

At least from my personal experience, I've used scratch images quite a lot for deployments of "pure Go" services, and I needed ca-certificates more often than I needed tzdata. They both fit the same bill in terms of system files which the standard library might depend on, depending on what the program is doing. A significant amount of what I've written in the past ends up needing ca-certificates in the end (e.g. because of outgoing HTTPS requests), so it's almost a default that I make sure it's available beforehand.

This is all to say - if tzdata was included, I think rootcerts should too, given how I think it's at least just as important.

[seankhliao]

examples of other third party packges that provide this:
https://github.com/certifi/gocertifi
https://github.com/gwatts/rootcerts
https://github.com/alexflint/stdroots

I'm not particularly convinced that having it in std is a good idea, since out of date tzdata is usually just a minor bug but out of date ca certs is a security issue. Maybe somewhere under x/crypto.

[mvdan]

I agree with what the proposal says, though. The current solutions people are using are also potentially vulnerable to not getting new CA roots for a long time. This includes cases where the binary runs on a system that's simply not being updated regularly.

At the end of the day, it's up to the developer to keep up to date with Go versions and rebuild/redeploy their binaries as needed. If that maintenance is not happening, the result is the same whether or not this proposal is accepted and used by them.

[breml]

examples of other third party packges that provide this:
https://github.com/certifi/gocertifi
https://github.com/gwatts/rootcerts
https://github.com/alexflint/stdroots

One small difference between https://github.com/breml/rootcerts and the other 3rd party packages mentioned above is, that it can be used in a "none intrusive" way, because no application code needs to be changed. A simple blank import (import _ "crypto/x509/rootcerts") in package main is enough.

This is implemented the same way it is done in time/tzdata (see: https://github.com/breml/rootcerts/blob/master/rootcerts.go#L28-L34).

[rolandshoemaker]

I think there are two interesting parts to this proposal: implementing an API to set the default root pool, and adding a standard library package that contains some set of roots. While I am generally in favor of the first part, I am somewhat opposed to the second.

I think adding an API to crypto/x509 that allows setting the default root pool (rather than always using the system pool when no roots are specified) could be quite valuable, and would more cleanly and safely provide the functionality you (and some other existing packages) are getting with go:linkname tricks. I think the major downside of adding this kind of functionality though is allowing the user(/developer) to understand where their root pool is being set, and opens up the rabbit hole of dependencies setting a malicious set of roots (although it could be argued that this is already the case with the go:linkname trick, so we'd just be adding a more explicit easy to understand API to replace this inherently unsafe approach).

Speaking as a member of the security team, who would likely end up being responsible for maintaining it, I'm hesitant to implement a root store in the standard library. Even if we are just copying some existing root program, there are a number of somewhat complicated policy enforcement decisions we'd become embroiled in. I'm also not sure what the update process for the store would be, for instance if a root was to become distrusted in the NSS program, would we need to issue a security release of Go in order to provide an updated root store to users? I think if we were to do something like this it'd most likely need to live in a x/ repository as a module so that updates could be made in a less rigorous cadence. (Side note: just using the NSS trust store is not as simple on its face as it seems, while it is possible to extract a list of 'trusted' certificates there are somewhat complex policies around what certificates may be used for what, and a chunk of policy that is implemented in code, meaning that just pulling their list and taking it at face value is likely to result in trusting things we may not always want to trust.)

[breml]

@rolandshoemaker Thanks for your thoughtful feedback.

The main intend of my proposal is a standard library package that contains some set of roots.

What I care about is, a simple, save and trusted way to embed a trusted set of root certificates into a Go binary.
I more than once observed inexperienced developers handling the missing ca-certificates.crt problem in awful ways (e.g. committing the file together with the code, where it gets forgotten and therefore is never updated). To be able to import the root certs in the same simple and straight forward way as the time zone data is very compelling.

Go positions it self as the language of choice for the cloud and cloud native applications. To cite @spf13 (Steve Francia in https://www.infoq.com/articles/go-language-13-years/):

Go’s strength is still the cloud/server applications that Go is such a good fit for, ...

The second, much more significant phase, will be the industry shifting to take advantage of the unique cloud offerings, increasingly moving to cloud-native application development. In these cases, Go is the clear choice.

Lots of these cloud native applications will run in container environments and lots of them will need to talk to external APIs, which then will need root certificates to establish the secure connection. There I see a clear benefit in the Go standard library providing a simple way of including the CA root certificates into the Go binary. It is maybe a little bit of a stretch, but if a Go application is run in a container FROM scratch, then the Go application sort of becomes the operating system and operating systems normally provide a set of root certificates.

I clearly understand your concern for you as a member of the security team and I completely agree, that is likely your team, who will end up being responsible for the maintenance. But if we look at this from users point of view, then we are definitively in a better place, when the Go security team takes care of this, than if a random open source contributor publishes a package with this functionality. If this proposal gets accepted, I will happily remove/deprecate my package.
With the still rapid growing number of Go developers around the world I feel the maintenance of these root certificates inside of the Go standard library is worth the effort to increase the overall security and trust of the Go and especially the cloud native / containerized eco system.

For the update process as well as the location of such a package, it is my belief, that the package should life in the standard library (and not in x/) for the following reasons:

• My observation in projects in the industry is, that updating the Go tool chain (and with this the standard library) is a well understood process. Updating the Go tool chain is (due to the great work of the Go team and the compatibility guarantee) a friction less process, which is often even automated.
• In contrary, the updating of Go modules is unfortunately sort of a forgotten and less understood topic. I often observe, that the dependencies are only updated, when this is necessary due to functional requirements. The minimal version selection does not help here and I fear that a root certificates package in x/ will get way less often updated than the same package in the standard library.
• It is already possible to day to ship minor updates for Go and it is done quite often. E.g. for Go 1.15 we currently are at 1.15.7, which makes more than a minor release per month. So I do not really see a problem to ship a new minor release, if there is a certificate, that needs to become distrusted.

In regards to the NSS trust, I did not go into full details here, but it is my observation, that this problem has relaxed over the last couple years. On one hand, all the mentioned Linux distributions have solved this problem one way or the other, so this is a pretty well understood problem with battle proved solutions available. On the other hand does Mozilla provide the PEM of Root Certificates in Mozilla's Root Store with the Websites (TLS/SSL) Trust Bit Enabled (see: CA/Included Certificates). In my understanding, this is already the correct list of certificates with the correct trust bits enabled.

The other part you have identified in my proposal (API to set the default root pool) is less relevant for the use cases I am outlining in the proposal, for the following reasons:

1. On Unix/Linux based systems, Go already today offers the possibility to add additional certificates to the default pool by setting the respective environment variables.
2. There are other proposals covering somewhat related aspects:
   1. crypto/x509: add ability to reload root certificates crypto/x509: add ability to reload root certificates #41888
   2. crypto/x509: Convenient, efficient way to reload a certificate pool after on-disk changes: crypto/x509: Convenient, efficient way to reload a certificate pool after on-disk changes #35887
3. For me, the way the system roots are loaded is not important. The go:linkname trick has been bluntly copied from time/tzdata, so nothing I invented myself.

[rsc]

How often do root sets change?
How quickly would one go stale in an old Go distribution?
How many different ones are there?
Is there an obvious one that we could pin ourselves to, like we pin to the standard timezone database.

[seankhliao]

Chrome's new initiative to maintain its own root store may be of interest https://www.chromium.org/Home/chromium-security/root-ca-policy/

[joeshaw]

How often do root sets change?

curl maintains a bundle of the Mozilla root certs: https://curl.se/docs/caextract.html

It may not be comprehensive but there were 5 updates in 2021 and 5 so far in 2022.

How many different ones are there?

The big ones are Mozilla, Google, Microsoft and Apple. There may be additional ones.

Is there an obvious one that we could pin ourselves to, like we pin to the standard timezone database.

The Mozilla one is the best candidate here. Their policy is documented here and they have an open process for inclusion. Most open source software and Linux distributions use the Mozilla store.

[breml]

My "proof of concept" (https://github.com/breml/rootcerts), mentioned above, does use the root certificates from Mozilla, similar to the way it is done for curl.

This root set does contain 146 certificates as of now. I quickly analyzed the list from Included CA Certificates (CSV) from https://wiki.mozilla.org/CA/Included_Certificates (with the email only certificates filtered out). The average validity of these certificates is 23.5 years (with a minimum of 9 years and a maximum of 35 years).

So the number of certificates that are needed to be updated will average in the range given by @joeshaw (~6 year).

In contrast to tzdata updates, this seams to be in a similar range (see: http://mm.icann.org/pipermail/tz-announce/):

2022 ytd: 1
2021: 5
2020: 6
2019: 3
2018: 9
2017: 3
2016: 10
2015: 7
2014: 10
2013: 9

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

/cc @golang/security

[rsc]

One reason this would be interesting to resolve is that currently "single-Go-binary containers" cannot make HTTPS connections. You have to side-load the root set or derive the container from a full Linux distribution. If we could optionally put the root set into the binary, that would make single-binary containers useful for more programs.

[FiloSottile]

I am conflicted about this.

I agree it makes single-binary deployments more convenient, and it feels like a natural feature to provide.

On the other hand

1. root stores are moving more and more away from being a list of homogeneous trusted anchors, and towards rich metadata that is only expressed in the code of the verifier they ship with (name constraints, time validity, CT requirements...) which is why we invested so much last year in moving to the platform verifier
2. root stores are aggressively moving towards higher root agility, meaning that past numbers on the frequency of root changes are not indicative of the next five years, and we'd be committing to maintaining this pretty much forever
3. distrust changes MUST be shipped as security releases (this is a whole debate in the Linux distribution world, and we don't want to become the next chapter of the problem of slow to update root stores), which will increase the security release load
4. it's not clear what store we should ship: the canonical answer is the Mozilla one, and it's not a bad answer; the Chrome one might be interesting too, but I'm not sure they would want us to decouple it from the verifier
5. trust is fundamentally a platform concern (which Windows and macOS and iOS understand) and the correct answer to this is arguably for container runtimes to synthesize an /etc/ssl/roots.pem file the same way the synthesize /etc/resolv.conf

This might make more sense as an auto-updated x/crypto package, which then would encourage adding x509.SetDefaultRoots() but I don't love the semantics of a runtime-changable singleton like that. I can see libraries arbitrarily using it in init() leading to conflicts and unexpected behavior that is hard to debug.

[qdm12]

From my narrowed github user point of view, having it as a module dependency together with dependabot is nice, so I'm alerted when it gets outdated. If we want it as part of x/crypto it would make tooling such as dependabot not alert us for outdated CAs.

[rolandshoemaker]

I agree with all of the points @FiloSottile makes (unsurprisingly.) This is really a platform problem, and in an ideal world platforms would solve it for us. That said this has been a problem for a while, and as far as I can tell, there has been no real movement towards a real platform fix for containers and such. It is probably better that we provide at least some centralized solution for now, in order to dissuade people from doing the less ideal runtime hacks that are described further up this thread.

I think if we are to go down this path it should absolutely be a separate x/crypto (or perhaps a completely new x/) module, which simply provides a pre-populated x509.CertPool. This would let us decouple any urgent updates from the Go release cadence (and if done outside of x/crypto any other changes people may be less interested in pulling down). Auto-updating the module would also allow us to embed some sort of auto-destruct staleness check, preventing usage of the module if it is too out of date.

Where the list comes from is a complicated question, Mozilla is almost definitely the easiest solution, and has a pretty good history of not being full of terrible stuff. There are some not-chrome-but-adjacent Google folks working on related stuff who it might be worth talking to (I'll need to trawl my inbox to try to remember who exactly to ping.)

I agree that the prospect of adding a SetDefaultRoots type API is not ideal, and will likely result in some people setting roots in places they probably shouldn't, but this can essentially already happen with runtime linking, without a clear API around the behavior of multiple modules attempting to change the default pool. At least with a clear public API we can document the expected behavior, i.e. only allowing a single call to the method, panicking otherwise ¯_(ツ)_/¯.

[breml]

From my narrowed github user point of view, having it as a module dependency together with dependabot is nice, so I'm alerted when it gets outdated. If we want it as part of x/crypto it would make tooling such as dependabot not alert us for outdated CAs.

If I understand this correctly, this is related to the fact, that there are no tagged releases for x/crypto, correct? For me, this raises the following questions:

• Is it planned to add version tags to x/crypto in the future (if not, why)?
• Does none of the x/ modules have version tags? Would it be possible to have a x/ package for the certificates, that has version tags (to allow automated tooling to track updates)?

[rsc]

It sounds like we may want to do something to nudge people's behavior here toward things we'd prefer.

It also sounds like we could add a very tiny hook in crypto/tls (Roland said SetDefaultRoots) which can be called just once (twice is a panic), and then the actual cert sets can be provided out of standard library by x/crypto, which is easier to update. Do I have that general sketch right?

Separately, Filippo said that verifiers are starting to add different kinds of new logic. So is the "pluggable root set" the right approach or is it "pluggable ____" where we still need to fill in the blank? Maybe a verifier function or interface makes more sense than a raw CertPool?

[rsc]

/cc @rolandshoemaker

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[rolandshoemaker]

I've been mostly sitting on the fence about the GODEBUG behavior, but I think we need to make a final decision and I'm going to propose the following (which is, essentially, what is implemented in my CL):

• When calling SetFallbackRoots with GODEBUG= x509usefallbackroots=0, if the system pool is empty it will be replaced with the fallback pool. On Windows and macOS we will inherently never fallback, since we don't know anything about the content of the system pool, as we use platform APIs for verification (so we essentially always consider the pool non-empty.)
• When calling SetFallbackRoots with GODEBUG= x509usefallbackroots=1, the system root pool will be replaced regardless of its content or the platform. On macOS and Windows this will inherently disable use of the platform verification APIs since the special system pool will be replaced with a generic pool (we could also create a hybrid pool, with would result in parallel verification both with the fallback pool, and the platform APIs, but I think this is more confusing.)
• If SetFallbackRoots is not called with GODEBUG= x509usefallbackroots=1, SystemCertPool acts as if a fallback pool was never set, and behaves normally.

This gives us (mostly) uniform behavior across platforms, and is the most obvious behavior. Given this is a GODEBUG flag, even though I'm sure people will end up relying on the semantics of this behavior in strange unexpected ways, we'll have a little bit more leeway to change it in the future in case we realize down the road this is wrong.

[andig]

I like the fallback name! Definitely more descriptive than bundle :)

I was gonna ask if OS certs would be used if present. That answers it- yes.
I really like this change. One of hour top support issue with Docker is people mounting anything into /etc and then complaining about apparently unrelated networking errors. I assume this change should fix the situation.

[pete-woods]

I'm not 100% sure I follow all the discussions, but one thing in particular we (at CircleCI) would like out of this (besides the FROM scratch Docker scenario) is to force the Go based certificate store, even when os-provided certificates are present. E.g. someone runs our software in an old Docker image with an out of date set of rootcerts in.

[breml]

Hi @pete-woods

I'm not 100% sure I follow all the discussions, but one thing in particular we (at CircleCI) would like out of this (besides the FROM scratch Docker scenario) is to force the Go based certificate store, even when os-provided certificates are present. E.g. someone runs our software in an old Docker image with an out of date set of rootcerts in.

#43958 (comment) summarizes how this will work. So I guess, your use case is (only) partly solved. Whenever a user needs to force the Go embedded certificates (e.g. because the os-provided roots are out of date), he will need to pass the GODEBUG= x509usefallbackroots=1 environment variable.
I pushed for a feature like the one you are requesting, but I had no support, so this was not included.

[breml]

@pete-woods Thinking of it a little bit more, there might be a way of setting the environment variable from code in an import, that is loaded before SetFallbackRoots is called (similar to how it is done in the unit tests in 04d6aa6#diff-aab8875f722653c06a57f1a4509bdad43391f1e2dc9555644a56beeaf96bb808R91).

[rsc]

The crypto/x509 API has landed and should be in Go 1.20. For the purposes of release notes we have to say what the supporting package will be. Roland and I talked and agreed on golang.org/x/crypto/x509roots/fallback.

[andig]

I'm struggelling using the fallback package (go1.20):

go get golang.org/x/crypto/x509roots/fallback@latest
go: module golang.org/x/crypto@latest found (v0.7.0), but does not contain package golang.org/x/crypto/x509roots/fallback

go get golang.org/x/crypto/x509roots/fallback@master
go: downloading golang.org/x/crypto v0.7.1-0.20230320203329-018c28f8a114
go: module golang.org/x/crypto@master found (v0.7.1-0.20230320203329-018c28f8a114), but does not contain package golang.org/x/crypto/x509roots/fallback

[pete-woods]

I don't think this package has actually been released yet (despite being referred to in the 1.20 release notes)

[dmitshur]

@rolandshoemaker With SetFallbackRoots present and documented in Go 1.20, and golang.org/x/crypto/x509roots/fallback being recently added as part of #57792, is it time to move this issue to Go1.20 milestone and close it? Or is there more that still needs to happen here?

[rolandshoemaker]

Yes, thanks for the reminder! 🎉