crypto/x509: TestSystemRoots failure #21416

martisch · 2017-08-12T14:34:56Z

go tip 23cd87e and earlier in 1.9 candidates

while running ./all.bash i have a reproducible test error on darwin:
--- FAIL: TestSystemRoots (1.41s)
root_darwin_test.go:31: cgo sys roots: 443.728642ms
root_darwin_test.go:32: non-cgo sys roots: 964.424609ms
root_darwin_test.go:44: got 169 roots
root_darwin_test.go:44: got 453 roots
root_darwin_test.go:73: insufficient overlap between cgo and non-cgo roots; want at least 226, have 168
FAIL

macOS Sierra 10.12.6 but i have seen that failure on macOS 10.11 too since i just upgraded.

Darwin ender 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64

another similar report here:
https://groups.google.com/forum/#!topic/golang-nuts/LZvj2N_8gs4

The text was updated successfully, but these errors were encountered:

martisch · 2017-08-12T14:47:20Z

happens since: 4dbcacd... crypto/x509: load all trusted certs on darwin (nocgo)

4dbcacda96... crypto/x509: load all trusted certs on darwin (nocgo)
ender:x509 martisch$ go test -run=TestSystemRoots
--- FAIL: TestSystemRoots (0.74s)
	root_darwin_test.go:31:     cgo sys roots: 124.339674ms
	root_darwin_test.go:32: non-cgo sys roots: 614.074109ms
	root_darwin_test.go:44: got 169 roots
	root_darwin_test.go:44: got 453 roots
	root_darwin_test.go:73: insufficient overlap between cgo and non-cgo roots; want at least 226, have 168
FAIL
exit status 1
FAIL	crypto/x509	0.752s

/cc @dsnet @ianlancetaylor

martisch · 2017-08-19T05:14:04Z

adding authors and more reviewers from the CL
/cc @mastercactapus @josharian @agl

martisch · 2017-08-19T05:30:10Z

I tagged this Go1.9 since i traced it to a CL merged at end of 1.9 cycle and it would be nice to have a working test suite on darwin with 1.9 release and also counter confusion and issue tickets from any others that will encounter the above failure when compiling 1.9 from source on darwin.

mastercactapus · 2017-08-19T14:59:37Z

I suspect this is due to the cgo counterpart to that fix not having made it into 1.9 (as it's still in-progress)
https://go-review.googlesource.com/c/36942

gopherbot · 2017-08-22T18:28:52Z

Change https://golang.org/cl/57830 mentions this issue: crypto/x509: skip TestSystemRoots

golang.org/cl/36941 enabled loading of all trusted certs on darwin for the non-cgo execSecurityRoots. The corresponding cgo version golang.org/cl/36942 for systemRootsPool has not been merged yet. This tests fails reliably on some darwin systems: --- FAIL: TestSystemRoots (1.28s) root_darwin_test.go:31: cgo sys roots: 353.552363ms root_darwin_test.go:32: non-cgo sys roots: 921.85297ms root_darwin_test.go:44: got 169 roots root_darwin_test.go:44: got 455 roots root_darwin_test.go:73: insufficient overlap between cgo and non-cgo roots; want at least 227, have 168 FAIL FAIL crypto/x509 2.445s Updates #16532 Updates #21416 Change-Id: I52c2c847651fb3621fdb6ab858ebe8e28894c201 Reviewed-on: https://go-review.googlesource.com/57830 Run-TryBot: Martin Möhrmann <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Joe Tsai <[email protected]>

adamryman · 2017-08-24T22:56:32Z

If this is a release blocker why did 1.9 get released without this?

martisch · 2017-08-25T04:07:49Z

Because the label was a suggestion by me but the as far as i know the decision was made by the go team that the issue should not hold up the release of 1.9 (which sounds like the right call to me). It can be considered for 1.9.x.

ianlancetaylor · 2017-10-13T18:01:45Z

If I'm reading this correctly, the fix for this test failure is to disable the test.

I guess that is OK for a backport to 1.9.

rsc · 2017-10-13T18:41:43Z

Sure, we can disable the test that non-cgo works here.

~~CL 57830 OK for Go 1.9.2.~~

rsc · 2017-10-13T19:57:51Z

I'm going to remilestone this to Go 1.10 for a proper fix, since there is good context in this thread, and I created #22256 for the Go 1.9.2 cherry-pick.

gopherbot · 2017-10-15T00:27:59Z

Change https://golang.org/cl/70847 mentions this issue: [release-branch.go1.9] crypto/x509: skip TestSystemRoots

golang.org/cl/36941 enabled loading of all trusted certs on darwin for the non-cgo execSecurityRoots. The corresponding cgo version golang.org/cl/36942 for systemRootsPool has not been merged yet. This tests fails reliably on some darwin systems: --- FAIL: TestSystemRoots (1.28s) root_darwin_test.go:31: cgo sys roots: 353.552363ms root_darwin_test.go:32: non-cgo sys roots: 921.85297ms root_darwin_test.go:44: got 169 roots root_darwin_test.go:44: got 455 roots root_darwin_test.go:73: insufficient overlap between cgo and non-cgo roots; want at least 227, have 168 FAIL FAIL crypto/x509 2.445s Updates #16532 Updates #21416 Change-Id: I52c2c847651fb3621fdb6ab858ebe8e28894c201 Reviewed-on: https://go-review.googlesource.com/57830 Run-TryBot: Martin Möhrmann <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Joe Tsai <[email protected]> Reviewed-on: https://go-review.googlesource.com/70847 Run-TryBot: Russ Cox <[email protected]> Reviewed-by: Martin Möhrmann <[email protected]>

gopherbot · 2018-06-07T15:19:12Z

Change https://golang.org/cl/117055 mentions this issue: crypto/x509: re-enable TestSystemRoots on darwin

It was apparently waiting on CL 36942, which was submitted. Fixes golang#21416 Change-Id: I8f4ccc5a3176070abf0df019c82700c5761b5f53 Reviewed-on: https://go-review.googlesource.com/117055 Run-TryBot: Brad Fitzpatrick <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Ian Lance Taylor <[email protected]>

martisch · 2018-07-06T07:23:26Z

reopening because after submission of https://golang.org/cl/117055
./all.bash on tip is broken again on my private macbook pro (Darwin ender 16.7.0 Darwin Kernel Version 16.7.0: Fri Apr 27 17:59:46 PDT 2018; root:xnu-3789.73.13~1/RELEASE_X86_64 x86_64
):

$ git checkout 2642df9
$ ./all.bash
...
--- FAIL: TestSystemRoots (4.97s)
root_darwin_test.go:31: cgo sys roots: 420.836945ms
root_darwin_test.go:32: non-cgo sys roots: 4.545627256s
root_darwin_test.go:44: got 177 roots
root_darwin_test.go:44: got 467 roots
root_darwin_test.go:73: insufficient overlap between cgo and non-cgo roots; want at least 233, have 172
FAIL
FAIL crypto/x509 6.895s

@bradfitz
can we rollback cl/117055 until the underlying issue is fixed (happy to provide more information until root cause of problem is found)?

gopherbot · 2018-07-06T07:31:41Z

Change https://golang.org/cl/122435 mentions this issue: Revert "crypto/x509: re-enable TestSystemRoots on darwin"

adamdecaf · 2018-07-06T13:59:13Z

@martisch Could you try with these CL's and see if the test is fixed?

#24652 (comment)

martisch · 2018-07-13T07:16:54Z

applied 3 patches from #24652 to tip:
ec09ea9fa4 (HEAD -> x509test) crypto/x509: on darwin verify-cert with ssl policy
54a3bdbe1e crypto/x509: collect keychain certs without explicit usage constraints
a6f6cfa570 crypto/x509: check darwin keychain policy in cgo path
8a33045 (origin/master, origin/HEAD, master) net/url: don't escape sub-delims in fragment

result:
--- FAIL: TestSystemRoots (1.58s)
root_darwin_test.go:31: cgo sys roots: 126.320357ms
root_darwin_test.go:32: non-cgo sys roots: 1.44816439s
root_darwin_test.go:44: got 177 roots
root_darwin_test.go:44: got 467 roots
root_darwin_test.go:73: insufficient overlap between cgo and non-cgo roots; want at least 233, have 171

any other CLs i should apply?

adamdecaf · 2018-07-13T18:33:19Z

@martisch 400+ roots seems like a lot compared to my system. I only have two custom CA's installed.

I tried from 8a33045 (in your comment) and ebdba42 with both passing:

// https://github.com/golang/go/commit/ebdba42d9e1de46ebf611baec98d53f01c534cac
$ ../bin/go test -v -count 1 -run TestSystemRoots crypto/x509 
=== RUN   TestSystemRoots
--- PASS: TestSystemRoots (3.03s)
    root_darwin_test.go:31:     cgo sys roots: 1.467407679s
    root_darwin_test.go:32: non-cgo sys roots: 1.5584415s
    root_darwin_test.go:44: got 169 roots
    root_darwin_test.go:44: got 173 roots
PASS
ok  	crypto/x509	3.043s

Could you explain more about the certificates picked up in the non-cgo route? Adding GODEBUG=x509roots=1 will list them out as they're picked up. (Please from from tip as there's better debugging.)

martisch · 2018-07-14T20:04:49Z

I have a lot of email certificates in my login key chain from smime signed emails from a common authority (e.g. DFN-PKI) . AFAIK apple mail adds them automatically to my keychain when i receive signed mails and they have been verified against a trusted authority. I just deleted one of those and the number of "got roots" decreased by one. The trust on those certificates is system default and they are not system root certificates.

GODEBUG=x509roots=1 go test
crypto/x509: 12 certs have a trust policy
Prints some custom roots for email signing within orgs, expired roots (accumulated over 10 years), system management software root certs. There are only ~20 in total listed with some expired. Some of them were double likely because i have them in two different key chains.

adamdecaf · 2018-07-17T14:25:18Z

It's been mentioned crypto/x509 is just for the web PKI, so we could probably exclude these certs from the SystemCertPool() result.

Do these certificates only have SubjectAltNames's for emails or Extended KeyUsage for email?

martisch · 2018-07-19T05:07:20Z

The certificates have:
X509v3 Key Usage: Digital Signature, Non Repudiation, Key Encipherment
X509v3 Extended Key Usage: TLS Web Client Authentication, E-mail Protection, Microsoft Smartcardlogin

and:
X509v3 Subject Alternative Name:
email: X, email:Y .....

(Where X and Y are placeholders here for actual email addresses)

adamdecaf · 2018-07-19T14:49:50Z

@FiloSottile Should we ignore picking up certs that only have email SANs?

Also, I don't really see crypto/x509 documented as "web pki only", but should we?

gopherbot · 2018-07-20T14:46:08Z

Change https://golang.org/cl/125259 mentions this issue: crypto/x509: skip TestSystemRoots

cgo and non-cgo code paths can disagree on the number of root certificates: === RUN TestSystemRoots --- FAIL: TestSystemRoots (0.31s) root_darwin_test.go:31: cgo sys roots: 93.605184ms root_darwin_test.go:32: non-cgo sys roots: 213.998586ms root_darwin_test.go:44: got 168 roots root_darwin_test.go:44: got 427 roots root_darwin_test.go:73: insufficient overlap between cgo and non-cgo roots; want at least 213, have 168 FAIL exit status 1 Updates #21416 Updates #24652 Change-Id: Idb6d35b17c142dfff79a10cf6b40a42d12f9d17e Reviewed-on: https://go-review.googlesource.com/125259 Run-TryBot: Martin Möhrmann <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Brad Fitzpatrick <[email protected]>

bcmills · 2018-12-11T14:39:36Z

This is failing again on the 10.11 builders (but not the other darwin builders) as of https://golang.org/cl/128117.

--- FAIL: TestSystemRoots (0.30s)
    root_darwin_test.go:34:     cgo sys roots: 66.259292ms
    root_darwin_test.go:35: non-cgo sys roots: 201.101857ms
    root_darwin_test.go:76: signed certificate only present in non-cgo pool (acceptable): CN=Developer ID Certification Authority,OU=Apple Certification Authority,O=Apple Inc.,C=US
    root_darwin_test.go:103: expired certificate only present in cgo pool (acceptable): OU=Equifax Secure Certificate Authority,O=Equifax,C=US
    root_darwin_test.go:107: certificate only present in cgo pool: CN=Thawte Premium Server CA,OU=Certification Services Division,O=Thawte Consulting cc,L=Cape Town,ST=Western Cape,C=ZA
    root_darwin_test.go:103: expired certificate only present in cgo pool (acceptable): CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
    root_darwin_test.go:107: certificate only present in cgo pool: CN=Thawte Server CA,OU=Certification Services Division,O=Thawte Consulting cc,L=Cape Town,ST=Western Cape,C=ZA
    root_darwin_test.go:107: certificate only present in cgo pool: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

FiloSottile · 2018-12-11T18:11:04Z

I'm working on it. I have so far not figured out where those three certificates come from.

katiehockman · 2019-01-02T16:39:23Z

Just a note that this is still failing the darwin 10.11 builds.

There is also a similar failure for darwin-amd64-race. Example: https://build.golang.org/log/a93b418cfdfb8f67374c79b7bb2fb2a804c53fc6

gopherbot · 2019-01-05T00:14:51Z

Change https://golang.org/cl/156329 mentions this issue: crypto/x509: ignore 5 phantom 1024-bit roots in TestSystemRoots

martisch added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels Aug 12, 2017

martisch added this to the Go1.9 milestone Aug 16, 2017

dsnet removed the release-blocker label Aug 25, 2017

aclements modified the milestones: Go1.9, Go1.9.1 Aug 25, 2017

rsc modified the milestones: Go1.9.1, Go1.9.2 Oct 4, 2017

rsc mentioned this issue Oct 13, 2017

crypto/x509: disable TestSystemRoots on non-cgo darwin #22256

Closed

rsc modified the milestones: Go1.9.2, Go1.10 Oct 13, 2017

rsc modified the milestones: Go1.10, Go1.11 Nov 22, 2017

FiloSottile added Testing An issue that has been verified to require only test changes, not just a test failure. release-blocker labels May 17, 2018

bradfitz self-assigned this Jun 7, 2018

gopherbot closed this as completed in 2642df9 Jun 14, 2018

martisch reopened this Jul 6, 2018

FiloSottile assigned FiloSottile and unassigned bradfitz Jul 27, 2018

rsc modified the milestones: Go1.11, Go1.12 Aug 17, 2018

FiloSottile mentioned this issue Dec 18, 2018

all: Go 1.12beta1 release status #29324

Closed

katiehockman added the OS-Darwin label Jan 2, 2019

gopherbot closed this as completed in 303a596 Jan 5, 2019

golang locked and limited conversation to collaborators Jan 5, 2020

gopherbot added the FrozenDueToAge label Jan 5, 2020

rsc unassigned FiloSottile Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crypto/x509: TestSystemRoots failure #21416

crypto/x509: TestSystemRoots failure #21416

martisch commented Aug 12, 2017 •

edited

Loading

martisch commented Aug 12, 2017 •

edited

Loading

martisch commented Aug 19, 2017 •

edited

Loading

martisch commented Aug 19, 2017

mastercactapus commented Aug 19, 2017

gopherbot commented Aug 22, 2017

adamryman commented Aug 24, 2017

martisch commented Aug 25, 2017 •

edited

Loading

ianlancetaylor commented Oct 13, 2017

rsc commented Oct 13, 2017 •

edited

Loading

rsc commented Oct 13, 2017

gopherbot commented Oct 15, 2017

gopherbot commented Jun 7, 2018

martisch commented Jul 6, 2018

gopherbot commented Jul 6, 2018

adamdecaf commented Jul 6, 2018

martisch commented Jul 13, 2018

adamdecaf commented Jul 13, 2018 •

edited

Loading

martisch commented Jul 14, 2018 •

edited

Loading

adamdecaf commented Jul 17, 2018

martisch commented Jul 19, 2018

adamdecaf commented Jul 19, 2018

gopherbot commented Jul 20, 2018

bcmills commented Dec 11, 2018

FiloSottile commented Dec 11, 2018

katiehockman commented Jan 2, 2019 •

edited

Loading

gopherbot commented Jan 5, 2019

crypto/x509: TestSystemRoots failure #21416

crypto/x509: TestSystemRoots failure #21416

Comments

martisch commented Aug 12, 2017 • edited Loading

martisch commented Aug 12, 2017 • edited Loading

martisch commented Aug 19, 2017 • edited Loading

martisch commented Aug 19, 2017

mastercactapus commented Aug 19, 2017

gopherbot commented Aug 22, 2017

adamryman commented Aug 24, 2017

martisch commented Aug 25, 2017 • edited Loading

ianlancetaylor commented Oct 13, 2017

rsc commented Oct 13, 2017 • edited Loading

rsc commented Oct 13, 2017

gopherbot commented Oct 15, 2017

gopherbot commented Jun 7, 2018

martisch commented Jul 6, 2018

gopherbot commented Jul 6, 2018

adamdecaf commented Jul 6, 2018

martisch commented Jul 13, 2018

adamdecaf commented Jul 13, 2018 • edited Loading

martisch commented Jul 14, 2018 • edited Loading

adamdecaf commented Jul 17, 2018

martisch commented Jul 19, 2018

adamdecaf commented Jul 19, 2018

gopherbot commented Jul 20, 2018

bcmills commented Dec 11, 2018

FiloSottile commented Dec 11, 2018

katiehockman commented Jan 2, 2019 • edited Loading

gopherbot commented Jan 5, 2019

martisch commented Aug 12, 2017 •

edited

Loading

martisch commented Aug 12, 2017 •

edited

Loading

martisch commented Aug 19, 2017 •

edited

Loading

martisch commented Aug 25, 2017 •

edited

Loading

rsc commented Oct 13, 2017 •

edited

Loading

adamdecaf commented Jul 13, 2018 •

edited

Loading

martisch commented Jul 14, 2018 •

edited

Loading

katiehockman commented Jan 2, 2019 •

edited

Loading