retrieving external modules on Go1.15 on s390x appears to have checksum and ECDSA verification issues

[justaugustus]

What version of Go are you using (go version)?

$ go version
go version go1.15 linux/s390x

Does this issue reproduce with the latest release?

Yep! go1.15 as of filing this issue (8/21/20).

What operating system and processor architecture are you using (go env)?

This is the s390x stage of a multi-arch container build running in Google Cloud Build (linux/amd64) using docker run --rm --privileged multiarch/qemu-user-static@sha256:c772ee1965aa0be9915ee1b018a0dd92ea361b4fa1bcab5bbc033517749b2af4 --reset -p yes to configure the environment.

go env Output

$ go env
GO111MODULE=""
GOARCH="s390x"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="s390x"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org|direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_s390x"
GCCGO="gccgo"
AR="ar"
CC="s390x-linux-gnu-gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/workspace/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -march=z196 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build401489570=/tmp/go-build -gno-record-gcc-switches"

What did you do?

For background, in the Kubernetes project, we recently switched an image called go-runner, which is essentially distroless/static with some minor modifications to support our logging and exec for our core binaries.

That image's Dockerfile builds the go-runner command in a golang:x.y container and then copies the binary into a gcr.io/distroless/static-debian10:latest image.

As we've recently updated to go1.15 on kubernetes/kubernetes (kubernetes/kubernetes#93939, kubernetes/release#1421), I wanted to update this image from go1.13.15 to go1.15 before our tentative release on 8/25.

The work-in-progress PR for this is here: kubernetes/release#1499

What did you expect to see?

All platforms in this container build should have succeeded.

What did you see instead?

Across several debugging runs of the container build, I saw issues with downloading github.com/pkg/[email protected]...

go: github.com/pkg/[email protected]: Get "https://proxy.golang.org/github.com/pkg/errors/@v/v0.9.1.mod": tls: invalid signature by the server certificate: ECDSA verification failure

With GOPROXY=direct

go mod download -json

{
	"Path": "github.com/pkg/errors",
	"Version": "v0.9.1",
	"Error": "zip: checksum error",
	"Info": "/go/pkg/mod/cache/download/github.com/pkg/errors/@v/v0.9.1.info",
	"GoMod": "/go/pkg/mod/cache/download/github.com/pkg/errors/@v/v0.9.1.mod",
	"GoModSum": "h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0="
}

I've tried with various permutations of GOPROXY:

• https://proxy.golang.org (what it was originally explicitly configured with)
• https://proxy.golang.org,direct (current go1.15 default)
• https://proxy.golang.org|direct (fallback to direct, new with go1.15)
• direct
• off

When curl-ing the proxy from within s390x build, I get the same values sum-wise:

#14 [builder  8/15] RUN curl -v https://proxy.golang.org/github.com/pkg/erro...
#14 0.182 * Expire in 0 ms for 6 (transfer 0x40017a4bb0)
#14 0.197 * Expire in 1 ms for 1 (transfer 0x40017a4bb0)
#14 0.204   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
#14 0.205                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Expire in 5 ms for 1 (transfer 0x40017a4bb0)
#14 0.209 * Expire in 2 ms for 1 (transfer 0x40017a4bb0)
#14 0.209 * Expire in 6 ms for 1 (transfer 0x40017a4bb0)
#14 0.210 * Expire in 6 ms for 1 (transfer 0x40017a4bb0)
#14 0.215 * Expire in 4 ms for 1 (transfer 0x40017a4bb0)
#14 0.215 * Expire in 8 ms for 1 (transfer 0x40017a4bb0)
#14 0.215 * Expire in 8 ms for 1 (transfer 0x40017a4bb0)
#14 0.216 * Expire in 4 ms for 1 (transfer 0x40017a4bb0)
#14 0.216 * Expire in 8 ms for 1 (transfer 0x40017a4bb0)
#14 0.216 * Expire in 8 ms for 1 (transfer 0x40017a4bb0)
#14 0.223 * Expire in 8 ms for 1 (transfer 0x40017a4bb0)
#14 0.223 * Expire in 10 ms for 1 (transfer 0x40017a4bb0)
#14 0.223 * Expire in 10 ms for 1 (transfer 0x40017a4bb0)
#14 0.224 * Expire in 8 ms for 1 (transfer 0x40017a4bb0)
#14 0.224 * Expire in 11 ms for 1 (transfer 0x40017a4bb0)
#14 0.224 * Expire in 11 ms for 1 (transfer 0x40017a4bb0)
#14 0.234 * Expire in 16 ms for 1 (transfer 0x40017a4bb0)
#14 0.234 * Expire in 14 ms for 1 (transfer 0x40017a4bb0)
#14 0.235 * Expire in 14 ms for 1 (transfer 0x40017a4bb0)
#14 0.236 * Expire in 16 ms for 1 (transfer 0x40017a4bb0)
#14 0.236 * Expire in 15 ms for 1 (transfer 0x40017a4bb0)
#14 0.236 * Expire in 15 ms for 1 (transfer 0x40017a4bb0)
#14 0.251 * Expire in 50 ms for 1 (transfer 0x40017a4bb0)
#14 0.253 *   Trying 173.194.214.141...
#14 0.253 * TCP_NODELAY set
#14 0.254 * Expire in 149967 ms for 3 (transfer 0x40017a4bb0)
#14 0.254 * Expire in 200 ms for 4 (transfer 0x40017a4bb0)
#14 0.256 * Connected to proxy.golang.org (173.194.214.141) port 443 (#0)
#14 0.284 * ALPN, offering h2
#14 0.284 * ALPN, offering http/1.1
#14 0.284 * successfully set certificate verify locations:
#14 0.284 *   CAfile: none
#14 0.284   CApath: /etc/ssl/certs
#14 0.300 } [5 bytes data]
#14 0.301 * TLSv1.3 (OUT), TLS handshake, Client hello (1):
#14 0.301 } [512 bytes data]
#14 0.305 * TLSv1.3 (IN), TLS handshake, Server hello (2):
#14 0.305 { [122 bytes data]
#14 0.320 * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
#14 0.320 { [15 bytes data]
#14 0.320 * TLSv1.3 (IN), TLS handshake, Certificate (11):
#14 0.320 { [7012 bytes data]
#14 0.368 * TLSv1.3 (IN), TLS handshake, CERT verify (15):
#14 0.368 { [79 bytes data]
#14 0.382 * TLSv1.3 (IN), TLS handshake, Finished (20):
#14 0.382 { [52 bytes data]
#14 0.383 * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
#14 0.383 } [1 bytes data]
#14 0.385 * TLSv1.3 (OUT), TLS handshake, Finished (20):
#14 0.385 } [52 bytes data]
#14 0.386 * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
#14 0.386 * ALPN, server accepted to use h2
#14 0.387 * Server certificate:
#14 0.388 *  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=misc-sni.google.com
#14 0.388 *  start date: Aug 11 08:54:10 2020 GMT
#14 0.389 *  expire date: Nov  3 08:54:10 2020 GMT
#14 0.391 *  subjectAltName: host "proxy.golang.org" matched cert's "*.golang.org"
#14 0.391 *  issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
#14 0.391 *  SSL certificate verify ok.
#14 0.393 * Using HTTP2, server supports multi-use
#14 0.393 * Connection state changed (HTTP/2 confirmed)
#14 0.394 * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
#14 0.395 } [5 bytes data]
#14 0.399 * Using Stream ID: 1 (easy handle 0x40017a4bb0)
#14 0.402 } [5 bytes data]
#14 0.402 > GET /github.com/pkg/errors/@v/v0.9.1.mod HTTP/2
#14 0.402 > Host: proxy.golang.org
#14 0.402 > User-Agent: curl/7.64.0
#14 0.402 > Accept: */*
#14 0.402 > 
#14 0.404 { [5 bytes data]
#14 0.405 * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
#14 0.405 } [5 bytes data]
#14 0.411 < HTTP/2 200 
#14 0.412 < accept-ranges: bytes
#14 0.412 < access-control-allow-origin: *
#14 0.412 < content-length: 29
#14 0.413 < content-type: text/plain; charset=UTF-8
#14 0.413 < date: Fri, 21 Aug 2020 12:19:38 GMT
#14 0.413 < etag: "df28c6a823f181d76179697177c0c5943c6ffb38f3c10b2dc53be360ee7d4589"
#14 0.413 < expires: Fri, 21 Aug 2020 15:19:38 GMT
#14 0.413 < last-modified: Tue, 14 Jan 2020 19:51:40 GMT
#14 0.413 < x-content-type-options: nosniff
#14 0.413 < x-frame-options: SAMEORIGIN
#14 0.413 < x-xss-protection: 0
#14 0.413 < age: 2562
#14 0.413 < cache-control: public, max-age=10800
#14 0.413 < 
#14 0.414 { [5 bytes data]
100    29  100    29    0     0    124      0 --:--:-- --:--:-- --:--:--   135
#14 0.419 * Connection #0 to host proxy.golang.org left intact
#14 0.420 module github.com/pkg/errors
#14 DONE 0.5s

#15 [builder  9/15] RUN curl -v https://sum.golang.org/lookup/github.com/pkg...
#15 0.165 * Expire in 0 ms for 6 (transfer 0x40017a4bb0)
#15 0.180 * Expire in 1 ms for 1 (transfer 0x40017a4bb0)
#15 0.186   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
#15 0.187                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Expire in 6 ms for 1 (transfer 0x40017a4bb0)
#15 0.198 * Expire in 2 ms for 1 (transfer 0x40017a4bb0)
#15 0.198 * Expire in 8 ms for 1 (transfer 0x40017a4bb0)
#15 0.199 * Expire in 8 ms for 1 (transfer 0x40017a4bb0)
#15 0.207 * Expire in 4 ms for 1 (transfer 0x40017a4bb0)
#15 0.207 * Expire in 11 ms for 1 (transfer 0x40017a4bb0)
#15 0.207 * Expire in 11 ms for 1 (transfer 0x40017a4bb0)
#15 0.217 * Expire in 8 ms for 1 (transfer 0x40017a4bb0)
#15 0.217 * Expire in 14 ms for 1 (transfer 0x40017a4bb0)
#15 0.217 * Expire in 14 ms for 1 (transfer 0x40017a4bb0)
#15 0.218 * Expire in 8 ms for 1 (transfer 0x40017a4bb0)
#15 0.218 * Expire in 15 ms for 1 (transfer 0x40017a4bb0)
#15 0.218 * Expire in 15 ms for 1 (transfer 0x40017a4bb0)
#15 0.234 * Expire in 50 ms for 1 (transfer 0x40017a4bb0)
#15 0.235 *   Trying 172.217.203.141...
#15 0.235 * TCP_NODELAY set
#15 0.236 * Expire in 149967 ms for 3 (transfer 0x40017a4bb0)
#15 0.236 * Expire in 200 ms for 4 (transfer 0x40017a4bb0)
#15 0.238 * Connected to sum.golang.org (172.217.203.141) port 443 (#0)
#15 0.266 * ALPN, offering h2
#15 0.266 * ALPN, offering http/1.1
#15 0.266 * successfully set certificate verify locations:
#15 0.267 *   CAfile: none
#15 0.267   CApath: /etc/ssl/certs
#15 0.282 } [5 bytes data]
#15 0.284 * TLSv1.3 (OUT), TLS handshake, Client hello (1):
#15 0.284 } [512 bytes data]
#15 0.286 * TLSv1.3 (IN), TLS handshake, Server hello (2):
#15 0.286 { [122 bytes data]
#15 0.302 * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
#15 0.302 { [15 bytes data]
#15 0.303 * TLSv1.3 (IN), TLS handshake, Certificate (11):
#15 0.303 { [7012 bytes data]
#15 0.349 * TLSv1.3 (IN), TLS handshake, CERT verify (15):
#15 0.349 { [79 bytes data]
#15 0.364 * TLSv1.3 (IN), TLS handshake, Finished (20):
#15 0.364 { [52 bytes data]
#15 0.365 * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
#15 0.365 } [1 bytes data]
#15 0.367 * TLSv1.3 (OUT), TLS handshake, Finished (20):
#15 0.367 } [52 bytes data]
#15 0.368 * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
#15 0.368 * ALPN, server accepted to use h2
#15 0.368 * Server certificate:
#15 0.370 *  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=misc-sni.google.com
#15 0.370 *  start date: Aug 11 08:54:10 2020 GMT
#15 0.370 *  expire date: Nov  3 08:54:10 2020 GMT
#15 0.372 *  subjectAltName: host "sum.golang.org" matched cert's "*.golang.org"
#15 0.373 *  issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
#15 0.373 *  SSL certificate verify ok.
#15 0.375 * Using HTTP2, server supports multi-use
#15 0.375 * Connection state changed (HTTP/2 confirmed)
#15 0.376 * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
#15 0.377 } [5 bytes data]
#15 0.380 * Using Stream ID: 1 (easy handle 0x40017a4bb0)
#15 0.383 } [5 bytes data]
#15 0.383 > GET /lookup/github.com/pkg/[email protected] HTTP/2
#15 0.383 > Host: sum.golang.org
#15 0.383 > User-Agent: curl/7.64.0
#15 0.383 > Accept: */*
#15 0.383 > 
#15 0.385 { [5 bytes data]
#15 0.386 * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
#15 0.387 } [5 bytes data]
#15 0.393 < HTTP/2 200 
#15 0.393 < accept-ranges: bytes
#15 0.393 < access-control-allow-origin: *
#15 0.393 < content-length: 356
#15 0.394 < content-type: text/plain; charset=UTF-8
#15 0.394 < date: Fri, 21 Aug 2020 10:44:27 GMT
#15 0.394 < expires: Fri, 21 Aug 2020 13:44:27 GMT
#15 0.394 < x-content-type-options: nosniff
#15 0.394 < x-frame-options: SAMEORIGIN
#15 0.394 < x-xss-protection: 0
#15 0.394 < cache-control: public, max-age=10800
#15 0.394 < age: 8273
#15 0.394 < 
#15 0.395 { [5 bytes data]
100   356  100   356    0     0   1534      0 --:--:-- --:--:-- --:--:--  1671
#15 0.400 * Connection #0 to host sum.golang.org left intact
#15 0.401 706716
#15 0.401 github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
#15 0.401 github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
#15 0.401 
#15 0.401 go.sum database tree
#15 0.401 1540930
#15 0.401 DixXf8ysWPUBPAyGyzq1QtEoWiWa+w6vBy8G7r4b7hc=
#15 0.401 
#15 0.401 — sum.golang.org Az3gruyZPrwSZ75Hbl3PVkKsDqLxrzAlZqX1pui0qTXKvs3FlgaYOIRU4V8Dc6Y57IZt5OmvXOPf06N1JOrAWB39oQg=
#15 DONE 0.5s

cc: @dims @liggitt @BenTheElder

[odeke-em]

Thank you for this report @justaugustus and welcome to the Go project!

There was a crypto/ed25519 s390x issue in #40475 which was spawned a revert in https://go-review.googlesource.com/c/go/+/245497 or 54e75e8

Kindly /cc-ing @mundaym @FiloSottile

[mundaym]

I don't think this has anything to do with crypto/ed25519.

I remember @mvdan running into a similar issue. It turned out to be a bug in qemu. Let me see if I can dig that up.

[justaugustus]

@odeke-em @mundaym -- Thanks for your help with this!

[mvdan]

I was hitting tls: bad record MAC errors, among some others. The upstream bug was https://bugs.launchpad.net/qemu/+bug/1847232/, which got fixed sometime earlier this year. The Moby (Docker) people also encountered it pretty often, and here is their issue to track the problem: moby/moby#40240

I would suggest to try the latest QEMU version, or at least check what version you're using and seeing if it includes the upstream fix.

[mvdan]

Also, I never ended up filing an issue on this tracker, as we ended up finding that it wasn't a bug in Go at all, like @mundaym said.

[liggitt]

That looks like it explains the tls errors, which is great. Does it also explain the zip checksum error with GOPROXY=direct?

With GOPROXY=direct

go mod download -json

{
	"Path": "github.com/pkg/errors",
	"Version": "v0.9.1",
	"Error": "zip: checksum error",
	"Info": "/go/pkg/mod/cache/download/github.com/pkg/errors/@v/v0.9.1.info",
	"GoMod": "/go/pkg/mod/cache/download/github.com/pkg/errors/@v/v0.9.1.mod",
	"GoModSum": "h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0="
}

[mvdan]

The QEMU bug was ultimately about hashing, so yeah, I would definitely expect that problem to be caused by the same QEMU bug.

[liggitt]

interesting, I didn't expect system hash functions to be used for go module hash calculations

[mvdan]

Ultimately I'm not sure, but it's a pretty rare problem to encounter, so I'd blame the same QEMU bug unless I have proof pointing in another direction :)

[mundaym]

The conversation in https://bugs.launchpad.net/qemu/+bug/1847232/ seems to indicate that they think it is a bug in vector instruction emulation (the 'vx' instruction set is an instruction set extension for s390x, similar to something like AVX for x86). The poly1305 code (ultimately used by crypto/tls) uses those instructions which was probably why the MAC was incorrect.

This looks a little different to me. There are a couple of issues in the multiarch/qemu-user-static repo related to s390x, though nothing I can see that exactly matches this (illegal instruction/hanging app).

We did make some changes to the crypto/ecdsa package this cycle but those are specific to the z15 model. I'd be suprised if qemu is emulating the z15 by default since that hardware is only a year old.

[mundaym]

But yes, if you are able to update qemu that is probably the best place to start. I'd also be interested to see if Go 1.14.x has the same issue if that is easy to try.

[mdempsky]

It seems like this is suspected to be a third-party tooling issue (qemu) without any action for the Go team. Also, all of the issues referencing this one have been closed, so I think we can close this one too.

Please ping if we should reopen. Thanks.