x/build/cmd/release: green-screen on Windows ARM64 builder with SkipTests: false

[toothrot]

It's unclear what is causing this issue on the Windows ARM64 builders, only during the release process. The builders are otherwise passing tests on the Go dashboard.

Discovered while testing #42604.

What version of Go are you using (go version)?

Go tip: 4711bf3

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

windows/arm64

What did you do?

@@ -206,10 +206,6 @@ var builds = []*Build{
                Arch:    "arm64",
                Race:    false, // Not supported as of 2021-06-01.
                Builder: "windows-arm64-10",
-               // TODO(golang.org/issue/46406, golang.org/issue/46502): Fix
-               // or skip failing tests, ensure the builder is fast enough to
-               // complete tests, then remove SkipTests here.
-               SkipTests: true,
        },

go run ./cmd/release -rev master -target windows-arm64 -version go1.17beta900

What did you expect to see?

A successful build with passing tests

What did you see instead?

2021/06/30 12:39:40 windows-arm64: Building (all.bash to ensure tests pass).
 /status
2021/06/30 12:45:17 user-rakoczy-windows-arm64-10-1: timeout after 10s waiting for headers for /status
2021/06/30 12:45:37 user-rakoczy-windows-arm64-10-1: timeout after 10s waiting for headers for /status
2021/06/30 12:45:37 Buildlet https://farmer.golang.org:443 failed three heartbeats; final error: timeout waiting for headers
2021/06/30 12:45:37 user-rakoczy-windows-arm64-10-1: peer dead with Buildlet https://farmer.golang.org:443 failed heartbeat after 10.001424152s; marking dead; err=timeout waiting for headers, waiting for headers for /halt
2021/06/30 12:45:37 windows-arm64: Error: Buildlet https://farmer.golang.org:443 failed heartbeat after 10.001424152s; marking dead; err=timeout waiting for headers

Stop code: MEMORY MANAGEMENT
Stop code: IRQL NOT LESS OR EQUAL

[dmitshur]

I tried to gather more information on this issue by making use of a physical windows/arm64 machine I have access to. I ran all.bat a number of times on it, and can confirm this issue doesn't reproduce on it, as all.bat passes okay (even with GOROOT_FINAL and other environment variables set as cmd/release does) without causing any green-screens. I tested with a recent Go tip commit, b7a85e0.

[dmitshur]

I've repeated this exercise using the windows-arm64-10 builder, and the green-screens no longer happen reproducibly. I've still observed at least 1 instance out of 5 runs, filed #47965 for that.

windows-arm64, 1.17 + CL 344674*
--------------------------------

$ go run . -target=windows-arm64 -version=go1.17beta700 -rev=12e8b254cbd6f8c2014473cc6f44fe9a2a07d90a -watch

run 1 = "ALL TESTS PASSED"
run 2 = "ALL TESTS PASSED"
run 3 = "ALL TESTS PASSED"

* This CL removes a test that is known to be flaky, which the windows/arm64
  port seemed to run into somewhat often last time I looked into this issue.

windows-arm64, 1.17
-------------------

$ go run . -target=windows-arm64 -version=go1.17beta710 -rev=ec5170397c724a8ae440b2bc529f857c86f0e6b1 -watch

run 1 = "ALL TESTS PASSED"
run 2 = runtime.TestGcLastTime failed: "gc_test.go:126: bad last GC time: got 1629865473098425100, want [1629865473098425100, 1629865473001007600]" (issue #44820)
run 3 = "windows-arm64: Error: Buildlet https://farmer.golang.org:443 failed heartbeat after 10.005360314s; marking dead; err=timeout waiting for headers"
run 4 = "ALL TESTS PASSED"
run 5 = "ALL TESTS PASSED"

windows-arm64, 1.18 (tested at commit de1c934b9709728b15cc821a055155ee13e1d0ab)
---------------------

$ go run . -target=windows-arm64 -version=go1.18beta720 -rev=de1c934b9709728b15cc821a055155ee13e1d0ab -watch

run 1 = "ALL TESTS PASSED"
run 2 = "ALL TESTS PASSED"
run 3 = "ALL TESTS PASSED"

As long as that failure doesn't happen too often, the 3 retries should allow to compensate for it, so we can remove the SkipTests now, which can help catch real failures. Sent CL 345129.

[gopherbot]

Change https://golang.org/cl/345129 mentions this issue: cmd/release: start running release tests for windows-arm64