windows/arm: fatal error: minpc or maxpc invalid

[zx2c4]

Now that we've got the windows/arm builder running, we can start looking at the first failures. So far there are no successful tests. That could indicate an issue with the test rig, or it might indicate a latent problem with windows/arm, while it's bitrotted without a builder.

Various issues seen on the farmer dashboard:

Building Go cmd/dist using C:\Program Files (Arm)\Go
Building Go toolchain1 using C:\Program Files (Arm)\Go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
fatal error: minpc or maxpc invalid
runtime: panic before malloc heap initialized

runtime stack:
fatal error: findfunc: bad findfunctab entry idx
runtime: panic before malloc heap initialized
panic during panic

runtime stack:
fatal error: findfunc: bad findfunctab entry idx
runtime: panic before malloc heap initialized
stack trace unavailable
go tool dist: FAILED: C:\workdir\go\pkg\tool\windows_arm\go_bootstrap install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 4

Building Go cmd/dist using C:\Program Files (Arm)\Go
Building Go toolchain1 using C:\Program Files (Arm)\Go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
fatal error: minpc or maxpc invalid
runtime: panic before malloc heap initialized

runtime stack:
fatal: morestack on g0
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.

Building Go cmd/dist using C:\Program Files (Arm)\Go
Building Go toolchain1 using C:\Program Files (Arm)\Go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
fatal error: minpc or maxpc invalid
runtime: panic before malloc heap initialized

runtime stack:
cmd/go/internal/work.init()
	C:/workdir/go/src/cmd/go/internal/work/security.go:198 +0x2400 fp=0x12ffc60 sp=0x12ffc4c pc=0x6f6c6c
cmd/go/internal/modcmd.runVendor(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	C:/workdir/go/src/cmd/go/internal/modcmd/vendor.go:124 +0x904 fp=0x12fffe0 sp=0x12ffc60 pc=0x710e4c
runtime: unexpected return pc for cmd/go/internal/clean.clean called from 0x0
stack: frame={sp:0x12fffe0, fp:0x1300198} stack=[0x12efd60,0x12ffcf8)

cmd/go/internal/clean.clean(0x0)
	C:/workdir/go/src/cmd/go/internal/clean/clean.go:237 +0x14 fp=0x1300198 sp=0x12fffe0 pc=0x6fa12c
go tool dist: FAILED: C:\workdir\go\pkg\tool\windows_arm\go_bootstrap install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 2

CC @bcmills @alexbrainman @bradfitz @cagedmantis

[ALTree]

Note that you reported essentially the same crash at #39465. I suggest keeping only one open (the old one has no additional info anyway).

[zx2c4]

Note that you reported essentially the same crash at #39465. I suggest keeping only one open (the old one has no additional info anyway).

Oh thanks! I had completely forgotten about that! I was in the process of beefing up the hardware, thinking that maybe this was just a out of memory issue, but now clearly that's not it. Time for me to jump into the actual bug I guess.

[zx2c4]

Looking at the farmer dashboard, things seem to be okay on the 1.14 release branch but are broken by 1.15.

[alexbrainman]

Looking at the farmer dashboard, things seem to be okay on the 1.14 release branch but are broken by 1.15.

Use git bisect to find commit that breaks. No?

Alex

[zx2c4]

Looking at the farmer dashboard, things seem to be okay on the 1.14 release branch but are broken by 1.15.

Use git bisect to find commit that breaks. No?

Seems reasonable. I'm in the process of installing a beefier drive so I can do that performantly.

[zx2c4]

Bisection identified c364079 as the first bad commit.

commit c364079a533c172bd071af38b6f6ffd7dc68186d
Author: Cherry Zhang <[email protected]>
Date:   Wed Apr 15 22:23:41 2020 -0400

    [dev.link] cmd/link: use function address directly in pclntab generation

    If we are internal linking a static executable, in pclntab
    generation, the function addresses are known, so we can just use
    them directly instead of emitting relocations.

    For external linking or other build modes,  we are generating a
    relocatable binary so we still need to emit relocations.

    Reduce some allocations: for linking cmd/compile,

    name           old alloc/op   new alloc/op   delta
    Pclntab_GC       38.8MB ± 0%    36.4MB ± 0%   -6.19%  (p=0.008 n=5+5)

    TODO: can we also do this in DWARF generation?

    Change-Id: I43920d930ab1da97c205871027e01844a07a5e60
    Reviewed-on: https://go-review.googlesource.com/c/go/+/228478
    Run-TryBot: Cherry Zhang <[email protected]>
    TryBot-Result: Gobot Gobot <[email protected]>
    Reviewed-by: Than McIntosh <[email protected]>

 src/cmd/link/internal/ld/pcln.go   | 28 ++++++++++++++++++++--------
 src/cmd/link/internal/ld/target.go |  4 ++++
 2 files changed, 24 insertions(+), 8 deletions(-)
bisect run success

cc @cherrymui @thanm

[jeremyfaller]

The flagged CL is almost certainly causing it. I will take a look and try to fix.

Is there gomotes available?

[zx2c4]

Is there gomotes available?

I'm not super familiar with how the gomote infrastructure works. I assume I run some sort of trojan on my arm machine, and then you're able to easily run commands on it through the testing infra? If so, let me know what you want me to run, and I'll do that. Otherwise, I'm happy to just test patches out manually. Or, can't you tell gobot to use the new builder for trybot for a CL?

[jeremyfaller]

I am not familiar with the infra beyond, "it just works, most of the time." This will likely be difficult to debug without access to a machine. Perhaps @dmitshur can fill us in on what needs to happen to get access to this machine? I checked gomote create and it was listing windows-arm-zx2c4 as a valid gomote, but gomote ssh failed.

[zx2c4]

Can you just use 'gomote put' and 'gomote run' instead? In theory that should be enough, right? There's also 'gomote rdp', though I'm not sure how authentication works there.

[zx2c4]

When it crashes, I'm seeing:

datap.minpc, datap.maxpc = 0xb11000, 0xe50630
datap.ftab[0].entry, datap.ftab[nftab].entry = 0x401000, 0x740630

So it looks like it's being relocated by a fixed offset.

[cherrymui]

It is always PIE on Windows/ARM, right? But it doesn't set buildmode to PIE. The CL above excludes PIE (by testing ctxt.IsExe()). So, if setting buildmode to PIE works, we probably want to do that. Or we could hardcode an alwaysPIE predicate.

[dmitshur]

Perhaps @dmitshur can fill us in on what needs to happen to get access to this machine? I checked gomote create and it was listing windows-arm-zx2c4 as a valid gomote, but gomote ssh failed.

The "windows-arm-zx2c4" builder was configured with SSH support turned off, because the HostConfig it uses has a zero value for SSHUsername field. If that configuration is changed, it should work, or it may fail for another reason. Also, gomote ssh uses a different authentication scheme, documented at https://golang.org/wiki/Gomote#gomote-ssh.

The gomote put and gomote run commands that @zx2c4 suggested are in general less finicky and easier to rely on.

[gopherbot]

Change https://golang.org/cl/273566 mentions this issue: link: windows/arm is all pie, so mark it as such

[zx2c4]

I think I've got it fixed!

[zx2c4]

@gopherbot please backport this to 1.15

[gopherbot]

Backport issue(s) opened: #42849 (for 1.15).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[gopherbot]

Change https://golang.org/cl/273567 mentions this issue: [release-branch.go1.15] cmd/link: windows/arm is all pie, so mark it as such