roachtest: kv50/enc=false/nodes=4/cpu=96/batch=64 failed [marked free object in span]

[cockroach-teamcity]

(roachtest).kv50/enc=false/nodes=4/cpu=96/batch=64 failed on release-21.1@23e7cb53bf5baede071832b59bd92ea8164531a6:

		  | main.registerKV.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:186
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2732
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2646
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5652
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	cluster.go:1667,context.go:140,cluster.go:1656,test_runner.go:848: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2808064-1616565608-89-n5cpu96 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		2: dead
		4: 6124
		3: 6160
		1: 6238
		Error: UNCLASSIFIED_PROBLEM: 2: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1852
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 2: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

More

Artifacts: /kv50/enc=false/nodes=4/cpu=96/batch=64

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[tbg]

Ugh:

runtime: marked free object in span 0x7f6a56e8a668, elemsize=128 freeindex=37 (bad use of unsafe.Pointer? try -d=checkptr)

cockroach-stderr.log

[tbg]

Kicking off ten runs on release-21.1 just in case this repros easily which I don't suppose it will.

[tbg]

They all passed.

[angelapwen]

I could not repro this error after 100 runs on 21.1 (I just got one failure which was a known clock skew failure 🕐 ).

I will look to see if I can repro on 100 runs on master 👀

[nvanbenschoten]

This looks similar to golang/go#44614. That issue references a bug in the go1.14+ compiler, which was fixed by golang/go@a429926. Our builder is currently running go1.15.6, which does not contain golang/go@f75ab2d. The earliest version of go1.15 with the fix is go1.15.10. So I think we'll want to bump our go version to the latest patch release of 1.15 before releasing.

[tbg]

#62873