roachtest: tpccbench/nodes=12/cpu=16 failed

[cockroach-teamcity]

(roachtest).tpccbench/nodes=12/cpu=16 failed on master@c36f7cbfe08d48025540d0af62637a2fdd1f7d89:

		          335.0s        0           88.6           88.9     16.8    121.6    218.1    260.0 orderStatus
		          335.0s        0          919.7          890.3     67.1   1677.7   2818.6   3758.1 payment
		          335.0s        0           85.6           88.8    117.4   2281.7   3355.4   3489.7 stockLevel
		          336.0s        0           86.9           88.6    167.8   1946.2   2281.7   2550.1 delivery
		          336.0s        0         1044.0          854.2   1208.0   2147.5   2818.6   5368.7 newOrder
		          336.0s        0          108.9           89.0     22.0    121.6    159.4    184.5 orderStatus
		          336.0s        0          945.1          890.5     79.7   1744.8   2415.9   5100.3 payment
		          336.0s        0           94.9           88.8    104.9   2818.6   9126.8  15569.3 stockLevel
		        _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		          337.0s        0           86.1           88.6    121.6   1946.2   2013.3   2147.5 delivery
		          337.0s        0          809.8          854.1    973.1   2080.4   2952.8   5100.3 newOrder
		          337.0s        0           88.1           89.0     16.3    109.1    192.9    260.0 orderStatus
		          337.0s        0          899.9          890.5     56.6   1543.5   2550.1   4563.4 payment
		          337.0s        0           88.1           88.8    125.8   5100.3  13421.8  18253.6 stockLevel
		          338.0s        0           79.9           88.6    125.8   2013.3   2147.5   2550.1 delivery
		          338.0s        0          951.5          854.4   1208.0   2147.5   2952.8   6174.0 newOrder
		          338.0s        0           81.9           88.9     14.7    113.2    201.3    209.7 orderStatus
		          338.0s        0          850.7          890.4     67.1   1744.8   2818.6   5100.3 payment
		          338.0s        0           90.9           88.8    218.1   3221.2  15032.4  16643.0 stockLevel
		          339.0s        0           89.0           88.6    184.5   1946.2   2281.7   2684.4 delivery
		          339.0s        0          936.0          854.6   1140.9   2080.4   3087.0   4831.8 newOrder
		          339.0s        0           84.0           88.9     15.7     88.1    134.2    142.6 orderStatus
		          339.0s        0          940.0          890.5    100.7   1677.7   2684.4   4160.7 payment
		          339.0s        0           85.0           88.8    151.0   3355.4   9126.8  11274.3 stockLevel
		          340.0s        0           82.5           88.6    167.8   1610.6   2080.4   2147.5 delivery
		          340.0s        0          889.3          854.8    973.1   1946.2   2684.4   3623.9 newOrder
		          340.0s        0           90.4           88.9     18.9     96.5    134.2    218.1 orderStatus
		          340.0s        0          864.5          890.5     83.9   1342.2   2415.9   3355.4 payment
		          340.0s        0           92.4           88.8    113.2   2550.1  12348.0  12348.0 stockLevel
		        _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		          341.0s        0           97.6           88.6    142.6   1342.2   1879.0   1946.2 delivery
		          341.0s        0         1040.8          855.3    939.5   1677.7   2415.9   4563.4 newOrder
		          341.0s        0          100.7           89.0     18.9     92.3    130.0    142.6 orderStatus
		          341.0s        0          942.2          890.6     75.5   1342.2   2550.1   3623.9 payment
		          341.0s        0           92.6           88.8    130.0   3355.4   6442.5  18253.6 stockLevel
		          342.0s        0           88.6           88.6    130.0   1610.6   1677.7   1744.8 delivery
		          342.0s        0          833.9          855.2    738.2   1543.5   2147.5   2952.8 newOrder
		          342.0s        0           93.5           89.0     15.7     75.5     88.1     96.5 orderStatus
		          342.0s        0          853.8          890.5     62.9   1140.9   1946.2   4831.8 payment
		          342.0s        0           78.6           88.8    209.7   4026.5  17179.9  18253.6 stockLevel:
		      - context canceled
		    error running tpcc load generator
		    main.runTPCCBench.func3.1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:818
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357

More

Artifacts: /tpccbench/nodes=12/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[nvanbenschoten]

SIGSEGV: segmentation violation
PC=0x761cd3 m=0 sigcode=0

goroutine 11824325 [syscall]:
runtime.notetsleepg(0x6f867a0, 0xb7bdca, 0x0)
	/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc01544c760 sp=0xc01544c730 pc=0x70b5f4
runtime.timerproc(0x6f86780)
	/usr/local/go/src/runtime/time.go:311 +0x2f1 fp=0xc01544c7d8 sp=0xc01544c760 pc=0x74e9f1
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc01544c7e0 sp=0xc01544c7d8 pc=0x75fdb1
created by runtime.(*timersBucket).addtimerLocked
	/usr/local/go/src/runtime/time.go:169 +0x10e

We're beein seeing this a lot lately: https://github.com/cockroachdb/cockroach/issues?q=is%3Aissue+addtimerLocked+is%3Aclosed. It's being tracked in #44069 (comment).