roachtest: restore2TB/nodes=32 failed

[cockroach-teamcity]

(roachtest).restore2TB/nodes=32 failed on master@b93fd531b93cb010729cb73fe679cdff9388cf27:

		13: 6169
		10: 6170
		3: 5913
		12: 5817
		18: 5686
		8: 5641
		29: 5877
		11: 5902
		19: dead
		28: 6389
		17: 5721
		7: 5813
		4: 6002
		14: 5816
		31: 5545
		2: 5917
		21: 5792
		27: 5972
		22: 6256
		25: 5805
		1: 6060
		20: 5908
		24: 5185
		5: 5674
		15: 6010
		16: 5684
		32: 5657
		9: 7338
		Error: UNCLASSIFIED_PROBLEM: 19: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1850
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 19: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

More

Artifacts: /restore2TB/nodes=32
Related:

• roachtest: restore2TB/nodes=32 failed #54500 roachtest: restore2TB/nodes=32 failed C-test-failure O-roachtest O-robot T-bulkio branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[adityamaru]

Restore failed because node 19 died with kv/kvserver/replica_proposal.go:783 ⋮ [n19,s19,r9756/2:‹×›] not using applied state key in v21.1

Roping in KV triage to shed some light on what could've caused this.

[cockroach-teamcity]

(roachtest).restore2TB/nodes=32 failed on master@d7bbe0060531063b9bee29f69bc4d23d41b84e3d:

		19: 5668
		15: 5239
		25: 5411
		21: 5386
		13: 5970
		20: 5742
		10: 5376
		17: dead
		16: 5839
		5: 5414
		26: 5413
		32: 5408
		9: 6265
		6: 6455
		31: 5287
		24: 5755
		1: 6592
		28: 5266
		3: 5290
		2: 5387
		7: 6026
		14: 5978
		12: 5357
		29: 5846
		27: 6494
		23: 5981
		18: 5280
		8: 5309
		Error: UNCLASSIFIED_PROBLEM: 17: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1850
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 17: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

More

Artifacts: /restore2TB/nodes=32
Related:

• roachtest: restore2TB/nodes=32 failed #54500 roachtest: restore2TB/nodes=32 failed C-test-failure O-roachtest O-robot T-bulkio branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[aayushshah15]

Hey @irfansharif, looks like it's one of your assertions firing. Assigning you on this.

[irfansharif]

Duplicate of #58378, the assertion exposed another bug. The assertion was downgraded in #58387 while we explore the other thing.