storage: addSSTable checksum failure during restore #63297

cockroach-teamcity · 2021-04-08T09:10:56Z

(roachtest).restore2TB/nodes=10 failed on release-21.1@389cbd4be0e9ce22ca7789cd61802f1f90392c97:

	cluster.go:2254,restore.go:257,cluster.go:2666,errgroup.go:57: output in run_080247.373_n1_cockroach_sql: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2859883-1617861597-21-n10cpu4:1 -- ./cockroach sql --insecure -e ": context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2332
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2252
		  | main.registerRestore.func1.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/restore.go:257
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2666
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_080247.373_n1_cockroach_sql
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2859883-1617861597-21-n10cpu4:1 -- ./cockroach sql --insecure -e "
		  | 				RESTORE csv.bank FROM
		  | 				'gs://cockroach-fixtures/workload/bank/version=1.0.0,payload-bytes=10240,ranges=0,rows=65104166,seed=1/bank'
		  | 				WITH into_db = 'restore2tb'" returned
		  | stderr:
		  |
		  | stdout:
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	cluster.go:2688,restore.go:264,test_runner.go:767: monitor failure: monitor task failed: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2676
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2684
		  | main.registerRestore.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/restore.go:264
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2732
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (4) monitor task failed
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

More

Artifacts: /restore2TB/nodes=10
Related:

roachtest: restore2TB/nodes=10 failed [stuck in kv] #61396 roachtest: restore2TB/nodes=10 failed [stuck in kv] C-test-failure GA-blocker O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

Jira issue: CRDB-6514

The text was updated successfully, but these errors were encountered:

dt · 2021-04-08T14:17:30Z

Looks like restore ground to a halt, with lots of messages about slow RPCs, some in the thousands of seconds, mostly involving n5, e.g.

[n1] 27700  slow range RPC: have been waiting 1816.15s (1 attempts) for RPC GC [‹/Table/53/1/64854841›,‹/Table/53/1/64854841/NULL›) to r36711:‹/Table/53/1/64854841{-/0}› [(n5,s5):1, (n1,s1):2, (n7,s7):3, next=4, gen=1163, sticky=1617874480.556828020,0]; resp: ‹(err: <nil>), *roachpb.GCResponse›

We see some AddSSTable and AdminSplits hanging for awhile too, which explains why the RESTORE hung.

n5's logs have lots of health alerts too around various metrics, so looks like that node got itself into a sad state.

cockroach-teamcity · 2021-04-08T20:55:52Z

(roachtest).restore2TB/nodes=10 failed on release-21.1@80709778b7c7de7b2b704972197022305bb3ca12:


	cluster.go:2688,restore.go:264,test_runner.go:767: monitor failure: unexpected node event: 3: dead
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2676
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2684
		  | main.registerRestore.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/restore.go:264
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 3: dead
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1667,context.go:140,cluster.go:1656,test_runner.go:848: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2862820-1617908229-21-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 7: 7500
		3: dead
		8: 7777
		5: 7297
		10: 6883
		2: 7648
		9: 7618
		4: 7543
		6: 7336
		1: 8420
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1852
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

More

Artifacts: /restore2TB/nodes=10
Related:

roachtest: restore2TB/nodes=10 failed [stuck in kv] #61396 roachtest: restore2TB/nodes=10 failed [stuck in kv] C-test-failure GA-blocker O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

tbg · 2021-04-12T14:13:21Z

On the first failure: yeah what David said, here's from health.log


health: 09:07:56 restore.go:126: n1/s1    2.00     metrics  ranges.underreplicated
n1/s1    1196.00  metrics  queue.raftsnapshot.pending
n2/s2    2.00     metrics  ranges.underreplicated
n2/s2    3.00     metrics  requests.slow.latch
n3/s3    6.00     metrics  ranges.underreplicated
n3/s3    2.00     metrics  requests.slow.latch
n4/s4    834.00   metrics  queue.raftsnapshot.pending
n5/s5    237.00   metrics  queue.raftsnapshot.pending
n5/s5    4.00     metrics  requests.slow.latch
n5/s5    1.00     metrics  ranges.underreplicated
n6/s6    609.00   metrics  queue.raftsnapshot.pending
n7/s7    1.00     metrics  ranges.underreplicated
n7/s7    353.00   metrics  queue.raftsnapshot.pending
n8/s8    1.00     metrics  ranges.underreplicated
n8/s8    204.00   metrics  queue.raftsnapshot.pending
n9/s9    1.00     metrics  ranges.underreplicated
n9/s9    389.00   metrics  queue.raftsnapshot.pending
n10/s10  2.00     metrics  requests.slow.latch
n10/s10  8.00     metrics  ranges.underreplicated
n10/s10  191.00   metrics  queue.raftsnapshot.pending

This does look a lot like #61396, which @aliher1911 is investigating.

tbg · 2021-04-12T14:16:44Z

The second failure:

F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868  checksum for AddSSTable at index term 6, index 21 does not match; at proposal time 60e930d (101618445), now 594f0f5a (1498353498)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !goroutine 171 [running]:
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x85f7a01, 0xf3a9d1, 0x469101, 0xffffffff0067c878)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0xb9
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0xc0015dc600, 0xc00101a600, 0x24, 0x3, 0x0, 0x0, 0x0, 0x1673fd64250b92b1, 0x400000000, 0x0, ...)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:279 +0xc32
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(0x59bf5c0, 0xc00bba5ce0, 0x1, 0x4, 0x4cf11a0, 0x68, 0xc0074c07f0, 0x6, 0x6)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/channels.go:58 +0x198
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_channels_generated.go:804
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.addSSTablePreApply(0x59bf5c0, 0xc00bba5ce0, 0xc000ed4000, 0x5b257a0, 0xc001339930, 0x5a147e0, 0xc0143853c0, 0x6, 0x15, 0xc011480000, ...)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:548 +0x10cd
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).runPreApplyTriggersAfterStagingWriteBatch(0xc00b4800e0, 0x59bf5c0, 0xc00bba5ce0, 0xc000f21008, 0x0, 0x0)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_application_state_machine.go:616 +0xcfc
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage(0xc00b4800e0, 0x59bfe80, 0xc000f21008, 0xc0074c10e8, 0x1e412eb, 0xc00b4802f8, 0xc00b480328)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_application_state_machine.go:507 +0x3c5
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter(0x5a06a40, 0xc00b4802f8, 0xc0074c1218, 0x0, 0x0, 0x0, 0x0)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:175 +0x131
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc0074c1798, 0x59bf5c0, 0xc00bba5ce0, 0x5a06a40, 0xc00b4802c8, 0x0, 0x0)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:280 +0x165
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc0074c1798, 0x59bf5c0, 0xc00bba5ce0, 0x2, 0x4c145aa)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:247 +0xb9
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(0xc00b480000, 0x59bf5c0, 0xc00bba5ce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:796 +0x108d
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(0xc00b480000, 0x59bf5c0, 0xc00bba5ce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:459 +0x113
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc000eb0700, 0x59bf5c0, 0xc0011645d0, 0x3429)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:523 +0x134
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0003da140, 0x59bf5c0, 0xc0011645d0)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:284 +0x2c2
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000de3180, 0x59bf5c0, 0xc0011645d0, 0xc000dff700, 0xc0005fc8d0)
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
F210408 20:54:57.716212 171 kv/kvserver/replica_proposal.go:548 â‹® [n3,s3,r13353/3:â€¹/Table/53/1/215{77359â€¦-80631}â€º] 8868 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

cc @dt

jbowens · 2021-04-13T19:50:03Z

Looks similar to #52720, #56186, #55106 and #41140. Some of the previous issues were assumed to be genuine bit flips. Maybe there's actually a longstanding race condition lurking somewhere.

tbg · 2021-04-14T08:08:43Z

Repro suggestions:

make a repro branch off the SHA that had the failure
adjust the test timeout for restore2TB to ~12h (since that test varies wildly in duration), this will be somewhere in ./pkg/cmd/roachtest
pick up log.CrashWithCore from kv: repro branch for closedts assertion #63533 and call it at replica_proposal.go:548
consider also doing the pebble archiver thing from that branch, though not sure if we'll run out of disk?
I would also make some change in addSSTablePreApply to create an extra hard link, making sure that all sideloaded SSTs remain on disk forever
add go.crdb.dev/roachstress to the branch, but with the --debug flag added and also an export GCE_PROJECT=andrei-jepsen ahead of the roachtest invocation. Btw I used a CPU quota of 1600 without issues on that project recently, so I think you can run with --parallelism 100 --cpu-quota 1600 and that should give you close to 40 concurrent runs.
push the branch (and never force push that branch)
on a permanently running workstation (and on the branch), caffeinate ./roachstress.sh

Note that you shouldn't use roachprod while repro'ing, as it mixes up the cluster lists and then the repro falls over. You have to only use the andrei-jepsen project. Mixing projects = confused roachprod. So just make sure to have export GCE_PROJECT=andrei-jepsen on all of the open terminals.

Watch the failures roll in (hopefully...)

When the repro run is aborted or fails for whatever reason, you need to clean up VMs manually, I do this:

export GCE_PROJECT=andrei-jepsen
roachprod list --mine | grep -Eo "^${USER}[^:]+"
# Grep out the clusters you want to keep (like the ones that repro'ed!)
 | grep -v cluster_name_repro1 | grep -v cluster_name_repro2 > vms.txt
for v in $(cat vms.txt); do roachprod destroy $v & done

I'm happy to review the branch before you spin up a repro attempt.

tbg · 2021-04-14T20:55:33Z

^- GOAWAY from upstream server

tbg · 2021-04-20T13:33:21Z

Renamed this issue so that the issue poster won't reuse it for future failures of this test, and moved out of KV backlog.

mwang1026 · 2022-03-02T20:27:55Z

Closing for now until we can get another repro

cockroach-teamcity added branch-release-21.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 8, 2021

adityamaru mentioned this issue Apr 8, 2021

roachtest: import/tpch/nodes=8 failed #63305

Closed

lunevalex assigned jbowens Apr 13, 2021

tbg added GA-blocker and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 14, 2021

This comment has been minimized.

Sign in to view

tbg changed the title ~~roachtest: restore2TB/nodes=10 failed~~ roachtest: restore2TB/nodes=10 failed [addsstable checksum failure] Apr 19, 2021

jbowens mentioned this issue Apr 19, 2021

storage: write test for ENOSPC #63847

Open

tbg changed the title ~~roachtest: restore2TB/nodes=10 failed [addsstable checksum failure]~~ storage: addSSTable checksum failure during restore Apr 20, 2021

dt removed the GA-blocker label May 10, 2021

jlinder added the T-storage Storage Team label Jun 16, 2021

msbutler mentioned this issue Nov 10, 2021

roachtest: backup/2TB/n10cpu4 failed #72558

Closed

mwang1026 closed this as completed Mar 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

storage: addSSTable checksum failure during restore #63297

storage: addSSTable checksum failure during restore #63297

cockroach-teamcity commented Apr 8, 2021 •

edited by cockroach-jira-scripts

Loading

dt commented Apr 8, 2021

cockroach-teamcity commented Apr 8, 2021

tbg commented Apr 12, 2021

tbg commented Apr 12, 2021

jbowens commented Apr 13, 2021

tbg commented Apr 14, 2021 •

edited

Loading

This comment has been minimized.

tbg commented Apr 14, 2021

tbg commented Apr 20, 2021

mwang1026 commented Mar 2, 2022

storage: addSSTable checksum failure during restore #63297

storage: addSSTable checksum failure during restore #63297

Comments

cockroach-teamcity commented Apr 8, 2021 • edited by cockroach-jira-scripts Loading

dt commented Apr 8, 2021

cockroach-teamcity commented Apr 8, 2021

tbg commented Apr 12, 2021

tbg commented Apr 12, 2021

jbowens commented Apr 13, 2021

tbg commented Apr 14, 2021 • edited Loading

This comment has been minimized.

tbg commented Apr 14, 2021

tbg commented Apr 20, 2021

mwang1026 commented Mar 2, 2022

cockroach-teamcity commented Apr 8, 2021 •

edited by cockroach-jira-scripts

Loading

tbg commented Apr 14, 2021 •

edited

Loading