sentry: replica.go:3759: %s: %s | while committing batch; replica.go:3577: caused by *errors.withMessage: caused by io error while open a file for appending <redacted> no <redacted> file or directory

[tbg]

https://sentry.io/cockroach-labs/cockroachdb/issues/722607997/

replica.go:3759: %s: %s | while committing batch; replica.go:3577: caused by *errors.withMessage: caused by io error while open a file for appending no file or directory

github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue.func1

stacktrace: {u'frames': [{u'function': u'func1', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go', u'pre_context': [u'// by the stopper. The function is run in a goroutine.', u'func (s *Stopper) RunWorker(ctx context.Context, f func(context.Context)) {', u'\ts.stop.Add(1)'], u'module': u'github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker', u'filename': u'github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go', u'post_context': [u'\t\t// Remove any associated span; we need to ensure this because the', u'\t\t// worker may run longer than the caller which presumably closes', u'\t\t// any spans it has created.'], u'in_app': False, u'context_line': u'\tgo func() {', u'lineno': 192}, {u'function': u'func2', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go', u'pre_context': [u'\tfor i := 0; i < s.numWorkers; i++ {', u'\t\tstopper.RunWorker(ctx, func(ctx context.Context) {', u'\t\t\ts.worker(ctx)'], u'module': u'github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start', u'filename': u'github.com/cockroachdb/cockroach/pkg/storage/scheduler.go', u'post_context': [u'\t}', u'}', u''], u'in_app': False, u'context_line': u'\t\t})', u'lineno': 166}, {u'function': u'worker', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go', u'pre_context': [u'\t\t// that case.', u'\t\tif state&stateRaftRequest != 0 {', u'\t\t\ts.processor.processRequestQueue(ctx, id)'], u'module': u'github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler)', u'filename': u'github.com/cockroachdb/cockroach/pkg/storage/scheduler.go', u'post_context': [u'', u'\t\ts.mu.Lock()', u'\t\tstate = s.mu.state[id]'], u'in_app': False, u'context_line': u'\t\t}', u'lineno': 226}, {u'function': u'processRequestQueue', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go', u'pre_context': [u'\t\t\t}', u'\t\t\t// If the replica ID in the error does match, we know the replica', u'\t\t\t// will be removed and we can cancel any pending commands. This is'], u'module': u'github.com/cockroachdb/cockroach/pkg/storage.(*Store)', u'filename': u'github.com/cockroachdb/cockroach/pkg/storage/store.go', u'post_context': [u'\t\t\t// necessary for the replica GC to succeed.', u'\t\t\trepl.cancelPendingCommandsLocked()', u'\t\t\t// The replica will be garbage collected soon (we are sure'], u'in_app': False, u'context_line': u'\t\t\t// sometimes necessary to unblock PushTxn operations that are', u'lineno': 3787}, {u'function': u'withReplicaForRequest', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go', u'pre_context': [u'\t\t\tshallowTxn.UpdateObservedTimestamp(ba.Replica.NodeID, now)', u'\t\t\tba.Txn = &shallowTxn', u'\t\t}'], u'module': u'github.com/cockroachdb/cockroach/pkg/storage.(*Store)', u'filename': u'github.com/cockroachdb/cockroach/pkg/storage/store.go', u'post_context': [u'', u'\tif log.V(1) {', u'\t\tlog.Eventf(ctx, "executing %s", ba)'], u'in_app': False, u'context_line': u'\t}', u'lineno': 3121}, {u'function': u'func1', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go', u'pre_context': [u'\t\t\t}', u'\t\t\trepl.mu.Unlock()', u''], u'module': u'github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue', u'filename': u'github.com/cockroachdb/cockroach/pkg/storage/store.go', u'post_context': [u'\t\t\t\tlog.Errorf(ctx, "unable to add to replica GC queue: %s", err)', u'\t\t\t} else {', u'\t\t\t\tlog.Infof(ctx, "added to replica GC queue (peer suggestion)")'], u'in_app': False, u'context_line': u'\t\t\tif _, err := s.replicaGCQueue.Add(repl, replicaGCPriorityRemoved); err != nil {', u'lineno': 3800}]}
type: *log.safeError
value: replica.go:3759: %s: %s | while committing batch; replica.go:3577: caused by *errors.withMessage: caused by io error while open a file for appending no file or directory

[benesch]

Isn't this just someone rm -rf cockroach-dataing while the server is running?

[tbg]

Yes, or run of the mill data corruption causes these too (see #31187)

These issues are posted automatically. When you see one in the future, feel free to close it. I should probably move this over to a bot account at some point.

[benesch]

Hah! Yeah, probably worth moving this to the @cockroach-teamcity account so it's more obvious that this might be spurious.