roachtest: disk-full failed

[cockroach-teamcity]

roachtest.disk-full failed with artifacts on master @ d926f1e51e23d477bcf3027adc6beac11513e58a:

(disk_full.go:124).2: COMMAND_PROBLEM: exit status 1
(cluster.go:2171).Run: context canceled
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/disk-full/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-32125

[jbowens]

COMMAND_PROBLEM: exit status 1
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerDiskFull.func1.2
  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/disk_full.go:124
  | main.(*monitorImpl).Go.func1
  | 	main/pkg/cmd/roachtest/monitor.go:119
  | golang.org/x/sync/errgroup.(*Group).Go.func1
  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
  | runtime.goexit
  | 	src/runtime/asm_amd64.s:1598
Wraps: (2) secondary error attachment
  | COMMAND_PROBLEM: exit status 1
  | (1) Node 1. Command with error:
  |   | ```
  |   | systemctl status cockroach.service | grep 'Main PID' | grep -oE '\((.+)\)'
  |   | ```
  |   | stdout: <empty>
  |   | stderr:Unit cockroach.service could not be found.
  | Wraps: (2) COMMAND_PROBLEM
  | Wraps: (3) exit status 1
  | Error types: (1) *hintdetail.withDetail (2) errors.Cmd (3) *exec.ExitError
Wraps: (3) COMMAND_PROBLEM: exit status 1
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError

[jbowens]

@cockroachdb/test-eng are you aware of any changes that might result in the systemd cockroach.service unit not being found?

[renatolabs]

Erm yes, that would be #111064 -- with virtualization, we might not have just one cockroach process in the node, so the unit names changed. It's not great that tests name these services directly as it's arguably "internal" API. Let me think about how to fix this.

[blathers-crl]

cc @cockroachdb/test-eng