roachtest: disk-full failed

[cockroach-teamcity]

roachtest.disk-full failed with artifacts on release-21.2 @ a5a88a4db163f0915ae65649aa0264c7a913fbfb:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)

Reproduce

See: roachtest README

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-14040

[nicktrav]

Looks like this got a backport of the fix to the recent issue, so it's probably not that: https://github.com/cockroachdb/cockroach/blob/release-21.2/pkg/cmd/roachtest/tests/disk_full.go

[nicktrav]

It looks like the monitor goroutine got stuck waiting for a query response, and this timed out after 600 minutes, failing the test:

cockroach/pkg/cmd/roachtest/tests/disk_full.go

Line 68 in 2c0a06e

err := db.QueryRow(`SELECT is_live FROM crdb_internal.gossip_nodes WHERE node_id = 1;`).Scan(&isLive)

goroutine 4050163 [IO wait, 600 minutes]:
internal/poll.runtime_pollWait(0x7ff475f65738, 0x72)
        /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00059ff00, 0xc000eba000, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00059ff00, {0xc000eba000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00059ff00, {0xc000eba000, 0x15a, 0x47})
        /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000f10820, {0xc000eba000, 0xc0045c7620, 0x0})
        /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc004d3ea20, {0xc0045c7620, 0x5, 0xc0019b7438})
        /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x5d836e0, 0xc004d3ea20}, {0xc0045c7620, 0x5, 0x200}, 0x5)
        /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:347
github.com/lib/pq.(*conn).recvMessage(0xc0045c7600, 0xc0019b75a0)
        /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:983 +0xca
github.com/lib/pq.(*conn).recv1Buf(0xc0045c7600, 0x0)
        /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:1038 +0x2e
github.com/lib/pq.(*conn).recv1(...)
        /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:1065
github.com/lib/pq.(*conn).simpleQuery(0xc0045c7600, {0x493a840, 0x41})
        /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:675 +0x231
github.com/lib/pq.(*conn).query(0xc0045c7600, {0x493a840, 0xc000126000}, {0x8988558, 0x0, 0x0})
        /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:868 +0x426
github.com/lib/pq.(*conn).QueryContext(0x8, {0x5e0d370, 0xc000126000}, {0x493a840, 0x41}, {0x8988558, 0x0, 0x8})
        /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn_go18.go:21 +0xd7
database/sql.ctxDriverQuery({0x5e0d370, 0xc000126000}, {0x7ff48b21af48, 0xc0045c7600}, {0x0, 0x0}, {0x493a840, 0xc001c7c990}, {0x8988558, 0x0, ...})
        /usr/local/go/src/database/sql/ctxutil.go:48 +0x17d
database/sql.(*DB).queryDC.func1()
        /usr/local/go/src/database/sql/sql.go:1722 +0x175
database/sql.withLock({0x5da4f18, 0xc001c7c990}, 0xc0019b7ae0)
        /usr/local/go/src/database/sql/sql.go:3396 +0x8c
database/sql.(*DB).queryDC(0xc0019b7b01, {0x5e0d370, 0xc000126000}, {0x0, 0x0}, 0xc001c7c990, 0xc001323470, {0x493a840, 0x41}, {0x0, ...})
        /usr/local/go/src/database/sql/sql.go:1717 +0x211
database/sql.(*DB).query(0x532fe5, {0x5e0d370, 0xc000126000}, {0x493a840, 0x41}, {0x0, 0x0, 0x0}, 0x58)
        /usr/local/go/src/database/sql/sql.go:1700 +0xfd
database/sql.(*DB).QueryContext(0x8, {0x5e0d370, 0xc000126000}, {0x493a840, 0x41}, {0x0, 0x0, 0x0})
        /usr/local/go/src/database/sql/sql.go:1674 +0xdf
database/sql.(*DB).QueryRowContext(...)
        /usr/local/go/src/database/sql/sql.go:1778
database/sql.(*DB).QueryRow(0xc0010023c0, {0x493a840, 0xc001cab600}, {0x0, 0x4821cdb, 0x300000002})
        /usr/local/go/src/database/sql/sql.go:1792 +0x4a
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerDiskFull.func1.2({0x5e0d338, 0xc001cab600})
        /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/disk_full.go:68 +0x205
main.(*monitorImpl).Go.func1()
        /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:106 +0xb0
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x92

I think we can chalk this one up to an infrastructure flake. Closing.

[nicktrav]

Closed by #78537.