Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: schemachange/mixed-versions-compat failed [unexpected EOF during cockroach debug declarative-corpus-validate] #134934

Open
cockroach-teamcity opened this issue Nov 12, 2024 · 11 comments
Open

roachtest: schemachange/mixed-versions-compat failed [unexpected EOF during cockroach debug declarative-corpus-validate] #134934

cockroach-teamcity opened this issue Nov 12, 2024 · 11 comments
Labels
branch-release-24.3.0-rc C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Nov 12, 2024 •
edited by cockroach-jira-scripts
Loading

roachtest.schemachange/mixed-versions-compat failed with artifacts on release-24.3.0-rc @ 6d0a627cbefef712d05815b90b738e01805a618e:

created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
	pkg/util/log/log_flush.go:79 +0x26
goroutine 35 gp=0xc00011a540 m=8 mp=0xc00136c008 [syscall]:
runtime.notetsleepg(0xd38be60, 0xffffffffffffffff)
	GOROOT/src/runtime/lock_futex.go:246 +0x29 fp=0xc0009b17a0 sp=0xc0009b1778 pc=0x46ee09
os/signal.signal_recv()
	GOROOT/src/runtime/sigqueue.go:152 +0x29 fp=0xc0009b17c0 sp=0xc0009b17a0 pc=0x4d4e09
os/signal.loop()
	GOROOT/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0009b17e0 sp=0xc0009b17c0 pc=0xd90dd3
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009b17e8 sp=0xc0009b17e0 pc=0x4d8d01
created by os/signal.Notify.func1.1 in goroutine 9
	GOROOT/src/os/signal/signal.go:151 +0x1f
goroutine 10 gp=0xc0010f8e00 m=nil [chan receive]:
runtime.gopark(0xc0023a4120?, 0x0?, 0x4?, 0x0?, 0xc0000dfed0?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0000dfe40 sp=0xc0000dfe20 pc=0x4a016e
runtime.chanrecv(0xc00109a000, 0xc0000dffb8, 0x1)
	GOROOT/src/runtime/chan.go:583 +0x3bf fp=0xc0000dfeb8 sp=0xc0000dfe40 pc=0x468ddf
runtime.chanrecv1(0x8614a00?, 0xc0000fc370?)
	GOROOT/src/runtime/chan.go:442 +0x12 fp=0xc0000dfee0 sp=0xc0000dfeb8 pc=0x4689f2
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
	pkg/util/goschedstats/runnable.go:190 +0xe5 fp=0xc0000dffe0 sp=0xc0000dfee0 pc=0x262bfe5
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000dffe8 sp=0xc0000dffe0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
	pkg/util/goschedstats/runnable.go:182 +0x1a
goroutine 11 gp=0xc0022db180 m=nil [runnable]:
runtime.gopark(0xc0009b2fb0?, 0x2?, 0x0?, 0x0?, 0xc0009b2f8c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009b2e38 sp=0xc0009b2e18 pc=0x4a016e
runtime.selectgo(0xc0009b2fb0, 0xc0009b2f88, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0009b2f58 sp=0xc0009b2e38 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func1()
	cli.go:333 +0x91 fp=0xc0009b2fe0 sp=0xc0009b2f58 pc=0x58501b1
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009b2fe8 sp=0xc0009b2fe0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:331 +0xe5
goroutine 12 gp=0xc0022db340 m=nil [runnable]:
runtime.gopark(0xc0009b37a8?, 0x2?, 0x0?, 0x0?, 0xc0009b371c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009b35a8 sp=0xc0009b3588 pc=0x4a016e
runtime.selectgo(0xc0009b37a8, 0xc0009b3718, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0009b36c8 sp=0xc0009b35a8 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
	cli.go:355 +0x9f fp=0xc0009b37e0 sp=0xc0009b36c8 pc=0x584fc3f
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009b37e8 sp=0xc0009b37e0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:353 +0x1a5
bash: line 8: 39888 Aborted                 (core dumped) bash -c "./cockroach debug declarative-corpus-validate /tmp/corpus2039357818/corpus"
test artifacts and logs in: /artifacts/schemachange/mixed-versions-compat/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-44300
@cockroach-teamcity cockroach-teamcity added branch-release-24.3.0-rc C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Nov 12, 2024
This was referenced Nov 12, 2024
Closed
Closed
@annrpom
Copy link
Contributor

annrpom commented Nov 12, 2024 

from test.log:
bash: line 8: 39888 Aborted                 (core dumped) bash -c "./cockroach debug declarative-corpus-validate /tmp/corpus2039357818/corpus"

also from cockroach_debug_decl.log:
COMMAND_PROBLEM: exit status 134
(1) Node 1. Command with error:
  | ```
  | ./cockroach debug declarative-corpus-validate /tmp/corpus2039357818/corpus
  | ```
  | <truncated> ... 501b1
  | runtime.goexit({})
  | 	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009b2fe8 sp=0xc0009b2fe0 pc=0x4d8d01
  | created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
  | 	cli.go:331 +0xe5
  |
  | goroutine 12 gp=0xc0022db340 m=nil [runnable]:
  | runtime.gopark(0xc0009b37a8?, 0x2?, 0x0?, 0x0?, 0xc0009b371c?)
  | 	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009b35a8 sp=0xc0009b3588 pc=0x4a016e
  | runtime.selectgo(0xc0009b37a8, 0xc0009b3718, 0x0?, 0x0, 0x0?, 0x1)
  | 	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0009b36c8 sp=0xc0009b35a8 pc=0x4b20c5
  | github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
  | 	cli.go:355 +0x9f fp=0xc0009b37e0 sp=0xc0009b36c8 pc=0x584fc3f
  | runtime.goexit({})
  | 	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009b37e8 sp=0xc0009b37e0 pc=0x4d8d01
  | created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
  | 	cli.go:353 +0x1a5
  | bash: line 8: 39888 Aborted                 (core dumped) bash -c "./cockroach debug declarative-corpus-validate /tmp/corpus2039357818/corpus"
Wraps: (2) COMMAND_PROBLEM
Wraps: (3) exit status 134
Error types: (1) *hintdetail.withDetail (2) errors.Cmd (3) *exec.ExitError

Dupe of #132523

@annrpom annrpom removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Nov 12, 2024
@annrpom annrpom changed the title roachtest: schemachange/mixed-versions-compat failed roachtest: schemachange/mixed-versions-compat failed [missing corpus file] Nov 12, 2024
@exalate-issue-sync exalate-issue-sync bot added the P-2 Issues/test failures with a fix SLA of 3 months label Nov 12, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions-compat failed with artifacts on release-24.3.0-rc @ 49d902f212d10c8482298d777061ba153d5c16c9:

created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
	pkg/util/log/log_flush.go:79 +0x26
goroutine 19 gp=0xc001164700 m=nil [select, locked to thread]:
runtime.gopark(0xc0009b77a8?, 0x2?, 0x60?, 0x76?, 0xc0009b7794?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009b7638 sp=0xc0009b7618 pc=0x4a016e
runtime.selectgo(0xc0009b77a8, 0xc0009b7790, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0009b7758 sp=0xc0009b7638 pc=0x4b20c5
runtime.ensureSigM.func1()
	GOROOT/src/runtime/signal_unix.go:1034 +0x19f fp=0xc0009b77e0 sp=0xc0009b7758 pc=0x4cf05f
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009b77e8 sp=0xc0009b77e0 pc=0x4d8d01
created by runtime.ensureSigM in goroutine 12
	GOROOT/src/runtime/signal_unix.go:1017 +0xc8
goroutine 13 gp=0xc0014af880 m=nil [chan receive]:
runtime.gopark(0xc000e00180?, 0x0?, 0x4?, 0x0?, 0xc0000deed0?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0000dee40 sp=0xc0000dee20 pc=0x4a016e
runtime.chanrecv(0xc000e00120, 0xc0000defb8, 0x1)
	GOROOT/src/runtime/chan.go:583 +0x3bf fp=0xc0000deeb8 sp=0xc0000dee40 pc=0x468ddf
runtime.chanrecv1(0x8618a80?, 0xc001196000?)
	GOROOT/src/runtime/chan.go:442 +0x12 fp=0xc0000deee0 sp=0xc0000deeb8 pc=0x4689f2
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
	pkg/util/goschedstats/runnable.go:190 +0xe5 fp=0xc0000defe0 sp=0xc0000deee0 pc=0x262c4a5
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000defe8 sp=0xc0000defe0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
	pkg/util/goschedstats/runnable.go:182 +0x1a
goroutine 16 gp=0xc0023e7880 m=nil [runnable]:
runtime.gopark(0xc0009b67a8?, 0x2?, 0x0?, 0x0?, 0xc0009b671c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009b65a8 sp=0xc0009b6588 pc=0x4a016e
runtime.selectgo(0xc0009b67a8, 0xc0009b6718, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0009b66c8 sp=0xc0009b65a8 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
	cli.go:355 +0x9f fp=0xc0009b67e0 sp=0xc0009b66c8 pc=0x5851c9f
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009b67e8 sp=0xc0009b67e0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:353 +0x1a5
goroutine 38 gp=0xc000148000 m=nil [runnable]:
runtime.gopark(0xc0009b1fb0?, 0x2?, 0x0?, 0x0?, 0xc0009b1f8c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009b1e38 sp=0xc0009b1e18 pc=0x4a016e
runtime.selectgo(0xc0009b1fb0, 0xc0009b1f88, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0009b1f58 sp=0xc0009b1e38 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func1()
	cli.go:333 +0x91 fp=0xc0009b1fe0 sp=0xc0009b1f58 pc=0x5852211
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009b1fe8 sp=0xc0009b1fe0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:331 +0xe5
bash: line 8: 33696 Aborted                 (core dumped) bash -c "./cockroach debug declarative-corpus-validate /tmp/corpus513357535/corpus"
test artifacts and logs in: /artifacts/schemachange/mixed-versions-compat/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions-compat failed with artifacts on release-24.3.0-rc @ 49d902f212d10c8482298d777061ba153d5c16c9:

created by runtime.ensureSigM in goroutine 20
	GOROOT/src/runtime/signal_unix.go:1017 +0xc8
goroutine 12 gp=0xc000178a80 m=8 mp=0xc0013c2008 [syscall]:
runtime.notetsleepg(0xd391ea0, 0xffffffffffffffff)
	GOROOT/src/runtime/lock_futex.go:246 +0x29 fp=0xc0000df7a0 sp=0xc0000df778 pc=0x46ee09
os/signal.signal_recv()
	GOROOT/src/runtime/sigqueue.go:152 +0x29 fp=0xc0000df7c0 sp=0xc0000df7a0 pc=0x4d4e09
os/signal.loop()
	GOROOT/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0000df7e0 sp=0xc0000df7c0 pc=0xd90dd3
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000df7e8 sp=0xc0000df7e0 pc=0x4d8d01
created by os/signal.Notify.func1.1 in goroutine 20
	GOROOT/src/os/signal/signal.go:151 +0x1f
goroutine 21 gp=0xc0010ffdc0 m=nil [chan receive]:
runtime.gopark(0xc0013821e0?, 0x0?, 0x4?, 0x0?, 0xc001305ed0?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc001305e40 sp=0xc001305e20 pc=0x4a016e
runtime.chanrecv(0xc001382180, 0xc001305fb8, 0x1)
	GOROOT/src/runtime/chan.go:583 +0x3bf fp=0xc001305eb8 sp=0xc001305e40 pc=0x468ddf
runtime.chanrecv1(0x8618a80?, 0xc000140190?)
	GOROOT/src/runtime/chan.go:442 +0x12 fp=0xc001305ee0 sp=0xc001305eb8 pc=0x4689f2
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
	pkg/util/goschedstats/runnable.go:190 +0xe5 fp=0xc001305fe0 sp=0xc001305ee0 pc=0x262c4a5
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc001305fe8 sp=0xc001305fe0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
	pkg/util/goschedstats/runnable.go:182 +0x1a
goroutine 25 gp=0xc0023b3a40 m=nil [runnable]:
runtime.gopark(0xc001302fa8?, 0x2?, 0x0?, 0x0?, 0xc001302f1c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc001302da8 sp=0xc001302d88 pc=0x4a016e
runtime.selectgo(0xc001302fa8, 0xc001302f18, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc001302ec8 sp=0xc001302da8 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
	cli.go:355 +0x9f fp=0xc001302fe0 sp=0xc001302ec8 pc=0x5851c9f
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc001302fe8 sp=0xc001302fe0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:353 +0x1a5
goroutine 13 gp=0xc001ef8000 m=nil [runnable]:
runtime.gopark(0xc0013087b0?, 0x2?, 0x0?, 0x0?, 0xc00130878c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc001308638 sp=0xc001308618 pc=0x4a016e
runtime.selectgo(0xc0013087b0, 0xc001308788, 0xc0023cd7e0?, 0x0, 0xc0023b7668?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc001308758 sp=0xc001308638 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func1()
	cli.go:333 +0x91 fp=0xc0013087e0 sp=0xc001308758 pc=0x5852211
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0013087e8 sp=0xc0013087e0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:331 +0xe5
bash: line 8: 251776 Aborted                 (core dumped) bash -c "./cockroach debug declarative-corpus-validate /tmp/corpus3303233088/corpus"
test artifacts and logs in: /artifacts/schemachange/mixed-versions-compat/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 20, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions-compat failed with artifacts on release-24.3.0-rc @ dd28642dbab32aeaf42d647f0ed0a08780e8f8b0:

created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
	pkg/util/log/log_flush.go:79 +0x26
goroutine 50 gp=0xc00011a8c0 m=nil [select, locked to thread]:
runtime.gopark(0xc00100d7a8?, 0x2?, 0x60?, 0xd6?, 0xc00100d794?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc00100d638 sp=0xc00100d618 pc=0x4a016e
runtime.selectgo(0xc00100d7a8, 0xc00100d790, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc00100d758 sp=0xc00100d638 pc=0x4b20c5
runtime.ensureSigM.func1()
	GOROOT/src/runtime/signal_unix.go:1034 +0x19f fp=0xc00100d7e0 sp=0xc00100d758 pc=0x4cf05f
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00100d7e8 sp=0xc00100d7e0 pc=0x4d8d01
created by runtime.ensureSigM in goroutine 36
	GOROOT/src/runtime/signal_unix.go:1017 +0xc8
goroutine 51 gp=0xc000054380 m=7 mp=0xc00111a008 [syscall]:
runtime.notetsleepg(0xd391ea0, 0xffffffffffffffff)
	GOROOT/src/runtime/lock_futex.go:246 +0x29 fp=0xc00100dfa0 sp=0xc00100df78 pc=0x46ee09
os/signal.signal_recv()
	GOROOT/src/runtime/sigqueue.go:152 +0x29 fp=0xc00100dfc0 sp=0xc00100dfa0 pc=0x4d4e09
os/signal.loop()
	GOROOT/src/os/signal/signal_unix.go:23 +0x13 fp=0xc00100dfe0 sp=0xc00100dfc0 pc=0xd90dd3
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00100dfe8 sp=0xc00100dfe0 pc=0x4d8d01
created by os/signal.Notify.func1.1 in goroutine 36
	GOROOT/src/os/signal/signal.go:151 +0x1f
goroutine 37 gp=0xc00027ea80 m=nil [chan receive]:
runtime.gopark(0xc001118240?, 0x0?, 0x4?, 0x0?, 0xc0012d5ed0?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0012d5e40 sp=0xc0012d5e20 pc=0x4a016e
runtime.chanrecv(0xc001118180, 0xc0012d5fb8, 0x1)
	GOROOT/src/runtime/chan.go:583 +0x3bf fp=0xc0012d5eb8 sp=0xc0012d5e40 pc=0x468ddf
runtime.chanrecv1(0x8618a80?, 0xc0000fc410?)
	GOROOT/src/runtime/chan.go:442 +0x12 fp=0xc0012d5ee0 sp=0xc0012d5eb8 pc=0x4689f2
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
	pkg/util/goschedstats/runnable.go:190 +0xe5 fp=0xc0012d5fe0 sp=0xc0012d5ee0 pc=0x262c4a5
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0012d5fe8 sp=0xc0012d5fe0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
	pkg/util/goschedstats/runnable.go:182 +0x1a
goroutine 38 gp=0xc0022fc8c0 m=nil [runnable]:
runtime.gopark(0xc0012d47a8?, 0x2?, 0x0?, 0x0?, 0xc0012d471c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0012d45a8 sp=0xc0012d4588 pc=0x4a016e
runtime.selectgo(0xc0012d47a8, 0xc0012d4718, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0012d46c8 sp=0xc0012d45a8 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
	cli.go:355 +0x9f fp=0xc0012d47e0 sp=0xc0012d46c8 pc=0x5851c9f
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0012d47e8 sp=0xc0012d47e0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:353 +0x1a5
bash: line 8: 40893 Aborted                 (core dumped) bash -c "./cockroach debug declarative-corpus-validate /tmp/corpus181828279/corpus"
test artifacts and logs in: /artifacts/schemachange/mixed-versions-compat/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions-compat failed with artifacts on release-24.3.0-rc @ ef2ebe96f3a26fd6d6716e2ec4a0c8c6fe0fb567:

created by runtime.ensureSigM in goroutine 24
	GOROOT/src/runtime/signal_unix.go:1017 +0xc8
goroutine 51 gp=0xc0014ce1c0 m=5 mp=0xc000118008 [syscall]:
runtime.notetsleepg(0xcf878e0, 0xffffffffffffffff)
	GOROOT/src/runtime/lock_futex.go:246 +0x29 fp=0xc0009b0fa0 sp=0xc0009b0f78 pc=0x46ee09
os/signal.signal_recv()
	GOROOT/src/runtime/sigqueue.go:152 +0x29 fp=0xc0009b0fc0 sp=0xc0009b0fa0 pc=0x4d4e09
os/signal.loop()
	GOROOT/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0009b0fe0 sp=0xc0009b0fc0 pc=0xd91ff3
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009b0fe8 sp=0xc0009b0fe0 pc=0x4d8d01
created by os/signal.Notify.func1.1 in goroutine 24
	GOROOT/src/os/signal/signal.go:151 +0x1f
goroutine 25 gp=0xc001288700 m=nil [chan receive]:
runtime.gopark(0x1?, 0x4?, 0xd0?, 0xce?, 0x2636eff?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc000fdce40 sp=0xc000fdce20 pc=0x4a016e
runtime.chanrecv(0xc00144a180, 0xc000fdcfb8, 0x1)
	GOROOT/src/runtime/chan.go:583 +0x3bf fp=0xc000fdceb8 sp=0xc000fdce40 pc=0x468ddf
runtime.chanrecv1(0x8632820?, 0xc000252190?)
	GOROOT/src/runtime/chan.go:442 +0x12 fp=0xc000fdcee0 sp=0xc000fdceb8 pc=0x4689f2
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
	pkg/util/goschedstats/runnable.go:190 +0xe5 fp=0xc000fdcfe0 sp=0xc000fdcee0 pc=0x26371a5
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000fdcfe8 sp=0xc000fdcfe0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
	pkg/util/goschedstats/runnable.go:182 +0x1a
goroutine 26 gp=0xc001fd6000 m=nil [runnable]:
runtime.gopark(0xc0009b47b0?, 0x2?, 0x0?, 0x0?, 0xc0009b478c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009b4638 sp=0xc0009b4618 pc=0x4a016e
runtime.selectgo(0xc0009b47b0, 0xc0009b4788, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0009b4758 sp=0xc0009b4638 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func1()
	cli.go:333 +0x91 fp=0xc0009b47e0 sp=0xc0009b4758 pc=0x587fc11
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009b47e8 sp=0xc0009b47e0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:331 +0xe5
goroutine 27 gp=0xc001fd61c0 m=nil [runnable]:
runtime.gopark(0xc0009b4fa8?, 0x2?, 0x0?, 0x0?, 0xc0009b4f1c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009b4da8 sp=0xc0009b4d88 pc=0x4a016e
runtime.selectgo(0xc0009b4fa8, 0xc0009b4f18, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0009b4ec8 sp=0xc0009b4da8 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
	cli.go:355 +0x9f fp=0xc0009b4fe0 sp=0xc0009b4ec8 pc=0x587f69f
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009b4fe8 sp=0xc0009b4fe0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:353 +0x1a5
bash: line 8: 105969 Aborted                 (core dumped) bash -c "./cockroach debug declarative-corpus-validate /tmp/corpus3652212671/corpus"
test artifacts and logs in: /artifacts/schemachange/mixed-versions-compat/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@rafiss
Copy link
Collaborator

rafiss commented Nov 21, 2024 

run_075539.754576284_n1_cockroach-debug-decl: 2024/11/21 07:55:39 cluster.go:2545: > ./cockroach debug declarative-corpus-validate /tmp/corpus3652212671/corpus
teamcity-17868557-1732171428-11-n1cpu4:[1]: ./cockroach debug declarati...
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1  a panic has occurred!
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +unexpected EOF
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +(1) attached stack trace
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  -- stack trace:
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.init.MaybeDecorateError.func68
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	pkg/cli/clierrorplus/decorate_error.go:67
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/spf13/cobra.(*Command).execute
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	external/com_github_spf13_cobra/command.go:916
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/spf13/cobra.(*Command).ExecuteC
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	external/com_github_spf13_cobra/command.go:1044
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/spf13/cobra.(*Command).Execute
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	external/com_github_spf13_cobra/command.go:968
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.Run
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	cli.go:301
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.doMain
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	cli.go:139
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.Main
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	cli.go:64
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | main.main
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	pkg/cmd/cockroach-short/main.go:21
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | runtime.main
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	GOROOT/src/runtime/proc.go:271
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | runtime.goexit
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	src/runtime/asm_amd64.s:1695
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +Wraps: (2) unexpected EOF
E241121 07:55:40.345441 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +Error types: (1) *withstack.withStack (2) *errors.errorString
panic: unexpected EOF [recovered]
	panic: unexpected EOF

goroutine 1 gp=0xc0000061c0 m=9 mp=0xc001380808 [running]:
panic({0x6230ce0?, 0xc710f20?})
	GOROOT/src/runtime/panic.go:779 +0x158 fp=0xc00493d548 sp=0xc00493d498 pc=0x49c6f8
github.com/cockroachdb/cockroach/pkg/util/log/logcrash.RecoverAndReportPanic({0x867e300, 0xcf85280}, 0xc0027b4000)
	pkg/util/log/logcrash/crash_reporting.go:135 +0x6e fp=0xc00493d598 sp=0xc00493d548 pc=0xf0d56e
github.com/cockroachdb/cockroach/pkg/cli.doMain.deferwrap1()
	cli.go:137 +0x28 fp=0xc00493d5c0 sp=0xc00493d598 pc=0x587eb28
panic({0x6230ce0?, 0xc710f20?})
	GOROOT/src/runtime/panic.go:770 +0x132 fp=0xc00493d670 sp=0xc00493d5c0 pc=0x49c6d2
github.com/cockroachdb/cockroach/pkg/cli.init.func17(0xc82d8e0, {0xc0028540f0?, 0x5127260?, 0xc00280cf00?})
	declarative_corpus.go:38 +0x54c fp=0xc00493db18 sp=0xc00493d670 pc=0x586fb4c
github.com/cockroachdb/cockroach/pkg/cli.init.MaybeDecorateError.func68(0xc82d8e0?, {0xc0028540f0?, 0x1?, 0x1?})
	pkg/cli/clierrorplus/decorate_error.go:67 +0x34 fp=0xc00493dc58 sp=0xc00493db18 pc=0x585c3d4
github.com/spf13/cobra.(*Command).execute(0xc82d8e0, {0xc0028540c0, 0x1, 0x1})
	external/com_github_spf13_cobra/command.go:916 +0x882 fp=0xc00493dd78 sp=0xc00493dc58 pc=0x511d782
github.com/spf13/cobra.(*Command).ExecuteC(0xc827a00)
	external/com_github_spf13_cobra/command.go:1044 +0x3a5 fp=0xc00493de30 sp=0xc00493dd78 pc=0x511dfc5
github.com/spf13/cobra.(*Command).Execute(...)
	external/com_github_spf13_cobra/command.go:968
github.com/cockroachdb/cockroach/pkg/cli.Run(...)
	cli.go:301
github.com/cockroachdb/cockroach/pkg/cli.doMain(0xc82d8e0, {0xc000dd5f2a, 0x21})
	cli.go:139 +0x1e6 fp=0xc00493deb0 sp=0xc00493de30 pc=0x587ea46
github.com/cockroachdb/cockroach/pkg/cli.Main()
	cli.go:64 +0x11f fp=0xc00493df40 sp=0xc00493deb0 pc=0x587e6ff
main.main()
	pkg/cmd/cockroach-short/main.go:21 +0xf fp=0xc00493df50 sp=0xc00493df40 pc=0x5a53e0f
runtime.main()
	GOROOT/src/runtime/proc.go:271 +0x29d fp=0xc00493dfe0 sp=0xc00493df50 pc=0x49fd1d
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00493dfe8 sp=0xc00493dfe0 pc=0x4d8d01

@rafiss rafiss changed the title roachtest: schemachange/mixed-versions-compat failed [missing corpus file] roachtest: schemachange/mixed-versions-compat failed [unexpected EOF during cockroach debug declarative-corpus-validate] Nov 21, 2024
@Dedej-Bergin Dedej-Bergin mentioned this issue Nov 21, 2024
Merged
@Dedej-Bergin
Copy link
Contributor

This is fixed by #135933 in master and 24.3, since this is low impact we are not fixing this in 24.3-rc and will close this once this branch gets closed.

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 22, 2024
Closed
@rafiss rafiss added P-3 Issues/test failures with no fix SLA and removed P-2 Issues/test failures with a fix SLA of 3 months labels Nov 23, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions-compat failed with artifacts on release-24.3.0-rc @ ef2ebe96f3a26fd6d6716e2ec4a0c8c6fe0fb567:

created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
	pkg/util/log/log_flush.go:79 +0x26
goroutine 37 gp=0xc00144e380 m=nil [select, locked to thread]:
runtime.gopark(0xc0000e57a8?, 0x2?, 0x60?, 0x56?, 0xc0000e5794?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0000e5638 sp=0xc0000e5618 pc=0x4a016e
runtime.selectgo(0xc0000e57a8, 0xc0000e5790, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0000e5758 sp=0xc0000e5638 pc=0x4b20c5
runtime.ensureSigM.func1()
	GOROOT/src/runtime/signal_unix.go:1034 +0x19f fp=0xc0000e57e0 sp=0xc0000e5758 pc=0x4cf05f
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000e57e8 sp=0xc0000e57e0 pc=0x4d8d01
created by runtime.ensureSigM in goroutine 22
	GOROOT/src/runtime/signal_unix.go:1017 +0xc8
goroutine 23 gp=0xc0010b41c0 m=nil [chan receive]:
runtime.gopark(0xc0000fa000?, 0x0?, 0x4?, 0x0?, 0xc00012bed0?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc00012be40 sp=0xc00012be20 pc=0x4a016e
runtime.chanrecv(0xc001010000, 0xc00012bfb8, 0x1)
	GOROOT/src/runtime/chan.go:583 +0x3bf fp=0xc00012beb8 sp=0xc00012be40 pc=0x468ddf
runtime.chanrecv1(0x8618840?, 0xc001014000?)
	GOROOT/src/runtime/chan.go:442 +0x12 fp=0xc00012bee0 sp=0xc00012beb8 pc=0x4689f2
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
	pkg/util/goschedstats/runnable.go:190 +0xe5 fp=0xc00012bfe0 sp=0xc00012bee0 pc=0x262c4a5
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00012bfe8 sp=0xc00012bfe0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
	pkg/util/goschedstats/runnable.go:182 +0x1a
goroutine 38 gp=0xc00201a700 m=nil [runnable]:
runtime.gopark(0xc00012f7b0?, 0x2?, 0x0?, 0x0?, 0xc00012f78c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc00012f638 sp=0xc00012f618 pc=0x4a016e
runtime.selectgo(0xc00012f7b0, 0xc00012f788, 0x4?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc00012f758 sp=0xc00012f638 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func1()
	cli.go:333 +0x91 fp=0xc00012f7e0 sp=0xc00012f758 pc=0x58520b1
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00012f7e8 sp=0xc00012f7e0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:331 +0xe5
goroutine 39 gp=0xc0022b7500 m=nil [runnable]:
runtime.gopark(0xc00012ffa8?, 0x2?, 0x0?, 0x0?, 0xc00012ff1c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0000f8da8 sp=0xc0000f8d88 pc=0x4a016e
runtime.selectgo(0xc0000f8fa8, 0xc00012ff18, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0000f8ec8 sp=0xc0000f8da8 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
	cli.go:355 +0x9f fp=0xc0000f8fe0 sp=0xc0000f8ec8 pc=0x5851b3f
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000f8fe8 sp=0xc0000f8fe0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:353 +0x1a5
bash: line 8: 17976 Aborted                 (core dumped) bash -c "./cockroach debug declarative-corpus-validate /tmp/corpus3816497245/corpus"
test artifacts and logs in: /artifacts/schemachange/mixed-versions-compat/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions-compat failed with artifacts on release-24.3.0-rc @ 330209cefef4383a9d67ce7d86e2d3a9794ccd4f:

created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
	pkg/util/log/log_flush.go:79 +0x26
goroutine 36 gp=0xc001288540 m=nil [select, locked to thread]:
runtime.gopark(0xc0009b17a8?, 0x2?, 0x60?, 0x16?, 0xc0009b1794?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009b1638 sp=0xc0009b1618 pc=0x4a016e
runtime.selectgo(0xc0009b17a8, 0xc0009b1790, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0009b1758 sp=0xc0009b1638 pc=0x4b20c5
runtime.ensureSigM.func1()
	GOROOT/src/runtime/signal_unix.go:1034 +0x19f fp=0xc0009b17e0 sp=0xc0009b1758 pc=0x4cf05f
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009b17e8 sp=0xc0009b17e0 pc=0x4d8d01
created by runtime.ensureSigM in goroutine 10
	GOROOT/src/runtime/signal_unix.go:1017 +0xc8
goroutine 50 gp=0xc0014c21c0 m=5 mp=0xc000118008 [syscall]:
runtime.notetsleepg(0xcf878e0, 0xffffffffffffffff)
	GOROOT/src/runtime/lock_futex.go:246 +0x29 fp=0xc000f877a0 sp=0xc000f87778 pc=0x46ee09
os/signal.signal_recv()
	GOROOT/src/runtime/sigqueue.go:152 +0x29 fp=0xc000f877c0 sp=0xc000f877a0 pc=0x4d4e09
os/signal.loop()
	GOROOT/src/os/signal/signal_unix.go:23 +0x13 fp=0xc000f877e0 sp=0xc000f877c0 pc=0xd91ff3
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000f877e8 sp=0xc000f877e0 pc=0x4d8d01
created by os/signal.Notify.func1.1 in goroutine 10
	GOROOT/src/os/signal/signal.go:151 +0x1f
goroutine 11 gp=0xc001382700 m=nil [chan receive]:
runtime.gopark(0x0?, 0x4?, 0xd0?, 0xee?, 0x2636eff?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0000dee40 sp=0xc0000dee20 pc=0x4a016e
runtime.chanrecv(0xc0012e7c20, 0xc0000defb8, 0x1)
	GOROOT/src/runtime/chan.go:583 +0x3bf fp=0xc0000deeb8 sp=0xc0000dee40 pc=0x468ddf
runtime.chanrecv1(0x8632820?, 0xc0002521e0?)
	GOROOT/src/runtime/chan.go:442 +0x12 fp=0xc0000deee0 sp=0xc0000deeb8 pc=0x4689f2
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
	pkg/util/goschedstats/runnable.go:190 +0xe5 fp=0xc0000defe0 sp=0xc0000deee0 pc=0x26371a5
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000defe8 sp=0xc0000defe0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
	pkg/util/goschedstats/runnable.go:182 +0x1a
goroutine 12 gp=0xc00200f340 m=nil [runnable]:
runtime.gopark(0xc0009b4fa8?, 0x2?, 0x0?, 0x0?, 0xc0009b4f1c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc003907da8 sp=0xc003907d88 pc=0x4a016e
runtime.selectgo(0xc003907fa8, 0xc0009b4f18, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc003907ec8 sp=0xc003907da8 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
	cli.go:355 +0x9f fp=0xc003907fe0 sp=0xc003907ec8 pc=0x587f69f
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc003907fe8 sp=0xc003907fe0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:353 +0x1a5
bash: line 8: 25318 Aborted                 (core dumped) bash -c "./cockroach debug declarative-corpus-validate /tmp/corpus2322765729/corpus"
test artifacts and logs in: /artifacts/schemachange/mixed-versions-compat/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions-compat failed with artifacts on release-24.3.0-rc @ 330209cefef4383a9d67ce7d86e2d3a9794ccd4f:

created by runtime.ensureSigM in goroutine 36
	GOROOT/src/runtime/signal_unix.go:1017 +0xc8
goroutine 12 gp=0xc0009aac40 m=3 mp=0xc0000e9008 [syscall]:
runtime.notetsleepg(0xcf878e0, 0xffffffffffffffff)
	GOROOT/src/runtime/lock_futex.go:246 +0x29 fp=0xc0000e1fa0 sp=0xc0000e1f78 pc=0x46ee09
os/signal.signal_recv()
	GOROOT/src/runtime/sigqueue.go:152 +0x29 fp=0xc0000e1fc0 sp=0xc0000e1fa0 pc=0x4d4e09
os/signal.loop()
	GOROOT/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0000e1fe0 sp=0xc0000e1fc0 pc=0xd91ff3
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000e1fe8 sp=0xc0000e1fe0 pc=0x4d8d01
created by os/signal.Notify.func1.1 in goroutine 36
	GOROOT/src/os/signal/signal.go:151 +0x1f
goroutine 37 gp=0xc0014dddc0 m=nil [chan receive]:
runtime.gopark(0x0?, 0x4?, 0xd0?, 0x6?, 0x2636eff?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0000e0640 sp=0xc0000e0620 pc=0x4a016e
runtime.chanrecv(0xc000da8d20, 0xc0000e07b8, 0x1)
	GOROOT/src/runtime/chan.go:583 +0x3bf fp=0xc0000e06b8 sp=0xc0000e0640 pc=0x468ddf
runtime.chanrecv1(0x8632820?, 0xc0000fd540?)
	GOROOT/src/runtime/chan.go:442 +0x12 fp=0xc0000e06e0 sp=0xc0000e06b8 pc=0x4689f2
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
	pkg/util/goschedstats/runnable.go:190 +0xe5 fp=0xc0000e07e0 sp=0xc0000e06e0 pc=0x26371a5
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000e07e8 sp=0xc0000e07e0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
	pkg/util/goschedstats/runnable.go:182 +0x1a
goroutine 13 gp=0xc002198540 m=nil [runnable]:
runtime.gopark(0xc00012e7b0?, 0x2?, 0x0?, 0x0?, 0xc00012e78c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc00012e638 sp=0xc00012e618 pc=0x4a016e
runtime.selectgo(0xc00012e7b0, 0xc00012e788, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc00012e758 sp=0xc00012e638 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func1()
	cli.go:333 +0x91 fp=0xc00012e7e0 sp=0xc00012e758 pc=0x587fc11
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00012e7e8 sp=0xc00012e7e0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:331 +0xe5
goroutine 14 gp=0xc002198700 m=nil [runnable]:
runtime.gopark(0xc00012efa8?, 0x2?, 0x0?, 0x0?, 0xc00012ef1c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc00012eda8 sp=0xc00012ed88 pc=0x4a016e
runtime.selectgo(0xc00012efa8, 0xc00012ef18, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc00012eec8 sp=0xc00012eda8 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
	cli.go:355 +0x9f fp=0xc00012efe0 sp=0xc00012eec8 pc=0x587f69f
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00012efe8 sp=0xc00012efe0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:353 +0x1a5
bash: line 8: 40630 Aborted                 (core dumped) bash -c "./cockroach debug declarative-corpus-validate /tmp/corpus3771052542/corpus"
test artifacts and logs in: /artifacts/schemachange/mixed-versions-compat/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions-compat failed with artifacts on release-24.3.0-rc @ 330209cefef4383a9d67ce7d86e2d3a9794ccd4f:

created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
	pkg/util/log/log_flush.go:79 +0x26
goroutine 35 gp=0xc0011528c0 m=nil [select, locked to thread]:
runtime.gopark(0xc00012afa8?, 0x2?, 0x60?, 0xae?, 0xc00012af94?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc00012ae38 sp=0xc00012ae18 pc=0x4a016e
runtime.selectgo(0xc00012afa8, 0xc00012af90, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc00012af58 sp=0xc00012ae38 pc=0x4b20c5
runtime.ensureSigM.func1()
	GOROOT/src/runtime/signal_unix.go:1034 +0x19f fp=0xc00012afe0 sp=0xc00012af58 pc=0x4cf05f
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00012afe8 sp=0xc00012afe0 pc=0x4d8d01
created by runtime.ensureSigM in goroutine 11
	GOROOT/src/runtime/signal_unix.go:1017 +0xc8
goroutine 12 gp=0xc0010be1c0 m=nil [chan receive]:
runtime.gopark(0x0?, 0x4?, 0xd0?, 0xf6?, 0x2636eff?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0000df640 sp=0xc0000df620 pc=0x4a016e
runtime.chanrecv(0xc000f80000, 0xc0000df7b8, 0x1)
	GOROOT/src/runtime/chan.go:583 +0x3bf fp=0xc0000df6b8 sp=0xc0000df640 pc=0x468ddf
runtime.chanrecv1(0x8632820?, 0xc000f84000?)
	GOROOT/src/runtime/chan.go:442 +0x12 fp=0xc0000df6e0 sp=0xc0000df6b8 pc=0x4689f2
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
	pkg/util/goschedstats/runnable.go:190 +0xe5 fp=0xc0000df7e0 sp=0xc0000df6e0 pc=0x26371a5
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000df7e8 sp=0xc0000df7e0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
	pkg/util/goschedstats/runnable.go:182 +0x1a
goroutine 20 gp=0xc002024380 m=nil [runnable]:
runtime.gopark(0xc00012efb0?, 0x2?, 0x0?, 0x0?, 0xc00012ef8c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc00012ee38 sp=0xc00012ee18 pc=0x4a016e
runtime.selectgo(0xc00012efb0, 0xc00012ef88, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc00012ef58 sp=0xc00012ee38 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func1()
	cli.go:333 +0x91 fp=0xc00012efe0 sp=0xc00012ef58 pc=0x587fc11
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00012efe8 sp=0xc00012efe0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:331 +0xe5
goroutine 21 gp=0xc002024540 m=nil [runnable]:
runtime.gopark(0xc00012f7a8?, 0x2?, 0x0?, 0x0?, 0xc00012f71c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc00012f5a8 sp=0xc00012f588 pc=0x4a016e
runtime.selectgo(0xc00012f7a8, 0xc00012f718, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc00012f6c8 sp=0xc00012f5a8 pc=0x4b20c5
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
	cli.go:355 +0x9f fp=0xc00012f7e0 sp=0xc00012f6c8 pc=0x587f69f
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00012f7e8 sp=0xc00012f7e0 pc=0x4d8d01
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:353 +0x1a5
bash: line 8: 41302 Aborted                 (core dumped) bash -c "./cockroach debug declarative-corpus-validate /tmp/corpus2501382890/corpus"
test artifacts and logs in: /artifacts/schemachange/mixed-versions-compat/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Dec 6, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-24.3.0-rc C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rafiss @cockroach-teamcity @annrpom @Dedej-Bergin