Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: schemachange/mixed-versions-compat failed [unexpected EOF during cockroach debug declarative-corpus-validate] #135807

Closed
cockroach-teamcity opened this issue Nov 20, 2024 · 4 comments · Fixed by #135933
Closed

roachtest: schemachange/mixed-versions-compat failed [unexpected EOF during cockroach debug declarative-corpus-validate] #135807

cockroach-teamcity opened this issue Nov 20, 2024 · 4 comments · Fixed by #135933
Assignees
@Dedej-Bergin
Labels
B-runtime-assertions-enabled branch-master Failures and bugs on the master branch. branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Nov 20, 2024 •
edited by cockroach-jira-scripts
Loading

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions-compat failed with artifacts on master @ 8eeb7f2ae3b2cede564b46ca47e2353fd147c061:

created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
	pkg/util/log/log_flush.go:79 +0x26
goroutine 22 gp=0xc0017c01c0 m=8 mp=0xc001746008 [syscall]:
runtime.notetsleepg(0xd5873e0, 0xffffffffffffffff)
	GOROOT/src/runtime/lock_futex.go:246 +0x29 fp=0xc0009bcfa0 sp=0xc0009bcf78 pc=0x46f1a9
os/signal.signal_recv()
	GOROOT/src/runtime/sigqueue.go:152 +0x29 fp=0xc0009bcfc0 sp=0xc0009bcfa0 pc=0x4d51a9
os/signal.loop()
	GOROOT/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0009bcfe0 sp=0xc0009bcfc0 pc=0xd98b53
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009bcfe8 sp=0xc0009bcfe0 pc=0x4d90a1
created by os/signal.Notify.func1.1 in goroutine 21
	GOROOT/src/os/signal/signal.go:151 +0x1f
goroutine 37 gp=0xc0011d0540 m=nil [chan receive]:
runtime.gopark(0x0?, 0x4?, 0xd0?, 0x1e?, 0x26a63bf?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009c1e40 sp=0xc0009c1e20 pc=0x4a050e
runtime.chanrecv(0xc0016fe8a0, 0xc0009c1fb8, 0x1)
	GOROOT/src/runtime/chan.go:583 +0x3bf fp=0xc0009c1eb8 sp=0xc0009c1e40 pc=0x468ddf
runtime.chanrecv1(0x878b440?, 0xc0016e7090?)
	GOROOT/src/runtime/chan.go:442 +0x12 fp=0xc0009c1ee0 sp=0xc0009c1eb8 pc=0x4689f2
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
	pkg/util/goschedstats/runnable.go:190 +0xe5 fp=0xc0009c1fe0 sp=0xc0009c1ee0 pc=0x26a6665
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009c1fe8 sp=0xc0009c1fe0 pc=0x4d90a1
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
	pkg/util/goschedstats/runnable.go:182 +0x1a
goroutine 38 gp=0xc001cd2380 m=nil [runnable]:
runtime.gopark(0xc0000defb0?, 0x2?, 0x0?, 0x0?, 0xc0000def8c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0000dee38 sp=0xc0000dee18 pc=0x4a050e
runtime.selectgo(0xc0000defb0, 0xc0000def88, 0xc00241a4f0?, 0x0, 0xc0023f7a70?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0000def58 sp=0xc0000dee38 pc=0x4b2465
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func1()
	cli.go:333 +0x91 fp=0xc0000defe0 sp=0xc0000def58 pc=0x5944311
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000defe8 sp=0xc0000defe0 pc=0x4d90a1
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:331 +0xe5
goroutine 39 gp=0xc001cd2540 m=nil [runnable]:
runtime.gopark(0xc0000e17a8?, 0x2?, 0x0?, 0x0?, 0xc0000e171c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0000e15a8 sp=0xc0000e1588 pc=0x4a050e
runtime.selectgo(0xc0000e17a8, 0xc0000e1718, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0000e16c8 sp=0xc0000e15a8 pc=0x4b2465
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
	cli.go:355 +0x9f fp=0xc0000e17e0 sp=0xc0000e16c8 pc=0x5943d9f
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000e17e8 sp=0xc0000e17e0 pc=0x4d90a1
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:353 +0x1a5
bash: line 8: 82770 Aborted                 (core dumped) bash -c "./cockroach debug declarative-corpus-validate /tmp/corpus680433151/corpus"
test artifacts and logs in: /artifacts/schemachange/mixed-versions-compat/run_1

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=4
  • encrypted=false
  • runtimeAssertionsBuild=true
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-44726
@cockroach-teamcity cockroach-teamcity added B-runtime-assertions-enabled branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Nov 20, 2024
This was referenced Nov 20, 2024
Closed
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions-compat failed with artifacts on master @ eb2d2e19eb29d2747d9e267bd0612a69d066adad:

created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
	pkg/util/log/log_flush.go:79 +0x26
goroutine 8 gp=0xc001382380 m=nil [select, locked to thread]:
runtime.gopark(0xc0009bf7a8?, 0x2?, 0x60?, 0xf6?, 0xc0009bf794?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009bf638 sp=0xc0009bf618 pc=0x4a050e
runtime.selectgo(0xc0009bf7a8, 0xc0009bf790, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0009bf758 sp=0xc0009bf638 pc=0x4b2465
runtime.ensureSigM.func1()
	GOROOT/src/runtime/signal_unix.go:1034 +0x19f fp=0xc0009bf7e0 sp=0xc0009bf758 pc=0x4cf3ff
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009bf7e8 sp=0xc0009bf7e0 pc=0x4d90a1
created by runtime.ensureSigM in goroutine 52
	GOROOT/src/runtime/signal_unix.go:1017 +0xc8
goroutine 53 gp=0xc000174540 m=nil [chan receive]:
runtime.gopark(0xc001568540?, 0x0?, 0x4?, 0x0?, 0xc0000df6d0?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0000df640 sp=0xc0000df620 pc=0x4a050e
runtime.chanrecv(0xc000136240, 0xc0000df7b8, 0x1)
	GOROOT/src/runtime/chan.go:583 +0x3bf fp=0xc0000df6b8 sp=0xc0000df640 pc=0x468ddf
runtime.chanrecv1(0x8760ec0?, 0xc0013100a0?)
	GOROOT/src/runtime/chan.go:442 +0x12 fp=0xc0000df6e0 sp=0xc0000df6b8 pc=0x4689f2
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
	pkg/util/goschedstats/runnable.go:190 +0xe5 fp=0xc0000df7e0 sp=0xc0000df6e0 pc=0x269c005
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000df7e8 sp=0xc0000df7e0 pc=0x4d90a1
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
	pkg/util/goschedstats/runnable.go:182 +0x1a
goroutine 36 gp=0xc004592a80 m=nil [runnable]:
runtime.gopark(0xc0009c37b0?, 0x2?, 0x0?, 0x0?, 0xc0009c378c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009c3638 sp=0xc0009c3618 pc=0x4a050e
runtime.selectgo(0xc0009c37b0, 0xc0009c3788, 0x4?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0009c3758 sp=0xc0009c3638 pc=0x4b2465
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func1()
	cli.go:333 +0x91 fp=0xc0009c37e0 sp=0xc0009c3758 pc=0x591c871
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009c37e8 sp=0xc0009c37e0 pc=0x4d90a1
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:331 +0xe5
goroutine 37 gp=0xc004592c40 m=nil [runnable]:
runtime.gopark(0xc0009c3fa8?, 0x2?, 0x0?, 0x0?, 0xc0009c3f1c?)
	GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009c3da8 sp=0xc0009c3d88 pc=0x4a050e
runtime.selectgo(0xc0009c3fa8, 0xc0009c3f18, 0x0?, 0x0, 0x0?, 0x1)
	GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0009c3ec8 sp=0xc0009c3da8 pc=0x4b2465
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
	cli.go:355 +0x9f fp=0xc0009c3fe0 sp=0xc0009c3ec8 pc=0x591c2ff
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009c3fe8 sp=0xc0009c3fe0 pc=0x4d90a1
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
	cli.go:353 +0x1a5
bash: line 8: 57593 Aborted                 (core dumped) bash -c "./cockroach debug declarative-corpus-validate /tmp/corpus4225922576/corpus"
test artifacts and logs in: /artifacts/schemachange/mixed-versions-compat/run_1

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=4
  • encrypted=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@rafiss
Copy link
Collaborator

rafiss commented Nov 21, 2024 

run_074604.592078636_n1_cockroach-debug-decl: 2024/11/21 07:46:04 cluster.go:2550: > ./cockroach debug declarative-corpus-validate /tmp/corpus4225922576/corpus
teamcity-17868467-1732171355-04-n1cpu4:[1]: ./cockroach debug declarati...
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1  a panic has occurred!
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +unexpected EOF
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +(1) attached stack trace
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  -- stack trace:
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.init.MaybeDecorateError.func69
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	pkg/cli/clierrorplus/decorate_error.go:67
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/spf13/cobra.(*Command).execute
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	external/com_github_spf13_cobra/command.go:916
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/spf13/cobra.(*Command).ExecuteC
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	external/com_github_spf13_cobra/command.go:1044
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/spf13/cobra.(*Command).Execute
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	external/com_github_spf13_cobra/command.go:968
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.Run
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	cli.go:301
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.doMain
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	cli.go:139
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.Main
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	cli.go:64
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | main.main
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	pkg/cmd/cockroach/main.go:21
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | runtime.main
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	GOROOT/src/runtime/proc.go:271
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | runtime.goexit
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | 	src/runtime/asm_amd64.s:1695
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +Wraps: (2) unexpected EOF
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +Error types: (1) *withstack.withStack (2) *errors.errorString
panic: unexpected EOF [recovered]
	panic: unexpected EOF

goroutine 1 gp=0xc0000061c0 m=10 mp=0xc001daa008 [running]:
panic({0x62eaa00?, 0xc8b3950?})
	GOROOT/src/runtime/panic.go:779 +0x158 fp=0xc003195548 sp=0xc003195498 pc=0x49ca98
github.com/cockroachdb/cockroach/pkg/util/log/logcrash.RecoverAndReportPanic({0x87aca60, 0xd54cf00}, 0xc0032c2000)
	pkg/util/log/logcrash/crash_reporting.go:135 +0x6e fp=0xc003195598 sp=0xc003195548 pc=0xf10a2e
github.com/cockroachdb/cockroach/pkg/cli.doMain.deferwrap1()
	cli.go:137 +0x28 fp=0xc0031955c0 sp=0xc003195598 pc=0x591b788
panic({0x62eaa00?, 0xc8b3950?})
	GOROOT/src/runtime/panic.go:770 +0x132 fp=0xc003195670 sp=0xc0031955c0 pc=0x49ca72
github.com/cockroachdb/cockroach/pkg/cli.init.func17(0xc9d20e0, {0xc00180e900?, 0x4be49e0?, 0xc003305800?})
	declarative_corpus.go:38 +0x54c fp=0xc003195b18 sp=0xc003195670 pc=0x590c88c
github.com/cockroachdb/cockroach/pkg/cli.init.MaybeDecorateError.func69(0xc9d20e0?, {0xc00180e900?, 0x1?, 0x1?})
	pkg/cli/clierrorplus/decorate_error.go:67 +0x34 fp=0xc003195c58 sp=0xc003195b18 pc=0x58f9114
github.com/spf13/cobra.(*Command).execute(0xc9d20e0, {0xc00180e8d0, 0x1, 0x1})
	external/com_github_spf13_cobra/command.go:916 +0x882 fp=0xc003195d78 sp=0xc003195c58 pc=0x4bdaf02
github.com/spf13/cobra.(*Command).ExecuteC(0xc9cc200)
	external/com_github_spf13_cobra/command.go:1044 +0x3a5 fp=0xc003195e30 sp=0xc003195d78 pc=0x4bdb745
github.com/spf13/cobra.(*Command).Execute(...)
	external/com_github_spf13_cobra/command.go:968
github.com/cockroachdb/cockroach/pkg/cli.Run(...)
	cli.go:301
github.com/cockroachdb/cockroach/pkg/cli.doMain(0xc9d20e0, {0xc00215120a, 0x21})
	cli.go:139 +0x1e6 fp=0xc003195eb0 sp=0xc003195e30 pc=0x591b6a6
github.com/cockroachdb/cockroach/pkg/cli.Main()
	cli.go:64 +0x11f fp=0xc003195f40 sp=0xc003195eb0 pc=0x591b35f
main.main()
	pkg/cmd/cockroach/main.go:21 +0xf fp=0xc003195f50 sp=0xc003195f40 pc=0x5b0542f
runtime.main()
	GOROOT/src/runtime/proc.go:271 +0x29d fp=0xc003195fe0 sp=0xc003195f50 pc=0x4a00bd
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc003195fe8 sp=0xc003195fe0 pc=0x4d90a1

@rafiss rafiss changed the title roachtest: schemachange/mixed-versions-compat failed roachtest: schemachange/mixed-versions-compat failed [unexpected EOF during cockroach debug declarative-corpus-validate] Nov 21, 2024
@rafiss rafiss removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Nov 21, 2024
@rafiss
Copy link
Collaborator

rafiss commented Nov 21, 2024

Assigning to @Dedej-Bergin as this is the same as #133003

@rafiss rafiss added the P-2 Issues/test failures with a fix SLA of 3 months label Nov 21, 2024
annrpom added a commit to annrpom/cockroach that referenced this issue Nov 21, 2024
@annrpom
scpb: ensure IndexZoneConfig fields do not overlap
b20c7c8 
In cockroachdb#134522, we added a fix to ensure that we cleaned up backrefs
properly for our `IndexZoneConfig` element, but in doing so, we
created an overlap of field ID `3` between v24.2+.
This patch ensures that we do not re-use the same field ID for
a different field in our IndexZoneConfig element by marking it
as `reserved`.

Fixes: cockroachdb#133003
Fixes: cockroachdb#135807

Release note: None
@annrpom annrpom mentioned this issue Nov 21, 2024
Merged
craig bot pushed a commit that referenced this issue Nov 21, 2024
@annrpom
Merge #135933
11743ed 
135933: scpb: ensure IndexZoneConfig fields do not overlap r=annrpom a=annrpom

In [#134522](#134472), we added a fix to ensure that we cleaned up backrefs properly for our `IndexZoneConfig` element, but in doing so, we created an overlap of field ID `3` between v24.2+. This patch ensures that we do not re-use the same field ID for a different field in our IndexZoneConfig element by marking it as `reserved`.

Fixes: #133003
Fixes: #135807

Release note: None

Co-authored-by: Annie Pompa <[email protected]>
@craig craig bot closed this as completed in d0d3a1e Nov 21, 2024
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Nov 21, 2024

Based on the specified backports for linked PR #135933, I applied the following new label(s) to this issue: branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added the branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 label Nov 21, 2024
@annrpom annrpom mentioned this issue Nov 21, 2024
Merged
annrpom added a commit to annrpom/cockroach that referenced this issue Nov 21, 2024
@annrpom
scpb: ensure IndexZoneConfig fields do not overlap
ade746b 
In cockroachdb#134522, we added a fix to ensure that we cleaned up backrefs
properly for our `IndexZoneConfig` element, but in doing so, we
created an overlap of field ID `3` between v24.2+.
This patch ensures that we do not re-use the same field ID for
a different field in our IndexZoneConfig element by marking it
as `reserved`.

Fixes: cockroachdb#133003
Fixes: cockroachdb#135807

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Dedej-Bergin Dedej-Bergin

Labels
B-runtime-assertions-enabled branch-master Failures and bugs on the master branch. branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

scpb: ensure IndexZoneConfig fields do not overlap annrpom/cockroach
3 participants
@rafiss @cockroach-teamcity @Dedej-Bergin