Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: acceptance/version-upgrade failed: invalid attempted write of database descriptor #58307

Closed
jbowens opened this issue Dec 28, 2020 · 13 comments
Closed

roachtest: acceptance/version-upgrade failed: invalid attempted write of database descriptor #58307

jbowens opened this issue Dec 28, 2020 · 13 comments
Assignees
@ajwerner
Labels
A-schema-descriptors Relating to SQL table/db descriptor handling. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-roachtest
Milestone
20.2

Comments

@jbowens
Copy link
Collaborator

jbowens commented Dec 28, 2020

While bumping Pebble in the release-20.2 branch, my PR #58304 hit this roachtest failure that appears to be unrelated to the change:

I201228 16:35:06.841433 201 server/server.go:1119 â‹® active cluster version is now 20.2 (up from 20.1)
I201228 16:35:08.772686 793 sql/sqlliveness/slstorage/slstorage.go:342 ⋮ [n2] inserted sqlliveness session ‹a418c755aebf4c96b7bbefa64d909a9b›
I201228 16:35:08.772738 793 sql/sqlliveness/slinstance/slinstance.go:143 ⋮ [n2] created new SQL liveness session ‹a418c755aebf4c96b7bbefa64d909a9b›
I201228 16:35:08.777500 5035 sql/table.go:102 ⋮ [n2,client=‹127.0.0.1:58956›,hostnossl,user=root] queued new database schema change job 619652411466579970 for database 162
F201228 16:35:08.777559 5035 sql/database.go:112 ⋮ [n2,client=‹127.0.0.1:58956›,hostnossl,user=root] invalid attempted write of database descriptor
goroutine 5035 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x95a7701, 0x742236, 0xc00130eb80, 0x20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0xb8
github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0x95a69a0, 0x4, 0x1654ee993bb9f652, 0x13ab, 0x874e1bf, 0xf, 0x70, 0xc0021cc750, 0x2e, 0xc00086eac0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:243 +0x9e5
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x618ba80, 0xc002726400, 0x4, 0x2, 0x512df6f, 0x2e, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:52 +0x1ac
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x618ba80, 0xc002726400, 0x1, 0xc000000004, 0x512df6f, 0x2e, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:44 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatal(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:173
github.com/cockroachdb/cockroach/pkg/sql.(*planner).writeDatabaseChangeToBatch(0xc001e9b418, 0x618ba80, 0xc002726400, 0xc0029f7980, 0xc000f9e000, 0x4e, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/database.go:112 +0x13f
github.com/cockroachdb/cockroach/pkg/sql.(*planner).writeNonDropDatabaseChange(0xc001e9b418, 0x618ba80, 0xc002726400, 0xc0029f7980, 0xc0009453b0, 0x4e, 0x4e, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/database.go:99 +0xeb
github.com/cockroachdb/cockroach/pkg/sql.(*planner).createUserDefinedSchema(0xc001e9b418, 0x618ba80, 0xc002726400, 0xc001f47340, 0xc001e9b418, 0xc00169f230, 0x7b5601, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/create_schema.go:136 +0x83f
github.com/cockroachdb/cockroach/pkg/sql.(*createSchemaNode).startExec(0xc001a69f08, 0x618ba80, 0xc002726400, 0xc001f47340, 0xc001e9b418, 0xc001c35300, 0x2749212)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/create_schema.go:36 +0x5b
github.com/cockroachdb/cockroach/pkg/sql.startExec.func2(0x5092b12, 0xd, 0x618d900, 0xc001a69f08, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:506 +0xb9
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1(0xc0027996e8, 0x5092b12, 0xd, 0x618d900, 0xc001a69f08)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:119 +0x5d
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal(0xc001c356e8, 0x618d900, 0xc001a69f08, 0x5092b12, 0xd)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:298 +0xbf
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit(0xc001c356e8, 0x618d900, 0xc001a69f08, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:86 +0x8d
github.com/cockroachdb/cockroach/pkg/sql.walkPlan(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:50
github.com/cockroachdb/cockroach/pkg/sql.startExec(0x618ba80, 0xc002726400, 0xc001f47340, 0xc001e9b418, 0x618d900, 0xc001a69f08, 0x0, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:509 +0x11f
github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start(0xc000f9ef00, 0x618ba80, 0xc002726400, 0x95f4e20, 0x48a1840)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:125 +0xd6
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBase).Run(0xc000f9ef00, 0x618ba80, 0xc002726400)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:774 +0x52
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run(0xc0019c6c60, 0x618ba80, 0xc002726400, 0x567fc80, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:392 +0x24d
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc0008914a0, 0xc0008477a0, 0xc001934a20, 0xc002a2ec00, 0xc00012ea80, 0xc001e9b518, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:422 +0x5d9
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc0008914a0, 0x618bb40, 0xc00169f290, 0xc001e9b518, 0xc0008477a0, 0xc001934a20, 0x618d900, 0xc001a69f08, 0x0, 0xc00012ea80, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1002 +0x1d1
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc001e9b000, 0x618bb40, 0xc00169f290, 0xc001e9b418, 0x1, 0x7f277540da78, 0xc001fda680, 0xc001c36100, 0xc000910ef8, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1001 +0x3c0
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc001e9b000, 0x618bb40, 0xc00169f290, 0xc001e9b418, 0x7f277540da78, 0xc001fda680, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:872 +0x6ff
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc001e9b000, 0x618bb40, 0xc00169f290, 0x61a9440, 0xc00169f230, 0xc002ca63b5, 0x24, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:639 +0xbb1
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc001e9b000, 0x618bb40, 0xc00169f290, 0x61a9440, 0xc00169f230, 0xc002ca63b5, 0x24, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:114 +0x924
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1(0x61a9440, 0xc00169f230, 0xc002ca63b5, 0x24, 0x0, 0x0, 0x362d543f, 0xed77c003a, 0x0, 0x362d5cb9, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1465 +0x2d5
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc001e9b000, 0x618ba80, 0xc002726340, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1467 +0x148d
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc001e9b000, 0x618ba80, 0xc002c8a080, 0xc000e34fa0, 0x5400, 0x15000, 0xc000e35040, 0xc002c82090, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1391 +0x1f2
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc0010b4680, 0x618ba80, 0xc002c8a080, 0xc001e9b000, 0x5400, 0x15000, 0xc000e35040, 0xc002c82090, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:508 +0x104
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc0026a003e, 0xc002a82080, 0x618ba80, 0xc002c8a080, 0xc002c82090, 0xc0010b4680, 0xc001fda000, 0x61bb380, 0xc0023cc330, 0xc00275a3c0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:626 +0x317
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:554 +0x17e

https://teamcity.cockroachdb.com/viewLog.html?buildId=2547260&buildTypeId=Cockroach_UnitTests_Roachtest&tab=buildResultsDiv
@jbowens jbowens added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). A-schema-descriptors Relating to SQL table/db descriptor handling. O-roachtest branch-release-20.2 labels Dec 28, 2020
@jbowens jbowens added this to the 20.2 milestone Dec 28, 2020
@ajwerner
Copy link
Contributor

Yikes, the version switch happens mid-statement and we hit the panic.

@ajwerner
Copy link
Contributor

Ah yeah so we're not doing the gate check for user defined schemas properly. I think the window for this crash is very small but it's still worthy of a fix.

@cockroach-teamcity cockroach-teamcity mentioned this issue Dec 29, 2020
Closed
This was referenced Jan 6, 2021
Closed
Closed
@cockroach-teamcity
Copy link
Member

(roachtest).acceptance/version-upgrade failed on release-20.2@91e4d8ab9d996a2c168c81371b3862115219a3fe:

		  |  1686.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1686.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1687.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1687.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1688.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1688.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1689.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1689.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1690.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1690.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1691.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  1691.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1692.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1692.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1693.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1693.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1694.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1694.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1695.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1695.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1696.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1696.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1697.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1697.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1698.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1698.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1699.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1699.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1700.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1700.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1701.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  1701.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1702.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1702.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1703.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1703.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1704.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1704.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1705.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1705.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1706.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1706.0s        2            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
More

Artifacts: /acceptance/version-upgrade
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 12, 2021
Closed
@yuzefovich yuzefovich mentioned this issue Feb 3, 2021
Closed
@yuzefovich
Copy link
Member

Hit an instance of this in the wild #59752.

@cockroach-teamcity
Copy link
Member

(roachtest).acceptance/version-upgrade failed on release-20.2@8c79e2bc4b35d36c8527f4c40c974f03d9034f46:

The test failed on branch=release-20.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/version-upgrade/run_1
	cluster.go:2212,mixed_version_schemachange.go:46,versionupgrade.go:189,versionupgrade.go:177,acceptance.go:58,acceptance.go:95,test_runner.go:755: output in run_074736.123_n1-4_workload_init_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2657161-1612856692-16-n4cpu4:1-4 -- ./workload init schemachange returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2290
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2210
		  | main.uploadAndInitSchemaChangeWorkload.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:46
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runVersionUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:177
		  | main.registerAcceptance.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:58
		  | main.registerAcceptance.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:95
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_074736.123_n1-4_workload_init_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2657161-1612856692-16-n4cpu4:1-4 -- ./workload init schemachange returned
		  | stderr:
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./workload init schemachange
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | teamcity-2657161-1612856692-16-n4cpu4: ./workload init schemachange
		  |    1: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    2: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    3: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    4: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: /acceptance/version-upgrade
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

This was referenced Feb 9, 2021
Closed
Closed
@cockroach-teamcity
Copy link
Member

(roachtest).acceptance/version-upgrade failed on release-20.2@b0012907c1bc9627ae2de83e6099c4930a32699e:

The test failed on branch=release-20.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/version-upgrade/run_1
	cluster.go:2212,mixed_version_schemachange.go:46,versionupgrade.go:189,versionupgrade.go:177,acceptance.go:58,acceptance.go:95,test_runner.go:755: output in run_071920.197_n1-4_workload_init_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2661584-1612941367-16-n4cpu4:1-4 -- ./workload init schemachange returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2290
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2210
		  | main.uploadAndInitSchemaChangeWorkload.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:46
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runVersionUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:177
		  | main.registerAcceptance.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:58
		  | main.registerAcceptance.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:95
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_071920.197_n1-4_workload_init_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2661584-1612941367-16-n4cpu4:1-4 -- ./workload init schemachange returned
		  | stderr:
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./workload init schemachange
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | teamcity-2661584-1612941367-16-n4cpu4: ./workload init schemachange
		  |    1: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    2: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    3: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    4: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: /acceptance/version-upgrade
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member

(roachtest).acceptance/version-upgrade failed on release-20.2@e0d26e817d5bdaa280a4692a3346f3eef0d4cce7:

		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	cluster.go:1657,context.go:135,cluster.go:1646,test_runner.go:836: dead node detection: /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes: exit status 1 4: dead
		3: dead
		2: 25299
		1: 25434
		Error: UNCLASSIFIED_PROBLEM: 4: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) secondary error attachment
		  | 3: dead
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | main.glob..func14
		  |   | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  |   | main.wrap.func1
		  |   | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  |   | github.com/spf13/cobra.(*Command).execute
		  |   | 	/go/pkg/mod/github.com/spf13/[email protected]/command.go:830
		  |   | github.com/spf13/cobra.(*Command).ExecuteC
		  |   | 	/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
		  |   | github.com/spf13/cobra.(*Command).Execute
		  |   | 	/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
		  |   | main.main
		  |   | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  |   | runtime.main
		  |   | 	/usr/local/go/src/runtime/proc.go:203
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		  | Wraps: (2) 3: dead
		  | Error types: (1) *withstack.withStack (2) *errutil.leafError
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/go/pkg/mod/github.com/spf13/[email protected]/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
		  | main.main
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (4) 4: dead
		Error types: (1) errors.Unclassified (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *errutil.leafError
More

Artifacts: /acceptance/version-upgrade
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 11, 2021
Closed
@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 19, 2021
Closed
@andreimatei andreimatei changed the title roachtest: acceptance/version-upgrade failed roachtest: acceptance/version-upgrade failed: invalid attempted write of database descriptor Mar 22, 2021
@andreimatei
Copy link
Contributor

Happened again at the tip of release-20.2:
https://teamcity.cockroachdb.com/viewLog.html?buildId=2800079&buildTypeId=Cockroach_UnitTests

@postamar did you look into this at all?
Sounds likely easy to repro since we've had all these reports from acceptance/version-upgrade. You can stress this roachtest locally - I can show you how if you want.

@postamar
Copy link
Contributor

No I haven't, it was in our backlog. That being said if it's a low-hanging fruit I'll take a look ASAP.

@ajwerner
Copy link
Contributor

This is low-hanging fruit for the 20.1->20.2 upgrade. We've got a problem with how we check the gate on User Defined Schemas. That being said, it's a transient error that is not super common to come up so we deprioritized it.

@cockroach-teamcity
Copy link
Member

(roachtest).acceptance/version-upgrade failed on release-20.2@e3bc5c226cae9970108803c743119619f3125f3e:

The test failed on branch=release-20.2, cloud=gce:
test artifacts and logs in: artifacts/acceptance/version-upgrade/run_1
	versionupgrade.go:260,versionupgrade.go:114,versionupgrade.go:189,versionupgrade.go:177,acceptance.go:58,acceptance.go:95,test_runner.go:755: dial tcp 127.0.0.1:26257: connect: connection refused

	cluster.go:1658,context.go:135,cluster.go:1647,test_runner.go:836: dead node detection: /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes: exit status 1 1: dead
		4: 27218
		3: 27083
		2: 26951
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/go/pkg/mod/github.com/spf13/[email protected]/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
		  | main.main
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
More

Artifacts: /acceptance/version-upgrade
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member

(roachtest).acceptance/version-upgrade failed on release-20.2@3ebdf340cb2e9f995a4e969d75dcc53fcc8daefa:

The test failed on branch=release-20.2, cloud=gce:
test artifacts and logs in: artifacts/acceptance/version-upgrade/run_1
	versionupgrade.go:283,versionupgrade.go:388,retry.go:197,versionupgrade.go:387,versionupgrade.go:189,versionupgrade.go:177,acceptance.go:58,acceptance.go:95,test_runner.go:755: dial tcp 127.0.0.1:26261: connect: connection refused

	cluster.go:1658,context.go:135,cluster.go:1647,test_runner.go:836: dead node detection: /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes: exit status 1 3: dead
		4: 26028
		1: 25761
		2: 25897
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/go/pkg/mod/github.com/spf13/[email protected]/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
		  | main.main
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
More

Artifacts: /acceptance/version-upgrade
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@ajwerner ajwerner mentioned this issue Apr 16, 2021
Merged
ajwerner added a commit to ajwerner/cockroach that referenced this issue Apr 19, 2021
@ajwerner
release-20.2: sql: fix version gate for user-defined schemas
8a50a26 
The cluster version which enables user-defined schemas comes after the
activation of the version which enables leasing database descriptors. However
the usability of leased database descriptors is checked at the beginning of
the transaction to ensure that the use remains stable throughout the life
of the transaction. The use of leased database descriptors is a requirement
to create user-defined schemas. This difference in when the two conditions
are checked leads to a window for a crash whereby we attempt to create a
new-style database due to a `CREATE SCHEMA` statement in a transaction which
does not have those databases enabled.

We fix this by checking both conditions in the `CREATE SCHEMA` execution.

Fixes cockroachdb#58307.

Release note (bug fix): Fixed a rare bug which can cause a crash if CREATE
SCHEMA is run in a transaction which began prior to a cluster version update
which enables that statement.
@cockroach-teamcity
Copy link
Member

(roachtest).acceptance/version-upgrade failed on release-20.2@dfe5778eddb408f0ceb8152538be7c34401866e2:

The test failed on branch=release-20.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/version-upgrade/run_1
	versionupgrade.go:260,versionupgrade.go:343,versionupgrade.go:189,versionupgrade.go:177,acceptance.go:58,acceptance.go:95,test_runner.go:755: write tcp 172.17.0.3:39796->34.122.44.53:26257: write: broken pipe
More

Artifacts: /acceptance/version-upgrade
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 30, 2021
Closed
@postamar postamar closed this as completed May 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ajwerner ajwerner

Labels
A-schema-descriptors Relating to SQL table/db descriptor handling. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-roachtest
Projects
None yet
Milestone
20.2
Development

No branches or pull requests
6 participants
@andreimatei @jbowens @ajwerner @postamar @yuzefovich @cockroach-teamcity