Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql/logictest: TestLogic failed #73788

Closed
cockroach-teamcity opened this issue Dec 14, 2021 · 5 comments · Fixed by #74188
Closed

sql/logictest: TestLogic failed #73788

cockroach-teamcity opened this issue Dec 14, 2021 · 5 comments · Fixed by #74188
Assignees
@ajwerner @RichardJCai
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

sql/logictest.TestLogic failed with artifacts on master @ 5cd5be83218475ef0cc3c62a856b8d9d22cf142b:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2fc2332]

Stack:

goroutine 8234028 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc00a12cb40, {0x51cc818, 0xc00c131c90})
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:254 +0xaa
panic({0x3849d20, 0x734c7f0})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:674 +0x192
panic({0x3849d20, 0x734c7f0})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*syntheticDescriptors).set(0xc00a0b8f38, {0x0, 0x1, 0xc0138d45f0})
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/synthetic_descriptors.go:39 +0x5a
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).SetSyntheticDescriptors(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/collection.go:384
github.com/cockroachdb/cockroach/pkg/sql.(*Server).newConnExecutorWithTxn(0x0, {0x51cc818, 0xc00df7ed90}, 0x203000, 0xc0003f7a10, {0x5310b20, 0xc02668bd00}, 0x0, {{0xc01439e7e0, 0xc0041fdda0}, ...}, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:931 +0x2e5
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx(0xc031222a20, {0x51cc818, 0xc00df7ed90}, 0xc005902a50, {0x51580d8, 0xc02668bbc0}, 0xc0119c3b80, 0x0, 0xc003a34940, 0xc0003f79e0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:187 +0x3ef
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal(0xc031222a20, {0x51b0c78, 0xc01fada540}, {0x3d562b3, 0xe}, 0xc02668bbc0, 0xc00b5dc240, {{{0x3d3cbda, 0x4}}, {0x0, ...}, ...}, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:732 +0x725
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered(0x0, {0x51b0c78, 0xc01fada540}, {0x3d562b3, 0xe}, 0x0, {{{0x3d3cbda, 0x4}}, {0x0, 0x0}, ...}, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:427 +0x19b
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols(0xc00d2cad50, {0x51b0c78, 0xc01fada540}, {0x3d562b3, 0x425965}, 0x8, {{{0x3d3cbda, 0x4}}, {0x0, 0x0}, ...}, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:485 +0xbb
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx(0x0, {0x51b0c78, 0xc01fada540}, {0x3d562b3, 0x0}, 0x0, {{{0x3d3cbda, 0x4}}, {0x0, 0x0}, ...}, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:471 +0xa5
github.com/cockroachdb/cockroach/pkg/jobs.(*Job).load.func1({0x51b0c78, 0xc01fada540}, 0x203002)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:776 +0x33c
github.com/cockroachdb/cockroach/pkg/jobs.(*Job).runInTxn(0x160, {0x51b0c78, 0xc01fada540}, 0x160, 0xc00074a000)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:737 +0x37
github.com/cockroachdb/cockroach/pkg/jobs.(*Job).load(0xc0158e6c60, {0x51b0c78, 0xc01fada540}, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:766 +0x9c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).LoadJobWithTxn(0xc020ee9440, {0x51b0c78, 0xc01fada540}, 0x9fa47fcb8300001, 0x1c28f57)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:722 +0x7f
github.com/cockroachdb/cockroach/pkg/sql/gcjob.initDetailsAndProgress.func1({0x51b0c78, 0xc01fada540}, 0xc00034c190)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job_utils.go:69 +0x74
github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1({0x51b0c78, 0xc01fada540}, 0xc00ba37680)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:849 +0x27
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0xc005902a50, {0x51b0c78, 0xc01fada540}, 0xc00123d3b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:961 +0xbf
github.com/cockroachdb/cockroach/pkg/kv.runTxn({0x51b0c78, 0xc01fada540}, 0xc01450cd80, 0x40)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:848 +0x5a
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(0x0, {0x51b0c78, 0xc01fada540}, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:830 +0x85
github.com/cockroachdb/cockroach/pkg/sql/gcjob.initDetailsAndProgress({0x51b0c78, 0xc01fada540}, 0xc0043d0400, 0x9fa47fcb8300001)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job_utils.go:67 +0xd5
github.com/cockroachdb/cockroach/pkg/sql/gcjob.schemaChangeGCResumer.Resume({0x9fa47fcb8300001}, {0x51b0c78, 0xc01fada540}, {0x3b75c00, 0xc000010858})
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job.go:181 +0x151
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2(0x51cc818, {0x5158920, 0xc02dbde478}, {0x51b0c78, 0xc01fada540}, {0x3b75c00, 0xc000010858}, 0xc00123d948)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1209 +0xba
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc020ee9440, {0x51cc818, 0xc00034c190}, {0x3b75c00, 0xc000010858}, {0x5158920, 0xc02dbde478}, 0xc018f77ce0, {0xc003a34290, 0x7}, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1210 +0x6b7
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc020ee9440, {0x51b0bd0, 0xc0042c0ac0}, {0x5158920, 0xc02dbde478}, 0xc018f77ce0, {0xc003a34290, 0x7}, {0xc0184a22a0, 0x16})
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:414 +0x54f
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1({0x51cc818, 0xc00f056790})
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:336 +0xfa
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:491 +0x12a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:481 +0x445
Log preceding fatal error 

=== RUN   TestLogic/fakedist-metadata/schema_change_in_txn/check_on_new_table

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)Parameters in this failure:

  • GOFLAGS=-parallel=2

Same failure on other branches

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Dec 14, 2021
@blathers-crl blathers-crl bot added the T-sql-queries SQL Queries Team label Dec 14, 2021
@blathers-crl blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label Dec 14, 2021
@cucaroach
Copy link
Contributor

Tagging schema since trace has schema Changer GC in it.

@cockroach-teamcity

This comment has been minimized.

Sign in to view
@rytaft

This comment has been minimized.

Sign in to view
This was referenced Dec 17, 2021
Merged
Merged
@cockroach-teamcity

This comment has been minimized.

Sign in to view
@ajwerner
Copy link
Contributor

This bug was introduced by #71246. The problem is that we're now sharing a global singleton and then we can do also sorts of shady things by racing and overwriting the synthetic descriptors.

@ajwerner ajwerner mentioned this issue Dec 22, 2021
Merged
craig bot pushed a commit that referenced this issue Jan 3, 2022
@ajwerner
Merge #71910 #74188
c1f1185 
71910: sql: add a cluster setting to avoid system config triggers r=ajwerner a=ajwerner

This is intended as a short-term workaround to improve performance in
situations of repeated schema changes, like ORM tests.

We have a plan to disable the system config trigger altogether in 22.1 with
#70560. 

This PR provides a cluster setting which allows schema change transactions
to bypass triggerring an update to the system config span. These updates
currently drive only the propagation of zone configs to KV and cluster
settings. The cluster setting behavior is retained until we address #70566.

We have a history of these sorts of unsafe settings in
`kv.raft_log.disable_synchronization_unsafe`.

Release note: None

74188: sql: fix InternalExecutor bug r=ajwerner a=ajwerner

Any time we use WithSyntheticDescriptors, it has to be on an unshared internal
executor. The move in #71246 to not have an internal executor hanging around
for the current session hurts here.

Fixes #73788

Release note: None

Co-authored-by: Andrew Werner <[email protected]>
@craig craig bot closed this as completed in 79c4c86 Jan 3, 2022
gustasva pushed a commit to gustasva/cockroach that referenced this issue Jan 4, 2022
@ajwerner
sql: fix InternalExecutor bug
f1373bf 
Any time we use WithSyntheticDescriptors, it has to be on an unshared internal
executor. The move in cockroachdb#71246 to not have an internal executor hanging around
for the current session hurts here.

Fixes cockroachdb#73788

Release note: None
@exalate-issue-sync exalate-issue-sync bot added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-schema-deprecated Use T-sql-foundations instead T-sql-queries SQL Queries Team labels May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ajwerner ajwerner

@RichardJCai RichardJCai

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sql: fix InternalExecutor bug ajwerner/cockroach
5 participants
@ajwerner @rytaft @cockroach-teamcity @RichardJCai @cucaroach