Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cdc: crdb_internal.complete_replication_stream causes panic: UpdateDeadline() called on leaf txn #78553

Closed
cockroach-teamcity opened this issue Mar 26, 2022 · 8 comments
Assignees
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 26, 2022

roachtest.sqlsmith/setup=seed/setting=no-mutations failed with artifacts on master @ 090d481f2ec1d1c3e668274f50e4f152b4596a1b:

				AS (
					SELECT
						*
					FROM
						(VALUES (1098543467:::OID), (3336355036:::OID), (28401922:::OID), (0:::OID), (3303751647:::OID))
							AS tab_64129 (col_157221)
				),
			with_26262 (col_157222)
				AS (
					SELECT
						*
					FROM
						(
							VALUES
								('1985-06-30':::DATE),
								('1996-12-31':::DATE),
								('1991-06-10':::DATE),
								('1974-05-18':::DATE),
								(('1981-05-11':::DATE::DATE - 9026887776678856910:::INT8::INT8)::DATE)
						)
							AS tab_64130 (col_157222)
				)
		SELECT
			tab_64131._bytes AS col_157223,
			NULL AS col_157224,
			tab_64131._decimal AS col_157225,
			'2003-04-17 02:17:55.000127':::TIMESTAMP AS col_157226,
			0:::OID AS col_157227,
			73600681407.17610711:::DECIMAL AS col_157228,
			tab_64131._jsonb AS col_157229,
			false AS col_157230,
			tab_64131._int8 AS col_157231,
			tab_64131._string AS col_157232,
			tab_64131._timestamptz AS col_157233,
			29585:::INT8 AS col_157234,
			tab_64131._enum AS col_157235,
			crdb_internal.complete_replication_stream(tab_64131._int2::INT8)::INT8 AS col_157236,
			'1988-03-26':::DATE AS col_157237,
			tab_64131._float8 AS col_157238,
			tab_64131._uuid AS col_157239,
			0.9110737708810559:::FLOAT8 AS col_157240,
			'77.198.180.245/0':::INET AS col_157241
		FROM
			defaultdb.public.seed@seed__int8__float8__date_idx AS tab_64131
		WHERE
			(tab_64131._bool AND true)
		ORDER BY
			tab_64131._string
		LIMIT
			54:::INT8;
Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-14177

Epic CRDB-11732

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Mar 26, 2022
@blathers-crl blathers-crl bot added the T-sql-queries SQL Queries Team label Mar 26, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.sqlsmith/setup=seed/setting=no-mutations failed with artifacts on master @ 834eaa0e83350486830867b5edd6e8809b52aa55:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/sqlsmith/setup=seed/setting=no-mutations/run_1
	sqlsmith.go:284,sqlsmith.go:304,test_runner.go:875: ping node 1: dial tcp 34.74.41.60:26257: connect: connection refused
		previous sql:
		SELECT
			tab_75328.tableoid AS col_184170,
			tab_75328._string AS col_184171,
			tab_75327._int8 AS col_184172,
			'1975-05-25':::DATE AS col_184173,
			'09d45faa-8906-4b4c-98ba-3b0cf10131f0':::UUID AS col_184174,
			7.679797610924093543E+25:::DECIMAL AS col_184175,
			tab_75327._bool AS col_184176,
			tab_75327._enum AS col_184177,
			tab_75327._enum AS col_184178,
			e' \x0bG:\x19Y\x1bu':::STRING AS col_184179,
			'd1d:23d6:77c5:1cf8:acdf:b601:3254:6e8a/42':::INET AS col_184180,
			crdb_internal.complete_replication_stream(tab_75328._int8::INT8)::INT8 AS col_184181,
			tab_75327._decimal AS col_184182,
			'045f23ad-5404-488c-b05b-ebd645474741':::UUID AS col_184183,
			tab_75327._int2 AS col_184184,
			tab_75327._inet AS col_184185,
			tab_75327._float8 AS col_184186,
			tab_75328._float4 AS col_184187,
			tab_75328.tableoid AS col_184188,
			tab_75328._int8 AS col_184189,
			tab_75328._timestamp AS col_184190,
			tab_75328._float8 AS col_184191,
			tab_75328._bytes AS col_184192
		FROM
			defaultdb.public.seed@seed__int8__float8__date_idx AS tab_75327,
			defaultdb.public.seed@seed__int8__float8__date_idx AS tab_75328
		WHERE
			tab_75327._bool;
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.sqlsmith/setup=seed/setting=no-mutations failed with artifacts on master @ 32b45c4bcf1ab41f0ba3abd36cb670eea7f450fd:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/sqlsmith/setup=seed/setting=no-mutations/run_1
	sqlsmith.go:284,sqlsmith.go:304,test_runner.go:875: ping node 1: dial tcp 35.237.146.227:26257: connect: connection refused
		previous sql:
		SELECT
			current_timestamp(crdb_internal.complete_replication_stream(2773075085751053386:::INT8::INT8)::INT8::INT8):::TIMESTAMP::TIMESTAMP
				AS col_402595
		FROM
			defaultdb.public.seed@seed__int8__float8__date_idx AS tab_165036
		WHERE
			tab_165036._bool
		ORDER BY
			tab_165036._bytes
		LIMIT
			99:::INT8;
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@rytaft
Copy link
Collaborator

rytaft commented Mar 28, 2022

All three of the above failures were due to panic: UpdateDeadline() called on leaf txn. Here is the stack trace from the first one:

panic: UpdateDeadline() called on leaf txn

goroutine 56071 [running]:
panic({0x4971f80, 0xc0045ff830})
	GOROOT/src/runtime/panic.go:1147 +0x3a8 fp=0xc004055fa0 sp=0xc004055ee0 pc=0x48a768
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).UpdateDeadline(0x8f93320, {0x6269df8, 0xc0045ff7d0}, {0x4d5a5e0, 0x2eebac0, 0xc0})
	github.com/cockroachdb/cockroach/pkg/kv/txn.go:782 +0x313 fp=0xc004056080 sp=0xc004055fa0 pc=0x13ceff3
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*leasedDescriptors).maybeUpdateDeadline(0xc001b93bd8, {0x6269df8, 0xc0045ff7d0}, {0x61e6b50, 0xc009648210}, {0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/leased_descriptors.go:220 +0x191 fp=0xc004056110 sp=0xc004056080 pc=0x1935e91
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*leasedDescriptors).getResult(0xc001b93bc8, {0x6269df8, 0xc0045ff7d0}, {0x61e6b50, 0xc009648210}, 0x1, {0x62d36e0, 0xc002eebac0}, {0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/leased_descriptors.go:186 +0x477 fp=0xc004056220 sp=0xc004056110 pc=0x1935c57
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*leasedDescriptors).getByName(0xc001b93bc8, {0x6269df8, 0xc0045ff7d0}, {0x61e6b50, 0xc009648210}, 0x1, 0x1d, {0xc004eac2e1, 0x4})
	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/leased_descriptors.go:108 +0x372 fp=0xc0040562f0 sp=0xc004056220 pc=0x1935392
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getByName(0xc001b93ba0, {0x6269df8, 0xc0045ff7d0}, 0xc009648210, {0x63734c0, 0xc000262370}, {0x63673b8, 0x8f81e70}, {0xc004eac2e1, 0x4}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:315 +0x2ed fp=0xc0040563e0 sp=0xc0040562f0 pc=0x192ef2d
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getObjectByNameIgnoringRequiredAndType(0xc001b93ba0, {0x6269df8, 0xc0045ff7d0}, 0xc009e35700, {0xc004eac2da, 0x6}, {0x4dd49f8, 0x6}, {0xc004eac2e1, 0x4}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/object.go:200 +0x3c7 fp=0xc004056540 sp=0xc0040563e0 pc=0x1937767
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getObjectByName(0x0, {0x6269df8, 0xc0045ff7d0}, 0x0, {0xc004eac2da, 0x0}, {0x4dd49f8, 0x0}, {0xc004eac2e1, 0x4}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/object.go:67 +0x1b9 fp=0xc004056708 sp=0xc004056540 pc=0x19367d9
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).GetObjectDesc(0x0, {0x6269df8, 0xc0045ff7d0}, 0x0, {0xc004eac2da, 0xc00a5f6950}, {0x4dd49f8, 0x0}, {0xc004eac2e1, 0x4}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/object.go:41 +0x9f fp=0xc004056808 sp=0xc004056708 pc=0x19364df
github.com/cockroachdb/cockroach/pkg/sql.(*planner).LookupObject(0x0, {0x6269df8, 0xc0045ff7d0}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, 0x0, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/resolver.go:200 +0x5d6 fp=0xc004056948 sp=0xc004056808 pc=0x2fe3d96
github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver.ResolveExisting({0x6269df8, 0xc0045ff7d0}, 0xc004d5bec0, {0x7fd71f1cbdb8, 0xc001b93fc0}, {{0x1, 0x0, 0x0, 0x0, 0x0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver/resolver.go:435 +0x33b fp=0xc004056b88 sp=0xc004056948 pc=0x1925dbb
github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver.ResolveExistingObject({0x6269df8, 0xc0045ff7d0}, {0x62e54a8, 0xc001b93fc0}, 0xc004d5bec0, {{0x1, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver/resolver.go:194 +0x173 fp=0xc004056d70 sp=0xc004056b88 pc=0x1923c13
github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver.ResolveExistingTableObject({0x6269df8, 0xc0045ff7d0}, {0x62e54a8, 0xc001b93fc0}, 0xc001b94800, {{0x1, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver/resolver.go:122 +0x15d fp=0xc004056e58 sp=0xc004056d70 pc=0x19230bd
github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).ResolveDataSource(0xc001b947e8, {0x6269df8, 0xc0045ff7d0}, {0x98, 0x4}, 0xc00a5f7100)
	github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:207 +0x20f fp=0xc004057090 sp=0xc004056e58 pc=0x2f9116f
github.com/cockroachdb/cockroach/pkg/sql/opt.(*Metadata).CheckDependencies(0xc0055b0d20, {0x6269df8, 0xc0045ff7d0}, {0x630d358, 0xc001b947e8})
	github.com/cockroachdb/cockroach/pkg/sql/opt/metadata.go:293 +0xf7 fp=0xc004057158 sp=0xc004057090 pc=0x1b03bb7
github.com/cockroachdb/cockroach/pkg/sql/opt/memo.(*Memo).IsStale(0xc0055b0d20, {0x6269df8, 0xc0045ff7d0}, 0xc001b94280, {0x630d358, 0xc001b947e8})
	github.com/cockroachdb/cockroach/pkg/sql/opt/memo/memo.go:333 +0x4c7 fp=0xc004057190 sp=0xc004057158 pc=0x1c9a367
github.com/cockroachdb/cockroach/pkg/sql.(*planner).prepareUsingOptimizer(0xc001b93fc0, {0x6269df8, 0xc0045ff7d0})
	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:119 +0x9ef fp=0xc004057310 sp=0xc004057190 pc=0x2fbb6cf
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).populatePrepared(0xc001b93900, {0x6269df8, 0xc0045ff7d0}, 0xc012859698, {0xc012859698, 0x1, 0x8}, 0xc001b93fc0)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_prepare.go:278 +0x1ab fp=0xc004057428 sp=0xc004057310 pc=0x2ed36eb
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).prepare.func1({0x6269df8, 0xc0045ff7d0}, 0x0)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_prepare.go:235 +0x48a fp=0xc0040575a8 sp=0xc004057428 pc=0x2ed32aa
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).prepare(0xc001b93900, {0x6269df8, 0xc0045ff7d0}, {{{0x629a098, 0xc001f82230}, {0xc003d0ae00, 0x73}, 0x1, 0x1}, {0xc003d0b200, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_prepare.go:240 +0x327 fp=0xc0040576c8 sp=0xc0040575a8 pc=0x2ed2ba7
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).addPreparedStmt(0xc001b93900, {0x6269df8, 0xc0045ff7d0}, {0x0, 0x0}, {{{0x629a098, 0xc001f82230}, {0xc003d0ae00, 0x73}, 0x1, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_prepare.go:102 +0x110 fp=0xc0040577f8 sp=0xc0040576c8 pc=0x2ed24b0
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPrepare(0xc001b93900, {0x6269df8, 0xc0045ff7a0}, {{0x0, 0x0}, {{0x629a098, 0xc001f82230}, {0xc003d0ae00, 0x73}, 0x1, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_prepare.go:63 +0x3b8 fp=0xc0040579c8 sp=0xc0040577f8 pc=0x2ed21d8
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc001b93900)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1962 +0x10c6 fp=0xc004057e60 sp=0xc0040579c8 pc=0x2eb8266
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc001b93900, {0x6269df8, 0xc0045ff440}, 0xc0045fe960, {0x0, 0x0, 0x0, 0x0, 0x0}, 0x0)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1786 +0x278 fp=0xc004057f30 sp=0xc004057e60 pc=0x2eb6f98
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx.func1()
	github.com/cockroachdb/cockroach/pkg/sql/internal.go:205 +0xa5 fp=0xc004057fe0 sp=0xc004057f30 pc=0x2f824a5
runtime.goexit()
	GOROOT/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc004057fe8 sp=0xc004057fe0 pc=0x4bf181
created by github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx
	github.com/cockroachdb/cockroach/pkg/sql/internal.go:204 +0x5f1

@rytaft
Copy link
Collaborator

rytaft commented Mar 28, 2022

All of these queries include crdb_internal.complete_replication_stream, which was merged three days ago in #78014

@blathers-crl blathers-crl bot added the T-cdc label Mar 28, 2022
@blathers-crl
Copy link

blathers-crl bot commented Mar 28, 2022

cc @cockroachdb/cdc

@rytaft rytaft changed the title roachtest: sqlsmith/setup=seed/setting=no-mutations failed cdc: crdb_internal.complete_replication_stream causes panic: UpdateDeadline() called on leaf txn Mar 28, 2022
rytaft added a commit to rytaft/cockroach that referenced this issue Mar 28, 2022
Don't use crdb_internal.complete_replication_stream when generating
sqlsmith queries, since it currently causes a panic.

Informs cockroachdb#78553

Release note: None
craig bot pushed a commit that referenced this issue Mar 28, 2022
78435: opt: fix deduplication of unique constraints in test catalog r=mgartner a=mgartner

This commit fixes a bug in the test catalog which incorrectly
de-duplicated unique constraints that had the same columns but different
predicates.

Fixes #76994

Release note: None

78524: zcfg: rename UpdateZoneConfigForTables to RefreshZoneConfigsForTable r=arulajmani a=RichardJCai

Release note: None

Fixes #77641

78533: ui: don't download .dll.js bundles r=sjbarag a=sjbarag

Loading the CockroachDB admin UI (served from port 8080) would previously download three JS files: `bundle.js`, `vendor.dll.js`, and `protos.dll.js`. Neither `vendor.dll.js` nor `protos.dll.js` need to be included via `<script/>` tag, as they're used only as build-time tooling to accelerate webpack builds. Only a single file -- `bundle.js` -- needs to be loaded for the admin UI to be functional, and Bazel-produced builds don't have any `.dll.js` files to serve. Don't load `.dll.js` files via index.html.

Since dll file aren't being served directly to users anymore, there's no need to embed them in the `cockroach` binary. Stop copying `.dll.js` files into `pkg/ui/dist{oss,ccl}` during builds, reducing the `cockroach` binary size by ~19MB (darwin, x86) when built with GNU make.

78607: sqlsmith: skip crdb_internal.complete_replication_stream r=rytaft a=rytaft

Don't use crdb_internal.complete_replication_stream when generating
sqlsmith queries, since it currently causes a panic.

Informs #78553

Release note: None

78609: cli: de-flake TestDockerCLI/test_demo_node_cmds.tcl r=ajwerner a=tbg

Touches #76391.
(Not closing since `@ajwerner` requested so)

This test is very flaky, so we need a stop-gap. It's also not really useful to get into the gossip info from a cli test.

Release note: None


Co-authored-by: Marcus Gartner <[email protected]>
Co-authored-by: richardjcai <[email protected]>
Co-authored-by: Sean Barag <[email protected]>
Co-authored-by: Rebecca Taft <[email protected]>
Co-authored-by: Tobias Grieger <[email protected]>
@exalate-issue-sync exalate-issue-sync bot removed the T-sql-queries SQL Queries Team label Mar 29, 2022
@amruss
Copy link
Contributor

amruss commented Mar 30, 2022

@rytaft can we close this?

@rytaft
Copy link
Collaborator

rytaft commented Mar 31, 2022

I updated SQLSmith so we won't get more failures like this, but I'm not sure if y'all are concerned about the failures that we saw, panic: UpdateDeadline() called on leaf txn. I would not recommend closing this until you have an idea about what caused those failures and how to avoid them.

@miretskiy
Copy link
Contributor

Seems to be fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc
Projects
None yet
Development

No branches or pull requests

5 participants