Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: sqlsmith/setup=rand-tables/setting=default failed #64180

Closed
cockroach-teamcity opened this issue Apr 24, 2021 · 5 comments · Fixed by #64258
Closed

roachtest: sqlsmith/setup=rand-tables/setting=default failed #64180

cockroach-teamcity opened this issue Apr 24, 2021 · 5 comments · Fixed by #64258
Assignees
@yuzefovich
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone
21.1

Comments

@cockroach-teamcity
Copy link
Member

roachtest.sqlsmith/setup=rand-tables/setting=default failed with artifacts on master @ 3cd29547884feb24b150b67ca2a88073798a35fd:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/sqlsmith/setup=rand-tables/setting=default/run_1
	sqlsmith.go:248,sqlsmith.go:262,test_runner.go:777: ping: dial tcp 34.68.31.39:26257: connect: connection refused
		previous sql:
		DELETE FROM
			defaultdb.public.table3 AS tab_338
		WHERE
			tab_338.col3_2
		ORDER BY
			tab_338.col3_4 ASC, tab_338.col3_4 DESC, tab_338.crdb_internal_mvcc_timestamp DESC
		LIMIT
			49:::INT8;

	cluster.go:1716,context.go:140,cluster.go:1705,test_runner.go:858: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2920165-1619291008-06-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 2: 10939
		3: 10356
		4: 10090
		1: dead
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1162
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:284
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2057
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh sqlsmith/setup=rand-tables/setting=default

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 24, 2021
@cockroach-teamcity cockroach-teamcity added this to the 21.1 milestone Apr 24, 2021
@yuzefovich
Copy link
Member 

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xeef862]

goroutine 2657 [running]:
panic(0x486a0a0, 0xc004292810)
	/usr/local/go/src/runtime/panic.go:1064 +0x545 fp=0xc003d02b60 sp=0xc003d02a98 pc=0x48b725
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).closeWrapper(0xc0024f8000, 0x5b6ba20, 0xc0021b9700, 0x4613c80, 0x7e301a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:814 +0x3a9 fp=0xc003d02be0 sp=0xc003d02b60 pc=0x3537ac9
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1(0xc0024f8000, 0x5b6ba20, 0xc0021b9700)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:482 +0x65 fp=0xc003d02c18 sp=0xc003d02be0 pc=0x370e4a5
runtime.call32(0x0, 0x53eddd8, 0xc0028a85e8, 0x1800000018)
	/usr/local/go/src/runtime/asm_amd64.s:540 +0x3e fp=0xc003d02c48 sp=0xc003d02c18 pc=0x4c30be
runtime.reflectcallSave(0xc003d02d88, 0x53eddd8, 0xc0028a85e8, 0x18)
	/usr/local/go/src/runtime/panic.go:881 +0x58 fp=0xc003d02c78 sp=0xc003d02c48 pc=0x48b118
runtime.runOpenDeferFrame(0xc0005dd680, 0xc0028a85a0, 0x0)
	/usr/local/go/src/runtime/panic.go:855 +0x2cd fp=0xc003d02d08 sp=0xc003d02c78 pc=0x48afcd
panic(0x4613c80, 0x7e301a0)
	/usr/local/go/src/runtime/panic.go:969 +0x1b9 fp=0xc003d02dd0 sp=0xc003d02d08 pc=0x48b399
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1(0xc003d03748)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58 +0x68b fp=0xc003d02f70 sp=0xc003d02dd0 pc=0x1d187cb
runtime.call32(0x0, 0x53eed80, 0xc0010e5448, 0x800000008)
	/usr/local/go/src/runtime/asm_amd64.s:540 +0x3e fp=0xc003d02fa0 sp=0xc003d02f70 pc=0x4c30be
runtime.reflectcallSave(0xc003d030e0, 0x53eed80, 0xc0010e5448, 0xc000000008)
	/usr/local/go/src/runtime/panic.go:881 +0x58 fp=0xc003d02fd0 sp=0xc003d02fa0 pc=0x48b118
runtime.runOpenDeferFrame(0xc0005dd680, 0xc0010e5400, 0xc003d03128)
	/usr/local/go/src/runtime/panic.go:855 +0x2cd fp=0xc003d03060 sp=0xc003d02fd0 pc=0x48afcd
panic(0x4613c80, 0x7e301a0)
	/usr/local/go/src/runtime/panic.go:969 +0x1b9 fp=0xc003d03128 sp=0xc003d03060 pc=0x48b399
runtime.panicmem(...)
	/usr/local/go/src/runtime/panic.go:212
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:742 +0x413 fp=0xc003d03158 sp=0xc003d03128 pc=0x4a2473
github.com/cockroachdb/cockroach/pkg/util/tracing.SpanFromContext(0x0, 0x0, 0xd8)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/context.go:25 +0x22 fp=0xc003d03190 sp=0xc003d03158 pc=0xeef862
github.com/cockroachdb/cockroach/pkg/sql/execinfra.GetTraceData(0x0, 0x0, 0xd8, 0x7f1d1d3be7d0, 0xc0040f2d20)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:243 +0x35 fp=0xc003d031c0 sp=0xc003d03190 pc=0x245c3f5
github.com/cockroachdb/cockroach/pkg/sql/execinfra.GetCumulativeContentionTime(0x0, 0x0, 0xc0011cf450)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/stats.go:35 +0x45 fp=0xc003d03238 sp=0xc003d031c0 pc=0x2465405
github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*ColBatchScan).GetCumulativeContentionTime(0xc0011cf400, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colfetcher/colbatch_scan.go:166 +0x37 fp=0xc003d03260 sp=0xc003d03238 pc=0x337ea57
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedStatsCollectorImpl).GetStats(0xc0040496c0, 0xc0040d3d18)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:188 +0x26d fp=0xc003d032d0 sp=0xc003d03260 pc=0x339dccd
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*drainHelper).Next(0xc002019aa0, 0x12e5496, 0x4ce9dd9, 0x4ce9dd9, 0x5)
...

@yuzefovich yuzefovich self-assigned this Apr 26, 2021
@yuzefovich yuzefovich removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Apr 26, 2021
@rytaft
Copy link
Collaborator

rytaft commented Apr 26, 2021

Is this a duplicate of #64193?

@yuzefovich
Copy link
Member

Yeah, it is, will close all of them together.

@cockroach-teamcity

This comment has been minimized.

Sign in to view
@yuzefovich
Copy link
Member

Second hit is:

panic: runtime error: index out of range [3] with length 3 [recovered]
	panic: runtime error: index out of range [3] with length 3

goroutine 2805 [running]:
panic(0x486a560, 0xc00252b350)
	/usr/local/go/src/runtime/panic.go:1064 +0x545 fp=0xc003d234e0 sp=0xc003d23418 pc=0x48b725
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).closeWrapper(0xc0011c1900, 0x5b6c160, 0xc001beb700, 0x4a40960, 0xc002b171a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:814 +0x3a9 fp=0xc003d23560 sp=0xc003d234e0 pc=0x3538309
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1(0xc0011c1900, 0x5b6c160, 0xc001beb700)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:482 +0x65 fp=0xc003d23598 sp=0xc003d23560 pc=0x370ed85
runtime.call32(0x0, 0x53ee3e0, 0xc000146288, 0x1800000018)
	/usr/local/go/src/runtime/asm_amd64.s:540 +0x3e fp=0xc003d235c8 sp=0xc003d23598 pc=0x4c30be
runtime.reflectcallSave(0xc003d23708, 0x53ee3e0, 0xc000146288, 0x18)
	/usr/local/go/src/runtime/panic.go:881 +0x58 fp=0xc003d235f8 sp=0xc003d235c8 pc=0x48b118
runtime.runOpenDeferFrame(0xc0011c4300, 0xc000146240, 0x0)
	/usr/local/go/src/runtime/panic.go:855 +0x2cd fp=0xc003d23688 sp=0xc003d235f8 pc=0x48afcd
panic(0x4a40960, 0xc002b171a0)
	/usr/local/go/src/runtime/panic.go:969 +0x1b9 fp=0xc003d23750 sp=0xc003d23688 pc=0x48b399
runtime.goPanicIndex(0x3, 0x3)
	/usr/local/go/src/runtime/panic.go:88 +0xa5 fp=0xc003d23798 sp=0xc003d23750 pc=0x489225
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).bufferRow(0xc0010f5600, 0x5b8a620, 0xc0034bee78, 0xc002db9fc0, 0x4, 0x4, 0x0, 0x0, 0x0, 0x200000000, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:1238 +0x3f0 fp=0xc003d23820 sp=0xc003d23798 pc=0x3bfdef0
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*commandResult).AddRow(0xc0010f5c80, 0x5b8a620, 0xc0034bee78, 0xc002db9fc0, 0x4, 0x4, 0x4, 0x3)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/command_result.go:206 +0x19b fp=0xc003d238c0 sp=0xc003d23820 pc=0x3bf675b
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLReceiver).Push(0xc003913880, 0xc002fcc240, 0x4, 0x4, 0x0, 0xc0034bf358)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:740 +0x8f6 fp=0xc003d23a20 sp=0xc003d238c0 pc=0x35bd2b6
github.com/cockroachdb/cockroach/pkg/sql/execinfra.Run(0x5b8a620, 0xc0034bf358, 0x5b90fe0, 0xc003441800, 0x5b34420, 0xc003913880)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:178 +0x5a fp=0xc003d23a60 sp=0xc003d23a20 pc=0x245c8ba
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBase).Run(0xc003441800, 0x5b6c160, 0xc002db92c0)
...

which is a dup of #64181.

@yuzefovich yuzefovich mentioned this issue Apr 27, 2021
Merged
@craig craig bot closed this as completed in 0312ae6 Apr 27, 2021
@mgartner mgartner moved this to Done in SQL Queries Jul 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yuzefovich yuzefovich

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
SQL Queries
Archived in project
Milestone
21.1
Development

Successfully merging a pull request may close this issue.

colbuilder,colflow: fixes around stats collection of uninitialized components yuzefovich/cockroach
3 participants
@rytaft @yuzefovich @cockroach-teamcity