Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rowexec: panic in HashMemRowContainer.AddRowToBucket #45975

Closed
cockroach-teamcity opened this issue Mar 11, 2020 · 8 comments · Fixed by #46162
Closed

rowexec: panic in HashMemRowContainer.AddRowToBucket #45975

cockroach-teamcity opened this issue Mar 11, 2020 · 8 comments · Fixed by #46162
Assignees
@yuzefovich
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Milestone
20.1

Comments

@cockroach-teamcity
Copy link
Member

(roachtest).sqlsmith/setup=seed/setting=no-ddl failed on master@5570c01402796edb7cd06eb8ce7f615371f22d42:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200311-1801614/sqlsmith/setup=seed/setting=no-ddl/run_1
	sqlsmith.go:185,sqlsmith.go:199,test_runner.go:747: ping: dial tcp 35.225.170.171:26257: connect: connection refused
		previous sql:
		INSERT
		INTO
			defaultdb.public.seed AS tab_827
		SELECT
			(-6769):::INT8 AS col_1997,
			(-1885658008):::INT8 AS col_1998,
			((-3617870019796154521):::INT8::INT8 % ((-7829511338012323029):::INT8::INT8 % (999754452227079239:::INT8::INT8 % 1457631285488654207:::INT8::INT8)::INT8::INT8)::INT8::INT8)::INT8
				AS col_1999,
			2.145972490310669:::FLOAT8 AS col_2000,
			(-0.19535191773842797):::FLOAT8 AS col_2001,
			'1981-07-24':::DATE AS col_2002,
			'1970-01-04 21:49:21.000481+00:00':::TIMESTAMP AS col_2003,
			NULL AS col_2004,
			age('0001-01-01 00:00:00+00:00':::TIMESTAMPTZ::TIMESTAMPTZ, '1970-01-10 22:07:11.000203+00:00':::TIMESTAMPTZ::TIMESTAMPTZ)::INTERVAL
				AS col_2005,
			(NOT false) AS col_2006,
			NULL AS col_2007,
			COALESCE('5':::STRING, NULL) AS col_2008,
			'\x18406acd731524e7':::BYTES AS col_2009,
			'b0f2d274-349b-4531-ad83-a3ada0835b0b':::UUID AS col_2010,
			'5020:9e29:e0f:fb13:59e5:29ec:1ed8:3217/57':::INET AS col_2011,
			'{"87/": [], ">]D8FaLq<k": [[null, {}], 1.6002049925945476, [], {}], "`H4duM[": [{"hSdIH<h{O": []}], "b": [[], "#3?~%$>54jh"], "vlEW": null}':::JSONB
				AS col_2012
		LIMIT
			39:::INT8;

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:801: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1801614-1583911542-16-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 2: 6656
		3: 6734
		4: 5083
		1: dead
		Error:  1: dead
More

Artifacts: /sqlsmith/setup=seed/setting=no-ddl

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Mar 11, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.1 milestone Mar 11, 2020
@rohany
Copy link
Contributor

rohany commented Mar 11, 2020

Looks like a node failed with this stack trace:

goroutine 3402 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc002fab3b0 sp=0xc002fab3a8 pc=0x7c7830
runtime.mallocgc(0xe0728000, 0x0, 0x0, 0x1291400)
	/usr/local/go/src/runtime/malloc.go:1032 +0x895 fp=0xc002fab450 sp=0xc002fab3b0 pc=0x7767c5
runtime.growslice(0x3c64080, 0xc2170dc000, 0x1671d800, 0x1671d800, 0x1671d801, 0x0, 0x0, 0x0)
	/usr/local/go/src/runtime/slice.go:175 +0x14e fp=0xc002fab4b8 sp=0xc002fab450 pc=0x7b00ee
github.com/cockroachdb/cockroach/pkg/sql/rowcontainer.(*HashMemRowContainer).addRowToBucket(0xc00515c000, 0x4b546e0, 0xc005ddca00, 0x74be4b8, 0x0, 0x0, 0x1671d800, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowcontainer/hash_row_container.go:262 +0x2dd fp=0xc002fab588 sp=0xc002fab4b8 pc=0x1fad75d
github.com/cockroachdb/cockroach/pkg/sql/rowcontainer.(*HashMemRowContainer).Init(0xc00515c000, 0x4b546e0, 0xc005ddca00, 0x74be400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowcontainer/hash_row_container.go:218 +0x1f5 fp=0xc002fab5f0 sp=0xc002fab588 pc=0x1fad215
github.com/cockroachdb/cockroach/pkg/sql/rowcontainer.(*HashDiskBackedRowContainer).Init(0xc006dcb200, 0x4b546e0, 0xc005ddca00, 0xc004570000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowcontainer/hash_row_container.go:818 +0x269 fp=0xc002fab6a8 sp=0xc002fab5f0 pc=0x1fafca9
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*hashJoiner).initStoredRows(0xc0031b2880, 0xc0031b0d80, 0x2c5f571)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/hashjoiner.go:752 +0x293 fp=0xc002fab778 sp=0xc002fab6a8 pc=0x20db9f3
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*hashJoiner).build.func1(0xc0031b2800, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/hashjoiner.go:274 +0x3e fp=0xc002fab7a8 sp=0xc002fab778 pc=0x211690e
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*hashJoiner).build(0xc0031b2880, 0x0, 0x0, 0x0, 0xc0062c9d50, 0xc000000000)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/hashjoiner.go:329 +0x50f fp=0xc002fab890 sp=0xc002fab7a8 pc=0x20d935f
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*hashJoiner).Next(0xc0031b2880, 0x0, 0x0, 0x0, 0xc0062c9d50)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/hashjoiner.go:235 +0x1bd fp=0xc002fab8f8 sp=0xc002fab890 pc=0x20d8ced
github.com/cockroachdb/cockroach/pkg/sql/execinfra.Run(0x4b546e0, 0xc005ddca00, 0x4b785a0, 0xc0031b2880, 0x4b32c20, 0xc0045ce380)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:170 +0x35 fp=0xc002fab938 sp=0xc002fab8f8 pc=0x20af025
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBase).Run(0xc0031b2880, 0x4b546e0, 0xc005ddca00)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:749 +0x92 fp=0xc002fab978 sp=0xc002fab938 pc=0x20b6ce2
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run(0xc006325e60, 0x4b546e0, 0xc005ddca00, 0x44544e8, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:370 +0x247 fp=0xc002faba28 sp=0xc002fab978 pc=0x20bfb67
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc0003efa40, 0xc0053c06c0, 0xc00531ebd0, 0xc005e3c3b0, 0xc0062fa580, 0xc0058e9b00, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:378 +0x426 fp=0xc002fabef8 sp=0xc002faba28 pc=0x2c5e396
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).planAndRunSubquery(0xc0003efa40, 0x4b547a0, 0xc0063ee960, 0x0, 0x4acef20, 0xc0063baae0, 0x4, 0x1, 0x4b56be0, 0xc0041ed180, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:899 +0x830 fp=0xc002fac498 sp=0xc002fabef8 pc=0x2c60f40
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunSubqueries(0xc0003efa40, 0x4b547a0, 0xc0063ee960, 0xc005e163b8, 0xc002fac6d8, 0xc004ef6e80, 0x2, 0x2, 0xc0047acdc0, 0xc005e16100, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:803 +0x1bd fp=0xc002fac5e0 sp=0xc002fac498 pc=0x2c6067d
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc005e16000, 0x4b547a0, 0xc0063ee960, 0xc005e163b8, 0x2, 0x7fc07a19fe60, 0xc005356f80, 0xc005e3c800, 0xc006461088, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:860 +0x5f1 fp=0xc002fac708 sp=0xc002fac5e0 pc=0x2c08b11
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc005e16000, 0x4b547a0, 0xc0063ee960, 0xc005e163b8, 0x7fc07a19fe60, 0xc005356f80, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:773 +0x6cd fp=0xc002fac9b0 sp=0xc002fac708 pc=0x2c07c5d
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc005e16000, 0x4b547a0, 0xc0063ee960, 0x4b6d8a0, 0xc006460ff0, 0xc003bed052, 0x6d5, 0x0, 0x9, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:467 +0xadf fp=0xc002fad2a0 sp=0xc002fac9b0 pc=0x2c04ccf
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc005e16000, 0x4b547a0, 0xc0063ee960, 0x4b6d8a0, 0xc006460ff0, 0xc003bed052, 0x6d5, 0x0, 0x9, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:117 +0x4f6 fp=0xc002fad550 sp=0xc002fad2a0 pc=0x2c03d16
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc005e16000, 0x4b546e0, 0xc00635ce00, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1394 +0x1aa8 fp=0xc002fadce8 sp=0xc002fad550 pc=0x2bfa7d8
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc005e16000, 0x4b546e0, 0xc005aae5c0, 0xc000107c00, 0x5400, 0x15000, 0xc000107c98, 0xc0055abf30, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1323 +0x1f2 fp=0xc002fadd88 sp=0xc002fadce8 pc=0x2bf8ae2
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc000f5a000, 0x4b546e0, 0xc005aae5c0, 0xc005e16000, 0x5400, 0x15000, 0xc000107c98, 0xc0055abf30, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:478 +0x104 fp=0xc002fade30 sp=0xc002fadd88 pc=0x2bf5024
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc005855546, 0xc005a58700, 0x4b546e0, 0xc005aae5c0, 0xc0055abf30, 0xc000f5a000, 0xc005356900, 0x4b781a0, 0xc005ab0d50, 0xc005a77a40, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:594 +0x2e8 fp=0xc002fadf60 sp=0xc002fade30 pc=0x2fe6358
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc002fadf68 sp=0xc002fadf60 pc=0x7c9901
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:526 +0x17e

@yuzefovich does this look familiar at all?

@rohany rohany changed the title roachtest: sqlsmith/setup=seed/setting=no-ddl failed rowexec: panic in HashMemRowContainer.AddRowToBucket Mar 11, 2020
@yuzefovich
Copy link
Member

Haven't seen this before, looks like memory allocation failed. Yeah, fatal error: runtime: out of memory which probably means that our memory accounting is off.

@rohany
Copy link
Contributor

rohany commented Mar 12, 2020 •
edited
Loading

#45812 has a failure with a similar stack

goroutine 324062 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc01ec377d0 sp=0xc01ec377c8 pc=0x7c8970
runtime.mallocgc(0x59400000, 0x3fb7fe0, 0xc0a23ea701, 0xc01ec378a8)
	/usr/local/go/src/runtime/malloc.go:1032 +0x895 fp=0xc01ec37870 sp=0xc01ec377d0 pc=0x777905
runtime.newarray(0x3fb7fe0, 0x440000, 0x20b2813)
	/usr/local/go/src/runtime/malloc.go:1173 +0x63 fp=0xc01ec378a0 sp=0xc01ec37870 pc=0x777d43
runtime.makeBucketArray(0x3d413c0, 0x16, 0x0, 0x4b8c320, 0xc0a23ea7c0)
	/usr/local/go/src/runtime/map.go:362 +0x183 fp=0xc01ec378d8 sp=0xc01ec378a0 pc=0x778c33
runtime.hashGrow(0x3d413c0, 0xc188939a10)
	/usr/local/go/src/runtime/map.go:1033 +0x89 fp=0xc01ec37928 sp=0xc01ec378d8 pc=0x77a8a9
runtime.mapassign_faststr(0x3d413c0, 0xc188939a10, 0xc0a22e3f6e, 0x7, 0x0)
	/usr/local/go/src/runtime/map_faststr.go:272 +0x122 fp=0xc01ec37990 sp=0xc01ec37928 pc=0x77e2e2
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*hashAggregator).accumulateRow(0xc0a6688400, 0xc01af82a48, 0x3, 0x3, 0x1, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/aggregator.go:782 +0x2e9 fp=0xc01ec37a40 sp=0xc01ec37990 pc=0x20db909
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*hashAggregator).accumulateRows(0xc0a6688400, 0x1, 0x0, 0x0, 0x0, 0xc013335ab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/aggregator.go:446 +0x244 fp=0xc01ec37b90 sp=0xc01ec37a40 pc=0x20d9044
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*hashAggregator).Next(0xc0a6688400, 0x0, 0x0, 0x7fe59934abf0, 0xc013335ab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/aggregator.go:656 +0x14c fp=0xc01ec37bf8 sp=0xc01ec37b90 pc=0x20dacac
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*hashJoiner).receiveNext(0xc005f93100, 0x0, 0x0, 0x0, 0x0, 0xc013335ab0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/hashjoiner.go:654 +0x194 fp=0xc01ec37c90 sp=0xc01ec37bf8 pc=0x20e6e24
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*hashJoiner).build(0xc005f93100, 0x0, 0x0, 0x0, 0xc013335ab0, 0xc000000000)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/hashjoiner.go:301 +0xb3 fp=0xc01ec37d78 sp=0xc01ec37c90 pc=0x20e4923
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*hashJoiner).Next(0xc005f93100, 0x0, 0x0, 0x0, 0xc013335ab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/hashjoiner.go:235 +0x1bd fp=0xc01ec37de0 sp=0xc01ec37d78 pc=0x20e470d
github.com/cockroachdb/cockroach/pkg/sql/execinfra.Run(0x4b67a20, 0xc0b092db00, 0x4b8b960, 0xc005f93100, 0x4b45f20, 0xc00e337880)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:170 +0x35 fp=0xc01ec37e20 sp=0xc01ec37de0 pc=0x20baa45
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBase).Run(0xc005f93100, 0x4b67a20, 0xc0b092db00)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:749 +0x92 fp=0xc01ec37e60 sp=0xc01ec37e20 pc=0x20c2702
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run(0xc00d685c20, 0x4b67a20, 0xc0b092db00, 0x4466260, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:370 +0x247 fp=0xc01ec37f10 sp=0xc01ec37e60 pc=0x20cb587
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc0007b0780, 0xc019c40ea0, 0xc008ae6000, 0xc0033004f8, 0xc0073e5080, 0xc013341498, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:378 +0x426 fp=0xc01ec383e0 sp=0xc01ec37f10 pc=0x2c6a6a6
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc0007b0780, 0x4b67ae0, 0xc187fff440, 0xc013341498, 0xc019c40ea0, 0xc008ae6000, 0x4b6a4a0, 0xc01ac34fa0, 0xc0073e5080, 0xc0133411b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:988 +0x225 fp=0xc01ec385e0 sp=0xc01ec383e0 pc=0x2c6de85
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc013341000, 0x4b67ae0, 0xc187fff440, 0xc0133413b8, 0x3, 0x7fe5c0fb3c50, 0xc003c18680, 0xc003300801, 0xc01ac34a98, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:869 +0x3cf fp=0xc01ec38708 sp=0xc01ec385e0 pc=0x2c146bf
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc013341000, 0x4b67ae0, 0xc187fff440, 0xc0133413b8, 0x7fe5c0fb3c50, 0xc003c18680, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:773 +0x6cd fp=0xc01ec389b0 sp=0xc01ec38708 pc=0x2c13a2d
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc013341000, 0x4b67ae0, 0xc187fff440, 0x4b812a0, 0xc01ac34a00, 0xc0041f927f, 0xf0, 0x0, 0x2, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:467 +0xadf fp=0xc01ec392a0 sp=0xc01ec389b0 pc=0x2c10a9f
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc013341000, 0x4b67ae0, 0xc187fff440, 0x4b812a0, 0xc01ac34a00, 0xc0041f927f, 0xf0, 0x0, 0x2, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:117 +0x4f6 fp=0xc01ec39550 sp=0xc01ec392a0 pc=0x2c0fae6
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc013341000, 0x4b67a20, 0xc00cf6a000, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1394 +0x1aa8 fp=0xc01ec39ce8 sp=0xc01ec39550 pc=0x2c065a8
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc013341000, 0x4b67a20, 0xc013742100, 0xc0005d5c00, 0x5400, 0x15000, 0xc0005d5c98, 0xc01fbea4e0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1323 +0x1f2 fp=0xc01ec39d88 sp=0xc01ec39ce8 pc=0x2c048b2
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc000d49180, 0x4b67a20, 0xc013742100, 0xc013341000, 0x5400, 0x15000, 0xc0005d5c98, 0xc01fbea4e0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:478 +0x104 fp=0xc01ec39e30 sp=0xc01ec39d88 pc=0x2c00df4
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc01e3402c1, 0xc00626fd60, 0x4b67a20, 0xc013742100, 0xc01fbea4e0, 0xc000d49180, 0xc003c18000, 0x4b8b560, 0xc0149d1800, 0xc0167f0120, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:594 +0x2e8 fp=0xc01ec39f60 sp=0xc01ec39e30 pc=0x2ff4408
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc01ec39f68 sp=0xc01ec39f60 pc=0x7caa41
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:526 +0x17e

@jordanlewis jordanlewis mentioned this issue Mar 12, 2020
Closed
83 tasks
@yuzefovich
Copy link
Member

Huh, why would we see a hash aggregator on tpcc workload?

@rohany
Copy link
Contributor

rohany commented Mar 12, 2020

It failed during the validation phase, not during the workload

@yuzefovich
Copy link
Member

Alright, I'll try to look into this later today.

@yuzefovich yuzefovich self-assigned this Mar 12, 2020
@yuzefovich
Copy link
Member

cc @jordanlewis

@rohany rohany mentioned this issue Mar 15, 2020
Closed
@yuzefovich yuzefovich mentioned this issue Mar 16, 2020
Merged
@yuzefovich
Copy link
Member

yuzefovich commented Mar 16, 2020 •
edited
Loading

I found a couple of things that we were not accounting for in the hash aggregation and opened up a PR to fix those. Running a CI now to check that I didn't break anything in the process :)

With that commit I was able to run EXPLAIN ANALYZE on the query from 3.3.2.6 check on 3 node roachprod cluster (without the commit one node OOMed), so I'm hopeful that the issue is getting fixed.

@craig craig bot closed this as completed in 47e5748 Mar 18, 2020
@yuzefovich yuzefovich mentioned this issue Mar 24, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yuzefovich yuzefovich

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Projects
None yet
Milestone
20.1
Development

Successfully merging a pull request may close this issue.

rowexec: fix some issues with aggregation yuzefovich/cockroach
4 participants
@andreimatei @yuzefovich @rohany @cockroach-teamcity