roachtest: timeouts due to missing cancellation checks (causes costfuzz failure)

[cockroach-teamcity]

roachtest.costfuzz failed with artifacts on master @ 2372698da1dfacb90f60c6a63f2c1298d1db16b8:

test artifacts and logs in: /artifacts/costfuzz/run_1
	test_runner.go:1028,test_runner.go:927: test timed out (1h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

• roachtest.costfuzz failed due to scientific notation v. decimal notation #86796 roachtest.costfuzz failed due to scientific notation v. decimal notation [C-test-failure T-sql-queries]

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-19365

[DrewKimball]

Heap profile doesn't look suspicious. Here's the cpu profile:

Maybe it would be a good idea to add some cancel-checking in the diskqueue methods?

[yuzefovich]

it would be a good idea to add some cancel-checking in the diskqueue methods?

Yes, I think so. I actually just ran into the missing cancellation checks in the external sorter when debugging #87510 where the external seems to proceed even after the connection that needed it is closed. We should also backport those cancel checks.

[DrewKimball]

I think we may have to add cancel checking for all operators that buffer before performing a lot of work, specifically in the loops that process the buffered data.

[cockroach-teamcity]

roachtest.costfuzz failed with artifacts on master @ e39111b2e714375faa0facc05e51e8f619a55b21:

		  |   	strings.Join({
		  |   		"-522806371908919723000000000000000.057361179212762389",
		  | - 		"0",
		  |   		",true",
		  |   	}, ""),
		  |   	"-5281573567465919405000050343168773.00290717,true",
		  |   	"-53008036059.78642014,true",
		  |   	"-53012568.62456803803,true",
		  |   	strings.Join({
		  |   		"-53930584384214236340831128.935",
		  | - 		"0",
		  |   		",true",
		  |   	}, ""),
		  | - 	"-54151595530771640900000000000000,true",
		  |   	"-553376140143201345099999531541229259.7143749,true",
		  |   	"-58828056393909638100000000010.1,true",
		  |   	"-598243635.3590026986,true",
		  |   	"-6.5336706868357837291E+37,true",
		  | + 	"-6.720430994736931567E+36,true",
		  |   	"-60667836961626125801.3612886988,true",
		  |   	"-625381854099381688.1239,true",
		  |   	"-63683.76094893934931,true",
		  |   	"-6601987234273.212696,true",
		  |   	"-6704655693005500528.2,true",
		  | - 	"-6720430994736931567000000000000000000,true",
		  |   	"-68479622903227596.51,true",
		  |   	strings.Join({
		  |   		"-68930104522260656.56688190538679",
		  | - 		"0",
		  |   		",true",
		  |   	}, ""),
		  | - 	"-716364259164750370200000000000000000,true",
		  | + 	"-7.163642591647503702E+35,true",
		  |   	... // 192 identical, 1 removed, 1 inserted, and 16 modified elements
		  |   }
		  | sql: SELECT
		  | 	tab_556.col2_13 AS col_1555, true AS col_1556
		  | FROM
		  | 	defaultdb.public.table2@[0] AS tab_556
		  | GROUP BY
		  | 	tab_556.col2_5, tab_556.col2_13
		  | ORDER BY
		  | 	tab_556.col2_5,
		  | 	tab_556.col2_5 ASC,
		  | 	tab_556.col2_5,
		  | 	tab_556.col2_13 ASC,
		  | 	tab_556.col2_13 DESC,
		  | 	tab_556.col2_13,
		  | 	tab_556.col2_5 ASC
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

• roachtest.costfuzz failed due to scientific notation v. decimal notation #86796 roachtest.costfuzz failed due to scientific notation v. decimal notation [C-test-failure T-sql-queries]

This test on roachdash | Improve this report!

[DrewKimball]

More decimal precision/format trouble, but the sorting looks suspicious; look at this snippet:

		  |   	"-6.5336706868357837291E+37,true",
		  | + 	"-6.720430994736931567E+36,true",
		  |   	"-60667836961626125801.3612886988,true",
		  |   	"-625381854099381688.1239,true",
		  |   	"-63683.76094893934931,true",
		  |   	"-6601987234273.212696,true",
		  |   	"-6704655693005500528.2,true",
		  | - 	"-6720430994736931567000000000000000000,true",
		  |   	"-68479622903227596.51,true",

[mgartner]

@yuzefovich @DrewKimball Is the timeout a release blocker, i.e., a regression?

[yuzefovich]

Missing cancellation checks is not a release blocker - it's a long standing problem, removing the label.

[mgartner]

Thanks!

[mgartner]

I created #87919 to track the last failure (hidden now). It looks like an actual correctness bug, not a flake.

[DrewKimball]

Most recent failure is #87919