pkg/sql/schemachanger/schemachanger_test: TestValidateMixedVersionElements_drop_column_computed_index failed

[cockroach-teamcity]

pkg/sql/schemachanger/schemachanger_test.TestValidateMixedVersionElements_drop_column_computed_index failed with artifacts on master @ bdaae6458a2388da03adbb979a25b5a07ea6edd9:

        	            				GOROOT/src/runtime/asm_arm64.s:1172
        	Error:      	Received unexpected error:
        	            	dial tcp 127.0.0.1:38811: connect: connection refused
        	Test:       	TestValidateMixedVersionElements_drop_column_computed_index/pause_upgrade_and_resume_at_each_stage/PostCommitPhase_1_of_7
        --- FAIL: TestValidateMixedVersionElements_drop_column_computed_index/pause_upgrade_and_resume_at_each_stage/PostCommitPhase_1_of_7 (48.06s)
=== RUN   TestValidateMixedVersionElements_drop_column_computed_index/pause_upgrade_and_resume_at_each_stage/PostCommitPhase_4_of_7
    sql_runner.go:104: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:117
        	            				github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:312
        	            				github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/end_to_end.go:488
        	            				github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1632
        	            				github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1729
        	Error:      	Received unexpected error:
        	            	query 'SELECT status, job_type, description FROM [SHOW JOBS] WHERE job_type IN ('SCHEMA CHANGE', 'TYPEDESC SCHEMA CHANGE', 'NEW SCHEMA CHANGE') AND status NOT IN ('succeeded', 'failed')': expected:
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:315
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
        	            	  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).succeedsWithin
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:117
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:312
        	            	  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest.waitForSchemaChangesToFinish
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/end_to_end.go:488
        	            	  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest.executeSchemaChangeTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1632
        	            	  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest.ValidateMixedVersionElements.func5.1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1729
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1446
        	            	  | runtime.goexit
        	            	  | 	GOROOT/src/runtime/asm_arm64.s:1172
        	            	Wraps: (2) query 'SELECT status, job_type, description FROM [SHOW JOBS] WHERE job_type IN ('SCHEMA CHANGE', 'TYPEDESC SCHEMA CHANGE', 'NEW SCHEMA CHANGE') AND status NOT IN ('succeeded', 'failed')': expected:
        	            	  |
        	            	  | got:
        	            	  | paused, NEW SCHEMA CHANGE, ALTER TABLE defaultdb.public.t DROP COLUMN j CASCADE
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
    cumulative.go:1724: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1724
        	            				GOROOT/src/runtime/asm_arm64.s:1172
        	Error:      	Received unexpected error:
        	            	dial tcp 127.0.0.1:33463: connect: connection refused
        	Test:       	TestValidateMixedVersionElements_drop_column_computed_index/pause_upgrade_and_resume_at_each_stage/PostCommitPhase_4_of_7
        --- FAIL: TestValidateMixedVersionElements_drop_column_computed_index/pause_upgrade_and_resume_at_each_stage/PostCommitPhase_4_of_7 (47.51s)

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-29642

[rafiss]

cc @chengxiong-ruan the connect: connection refused error is back. Do you know if this relates to liveness?

[chengxiong-ruan]

I still have no luck on reproducing this... but we could try to lower the liveness heartbeat interval and increase the liveness TTL and how that would help.

[chengxiong-ruan]

Looks like we can clam victory on this. It's been a week... (hope it's not too early!)