Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql/tests: TestRandomSyntaxSchemaChangeColumn failed #63944

Closed
cockroach-teamcity opened this issue Apr 20, 2021 · 31 comments
Closed

sql/tests: TestRandomSyntaxSchemaChangeColumn failed #63944

cockroach-teamcity opened this issue Apr 20, 2021 · 31 comments
Assignees
@fqazi
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

(sql/tests).TestRandomSyntaxSchemaChangeColumn failed on release-21.1@0eb5ba6e6c433f57ed1851a0c160f7739e7a7725:

        	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2306 +0x1eb
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00baea300, 0x5afec80, 0xc00e2026c0, 0x0, 0xc015b54620)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8339352 [chan receive, 5 minutes]:
        google.golang.org/grpc.(*addrConn).resetTransport(0xc02542d080)
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1156 +0x769
        created by google.golang.org/grpc.(*addrConn).connect
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:799 +0x12a
        
        goroutine 8339419 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc00fb78110, 0x2835)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc00fb78100)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc00ed4c0a0, 0x5afec80, 0xc00fb62fc0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00baea300, 0x5afec80, 0xc00fb62fc0, 0x0, 0xc01377dbf0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8339262 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc00fb78110, 0x2840)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc00fb78100)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc00ed4c0a0, 0x5afec80, 0xc00fb62fc0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00baea300, 0x5afec80, 0xc00fb62fc0, 0x0, 0xc01377d920)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident DROP COLUMN EXISTS RESTRICT
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident DROP COLUMN EXISTS RESTRICT". currently executing: map[ALTER TABLE ident.ident DROP COLUMN EXISTS RESTRICT:1]
    rsg_test.go:764: 4840 executions, 902 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn397845159
--- FAIL: TestRandomSyntaxSchemaChangeColumn (356.71s)
More

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-release-21.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 20, 2021
@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 20, 2021
Closed
@rafiss rafiss unassigned tbg Apr 20, 2021
@rafiss
Copy link
Collaborator

rafiss commented Apr 20, 2021

something similar happens on master: #63393

@cockroach-teamcity
Copy link
Member Author

(sql/tests).TestRandomSyntaxSchemaChangeColumn failed on release-21.1@f3b14a95b781f57ac53a472b9979270b0ecbedfb:

        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8542860 [select]:
        github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x5afee80, 0xc00fae9b30)
        	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1374 +0x345
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc001a58200, 0x5afee80, 0xc00fae9b30, 0x0, 0xc01d387fc0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8543834 [select]:
        github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1(0x5afee80, 0xc00f68a600)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:151 +0x2e9
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc001a58200, 0x5afee80, 0xc00f68a600, 0x0, 0xc0222fd1e0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8605711 [select]:
        github.com/cockroachdb/cockroach/pkg/util/retry.(*Retry).Next(0xc0029a5378, 0x1)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:128 +0x187
        github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume.func1(0xab00000035, 0xc000000000, 0xc0082f5ea0, 0x5b635a0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2158 +0x5f3
        github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume(0xc00ff44280, 0x5afee80, 0xc009f25950, 0x493b800, 0xc00f00a270, 0xc00ff44280, 0xc00497f838)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2279 +0x803
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1(0xc0175aa040, 0x5aa7700, 0xc00f00a260, 0x5afee80, 0xc009f25950, 0x493b800, 0xc00f00a270, 0xc0029a5a50)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1193 +0xae
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc01e4aeb60, 0x5afee80, 0xc009f25890, 0x493b800, 0xc00f00a270, 0x5aa7700, 0xc00f00a260, 0xc00ff44280, 0xc00fa22a60, 0x7, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1194 +0x885
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc01e4aeb60, 0x5afedc0, 0xc01396e040, 0x5aa7700, 0xc00f00a260, 0xc00ff44280, 0xc00fa22a60, 0x7, 0xc019203ae0, 0x16, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:255 +0x378
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1(0x5afedc0, 0xc010086700)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:214 +0x165
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc001a58200, 0x5afedc0, 0xc010086700, 0x0, 0xc0083a7540)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ADD IF NOT EXISTS SURVIVAL VARBIT ( 434 ) ARRAY
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ADD IF NOT EXISTS SURVIVAL VARBIT ( 434 ) ARRAY". currently executing: map[ALTER TABLE ident.ident ADD IF NOT EXISTS SURVIVAL VARBIT ( 434 ) ARRAY:1]
    rsg_test.go:764: 6188 executions, 1179 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn733038098
--- FAIL: TestRandomSyntaxSchemaChangeColumn (342.86s)
More

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 21, 2021
Closed
@cockroach-teamcity
Copy link
Member Author

(sql/tests).TestRandomSyntaxSchemaChangeColumn failed on release-21.1@addc4bbefd58089df63fe4d3c930cd9ac6a4b186:

        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1194 +0x885
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc005870d00, 0x5b11520, 0xc00c891900, 0x5ab9d60, 0xc00f387b70, 0xc00a473680, 0xc009eda6d0, 0x7, 0xc00cf63e20, 0x16, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:255 +0x378
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1(0x5b11520, 0xc0122e0540)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:214 +0x165
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00fc7e280, 0x5b11520, 0xc0122e0540, 0x0, 0xc007f934a0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8340218 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc000b90ad0, 0x29ae)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc000b90ac0)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0125be140, 0x5b115e0, 0xc00eb7d860)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00fc7e280, 0x5b115e0, 0xc00eb7d860, 0x0, 0xc004309f10)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8346709 [chan receive, 5 minutes]:
        google.golang.org/grpc.(*addrConn).resetTransport(0xc0057c6b00)
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1156 +0x769
        created by google.golang.org/grpc.(*addrConn).connect
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:799 +0x12a
        
        goroutine 8340059 [chan receive, 5 minutes]:
        github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x5b115e0, 0xc00eb7c210)
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
        runtime/pprof.Do(0x5b115e0, 0xc00eb7c210, 0xc000110ec0, 0x1, 0x1, 0xc0048d1fb8)
        	/usr/local/go/src/runtime/pprof/runtime.go:40 +0xcc
        github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc0028c8800)
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
        created by github.com/cockroachdb/pebble.(*tableCacheShard).init
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ADD COLUMN ident CHARACTER VARYING ( 25 ) ARRAY CONSTRAINT IMMEDIATE NULL
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ADD COLUMN ident CHARACTER VARYING ( 25 ) ARRAY CONSTRAINT IMMEDIATE NULL". currently executing: map[ALTER TABLE ident.ident ADD COLUMN ident CHARACTER VARYING ( 25 ) ARRAY CONSTRAINT IMMEDIATE NULL:1]
    rsg_test.go:764: 5391 executions, 1019 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn378718122
--- FAIL: TestRandomSyntaxSchemaChangeColumn (357.32s)
More

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(sql/tests).TestRandomSyntaxSchemaChangeColumn failed on release-21.1@053ecdd8309001f96a0ec59fce2468846e20e1e0:

        created by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed
        	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:574 +0x11e
        
        goroutine 8500787 [select]:
        github.com/cockroachdb/cockroach/pkg/util/retry.(*Retry).Next(0xc012bb1508, 0xc000000003)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:128 +0x187
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).WaitForJobs(0xc010da96c0, 0x5b115a0, 0xc007b20c80, 0x5b7d400, 0xc00c5fd440, 0xc00e946430, 0x1, 0x1, 0x1, 0x1)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:328 +0x2fb
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Run(0xc010da96c0, 0x5b115a0, 0xc007b20c80, 0x5b7d400, 0xc00c5fd440, 0xc00e946430, 0x1, 0x1, 0x7f80ae8, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:383 +0x16f
        github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).txnStateTransitionsApplyWrapper(0xc001f52600, 0x5a75880, 0x7f80ae8, 0x0, 0x0, 0x7fe3733c6158, 0xc000defc80, 0xaea, 0x0, 0x0, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2379 +0x765
        github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc001f52600, 0x5b115a0, 0xc01ad61a80, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1661 +0xb77
        github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc001f52600, 0x5b115a0, 0xc007b20c80, 0xc0134d4640, 0x5400, 0x15000, 0xc0134d46e0, 0xc006caedf0, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1438 +0x228
        github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc001189700, 0x5b115a0, 0xc007b20c80, 0xc001f52600, 0x5400, 0x15000, 0xc0134d46e0, 0xc006caedf0, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:482 +0xce
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc013cb96ef, 0xc02487cc60, 0x5b115a0, 0xc007b20c80, 0xc006caedf0, 0xc001189700, 0xc000def600, 0x5b78a20, 0xc0148c7000, 0xc0158b6fc0, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:625 +0x3ea
        created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:553 +0x185
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ALTER MULTILINESTRINGZM DROP NOT NULL
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ALTER MULTILINESTRINGZM DROP NOT NULL". currently executing: map[ALTER TABLE ident.ident ADD COLUMN IFNULL BIT VARYING ( 45 ) ARRAY CREATE FAMILY:1 ALTER TABLE ident.ident ADD COLUMN STRING INT ARRAY [ 12 ]:1 ALTER TABLE ident.ident ADD COLUMN ident SMALLINT:1 ALTER TABLE ident.ident ALTER MULTILINESTRINGZM DROP NOT NULL:1 ALTER TABLE ident.ident ALTER SUBSTRING SET NOT NULL:1 ALTER TABLE ident.ident DROP COLUMN IF EXISTS IFNULL RESTRICT:1 ALTER TABLE ident.ident DROP COLUMN LANGUAGE CASCADE:1 ALTER TABLE ident.ident DROP IF EXISTS FUNCTION:1]
    rsg_test.go:731: 4m15s of 5m0s: 8122 executions, 1587 successful
    rsg_test.go:731: 4m20s of 5m0s: 8154 executions, 1596 successful
    rsg_test.go:731: 4m25s of 5m0s: 8304 executions, 1618 successful
    rsg_test.go:731: 4m30s of 5m0s: 8311 executions, 1620 successful
    rsg_test.go:731: 4m35s of 5m0s: 8365 executions, 1626 successful
    rsg_test.go:731: 4m40s of 5m0s: 8365 executions, 1626 successful
    rsg_test.go:731: 4m45s of 5m0s: 8365 executions, 1626 successful
    rsg_test.go:731: 4m50s of 5m0s: 8366 executions, 1627 successful
    rsg_test.go:731: 4m55s of 5m0s: 8391 executions, 1630 successful
    rsg_test.go:731: 5m0s of 5m0s: 8425 executions, 1636 successful
    rsg_test.go:731: 5m5s of 5m0s: 8427 executions, 1638 successful
    rsg_test.go:731: 5m10s of 5m0s: 8428 executions, 1639 successful
    rsg_test.go:731: 5m15s of 5m0s: 8429 executions, 1640 successful
    rsg_test.go:731: 5m20s of 5m0s: 8429 executions, 1640 successful
    rsg_test.go:731: 5m25s of 5m0s: 8429 executions, 1640 successful
    rsg_test.go:731: 5m30s of 5m0s: 8430 executions, 1641 successful
    rsg_test.go:731: 5m35s of 5m0s: 8431 executions, 1642 successful
    rsg_test.go:764: 8433 executions, 1644 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn501245122
--- FAIL: TestRandomSyntaxSchemaChangeColumn (339.50s)
More

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(sql/tests).TestRandomSyntaxSchemaChangeColumn failed on release-21.1@affd857ffc05591f5898701690b8a9136859ab1b:

        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
        created by github.com/cockroachdb/pebble.(*tableCacheShard).init
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5
        
        goroutine 8291189 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc0024821d0, 0x2b15)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc0024821c0)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0144980a0, 0x5b11660, 0xc0145eac90)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc002358200, 0x5b11660, 0xc0145eac90, 0x0, 0xc0078e7cd0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8291154 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc0024821d0, 0x2b06)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc0024821c0)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0144980a0, 0x5b11660, 0xc0145eac90)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc002358200, 0x5b11660, 0xc0145eac90, 0x0, 0xc0078e7aa0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8291900 [select]:
        github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*Sender).Run.func2(0x5b11660, 0xc0117b6660)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport/sender.go:253 +0x1e5
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc002358200, 0x5b11660, 0xc0117b6660, 0x0, 0xc010adb9e0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ADD ALTER INTERVAL DAY
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ADD ALTER INTERVAL DAY". currently executing: map[ALTER TABLE ident.ident ADD ALTER INTERVAL DAY:1]
    rsg_test.go:764: 5193 executions, 939 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn685700363
--- FAIL: TestRandomSyntaxSchemaChangeColumn (356.71s)
More

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(sql/tests).TestRandomSyntaxSchemaChangeColumn failed on release-21.1@3447b13373c947d4b70d33dacc3eda7e5c15e753:

        	/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
        net.(*netFD).Read(0xc00bcdfe00, 0xc00c57a000, 0x731, 0x731, 0x203008, 0x5a73280, 0x7f802a0)
        	/usr/local/go/src/net/fd_posix.go:55 +0x4f
        net.(*conn).Read(0xc01d623948, 0xc00c57a000, 0x731, 0x731, 0x0, 0x0, 0x0)
        	/usr/local/go/src/net/net.go:182 +0x8e
        crypto/tls.(*atLeastReader).Read(0xc02116f480, 0xc00c57a000, 0x731, 0x731, 0x0, 0x5a73500, 0xc001af0fb8)
        	/usr/local/go/src/crypto/tls/conn.go:779 +0x62
        bytes.(*Buffer).ReadFrom(0xc00c3be600, 0x5a6a460, 0xc02116f480, 0x450245, 0x4631900, 0x4bc0520)
        	/usr/local/go/src/bytes/buffer.go:204 +0xb1
        crypto/tls.(*Conn).readFromUntil(0xc00c3be380, 0x5a73560, 0xc01d623948, 0x5, 0xc01d623948, 0xc0075d0101)
        	/usr/local/go/src/crypto/tls/conn.go:801 +0xf3
        crypto/tls.(*Conn).readRecordOrCCS(0xc00c3be380, 0x0, 0x0, 0x6e81d1)
        	/usr/local/go/src/crypto/tls/conn.go:608 +0x115
        crypto/tls.(*Conn).readRecord(...)
        	/usr/local/go/src/crypto/tls/conn.go:576
        crypto/tls.(*Conn).Read(0xc00c3be380, 0xc00c91a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/conn.go:1252 +0x15f
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*readTimeoutConn).Read(0xc01d00b7a0, 0xc00c91a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:1672 +0x129
        bufio.(*Reader).fill(0xc00c524948)
        	/usr/local/go/src/bufio/bufio.go:101 +0x105
        bufio.(*Reader).ReadByte(0xc00c524948, 0x10d4ae65, 0xed81901fe, 0x0)
        	/usr/local/go/src/bufio/bufio.go:253 +0x39
        github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase.(*ReadBuffer).ReadTypedMsg(0xc00c525108, 0x5acb160, 0xc00c524948, 0x5ac0c60, 0x7f80ae8, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase/encoding.go:203 +0x35
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl.func4(0xc00c524900, 0xc01cb414d0, 0xc000cfe100, 0xc001af1815, 0x5add620, 0xc00c2cbc00, 0xc001af1870, 0xc001af1814, 0xc009916500, 0x0, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:327 +0x71
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl(0xc00c524900, 0x5b11720, 0xc0104009c0, 0xc01cb414c0, 0xc000cfe100, 0x5400, 0x15000, 0xc010eca640, 0x0, 0x4, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:458 +0x565
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveConn(0xc008cdab00, 0x5b11720, 0xc0104009c0, 0x5b757c0, 0xc00c3be380, 0xc00c57c02d, 0x4, 0xc010400990, 0x5abe4a0, 0xc0109dd020, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:152 +0x1b8
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc008cdab00, 0x5b11720, 0xc0104009c0, 0x5b79660, 0xc01d623948, 0x5b11701, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:658 +0xc18
        github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).startServeSQL.func1.1(0x5b79660, 0xc01d623948)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2080 +0xf7
        github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1(0xc00e608200, 0x5b11720, 0xc00423eab0, 0xc01bd5a248, 0x5b79660, 0xc01d623948, 0xc01241e9f0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:156 +0xaf
        created by github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:153 +0x24a
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ALTER COLUMN ident SET NOT NULL
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ALTER COLUMN ident SET NOT NULL". currently executing: map[ALTER TABLE ident.ident ALTER COLUMN ident SET NOT NULL:1]
    rsg_test.go:764: 9786 executions, 1936 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn357684128
--- FAIL: TestRandomSyntaxSchemaChangeColumn (341.91s)
More

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 44a4b10cabb511842a9fe30285b9f7d1041870a1:

        	/usr/local/go/src/net/pipe.go:159 +0x1a5
        net.(*pipe).Read(0xc02a748100, 0xc00acb2000, 0x8b5, 0x8b5, 0xc00acb2000, 0x5, 0x7f1fd455c7f0)
        	/usr/local/go/src/net/pipe.go:142 +0x4d
        crypto/tls.(*atLeastReader).Read(0xc01aac8a80, 0xc00acb2000, 0x8b5, 0x8b5, 0x22, 0x8b0, 0xc0000ab7f0)
        	/usr/local/go/src/crypto/tls/conn.go:779 +0x62
        bytes.(*Buffer).ReadFrom(0xc02cef4980, 0x5a6e300, 0xc01aac8a80, 0x450245, 0x4634a60, 0x4a1ebe0)
        	/usr/local/go/src/bytes/buffer.go:204 +0xb1
        crypto/tls.(*Conn).readFromUntil(0xc02cef4700, 0x7f1fd8b93c40, 0xc02a748100, 0x5, 0xc02a748100, 0x11)
        	/usr/local/go/src/crypto/tls/conn.go:801 +0xf3
        crypto/tls.(*Conn).readRecordOrCCS(0xc02cef4700, 0x0, 0x0, 0xc0ca3e)
        	/usr/local/go/src/crypto/tls/conn.go:608 +0x115
        crypto/tls.(*Conn).readRecord(...)
        	/usr/local/go/src/crypto/tls/conn.go:576
        crypto/tls.(*Conn).Read(0xc02cef4700, 0xc002010000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/conn.go:1252 +0x15f
        bufio.(*Reader).Read(0xc02a2b7020, 0xc01b8c5b58, 0x9, 0x9, 0xc001d46480, 0xc0000abe08, 0x44a7a9)
        	/usr/local/go/src/bufio/bufio.go:227 +0x222
        io.ReadAtLeast(0x5a6e0c0, 0xc02a2b7020, 0xc01b8c5b58, 0x9, 0x9, 0x9, 0xbe0045, 0x4a9b20, 0xc0291ca780)
        	/usr/local/go/src/io/io.go:314 +0x87
        io.ReadFull(...)
        	/usr/local/go/src/io/io.go:333
        golang.org/x/net/http2.readFrameHeader(0xc01b8c5b58, 0x9, 0x9, 0x5a6e0c0, 0xc02a2b7020, 0x0, 0xc000000000, 0xc0000abe01, 0x100000000452e98)
        	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x89
        golang.org/x/net/http2.(*Framer).ReadFrame(0xc01b8c5b20, 0xc0064ff680, 0xc0064ff680, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa5
        google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc006f4f500)
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1273 +0x179
        created by google.golang.org/grpc/internal/transport.newHTTP2Client
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0xd31
        
        goroutine 8784999 [chan receive, 5 minutes]:
        github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x5b15640, 0xc01c442270)
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
        runtime/pprof.Do(0x5b15640, 0xc01c442270, 0xc000111300, 0x1, 0x1, 0xc001a10fb8)
        	/usr/local/go/src/runtime/pprof/runtime.go:40 +0xcc
        github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc00aa0e600)
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
        created by github.com/cockroachdb/pebble.(*tableCacheShard).init
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ADD ident INTERVAL
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ADD ident INTERVAL". currently executing: map[ALTER TABLE ident.ident ADD ident INTERVAL:1]
    rsg_test.go:764: 4972 executions, 970 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn517827145
--- FAIL: TestRandomSyntaxSchemaChangeColumn (347.08s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 5c5bcf11bbc69161a9159eed027a00ced33f4c6e:

        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x172
        
        goroutine 8262601 [sync.Cond.Wait, 1 minutes]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc00eb9e090, 0x3587)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc00eb9e080)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc027c1c8c0, 0x5b174a0, 0xc0150ddd70)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc01d838300, 0x5b174a0, 0xc0150ddd70, 0x0, 0xc0083fbf40)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8262493 [select, 1 minutes]:
        github.com/cockroachdb/cockroach/pkg/sql/gcjob/gcjobnotifier.(*Notifier).run(0xc018d7f380, 0x5b174a0, 0xc021ba4600)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gcjobnotifier/notifier.go:136 +0x115
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc01d838300, 0x5b174a0, 0xc021ba4600, 0x0, 0xc009c1ce10)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8262496 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/sql.(*Server).PeriodicallyClearSQLStats.func1(0x5b174a0, 0xc021ba4600)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:754 +0x265
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc01d838300, 0x5b174a0, 0xc021ba4600, 0x0, 0xc027fd2060)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8262539 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x5b174a0, 0xc021ba4600)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc01d838300, 0x5b174a0, 0xc021ba4600, 0x0, 0xc00acda5e0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ADD COLUMN IF NOT EXISTS GROUPING INTERVAL ARRAY
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ADD COLUMN IF NOT EXISTS GROUPING INTERVAL ARRAY". currently executing: map[ALTER TABLE ident.ident ADD COLUMN IF NOT EXISTS GROUPING INTERVAL ARRAY:1]
    rsg_test.go:764: 6660 executions, 1300 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn801716346
--- FAIL: TestRandomSyntaxSchemaChangeColumn (353.95s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 2fb7b6a9af22779a20e3354e81dc90d5270b641f:

Fatal error:

panic: runtime error: slice bounds out of range [2:1] [recovered]
	panic: runtime error: slice bounds out of range [2:1]

Stack:

goroutine 8677511 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc00f686200, 0x5b19b20, 0xc00995c140)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:233 +0x126
panic(0x4a14e40, 0xc020c076c0)
	/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*Mutable).MakeMutationComplete(0xc002f74400, 0x5b1c960, 0xc000f486a0, 0x100000002, 0x5, 0xc002f74400, 0x5b1c960)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/structured.go:1582 +0x1dae
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).done.func1(0x5b19be0, 0xc00bb8b860, 0xc002b8c870, 0xc00f6ff130, 0x203000, 0xc002b8c8a8)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1184 +0x49d
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).txnWithModified.func1(0x5b19be0, 0xc00bb8b860, 0xc002b8c870, 0xc00f6ff130, 0x452530, 0xc022772200)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2020 +0x5a
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.Txn.func1(0x5b19be0, 0xc00bb8b860, 0xc002b8c870, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:52 +0x132
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn.func1(0x5b19be0, 0xc00bb8b860, 0xc002b8c870, 0x3ff0000000000000, 0x5c712e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:773 +0x43
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0xc002b8c870, 0x5b19be0, 0xc00bb8b860, 0xc001c98bc8, 0xc002b8c870, 0xc012b99260)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:828 +0xdc
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(0xc0032df730, 0x5b19be0, 0xc00bb8b860, 0xc001c98c30, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:772 +0x105
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.Txn(0x5b19be0, 0xc00bb8b860, 0xc000534000, 0xc00b8cc580, 0x5b85ae0, 0xc012b99260, 0xc0032df730, 0xc001c98cc8, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:47 +0x133
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).txnWithModified(0xc023b47b90, 0x5b19be0, 0xc00bb8b860, 0xc001c98d78, 0x0, 0x0, 0x0, 0x0, 0x846887)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2017 +0x115
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).done(0xc023b47b90, 0x5b19be0, 0xc00bb8b860, 0x1, 0x4d3174f)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1052 +0xbe
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runStateMachineAndBackfill(0xc023b47b90, 0x5b19be0, 0xc00bb8b860, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1492 +0x125
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).exec(0xc023b47b90, 0x5b19be0, 0xc00b3b17d0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:682 +0x5ea
github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume.func1(0x500000035, 0xc000000000, 0xc0104e6900, 0x5b7e4c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2167 +0x629
github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume(0xc00c9df2c0, 0x5b19be0, 0xc00b3b17d0, 0x4950ca0, 0xc0179dafe8, 0xc00c9df2c0, 0xc005d21838)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2285 +0x803
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1(0xc0165be200, 0x5ac2360, 0xc0179dafd8, 0x5b19be0, 0xc00b3b17d0, 0x4950ca0, 0xc0179dafe8, 0xc001c99a50)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1193 +0xae
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc029b2a820, 0x5b19be0, 0xc00b3b1710, 0x4950ca0, 0xc0179dafe8, 0x5ac2360, 0xc0179dafd8, 0xc00c9df2c0, 0xc0163aac40, 0x7, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1194 +0x885
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc029b2a820, 0x5b19b20, 0xc017bdeac0, 0x5ac2360, 0xc0179dafd8, 0xc00c9df2c0, 0xc0163aac40, 0x7, 0xc0145d4d20, 0x16, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:255 +0x378
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1(0x5b19b20, 0xc00995c140)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:214 +0x165
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00f686200, 0x5b19b20, 0xc00995c140, 0x0, 0xc02d396780)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
Log preceding fatal error 

E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1  a panic has occurred!
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +runtime error: slice bounds out of range [2:1]
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +(1) attached stack trace
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  -- stack trace:
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | runtime.gopanic
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/usr/local/go/src/runtime/panic.go:969
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | runtime.goPanicSliceB
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/usr/local/go/src/runtime/panic.go:116
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*Mutable).MakeMutationComplete
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/structured.go:1582
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).done.func1
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1184
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).txnWithModified.func1
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2020
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.Txn.func1
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:52
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn.func1
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:773
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:828
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:772
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.Txn
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:47
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).txnWithModified
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2017
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).done
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1052
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runStateMachineAndBackfill
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1492
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).exec
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:682
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume.func1
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2167
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2285
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1193
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1194
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:255
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:214
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | runtime.goexit
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +Wraps: (2) runtime error: slice bounds out of range [2:1]
E210430 18:44:56.916733 8677511 1@util/log/logcrash/crash_reporting.go:140  [-] 1 +Error types: (1) *withstack.withStack (2) runtime.boundsError

Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@rafiss rafiss mentioned this issue Apr 30, 2021
Closed
@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ f05a57facec0c2c3f623977b90506b76f94771df:

        	/usr/local/go/src/database/sql/sql.go:1528 +0xe5
        github.com/cockroachdb/cockroach/pkg/sql/tests_test.(*verifyFormatDB).exec.func1(0xc0145f5140, 0x5b19ae0, 0xc00a3334c0, 0xc0124ba840, 0x30, 0xc01357de60)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/tests/rsg_test.go:134 +0x7f
        created by github.com/cockroachdb/cockroach/pkg/sql/tests_test.(*verifyFormatDB).exec
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/tests/rsg_test.go:133 +0x18a
        
        goroutine 8412734 [select]:
        github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x5b19ba0, 0xc013eff980)
        	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1374 +0x345
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc011504200, 0x5b19ba0, 0xc013eff980, 0x0, 0xc002685a50)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8412763 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc009274250, 0x2f14)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc009274240)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0186dcc80, 0x5b19ba0, 0xc01212f470)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc011504200, 0x5b19ba0, 0xc01212f470, 0x0, 0xc00616dda0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8412602 [select]:
        github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc00b1ee900, 0x5b19b20, 0xc000074120)
        	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:466 +0x370
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc011504200, 0x5b19b20, 0xc000074120, 0x0, 0xc00616c680)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident DROP ident
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident DROP ident". currently executing: map[ALTER TABLE ident.ident ALTER COLUMN IGNORE_FOREIGN_KEYS SET NOT NULL:1 ALTER TABLE ident.ident DROP IF EXISTS FLOAT:1 ALTER TABLE ident.ident DROP ident:1]
    rsg_test.go:731: 5m35s of 5m0s: 6021 executions, 1169 successful
    rsg_test.go:731: 5m40s of 5m0s: 6021 executions, 1169 successful
    rsg_test.go:731: 5m45s of 5m0s: 6022 executions, 1170 successful
    rsg_test.go:731: 5m50s of 5m0s: 6022 executions, 1170 successful
    rsg_test.go:764: 6023 executions, 1171 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn059674099
--- FAIL: TestRandomSyntaxSchemaChangeColumn (351.00s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

postamar pushed a commit to postamar/cockroach that referenced this issue May 4, 2021
tabledesc: prevent panic when removing constraint from Checks slice
f97cf8a 
Recently, a run of the TestRandomSyntaxSchemaChangeColumn test failed
with a panic in tabledesc.MakeMutationComplete, see cockroachdb#63944. The panic
was triggered by removing multiple elements from the Checks slice in the
descriptor. While the underlying issue of _how_ this slice could
possibly contain two constraints sharing the same name has not yet been
resolved, deleting only the first constraint matching the target name
will prevent future panics and perhaps help diagnose the underlying
issue.

Release note: None
@postamar postamar mentioned this issue May 4, 2021
Merged
craig bot pushed a commit that referenced this issue May 4, 2021
Merge #64653
3761b5c 
64653: tabledesc: prevent panic when removing constraint from Checks slice r=postamar a=postamar

Recently, a run of the TestRandomSyntaxSchemaChangeColumn test failed
with a panic in tabledesc.MakeMutationComplete, see #63944. The panic
was triggered by removing multiple elements from the Checks slice in the
descriptor. While the underlying issue of _how_ this slice could
possibly contain two constraints sharing the same name has not yet been
resolved, deleting only the first constraint matching the target name
will prevent future panics and perhaps help diagnose the underlying
issue.

Release note: None

Co-authored-by: Marius Posta <[email protected]>
@postamar postamar mentioned this issue May 4, 2021
Merged
@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ aedf03967e1d9179943f403567433c9c1c6d6f23:

        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
        runtime/pprof.Do(0x5b1f400, 0xc01f0661b0, 0xc00000ff80, 0x1, 0x1, 0xc005febfb8)
        	/usr/local/go/src/runtime/pprof/runtime.go:40 +0xcc
        github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc016d24600)
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
        created by github.com/cockroachdb/pebble.(*tableCacheShard).init
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5
        
        goroutine 8314566 [chan receive, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func1(0x5b1f400, 0xc01f067560)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:193 +0x4c
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc01e74a200, 0x5b1f400, 0xc01f067560, 0x0, 0xc014c94380)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8314563 [select]:
        github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc01e05d0e0, 0x5b1f380, 0xc000074120)
        	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:466 +0x370
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc01e74a200, 0x5b1f380, 0xc000074120, 0x0, 0xc009ffc650)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8321358 [select]:
        github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc01996a240, 0xc00c3f2960, 0xc025c49fb0, 0xf, 0xc005d21560, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:1136 +0x2e7
        github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1(0x5b1f340, 0xc015c4ea40)
        	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:1065 +0x79
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc01e74a200, 0x5b1f340, 0xc015c4ea40, 0x0, 0xc00c3f2a50)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8321394 [select]:
        google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc01c9f48c0)
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1339 +0x193
        created by google.golang.org/grpc/internal/transport.newHTTP2Client
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:295 +0x16fc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ALTER FOLLOWING SET NOT NULL
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ALTER FOLLOWING SET NOT NULL". currently executing: map[ALTER TABLE ident.ident ALTER COLUMN RESET DROP NOT NULL:1 ALTER TABLE ident.ident ALTER FOLLOWING SET NOT NULL:1]
    rsg_test.go:764: 8823 executions, 1706 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn990932004
--- FAIL: TestRandomSyntaxSchemaChangeColumn (359.59s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 3bf34abe95cf99091c7ddea2eb767cdf0a297bd8:

        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc02bd9c200, 0x5b31da0, 0xc012be1020, 0x0, 0xc025213cc0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8725110 [select]:
        github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1(0x5b31da0, 0xc01278e630)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:151 +0x2e9
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc02bd9c200, 0x5b31da0, 0xc01278e630, 0x0, 0xc02f4eea80)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8724188 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc013bd6190, 0x3f8f)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc013bd6180)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc01ff86140, 0x5b31da0, 0xc012be1020)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc02bd9c200, 0x5b31da0, 0xc012be1020, 0x0, 0xc025213f10)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8802226 [select]:
        github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00cb22900, 0x5b31d20, 0xc000124010, 0x6, 0x5ba8160, 0xc02b867740, 0x0, 0x1, 0x10, 0x10)
        	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:248 +0x1fc
        github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).gcTxnRecord(0xc01e0865b0, 0x5b31d20, 0xc000124010, 0x6, 0xc009400a20, 0x0, 0xc01e6d5070)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:716 +0x35e
        github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x5b31d20, 0xc000124010)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:752 +0x68
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc02bd9c200, 0x5b31d20, 0xc000124010, 0x0, 0xc03934b620)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ALTER ident SET NOT NULL
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ALTER ident SET NOT NULL". currently executing: map[ALTER TABLE ident.ident ALTER ident SET NOT NULL:1]
    rsg_test.go:764: 8100 executions, 1523 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn788804837
--- FAIL: TestRandomSyntaxSchemaChangeColumn (357.35s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

postamar pushed a commit to postamar/cockroach that referenced this issue May 13, 2021
tabledesc: prevent panic when removing constraint from Checks slice
bde36d9 
Recently, a run of the TestRandomSyntaxSchemaChangeColumn test failed
with a panic in tabledesc.MakeMutationComplete, see cockroachdb#63944. The panic
was triggered by removing multiple elements from the Checks slice in the
descriptor. While the underlying issue of _how_ this slice could
possibly contain two constraints sharing the same name has not yet been
resolved, deleting only the first constraint matching the target name
will prevent future panics and perhaps help diagnose the underlying
issue.

Release note: None
@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 8fecaf91b33f63780880f9b0fd715e15ceecede8:

        goroutine 8496986 [select]:
        github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc0001810e0, 0x5b65440, 0xc000074120)
        	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:466 +0x370
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00e3b2200, 0x5b65440, 0xc000074120, 0x0, 0xc005b40600)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8497049 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x5b654c0, 0xc00e6686c0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00e3b2200, 0x5b654c0, 0xc00e6686c0, 0x0, 0xc0005c1500)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8497030 [chan receive, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1(0x5b65400, 0xc00f3b64c0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:400 +0x51
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00e3b2200, 0x5b65400, 0xc00f3b64c0, 0x0, 0xc014379e60)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8496984 [select]:
        github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc000180a20, 0x5b65440, 0xc000074120)
        	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:466 +0x370
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00e3b2200, 0x5b65440, 0xc000074120, 0x0, 0xc005b405e0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8497208 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/sql.(*Server).PeriodicallyClearSQLStats.func1(0x5b654c0, 0xc00e6686c0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:754 +0x265
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00e3b2200, 0x5b654c0, 0xc00e6686c0, 0x0, 0xc011157170)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ALTER TRIM SET NOT NULL
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ALTER TRIM SET NOT NULL". currently executing: map[ALTER TABLE ident.ident ALTER TRIM SET NOT NULL:1]
    rsg_test.go:764: 8233 executions, 1513 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn628955036
--- FAIL: TestRandomSyntaxSchemaChangeColumn (350.29s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ c1e87bb93b8782dc71ebba12145ca0446dd6a93b:

        goroutine 8367020 [chan receive, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func1(0x5b69d40, 0xc00258e600)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2023 +0x53
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0041bc200, 0x5b69d40, 0xc00258e600, 0x0, 0xc012120a20)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8367190 [select]:
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func8(0x5b69cc0, 0xc000074120)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:727 +0x20d
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0041bc200, 0x5b69cc0, 0xc000074120, 0x0, 0xc013cf1f20)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8366888 [select]:
        google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc009433d60, 0x1, 0x0, 0x0, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x125
        google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc027624660, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1d3
        google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0121d4300)
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:291 +0xd7
        created by google.golang.org/grpc/internal/transport.newHTTP2Server
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:288 +0x10d6
        
        goroutine 8367066 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc014a3a1d0, 0x39d6)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc014a3a1c0)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc01369fc20, 0x5b69d40, 0xc00c865d40)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0041bc200, 0x5b69d40, 0xc00c865d40, 0x0, 0xc0040ddc80)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ALTER COLUMN PAUSE SET NOT NULL
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ALTER COLUMN PAUSE SET NOT NULL". currently executing: map[ALTER TABLE ident.ident ALTER COLUMN PAUSE SET NOT NULL:1]
    rsg_test.go:764: 8160 executions, 1434 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn450486157
--- FAIL: TestRandomSyntaxSchemaChangeColumn (358.16s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ c6917a3058604cc42381915f108428533e1d47ff:

        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:482 +0xce
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc0126a7e70, 0xc01e6177a0, 0x5b69ca0, 0xc0270a9240, 0xc012cc66f0, 0xc000fda680, 0xc010dba400, 0x5bd1740, 0xc0270cad80, 0xc001880900, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:625 +0x3ea
        created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:553 +0x185
        
        goroutine 8922317 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x5b69d60, 0xc01bffe6f0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc025860280, 0x5b69d60, 0xc01bffe6f0, 0x0, 0xc027d7e820)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8922422 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc00049fe10, 0x2d11)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc00049fe00)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc00134b540, 0x5b69d60, 0xc02a466e10)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc025860280, 0x5b69d60, 0xc02a466e10, 0x0, 0xc026ce1e30)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8922366 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/server.(*loopbackListener).Accept(0xc029d1f890, 0x5407118, 0xc014eac1a0, 0x5bd2340, 0xc029c28680)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/loopback.go:49 +0xff
        google.golang.org/grpc.(*Server).Serve(0xc014eac1a0, 0x5b2d860, 0xc029d1f890, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:621 +0x216
        github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart.func2(0x5b69d60, 0xc01bffe6f0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1339 +0x49
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc025860280, 0x5b69d60, 0xc01bffe6f0, 0x0, 0xc02eb18ea0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident DROP COLUMN ANNOTATE_TYPE RESTRICT
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident DROP COLUMN ANNOTATE_TYPE RESTRICT". currently executing: map[ALTER TABLE ident.ident DROP COLUMN ANNOTATE_TYPE RESTRICT:1]
    rsg_test.go:764: 5916 executions, 1077 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn387761044
--- FAIL: TestRandomSyntaxSchemaChangeColumn (359.66s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 8ae1b4c3342a1aa9d187219a4f4eb0407452ab3a:

        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase/encoding.go:203 +0x35
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl.func4(0xc00638e000, 0xc008ca8fe0, 0xc000cf2580, 0xc001a27815, 0x5b36a20, 0xc010db4680, 0xc001a27870, 0xc001a27814, 0xc00ea9a200, 0x0, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:327 +0x71
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl(0xc00638e000, 0x5b6b1e0, 0xc024536ff0, 0xc008ca8fd0, 0xc000cf2580, 0x5400, 0x15000, 0xc00cec41e0, 0x0, 0x4, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:458 +0x565
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveConn(0xc000de8b00, 0x5b6b1e0, 0xc024536ff0, 0x5bcf7a0, 0xc00fc8e000, 0xc00db3d009, 0x4, 0xc024536fc0, 0x5b177e0, 0xc024536420, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:152 +0x1b8
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc000de8b00, 0x5b6b1e0, 0xc024536ff0, 0x5bd36a0, 0xc02c1d7dd0, 0x5b6b101, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:658 +0xc18
        github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).startServeSQL.func1.1(0x5bd36a0, 0xc02c1d7dd0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2096 +0xf7
        github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1(0xc00cc7c200, 0x5b6b1e0, 0xc01f328600, 0xc024b93160, 0x5bd36a0, 0xc02c1d7dd0, 0xc01604c360)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:156 +0xaf
        created by github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:153 +0x24a
        
        goroutine 9089733 [IO wait, 1 minutes]:
        internal/poll.runtime_pollWait(0x7f9d72675db0, 0x72, 0x0)
        	/usr/local/go/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc016bbe718, 0x72, 0x0, 0x0, 0x4d04641)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Accept(0xc016bbe700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        	/usr/local/go/src/internal/poll/fd_unix.go:394 +0x1fc
        net.(*netFD).accept(0xc016bbe700, 0x37e11d600, 0x0, 0x0)
        	/usr/local/go/src/net/fd_unix.go:172 +0x45
        net.(*TCPListener).accept(0xc00b0b8c20, 0xc001f4ce30, 0xc001f4ce38, 0x38)
        	/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
        net.(*TCPListener).Accept(0xc00b0b8c20, 0x54062f0, 0xc00cc7c200, 0x5b6b1e0, 0xc01f328600)
        	/usr/local/go/src/net/tcpsock.go:261 +0x65
        github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith(0xc024b93160, 0x5b6b1e0, 0xc01f328600, 0xc00cc7c200, 0x5b42be0, 0xc00b0b8c20, 0xc01604c360, 0xc0304605a0, 0xc005f69d40)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:135 +0x50
        github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).startServeSQL.func1(0x5b6b1e0, 0xc01f328600)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2092 +0xf4
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00cc7c200, 0x5b6b1e0, 0xc01f328600, 0x0, 0xc01d700940)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ADD COLUMN OPERATOR INTERVAL MINUTE TO SECOND ARRAY CREATE IF NOT EXISTS FAMILY ident
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ADD COLUMN OPERATOR INTERVAL MINUTE TO SECOND ARRAY CREATE IF NOT EXISTS FAMILY ident". currently executing: map[ALTER TABLE ident.ident ADD COLUMN OPERATOR INTERVAL MINUTE TO SECOND ARRAY CREATE IF NOT EXISTS FAMILY ident:1]
    rsg_test.go:764: 9184 executions, 1713 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn136741240
--- FAIL: TestRandomSyntaxSchemaChangeColumn (346.88s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 803e71d5b92591547f058cc731482cc2ccdd032a:

        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc016abc180)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc01a226e60, 0x5b6c720, 0xc0106519e0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc011ed6200, 0x5b6c720, 0xc0106519e0, 0x0, 0xc000f6bcc0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8744324 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x5b6c720, 0xc010b96660)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc011ed6200, 0x5b6c720, 0xc010b96660, 0x0, 0xc00629d100)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8744420 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc016abc190, 0x5ea7)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc016abc180)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc01a226e60, 0x5b6c720, 0xc0106519e0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc011ed6200, 0x5b6c720, 0xc0106519e0, 0x0, 0xc000f6bf70)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8744482 [select, 1 minutes]:
        github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache.(*Cache).periodicallyRefreshProtectedtsCache(0xc00b9751e0, 0x5b6c720, 0xc010b96660)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache/cache.go:160 +0x25b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc011ed6200, 0x5b6c720, 0xc010b96660, 0x0, 0xc00aef30d0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ADD COLUMN DETACHED BIT ( 398 ) ARRAY
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ADD COLUMN DETACHED BIT ( 398 ) ARRAY". currently executing: map[ALTER TABLE ident.ident ADD COLUMN DETACHED BIT ( 398 ) ARRAY:1]
    rsg_test.go:764: 11870 executions, 2248 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn847456501
--- FAIL: TestRandomSyntaxSchemaChangeColumn (358.09s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 3597e1e739382a7f067d86f6fba9ec39841dbb2c:

        goroutine 8882166 [IO wait]:
        internal/poll.runtime_pollWait(0x7f8c9e0b1390, 0x72, 0x5ad3e60)
        	/usr/local/go/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc01ef82398, 0x72, 0x5ad3e00, 0x7c4cbd8, 0x0)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc01ef82380, 0xc00bfe4000, 0x8b5, 0x8b5, 0x0, 0x0, 0x0)
        	/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
        net.(*netFD).Read(0xc01ef82380, 0xc00bfe4000, 0x8b5, 0x8b5, 0x203008, 0x6e8c3b, 0xc01edd52e0)
        	/usr/local/go/src/net/fd_posix.go:55 +0x4f
        net.(*conn).Read(0xc03be74f00, 0xc00bfe4000, 0x8b5, 0x8b5, 0x0, 0x0, 0x0)
        	/usr/local/go/src/net/net.go:182 +0x8e
        crypto/tls.(*atLeastReader).Read(0xc022345140, 0xc00bfe4000, 0x8b5, 0x8b5, 0x62, 0x8b0, 0xc01f0877a8)
        	/usr/local/go/src/crypto/tls/conn.go:779 +0x62
        bytes.(*Buffer).ReadFrom(0xc01edd5400, 0x5ac6460, 0xc022345140, 0x4502a5, 0x467e180, 0x4c11f20)
        	/usr/local/go/src/bytes/buffer.go:204 +0xb1
        crypto/tls.(*Conn).readFromUntil(0xc01edd5180, 0x5acf620, 0xc03be74f00, 0x5, 0xc03be74f00, 0x51)
        	/usr/local/go/src/crypto/tls/conn.go:801 +0xf3
        crypto/tls.(*Conn).readRecordOrCCS(0xc01edd5180, 0x0, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/conn.go:608 +0x115
        crypto/tls.(*Conn).readRecord(...)
        	/usr/local/go/src/crypto/tls/conn.go:576
        crypto/tls.(*Conn).Read(0xc01edd5180, 0xc008202000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/conn.go:1252 +0x15f
        bufio.(*Reader).Read(0xc013133140, 0xc01f2f0498, 0x9, 0x9, 0x0, 0xc01f087d01, 0x100050090)
        	/usr/local/go/src/bufio/bufio.go:227 +0x222
        io.ReadAtLeast(0x5ac6220, 0xc013133140, 0xc01f2f0498, 0x9, 0x9, 0x9, 0xc01659c2e0, 0xc006d02ed8, 0x0)
        	/usr/local/go/src/io/io.go:314 +0x87
        io.ReadFull(...)
        	/usr/local/go/src/io/io.go:333
        golang.org/x/net/http2.readFrameHeader(0xc01f2f0498, 0x9, 0x9, 0x5ac6220, 0xc013133140, 0x0, 0x0, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x89
        golang.org/x/net/http2.(*Framer).ReadFrame(0xc01f2f0460, 0xc013b75b90, 0x7fa6de0, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa5
        google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc016b776c0)
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1273 +0x179
        created by google.golang.org/grpc/internal/transport.newHTTP2Client
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0xd31
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ALTER DOMAIN SET NOT NULL
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ALTER DOMAIN SET NOT NULL". currently executing: map[ALTER TABLE ident.ident ALTER DOMAIN SET NOT NULL:1]
    rsg_test.go:764: 7280 executions, 1331 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn883791930
--- FAIL: TestRandomSyntaxSchemaChangeColumn (352.56s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 8a9909257235309ea01fff1ef9ce73d00fec5b97:

        	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2096 +0xf7
        github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1(0xc00ed4c400, 0x5b728e0, 0xc00e52c600, 0xc0145fc610, 0x5bdaea0, 0xc00ec7e170, 0xc01445cd20)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:156 +0xaf
        created by github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:153 +0x24a
        
        goroutine 8948714 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc00aa63e10, 0x2d42)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc00aa63e00)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0100383c0, 0x5b728e0, 0xc015709a40)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00ed4c400, 0x5b728e0, 0xc015709a40, 0x0, 0xc01426b550)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 9000341 [select]:
        github.com/cockroachdb/cockroach/pkg/util/retry.(*Retry).Next(0xc008625358, 0x1)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:128 +0x187
        github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume.func1(0x7d00000035, 0xc000000000, 0xc00e804c00, 0x5bd7780)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2170 +0x5f3
        github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume(0xc010a78c80, 0x5b728e0, 0xc01226e6f0, 0x499c120, 0xc010ae8d38, 0xc010a78c80, 0xc006723838)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2291 +0x803
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1(0xc01062c280, 0x5b1a7a0, 0xc012009000, 0x5b728e0, 0xc01226e6f0, 0x499c120, 0xc010ae8d38, 0xc008625a50)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1190 +0xae
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc010539380, 0x5b728e0, 0xc01226e630, 0x499c120, 0xc010ae8d38, 0x5b1a7a0, 0xc012009000, 0xc010a78c80, 0xc00555e650, 0x7, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1191 +0x885
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc010539380, 0x5b72820, 0xc00f57e700, 0x5b1a7a0, 0xc012009000, 0xc010a78c80, 0xc00555e650, 0x7, 0xc00466b500, 0x16, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:255 +0x378
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1(0x5b72820, 0xc014d058c0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:214 +0x165
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00ed4c400, 0x5b72820, 0xc014d058c0, 0x0, 0xc0085f2fa0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ADD IF NOT EXISTS ident BIT VARYING ( 377 ) ARRAY [ 412 ]
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ADD IF NOT EXISTS ident BIT VARYING ( 377 ) ARRAY [ 412 ]". currently executing: map[ALTER TABLE ident.ident ADD IF NOT EXISTS ident BIT VARYING ( 377 ) ARRAY [ 412 ]:1]
    rsg_test.go:764: 5815 executions, 1088 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn081644013
--- FAIL: TestRandomSyntaxSchemaChangeColumn (358.42s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 9a6c8c0a3fecb76585843fb8ee7c9e5b10b7630a:

        	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:128 +0x187
        github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume.func1(0x6b00000035, 0xc000000000, 0xc007f85700, 0x5bdd040)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2170 +0x5f3
        github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume(0xc015739180, 0x5b78140, 0xc01b5011a0, 0x49a13c0, 0xc00f9ae718, 0xc015739180, 0xc002e2d838)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2291 +0x803
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1(0xc00f730200, 0x5b20000, 0xc00f9ae708, 0x5b78140, 0xc01b5011a0, 0x49a13c0, 0xc00f9ae718, 0xc001319a50)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1190 +0xae
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc00b190b60, 0x5b78140, 0xc01b5010e0, 0x49a13c0, 0xc00f9ae718, 0x5b20000, 0xc00f9ae708, 0xc015739180, 0xc010fc3f70, 0x7, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1191 +0x885
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc00b190b60, 0x5b78080, 0xc008e8b5c0, 0x5b20000, 0xc00f9ae708, 0xc015739180, 0xc010fc3f70, 0x7, 0xc006d416a0, 0x16, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:255 +0x378
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1(0x5b78080, 0xc011a1ecc0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:214 +0x165
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00cc2a200, 0x5b78080, 0xc011a1ecc0, 0x0, 0xc0053c91d0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8214191 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc00047bb90, 0x249e)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc00047bb80)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc010bdfd60, 0x5b78140, 0xc011cc7320)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00cc2a200, 0x5b78140, 0xc011cc7320, 0x0, 0xc001361ad0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8214146 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x5b78140, 0xc010498600)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00cc2a200, 0x5b78140, 0xc010498600, 0x0, 0xc007bde0d0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ALTER ident SET NOT NULL
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ALTER ident SET NOT NULL". currently executing: map[ALTER TABLE ident.ident ALTER ident SET NOT NULL:1]
    rsg_test.go:764: 3981 executions, 762 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn063093835
--- FAIL: TestRandomSyntaxSchemaChangeColumn (357.27s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 349b8be9d101ec656e61d0b6a49dda0dc26bad57:

        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc010d4e380, 0x5b78140, 0xc01142b290, 0x0, 0xc0257d1980)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8960380 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc024d1c190, 0x3650)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc024d1c180)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc000b24f00, 0x5b78140, 0xc01142b290)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc010d4e380, 0x5b78140, 0xc01142b290, 0x0, 0xc0257d18d0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8970461 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc023238b10, 0x331)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc023238b00)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/sql.(*StmtBuf).CurCmd(0xc00438e620, 0x0, 0x0, 0x0, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_io.go:440 +0x9e
        github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc003b1a600, 0x5b78080, 0xc023238b40, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1462 +0x65
        github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc003b1a600, 0x5b78080, 0xc023238b40, 0xc014366320, 0x5400, 0x15000, 0xc0143663c0, 0xc005369950, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1438 +0x228
        github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc000f92000, 0x5b78080, 0xc023238b40, 0xc003b1a600, 0x5400, 0x15000, 0xc0143663c0, 0xc005369950, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:482 +0xce
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc012ad471f, 0xc004aa5780, 0x5b78080, 0xc023238b40, 0xc005369950, 0xc000f92000, 0xc00438e000, 0x5bdfc80, 0xc00ecc9b80, 0xc02a111e00, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:625 +0x3ea
        created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:553 +0x185
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident DROP IF EXISTS IFERROR RESTRICT
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident DROP IF EXISTS IFERROR RESTRICT". currently executing: map[ALTER TABLE ident.ident DROP IF EXISTS IFERROR RESTRICT:1]
    rsg_test.go:764: 7596 executions, 1395 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn465230500
--- FAIL: TestRandomSyntaxSchemaChangeColumn (354.54s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue May 24, 2021
Closed
@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ b600d9c2283e19886406e658369e70b5c61a612f:

        
        goroutine 9119604 [select]:
        github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10(0x5b7c0c0, 0xc000074120)
        	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:792 +0x20c
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0116f2280, 0x5b7c0c0, 0xc000074120, 0x0, 0xc0154ec840)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 9119454 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x5b7c140, 0xc00fa68600)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0116f2280, 0x5b7c140, 0xc00fa68600, 0x0, 0xc00a530a70)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 9119590 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/sql.(*TemporaryObjectCleaner).Start.func1(0x5b7c140, 0xc00fa68600)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/temporary_schema.go:596 +0x32c
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0116f2280, 0x5b7c140, 0xc00fa68600, 0x0, 0xc014e766e0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 9119521 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc000a15a90, 0x41a2)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc000a15a80)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc016810140, 0x5b7c140, 0xc01760cf90)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0116f2280, 0x5b7c140, 0xc01760cf90, 0x0, 0xc009755b40)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ADD COLUMN GEOMETRYCOLLECTIONM INTERVAL YEAR TO MONTH ARRAY
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ADD COLUMN GEOMETRYCOLLECTIONM INTERVAL YEAR TO MONTH ARRAY". currently executing: map[ALTER TABLE ident.ident ADD COLUMN GEOMETRYCOLLECTIONM INTERVAL YEAR TO MONTH ARRAY:1 ALTER TABLE ident.ident DROP COLUMN IF EXISTS POLYGON RESTRICT:1]
    rsg_test.go:731: 5m50s of 5m0s: 8601 executions, 1591 successful
    rsg_test.go:764: 8602 executions, 1592 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn834437891
--- FAIL: TestRandomSyntaxSchemaChangeColumn (353.89s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@rafiss rafiss mentioned this issue May 26, 2021
Merged
@cockroach-teamcity cockroach-teamcity mentioned this issue May 26, 2021
Closed
@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 0f9ad772b496f0b477c83b2e692607adcf6da220:

        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc011f58190, 0x27d9)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc011f58180)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0228a61e0, 0x5b81d20, 0xc0071db4d0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc006404580, 0x5b81d20, 0xc0071db4d0, 0x0, 0xc005227bf0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 9298298 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc011f58190, 0x27de)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc011f58180)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0228a61e0, 0x5b81d20, 0xc0071db4d0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc006404580, 0x5b81d20, 0xc0071db4d0, 0x0, 0xc005227b00)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 9298292 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc011f58190, 0x2812)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc011f58180)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0228a61e0, 0x5b81d20, 0xc0071db4d0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc006404580, 0x5b81d20, 0xc0071db4d0, 0x0, 0xc005227aa0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ALTER COLUMN OVERLAY SET NOT NULL
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ALTER COLUMN OVERLAY SET NOT NULL". currently executing: map[ALTER TABLE ident.ident ALTER COLUMN OVERLAY SET NOT NULL:1]
    rsg_test.go:764: 5310 executions, 960 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn222251765
--- FAIL: TestRandomSyntaxSchemaChangeColumn (358.46s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@fqazi fqazi self-assigned this May 28, 2021
@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 4837b15a513fe9f0283b1583fece7fe8d3cd49ae:

        github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0xc011a97ef0, 0x5b83020, 0xc019446740, 0xc010dc1c88, 0xc011a97ef0, 0x7f87e805d108)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:843 +0xdc
        github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(0xc023ed93b0, 0x5b83020, 0xc019446740, 0xc02d9a7cf0, 0xc027642e58, 0x541f320)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:772 +0x105
        github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.storage.acquire(0xc003fdce80, 0xc023ed93b0, 0xc008716460, 0x5bef6a0, 0xc019d74720, 0xc000be4000, 0xc0297d9d80, 0xc0297d9d80, 0x0, 0xc002133580, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/lease.go:212 +0xc5
        github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.acquireNodeLease.func2(0x0, 0x0, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/lease.go:902 +0x2a6
        github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall(0xc002133580, 0xc00d812050, 0xc0166d5180, 0x9, 0xc0103d0720)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:128 +0x2e
        created by github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).DoChan
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:121 +0x2cc
        
        goroutine 9030528 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc01405bb10, 0x2976)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc01405bb00)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc02933a6e0, 0x5b830e0, 0xc0242e97d0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc02800a280, 0x5b830e0, 0xc0242e97d0, 0x0, 0xc00967ba60)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 9030424 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x5b830e0, 0xc01aabea80)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc02800a280, 0x5b830e0, 0xc01aabea80, 0x0, 0xc02faa4d40)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident DROP COLUMN GEOMETRY CASCADE
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident DROP COLUMN GEOMETRY CASCADE". currently executing: map[ALTER TABLE ident.ident ADD ident CHAR VARYING ( 410 ) ARRAY [ 251 ]:1 ALTER TABLE ident.ident ALTER COLUMN ident SET NOT NULL:1 ALTER TABLE ident.ident DROP COLUMN EXTRACT_DURATION:1 ALTER TABLE ident.ident DROP COLUMN GEOMETRY CASCADE:1 ALTER TABLE ident.ident DROP COLUMN IF EXISTS IDENTITY:1]
    rsg_test.go:731: 5m20s of 5m0s: 4864 executions, 926 successful
    rsg_test.go:731: 5m25s of 5m0s: 4865 executions, 927 successful
    rsg_test.go:731: 5m30s of 5m0s: 4866 executions, 928 successful
    rsg_test.go:731: 5m35s of 5m0s: 4867 executions, 929 successful
    rsg_test.go:731: 5m40s of 5m0s: 4867 executions, 929 successful
    rsg_test.go:764: 4868 executions, 930 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn343720451
--- FAIL: TestRandomSyntaxSchemaChangeColumn (344.45s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 587aaedda1a00fe03dd16317ec6da329f69a3ac9:

        	/usr/local/go/src/runtime/sema.go:56 +0x45
        sync.(*WaitGroup).Wait(0xc015425268)
        	/usr/local/go/src/sync/waitgroup.go:130 +0x65
        golang.org/x/sync/errgroup.(*Group).Wait(0xc015425260, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x31
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait(0xc015425260, 0x5b834e0, 0xc00bc4ea00, 0xc0157bea20, 0xc00bc4ea00)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:142 +0x57
        github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc00b94c400, 0x5b835a0, 0xc015425200, 0xc00f8df8a0, 0x1, 0x8, 0xc00f8df8a8, 0x1, 0x1, 0x0, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:83 +0x3c7
        github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0xc017132420, 0x5b834e0, 0xc015bf1580, 0xc00f8df8a0, 0x1, 0x8, 0xc00f8df8a8, 0x1, 0x1, 0x0, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:69 +0xb4
        github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1(0x5b834e0, 0xc015bf1580)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:208 +0xe3
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0139ae200, 0x5b834e0, 0xc015bf1580, 0x0, 0xc0154251d0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8891475 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2(0x5b835a0, 0xc0156c4870)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:818 +0x15f
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0139ae200, 0x5b835a0, 0xc0156c4870, 0x0, 0xc00f6f00c0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8891681 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/sql.(*Server).PeriodicallyClearSQLStats.func1(0x5b835a0, 0xc00e0e8600)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:756 +0x265
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0139ae200, 0x5b835a0, 0xc00e0e8600, 0x0, 0xc015b98990)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8953766 [chan send, 1 minutes]:
        github.com/lib/pq.(*conn).watchCancel.func1(0xc00a820180, 0xc00841cdc0, 0xc0123b3380)
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn_go18.go:104 +0x185
        created by github.com/lib/pq.(*conn).watchCancel
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn_go18.go:93 +0xa5
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ADD POSITION INTERVAL [ ] CREATE FAMILY
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ADD POSITION INTERVAL [ ] CREATE FAMILY". currently executing: map[ALTER TABLE ident.ident ADD POSITION INTERVAL [ ] CREATE FAMILY:1]
    rsg_test.go:764: 7230 executions, 1332 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn445203883
--- FAIL: TestRandomSyntaxSchemaChangeColumn (353.36s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ babd542104317a6dec12e29d2b93ca973f9f888c:

        	/usr/local/go/src/net/http/server.go:2937 +0x266
        github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func4(0x5b85f20, 0xc010e2a630)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2071 +0x3f
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc01121e280, 0x5b85f20, 0xc010e2a630, 0x0, 0xc039905ae0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8899485 [chan send, 1 minutes]:
        github.com/lib/pq.(*conn).watchCancel.func1(0xc01e7b87e0, 0xc001778dc0, 0xc00c1940c0)
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn_go18.go:104 +0x185
        created by github.com/lib/pq.(*conn).watchCancel
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn_go18.go:93 +0xa5
        
        goroutine 8835875 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x5b85f20, 0xc010e2a630)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc01121e280, 0x5b85f20, 0xc010e2a630, 0x0, 0xc012362800)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8836069 [select]:
        github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run(0xc00a876160, 0x5b85f20, 0xc010d44810, 0xc00d55fbe0, 0xc01121e280)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:238 +0x438
        github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1(0x5b85f20, 0xc010d44810)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:196 +0x50
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc01121e280, 0x5b85f20, 0xc010d44810, 0x0, 0xc00d55fc20)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8836001 [select]:
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*StoreRebalancer).Start.func1(0x5b85f20, 0xc010b66c30)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_rebalancer.go:190 +0x192
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc01121e280, 0x5b85f20, 0xc010b66c30, 0x0, 0xc034ce8fa0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ADD COLUMN LEVEL VARBIT ARRAY
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ADD COLUMN LEVEL VARBIT ARRAY". currently executing: map[ALTER TABLE ident.ident ADD COLUMN LEVEL VARBIT ARRAY:1]
    rsg_test.go:764: 7300 executions, 1366 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn761628817
--- FAIL: TestRandomSyntaxSchemaChangeColumn (357.28s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 1e2234a8449f6fe886387128dc5103866c4d244e:

        	/usr/local/go/src/crypto/tls/conn.go:1252 +0x15f
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*readTimeoutConn).Read(0xc015a96c40, 0xc008105000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:1679 +0x129
        bufio.(*Reader).fill(0xc008106048)
        	/usr/local/go/src/bufio/bufio.go:101 +0x105
        bufio.(*Reader).ReadByte(0xc008106048, 0x14dca1c, 0xed8491308, 0x0)
        	/usr/local/go/src/bufio/bufio.go:253 +0x39
        github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase.(*ReadBuffer).ReadTypedMsg(0xc008106808, 0x5b3f3c0, 0xc008106048, 0x5b34ec0, 0x801fe48, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase/encoding.go:203 +0x35
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl.func4(0xc008106000, 0xc006447650, 0xc000e7c000, 0xc0013f1815, 0x5b51a00, 0xc014b5c400, 0xc0013f1870, 0xc0013f1814, 0xc008c2c000, 0x0, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:328 +0x71
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl(0xc008106000, 0x5b86280, 0xc018718ae0, 0xc006447640, 0xc000e7c000, 0x5400, 0x15000, 0xc0150ed7c0, 0x0, 0x4, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:459 +0x565
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveConn(0xc0011dc400, 0x5b86280, 0xc018718ae0, 0x5beaa40, 0xc007028380, 0xc00742f01e, 0x4, 0xc018718ab0, 0x5b32680, 0xc01da0cd20, ...)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:153 +0x1b8
        github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc0011dc400, 0x5b86280, 0xc018718ae0, 0x5bee940, 0xc014d14b50, 0x5b86201, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:658 +0xc18
        github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).startServeSQL.func1.1(0x5bee940, 0xc014d14b50)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2096 +0xf7
        github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1(0xc015da8200, 0x5b86280, 0xc027e2a6c0, 0xc0024f8f90, 0x5bee940, 0xc014d14b50, 0xc01626ccc0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:156 +0xaf
        created by github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:153 +0x24a
        
        goroutine 9173130 [select, 1 minutes]:
        github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports.(*Reporter).Start.func2(0x5b86280, 0xc027e2a6c0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports/reporter.go:152 +0x206
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc015da8200, 0x5b86280, 0xc027e2a6c0, 0x0, 0xc01fe387e0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 9172929 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2(0x5b86280, 0xc014c69ec0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:818 +0x15f
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc015da8200, 0x5b86280, 0xc014c69ec0, 0x0, 0xc0150fe140)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident DROP COLUMN ANNOTATE_TYPE RESTRICT
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident DROP COLUMN ANNOTATE_TYPE RESTRICT". currently executing: map[ALTER TABLE ident.ident DROP COLUMN ANNOTATE_TYPE RESTRICT:1]
    rsg_test.go:764: 8398 executions, 1575 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn249719528
--- FAIL: TestRandomSyntaxSchemaChangeColumn (352.60s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ df9809c830c5cc26c9ebca67fba9ca4fcfa045ee:

        goroutine 8888176 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc00437c5e0, 0x12c3)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc00437c5d0)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/pebble/internal/record.(*flusherCond).Wait(...)
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:193
        github.com/cockroachdb/pebble/internal/record.(*LogWriter).flushLoop(0xc00437c500, 0x5b862c0, 0xc00c600b10)
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:396 +0x10b
        runtime/pprof.Do(0x5b862c0, 0xc00c600b10, 0xc00000f280, 0x1, 0x1, 0xc001990fb8)
        	/usr/local/go/src/runtime/pprof/runtime.go:40 +0xcc
        github.com/cockroachdb/pebble/internal/record.NewLogWriter.func2(0xc00437c500)
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:314 +0x89
        created by github.com/cockroachdb/pebble/internal/record.NewLogWriter
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:313 +0x2ea
        
        goroutine 8848489 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc0151d36d0, 0x3419)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc0151d36c0)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0156b5900, 0x5b862c0, 0xc014489470)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc01562c400, 0x5b862c0, 0xc014489470, 0x0, 0xc00769b130)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8848549 [select]:
        github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1(0x5b862c0, 0xc014f6a6c0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:660 +0x125
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc01562c400, 0x5b862c0, 0xc014f6a6c0, 0x0, 0xc0134f13c0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ADD COLUMN START DATE
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ADD COLUMN START DATE". currently executing: map[ALTER TABLE ident.ident ADD COLUMN START DATE:1]
    rsg_test.go:764: 7050 executions, 1221 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn682660257
--- FAIL: TestRandomSyntaxSchemaChangeColumn (359.24s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ 00b770d51383793d83c6c886357788dc211f7158:

        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc01832a180)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc015a2e140, 0x5b8a1e0, 0xc015dd6c60)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc013cfc280, 0x5b8a1e0, 0xc015dd6c60, 0x0, 0xc014771880)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8941121 [select]:
        github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc0012db180, 0x5b8a120, 0xc014360f80, 0xc014360f80, 0xc013c3cf50)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:254 +0x1f9
        github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc0012db180, 0x5b8a120, 0xc014360f80)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:271 +0xc5
        github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1(0x5b8a120, 0xc014360e40)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:259 +0x65
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc013cfc280, 0x5b8a120, 0xc014360e40, 0x0, 0xc00b9396e0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8947839 [select]:
        google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00ade1ae0, 0x1, 0x0, 0x0, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x125
        google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0193c0060, 0x0, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1d3
        google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc018923c00)
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x7b
        created by google.golang.org/grpc/internal/transport.newHTTP2Client
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:344 +0xefc
        
        goroutine 8941242 [chan receive, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart.func5(0x5b8a1e0, 0xc0043f6660)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1372 +0x53
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc013cfc280, 0x5b8a1e0, 0xc0043f6660, 0x0, 0xc00a245440)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ADD COLUMN IF NOT EXISTS ident INTERVAL MONTH
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ADD COLUMN IF NOT EXISTS ident INTERVAL MONTH". currently executing: map[ALTER TABLE ident.ident ADD COLUMN IF NOT EXISTS ident INTERVAL MONTH:1]
    rsg_test.go:764: 6930 executions, 1263 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn017411542
--- FAIL: TestRandomSyntaxSchemaChangeColumn (355.45s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql/tests.TestRandomSyntaxSchemaChangeColumn failed with artifacts on release-21.1 @ ac27d325ed08b76408e3bfdec3707c74d5a1c565:

        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8774888 [chan receive, 5 minutes]:
        google.golang.org/grpc.(*addrConn).resetTransport(0xc012120000)
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1156 +0x769
        created by google.golang.org/grpc.(*addrConn).connect
        	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:799 +0x12a
        
        goroutine 8767890 [chan receive, 5 minutes]:
        github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x5b8a1a0, 0xc011ea6570)
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
        runtime/pprof.Do(0x5b8a1a0, 0xc011ea6570, 0xc00000ff80, 0x1, 0x1, 0xc00eefcfb8)
        	/usr/local/go/src/runtime/pprof/runtime.go:40 +0xcc
        github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc00801f200)
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
        created by github.com/cockroachdb/pebble.(*tableCacheShard).init
        	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5
        
        goroutine 8768001 [sync.Cond.Wait]:
        runtime.goparkunlock(...)
        	/usr/local/go/src/runtime/proc.go:312
        sync.runtime_notifyListWait(0xc022a26110, 0x2fb8)
        	/usr/local/go/src/runtime/sema.go:513 +0xf8
        sync.(*Cond).Wait(0xc022a26100)
        	/usr/local/go/src/sync/cond.go:56 +0x9d
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc016df0140, 0x5b8a1a0, 0xc017e22fc0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:251 +0x8b
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc02ebc8300, 0x5b8a1a0, 0xc017e22fc0, 0x0, 0xc012785ca0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
        goroutine 8768045 [select, 5 minutes]:
        github.com/cockroachdb/cockroach/pkg/sql.(*TemporaryObjectCleaner).Start.func1(0x5b8a1a0, 0xc0085d4600)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/temporary_schema.go:596 +0x32c
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc02ebc8300, 0x5b8a1a0, 0xc0085d4600, 0x0, 0xc006a6f220)
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
        created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc
        
    rsg_test.go:186: SELECT 1 executed successfully: probably a slow statement
    rsg_test.go:755: Crash detected: 
        ALTER TABLE ident.ident ADD CHARACTERISTICS VARCHAR ( 253 ) [ ]
        
        Stack trace:
        timeout: "ALTER TABLE ident.ident ADD CHARACTERISTICS VARCHAR ( 253 ) [ ]". currently executing: map[ALTER TABLE ident.ident ADD CHARACTERISTICS VARCHAR ( 253 ) [ ]:1]
    rsg_test.go:764: 6066 executions, 1141 successful
    rsg_test.go:401: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomSyntaxSchemaChangeColumn247800487
--- FAIL: TestRandomSyntaxSchemaChangeColumn (358.70s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestRandomSyntaxSchemaChangeColumn PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Same failure on other branches

Internal log 

mjibson marked as alumn{us/a}; resolving to rafiss instead

/cc @rafiss

This test on roachdash | Improve this report!

@fqazi
Copy link
Collaborator

fqazi commented Jun 7, 2021

Fixed with: #65839

@fqazi fqazi closed this as completed Jun 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fqazi fqazi

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rafiss @fqazi @tbg @cockroach-teamcity