ccl/backupccl: TestDataDriven failed #96149

cockroach-teamcity · 2023-01-29T07:50:48Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 69dd453d0e61e258f402c5751de310405743cd18:

Fatal error:

panic: for 3 in test_certs: tenant client cert not found [recovered]
	panic: for 3 in test_certs: tenant client cert not found

Stack:

goroutine 543259385 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x3fe8020?, {0x6e7c438, 0xc070408990})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:229 +0x6a
panic({0x4fccb00, 0xc00790bf00})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx(0xc04581a000, {0xc01afac4c0, 0x1, 0x1?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:213 +0x10de
github.com/cockroachdb/cockroach/pkg/server.newGRPCServer(0x6e7c438?)
	github.com/cockroachdb/cockroach/pkg/server/grpc_server.go:36 +0xaa
github.com/cockroachdb/cockroach/pkg/server.makeTenantSQLServerArgs({_, _}, _, {0xc01e744000, 0xc0dea2a200, 0xc010224620, 0xc038830eb0, 0xc017e8b300, 0xc01a395ce0, {0xc010224620, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/tenant.go:960 +0x1686
github.com/cockroachdb/cockroach/pkg/server.NewTenantServer({_, _}, _, {0xc01e744000, 0xc0dea2a200, 0xc010224620, 0xc038830eb0, 0xc017e8b300, 0xc01a395ce0, {0xc010224620, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/tenant.go:172 +0x165
github.com/cockroachdb/cockroach/pkg/server.(*Server).startTenantServerInternal(_, {_, _}, {0xc01e744000, 0xc0dea2a200, 0xc010224620, 0xc038830eb0, 0xc017e8b300, 0xc01a395ce0, {0xc010224620, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:148 +0x21b
github.com/cockroachdb/cockroach/pkg/server.(*Server).newTenantServer(_, {_, _}, _, _, {{0x0, 0x0}, {0x0}, {{0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:82 +0x2fb
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startServerInternal(0xc0380cd140, {0x6e7c438, 0xc070408990}, 0xc040349f30)
	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:361 +0x196
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startControlledServer.func2({0x6e7c438, 0xc070408990})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:233 +0x296
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x43b

Log preceding fatal error

        ----
        2 newname 2
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4365/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-tenants:125:
        exec-sql [0 args]
        BACKUP TENANT 2 INTO 'nodelocal://1/tenant2'
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4365/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-tenants:129:
        exec-sql [0 args]
        RESTORE TENANT 2 FROM LATEST IN 'nodelocal://1/tenant2' WITH tenant_name = 'another-name';
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4365/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-tenants:133:
        query-sql [0 args]
        SELECT id,name,service_mode FROM system.tenants WHERE name = 'another-name';
        ----
        3 another-name 2
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
*
* ERROR: a panic has occurred!
* for 3 in test_certs: tenant client cert not found
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:884
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:213
*   | github.com/cockroachdb/cockroach/pkg/server.newGRPCServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/grpc_server.go:36
*   | github.com/cockroachdb/cockroach/pkg/server.makeTenantSQLServerArgs
*   | 	github.com/cockroachdb/cockroach/pkg/server/tenant.go:960
*   | github.com/cockroachdb/cockroach/pkg/server.NewTenantServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/tenant.go:172
*   | github.com/cockroachdb/cockroach/pkg/server.(*Server).startTenantServerInternal
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:148
*   | github.com/cockroachdb/cockroach/pkg/server.(*Server).newTenantServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:82
*   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startServerInternal
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:361
*   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startControlledServer.func2
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:233
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
*   | runtime.goexit
*   | 	GOROOT/src/runtime/asm_amd64.s:1594
* Wraps: (2) for 3 in test_certs: tenant client cert not found
* Error types: (1) *withstack.withStack (2) *security.Error
*

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery _{This test on roachdash | Improve this report!

Jira issue: CRDB-23967}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2023-01-29T07:59:49Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 69dd453d0e61e258f402c5751de310405743cd18:

*   | github.com/cockroachdb/cockroach/pkg/server.NewTenantServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/tenant.go:172
*   | github.com/cockroachdb/cockroach/pkg/server.(*Server).startTenantServerInternal
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:148
*   | github.com/cockroachdb/cockroach/pkg/server.(*Server).newTenantServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:82
*   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startServerInternal
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:361
*   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startControlledServer.func2
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:233
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
*   | runtime.goexit
*   | 	GOROOT/src/runtime/asm_amd64.s:1594
* Wraps: (2) for 3 in test_certs: tenant client cert not found
* Error types: (1) *withstack.withStack (2) *security.Error
*
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10281/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-tenants:129:
        exec-sql [0 args]
        RESTORE TENANT 2 FROM LATEST IN 'nodelocal://1/tenant2' WITH tenant_name = 'another-name';
        ----
panic: for 3 in test_certs: tenant client cert not found [recovered]
	panic: for 3 in test_certs: tenant client cert not found

goroutine 653810 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x3fe0cc0?, {0x6e75678, 0xc01f82b4d0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:229 +0x6a
panic({0x4fc4740, 0xc022d2a2a0})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx(0xc01dcb1180, {0xc0074ba4c0, 0x1, 0x1?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:213 +0x10de
github.com/cockroachdb/cockroach/pkg/server.newGRPCServer(0x6e75678?)
	github.com/cockroachdb/cockroach/pkg/server/grpc_server.go:36 +0xaa
github.com/cockroachdb/cockroach/pkg/server.makeTenantSQLServerArgs({_, _}, _, {0xc007424000, 0xc0048ee000, 0xc00b8dc540, 0xc00375bf40, 0xc00283faa0, 0xc0207a5e78, {0xc00b8dc540, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/tenant.go:960 +0x1686
github.com/cockroachdb/cockroach/pkg/server.NewTenantServer({_, _}, _, {0xc007424000, 0xc0048ee000, 0xc00b8dc540, 0xc00375bf40, 0xc00283faa0, 0xc0207a5e78, {0xc00b8dc540, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/tenant.go:172 +0x165
github.com/cockroachdb/cockroach/pkg/server.(*Server).startTenantServerInternal(_, {_, _}, {0xc007424000, 0xc0048ee000, 0xc00b8dc540, 0xc00375bf40, 0xc00283faa0, 0xc0207a5e78, {0xc00b8dc540, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:148 +0x21b
github.com/cockroachdb/cockroach/pkg/server.(*Server).newTenantServer(_, {_, _}, _, _, {{0x0, 0x0}, {0x0}, {{0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:82 +0x2fb
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startServerInternal(0xc0128ef860, {0x6e75678, 0xc01f82b4d0}, 0xc021251160)
	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:361 +0x196
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startControlledServer.func2({0x6e75678, 0xc01f82b4d0})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:233 +0x296
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x43b

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-29T07:59:50Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 69dd453d0e61e258f402c5751de310405743cd18:

        with schedules as (show schedules) select id, state, recurrence, owner, command from schedules where label='datatest' order by command->>'backup_type' asc;
        ----
        835281516055592961 <nil> @weekly root {"backup_statement": "BACKUP INTO 'nodelocal://1/example-schedule' WITH detached", "chain_protected_timestamp_records": true, "dependent_schedule_id": 835281516046516225, "unpause_on_success": 835281516046516225}
        835281516046516225 Waiting for initial backup to complete @daily root {"backup_statement": "BACKUP INTO LATEST IN 'nodelocal://1/example-schedule' WITH detached", "backup_type": 1, "chain_protected_timestamp_records": true, "dependent_schedule_id": 835281516055592961}
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11648/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/alter-schedule/missing-schedule:20:
        exec-sql [0 args]
        drop schedule 835281516055592961
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11648/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/alter-schedule/missing-schedule:26:
        exec-sql [0 args]
        create schedule datatest for backup into 'nodelocal://1/example-schedule' recurring '@daily' full backup '@weekly';
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11648/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/alter-schedule/missing-schedule:30:
        let [2 args]
        with schedules as (show schedules) select id from schedules where label='datatest' order by command->>'backup_type' asc;
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11648/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/alter-schedule/missing-schedule:34:
        query-sql [0 args]
        with schedules as (show schedules) select id, state, recurrence, owner, command from schedules where label='datatest' order by command->>'backup_type' asc;
        ----
        835281516304826369 <nil> @weekly root {"backup_statement": "BACKUP INTO 'nodelocal://1/example-schedule' WITH detached", "chain_protected_timestamp_records": true, "dependent_schedule_id": 835281516297519105, "unpause_on_success": 835281516297519105}
        835281516297519105 Waiting for initial backup to complete @daily root {"backup_statement": "BACKUP INTO LATEST IN 'nodelocal://1/example-schedule' WITH detached", "backup_type": 1, "chain_protected_timestamp_records": true, "dependent_schedule_id": 835281516304826369}
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11648/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/alter-schedule/missing-schedule:40:
        exec-sql [0 args]
        drop schedule 835281516297519105
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11648/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/alter-schedule/missing-schedule:46:
        query-sql [0 args]
        with schedules as (show schedules) select id, state, recurrence, owner, command from schedules where label='datatest' order by command->>'backup_type' asc;
        ----
        835281516304826369 <nil> @weekly root {"backup_statement": "BACKUP INTO 'nodelocal://1/example-schedule' WITH detached", "chain_protected_timestamp_records": true}
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11648/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/alter-schedule/missing-schedule:51:
        exec-sql [0 args]
        alter backup schedule 835281516304826369 set recurring '@daily';
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11648/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/alter-schedule/missing-schedule:55:
        query-sql [0 args]
        with schedules as (show schedules) select id, state, recurrence, owner, command from schedules where label='datatest' order by command->>'backup_type' asc;
        ----
        835281516304826369 <nil> @daily root {"backup_statement": "BACKUP INTO 'nodelocal://1/example-schedule' WITH detached", "chain_protected_timestamp_records": true}
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-29T08:00:26Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 69dd453d0e61e258f402c5751de310405743cd18:

Fatal error:

panic: SQL instance ID not set [recovered]
	panic: SQL instance ID not set

Stack:

goroutine 2166484 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00e93e820?, {0x6e75510, 0xc00a5ac0c0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:229 +0x6a
panic({0x4e95d80, 0x6e1d160})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.storage.acquire.func1({0x6e755b8, 0xc00bdbdf20}, 0x6e4a5b0?)
	github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/storage.go:123 +0x77a
github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1({0x6e755b8?, 0xc00bdbdf20?}, 0xc00ba2cc30?)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:960 +0x27
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0xc00f3ec6e0, {0x6e755b8, 0xc00bdbdf20}, 0xc0163f9718)
	github.com/cockroachdb/cockroach/pkg/kv/txn.go:927 +0xae
github.com/cockroachdb/cockroach/pkg/kv.runTxn({0x6e755b8, 0xc00bdbdf20}, 0xc00bdbdf20?, 0x33333333?)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:959 +0x6b
github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl(0x6e755b8?, {0x6e755b8, 0xc00bdbdf20}, 0x77359400?, 0x0?, 0x0?, 0x4000000000000000?)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:922 +0xa7
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(...)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:897
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.storage.acquire({0xc01d424280, {0x6e61bf0, 0xc01be60c00}, 0xc00ea2ec80, 0xc004832000, {{0xc0079e7020}, {0xc0079e7020}, 0x0}, 0xc027d4d380, 0xc01833cb40, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/storage.go:177 +0x314
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.acquireNodeLease.func1({0x6e75510, 0xc00a5ac0c0})
	github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/lease.go:522 +0x197
github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall.func1({0x6e75510?, 0xc00a5ac0c0?})
	github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:387 +0x34
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask(0xc018f5d000, {0x6e75510, 0xc00a5ac0c0}, {0x6e13a70?, 0x0?}, 0xc002114e70)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:305 +0xd1
github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall(0xc01bc95560, {0x6e755b8?, 0xc00bdbde90?}, 0xc0142618c0, {0x5944eb0, 0x1}, {0xc018f5d000?, 0x80?}, 0xc011ed96a0)
	github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:386 +0x2b0
created by github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).DoChan
	github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:356 +0x55e

Log preceding fatal error

* github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
* golang.org/x/sync/errgroup.(*Group).Go.func1()
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
* created by golang.org/x/sync/errgroup.(*Group).Go
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
* 
* goroutine 1264897 [select, 2 minutes]:
* github.com/cockroachdb/cockroach/pkg/sql.(*runnerCoordinator).init.func1({0xc01da5ac00, 0xc01201c120})
* 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:146 +0xf8
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x146
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x43b
* 
*
*
* ERROR: a panic has occurred!
* panic: SQL instance ID not set
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:884
*   | github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.storage.acquire.func1
*   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/storage.go:123
*   | github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1
*   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:960
*   | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
*   | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:927
*   | github.com/cockroachdb/cockroach/pkg/kv.runTxn
*   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:959
*   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
*   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:922
*   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
*   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:897
*   | github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.storage.acquire
*   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/storage.go:177
*   | github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.acquireNodeLease.func1
*   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/lease.go:522
*   | github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall.func1
*   | 	github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:387
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:305
*   | github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall
*   | 	github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:386
*   | runtime.goexit
*   | 	GOROOT/src/runtime/asm_amd64.s:1594
* Wraps: (2) panic: SQL instance ID not set
* Error types: (1) *withstack.withStack (2) *errutil.leafError
*

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-30T19:21:02Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 10ef5d9d7a646831799faa0f0611834fe032b93b:

        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1783/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-dropped-descriptors:78:
        exec-sql [0 args]
        USE d1;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1783/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-dropped-descriptors:82:
        query-sql [0 args]
        SELECT schema_name,table_name FROM [SHOW TABLES];
        ----
        public bar
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1783/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-dropped-descriptors:90:
        exec-sql [0 args]
        RESTORE DATABASE d FROM LATEST IN 'nodelocal://0/cluster/dropped-database' WITH new_db_name = 'd2';
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1783/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-dropped-descriptors:94:
        exec-sql [0 args]
        USE d2;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1783/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-dropped-descriptors:98:
        query-sql [0 args]
        SELECT schema_name,table_name FROM [SHOW TABLES];
        ----
        public bar
=== RUN   TestDataDriven/external-connections-nodelocal
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1783/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/external-connections-nodelocal:1:
        new-cluster [1 args]
        <no input to command>
        ----
=== CONT  TestDataDriven/external-connections-nodelocal
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
=== RUN   TestDataDriven/external-connections-userfile
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1783/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/external-connections-userfile:1:
        new-cluster [1 args]
        <no input to command>
        ----
=== CONT  TestDataDriven/external-connections-userfile
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T00:54:59Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 53f394da85573530c38c4197be2dacc6d42af10e:

        public schema full
        public schema full
        bank table incremental
        baz table incremental
        foo table incremental
        public schema incremental
        public schema incremental
        public schema incremental
        public schema incremental
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11771/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/encrypted-backups:159:
        exec-sql [0 args]
        RESTORE DATABASE d FROM LATEST IN 'nodelocal://1/full' WITH new_db_name='d2', encryption_passphrase='123';
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11771/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/encrypted-backups:163:
        exec-sql [0 args]
        RESTORE DATABASE d FROM LATEST IN 'nodelocal://1/full2' WITH new_db_name='d3', encryption_passphrase='456', incremental_location='nodelocal://1/inc';
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11771/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/encrypted-backups:167:
        exec-sql [0 args]
        RESTORE DATABASE d FROM LATEST IN 'nodelocal://1/full3' WITH new_db_name='d4', kms='testkms:///cmk?AUTH=implicit';
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11771/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/encrypted-backups:171:
        query-sql [0 args]
        USE d2;
        SELECT table_name FROM [SHOW TABLES] ORDER BY table_name;
        ----
        baz
        foo
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11771/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/encrypted-backups:178:
        query-sql [0 args]
        USE d3;
        SELECT table_name FROM [SHOW TABLES] ORDER BY table_name;
        ----
        baz
        foo
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11771/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/encrypted-backups:185:
        query-sql [0 args]
        USE d4;
        SELECT table_name FROM [SHOW TABLES] ORDER BY table_name;
        ----
        baz
        foo
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T00:55:02Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 53f394da85573530c38c4197be2dacc6d42af10e:

        query-sql [0 args]
        select role_name, settings FROM system.database_role_settings AS drs JOIN system.namespace AS ns ON ns.id = drs.database_id WHERE ns.name = 'foo';
        ----
        hamburger {application_name=helper}
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
=== RUN   TestDataDriven/restore-schema-only-mixed-version
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11783/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-schema-only-mixed-version:5:
        new-cluster [3 args]
        <no input to command>
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11783/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-schema-only-mixed-version:8:
        exec-sql [0 args]
        CREATE DATABASE d;
        USE d;
        CREATE TABLE foo (i INT PRIMARY KEY, s STRING);
        INSERT INTO foo VALUES (1, 'x'),(2,'y');
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11783/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-schema-only-mixed-version:15:
        exec-sql [0 args]
        BACKUP Database d INTO 'nodelocal://1/full_database_backup/';
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11783/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-schema-only-mixed-version:19:
        exec-sql [0 args]
        RESTORE DATABASE d FROM LATEST IN 'nodelocal://0/full_database_backup/' with schema_only, new_db_name='d2';
        ----
        pq: cannot run RESTORE with schema_only until cluster has fully upgraded to 22.2
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11783/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-schema-only-mixed-version:24:
        upgrade-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11783/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-schema-only-mixed-version:27:
        exec-sql [0 args]
        RESTORE DATABASE d FROM LATEST IN 'nodelocal://0/full_database_backup/' with schema_only, new_db_name='d2';
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11783/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-schema-only-mixed-version:32:
        query-sql [0 args]
        SELECT * FROM d2.foo;
        ----
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T06:24:47Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 1fb7236f7b1f53e28bde833c28bed83a976138cf:

Fatal error:

panic: for 3 in test_certs: tenant client cert not found [recovered]
	panic: for 3 in test_certs: tenant client cert not found

Stack:

goroutine 1368323 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x3fe0f60?, {0x6e75bd8, 0xc01a5572c0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:229 +0x6a
panic({0x4fc4aa0, 0xc00ab353a0})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx(0xc023c83900, {0xc0032a04c0, 0x1, 0x1?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:213 +0x10de
github.com/cockroachdb/cockroach/pkg/server.newGRPCServer(0x6e75bd8?)
	github.com/cockroachdb/cockroach/pkg/server/grpc_server.go:36 +0xaa
github.com/cockroachdb/cockroach/pkg/server.makeTenantSQLServerArgs({_, _}, _, {0xc001ee6000, 0xc00fc16d00, 0xc00b9fe2a0, 0xc016ee9540, 0xc00ab34b00, 0xc021512528, {0xc00b9fe2a0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/tenant.go:963 +0x16c6
github.com/cockroachdb/cockroach/pkg/server.NewTenantServer({_, _}, _, {0xc001ee6000, 0xc00fc16d00, 0xc00b9fe2a0, 0xc016ee9540, 0xc00ab34b00, 0xc021512528, {0xc00b9fe2a0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/tenant.go:172 +0x165
github.com/cockroachdb/cockroach/pkg/server.(*Server).startTenantServerInternal(_, {_, _}, {0xc001ee6000, 0xc00fc16d00, 0xc00b9fe2a0, 0xc016ee9540, 0xc00ab34b00, 0xc021512528, {0xc00b9fe2a0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:148 +0x21b
github.com/cockroachdb/cockroach/pkg/server.(*Server).newTenantServer(_, {_, _}, _, _, {{0x0, 0x0}, {0x0}, {{0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:82 +0x2fb
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startServerInternal(0xc003e05f80, {0x6e75bd8, 0xc01a5572c0}, 0xc0067d34b0)
	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:361 +0x196
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startControlledServer.func2({0x6e75bd8, 0xc01a5572c0})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:233 +0x296
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x43b

Log preceding fatal error

        ----
        2 newname 2
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3651/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-tenants:125:
        exec-sql [0 args]
        BACKUP TENANT 2 INTO 'nodelocal://1/tenant2'
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3651/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-tenants:129:
        exec-sql [0 args]
        RESTORE TENANT 2 FROM LATEST IN 'nodelocal://1/tenant2' WITH tenant_name = 'another-name';
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3651/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-tenants:133:
        query-sql [0 args]
        SELECT id,name,service_mode FROM system.tenants WHERE name = 'another-name';
        ----
        3 another-name 2
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
*
* ERROR: a panic has occurred!
* for 3 in test_certs: tenant client cert not found
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:884
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:213
*   | github.com/cockroachdb/cockroach/pkg/server.newGRPCServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/grpc_server.go:36
*   | github.com/cockroachdb/cockroach/pkg/server.makeTenantSQLServerArgs
*   | 	github.com/cockroachdb/cockroach/pkg/server/tenant.go:963
*   | github.com/cockroachdb/cockroach/pkg/server.NewTenantServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/tenant.go:172
*   | github.com/cockroachdb/cockroach/pkg/server.(*Server).startTenantServerInternal
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:148
*   | github.com/cockroachdb/cockroach/pkg/server.(*Server).newTenantServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:82
*   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startServerInternal
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:361
*   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startControlledServer.func2
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:233
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
*   | runtime.goexit
*   | 	GOROOT/src/runtime/asm_amd64.s:1594
* Wraps: (2) for 3 in test_certs: tenant client cert not found
* Error types: (1) *withstack.withStack (2) *security.Error
*

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T07:47:38Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 1fb7236f7b1f53e28bde833c28bed83a976138cf:

Fatal error:

panic: for 3 in test_certs: tenant client cert not found [recovered]
	panic: for 3 in test_certs: tenant client cert not found

Stack:

goroutine 540586044 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x3fe8300?, {0x6e7cab8, 0xc0228fd1d0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:229 +0x6a
panic({0x4fccea0, 0xc01ddf3c00})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx(0xc024e5ea00, {0xc002fa44c0, 0x1, 0x1?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:213 +0x10de
github.com/cockroachdb/cockroach/pkg/server.newGRPCServer(0x6e7cab8?)
	github.com/cockroachdb/cockroach/pkg/server/grpc_server.go:36 +0xaa
github.com/cockroachdb/cockroach/pkg/server.makeTenantSQLServerArgs({_, _}, _, {0xc00305a000, 0xc010c41400, 0xc01e2b87e0, 0xc02d2af130, 0xc0076aaf20, 0xc01b691cf8, {0xc01e2b87e0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/tenant.go:963 +0x16c6
github.com/cockroachdb/cockroach/pkg/server.NewTenantServer({_, _}, _, {0xc00305a000, 0xc010c41400, 0xc01e2b87e0, 0xc02d2af130, 0xc0076aaf20, 0xc01b691cf8, {0xc01e2b87e0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/tenant.go:172 +0x165
github.com/cockroachdb/cockroach/pkg/server.(*Server).startTenantServerInternal(_, {_, _}, {0xc00305a000, 0xc010c41400, 0xc01e2b87e0, 0xc02d2af130, 0xc0076aaf20, 0xc01b691cf8, {0xc01e2b87e0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:148 +0x21b
github.com/cockroachdb/cockroach/pkg/server.(*Server).newTenantServer(_, {_, _}, _, _, {{0x0, 0x0}, {0x0}, {{0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:82 +0x2fb
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startServerInternal(0xc02abc19e0, {0x6e7cab8, 0xc0228fd1d0}, 0xc0a6dc51f0)
	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:361 +0x196
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startControlledServer.func2({0x6e7cab8, 0xc0228fd1d0})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:233 +0x296
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x43b

Log preceding fatal error

        1 system 1 2 true {"capabilities": {"canAdminSplit": false}, "deprecatedDataState": "READY", "deprecatedId": "1", "droppedName": "", "tenantReplicationJobId": "0"}
        2 newname 1 1 true {"capabilities": {"canAdminSplit": false}, "deprecatedDataState": "READY", "deprecatedId": "2", "droppedName": "", "tenantReplicationJobId": "0"}
        5 tenant-5 2 0 false {"capabilities": {"canAdminSplit": false}, "deprecatedDataState": "DROP", "deprecatedId": "5", "droppedName": "tenant-5", "tenantReplicationJobId": "0"}
        6 tenant-6 1 1 true {"capabilities": {"canAdminSplit": false}, "deprecatedDataState": "READY", "deprecatedId": "6", "droppedName": "", "tenantReplicationJobId": "0"}
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4365/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-tenants:116:
        exec-sql [0 args]
        ALTER TENANT newname STOP SERVICE; ALTER TENANT newname START SERVICE SHARED
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4365/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-tenants:120:
        query-sql [0 args]
        SELECT id,name,service_mode FROM system.tenants WHERE name = 'newname';
        ----
        2 newname 2
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4365/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-tenants:125:
        exec-sql [0 args]
        BACKUP TENANT 2 INTO 'nodelocal://1/tenant2'
        ----
*
* ERROR: a panic has occurred!
* for 3 in test_certs: tenant client cert not found
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:884
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:213
*   | github.com/cockroachdb/cockroach/pkg/server.newGRPCServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/grpc_server.go:36
*   | github.com/cockroachdb/cockroach/pkg/server.makeTenantSQLServerArgs
*   | 	github.com/cockroachdb/cockroach/pkg/server/tenant.go:963
*   | github.com/cockroachdb/cockroach/pkg/server.NewTenantServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/tenant.go:172
*   | github.com/cockroachdb/cockroach/pkg/server.(*Server).startTenantServerInternal
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:148
*   | github.com/cockroachdb/cockroach/pkg/server.(*Server).newTenantServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:82
*   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startServerInternal
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:361
*   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startControlledServer.func2
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:233
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
*   | runtime.goexit
*   | 	GOROOT/src/runtime/asm_amd64.s:1594
* Wraps: (2) for 3 in test_certs: tenant client cert not found
* Error types: (1) *withstack.withStack (2) *security.Error
*

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T07:56:35Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 639decf058c78f0afe0dba613d60670628c92e81:

        ----
        NOTICE: setting the PRIMARY REGION as eu-north-1 on database defaultdb
        HINT: to change the default primary region, use SET CLUSTER SETTING sql.defaults.primary_region = 'region' or use RESET CLUSTER SETTING sql.defaults.primary_region to disable this behavior
        NOTICE: setting the PRIMARY REGION as eu-north-1 on database postgres
        HINT: to change the default primary region, use SET CLUSTER SETTING sql.defaults.primary_region = 'region' or use RESET CLUSTER SETTING sql.defaults.primary_region to disable this behavior
        NOTICE: setting the PRIMARY REGION as eu-north-1 on database no_region_db
        HINT: to change the default primary region, use SET CLUSTER SETTING sql.defaults.primary_region = 'region' or use RESET CLUSTER SETTING sql.defaults.primary_region to disable this behavior
        NOTICE: setting the PRIMARY REGION as eu-north-1 on database no_region_db_2
        HINT: to change the default primary region, use SET CLUSTER SETTING sql.defaults.primary_region = 'region' or use RESET CLUSTER SETTING sql.defaults.primary_region to disable this behavior
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-schema-only-multiregion:164:
        query-sql [0 args]
        SHOW DATABASES;
        ----
        defaultdb root eu-north-1  {eu-north-1} zone
        no_region_db root eu-north-1  {eu-north-1} zone
        no_region_db_2 root eu-north-1  {eu-north-1} zone
        postgres root eu-north-1  {eu-north-1} zone
        system node <nil> <nil> {} <nil>
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-schema-only-multiregion:173:
        query-sql [0 args]
        USE no_region_db;
        SELECT schema_name, table_name, type, owner, locality FROM [SHOW TABLES];
        ----
        public t table root REGIONAL BY TABLE IN PRIMARY REGION
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-schema-only-multiregion:180:
        exec-sql [0 args]
        RESTORE DATABASE eu_central_db FROM LATEST IN 'nodelocal://1/eu_central_database_backup/' with schema_only;
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-schema-only-multiregion:184:
        query-sql [0 args]
        SHOW DATABASES;
        ----
        defaultdb root eu-north-1  {eu-north-1} zone
        eu_central_db root eu-central-1  {eu-central-1} zone
        no_region_db root eu-north-1  {eu-north-1} zone
        no_region_db_2 root eu-north-1  {eu-north-1} zone
        postgres root eu-north-1  {eu-north-1} zone
        system node <nil> <nil> {} <nil>
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T07:56:36Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 639decf058c78f0afe0dba613d60670628c92e81:

        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12358/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-retry:20:
        new-cluster [4 args]
        <no input to command>
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12358/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-retry:23:
        exec-sql [0 args]
        SET CLUSTER SETTING storage.mvcc.range_tombstones.enabled = true;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12358/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-retry:27:
        exec-sql [0 args]
        SELECT crdb_internal.set_vmodule('lease=3');
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12358/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-retry:35:
        exec-sql [0 args]
        SET CLUSTER SETTING sql.catalog.descriptor_lease_duration = '1s';
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12358/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-retry:39:
        exec-sql [0 args]
        SET CLUSTER SETTING jobs.debug.pausepoints = 'restore.after_publishing_descriptors';
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12358/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-retry:43:
        restore [2 args]
        RESTORE FROM LATEST IN 'nodelocal://0/cluster_backup';
        ----
        job paused at pausepoint
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12358/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-retry:48:
        exec-sql [0 args]
        SET CLUSTER SETTING jobs.debug.pausepoints = '';
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12358/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-retry:53:
        job [1 args]
        <no input to command>
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12358/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-retry:58:
        sleep [1 args]
        <no input to command>
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12358/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-retry:61:
        restore [0 args]
        RESTORE FROM LATEST IN 'nodelocal://0/cluster_backup';
        ----

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T08:34:59Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 639decf058c78f0afe0dba613d60670628c92e81:

        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2472/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-ttl:23:
        exec-sql [0 args]
        DROP DATABASE d;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2472/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-ttl:27:
        restore [2 args]
        RESTORE DATABASE d FROM LATEST IN 'userfile:///foo';
        ----
        job paused at pausepoint
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2472/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-ttl:33:
        job [1 args]
        <no input to command>
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2472/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-ttl:36:
        query-sql [0 args]
        SELECT count(1) FROM [SHOW SCHEDULES] WHERE label LIKE 'row-level-ttl-%';
        ----
        0
=== RUN   TestDataDriven/restore-permissions/cluster-restore
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2472/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-permissions:180:
        new-cluster [3 args]
        <no input to command>
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2472/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-permissions:183:
        exec-sql [0 args]
        CREATE USER testuser;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2472/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-permissions:187:
        exec-sql [1 args]
        RESTORE FROM LATEST IN 'nodelocal://1/foo';
        ----
        pq: only users with the admin role or the RESTORE system privilege are allowed to perform a cluster restore: user testuser does not have RESTORE privilege on global 
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2472/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-permissions:192:
        exec-sql [0 args]
        GRANT SYSTEM RESTORE TO testuser;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2472/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-permissions:196:
        exec-sql [1 args]
        RESTORE FROM LATEST IN 'nodelocal://1/foo';
        ----

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T08:35:00Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 639decf058c78f0afe0dba613d60670628c92e81:

        <no input to command>
        ----
=== CONT  TestDataDriven/restore-on-fail-or-cancel-schema-objects
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
=== RUN   TestDataDriven/restore-on-fail-or-cancel-ttl/restore-fail-cleans-up-ttl-schedules-after-publishing
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2483/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-ttl:46:
        exec-sql [0 args]
        CREATE DATABASE d;
        CREATE TABLE d.tb (id INT PRIMARY KEY) WITH (ttl_expire_after = '10 minutes');
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2483/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-ttl:51:
        exec-sql [0 args]
        BACKUP DATABASE d INTO 'userfile:///foo'
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2483/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-ttl:56:
        exec-sql [0 args]
        SET CLUSTER SETTING jobs.debug.pausepoints = '';
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2483/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-ttl:60:
        exec-sql [0 args]
        SET CLUSTER SETTING jobs.debug.pausepoints = 'restore.after_publishing_descriptors';
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2483/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-ttl:64:
        exec-sql [0 args]
        DROP DATABASE d;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2483/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-ttl:68:
        restore [2 args]
        RESTORE DATABASE d FROM LATEST IN 'userfile:///foo';
        ----
        job paused at pausepoint
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2483/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-ttl:74:
        job [1 args]
        <no input to command>
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2483/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-ttl:77:
        query-sql [0 args]
        SELECT count(1) FROM [SHOW SCHEDULES] WHERE label LIKE 'row-level-ttl-%';
        ----
        0

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T08:35:02Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 639decf058c78f0afe0dba613d60670628c92e81:

        CREATE STATISTICS __auto__ ON a FROM tab;
        CREATE STATISTICS __auto__ ON b FROM tab;
        CREATE STATISTICS __auto__ ON a FROM tab;
        CREATE STATISTICS __auto__ ON b FROM tab;
        CREATE STATISTICS partial  ON a FROM tab USING EXTREMES;
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2484/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/metadata:37:
        query-sql [0 args]
        SELECT count(1) FROM [ SHOW STATISTICS FOR TABLE tab ]
        ----
        7
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2484/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/metadata:42:
        query-sql [0 args]
        SELECT count(1) FROM [ SHOW STATISTICS FOR TABLE tab WITH FORECAST ] WHERE statistics_name = '__forecast__'
        ----
        2
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2484/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/metadata:47:
        query-sql [0 args]
        SELECT count(1) FROM [ SHOW STATISTICS FOR TABLE tab WITH MERGE ] WHERE statistics_name = '__merged__'
        ----
        1
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2484/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/metadata:52:
        exec-sql [0 args]
        BACKUP DATABASE db1 INTO 'nodelocal://0/test/'
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2484/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/metadata:56:
        query-sql [0 args]
        SELECT
            json_array_length(
                crdb_internal.pb_to_json(
                    'cockroach.ccl.backupccl.StatsTable',
                    crdb_internal.read_file(
                        concat(
                            'nodelocal://0/test/',
                            (SELECT PATH FROM [SHOW BACKUPS IN 'nodelocal://0/test/']),
                            '/BACKUP-STATISTICS'
                        )
                    )
                )
                -> 'statistics'
            )
        ----
        7
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T08:48:15Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 639decf058c78f0afe0dba613d60670628c92e81:

        CREATE TABLE d.t (x INT);
        INSERT INTO d.t VALUES (1), (2), (3);
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12361/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/feature-flags:15:
        exec-sql [0 args]
        SET CLUSTER SETTING feature.backup.enabled = FALSE;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12361/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/feature-flags:19:
        exec-sql [0 args]
        BACKUP INTO 'nodelocal://0/test-root/';
        ----
        pq: feature BACKUP was disabled by the database administrator
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12361/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/feature-flags:25:
        exec-sql [0 args]
        SET CLUSTER SETTING feature.backup.enabled = TRUE;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12361/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/feature-flags:29:
        exec-sql [0 args]
        BACKUP INTO 'nodelocal://0/test-root/';
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12361/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/feature-flags:33:
        exec-sql [0 args]
        DROP TABLE d.t;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12361/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/feature-flags:38:
        exec-sql [0 args]
        SET CLUSTER SETTING feature.restore.enabled = FALSE;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12361/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/feature-flags:42:
        exec-sql [0 args]
        RESTORE TABLE d.t FROM LATEST IN 'nodelocal://0/test-root/';
        ----
        pq: feature RESTORE was disabled by the database administrator
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12361/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/feature-flags:48:
        exec-sql [0 args]
        SET CLUSTER SETTING feature.restore.enabled = TRUE;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12361/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/feature-flags:52:
        exec-sql [0 args]
        RESTORE TABLE d.t FROM LATEST IN 'nodelocal://0/test-root/';
        ----

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T08:48:16Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 639decf058c78f0afe0dba613d60670628c92e81:

github.com/cockroachdb/cockroach/pkg/server.(*Server).newTenantServer(_, {_, _}, _, _, {{0x0, 0x0}, {0x0}, {{0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:82 +0x2fb
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startServerInternal(0xc01805dec0, {0x6e769b8, 0xc018fc7d40}, 0xc00aa4af20)
	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:361 +0x196
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startControlledServer.func2({0x6e769b8, 0xc018fc7d40})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:233 +0x296
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x43b
=== RUN   TestDataDriven/drop-schedule-backup
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12362/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/drop-schedule-backup:4:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12362/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/drop-schedule-backup:7:
        exec-sql [0 args]
        CREATE DATABASE foo;
        CREATE TABLE foo.t (x INT);
        INSERT INTO foo.t VALUES (1), (2), (3);
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12362/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/drop-schedule-backup:13:
        exec-sql [0 args]
        CREATE SCHEDULE 'hello' FOR BACKUP DATABASE foo INTO 'userfile:///newbackup' WITH revision_history RECURRING '@daily';
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12362/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/drop-schedule-backup:17:
        query-sql [0 args]
        SELECT recurrence FROM [SHOW SCHEDULES] WHERE label='hello';
        ----
        @daily
        @weekly
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12362/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/drop-schedule-backup:23:
        exec-sql [0 args]
        DROP SCHEDULES WITH x AS (SHOW SCHEDULES) SELECT id FROM x WHERE label='hello' AND recurrence='@weekly';
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12362/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/drop-schedule-backup:27:
        query-sql [0 args]
        SELECT * FROM [SHOW SCHEDULES] WHERE label='hello';
        ----
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T14:38:15Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ c410d8d740e6943114fcbfde5355e7ba9bfe7c35:

        with schedules as (show schedules) select id, state, recurrence, owner, command from schedules where label='datatest' order by command->>'backup_type' asc;
        ----
        835927874968813569 <nil> 0 0 1 * * root {"backup_statement": "BACKUP INTO 'nodelocal://1/example-schedule' WITH detached", "chain_protected_timestamp_records": true, "dependent_schedule_id": 835927876304797697, "unpause_on_success": 835927876304797697}
        835927876304797697 Waiting for initial backup to complete @weekly root {"backup_statement": "BACKUP INTO LATEST IN 'nodelocal://1/example-schedule' WITH detached", "backup_type": 1, "chain_protected_timestamp_records": true, "dependent_schedule_id": 835927874968813569}
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11753/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/alter-schedule/recurrence:166:
        exec-sql [0 args]
        alter backup schedule 835927876304797697 set full backup '0 0 1 * *', set recurring '@weekly';
        alter backup schedule 835927876304797697 set full backup '0 0 1 * *', set recurring '@weekly';
        alter backup schedule 835927876304797697 set full backup '0 0 1 * *', set recurring '@weekly';
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11753/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/alter-schedule/recurrence:172:
        query-sql [0 args]
        with schedules as (show schedules) select id, state, recurrence, owner, command from schedules where label='datatest' order by command->>'backup_type' asc;
        ----
        835927874968813569 <nil> 0 0 1 * * root {"backup_statement": "BACKUP INTO 'nodelocal://1/example-schedule' WITH detached", "chain_protected_timestamp_records": true, "dependent_schedule_id": 835927876304797697, "unpause_on_success": 835927876304797697}
        835927876304797697 Waiting for initial backup to complete @weekly root {"backup_statement": "BACKUP INTO LATEST IN 'nodelocal://1/example-schedule' WITH detached", "backup_type": 1, "chain_protected_timestamp_records": true, "dependent_schedule_id": 835927874968813569}
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11753/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/alter-schedule/recurrence:180:
        exec-sql [1 args]
        alter backup schedule 835927874968813569 set recurring '@weekly', set full backup '@daily';
        ----
        regex matches error
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11753/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/alter-schedule/recurrence:187:
        exec-sql [0 args]
        alter backup schedule 835927874968813569 set full backup always, set recurring '@daily';
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11753/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/alter-schedule/recurrence:191:
        query-sql [0 args]
        with schedules as (show schedules) select id, state, recurrence, owner, command from schedules where label='datatest' order by command->>'backup_type' asc;
        ----
        835927874968813569 <nil> @daily root {"backup_statement": "BACKUP INTO 'nodelocal://1/example-schedule' WITH detached", "chain_protected_timestamp_records": true}
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11753/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/alter-schedule/recurrence:198:
        exec-sql [0 args]
        alter backup schedule 835927874968813569 set full backup always, set recurring '@daily';
        alter backup schedule 835927874968813569 set full backup always, set recurring '@daily';
        alter backup schedule 835927874968813569 set full backup always, set recurring '@daily';
        ----
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11753/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/alter-schedule/recurrence:204:
        query-sql [0 args]
        with schedules as (show schedules) select id, state, recurrence, owner, command from schedules where label='datatest' order by command->>'backup_type' asc;
        ----
        835927874968813569 <nil> @daily root {"backup_statement": "BACKUP INTO 'nodelocal://1/example-schedule' WITH detached", "chain_protected_timestamp_records": true}
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T14:38:17Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ c410d8d740e6943114fcbfde5355e7ba9bfe7c35:

    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-permissions:196:
        exec-sql [1 args]
        RESTORE FROM LATEST IN 'nodelocal://1/foo';
        ----
=== RUN   TestDataDriven/in-progress-restores
    datadriven_test.go:463: [https://github.com/cockroachdb/cockroach/issues/93731]
=== RUN   TestDataDriven/lock-concurrent-backups/backup-does-not-read-its-own-lock
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/lock-concurrent-backups:9:
        exec-sql [0 args]
        SET CLUSTER SETTING jobs.debug.pausepoints = 'backup.after.write_lock';
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/lock-concurrent-backups:13:
        backup [2 args]
        BACKUP INTO 'userfile://defaultdb.public.foo/foo';
        ----
        job paused at pausepoint
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/lock-concurrent-backups:20:
        query-sql [0 args]
        SELECT regexp_replace(filename, '.*BACKUP-LOCK-[0-9]+$', 'BACKUP-LOCK') FROM defaultdb.public.foo_upload_files;
        ----
        BACKUP-LOCK
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/lock-concurrent-backups:25:
        exec-sql [0 args]
        SET CLUSTER SETTING jobs.debug.pausepoints = '';
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/lock-concurrent-backups:30:
        job [1 args]
        <no input to command>
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/lock-concurrent-backups:33:
        job [2 args]
        <no input to command>
        ----
=== RUN   TestDataDriven/restore-on-fail-or-cancel-retry
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-retry:2:
        new-cluster [3 args]
        <no input to command>
        ----
=== CONT  TestDataDriven/restore-on-fail-or-cancel-retry
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T14:38:19Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ c410d8d740e6943114fcbfde5355e7ba9bfe7c35:

        DROP TABLE d.tb;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-schema-objects:115:
        exec-sql [0 args]
        SET CLUSTER SETTING jobs.debug.pausepoints = 'restore.after_publishing_descriptors';
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-schema-objects:119:
        restore [2 args]
        RESTORE d.tb FROM LATEST IN 'nodelocal://1/bar';
        ----
        job paused at pausepoint
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-schema-objects:126:
        job [1 args]
        <no input to command>
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-schema-objects:131:
        exec-sql [0 args]
        DROP TYPE d.ty;
        ----
=== RUN   TestDataDriven/restore-permissions/cluster-restore
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-permissions:180:
        new-cluster [3 args]
        <no input to command>
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-permissions:183:
        exec-sql [0 args]
        CREATE USER testuser;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-permissions:187:
        exec-sql [1 args]
        RESTORE FROM LATEST IN 'nodelocal://1/foo';
        ----
        pq: only users with the admin role or the RESTORE system privilege are allowed to perform a cluster restore: user testuser does not have RESTORE privilege on global 
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-permissions:192:
        exec-sql [0 args]
        GRANT SYSTEM RESTORE TO testuser;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-permissions:196:
        exec-sql [1 args]
        RESTORE FROM LATEST IN 'nodelocal://1/foo';
        ----

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T14:39:59Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 8e15b54e3e03999973c59c6b7fe2e0e1c669ad18:

    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12273/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-grants:476:
        query-sql [0 args]
        SHOW GRANTS ON TYPE greeting_usage;
        ----
        testdb public greeting_usage admin ALL true
        testdb public greeting_usage root ALL true
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12273/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-grants:482:
        query-sql [0 args]
        SHOW GRANTS ON testdb.testtable_greeting_usage;
        ----
        testdb public testtable_greeting_usage admin ALL true
        testdb public testtable_greeting_usage root ALL true
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12273/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-grants:488:
        query-sql [0 args]
        SHOW GRANTS ON TYPE greeting_owner;
        ----
        testdb public greeting_owner admin ALL true
        testdb public greeting_owner root ALL true
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12273/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-grants:494:
        query-sql [0 args]
        SHOW GRANTS ON testdb.testtable_greeting_owner;
        ----
        testdb public testtable_greeting_owner admin ALL true
        testdb public testtable_greeting_owner root ALL true
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12273/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-grants:501:
        query-sql [0 args]
        SELECT owner FROM [SHOW TYPES] WHERE name = 'greeting_owner';
        ----
        root
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12273/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-grants:506:
        query-sql [0 args]
        SELECT owner FROM [SHOW TABLES] WHERE table_name = 'testtable_greeting_owner';
        ----
        root
=== RUN   TestDataDriven/restore-on-fail-or-cancel-retry
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12273/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-on-fail-or-cancel-retry:2:
        new-cluster [3 args]
        <no input to command>
        ----
=== CONT  TestDataDriven/restore-on-fail-or-cancel-retry
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T14:40:01Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 8e15b54e3e03999973c59c6b7fe2e0e1c669ad18:

    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-permissions:33:
        exec-sql [1 args]
        BACKUP DATABASE d INTO 'userfile:///test-nonroot-db';
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-permissions:37:
        exec-sql [1 args]
        BACKUP TABLE d.t INTO 'userfile:///test-nonroot-table';
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-permissions:41:
        exec-sql [0 args]
        REVOKE ADMIN FROM testuser
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-permissions:46:
        exec-sql [1 args]
        BACKUP INTO 'userfile:///test-nonroot-cluster';
        ----
        pq: only users with the admin role or the BACKUP system privilege are allowed to perform full cluster backups: user testuser does not have BACKUP privilege on global 
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-permissions:52:
        exec-sql [0 args]
        GRANT SYSTEM BACKUP TO testuser;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-permissions:56:
        exec-sql [1 args]
        BACKUP INTO 'userfile:///test-nonroot-cluster';
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-permissions:62:
        exec-sql [1 args]
        BACKUP DATABASE d INTO 'userfile:///test-nonroot-db';
        ----
        pq: user testuser does not have SELECT privilege on relation t
        HINT: The existing privileges are being deprecated in favour of a fine-grained privilege model explained here https://www.cockroachlabs.com/docs/stable/backup.html#required-privileges. In a future release, to run BACKUP DATABASE, user testuser will exclusively require the BACKUP privilege on database d.
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-permissions:68:
        exec-sql [1 args]
        BACKUP TABLE d.t INTO 'userfile:///test-nonroot-table';
        ----
        pq: user testuser does not have SELECT privilege on relation t
        HINT: The existing privileges are being deprecated in favour of a fine-grained privilege model explained here https://www.cockroachlabs.com/docs/stable/backup.html#required-privileges. In a future release, to run BACKUP TABLE, user testuser will exclusively require the BACKUP privilege on tables: t.
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12356/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-permissions:74:
        exec-sql [0 args]
        REVOKE SYSTEM BACKUP FROM testuser;
        ----

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T14:40:02Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 8e15b54e3e03999973c59c6b7fe2e0e1c669ad18:

*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:213
*   | github.com/cockroachdb/cockroach/pkg/server.newGRPCServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/grpc_server.go:36
*   | github.com/cockroachdb/cockroach/pkg/server.makeTenantSQLServerArgs
*   | 	github.com/cockroachdb/cockroach/pkg/server/tenant.go:963
*   | github.com/cockroachdb/cockroach/pkg/server.NewTenantServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/tenant.go:172
*   | github.com/cockroachdb/cockroach/pkg/server.(*Server).startTenantServerInternal
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:148
*   | github.com/cockroachdb/cockroach/pkg/server.(*Server).newTenantServer
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:82
*   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startServerInternal
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:361
*   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startControlledServer.func2
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:233
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
*   | runtime.goexit
*   | 	GOROOT/src/runtime/asm_arm64.s:1172
* Wraps: (2) for 3 in test_certs: tenant client cert not found
* Error types: (1) *withstack.withStack (2) *security.Error
*
panic: for 3 in test_certs: tenant client cert not found [recovered]
	panic: for 3 in test_certs: tenant client cert not found

goroutine 541137 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x4003600098?, {0x6666bf8, 0x400efc7ef0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:229 +0x68
panic({0x47b3d00, 0x400bdc5980})
	GOROOT/src/runtime/panic.go:884 +0x20c
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx(0x4002905b80, {0x4003600478, 0x1, 0x60?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:213 +0xf24
github.com/cockroachdb/cockroach/pkg/server.newGRPCServer(0x6666bf8?)
	github.com/cockroachdb/cockroach/pkg/server/grpc_server.go:36 +0x9c
github.com/cockroachdb/cockroach/pkg/server.makeTenantSQLServerArgs({_, _}, _, {0x4003452000, 0x4016b46700, 0x400f17b340, 0x40180d0eb0, 0x400bdc50e0, 0x4019e92540, {0x400f17b340, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/tenant.go:963 +0x11c0
github.com/cockroachdb/cockroach/pkg/server.NewTenantServer({_, _}, _, {0x4003452000, 0x4016b46700, 0x400f17b340, 0x40180d0eb0, 0x400bdc50e0, 0x4019e92540, {0x400f17b340, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/tenant.go:172 +0xf0
github.com/cockroachdb/cockroach/pkg/server.(*Server).startTenantServerInternal(_, {_, _}, {0x4003452000, 0x4016b46700, 0x400f17b340, 0x40180d0eb0, 0x400bdc50e0, 0x4019e92540, {0x400f17b340, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:148 +0x190
github.com/cockroachdb/cockroach/pkg/server.(*Server).newTenantServer(_, {_, _}, _, _, {{0x0, 0x0}, {0x0}, {{0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_new_server.go:82 +0x20c
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startServerInternal(0x400a25e240, {0x6666bf8, 0x400efc7ef0}, 0x4008732820)
	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:361 +0x14c
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startControlledServer.func2({0x6666bf8, 0x400efc7ef0})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:233 +0x1a4
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x128
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x358

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T16:36:48Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 2fd74fcd3a2dbe8eecbc596064ea1ede898b9c2d:

        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/file_table_read_write:22:
        exec-sql [0 args]
        USE backups;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/file_table_read_write:26:
        query-sql [0 args]
        SELECT * FROM pg_catalog.pg_tables where schemaname='public';
        ----
        public userfiles_$user_upload_files root <nil> true false false false
        public userfiles_$user_upload_payload root <nil> true false false false
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/file_table_read_write:32:
        query-sql [0 args]
        SELECT conname FROM pg_catalog.pg_constraint con
        INNER JOIN pg_catalog.pg_class rel   ON rel.oid = con.conrelid
        INNER JOIN pg_catalog.pg_namespace nsp
        ON nsp.oid = connamespace
        WHERE rel.relname='userfiles_$user_upload_payload'
        ORDER BY conname;
        ----
        file_id_fk
        userfiles_$user_upload_payload_pkey
=== RUN   TestDataDriven/lock-concurrent-backups/backup-does-not-read-its-own-checkpoint
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/lock-concurrent-backups:117:
        exec-sql [0 args]
        SET CLUSTER SETTING jobs.debug.pausepoints = 'backup.after.write_first_checkpoint';
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/lock-concurrent-backups:121:
        backup [2 args]
        BACKUP TO 'userfile://defaultdb.public.bat/bat';
        ----
        job paused at pausepoint
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/lock-concurrent-backups:126:
        exec-sql [0 args]
        SET CLUSTER SETTING jobs.debug.pausepoints = '';
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/lock-concurrent-backups:131:
        job [1 args]
        <no input to command>
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12357/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/lock-concurrent-backups:134:
        job [2 args]
        <no input to command>
        ----

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-31T16:36:49Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 2fd74fcd3a2dbe8eecbc596064ea1ede898b9c2d:

        SHOW GRANTS ON TYPE testdb.greeting_usage;
        ----
        testdb public greeting_usage admin ALL true
        testdb public greeting_usage public USAGE false
        testdb public greeting_usage root ALL true
        testdb public greeting_usage testuser USAGE false
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12358/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-grants:134:
        query-sql [0 args]
        SHOW GRANTS ON TABLE testdb.testtable_greeting_usage;
        ----
        testdb public testtable_greeting_usage admin ALL true
        testdb public testtable_greeting_usage root ALL true
        testdb public testtable_greeting_usage testuser UPDATE false
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12358/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-grants:141:
        query-sql [0 args]
        SHOW GRANTS ON TYPE testdb.greeting_owner;
        ----
        testdb public greeting_owner admin ALL true
        testdb public greeting_owner public USAGE false
        testdb public greeting_owner root ALL true
        testdb public greeting_owner testuser ALL true
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12358/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-grants:149:
        query-sql [0 args]
        SHOW GRANTS ON TABLE testdb.testtable_greeting_owner;
        ----
        testdb public testtable_greeting_owner admin ALL true
        testdb public testtable_greeting_owner root ALL true
        testdb public testtable_greeting_owner testuser ALL true
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12358/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-grants:158:
        exec-sql [0 args]
        BACKUP INTO 'nodelocal://0/test/'
        ----
=== CONT  TestDataDriven/restore-grants
    datadriven_test.go:411: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12358/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-grants:514:
        exec-sql [0 args]
        USE defaultdb;
        DROP DATABASE testdb CASCADE;
        ALTER USER testuser CREATEDB;
        ----
=== CONT  TestDataDriven/restore-grants
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

blathers-crl · 2023-02-20T14:21:50Z

cc @cockroachdb/replication

tbg · 2023-02-20T14:42:52Z

finished proposal inserted into map

F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1  finished proposal inserted into map: &{ctx:0xc000082048 sp:<nil> idKey:�úTW,{P$ proposedAtTicks:2111 createdAtTicks:2068 command:0xc005b3a300 encodedCommand:[0 11 250 84 87 44 123 80 36 48 4 106 97 18 44 34 40 10 10 8 136 170 130 148 194 220 222 162 23 26 8 8 3 16 3 24 3 32 0 42 10 8 136 170 130 148 194 220 222 162 23 48 1 56 5 64 2 106 0 48 1 66 0 146 1 0 162 1 10 8 235 227 134 173 249 219 222 162 23 178 1 28 10 12 10 10 8 136 170 130 148 194 220 222 162 23 18 12 10 10 8 136 170 130 148 194 220 222 162 23 114 145 1 10 142 1 0 0 0 0 0 0 0 0 2 0 0 0 1 9 1 105 177 114 114 108 108 45 0 59 18 4 8 0 16 0 24 0 32 0 40 0 50 45 50 24 244 21 3 10 10 8 136 170 130 148 194 220 222 162 23 26 8 8 3 16 3 24 3 32 0 42 10 8 136 170 130 148 194 220 222 162 23 48 1 56 5 64 2 1 9 1 105 177 114 114 112 114 115 0 47 18 4 8 0 16 0 24 0 32 0 40 0 50 33 233 125 191 152 3 10 12 10 10 8 136 170 130 148 194 220 222 162 23 18 12 10 10 8 136 170 130 148 194 220 222 162 23 122 0 138 1 0] quotaAlloc:<nil> ec:{repl:<nil> g:<nil> st:{Lease:{Start:{WallTime:0 Logical:0 Synthetic:false} Expiration:<nil> Replica:{NodeID:0 StoreID:0 ReplicaID:0 Type:0} DeprecatedStartStasis:<nil> ProposedTS:<nil> Epoch:0 Sequence:0 AcquisitionType:0} Now:{WallTime:0 Logical:0 Synthetic:false} RequestTime:{WallTime:0 Logical:0 Synthetic:false} State:0 ErrInfo: Liveness:{NodeID:0 Epoch:0 Expiration:{WallTime:0 Logical:0 Synthetic:<nil>} Draining:false Membership:0} MinValidObservedTimestamp:{WallTime:0 Logical:0 Synthetic:false}}} applied:false doneCh:<nil> Local:LocalResult (reply: (err: <nil>), *kvpb.RequestLeaseResponse, #encountered intents: 0, #acquired locks: 0, #resolved locks: 0#updated txns: 0 #end txns: 0, GossipFirstRange:false MaybeGossipSystemConfig:false MaybeGossipSystemConfigIfHaveFailure:false MaybeAddToSplitQueue:false MaybeGossipNodeLiveness:<nil> Request:RequestLease [/Table/39,/Min) leaseStatus:{Lease:{Start:{WallTime:1676881500768851400 Logical:0 Synthetic:false} Expiration:<nil> Replica:{NodeID:2 StoreID:2 ReplicaID:2 Type:0} DeprecatedStartStasis:<nil> ProposedTS:0xc044be7b50 Epoch:2 Sequence:4 AcquisitionType:2} Now:{WallTime:1676881556572345751 Logical:0 Synthetic:false} RequestTime:{WallTime:1676881556572209259 Logical:0 Synthetic:false} State:3 ErrInfo: Liveness:{NodeID:2 Epoch:3 Expiration:{WallTime:1676881542326992778 Logical:0 Synthetic:<nil>} Draining:false Membership:0} MinValidObservedTimestamp:{WallTime:0 Logical:0 Synthetic:false}} tok:{done:true tok:<nil> b:<nil>}}
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !goroutine 448733787 [running]:
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x1)
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !	github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0x89
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0xc02bacd880, {{{0xc05338e1b0, 0x24}, {0x0, 0x0}, {0x5ab76f6, 0x1}, {0x5ab76f8, 0x1}}, 0x17457ae64c1c6981, ...})
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !	github.com/cockroachdb/cockroach/pkg/util/log/clog.go:262 +0x97
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x6f2dd58, 0xc03b6909c0}, 0x2, 0x4, 0x0, 0x0?, {0x5ae12d5, 0x28}, {0xc03f06f728, 0x1, ...})
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:106 +0x605
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:848
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaProposer).registerProposalLocked(0xc004080580, 0xc00aede6c0)
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:1193 +0xcb
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup(0xc004080da0, {0x6f2dd58, 0xc0a0dfe540}, {0x6f5a4c0, 0xc03faaf310})
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:435 +0x485
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2(0x51647a0?)
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:764 +0xd4
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked.func1(...)
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1934
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked(0xc004080580, 0xe0, 0xc004080b78)
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1935 +0x26a
F230220 08:26:05.803374 448733787 kv/kvserver/replica_proposal_buf.go:1193  [T1,n3,s3] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})

It's a LeaseRequest (we can see a LeaseResponse local result).

tbg · 2023-02-20T16:02:01Z

No simple repro via

$ ./dev test --cpus 4 --stress ./pkg/ccl/backupccl/ --filter='^TestDataDriven$' 2>&1 | tee stress.log
36 runs so far, 0 failures, over 42m25s

cockroach-teamcity · 2023-02-22T07:51:12Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 286b3e235171a39b8f9910555affcc7ce310741a:

=== RUN   TestDataDriven/backup-permissions/external-io-implicit-access
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4419/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-permissions:219:
        exec-sql [1 args]
        BACKUP TABLE d.foo.t1 INTO 'nodelocal://1/test-nonroot-table'
        ----
        pq: only users with the admin role or the EXTERNALIOIMPLICITACCESS system privilege are allowed to access the specified nodelocal URI
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4419/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-permissions:224:
        exec-sql [0 args]
        GRANT SYSTEM EXTERNALIOIMPLICITACCESS TO testuser;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4419/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-permissions:229:
        exec-sql [1 args]
        BACKUP TABLE d.foo.t1 INTO 'nodelocal://1/test-nonroot-table'
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4419/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-permissions:233:
        query-sql [1 args]
        select object_name from [show backup latest in 'nodelocal://1/test-nonroot-table'] order by object_name
        ----
        _type1
        d
        foo
        t1
        type1
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4419/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-permissions:242:
        exec-sql [0 args]
        REVOKE SYSTEM EXTERNALIOIMPLICITACCESS FROM testuser;
        ----
    datadriven.go:242: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4419/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/backup-permissions:246:
        query-sql [1 args]
        select object_name from [show backup latest in 'nodelocal://1/test-nonroot-table'] order by object_name
        ----
        pq: only users with the admin role or the EXTERNALIOIMPLICITACCESS system privilege are allowed to access the specified nodelocal URI
=== RUN   TestDataDriven/lock-concurrent-backups
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4419/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/lock-concurrent-backups:1:
        new-cluster [1 args]
        <no input to command>
        ----
=== CONT  TestDataDriven/lock-concurrent-backups
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-02-23T07:38:43Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ e028ce5b14505dfd17ef8b13001c0ab8ac811e3c:

        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4423/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/in-progress-imports:579:
        query-sql [0 args]
        SELECT table_name FROM [SHOW TABLES FROM d];
        ----
        foo
        foofoo
        baz
        show_cluster_backup
        show_database_backup
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
=== RUN   TestDataDriven/restore-permissions
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4423/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-permissions:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4423/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-permissions:4:
        exec-sql [0 args]
        CREATE DATABASE d;
        CREATE SCHEMA d.ds;
        CREATE TYPE d.ds.typ AS ENUM ('hello', 'hi');
        CREATE TABLE d.ds.t (x INT PRIMARY KEY, y d.ds.typ);
        INSERT INTO d.ds.t VALUES (1, 'hi'), (2, 'hello'), (3, 'hi');
        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4423/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/restore-permissions:14:
        exec-sql [0 args]
        BACKUP INTO 'nodelocal://1/foo';
        ----
=== CONT  TestDataDriven/restore-permissions
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-02-24T00:21:47Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ dda41f8bf34e20813a066c59840044530945ded4:

        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10581/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:11:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE FROM 'invalidDependOnBy_21.1' IN 'nodelocal://0/'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10581/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:16:
        link-backup [3 args]
        <no input to command>
        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10581/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:20:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE FROM 'valid-22.2' IN 'nodelocal://0/'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10581/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:25:
        link-backup [3 args]
        <no input to command>
        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10581/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:30:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE FROM 'valid-22.2-with-job' IN 'nodelocal://0/'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10581/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:36:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE 'nodelocal://0/valid-22.2-with-job'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10581/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:41:
         SELECT node, locality, ok, error, can_delete FROM [SHOW BACKUP CONNECTION 'nodelocal://1/conn-test' WITH TRANSFER='1'] ORDER BY node;
        expected:
        1 region=eu-central-1,availability-zone=eu-central-1 true  true
        2 region=eu-north-1,availability-zone=eu-north-1 true  true
        3 region=us-east-1,availability-zone=us-east1 true  true
        
        found:
        1 region=eu-central-1,availability-zone=eu-central-1 true  true
        2 region=eu-north-1,availability-zone=eu-north-1 true  true
        2 region=eu-north-1,availability-zone=eu-north-1 true  true
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    --- FAIL: TestDataDriven/show_backup (5.29s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-02-24T00:21:49Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ dda41f8bf34e20813a066c59840044530945ded4:

        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11616/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:11:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE FROM 'invalidDependOnBy_21.1' IN 'nodelocal://0/'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11616/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:16:
        link-backup [3 args]
        <no input to command>
        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11616/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:20:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE FROM 'valid-22.2' IN 'nodelocal://0/'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11616/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:25:
        link-backup [3 args]
        <no input to command>
        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11616/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:30:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE FROM 'valid-22.2-with-job' IN 'nodelocal://0/'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11616/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:36:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE 'nodelocal://0/valid-22.2-with-job'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11616/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:41:
         SELECT node, locality, ok, error, can_delete FROM [SHOW BACKUP CONNECTION 'nodelocal://1/conn-test' WITH TRANSFER='1'] ORDER BY node;
        expected:
        1 region=eu-central-1,availability-zone=eu-central-1 true  true
        2 region=eu-north-1,availability-zone=eu-north-1 true  true
        3 region=us-east-1,availability-zone=us-east1 true  true
        
        found:
        1 region=eu-central-1,availability-zone=eu-central-1 true  true
        2 region=eu-north-1,availability-zone=eu-north-1 true  true
        2 region=eu-north-1,availability-zone=eu-north-1 true  true
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    --- FAIL: TestDataDriven/show_backup (3.46s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-02-24T00:21:50Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ dda41f8bf34e20813a066c59840044530945ded4:

        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12723/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:11:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE FROM 'invalidDependOnBy_21.1' IN 'nodelocal://0/'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12723/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:16:
        link-backup [3 args]
        <no input to command>
        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12723/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:20:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE FROM 'valid-22.2' IN 'nodelocal://0/'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12723/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:25:
        link-backup [3 args]
        <no input to command>
        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12723/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:30:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE FROM 'valid-22.2-with-job' IN 'nodelocal://0/'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12723/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:36:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE 'nodelocal://0/valid-22.2-with-job'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12723/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:41:
         SELECT node, locality, ok, error, can_delete FROM [SHOW BACKUP CONNECTION 'nodelocal://1/conn-test' WITH TRANSFER='1'] ORDER BY node;
        expected:
        1 region=eu-central-1,availability-zone=eu-central-1 true  true
        2 region=eu-north-1,availability-zone=eu-north-1 true  true
        3 region=us-east-1,availability-zone=us-east1 true  true
        
        found:
        1 region=eu-central-1,availability-zone=eu-central-1 true  true
        2 region=eu-north-1,availability-zone=eu-north-1 true  true
        2 region=eu-north-1,availability-zone=eu-north-1 true  true
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    --- FAIL: TestDataDriven/show_backup (3.20s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-02-24T07:30:00Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 0d3393b0623a5c258b25725f64f3689e2f54667b:

        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10284/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:11:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE FROM 'invalidDependOnBy_21.1' IN 'nodelocal://0/'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10284/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:16:
        link-backup [3 args]
        <no input to command>
        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10284/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:20:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE FROM 'valid-22.2' IN 'nodelocal://0/'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10284/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:25:
        link-backup [3 args]
        <no input to command>
        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10284/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:30:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE FROM 'valid-22.2-with-job' IN 'nodelocal://0/'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10284/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:36:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE 'nodelocal://0/valid-22.2-with-job'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10284/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:41:
         SELECT node, locality, ok, error, can_delete FROM [SHOW BACKUP CONNECTION 'nodelocal://1/conn-test' WITH TRANSFER='1'] ORDER BY node;
        expected:
        1 region=eu-central-1,availability-zone=eu-central-1 true  true
        2 region=eu-north-1,availability-zone=eu-north-1 true  true
        3 region=us-east-1,availability-zone=us-east1 true  true
        
        found:
        1 region=eu-central-1,availability-zone=eu-central-1 true  true
        3 region=us-east-1,availability-zone=us-east1 true  true
        3 region=us-east-1,availability-zone=us-east1 true  true
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    --- FAIL: TestDataDriven/show_backup (4.56s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

tbg · 2023-02-24T07:40:36Z

#97606 disables the "finished proposal inserted" assertion, so once that PR merges it won't occur here. It's tracked separately in #97605.

cockroach-teamcity · 2023-02-24T09:33:59Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 0d3393b0623a5c258b25725f64f3689e2f54667b:

        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3711/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:11:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE FROM 'invalidDependOnBy_21.1' IN 'nodelocal://0/'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3711/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:16:
        link-backup [3 args]
        <no input to command>
        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3711/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:20:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE FROM 'valid-22.2' IN 'nodelocal://0/'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3711/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:25:
        link-backup [3 args]
        <no input to command>
        ----
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3711/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:30:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE FROM 'valid-22.2-with-job' IN 'nodelocal://0/'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3711/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:36:
        query-sql [1 args]
        SELECT * FROM [SHOW BACKUP VALIDATE 'nodelocal://0/valid-22.2-with-job'];
        ----
        true
    datadriven_test.go:421: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3711/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/show_backup:41:
         SELECT node, locality, ok, error, can_delete FROM [SHOW BACKUP CONNECTION 'nodelocal://1/conn-test' WITH TRANSFER='1'] ORDER BY node;
        expected:
        1 region=eu-central-1,availability-zone=eu-central-1 true  true
        2 region=eu-north-1,availability-zone=eu-north-1 true  true
        3 region=us-east-1,availability-zone=us-east1 true  true
        
        found:
        1 region=eu-central-1,availability-zone=eu-central-1 true  true
        3 region=us-east-1,availability-zone=us-east1 true  true
        3 region=us-east-1,availability-zone=us-east1 true  true
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    --- FAIL: TestDataDriven/show_backup (9.08s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

tbg · 2023-02-27T08:46:08Z

@stevendanna tossing this back to you since we shouldn't be seeing the assertion here any longer (#96149 (comment)).

stevendanna · 2023-03-03T10:01:37Z

Thanks. This hasn't failed in a week since that fix has gone in. Closing.

cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jan 29, 2023

cockroach-teamcity added this to the 23.1 milestone Jan 29, 2023

blathers-crl bot added the T-disaster-recovery label Jan 29, 2023

exalate-issue-sync bot assigned stevendanna Jan 31, 2023

tbg added T-kv-replication and removed T-kv KV Team labels Feb 20, 2023

exalate-issue-sync bot unassigned tbg Feb 20, 2023

exalate-issue-sync bot removed the T-disaster-recovery label Feb 20, 2023

exalate-issue-sync bot assigned tbg and unassigned stevendanna Feb 20, 2023

This comment was marked as duplicate.

Sign in to view

matthewtodd mentioned this issue Feb 23, 2023

sqlstats: include regions in statement_statistics #95449

Merged

tbg mentioned this issue Feb 24, 2023

kvserver: finished proposal inserted into map #97605

Closed

tbg assigned stevendanna and unassigned tbg Feb 27, 2023

tbg removed the T-kv-replication label Feb 27, 2023

stevendanna closed this as completed Mar 3, 2023

github-project-automation bot added this to Disaster Recovery Backlog Aug 28, 2024

github-project-automation bot moved this to Done in Disaster Recovery Backlog Aug 28, 2024

ccl/backupccl: TestDataDriven failed #96149

ccl/backupccl: TestDataDriven failed #96149

Comments

cockroach-teamcity commented Jan 29, 2023 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Jan 29, 2023

cockroach-teamcity commented Jan 29, 2023

cockroach-teamcity commented Jan 29, 2023

cockroach-teamcity commented Jan 30, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

cockroach-teamcity commented Jan 31, 2023

blathers-crl bot commented Feb 20, 2023

tbg commented Feb 20, 2023 • edited Loading

tbg commented Feb 20, 2023

This comment was marked as duplicate.

cockroach-teamcity commented Feb 22, 2023

cockroach-teamcity commented Feb 23, 2023

cockroach-teamcity commented Feb 24, 2023

cockroach-teamcity commented Feb 24, 2023

cockroach-teamcity commented Feb 24, 2023

cockroach-teamcity commented Feb 24, 2023

tbg commented Feb 24, 2023

cockroach-teamcity commented Feb 24, 2023

tbg commented Feb 27, 2023

stevendanna commented Mar 3, 2023

cockroach-teamcity commented Jan 29, 2023 •

edited by cockroach-jira-scripts

Loading

tbg commented Feb 20, 2023 •

edited

Loading