Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql/gcjob_test: TestSchemaChangeGCJob failed #46797

Closed
cockroach-teamcity opened this issue Mar 31, 2020 · 3 comments
Closed

sql/gcjob_test: TestSchemaChangeGCJob failed #46797

cockroach-teamcity opened this issue Mar 31, 2020 · 3 comments
Assignees
@pbardea
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
20.1

Comments

@cockroach-teamcity
Copy link
Member

(sql/gcjob_test).TestSchemaChangeGCJob failed on master@1320e1360a2cbed8df8fbaa5e7100155999e534e:

I200331 17:02:07.481671 5668 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (merge): node unavailable; try another peer
I200331 17:02:07.481683 5668 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (split): node unavailable; try another peer
I200331 17:02:07.481695 5668 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (replicate): node unavailable; try another peer
I200331 17:02:07.481706 5668 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (replicaGC): node unavailable; try another peer
I200331 17:02:07.481717 5668 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (raftlog): node unavailable; try another peer
I200331 17:02:07.481727 5668 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (raftsnapshot): node unavailable; try another peer
I200331 17:02:07.481737 5668 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (consistencyChecker): node unavailable; try another peer
I200331 17:02:07.481747 5668 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (timeSeriesMaintenance): node unavailable; try another peer
E200331 17:02:07.560516 5731 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
W200331 17:02:07.564895 5713 jobs/registry.go:1215  job 542650144771047425: canceling due to liveness failure
E200331 17:02:07.565012 6848 jobs/registry.go:946  [n1] job 542650144771047425: adoption completed with error job 542650144771047425: node liveness error: restarting in background
W200331 17:02:07.565059 6848 kv/txn.go:603  [n1] failure aborting transaction: node unavailable; try another peer; abort caused by: context canceled
E200331 17:02:07.565079 6848 jobs/registry.go:950  [n1] job 542650144771047425: failed querying status: context canceled
I200331 17:02:07.569754 131 util/stop/stopper.go:539  quiescing
I200331 17:02:07.602089 4039 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (gc): node unavailable; try another peer
I200331 17:02:07.602159 4039 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (merge): node unavailable; try another peer
I200331 17:02:07.602171 4039 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (split): node unavailable; try another peer
I200331 17:02:07.602182 4039 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (replicate): node unavailable; try another peer
I200331 17:02:07.602191 4039 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (replicaGC): node unavailable; try another peer
I200331 17:02:07.602202 4039 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (raftlog): node unavailable; try another peer
I200331 17:02:07.602213 4039 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (raftsnapshot): node unavailable; try another peer
I200331 17:02:07.602224 4039 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (consistencyChecker): node unavailable; try another peer
I200331 17:02:07.602235 4039 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (timeSeriesMaintenance): node unavailable; try another peer
W200331 17:02:07.616940 4055 jobs/registry.go:1215  job 542650141505323009: canceling due to liveness failure
E200331 17:02:07.617085 5420 jobs/registry.go:946  [n1] job 542650141505323009: adoption completed with error job 542650141505323009: node liveness error: restarting in background
W200331 17:02:07.617132 5420 kv/txn.go:603  [n1] failure aborting transaction: node unavailable; try another peer; abort caused by: context canceled
E200331 17:02:07.617196 5420 jobs/registry.go:950  [n1] job 542650141505323009: failed querying status: context canceled
I200331 17:02:07.624414 131 util/stop/stopper.go:539  quiescing
I200331 17:02:07.634086 131 util/stop/stopper.go:539  quiescing
--- FAIL: TestSchemaChangeGCJob (49.64s)
    gc_job_test.go:196: condition failed to evaluate within 45s: query 'SELECT status FROM [SHOW JOBS] WHERE job_id = 542650144771047425': expected:
        succeeded
        
        got:
        running
        
        
        goroutine 131 [running]:
        runtime/debug.Stack(0xc001ef77d8, 0x4205f40, 0xc004d81f80)
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon(0x42ea400, 0xc000103100, 0xc001ef77d8)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37 +0x6b
        github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry(0xc001ef7c48, 0x42ea400, 0xc000103100, 0xc003906680, 0x40, 0xc00404c260, 0x1, 0x1)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:199 +0xde
        github.com/cockroachdb/cockroach/pkg/sql/gcjob_test_test.TestSchemaChangeGCJob(0xc000103100)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob_test/gc_job_test.go:196 +0xfa8
        testing.tRunner(0xc000103100, 0x3c15f48)
        	/usr/local/go/src/testing/testing.go:909 +0xc9
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:960 +0x350
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestSchemaChangeGCJob PKG=./pkg/sql/gcjob_test TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 31, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.1 milestone Mar 31, 2020
@cockroach-teamcity
Copy link
Member Author

(sql/gcjob_test).TestSchemaChangeGCJob failed on master@f8d26e75cc3d83eaf4c47ea279d5b22b94bef42e:

I200331 18:59:37.852380 260 server/status/runtime.go:498  [n1] runtime stats: 182 MiB RSS, 508 goroutines, 40 MiB/67 MiB/97 MiB GO alloc/idle/total, 38 MiB/55 MiB CGO alloc/total, 1192.9 CGO/sec, 14.6/0.9 %(u/s)time, 0.0 %gc (3x), 4.7 MiB/4.7 MiB (r/w)net
W200331 18:59:37.898244 4043 kv/kvserver/store.go:1541  [n1,s1,r1/1:/{Min-System/NodeL…}] could not gossip first range descriptor: node unavailable; try another peer
E200331 18:59:37.944446 4111 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
W200331 18:59:38.005628 4043 kv/kvserver/store.go:1541  [n1,s1,r1/1:/{Min-System/NodeL…}] could not gossip first range descriptor: node unavailable; try another peer
E200331 18:59:38.045750 4111 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
E200331 18:59:38.146742 4111 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
W200331 18:59:38.210642 4043 kv/kvserver/store.go:1541  [n1,s1,r1/1:/{Min-System/NodeL…}] could not gossip first range descriptor: node unavailable; try another peer
E200331 18:59:38.247598 4111 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
E200331 18:59:38.349647 4111 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
E200331 18:59:38.450670 4111 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
E200331 18:59:38.551550 4111 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
I200331 18:59:38.557043 2034 server/status/runtime.go:498  [n1] runtime stats: 182 MiB RSS, 508 goroutines, 48 MiB/61 MiB/97 MiB GO alloc/idle/total, 38 MiB/55 MiB CGO alloc/total, 1156.0 CGO/sec, 14.1/1.0 %(u/s)time, 0.0 %gc (3x), 4.5 MiB/4.5 MiB (r/w)net
I200331 18:59:38.634477 4091 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (gc): node unavailable; try another peer
I200331 18:59:38.634547 4091 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (merge): node unavailable; try another peer
I200331 18:59:38.634562 4091 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (split): node unavailable; try another peer
I200331 18:59:38.634572 4091 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (replicate): node unavailable; try another peer
I200331 18:59:38.634583 4091 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (replicaGC): node unavailable; try another peer
I200331 18:59:38.634593 4091 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (raftlog): node unavailable; try another peer
I200331 18:59:38.634602 4091 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (raftsnapshot): node unavailable; try another peer
I200331 18:59:38.634611 4091 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (consistencyChecker): node unavailable; try another peer
I200331 18:59:38.634621 4091 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (timeSeriesMaintenance): node unavailable; try another peer
W200331 18:59:38.652455 4043 kv/kvserver/store.go:1541  [n1,s1,r1/1:/{Min-System/NodeL…}] could not gossip first range descriptor: node unavailable; try another peer
E200331 18:59:38.652520 4111 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
W200331 18:59:38.675672 4109 jobs/registry.go:1215  job 542673243375894529: canceling due to liveness failure
E200331 18:59:38.675780 4832 jobs/registry.go:946  [n1] job 542673243375894529: adoption completed with error job 542673243375894529: node liveness error: restarting in background
W200331 18:59:38.675819 4832 kv/txn.go:603  [n1] failure aborting transaction: node unavailable; try another peer; abort caused by: context canceled
E200331 18:59:38.675827 4832 jobs/registry.go:950  [n1] job 542673243375894529: failed querying status: context canceled
I200331 18:59:38.680177 14 util/stop/stopper.go:539  quiescing
I200331 18:59:38.690735 14 util/stop/stopper.go:539  quiescing
--- FAIL: TestSchemaChangeGCJob (50.95s)
    gc_job_test.go:196: condition failed to evaluate within 45s: query 'SELECT status FROM [SHOW JOBS] WHERE job_id = 542673246714953729': expected:
        succeeded
        
        got:
        running
        
        
        goroutine 14 [running]:
        runtime/debug.Stack(0xc000e177d8, 0x4206d20, 0xc0047e5fa0)
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon(0x42eb220, 0xc000129b00, 0xc000e177d8)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37 +0x6b
        github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry(0xc000e17c48, 0x42eb220, 0xc000129b00, 0xc002b09a80, 0x40, 0xc003be80a0, 0x1, 0x1)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:199 +0xde
        github.com/cockroachdb/cockroach/pkg/sql/gcjob_test_test.TestSchemaChangeGCJob(0xc000129b00)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob_test/gc_job_test.go:196 +0xfa8
        testing.tRunner(0xc000129b00, 0x3c16c30)
        	/usr/local/go/src/testing/testing.go:909 +0xc9
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:960 +0x350
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestSchemaChangeGCJob PKG=./pkg/sql/gcjob_test TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(sql/gcjob_test).TestSchemaChangeGCJob failed on master@229fcae74dcc2b72b42cf83c13afaab9769ded28:

W200331 21:32:56.516942 9144 kv/txn.go:603  [n1] failure aborting transaction: node unavailable; try another peer; abort caused by: context canceled
E200331 21:32:56.516963 9144 jobs/registry.go:950  [n1] job 542703389726965761: failed querying status: context canceled
I200331 21:32:56.522076 45 util/stop/stopper.go:539  quiescing
E200331 21:32:56.523507 5834 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
I200331 21:32:56.529452 45 util/stop/stopper.go:539  quiescing
I200331 21:32:56.609841 2006 server/status/runtime.go:498  [n1] runtime stats: 201 MiB RSS, 508 goroutines, 82 MiB/37 MiB/113 MiB GO alloc/idle/total, 38 MiB/58 MiB CGO alloc/total, 1508.9 CGO/sec, 16.3/1.1 %(u/s)time, 0.0 %gc (2x), 5.0 MiB/5.0 MiB (r/w)net
E200331 21:32:56.626368 4082 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
E200331 21:32:56.727498 4082 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
E200331 21:32:56.828970 4082 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
E200331 21:32:56.930083 4082 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
E200331 21:32:57.030912 4082 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
E200331 21:32:57.131675 4082 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
E200331 21:32:57.232836 4082 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
I200331 21:32:57.270534 3952 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (gc): node unavailable; try another peer
I200331 21:32:57.270604 3952 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (merge): node unavailable; try another peer
I200331 21:32:57.270616 3952 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (split): node unavailable; try another peer
I200331 21:32:57.270627 3952 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (replicate): node unavailable; try another peer
I200331 21:32:57.270636 3952 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (replicaGC): node unavailable; try another peer
I200331 21:32:57.270646 3952 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (raftlog): node unavailable; try another peer
I200331 21:32:57.270656 3952 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (raftsnapshot): node unavailable; try another peer
I200331 21:32:57.270665 3952 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (consistencyChecker): node unavailable; try another peer
I200331 21:32:57.270676 3952 kv/kvserver/queue.go:578  [n1,s1] rate limited in MaybeAdd (timeSeriesMaintenance): node unavailable; try another peer
E200331 21:32:57.333710 4082 jobs/registry.go:500  error while adopting jobs: failed querying for jobs: adopt-job: node unavailable; try another peer
W200331 21:32:57.382136 4080 jobs/registry.go:1215  job 542703382165028865: canceling due to liveness failure
E200331 21:32:57.382232 5477 jobs/registry.go:946  [n1] job 542703382165028865: adoption completed with error job 542703382165028865: node liveness error: restarting in background
W200331 21:32:57.382278 5477 kv/txn.go:603  [n1] failure aborting transaction: node unavailable; try another peer; abort caused by: context canceled
E200331 21:32:57.382288 5477 jobs/registry.go:950  [n1] job 542703382165028865: failed querying status: context canceled
I200331 21:32:57.385991 45 util/stop/stopper.go:539  quiescing
I200331 21:32:57.408740 45 util/stop/stopper.go:539  quiescing
--- FAIL: TestSchemaChangeGCJob (51.36s)
    gc_job_test.go:196: condition failed to evaluate within 45s: query 'SELECT status FROM [SHOW JOBS] WHERE job_id = 542703389726965761': expected:
        succeeded
        
        got:
        running
        
        
        goroutine 45 [running]:
        runtime/debug.Stack(0xc00368d7d8, 0x4206d40, 0xc00357f4c0)
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon(0x42eb280, 0xc00048f900, 0xc00368d7d8)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37 +0x6b
        github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry(0xc00368dc48, 0x42eb280, 0xc00048f900, 0xc0033b1340, 0x40, 0xc003870760, 0x1, 0x1)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:199 +0xde
        github.com/cockroachdb/cockroach/pkg/sql/gcjob_test_test.TestSchemaChangeGCJob(0xc00048f900)
        	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob_test/gc_job_test.go:196 +0xfa8
        testing.tRunner(0xc00048f900, 0x3c16c70)
        	/usr/local/go/src/testing/testing.go:909 +0xc9
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:960 +0x350
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestSchemaChangeGCJob PKG=./pkg/sql/gcjob_test TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@pbardea
Copy link
Contributor

pbardea commented Apr 1, 2020

Fixed by #46792.

@pbardea pbardea closed this as completed Apr 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pbardea pbardea

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
20.1
Development

No branches or pull requests
2 participants
@pbardea @cockroach-teamcity