Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: kv/gracefuldraining/nodes=3 failed #53725

Closed
cockroach-teamcity opened this issue Sep 1, 2020 · 16 comments · Fixed by #58419
Closed

roachtest: kv/gracefuldraining/nodes=3 failed #53725

cockroach-teamcity opened this issue Sep 1, 2020 · 16 comments · Fixed by #58419
Assignees
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@ff3749579c10bb948e89598a5b58bfdcfeb432be:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:453,test_runner.go:754: QPS of 416.40 at time 2020-09-01 06:08:00 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1598940470000000000 Value:0.9} {TimestampNanos:1598940480000000000 Value:416.40000000000003} {TimestampNanos:1598940490000000000 Value:999.8} {TimestampNanos:1598940500000000000 Value:998.9000000000001} {TimestampNanos:1598940510000000000 Value:1000.3000000000001} {TimestampNanos:1598940520000000000 Value:1000.1000000000001} {TimestampNanos:1598940530000000000 Value:999.7} {TimestampNanos:1598940540000000000 Value:998.4000000000001} {TimestampNanos:1598940550000000000 Value:999.4000000000001} {TimestampNanos:1598940560000000000 Value:998.8} {TimestampNanos:1598940570000000000 Value:998.7} {TimestampNanos:1598940580000000000 Value:998.1} {TimestampNanos:1598940590000000000 Value:998.5} {TimestampNanos:1598940600000000000 Value:997.6} {TimestampNanos:1598940610000000000 Value:999.3000000000001} {TimestampNanos:1598940620000000000 Value:999.9000000000001} {TimestampNanos:1598940630000000000 Value:1000.2} {TimestampNanos:1598940640000000000 Value:1000} {TimestampNanos:1598940650000000000 Value:999.9000000000001} {TimestampNanos:1598940660000000000 Value:1000.3000000000001} {TimestampNanos:1598940670000000000 Value:999.6} {TimestampNanos:1598940680000000000 Value:999.5} {TimestampNanos:1598940690000000000 Value:997.9000000000001} {TimestampNanos:1598940700000000000 Value:999.6000000000001} {TimestampNanos:1598940710000000000 Value:999.5} {TimestampNanos:1598940720000000000 Value:999.7} {TimestampNanos:1598940730000000000 Value:998.8000000000001}]

More

Artifacts: /kv/gracefuldraining/nodes=3

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity added branch-release-20.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 1, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.2 milestone Sep 1, 2020
@knz knz self-assigned this Sep 1, 2020
@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@e9efaa35e15b32979f9deb3914ddd7b813558dd1:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:453,test_runner.go:754: QPS of 390.00 at time 2020-09-03 06:07:30 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1599113240000000000 Value:0.9} {TimestampNanos:1599113250000000000 Value:390} {TimestampNanos:1599113260000000000 Value:999.6} {TimestampNanos:1599113270000000000 Value:999.7} {TimestampNanos:1599113280000000000 Value:1000.7} {TimestampNanos:1599113290000000000 Value:999.3} {TimestampNanos:1599113300000000000 Value:999.1000000000001} {TimestampNanos:1599113310000000000 Value:999.6} {TimestampNanos:1599113320000000000 Value:999} {TimestampNanos:1599113330000000000 Value:999.8000000000001} {TimestampNanos:1599113340000000000 Value:998.6} {TimestampNanos:1599113350000000000 Value:999.4000000000001} {TimestampNanos:1599113360000000000 Value:996.8000000000001} {TimestampNanos:1599113370000000000 Value:996.9000000000001} {TimestampNanos:1599113380000000000 Value:998.8} {TimestampNanos:1599113390000000000 Value:998.7} {TimestampNanos:1599113400000000000 Value:1000.3000000000001} {TimestampNanos:1599113410000000000 Value:999} {TimestampNanos:1599113420000000000 Value:1000} {TimestampNanos:1599113430000000000 Value:1000.3000000000001} {TimestampNanos:1599113440000000000 Value:999} {TimestampNanos:1599113450000000000 Value:999} {TimestampNanos:1599113460000000000 Value:998.6} {TimestampNanos:1599113470000000000 Value:915.2} {TimestampNanos:1599113480000000000 Value:997.6} {TimestampNanos:1599113490000000000 Value:998.9000000000001} {TimestampNanos:1599113500000000000 Value:999}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@73bdcc37e28aeb8d9f71f721dfbe4b69dc14955d:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:453,test_runner.go:754: QPS of 405.20 at time 2020-09-05 06:01:20 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1599285670000000000 Value:0.9} {TimestampNanos:1599285680000000000 Value:405.20000000000005} {TimestampNanos:1599285690000000000 Value:1000.4000000000001} {TimestampNanos:1599285700000000000 Value:998.9000000000001} {TimestampNanos:1599285710000000000 Value:1000.3000000000001} {TimestampNanos:1599285720000000000 Value:1000.6} {TimestampNanos:1599285730000000000 Value:998.2} {TimestampNanos:1599285740000000000 Value:942.7} {TimestampNanos:1599285750000000000 Value:560.7} {TimestampNanos:1599285760000000000 Value:893.3000000000001} {TimestampNanos:1599285770000000000 Value:997.9000000000001} {TimestampNanos:1599285780000000000 Value:998.2} {TimestampNanos:1599285790000000000 Value:993} {TimestampNanos:1599285800000000000 Value:997.5} {TimestampNanos:1599285810000000000 Value:999.2} {TimestampNanos:1599285820000000000 Value:998.7} {TimestampNanos:1599285830000000000 Value:1000.4000000000001} {TimestampNanos:1599285840000000000 Value:1000.4000000000001} {TimestampNanos:1599285850000000000 Value:999.9000000000001} {TimestampNanos:1599285860000000000 Value:1000.8} {TimestampNanos:1599285870000000000 Value:998.5} {TimestampNanos:1599285880000000000 Value:998.8000000000001} {TimestampNanos:1599285890000000000 Value:998.4000000000001} {TimestampNanos:1599285900000000000 Value:998.7} {TimestampNanos:1599285910000000000 Value:999.5} {TimestampNanos:1599285920000000000 Value:998.3} {TimestampNanos:1599285930000000000 Value:999.5}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@273b49d542d897d6f3b4f06d52322cbe77b0b436:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:453,test_runner.go:754: QPS of 657.30 at time 2020-09-09 06:03:30 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1599631400000000000 Value:58.6} {TimestampNanos:1599631410000000000 Value:657.3} {TimestampNanos:1599631420000000000 Value:999.4000000000001} {TimestampNanos:1599631430000000000 Value:999.6} {TimestampNanos:1599631440000000000 Value:996.5} {TimestampNanos:1599631450000000000 Value:999.4000000000001} {TimestampNanos:1599631460000000000 Value:998.4000000000001} {TimestampNanos:1599631470000000000 Value:999.8} {TimestampNanos:1599631480000000000 Value:999.3} {TimestampNanos:1599631490000000000 Value:1000.1000000000001} {TimestampNanos:1599631500000000000 Value:999} {TimestampNanos:1599631510000000000 Value:998.4000000000001} {TimestampNanos:1599631520000000000 Value:997.8} {TimestampNanos:1599631530000000000 Value:969} {TimestampNanos:1599631540000000000 Value:870.1000000000001} {TimestampNanos:1599631550000000000 Value:970.3} {TimestampNanos:1599631560000000000 Value:998.3000000000001} {TimestampNanos:1599631570000000000 Value:999.6000000000001} {TimestampNanos:1599631580000000000 Value:999.3000000000001} {TimestampNanos:1599631590000000000 Value:999.7} {TimestampNanos:1599631600000000000 Value:998.5} {TimestampNanos:1599631610000000000 Value:999} {TimestampNanos:1599631620000000000 Value:999.3} {TimestampNanos:1599631630000000000 Value:998.9000000000001} {TimestampNanos:1599631640000000000 Value:999.5} {TimestampNanos:1599631650000000000 Value:998.3} {TimestampNanos:1599631660000000000 Value:998.2}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@e0256086e4f37cf120a86f8684072e91fee5b692:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:453,test_runner.go:754: QPS of 876.90 at time 2020-09-10 06:13:30 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1599718400000000000 Value:285.8} {TimestampNanos:1599718410000000000 Value:876.9000000000001} {TimestampNanos:1599718420000000000 Value:1000.3000000000001} {TimestampNanos:1599718430000000000 Value:999.8000000000001} {TimestampNanos:1599718440000000000 Value:1000.2} {TimestampNanos:1599718450000000000 Value:1000.2} {TimestampNanos:1599718460000000000 Value:998.7} {TimestampNanos:1599718470000000000 Value:999.5} {TimestampNanos:1599718480000000000 Value:999.5} {TimestampNanos:1599718490000000000 Value:999.6000000000001} {TimestampNanos:1599718500000000000 Value:999} {TimestampNanos:1599718510000000000 Value:999.2} {TimestampNanos:1599718520000000000 Value:999} {TimestampNanos:1599718530000000000 Value:984.2} {TimestampNanos:1599718540000000000 Value:999.8} {TimestampNanos:1599718550000000000 Value:1000.3000000000001} {TimestampNanos:1599718560000000000 Value:999.7} {TimestampNanos:1599718570000000000 Value:1000.3000000000001} {TimestampNanos:1599718580000000000 Value:999.4000000000001} {TimestampNanos:1599718590000000000 Value:999.8} {TimestampNanos:1599718600000000000 Value:998.9000000000001} {TimestampNanos:1599718610000000000 Value:999.4000000000001} {TimestampNanos:1599718620000000000 Value:999.4000000000001} {TimestampNanos:1599718630000000000 Value:999.6} {TimestampNanos:1599718640000000000 Value:999.1000000000001} {TimestampNanos:1599718650000000000 Value:999.4000000000001} {TimestampNanos:1599718660000000000 Value:999.9000000000001}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@63cd4307bf21758d0d6c5663b39b8a73020843fd:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:453,test_runner.go:754: QPS of 326.50 at time 2020-09-12 06:06:50 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1599890800000000000 Value:1} {TimestampNanos:1599890810000000000 Value:326.5} {TimestampNanos:1599890820000000000 Value:1000} {TimestampNanos:1599890830000000000 Value:999.4000000000001} {TimestampNanos:1599890840000000000 Value:989.1000000000001} {TimestampNanos:1599890850000000000 Value:905.3000000000001} {TimestampNanos:1599890860000000000 Value:999.5} {TimestampNanos:1599890870000000000 Value:1000} {TimestampNanos:1599890880000000000 Value:998.9000000000001} {TimestampNanos:1599890890000000000 Value:998.4000000000001} {TimestampNanos:1599890900000000000 Value:998.7} {TimestampNanos:1599890910000000000 Value:997.8000000000001} {TimestampNanos:1599890920000000000 Value:999.3000000000001} {TimestampNanos:1599890930000000000 Value:997.8000000000001} {TimestampNanos:1599890940000000000 Value:999.1} {TimestampNanos:1599890950000000000 Value:998.7} {TimestampNanos:1599890960000000000 Value:999.6000000000001} {TimestampNanos:1599890970000000000 Value:1000} {TimestampNanos:1599890980000000000 Value:999.3000000000001} {TimestampNanos:1599890990000000000 Value:999.4000000000001} {TimestampNanos:1599891000000000000 Value:999} {TimestampNanos:1599891010000000000 Value:999} {TimestampNanos:1599891020000000000 Value:998.4000000000001} {TimestampNanos:1599891030000000000 Value:999.4000000000001} {TimestampNanos:1599891040000000000 Value:998.5} {TimestampNanos:1599891050000000000 Value:998} {TimestampNanos:1599891060000000000 Value:999}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@9910e7188c024120ecf8d0ffe64c8e58c2a4b3c0:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:446,test_runner.go:754: QPS of 331.60 at time 2020-09-16 05:59:40 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1600235970000000000 Value:1} {TimestampNanos:1600235980000000000 Value:331.6} {TimestampNanos:1600235990000000000 Value:1000.2} {TimestampNanos:1600236000000000000 Value:999.7} {TimestampNanos:1600236010000000000 Value:999.9000000000001} {TimestampNanos:1600236020000000000 Value:1000.2} {TimestampNanos:1600236030000000000 Value:998.9000000000001} {TimestampNanos:1600236040000000000 Value:999.9000000000001} {TimestampNanos:1600236050000000000 Value:998.6000000000001} {TimestampNanos:1600236060000000000 Value:973.1} {TimestampNanos:1600236070000000000 Value:998.5} {TimestampNanos:1600236080000000000 Value:998.9000000000001} {TimestampNanos:1600236090000000000 Value:998} {TimestampNanos:1600236100000000000 Value:997.1000000000001} {TimestampNanos:1600236110000000000 Value:997.5} {TimestampNanos:1600236120000000000 Value:999.3} {TimestampNanos:1600236130000000000 Value:999.2} {TimestampNanos:1600236140000000000 Value:998.8000000000001} {TimestampNanos:1600236150000000000 Value:1000.4000000000001} {TimestampNanos:1600236160000000000 Value:999.9000000000001} {TimestampNanos:1600236170000000000 Value:999.4000000000001} {TimestampNanos:1600236180000000000 Value:998.3000000000001} {TimestampNanos:1600236190000000000 Value:998.2} {TimestampNanos:1600236200000000000 Value:998.6} {TimestampNanos:1600236210000000000 Value:994.8} {TimestampNanos:1600236220000000000 Value:997.1000000000001} {TimestampNanos:1600236230000000000 Value:997.7}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@56a0bd4a62fa28d52521e51996958965b3c914d2:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:446,test_runner.go:754: QPS of 809.00 at time 2020-09-18 06:10:10 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1600409280000000000 Value:441.50000000000006} {TimestampNanos:1600409290000000000 Value:999.9000000000001} {TimestampNanos:1600409300000000000 Value:999.8000000000001} {TimestampNanos:1600409310000000000 Value:1000.2} {TimestampNanos:1600409320000000000 Value:999.8000000000001} {TimestampNanos:1600409330000000000 Value:1000} {TimestampNanos:1600409340000000000 Value:998.1000000000001} {TimestampNanos:1600409350000000000 Value:999.3000000000001} {TimestampNanos:1600409360000000000 Value:998.7} {TimestampNanos:1600409370000000000 Value:997.4000000000001} {TimestampNanos:1600409380000000000 Value:997.8000000000001} {TimestampNanos:1600409390000000000 Value:999.4000000000001} {TimestampNanos:1600409400000000000 Value:995.3000000000001} {TimestampNanos:1600409410000000000 Value:809} {TimestampNanos:1600409420000000000 Value:998.8000000000001} {TimestampNanos:1600409430000000000 Value:1000.3} {TimestampNanos:1600409440000000000 Value:998.9000000000001} {TimestampNanos:1600409450000000000 Value:1000.1000000000001} {TimestampNanos:1600409460000000000 Value:999.5} {TimestampNanos:1600409470000000000 Value:999.3000000000001} {TimestampNanos:1600409480000000000 Value:999.4000000000001} {TimestampNanos:1600409490000000000 Value:999.5} {TimestampNanos:1600409500000000000 Value:999.1000000000001} {TimestampNanos:1600409510000000000 Value:998.8000000000001} {TimestampNanos:1600409520000000000 Value:998.9000000000001} {TimestampNanos:1600409530000000000 Value:998.8} {TimestampNanos:1600409540000000000 Value:999.6000000000001}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@6f671e63f52f277b79faae939a11bb746b413c40:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:446,test_runner.go:754: QPS of 802.30 at time 2020-09-19 06:11:00 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1600495730000000000 Value:325.6} {TimestampNanos:1600495740000000000 Value:999.5} {TimestampNanos:1600495750000000000 Value:999.9000000000001} {TimestampNanos:1600495760000000000 Value:999.8} {TimestampNanos:1600495770000000000 Value:999.8000000000001} {TimestampNanos:1600495780000000000 Value:999.1000000000001} {TimestampNanos:1600495790000000000 Value:1000.2} {TimestampNanos:1600495800000000000 Value:999.2} {TimestampNanos:1600495810000000000 Value:998.2} {TimestampNanos:1600495820000000000 Value:997.5} {TimestampNanos:1600495830000000000 Value:997.9000000000001} {TimestampNanos:1600495840000000000 Value:997} {TimestampNanos:1600495850000000000 Value:998.7} {TimestampNanos:1600495860000000000 Value:802.3000000000001} {TimestampNanos:1600495870000000000 Value:647.5} {TimestampNanos:1600495880000000000 Value:999.9000000000001} {TimestampNanos:1600495890000000000 Value:1000.7} {TimestampNanos:1600495900000000000 Value:999.8000000000001} {TimestampNanos:1600495910000000000 Value:1000.9000000000001} {TimestampNanos:1600495920000000000 Value:998.9000000000001} {TimestampNanos:1600495930000000000 Value:998.8} {TimestampNanos:1600495940000000000 Value:998.3} {TimestampNanos:1600495950000000000 Value:998.7} {TimestampNanos:1600495960000000000 Value:998.5} {TimestampNanos:1600495970000000000 Value:998.3000000000001} {TimestampNanos:1600495980000000000 Value:998.9000000000001} {TimestampNanos:1600495990000000000 Value:999.6000000000001}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@85c8a57e88141fc99ce00aa4a3e6cc5079a1246a:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:446,test_runner.go:755: QPS of 688.40 at time 2020-09-22 06:07:10 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1600754640000000000 Value:421.8} {TimestampNanos:1600754650000000000 Value:999.7} {TimestampNanos:1600754660000000000 Value:1000.7} {TimestampNanos:1600754670000000000 Value:999.5} {TimestampNanos:1600754680000000000 Value:1000} {TimestampNanos:1600754690000000000 Value:999} {TimestampNanos:1600754700000000000 Value:999.2} {TimestampNanos:1600754710000000000 Value:998.8000000000001} {TimestampNanos:1600754720000000000 Value:999.7} {TimestampNanos:1600754730000000000 Value:999.5} {TimestampNanos:1600754740000000000 Value:996.6} {TimestampNanos:1600754750000000000 Value:998.9000000000001} {TimestampNanos:1600754760000000000 Value:997.7} {TimestampNanos:1600754770000000000 Value:999} {TimestampNanos:1600754780000000000 Value:998.7} {TimestampNanos:1600754790000000000 Value:999} {TimestampNanos:1600754800000000000 Value:1000} {TimestampNanos:1600754810000000000 Value:1000.3000000000001} {TimestampNanos:1600754820000000000 Value:1000.6000000000001} {TimestampNanos:1600754830000000000 Value:688.4000000000001} {TimestampNanos:1600754840000000000 Value:974.1} {TimestampNanos:1600754850000000000 Value:998.5} {TimestampNanos:1600754860000000000 Value:999.2} {TimestampNanos:1600754870000000000 Value:1000.1} {TimestampNanos:1600754880000000000 Value:999} {TimestampNanos:1600754890000000000 Value:999.7} {TimestampNanos:1600754900000000000 Value:1000}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@85c8a57e88141fc99ce00aa4a3e6cc5079a1246a:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:446,test_runner.go:755: QPS of 427.70 at time 2020-09-22 08:45:30 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1600764320000000000 Value:0.9} {TimestampNanos:1600764330000000000 Value:427.70000000000005} {TimestampNanos:1600764340000000000 Value:1000.6000000000001} {TimestampNanos:1600764350000000000 Value:1000.3000000000001} {TimestampNanos:1600764360000000000 Value:999.8} {TimestampNanos:1600764370000000000 Value:1000.5} {TimestampNanos:1600764380000000000 Value:998.6000000000001} {TimestampNanos:1600764390000000000 Value:999.9000000000001} {TimestampNanos:1600764400000000000 Value:999.3000000000001} {TimestampNanos:1600764410000000000 Value:1000.4000000000001} {TimestampNanos:1600764420000000000 Value:998.4000000000001} {TimestampNanos:1600764430000000000 Value:999.6000000000001} {TimestampNanos:1600764440000000000 Value:998.3} {TimestampNanos:1600764450000000000 Value:998.8000000000001} {TimestampNanos:1600764460000000000 Value:1000} {TimestampNanos:1600764470000000000 Value:999.4000000000001} {TimestampNanos:1600764480000000000 Value:999.9000000000001} {TimestampNanos:1600764490000000000 Value:999.8} {TimestampNanos:1600764500000000000 Value:1000} {TimestampNanos:1600764510000000000 Value:999.9000000000001} {TimestampNanos:1600764520000000000 Value:999} {TimestampNanos:1600764530000000000 Value:998.8000000000001} {TimestampNanos:1600764540000000000 Value:999.8000000000001} {TimestampNanos:1600764550000000000 Value:998.9000000000001} {TimestampNanos:1600764560000000000 Value:999.2} {TimestampNanos:1600764570000000000 Value:998.6000000000001} {TimestampNanos:1600764580000000000 Value:1000.1}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@tbg tbg removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Sep 22, 2020
@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@1ee84232a825c9fd69437c00c1dbe2dfb950d009:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:446,test_runner.go:755: QPS of 672.10 at time 2020-09-25 06:08:50 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1601014050000000000 Value:336.9} {TimestampNanos:1601014060000000000 Value:1000} {TimestampNanos:1601014070000000000 Value:1000} {TimestampNanos:1601014080000000000 Value:1000} {TimestampNanos:1601014090000000000 Value:999.8000000000001} {TimestampNanos:1601014100000000000 Value:999.6} {TimestampNanos:1601014110000000000 Value:1000} {TimestampNanos:1601014120000000000 Value:984} {TimestampNanos:1601014130000000000 Value:672.1} {TimestampNanos:1601014140000000000 Value:998.9000000000001} {TimestampNanos:1601014150000000000 Value:998.9000000000001} {TimestampNanos:1601014160000000000 Value:998.9000000000001} {TimestampNanos:1601014170000000000 Value:999} {TimestampNanos:1601014180000000000 Value:999.7} {TimestampNanos:1601014190000000000 Value:998.9000000000001} {TimestampNanos:1601014200000000000 Value:1000.5} {TimestampNanos:1601014210000000000 Value:998.9000000000001} {TimestampNanos:1601014220000000000 Value:1000.4000000000001} {TimestampNanos:1601014230000000000 Value:999.6} {TimestampNanos:1601014240000000000 Value:999.1} {TimestampNanos:1601014250000000000 Value:998.6000000000001} {TimestampNanos:1601014260000000000 Value:998.9000000000001} {TimestampNanos:1601014270000000000 Value:999.5} {TimestampNanos:1601014280000000000 Value:998.9000000000001} {TimestampNanos:1601014290000000000 Value:999.3000000000001} {TimestampNanos:1601014300000000000 Value:999.5} {TimestampNanos:1601014310000000000 Value:998.6}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@a84063ed9e7e3c43c710534629399bb957f17cca:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:446,test_runner.go:755: QPS of 759.90 at time 2020-09-26 06:07:50 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1601100460000000000 Value:162.3} {TimestampNanos:1601100470000000000 Value:759.9000000000001} {TimestampNanos:1601100480000000000 Value:999.7} {TimestampNanos:1601100490000000000 Value:999.9000000000001} {TimestampNanos:1601100500000000000 Value:1000.1} {TimestampNanos:1601100510000000000 Value:999.9000000000001} {TimestampNanos:1601100520000000000 Value:998.8000000000001} {TimestampNanos:1601100530000000000 Value:999.9000000000001} {TimestampNanos:1601100540000000000 Value:998.7} {TimestampNanos:1601100550000000000 Value:999.8} {TimestampNanos:1601100560000000000 Value:999.5} {TimestampNanos:1601100570000000000 Value:998} {TimestampNanos:1601100580000000000 Value:991.9000000000001} {TimestampNanos:1601100590000000000 Value:986.8000000000001} {TimestampNanos:1601100600000000000 Value:997} {TimestampNanos:1601100610000000000 Value:1000.4000000000001} {TimestampNanos:1601100620000000000 Value:998.5} {TimestampNanos:1601100630000000000 Value:999.4000000000001} {TimestampNanos:1601100640000000000 Value:999.7} {TimestampNanos:1601100650000000000 Value:999.4000000000001} {TimestampNanos:1601100660000000000 Value:999.7} {TimestampNanos:1601100670000000000 Value:998.9000000000001} {TimestampNanos:1601100680000000000 Value:996.8} {TimestampNanos:1601100690000000000 Value:993.6000000000001} {TimestampNanos:1601100700000000000 Value:997} {TimestampNanos:1601100710000000000 Value:999.7} {TimestampNanos:1601100720000000000 Value:999.2}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@5a4e41010f8faa095c88e2e5b142e0b6880c9f44:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:446,test_runner.go:755: QPS of 718.10 at time 2020-09-29 06:09:20 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1601359750000000000 Value:119.9} {TimestampNanos:1601359760000000000 Value:718.1} {TimestampNanos:1601359770000000000 Value:1000.7} {TimestampNanos:1601359780000000000 Value:998.6} {TimestampNanos:1601359790000000000 Value:1001} {TimestampNanos:1601359800000000000 Value:997.9000000000001} {TimestampNanos:1601359810000000000 Value:995} {TimestampNanos:1601359820000000000 Value:999.8} {TimestampNanos:1601359830000000000 Value:997.9000000000001} {TimestampNanos:1601359840000000000 Value:999.9000000000001} {TimestampNanos:1601359850000000000 Value:998.5} {TimestampNanos:1601359860000000000 Value:997.4000000000001} {TimestampNanos:1601359870000000000 Value:998} {TimestampNanos:1601359880000000000 Value:1000} {TimestampNanos:1601359890000000000 Value:999.2} {TimestampNanos:1601359900000000000 Value:999.4000000000001} {TimestampNanos:1601359910000000000 Value:1000.1000000000001} {TimestampNanos:1601359920000000000 Value:999.7} {TimestampNanos:1601359930000000000 Value:1000.3000000000001} {TimestampNanos:1601359940000000000 Value:999.4000000000001} {TimestampNanos:1601359950000000000 Value:998.4000000000001} {TimestampNanos:1601359960000000000 Value:998.9000000000001} {TimestampNanos:1601359970000000000 Value:998.3000000000001} {TimestampNanos:1601359980000000000 Value:998.4000000000001} {TimestampNanos:1601359990000000000 Value:998.8000000000001} {TimestampNanos:1601360000000000000 Value:998.8000000000001} {TimestampNanos:1601360010000000000 Value:998.8000000000001}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@d89343cdb6dcd86584584b098889795620dfd534:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:446,test_runner.go:755: QPS of 826.50 at time 2020-09-30 06:12:30 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1601446290000000000 Value:429.9} {TimestampNanos:1601446300000000000 Value:999.7} {TimestampNanos:1601446310000000000 Value:999.2} {TimestampNanos:1601446320000000000 Value:998} {TimestampNanos:1601446330000000000 Value:1000.2} {TimestampNanos:1601446340000000000 Value:999} {TimestampNanos:1601446350000000000 Value:826.5} {TimestampNanos:1601446360000000000 Value:829} {TimestampNanos:1601446370000000000 Value:998.2} {TimestampNanos:1601446380000000000 Value:997.7} {TimestampNanos:1601446390000000000 Value:997.6000000000001} {TimestampNanos:1601446400000000000 Value:997.5} {TimestampNanos:1601446410000000000 Value:998.2} {TimestampNanos:1601446420000000000 Value:998.9000000000001} {TimestampNanos:1601446430000000000 Value:999.3000000000001} {TimestampNanos:1601446440000000000 Value:1000} {TimestampNanos:1601446450000000000 Value:1000.6} {TimestampNanos:1601446460000000000 Value:999.5} {TimestampNanos:1601446470000000000 Value:1000} {TimestampNanos:1601446480000000000 Value:998.8000000000001} {TimestampNanos:1601446490000000000 Value:998.1} {TimestampNanos:1601446500000000000 Value:997.5} {TimestampNanos:1601446510000000000 Value:998.2} {TimestampNanos:1601446520000000000 Value:998.6} {TimestampNanos:1601446530000000000 Value:999.1} {TimestampNanos:1601446540000000000 Value:999.4000000000001} {TimestampNanos:1601446550000000000 Value:998.8000000000001}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@05fa4f5ac66d1672cc9f33752eea497ea163193d:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:446,test_runner.go:755: QPS of 714.30 at time 2020-10-01 06:14:20 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1601532850000000000 Value:122.7} {TimestampNanos:1601532860000000000 Value:714.3000000000001} {TimestampNanos:1601532870000000000 Value:999.7} {TimestampNanos:1601532880000000000 Value:999.5} {TimestampNanos:1601532890000000000 Value:1000} {TimestampNanos:1601532900000000000 Value:999.3000000000001} {TimestampNanos:1601532910000000000 Value:1000.2} {TimestampNanos:1601532920000000000 Value:999} {TimestampNanos:1601532930000000000 Value:999.7} {TimestampNanos:1601532940000000000 Value:1000.2} {TimestampNanos:1601532950000000000 Value:999.5} {TimestampNanos:1601532960000000000 Value:999.3000000000001} {TimestampNanos:1601532970000000000 Value:999.7} {TimestampNanos:1601532980000000000 Value:998} {TimestampNanos:1601532990000000000 Value:999.6000000000001} {TimestampNanos:1601533000000000000 Value:1000.2} {TimestampNanos:1601533010000000000 Value:1000.1000000000001} {TimestampNanos:1601533020000000000 Value:999.7} {TimestampNanos:1601533030000000000 Value:1000.1000000000001} {TimestampNanos:1601533040000000000 Value:999.7} {TimestampNanos:1601533050000000000 Value:998.8} {TimestampNanos:1601533060000000000 Value:998.9000000000001} {TimestampNanos:1601533070000000000 Value:998.7} {TimestampNanos:1601533080000000000 Value:999.6000000000001} {TimestampNanos:1601533090000000000 Value:998.8} {TimestampNanos:1601533100000000000 Value:998.6000000000001} {TimestampNanos:1601533110000000000 Value:999.2}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/gracefuldraining/nodes=3 failed on release-20.1@406792b58d32dc053008cc129b409252798f509c:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv/gracefuldraining/nodes=3/run_1
	kv.go:446,test_runner.go:755: QPS of 319.80 at time 2020-10-03 06:11:20 +0000 UTC is below minimum allowable QPS of 900.00; entire timeseries: [{TimestampNanos:1601705470000000000 Value:1} {TimestampNanos:1601705480000000000 Value:319.8} {TimestampNanos:1601705490000000000 Value:999.1} {TimestampNanos:1601705500000000000 Value:1000.6} {TimestampNanos:1601705510000000000 Value:999.9000000000001} {TimestampNanos:1601705520000000000 Value:999.5} {TimestampNanos:1601705530000000000 Value:998} {TimestampNanos:1601705540000000000 Value:1000.2} {TimestampNanos:1601705550000000000 Value:997.2} {TimestampNanos:1601705560000000000 Value:998.8} {TimestampNanos:1601705570000000000 Value:999.5} {TimestampNanos:1601705580000000000 Value:999.1} {TimestampNanos:1601705590000000000 Value:998.5} {TimestampNanos:1601705600000000000 Value:997.6} {TimestampNanos:1601705610000000000 Value:999.1000000000001} {TimestampNanos:1601705620000000000 Value:999} {TimestampNanos:1601705630000000000 Value:1000.2} {TimestampNanos:1601705640000000000 Value:997.5} {TimestampNanos:1601705650000000000 Value:999.9000000000001} {TimestampNanos:1601705660000000000 Value:1000.2} {TimestampNanos:1601705670000000000 Value:999.2} {TimestampNanos:1601705680000000000 Value:997.9000000000001} {TimestampNanos:1601705690000000000 Value:998.7} {TimestampNanos:1601705700000000000 Value:998.4000000000001} {TimestampNanos:1601705710000000000 Value:999.7} {TimestampNanos:1601705720000000000 Value:999.6000000000001} {TimestampNanos:1601705730000000000 Value:999.2}]

More

Artifacts: /kv/gracefuldraining/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

tbg added a commit to tbg/cockroach that referenced this issue Oct 6, 2020
It's reliably flaky.

Touches cockroachdb#53811.
Touches cockroachdb#54088.
Touches cockroachdb#53760.
Touches cockroachdb#53725.

Release note: None
jayshrivastava pushed a commit that referenced this issue Oct 8, 2020
It's reliably flaky.

Touches #53811.
Touches #54088.
Touches #53760.
Touches #53725.

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants