Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: admission-control/multitenant-fairness/read-heavy/skewed failed #97448

Closed
cockroach-teamcity opened this issue Feb 22, 2023 · 28 comments · Fixed by #98785
Closed

roachtest: admission-control/multitenant-fairness/read-heavy/skewed failed #97448

cockroach-teamcity opened this issue Feb 22, 2023 · 28 comments · Fixed by #98785
Assignees
@irfansharif
Labels
A-kv Anything in KV that doesn't belong in a more specific category. branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). GA-blocker O-roachtest O-robot Originated from a bot. T-kv KV Team
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Feb 22, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 286b3e235171a39b8f9910555affcc7ce310741a:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_072327.428700283_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_072327.440312850_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-24702
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Feb 22, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Feb 22, 2023
@blathers-crl blathers-crl bot added the T-kv KV Team label Feb 22, 2023
@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ e028ce5b14505dfd17ef8b13001c0ab8ac811e3c:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_072515.503077385_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_072515.507927573_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 0d3393b0623a5c258b25725f64f3689e2f54667b:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_072113.582847639_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_072113.589806149_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@pav-kv
Copy link
Collaborator

pav-kv commented Feb 24, 2023

Workload fails with a context canceled for some reason.

@pav-kv
Copy link
Collaborator

pav-kv commented Feb 24, 2023

This test has previously failed in workload (with a different error) in #93304.

@irfansharif irfansharif self-assigned this Feb 24, 2023
@pav-kv pav-kv added A-kv Anything in KV that doesn't belong in a more specific category. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels Feb 24, 2023
@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ fca69b046f6389fbc4d2b6db428e27a5d7277773:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_072537.758270627_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_072537.766610340_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 39c06b5a438c01c93ffbfeeefe702d3f9b620eaf:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_071929.290385126_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_071929.302276371_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 13c58f621519794e775b7cfc4d8b557bc99eeca0:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_071911.050349312_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_071911.060769823_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ b0e5507f74c07e13cfda8cda8b9079b457a9f37d:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_071630.537849813_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_071630.542828030_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 21786aa112e6b822858f281c1cc59608987c5c0a:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_071903.350266310_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_071903.361584420_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled
(cluster.go:1956).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 20e2adda3c76c7172dd986c871df0ae9a346918f:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 134)
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ fcea283ebca17a6d923c5d4b0401697438b77dbd:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_072424.375609995_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_072424.397997479_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ cf14ad694ee562676f53e36fa8495206c3aed61f:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_071916.219833192_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_071916.228415434_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 14b43be03c1c246765be17aa15245493a42c401d:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_072257.294484424_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_072257.303609471_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 43a25a09f92408accdeb3cfed1eb35e02d375877:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_072232.068919955_n4_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=750 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_072232.075672980_n4_cockroach-workload-r.log: exit status 1
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 1b162d1b274eec7b307fbbfca7294460bfdef025:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_073003.385331303_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_073003.408553868_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 4dc9e98d8cc1b137ee4cc950a4a42e5895bb890b:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_072446.870589637_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_072446.886828552_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ d4a584e49f0b1ca89738376090939d7669c3b3db:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_072933.886395755_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_072933.894352514_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 5b2a5670cbbe895d76602c230390816e783e0caa:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_073147.460896304_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_073147.469866228_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled
(cluster.go:1968).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ e4924e2b9be4a36d466beab53a80df9241df4783:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_063124.792496039_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_063124.797429089_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 10c1e3e01b7da4cced0478e1bfd711a1c9be9afc:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_063323.462129447_n4_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=750 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_063323.470337974_n4_cockroach-workload-r.log: exit status 1
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 9b43dc50d468171bc9919dc684b7274b63e13da2:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_062717.909898936_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_062717.917818513_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@irfansharif
Copy link
Contributor

This is not a release blocker -- it's the same as #78691 (these tests were recently re-enabled, and we're running into the same errors that made us skip them in the first place). I'll fix, but getting rid of these labels.

err:ERROR: liveness session expired 42.991119ms before transaction (SQLSTATE XXUUU) pid:111314

@irfansharif irfansharif removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Mar 15, 2023
@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ d68f5cdfb25476aaf874bff0bb6515eba02efcd2:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_063305.610518852_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_063305.636437602_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@irfansharif irfansharif mentioned this issue Mar 16, 2023
Closed
@irfansharif irfansharif mentioned this issue Mar 16, 2023
Merged
irfansharif added a commit to irfansharif/cockroach that referenced this issue Mar 16, 2023
@irfansharif
roachtest: de-flake multitenant-fairness/read-heavy/skewed
3e85516 
Fixes cockroachdb#97448 (possibly).
Fixes cockroachdb#78691.

These tests run under severe CPU overload, and we see the workload
getting observing the following errors:

  ERROR: liveness session expired 571.043163ms before transaction

The SQL liveness lease extension work ends up getting severely starved,
despite extending leases by 40s every 5s. It turns out for tenant
SQL liveness work, we were using admissionpb.NormalPri, so such
starvation was possible. This wasn't true for the system tenant where we
bypassed AC altogether.

Release note: None
@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ f0489334a0ee6980a9d365b361d2fce4b2cdc05b:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_062610.265015075_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_062610.270683437_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 6c99966f604f3521acdb925b9f689529ffd46df3:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_061634.177006509_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_061634.185620079_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 53dbb86acb1d48309530181b94838faf937084d3:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_063149.051789751_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_063149.068300755_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ 9acc75317aebfdfe69fb097a8a28e0040c7a67fd:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_062911.478275829_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_062911.487776421_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/multitenant-fairness/read-heavy/skewed failed with artifacts on master @ b89fa2cc4bc1fb9447ab1009b34b6f354f9618f0:

test artifacts and logs in: /artifacts/admission-control/multitenant-fairness/read-heavy/skewed/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_062933.177805369_n5_cockroach-workload-r: ./cockroach workload run kv 'postgres://[email protected]:26259?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' --write-seq=R10000000 --secure --min-block-bytes 5 --max-block-bytes 5 --batch 100 --duration=20m0s --read-percent=95 --concurrency=1000 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_062933.195691374_n5_cockroach-workload-r.log: exit status 1
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled
(cluster.go:1969).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue Mar 21, 2023
@stevendanna @irfansharif
@cucaroach @tbg @matthewtodd
Merge #98207 #98785 #98821 #99109 #99122
eab416d 
98207: sql: add REPLICATION and MANAGETENANT system privileges  r=msbutler a=stevendanna

This adds two new system privileges:

- `REPLICATION`: Allows the user to call the internal functions that
produce a cross-cluster replication stream.

- `MANAGETENANT`: Allows the user to create and manage
  tenants.

A user with the MANAGETENANT privileges is now able to execute
the following statements:

- SHOW TENANT
- SHOW TENANTS
- CREATE TENANT
- CREATE TENANT FROM REPLICATION STREAM
- DROP TENANT (if it is part of an active stream)
- ALTER TENANT

A user with the REPLICATION privileges is able to call the following
functions:

- crdb_internal.start_replication_stream
- crdb_internal.replication_stream_progress
- crdb_internal.stream_partition
- crdb_internal.replication_stream_spec
- crdb_internal.complete_replication_stream

Fixes #95425

Release note: None

98785: roachtest:  de-flake multitenant-fairness/read-heavy/skewed r=irfansharif a=irfansharif

Fixes #97448 (possibly).
Fixes #78691.

These tests run under severe CPU overload, and we see the workload
getting observing the following errors:

  ERROR: liveness session expired 571.043163ms before transaction

The SQL liveness lease extension work ends up getting severely starved,
despite extending leases by 40s every 5s. It turns out for tenant
SQL liveness work, we were using admissionpb.NormalPri, so such
starvation was possible. This wasn't true for the system tenant where we
bypassed AC altogether.

Release note: None

98821: sql: TestRandomSyntaxGeneration fixes r=cucaroach a=cucaroach

### sql: fix some TestRandomSyntaxGeneration bugs

The RSG works by calling format on the AST's it generates so its good
at finding Format bugs.

Fix a missing separator in ShowBackupOptions.  Example:

```
SHOW BACKUP 'family' IN ('string', 'placeholder', 'placeholder', 'placeholder', 'string', 'placeholder', 'string', 'placeholder') WITH incremental_location = 'nullif', privilegesdebug_dump_metadata_sst
```

Fix bad construction in ShowTenant.  Example:

```
SHOW TENANT [B'10010'] WITH REPLICATION STATUS WITH CAPABILITIES
```

Epic: none
Release note: None

### copy: fix copy grammar to match PG

Previously COPY would allow a wide range of syntax in the COPY
TO substatement.  Now like PG we limit it to a few things.

PG grammar is:
```
PreparableStmt:
                        SelectStmt
                        | InsertStmt
                        | UpdateStmt
                        | DeleteStmt
                        | MergeStmt
```

And now we do something similar. This prevents the wheels from coming
off when RSG generates EXPLAIN's in the substatement for instance.

Release note: none
Epic: none


99109: kvserver: fortify TestReplicaClosedTimestamp r=erikgrinaker a=tbg

This test was flaky until ~Feb 2nd. This has since resolved, likely as a
result of some other change, however there's an easy way to make the
test a bit more resilient by widening a critical section.

Closes #93864

Epic: none
Release note: None


99122: ccl/multiregionccl: skip flaky secondary_region test r=matthewtodd a=matthewtodd

Part of #92235.
Part of #98020.

It [flaked][1] this morning, after last night's [other skip][2] landed.

[1]: #98020 (comment)
[2]: #99031

Release note: None

Co-authored-by: Steven Danna <[email protected]>
Co-authored-by: irfan sharif <[email protected]>
Co-authored-by: Tommy Reilly <[email protected]>
Co-authored-by: Tobias Grieger <[email protected]>
Co-authored-by: Matthew Todd <[email protected]>
@craig craig bot closed this as completed in 15dc772 Mar 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@irfansharif irfansharif

Labels
A-kv Anything in KV that doesn't belong in a more specific category. branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). GA-blocker O-roachtest O-robot Originated from a bot. T-kv KV Team
Projects
None yet
Milestone
23.1
Development

Successfully merging a pull request may close this issue.

roachtest: de-flake multitenant-fairness/read-heavy/skewed irfansharif/cockroach
3 participants
@pav-kv @irfansharif @cockroach-teamcity