roachtest: re-enable admission control fairness tests #82033
Assignees
Labels
branch-master
Failures and bugs on the master branch.
C-test-failure
Broken test (automatically or manually discovered).
O-roachtest
O-robot
Originated from a bot.
T-sql-queries
SQL Queries Team
Comments
cockroach-teamcity
added
branch-master
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ e6815947a050e32f21e983aa30dc74ab2a247af3: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/admission failed with artifacts on master @ 1cea73c8a18623949b81705eb5f75179e6cd8d86: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ 1cea73c8a18623949b81705eb5f75179e6cd8d86: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ 2181204e9c7ac6b316573073b6b8010f43920f8b: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ 90b5db50e1e1cdb0315d8b094081d261e6dcb336: |
|
Seems the kvserver is falling over w/o admission control. I will adjust test to allow failure on no-admission cases. |
cucaroach
added a commit
to cucaroach/cockroach
that referenced
this issue
Jun 9, 2022
Fixes: cockroachdb#82033 In further validation of admission control the kvserver was falling over with super high concurrency of the concurrency-skew workload. Dial it back to be inline with "same" workload, ie 1000 for kv and 200 for store. Release note: None
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ 45ee8026c1997df738ee184fe6a606e927c0a8c5: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ 87da966cbc15e058591d80c477c78d6059483a5c: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ b0a8d28feb8befa417933d0a2ee99511b16414a1: Parameters: |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
|
roachtest.multitenant/fairness/kv/concurrency-skew/admission failed with artifacts on master @ 8d34ef1ea15850ee1c70470610b6652df4c317de: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ 8d34ef1ea15850ee1c70470610b6652df4c317de: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/admission failed with artifacts on master @ 8d34ef1ea15850ee1c70470610b6652df4c317de: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ 8d34ef1ea15850ee1c70470610b6652df4c317de: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/admission failed with artifacts on master @ 8d34ef1ea15850ee1c70470610b6652df4c317de: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ 8d34ef1ea15850ee1c70470610b6652df4c317de: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ 377b113bd7e7b679c55a0f93e0d8ab5dbb6f2e97: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/admission failed with artifacts on master @ 13cb2f6c40e3146fed8d931f65f89da9b42ce2c1: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ 13cb2f6c40e3146fed8d931f65f89da9b42ce2c1: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/admission failed with artifacts on master @ 7a1ced4a49d28f66f381e78733bcab510e2cbafe: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ 7a1ced4a49d28f66f381e78733bcab510e2cbafe: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/admission failed with artifacts on master @ 7af6c0d3e2085cafda90d5f0b15d06f6a360f154: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ 7af6c0d3e2085cafda90d5f0b15d06f6a360f154: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/admission failed with artifacts on master @ 03d5260a467b8640de41ee08054c1ce05f91966c: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ 03d5260a467b8640de41ee08054c1ce05f91966c: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/admission failed with artifacts on master @ 88789feea153b61b379c0240a07482bdfaf0c932: Parameters: |
|
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ 88789feea153b61b379c0240a07482bdfaf0c932: Parameters: |
cucaroach
added a commit
to cucaroach/cockroach
that referenced
this issue
Jul 8, 2022
Fixes: cockroachdb#82033 In further validation of admission control the kvserver was falling over with super high concurrency of the concurrency-skew workload. Dial it back to be inline with "same" workload, ie 1000 for kv and 200 for store. Release note: None
|
@cucaroach is this a release blocker? |
|
No, the test only malfunctions when admission control is off, so its more of a release encourager ;-) |
rytaft
removed
the
release-blocker
rytaft
changed the title
craig bot
pushed a commit
that referenced
this issue
Dec 5, 2022
89721: multitenant: re-enable admission control fairness tests r=irfansharif a=cucaroach Previously these tests were disabled for being flakey. Re-enable them and increase tenant resource limits to prevent throughput collapse, not sure why this wasn't an issue originally. Also disable running the tests w/o admission control as that mode is flakey and no longer of interest. Also includes some commented out code to attempt to use prometheus graphana, I couldn't get it to work but its probably close. Fixes: #82033, #83994 Release note: None 91324: server: implement unified status server r=knz,abarganier a=dhartunian Previously, we had a separate tenant status server that implemented a subset of status server RPC handlers for SQL tenants to use. This commit modifies the existing status server to create a single implementation that can be shared between app and system tenants. There are two primary reasons why the two implementations diverged: 1. The tenant server simply does not have many capabilities that would allow it to serve certain StatusServer requests. For example: gossip. This is simple to reconcile as certain RPCs will return errors. Some additional work is necessary to ensure that tenant implementations don't panic and return proper responses. Previously, this was accomplished via the base implementation that would return "Unimplemented" errors for all handlers, now we will have to modify our handlers to know when they are run as tenants. 2. Requests that require fan-out to either nodes in a cluster or instances in a tenant, require different code to execute. This has now been moved behind an interface called `ServerIterator` that has two implementations: one for nodes, and another for tenant instances. Once we have the admin server migrated as well, the full API V2 server can be implemented on tenants and we should have simpler feature parity between the two. Contributes to: #80789 Epic: [CRDB-17356](https://cockroachlabs.atlassian.net/browse/CRDB-17356) Release note: None 91991: backupccl: issue protected timestamps during on restore spans r=adityamaru a=msbutler Fixes #91148 Release note: None 93008: rowexec: high frequency cancel checking for row exec engine r=DrewKimball,yuzefovich a=msirek Informs #92753 The row execution engine is slower than the vectorized one, and any additional slowdowns caused by contention or other factors may make the cancel checker unresponsive because each call to `Check()` could occur in 350 ms or longer intervals. This can impact SQLSmith tests which expect a 1 minute statement timeout to be honored, timing out the test with error after 5 minutes have elapsed. The solution is to increase the frequency of the cancel checker for calls to `Check()` from the row engine from once every 1024 calls to once every 128 calls. Release note: None 93063: roachtest: update version map for 22.2.0 r=ZhouXing19 a=ZhouXing19 links epic https://cockroachlabs.atlassian.net/browse/REL-184 Release note: None 93069: vendor: bump Pebble to 4a63cdb3a71e r=coolcom200 a=jbowens ``` 4a63cdb3 crossversion: gracefully handle parallel test failures 0fd6d402 docs: update virtual sstables RFC filename, status a08baf44 ci: temporarily skip the linux-race job a3c599e2 crossversion: allow run dir to not exist fb84a7b8 db: change LazyFetcher.ValueFetcher to an interface 8e5e7973 db: make EnableValueBlocks dynamically configurable 7d9a5b2e db: read path for values in value blocks 936e011b rfc: virtual sstables in the ingestion path fcf9e404 internal/rangekey: avoid Transform allocation fece1a6f db: use bytealloc.A for key buffering ec94ead4 internal/rangekey: reuse merging buffers 630e6e90 internal/rangekey: reuse defragmenting buffers 4613f12b db: reuse RangeKeyData slice 6ee5cca6 vfs: fix typo b9289d76 base: add LazyValue.TryGetShortAttribute 87eccabb metamorphic: prevent directory collisions ``` Release note: None Epic: None 93085: ptcache: use simpler TestServer for tests r=andreimatei a=andreimatei These tests were creating a 1-node TestCluster instead of a simpler TestServer for no apparent reason. This patch switches to TestServer. Besides being more straight-forward, the TestServer is better because the test is doing funky stuff, combining the server with an external Cache using the server's stopper. Before, it was using the cluster's stopper, which is different from the server's stopper, which in turn was causing problems because their tracers are different. Release note: None Epic: None Co-authored-by: Tommy Reilly <[email protected]> Co-authored-by: irfan sharif <[email protected]> Co-authored-by: David Hartunian <[email protected]> Co-authored-by: Michael Butler <[email protected]> Co-authored-by: Mark Sirek <[email protected]> Co-authored-by: Jane Xing <[email protected]> Co-authored-by: Jackson Owens <[email protected]> Co-authored-by: Andrei Matei <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
roachtest.multitenant/fairness/kv/concurrency-skew/no-admission failed with artifacts on master @ e6815947a050e32f21e983aa30dc74ab2a247af3:
Help
See: roachtest README
See: How To Investigate (internal)
This test on roachdash | Improve this report!
Jira issue: CRDB-16176
The text was updated successfully, but these errors were encountered: