ccl/multitenantccl/tenantcostclient: TestConsumption failed #94286

cockroach-teamcity · 2022-12-25T08:16:07Z

ccl/multitenantccl/tenantcostclient.TestConsumption failed with artifacts on release-22.2 @ ea0e966fe05a788fd123a08cec8ef5e935fa5496:

=== RUN   TestConsumption
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/0cce69e026224ea71be99da3223b988f/logTestConsumption57809745
    test_log_scope.go:79: use -show-logs to present logs inline
    tenant_side_test.go:885: usage after write: r_u:35.13038335253964 kv_r_u:5.1303863525390625 read_batches:1 read_requests:34 read_bytes:24929 sql_pods_cpu_seconds:0.09000000000000075 
    tenant_side_test.go:909: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/0cce69e026224ea71be99da3223b988f/logTestConsumption57809745
--- FAIL: TestConsumption (2.23s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

ccl/multitenantccl/tenantcostclient: TestConsumption failed #92450 ccl/multitenantccl/tenantcostclient: TestConsumption failed [C-test-failure O-robot branch-release-22.1]

/cc @cockroachdb/multi-tenant _{This test on roachdash | Improve this report!

Jira issue: CRDB-22797}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2022-12-31T07:51:02Z

ccl/multitenantccl/tenantcostclient.TestConsumption failed with artifacts on release-22.2 @ 07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1:

=== RUN   TestConsumption
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/0cce69e026224ea71be99da3223b988f/logTestConsumption2414672614
    test_log_scope.go:79: use -show-logs to present logs inline
    tenant_side_test.go:885: usage after write: r_u:13.333332000000155 sql_pods_cpu_seconds:0.040000000000000036 
    tenant_side_test.go:909: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/0cce69e026224ea71be99da3223b988f/logTestConsumption2414672614
--- FAIL: TestConsumption (1.74s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

ccl/multitenantccl/tenantcostclient: TestConsumption failed #92450 ccl/multitenantccl/tenantcostclient: TestConsumption failed [C-test-failure O-robot branch-release-22.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-04-28T09:13:02Z

ccl/multitenantccl/tenantcostclient.TestConsumption failed with artifacts on release-22.2 @ f567e817548073868fdb9c0ec942062f4b5007d7:

F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !	github.com/cockroachdb/cockroach/pkg/kv/txn.go:961 +0xb1
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !github.com/cockroachdb/cockroach/pkg/kv.runTxn({0xab759f0, 0xc000762060}, 0x7?, 0xc00192e920)
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !	github.com/cockroachdb/cockroach/pkg/kv/db.go:979 +0x85
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl(0xc0011f74d0, {0xab759f0, 0xc000762060}, 0x4a8ee0?, 0x0?, 0x15?)
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !	github.com/cockroachdb/cockroach/pkg/kv/db.go:942 +0xd4
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(...)
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !	github.com/cockroachdb/cockroach/pkg/kv/db.go:921
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !github.com/cockroachdb/cockroach/pkg/startupmigrations/leasemanager.(*LeaseManager).AcquireLease(0xc002fb3470?, {0xab759f0?, 0xc000762060?}, {0xc00332e240?, 0x15?, 0x18?})
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !	github.com/cockroachdb/cockroach/pkg/startupmigrations/leasemanager/pkg/startupmigrations/leasemanager/lease.go:97 +0x1e5
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !github.com/cockroachdb/cockroach/pkg/startupmigrations.(*Manager).EnsureMigrations(0xc000950280, {0xab759f0, 0xc000762060}, {0x17466c0?, 0xc0?, 0x17?, 0x0?})
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !	github.com/cockroachdb/cockroach/pkg/startupmigrations/migrations.go:523 +0x658
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart(_, {_, _}, _, {{0xab32660, 0xc000d43600}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...)
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1387 +0x1437
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart(0xc00003ad00, {0xab75980, 0xc000086098})
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !	github.com/cockroachdb/cockroach/pkg/server/server.go:1652 +0x5738
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !github.com/cockroachdb/cockroach/pkg/server.(*Server).Start(0x0?, {0xab75980, 0xc000086098})
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !	github.com/cockroachdb/cockroach/pkg/server/server.go:1012 +0x45
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start(0xc00023a580, {0xab75980, 0xc000086098})
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !	github.com/cockroachdb/cockroach/pkg/server/testserver.go:590 +0x67
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer({_, _}, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !	github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:332 +0x1b7
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !github.com/cockroachdb/cockroach/pkg/ccl/multitenantccl/tenantcostclient_test.TestConsumption(0xc001630000)
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !	github.com/cockroachdb/cockroach/pkg/ccl/multitenantccl/tenantcostclient_test/pkg/ccl/multitenantccl/tenantcostclient/tenant_side_test.go:849 +0x11a
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !testing.tRunner(0xc001630000, 0x82015f0)
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !	GOROOT/src/testing/testing.go:1446 +0x217
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !created by testing.(*T).Run
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !	GOROOT/src/testing/testing.go:1493 +0x75e
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !For more context, check log files in: /artifacts/tmp/_tmp/0cce69e026224ea71be99da3223b988f/logTestConsumption2007468078
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !****************************************************************************
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !This node experienced a fatal error (printed above), and as a result the
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !process is terminating.
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !will try to determine the root cause, recommend next steps, and we can
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !improve CockroachDB based on your report.
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !Please submit a crash report by following the instructions here:
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !If you would rather not post publicly, please contact us directly at:
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !    [email protected]
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !
F230428 09:13:01.369998 184 util/startup/retry.go:180  [n1] 1 !The Cockroach Labs team appreciates your feedback.

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

ccl/multitenantccl/tenantcostclient: TestConsumption failed #92450 ccl/multitenantccl/tenantcostclient: TestConsumption failed [C-test-failure O-robot branch-release-22.1]

_{This test on roachdash | Improve this report!}

aliher1911 · 2023-04-28T12:02:55Z

I'm not sure why TC bundled this failure together. Last test failure is caused by startup check failure, but previous ones are genuinely different and not KV related.
I'll remove KV from it and keep as it is.

KV failure was caused by same as #102441

kvoli · 2023-05-25T19:30:57Z

I'm not sure why TC bundled this failure together. Last test failure is caused by startup check failure, but previous ones are genuinely different and not KV related.
I'll remove KV from it and keep as it is.

Looks like it was still in KV, moved to multi-tenant and removed KV.

cockroach-teamcity · 2023-05-28T07:59:13Z

ccl/multitenantccl/tenantcostclient.TestConsumption failed with artifacts on release-22.2 @ afe2d46600257f6c022ce89168bebdc0023c8215:

=== RUN   TestConsumption
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/0cce69e026224ea71be99da3223b988f/logTestConsumption1792693212
    test_log_scope.go:79: use -show-logs to present logs inline
    tenant_side_test.go:891: usage after write: r_u:289.5164809980471 kv_r_u:202.84982299804688 read_batches:6 read_requests:3 read_bytes:14 write_batches:1 write_requests:18 write_bytes:184806 sql_pods_cpu_seconds:0.2600000000000007 
    tenant_side_test.go:915: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/0cce69e026224ea71be99da3223b988f/logTestConsumption1792693212
--- FAIL: TestConsumption (2.79s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

ccl/multitenantccl/tenantcostclient: TestConsumption failed #92450 ccl/multitenantccl/tenantcostclient: TestConsumption failed [C-test-failure O-robot branch-release-22.1]

_{This test on roachdash | Improve this report!}

knz · 2023-05-28T11:05:20Z

Issues related by rate consumption are now owned by SQL queries.

DrewKimball · 2023-06-06T03:56:53Z

Maybe the problem here is that the test waits for a bucket request, but that isn't necessarily the one triggered by the write in the test when run under stress. If that's the case, SucceedsSoon would probably fix the flake.

This patch adds sleeps to the `TestConsumption` test before retrieving the total consumption in order to ensure that a bucket flush has occurred because of the test query, and not because of some background activity. Fixes cockroachdb#94286 Release note: None

This patch adds retries to the `TestConsumption` test for retrieving the total tenant RU consumption in order to ensure that the delta includes the bucket flush that was triggered by the test query, instead of background activity. Fixes cockroachdb#94286 Release note: None

105552: multitenant: deflake TestConsumption r=yuzefovich a=DrewKimball This patch adds retries to the `TestConsumption` test for retrieving the total tenant RU consumption in order to ensure that the delta includes the bucket flush that was triggered by the test query, instead of background activity. Fixes #94286 Fixes #106572 Release note: None 107719: ui: add latency info to explain plan tab r=maryliag a=maryliag This commit adds columns for latency information (p50, p90, p99, pMax, pMin) on Explain Plan tab under Statement Details page. Fixes #105371 <img width="1707" alt="Screenshot 2023-07-27 at 10 59 41 AM" src="https://github.com/cockroachdb/cockroach/assets/1017486/81a2ce3d-655e-47c1-931b-00e34433b3a1"> Release note (ui change): Add columns for p50, p90, p99 percentiles and latency min and max on Explain Plan tab on Statement Details page. 107732: valueside: use redact type name when can't decode r=yuzefovich a=yuzefovich Informs: #106158 Epic: None Release note: None Co-authored-by: Drew Kimball <[email protected]> Co-authored-by: maryliag <[email protected]> Co-authored-by: Yahor Yuzefovich <[email protected]>

This patch adds retries to the `TestConsumption` test for retrieving the total tenant RU consumption in order to ensure that the delta includes the bucket flush that was triggered by the test query, instead of background activity. Fixes #94286 Fixes #106572 Release note: None

cockroach-teamcity added branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Dec 25, 2022

cockroach-teamcity added this to the 22.2 milestone Dec 25, 2022

cockroach-teamcity mentioned this issue Jan 19, 2023

ccl/multitenantccl/tenantcostclient: TestConsumption failed #92450

Closed

cockroach-teamcity mentioned this issue Jan 26, 2023

ccl/multitenantccl/tenantcostclient: TestConsumption failed #95921

Closed

knz assigned aliher1911 Apr 28, 2023

blathers-crl bot added the T-kv KV Team label Apr 28, 2023

aliher1911 added X-duplicate Closed as a duplicate of another issue. and removed X-duplicate Closed as a duplicate of another issue. labels Apr 28, 2023

aliher1911 removed their assignment Apr 28, 2023

kvoli added T-multitenant Issues owned by the multi-tenant virtual team and removed T-kv KV Team labels May 25, 2023

knz added T-sql-queries SQL Queries Team and removed T-multitenant Issues owned by the multi-tenant virtual team labels May 28, 2023

michae2 assigned mgartner May 30, 2023

mgartner assigned DrewKimball and unassigned mgartner Jun 6, 2023

DrewKimball mentioned this issue Jun 26, 2023

multitenant: deflake TestConsumption #105552

Merged

cockroach-teamcity mentioned this issue Jul 11, 2023

ccl/multitenantccl/tenantcostclient: TestConsumption failed #106572

Closed

mgartner added this to SQL Queries Jul 24, 2023

mgartner moved this to Active in SQL Queries Jul 24, 2023

craig bot closed this as completed in d99cd3d Jul 28, 2023

github-project-automation bot moved this from Active to Done in SQL Queries Jul 28, 2023

blathers-crl bot mentioned this issue Jul 28, 2023

release-22.2: multitenant: deflake TestConsumption #107784

Merged

blathers-crl bot mentioned this issue Jul 28, 2023

release-23.1: multitenant: deflake TestConsumption #107786

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ccl/multitenantccl/tenantcostclient: TestConsumption failed #94286

ccl/multitenantccl/tenantcostclient: TestConsumption failed #94286

cockroach-teamcity commented Dec 25, 2022 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Dec 31, 2022

cockroach-teamcity commented Apr 28, 2023

aliher1911 commented Apr 28, 2023

kvoli commented May 25, 2023

cockroach-teamcity commented May 28, 2023

knz commented May 28, 2023

DrewKimball commented Jun 6, 2023

ccl/multitenantccl/tenantcostclient: TestConsumption failed #94286

ccl/multitenantccl/tenantcostclient: TestConsumption failed #94286

Comments

cockroach-teamcity commented Dec 25, 2022 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Dec 31, 2022

cockroach-teamcity commented Apr 28, 2023

aliher1911 commented Apr 28, 2023

kvoli commented May 25, 2023

cockroach-teamcity commented May 28, 2023

knz commented May 28, 2023

DrewKimball commented Jun 6, 2023

cockroach-teamcity commented Dec 25, 2022 •

edited by cockroach-jira-scripts

Loading