Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: splits/load/ycsb/e/nodes=3/obj=cpu failed #120163

Closed
cockroach-teamcity opened this issue Mar 9, 2024 · 3 comments · Fixed by #120361
Closed

roachtest: splits/load/ycsb/e/nodes=3/obj=cpu failed #120163

cockroach-teamcity opened this issue Mar 9, 2024 · 3 comments · Fixed by #120361
Assignees
@nvanbenschoten
Labels
A-testing Testing tools and infrastructure branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-kv KV Team
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 9, 2024 •
edited by cockroach-jira-scripts
Loading

roachtest.splits/load/ycsb/e/nodes=3/obj=cpu failed with artifacts on master @ 72646a555214c0705781e440b9df585d5eea9511:

(monitor.go:154).Wait: monitor failure: 14 splits, expected between 3 and 13 splits (ranges 16 initial 2)
test artifacts and logs in: /artifacts/splits/load/ycsb/e/nodes=3/obj=cpu/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-36514
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Mar 9, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Mar 9, 2024
@nvanbenschoten
Copy link
Member

@kvoli we see 14 splits here when the test configures the maximum number of splits to be 13. Is that cause for concern? Should we just bump the maximumRanges? I'm not sure how scientifically these initial bounds were chosen when they were added in aa95836.

@kvoli
Copy link
Collaborator

kvoli commented Mar 12, 2024

@kvoli we see 14 splits here when the test configures the maximum number of splits to be 13. Is that cause for concern? Should we just bump the maximumRanges? I'm not sure how scientifically these initial bounds were chosen when they were added in aa95836.

They weren't that scientific, we can bump the threshold up by one since I don't think there were any changes that went in to affect the number. The original thresholds were determined experimentally, see this (internal) spreadsheet.

@nvanbenschoten nvanbenschoten removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Mar 12, 2024
@nvanbenschoten nvanbenschoten self-assigned this Mar 12, 2024
@nvanbenschoten
Copy link
Member

Sounds good, I'll send a PR.

@nvanbenschoten nvanbenschoten mentioned this issue Mar 12, 2024
Merged
craig bot pushed a commit that referenced this issue Mar 12, 2024
@msbutler @dt
@aadityasondhi @yuzefovich @rickystewart @nvanbenschoten @jbowens
Merge #85138 #120080 #120342 #120350 #120356 #120359 #120361 #120363
9adee4e 
85138: importer/kv:  add ImportEpoch field to MVCCValueHeader and write to it during IMPORT INTO r=dt,erikgrinaker a=msbutler

storage: add ImportEpoch field to MVCCValueHeader
This patch adds the ImportEpoch field to an MVCCValue's MVCCValueHeader,
which allows kv clients (namely the sst_batcher in an IMPORT INTO) to write
the importing table's ImportEpoch to the metadata of each ingesting MVCCValue.

Unlike the MVCCValueHeader.LocalTimestamp field, the ImportEpoch field should
be exported to other clusters (e.g. via ExportRequests from BACKUP/RESTORE and
streaming). Consequently, this PR relaxes the invariant that the
MVCCValueHeader field must be stripped in an Export Request and must be empty
in an AddSSTable Request. Now, Export Request only strips the
MVCCValueHeader.LocalTimestamp field and AddSSTable will only require the
LocalTimestamp to be empty.

Release note: none

----

bulk/kv write the table's ImportEpoch to each MVCCValue during IMPORT
This patch makes IMPORT INTO on a non-empty table write the table's ImportEpoch
to each ingested MVCC Value, via the SSTBatcher. In a future PR, the
ImportEpoch will be used to track and rollback an IMPORT INTO. This additional
information will allow IMPORTing tables to be backed up and restored, as
described in this [RFC](https://docs.google.com/document/d/16TbkFznqbsu3mialSw6o1sxOEn1pKhhJ9HTxNdw0-WE/edit#heading=h.bpox0bmkz77i).

Informs #76722

Release note: None

120080: sql: remove default_target_cluster.check_service.enabled r=dt a=dt

Release note (enterprise change): default_target_cluster can now be set to any tenant name by default, including a tenant yet to be created or have service started.
Epic: none.

120342: roachtest: admission-control/elastic-io deflake r=sumeerbhola a=aadityasondhi

Similar to #114446, we now take the mean over the last two minutes for determining high L0 sublevel count.

Fixes #119838.

Release note: None

120350: backfill: finish the tracing span after closing the account r=yuzefovich a=yuzefovich

We recently fixed an issue where we forgot to stop the index backfill merger monitor, but we had a minor bug in that fix - we captured the context that contains the tracing span that is finished before the account is closed leading to "use after finish" assertions. This is now fixed.

Fixes: #120266.

Release note: None

120356: cmd: link on the `large` pool r=jlinder a=rickystewart

The `default` pool seems to be too small to perform linking efficiently. This should speed things up.

Epic: CRDB-8308

Release note: None

120359: logictest: skip some tests under `race` r=jlinder a=rickystewart

These tests specifically are prone to failing/timing out under `race`.

Epic: CRDB-8308
Release note: None

120361: roachtest: bump max ranges threshold in splits/load/ycsb/e/nodes=3/obj=cpu r=nvanbenschoten a=nvanbenschoten

Fixes #120163.

Avoids rare test flakes.

Release note: None

120363: go.mod: bump Pebble to 51faab0a3555 r=aadityasondhi a=jbowens

Changes:

 * [`51faab0a`](cockroachdb/pebble@51faab0a) tool: add DirectoryLock option
 * [`ec69e9a2`](cockroachdb/pebble@ec69e9a2) ingest test: fix merge skew
 * [`635c6003`](cockroachdb/pebble@635c6003) manifest: add VersionEdit tests with virtual tables
 * [`cb660884`](cockroachdb/pebble@cb660884) manifest: improve VersionEdit stringification
 * [`31b37248`](cockroachdb/pebble@31b37248) ingest_test: support reopening in ingest tests
 * [`64ebec94`](cockroachdb/pebble@64ebec94) ingest_test: set correct sizes for external ingests
 * [`4bf09d5e`](cockroachdb/pebble@4bf09d5e) manifest: improve VersionEdit tests
 * [`a034560d`](cockroachdb/pebble@a034560d) manifest: add a helper for DebugString parsing
 * [`623524f1`](cockroachdb/pebble@623524f1) manifest: use Lx for levels in version String/DebugString

Release note: none.
Epic: none.

Co-authored-by: Michael Butler <[email protected]>
Co-authored-by: David Taylor <[email protected]>
Co-authored-by: Aaditya Sondhi <[email protected]>
Co-authored-by: Yahor Yuzefovich <[email protected]>
Co-authored-by: Ricky Stewart <[email protected]>
Co-authored-by: Nathan VanBenschoten <[email protected]>
Co-authored-by: Jackson Owens <[email protected]>
@craig craig bot closed this as completed in de41ff6 Mar 12, 2024
blathers-crl bot pushed a commit that referenced this issue Mar 12, 2024
@nvanbenschoten
roachtest: bump max ranges threshold in splits/load/ycsb/e/nodes=3/ob…
4148e8f 
…j=cpu

Fixes #120163.

Avoids rare test flakes.

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Mar 12, 2024
Merged
@kvoli kvoli added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-testing Testing tools and infrastructure labels Mar 18, 2024
jasminejsun pushed a commit to jasminejsun/cockroach that referenced this issue Mar 18, 2024
@nvanbenschoten @jasminejsun
roachtest: bump max ranges threshold in splits/load/ycsb/e/nodes=3/ob…
361de73 
…j=cpu

Fixes cockroachdb#120163.

Avoids rare test flakes.

Release note: None
@github-project-automation github-project-automation bot moved this to Closed in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nvanbenschoten nvanbenschoten

Labels
A-testing Testing tools and infrastructure branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-kv KV Team
Projects
No open projects
KV
Status: Closed
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

roachtest: bump max ranges threshold in splits/load/ycsb/e/nodes=3/obj=cpu nvanbenschoten/cockroach
3 participants
@nvanbenschoten @cockroach-teamcity @kvoli