Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: tpcc/large-schema-benchmark/multiregion=false/tables=25000 failed #132035

Closed
cockroach-teamcity opened this issue Oct 6, 2024 · 4 comments · Fixed by #132766
Closed

roachtest: tpcc/large-schema-benchmark/multiregion=false/tables=25000 failed #132035

cockroach-teamcity opened this issue Oct 6, 2024 · 4 comments · Fixed by #132766
Assignees
@itsbilal
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-storage Storage Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Oct 6, 2024 •
edited by cockroach-jira-scripts
Loading

roachtest.tpcc/large-schema-benchmark/multiregion=false/tables=25000 failed with artifacts on master @ f842c3b4b5adc040d411bd17d7d10005273fc1b6:

(test_runner.go:1308).runTest: test timed out (19h0m0s)
test artifacts and logs in: /artifacts/tpcc/large-schema-benchmark/multiregion=false/tables=25000/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-42796
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Oct 6, 2024
@exalate-issue-sync exalate-issue-sync bot removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Oct 8, 2024
@rafiss
Copy link
Collaborator

rafiss commented Oct 8, 2024

This seems to have gotten stuck during importing initial data

bash: line 8: 15549 Killed                  bash -c "./cockroach workload init tpccmultidb --db-list-file=populate_0  --import-concurrency-limit=32 'postgres://roachprod:[email protected]:26257?sslcert=.%2Fcerts%2Fclient.roachprod.crt&sslkey=.%2Fcerts%2Fclient.roachprod.key&sslmode=verify-full&sslrootcert=.%2Fcerts%2Fca.crt' --db warehouse_37 --warehouses 9"

@exalate-issue-sync exalate-issue-sync bot added the P-2 Issues/test failures with a fix SLA of 3 months label Oct 8, 2024
@fqazi fqazi added the T-kv KV Team label Oct 9, 2024
@fqazi
Copy link
Collaborator

fqazi commented Oct 9, 2024

Digging into this there is some sort of problem that exists on node 9. There are quite a few unavailable ranges and I also see a pile of quorum errors. Can someone from KV take a peak, this test got stuck importing large quantities of data. I also see circuit breaker errors in the log files, so would be good to understand what we ran into and why it didn't recover.

@fqazi fqazi removed their assignment Oct 9, 2024
@fqazi fqazi removed the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label Oct 9, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.tpcc/large-schema-benchmark/multiregion=false/tables=25000 failed with artifacts on master @ 30dbb173d0f083b35cf9eb8093832a5dd764c5af:

(test_runner.go:1308).runTest: test timed out (19h0m0s)
test artifacts and logs in: /artifacts/tpcc/large-schema-benchmark/multiregion=false/tables=25000/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@andrewbaptist andrewbaptist added the T-storage Storage Team label Oct 14, 2024
@blathers-crl blathers-crl bot added the A-storage Relating to our storage engine (Pebble) on-disk storage. label Oct 14, 2024
@andrewbaptist andrewbaptist removed A-storage Relating to our storage engine (Pebble) on-disk storage. T-kv KV Team labels Oct 14, 2024
@andrewbaptist andrewbaptist removed their assignment Oct 14, 2024
@andrewbaptist
Copy link
Collaborator

Appears to be caused be 124e6c86c10.

@aadityasondhi aadityasondhi mentioned this issue Oct 15, 2024
Closed
@itsbilal itsbilal self-assigned this Oct 16, 2024
@itsbilal itsbilal mentioned this issue Oct 16, 2024
Merged
@craig craig bot closed this as completed in #132766 Oct 16, 2024
@craig craig bot closed this as completed in 8cd457d Oct 16, 2024
@github-project-automation github-project-automation bot moved this from Tests (failures, skipped, flakes) to Done in [Deprecated] Storage Oct 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@itsbilal itsbilal

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-storage Storage Team
Projects
[Deprecated] Storage
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

go.mod: bump Pebble to 8b6d64f23a33 itsbilal/cockroach
6 participants
@itsbilal @rafiss @fqazi @aadityasondhi @cockroach-teamcity @andrewbaptist