Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: follower-reads/mixed-version/single-region failed #129546

Closed
cockroach-teamcity opened this issue Aug 23, 2024 · 9 comments · Fixed by #133092
Closed

roachtest: follower-reads/mixed-version/single-region failed #129546

cockroach-teamcity opened this issue Aug 23, 2024 · 9 comments · Fixed by #133092
Assignees
Labels
branch-master Failures and bugs on the master branch. branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-kv KV Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 23, 2024

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ c57e04527fbe285402bcadb7f73ce559e85d0c27:

(follower_reads.go:873).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 23 intervals > 4 threshold. Bad intervals:
interval 07:38:10-07:38:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.002 n4 ratio: 0.000 
interval 07:38:20-07:38:30: n1 ratio: 0.008 n2 ratio: 0.001 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:38:30-07:38:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:38:40-07:38:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:38:50-07:39:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:39:00-07:39:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:39:10-07:39:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.001 
interval 07:39:20-07:39:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:39:30-07:39:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:39:40-07:39:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.001 n4 ratio: 0.000 
interval 07:39:50-07:40:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:40:00-07:40:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:40:10-07:40:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:40:20-07:40:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:40:30-07:40:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:40:40-07:40:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.001 n4 ratio: 0.000 
interval 07:40:50-07:41:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.999 n4 ratio: 0.000 
interval 07:41:00-07:41:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:41:10-07:41:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:41:20-07:41:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:41:30-07:41:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:41:40-07:41:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:41:50-07:42:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
(mixedversion.go:710).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

Same failure on other branches

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-41588

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Aug 23, 2024
@andrewbaptist andrewbaptist added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Aug 26, 2024
@arulajmani
Copy link
Collaborator

arulajmani commented Aug 26, 2024

This one seems to be because of shared process multi-tenancy. From the logs:

Deployment mode:    shared-process

Seems like we're using #129167 as the main issue for this class of failures, so I'll mark this as a duplicate. We're also not treating these shared process multi-tenancy failures as release blockers, so I'll remove the label.

@arulajmani arulajmani removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Aug 26, 2024
@arulajmani arulajmani added the P-1 Issues/test failures with a fix SLA of 1 month label Aug 26, 2024
@github-project-automation github-project-automation bot moved this to roachtest/unit test backlog in KV Aug 28, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

(follower_reads.go:873).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 9 intervals > 4 threshold. Bad intervals:
interval 09:25:10-09:25:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.005 n4 ratio: 0.000 
interval 09:25:20-09:25:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.001 n4 ratio: 0.000 
interval 09:25:30-09:25:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 09:25:40-09:25:50: n1 ratio: 0.002 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 09:25:50-09:26:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 09:26:00-09:26:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 09:26:10-09:26:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 09:26:20-09:26:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 09:26:30-09:26:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.234 
(mixedversion.go:710).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ 67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867:

(follower_reads.go:873).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 14 intervals > 4 threshold. Bad intervals:
interval 07:12:00-07:12:10: n1 ratio: 0.825 n2 ratio: 0.201 n3 ratio: 1.774 n4 ratio: 0.000 
interval 07:12:10-07:12:20: n1 ratio: 0.805 n2 ratio: 0.000 n3 ratio: 1.874 n4 ratio: 0.000 
interval 07:12:20-07:12:30: n1 ratio: 0.811 n2 ratio: 0.000 n3 ratio: 1.971 n4 ratio: 0.000 
interval 07:12:30-07:12:40: n1 ratio: 0.807 n2 ratio: 0.000 n3 ratio: 1.976 n4 ratio: 0.000 
interval 07:12:40-07:12:50: n1 ratio: 0.830 n2 ratio: 0.000 n3 ratio: 1.866 n4 ratio: 0.000 
interval 07:12:50-07:13:00: n1 ratio: 0.833 n2 ratio: 0.000 n3 ratio: 1.876 n4 ratio: 0.000 
interval 07:13:00-07:13:10: n1 ratio: 0.886 n2 ratio: 0.000 n3 ratio: 1.835 n4 ratio: 0.000 
interval 07:13:10-07:13:20: n1 ratio: 0.726 n2 ratio: 0.000 n3 ratio: 1.875 n4 ratio: 0.000 
interval 07:13:20-07:13:30: n1 ratio: 0.534 n2 ratio: 0.000 n3 ratio: 2.413 n4 ratio: 0.000 
interval 07:13:30-07:13:40: n1 ratio: 0.553 n2 ratio: 0.000 n3 ratio: 2.195 n4 ratio: 0.000 
interval 07:14:00-07:14:10: n1 ratio: 0.449 n2 ratio: 0.000 n3 ratio: 2.485 n4 ratio: 0.000 
interval 07:14:10-07:14:20: n1 ratio: 0.895 n2 ratio: 0.000 n3 ratio: 1.999 n4 ratio: 0.000 
interval 07:14:20-07:14:30: n1 ratio: 0.606 n2 ratio: 0.000 n3 ratio: 2.414 n4 ratio: 0.000 
interval 07:14:30-07:14:40: n1 ratio: 0.629 n2 ratio: 0.000 n3 ratio: 2.088 n4 ratio: 0.000 
(mixedversion.go:710).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ 197c6ee5537ffb211ebd8dbcbe49edc6d5c710e1:

(follower_reads.go:873).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 23 intervals > 4 threshold. Bad intervals:
interval 07:48:20-07:48:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.698 n4 ratio: 0.011 
interval 07:48:30-07:48:40: n1 ratio: 0.000 n2 ratio: 0.001 n3 ratio: 2.747 n4 ratio: 0.000 
interval 07:48:40-07:48:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.772 n4 ratio: 0.000 
interval 07:48:50-07:49:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.795 n4 ratio: 0.000 
interval 07:49:00-07:49:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.772 n4 ratio: 0.000 
interval 07:49:10-07:49:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.761 n4 ratio: 0.000 
interval 07:49:20-07:49:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.667 n4 ratio: 0.000 
interval 07:49:30-07:49:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.664 n4 ratio: 0.000 
interval 07:49:40-07:49:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.826 n4 ratio: 0.000 
interval 07:49:50-07:50:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.853 n4 ratio: 0.000 
interval 07:50:00-07:50:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.835 n4 ratio: 0.000 
interval 07:50:10-07:50:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.813 n4 ratio: 0.000 
interval 07:50:20-07:50:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.916 n4 ratio: 0.000 
interval 07:50:30-07:50:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 3.042 n4 ratio: 0.000 
interval 07:50:40-07:50:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 3.061 n4 ratio: 0.000 
interval 07:50:50-07:51:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.869 n4 ratio: 0.000 
interval 07:51:00-07:51:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.764 n4 ratio: 0.000 
interval 07:51:10-07:51:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.856 n4 ratio: 0.000 
interval 07:51:20-07:51:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.794 n4 ratio: 0.000 
interval 07:51:30-07:51:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.737 n4 ratio: 0.000 
interval 07:51:40-07:51:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.986 n4 ratio: 0.000 
interval 07:51:50-07:52:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.642 n4 ratio: 0.000 
interval 07:52:00-07:52:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.627 n4 ratio: 0.000 
(mixedversion.go:720).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ 83589fb87caa92fb42e83994f1691978f37e4cbb:

(follower_reads.go:873).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 21 intervals > 4 threshold. Bad intervals:
interval 09:56:00-09:56:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 3.609 
interval 09:56:10-09:56:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.969 
interval 09:56:20-09:56:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.976 
interval 09:56:30-09:56:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.964 
interval 09:56:40-09:56:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.206 
interval 09:56:50-09:57:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.416 
interval 09:57:00-09:57:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.685 
interval 09:57:10-09:57:20: n1 ratio: 0.000 n2 ratio: 0.005 n3 ratio: 0.000 n4 ratio: 2.695 
interval 09:57:20-09:57:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.062 
interval 09:57:30-09:57:40: n1 ratio: 0.671 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.442 
interval 09:57:40-09:57:50: n1 ratio: 0.266 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.966 
interval 09:57:50-09:58:00: n1 ratio: 0.336 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.713 
interval 09:58:00-09:58:10: n1 ratio: 0.858 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.618 
interval 09:58:10-09:58:20: n1 ratio: 0.511 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.352 
interval 09:58:20-09:58:30: n1 ratio: 0.677 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.996 
interval 09:58:30-09:58:40: n1 ratio: 0.576 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.360 
interval 09:58:50-09:59:00: n1 ratio: 0.716 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.218 
interval 09:59:00-09:59:10: n1 ratio: 0.794 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.013 
interval 09:59:10-09:59:20: n1 ratio: 0.061 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.165 
interval 09:59:30-09:59:40: n1 ratio: 0.897 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.047 
interval 09:59:40-09:59:50: n1 ratio: 0.535 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.029 
(mixedversion.go:720).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ 8e6e4090457565a41bc3bd8ea954e437030d1c49:

(follower_reads.go:873).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 23 intervals > 4 threshold. Bad intervals:
interval 09:04:10-09:04:20: n1 ratio: 0.007 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:04:20-09:04:30: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:04:30-09:04:40: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:04:40-09:04:50: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:04:50-09:05:00: n1 ratio: 0.000 n2 ratio: 1.411 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:05:00-09:05:10: n1 ratio: 0.000 n2 ratio: 1.980 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:05:10-09:05:20: n1 ratio: 0.000 n2 ratio: 1.980 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:05:20-09:05:30: n1 ratio: 0.000 n2 ratio: 1.803 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:05:30-09:05:40: n1 ratio: 0.004 n2 ratio: 1.195 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:05:40-09:05:50: n1 ratio: 0.000 n2 ratio: 1.693 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:05:50-09:06:00: n1 ratio: 0.000 n2 ratio: 1.810 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:06:00-09:06:10: n1 ratio: 0.000 n2 ratio: 1.981 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:06:10-09:06:20: n1 ratio: 0.000 n2 ratio: 1.203 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:06:20-09:06:30: n1 ratio: 0.000 n2 ratio: 1.212 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:06:30-09:06:40: n1 ratio: 0.000 n2 ratio: 1.814 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:06:40-09:06:50: n1 ratio: 0.000 n2 ratio: 1.062 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:06:50-09:07:00: n1 ratio: 0.000 n2 ratio: 1.593 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:07:00-09:07:10: n1 ratio: 0.000 n2 ratio: 1.468 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:07:10-09:07:20: n1 ratio: 0.000 n2 ratio: 1.182 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:07:20-09:07:30: n1 ratio: 0.000 n2 ratio: 1.584 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:07:30-09:07:40: n1 ratio: 0.000 n2 ratio: 1.731 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:07:40-09:07:50: n1 ratio: 0.000 n2 ratio: 1.978 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:07:50-09:08:00: n1 ratio: 0.000 n2 ratio: 1.979 n3 ratio: 0.000 n4 ratio: 0.000 
(mixedversion.go:720).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ ec2573dc6aaeefc226440bb2c5a7c94a63989868:

(follower_reads.go:379).runFollowerReadsTest: fewer than 2 follower reads occurred: saw [63 0 6 1] before and [63 640 6 1] after
(mixedversion.go:737).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ 4de315c9ca4ccf7c3bdbf53a5226e8c14c84a68e:

(follower_reads.go:870).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 23 intervals > 4 threshold. Bad intervals:
interval 10:19:40-10:19:50: n1 ratio: 0.000 n2 ratio: 0.999 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:19:50-10:20:00: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:20:00-10:20:10: n1 ratio: 0.000 n2 ratio: 1.012 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:20:10-10:20:20: n1 ratio: 0.000 n2 ratio: 1.001 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:20:20-10:20:30: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:20:30-10:20:40: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:20:40-10:20:50: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.005 
interval 10:20:50-10:21:00: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:21:00-10:21:10: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:21:10-10:21:20: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:21:20-10:21:30: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:21:30-10:21:40: n1 ratio: 0.000 n2 ratio: 0.999 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:21:40-10:21:50: n1 ratio: 0.000 n2 ratio: 1.001 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:21:50-10:22:00: n1 ratio: 0.000 n2 ratio: 0.999 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:22:00-10:22:10: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:22:10-10:22:20: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:22:20-10:22:30: n1 ratio: 0.000 n2 ratio: 1.001 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:22:30-10:22:40: n1 ratio: 0.000 n2 ratio: 0.999 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:22:40-10:22:50: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:22:50-10:23:00: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:23:00-10:23:10: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:23:10-10:23:20: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:23:20-10:23:30: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
(mixedversion.go:732).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@arulajmani arulajmani added P-2 Issues/test failures with a fix SLA of 3 months and removed P-1 Issues/test failures with a fix SLA of 1 month labels Oct 14, 2024
andrewbaptist added a commit to andrewbaptist/cockroach that referenced this issue Oct 21, 2024
After introducing shared process testing of follower reads, the tests
follower-reads/mixed-version/* have been flaky. This commit starts
testing from 23.2 to make sure that any other failures are caught and
triaged correctly.

Epic: none
Fixes: cockroachdb#133000
Fixes: cockroachdb#132999
Fixes: cockroachdb#129546
Fixes: cockroachdb#129167

Release note: None
andrewbaptist added a commit to andrewbaptist/cockroach that referenced this issue Oct 21, 2024
After introducing shared process testing of follower reads, the tests
follower-reads/mixed-version/* have been flaky. This commit starts
testing from 23.2 to make sure that any other failures are caught and
triaged correctly.

Epic: none
Fixes: cockroachdb#133000
Fixes: cockroachdb#132999
Fixes: cockroachdb#129546
Fixes: cockroachdb#129167

Release note: None
craig bot pushed a commit that referenced this issue Oct 23, 2024
132851: kvserver/rangefeed: remove context from kvpb.RangeFeedEventSink r=stevendanna a=wenyihu6

Previously, `node.MuxRangefeed` created a child context for each rangefeed
request, storing it in the stream interface to allow the node level to be able
to shut down registration goroutines. This patch simplifies the approach by
passing the stream context directly to `p.Register`, eliminating the need to
store context in `streamSink` or return context via the interface. So this patch
also removes context from `kvpb.RangeFeedEventSink`.

Epic: none
Release note: none

133092: roachtest: disable 23.1 -> 23.2 testing for follower reads r=arulajmani a=andrewbaptist

After introducing shared process testing of follower reads, the tests follower-reads/mixed-version/* have been flaky. This commit starts testing from 23.2 to make sure that any other failures are caught and triaged correctly.

Epic: none
Fixes: #133000
Fixes: #132999
Fixes: #129546
Fixes: #129167

Release note: None

Co-authored-by: Wenyi Hu <[email protected]>
Co-authored-by: Andrew Baptist <[email protected]>
craig bot pushed a commit that referenced this issue Oct 23, 2024
132443: crosscluster/logical: add KV write path metrics r=msbutler a=stevendanna

It would be nice to know how often we are hitting these particular errors.

Epic: none
Release note: None

133092: roachtest: disable 23.1 -> 23.2 testing for follower reads r=arulajmani a=andrewbaptist

After introducing shared process testing of follower reads, the tests follower-reads/mixed-version/* have been flaky. This commit starts testing from 23.2 to make sure that any other failures are caught and triaged correctly.

Epic: none
Fixes: #133000
Fixes: #132999
Fixes: #129546
Fixes: #129167

Release note: None

133273: ui: bump cluster-ui to 24.3.0-prerelease.4 r=xinhaoz a=xinhaoz

This commit bumps cluster-ui version to 24.3.0-prerelease.4 and moves crdb-protobuf-client pkg from dependencies back to dev dependencies in an attempt to resolve build issues when exporting the package.

Epic: none

Release note: None

Co-authored-by: Steven Danna <[email protected]>
Co-authored-by: Andrew Baptist <[email protected]>
Co-authored-by: Xin Hao Zhang <[email protected]>
@craig craig bot closed this as completed in 30a21f3 Oct 23, 2024
Copy link

blathers-crl bot commented Oct 23, 2024

Based on the specified backports for linked PR #133092, I applied the following new label(s) to this issue: branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added the branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 label Oct 23, 2024
blathers-crl bot pushed a commit that referenced this issue Oct 23, 2024
After introducing shared process testing of follower reads, the tests
follower-reads/mixed-version/* have been flaky. This commit starts
testing from 23.2 to make sure that any other failures are caught and
triaged correctly.

This test is flaky when upgrading from v23.1 to v23.2 for follower reads
in shared-process deployments. There were a number of changes to tenant
health checks since then which appear to have addressed this issue.

Epic: none
Fixes: #133000
Fixes: #132999
Fixes: #129546
Fixes: #129167

Release note: None
miraradeva pushed a commit to miraradeva/cockroach that referenced this issue Dec 11, 2024
After introducing shared process testing of follower reads, the tests
follower-reads/mixed-version/* have been flaky. This commit starts
testing from 23.2 to make sure that any other failures are caught and
triaged correctly.

This test is flaky when upgrading from v23.1 to v23.2 for follower reads
in shared-process deployments. There were a number of changes to tenant
health checks since then which appear to have addressed this issue.

Epic: none
Fixes: cockroachdb#133000
Fixes: cockroachdb#132999
Fixes: cockroachdb#129546
Fixes: cockroachdb#129167

Release note: None
miraradeva pushed a commit to miraradeva/cockroach that referenced this issue Dec 11, 2024
After introducing shared process testing of follower reads, the tests
follower-reads/mixed-version/* have been flaky. This commit starts
testing from 23.2 to make sure that any other failures are caught and
triaged correctly.

This test is flaky when upgrading from v23.1 to v23.2 for follower reads
in shared-process deployments. There were a number of changes to tenant
health checks since then which appear to have addressed this issue.

Epic: none
Fixes: cockroachdb#133000
Fixes: cockroachdb#132999
Fixes: cockroachdb#129546
Fixes: cockroachdb#129167

Release note: None
miraradeva pushed a commit that referenced this issue Dec 12, 2024
After introducing shared process testing of follower reads, the tests
follower-reads/mixed-version/* have been flaky. This commit starts
testing from 23.2 to make sure that any other failures are caught and
triaged correctly.

This test is flaky when upgrading from v23.1 to v23.2 for follower reads
in shared-process deployments. There were a number of changes to tenant
health checks since then which appear to have addressed this issue.

Epic: none
Fixes: #133000
Fixes: #132999
Fixes: #129546
Fixes: #129167

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-kv KV Team
Projects
No open projects
Status: roachtest/unit test backlog
Development

Successfully merging a pull request may close this issue.

3 participants