Require conn_quota::units stay on a single shard

[rockwotj]

This makes the API contract more explicit since we now require the
connection quota units doesn't change shards.

Related: #10544

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.1.x
• v22.3.x
• v22.2.x

Release Notes

• none

[rockwotj]

/ci-repeat 5
skip-units
dt-repeat=100
rptest.tests.services_self_test.py::KgoVerifierSelfTest.test_kgo_verifier

[rockwotj]

/ci-repeat 10
debug
skip-units
dt-repeat=30
tests/rptest/tests/services_self_test.py

[rockwotj]

/ci-repeat 1
release
skip-units
dt-repeat=10
dt-log-level=TRACE
tests/rptest/tests/services_self_test.py

[vbotbuildovich]

Invalid ducktape log level TRACE. Must be one of trace,debug,info,warn,error.

Workflow run logs.

[rockwotj]

/ci-repeat 1
release
skip-units
dt-repeat=10
dt-log-level=trace
tests/rptest/tests/services_self_test.py

[rockwotj]

/ci-repeat 5
skip-units
dt-repeat=20
rptest/tests/services_self_test.py::KgoVerifierSelfTest.test_kgo_verifier

[dlex]

Please correct me from wrong, do_put() explicitly supports returning units cross-shard. If there is a suspiction that this is somehow correlated with segfaults, maybe it's safer to add warns/errors to the log about that.

[rockwotj]

The while conn_quota supports cross shard, it does not support being used on a different shard than it's home shard. The stacktrace in the linked bug makes it look like total_home is a bad pointer or pointing to junk. conn_quota::units captures a reference to a single shard's conn_quota which is a way the stacktrace could happen.

total_home is only defined on shard0, if another shard's service is called on shard0 then it will assume that total_home is a valid pointer and then dereference nullptr. The log statement I added should help determine this too.

[rockwotj]

CI Failures:

• Reactor stall in redpanda #1725
• CI Failure (failed to hydrate chunk starting at 0) in CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves #10868

[dotnwat]

I have a theory total_home (only set on shard0's service) could be ran
using a service from another shard, but on shard0.

do you have a candidate execution schedule that would lead to the violation? i recall thinking the same thing a long time ago, but was never able to find a counterexample. i still remained a bit suspicious.

[rockwotj]

The stack trace in #10544 (comment) hints that the total_home pointer on shard0 is bad from the logs. And we do grab a reference in units to a local shard, so seems like it could be a smoking gun. From my reading of the code it doesn't look like we switch shards however.

[dotnwat]

The stack trace in #10544 (comment) hints that the total_home pointer on shard0 is bad from the logs. And we do grab a reference in units to a local shard, so seems like it could be a smoking gun. From my reading of the code it doesn't look like we switch shards however.

this is wild. do you think maybe we aren't switching shards, but a pointer is leaking across shards some how?

[rockwotj]

CI Failures:

• CI Failure (retries) in rptest.tests.e2e_iam_role_test #11107
• CI Failure (writes got stuck) in CompactedVerifierTest.test_tx #11065

[rockwotj]

this is wild. do you think maybe we aren't switching shards, but a pointer is leaking across shards some how?

Hrm, that seems unlikely? I'm not thinking that directly I supposed. This is more to prevent this from accidentally happening in the future, and guarding against any weird bugs here.

[rockwotj]

/ci-repeat 10
skip-units
dt-repeat=10
rptest/tests/services_self_test.py::KgoVerifierSelfTest.test_kgo_verifier

[rockwotj]

/ci-repeat 10
skip-units
skip-rebase
dt-repeat=10
tests/rptest/tests/services_self_test.py::KgoVerifierSelfTest.test_kgo_verifier

[rockwotj]

/ci-repeat 10
skip-units
skip-rebase
dt-repeat=200
tests/rptest/tests/services_self_test.py::KgoVerifierSelfTest.test_kgo_verifier

[dotnwat]

lgtm!