Optimize thread_worker

[rockwotj]

Use a ring buffer to bulk pass tasks around, and also support creating a
twin worker thread for every reactor thread.

This amortizes the cost of running a task on an alien thread, this came about because we're going to be switching transforms over to Wasmtime running on an alien thread.

Before:

test                                      iterations      median         mad         min         max      allocs       tasks        inst
thread_worker_test.1                            3354   216.749us    47.062ns   216.590us   216.951us       5.000       4.000   1610844.2
thread_worker_test.10                            460     2.088ms   870.335ns     2.087ms     2.089ms      70.000      58.000  25391981.4
thread_worker_test.100                            61    16.465ms     1.733us    16.461ms    16.469ms     700.000     598.000 208106393.1
thread_worker_test.1000                           50    19.925ms     6.187us    19.919ms    19.937ms    7007.000    5998.000 214776807.2

After:

test                                      iterations      median         mad         min         max      allocs       tasks        inst
thread_worker_test.1                            1855   216.515us    77.552ns   216.330us   216.593us       4.000       4.000   1549639.0
thread_worker_test.10                            413     2.085ms   394.971ns     2.084ms     2.086ms      40.000      40.000  24921287.6
thread_worker_test.100                           113     8.518ms     3.891us     8.513ms     8.524ms     400.000     400.000 106121583.5
thread_worker_test.1000                           93    10.424ms   605.968ns    10.423ms    10.426ms    4880.000    5744.000 127114775.7

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.2.x
• v23.1.x
• v22.3.x

Release Notes

• none

[dotnwat]

lgtm

[BenPope]

Looks great.

I wonder if it makes sense to add an abort_source, or fail pending requests on stop()?

[rockwotj]

I wonder if it makes sense to add an abort_source, or fail pending requests on stop()?

I am not really sure how to do this correctly, as I don't think seastar classes are designed to support work outside of seastar, unless there is a different technique you're suggesting.

Same goes for failing pending stuff on stop, I am not sure how to tell the alien thread that it needs to stop, maybe with an atomic boolean it checks before every task? Thoughts?

[BenPope]

I wonder if it makes sense to add an abort_source, or fail pending requests on stop()?

I am not really sure how to do this correctly, as I don't think seastar classes are designed to support work outside of seastar, unless there is a different technique you're suggesting.

Same goes for failing pending stuff on stop, I am not sure how to tell the alien thread that it needs to stop, maybe with an atomic boolean it checks before every task? Thoughts?

The stop signal was previously passed via the ss::writeable_eventfd, once stop is detected, I imagine it would be possible to call fail_tasks.

I'm not saying it has to be done, it's just a thought. Propagating an abort (somehow) might be more useful than just cancelling the work. Or it might just make sense to drain the queue; it probably depends what kind of work it's running.

[rockwotj]

The stop signal was previously passed via the ss::writeable_eventfd, once stop is detected, I imagine it would be possible to call fail_tasks.

That's a little hard to do now that there are multiple signals in the normal submit path. signal adds values up and then read drains the value from my reading of the man pages, so calling signal(1) twice means that you'll read 2 as the result.

it probably depends what kind of work it's running.

Agreed, I don't know about kerberos, but for wasm, cancelling does make since, but that will have already happened as the transform processor needs to shutdown before wasm does.

[BenPope]

it probably depends what kind of work it's running.

Agreed, I don't know about kerberos, but for wasm, cancelling does make since, but that will have already happened as the transform processor needs to shutdown before wasm does.

Kerberos is usually just reading a local file, so I/O, but not too bad, and unlikely to build a large backlog.

[BenPope]

LGTM, let's keep an eye on the queue length at shutdown. Should there be metrics for queue length, time in queue, etc?

Feel free to add a followup issue.

[rockwotj]

CI Failures: #12120 and #12659

[rockwotj]

LGTM, let's keep an eye on the queue length at shutdown.

I added a followup task for metrics - Also the queue is limited to 128 items for now.