thingbuf::mpsc::Sender hanging up for parallel try_send_ref and send / send_ref from sync thread and async tokio::task
#83
Comments
|
I encountered the same problem. |
|
We are also using StaticThingbuf in the DragonOS kernel and have encountered the exact same problem. The issue appears randomly and is difficult to reproduce after the system kernel is restarted. In our use case:
Once the buffer is full, there is a very sluggish phenomenon. After a period of time, the performance drops off a cliff. Ultimately, it freezes at the |
This was referenced Apr 6, 2024
hawkw
added a commit
that referenced
this issue
Apr 6, 2024
## v0.1.5 (2024-04-06) #### Features * **mpsc:** add `len`, `capacity`, and `remaining` methods to mpsc (#72) ([00213c1](00213c1), closes [#71](#71)) #### Bug Fixes * unused import with `alloc` enabled ([ac1eafc](ac1eafc)) * skip slots with active reading `Ref`s in `push_ref` (#81) ([a72a286](a72a286), closes [#83](#83), [#80](#80))
|
After updating the repository, the performance issue still remains unresolved. Below is the backtrace of the program execution: We made some minor modifications to the code to adapt it to our requirements. Below is the repository link after merging the latest version:https://github.com/xiaolin2004/thingbuf.git |
|
Hmm, that's unfortunate. Thanks for following up --- I'll keep looking. |
|
It looks like, after 11089327 iterations under Full logs from that iteration |
I have an idea what's wrong with the test but I have to debug it. Will try to find some time this week to work on it. |
|
I investigated the test and the code in the https://github.com/sgasse/thingbuf_hangup/. The test fails because of an edge case, and my initial thought is to adjust the test itself. Here's a breakdown of the issue: We have a buffer with two slots. Initially, we write to slot [0], then to slot [1], and start reading from slot [0]. If we try writing again, slot [0] is unavailable (as it's still being read), so we skip it. Once the reading from slot [0] is completed, the head pointer moves to slot [1]. We attempt to write to slot [1] but can't, since it's not yet read, and the head pointer is already on slot [1]. Consequently, we declare the buffer as full. The problem arises in my test when starting to read from slot [1] – it never gets released, making it perpetually empty for reading and full for writing. However, I identified a genuine issue within the code at https://github.com/sgasse/thingbuf_hangup/, which I've temporarily fixed on my local setup. I executed I will re-examine my findings and aim to submit a PR later this week. |
hawkw
pushed a commit
that referenced
this issue
Apr 18, 2024
Fixes #83 Previously, to determine if the buffer was full, we checked whether the head and tail were pointing to the same slot with the head one generation behind. However, this check fails if we skip slots, leading to scenarios where the `head` and `tail` point to different slots even though the buffer is full. For example, consider a buffer with 3 slots. Initially, we write to the buffer three times (gen + 0). Then, we read from slot 0 and slot 1, holding the reference from slot 1, and read from slot 2 (gen + 0). Next, we write to slot 0 (gen + 1) and read from slot 0 (gen + 1), which moves our `head` to slot 1 (gen + 1). Then we try to write to slot 1 (gen + 1) and skip it, so we write to slot 2 (gen + 1). Then again we write to slot 0 (gen + 2). And then we attempt to write to slot 1 but we skip and attempt to write to slot 2 (gen + 2). However, we can’t write into it because it still contains data from the previous generation (gen + 1), and our `head` points to slot 1 instead of slot 2. This fix ensures the buffer full condition accurately reflects the actual status of the slots, particularly when writes are skipped.
hawkw
added a commit
that referenced
this issue
Apr 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi! We are using
thingbufin a performance-sensitive application for its speed while supporting sync and async interaction on the same sender type. Recently I ran into an issue wherethingbufseems to hang up our async application completely in busy loops. The profiling in the degraded state showed almost only calls tothingbuf.I was able to reproduce the issue in a minimum example. You can run several variations of it yourself by checking out this repo:
https://github.com/sgasse/thingbuf_hangup/
The initial setup (binary
thingbuf_sendref) to get into the hangup was this:std::threadsends withtry_send_refin a loop every 10ms.tokio::taskreceives on the channel withrecv_ref().awaitin a loop. After 10s, there is a delay of 10s introduced between receive calls. This simulates badly handled backpressure from a downstream task.tokio::taskstarts sending withsend(..).awaitin a loop after 20s.tokio::tasklogs an alive message every second.Once the second sender becomes active, we no longer see any alive logs. Introducing logs to
thingbufshows that two threads (one tokio worker and the self-spawned thread) are both stuck in this loop inpush_ref.Here is some log output with line numbers from
src/lib.rsfromthingbuffor the hang-up scenario, this is infinitely:The initial setup mimics the behavior of a part of our real application. However I varied the setup in other examples, here are some findings:
try_send_ref()(sync),send_ref().await()orsend(..).await(async) does not seem to matter, see examplesthingbuf_sendref,thingbuf_sendref_pure,thingbuf_send_recvrefandthingbuf_send_no_try_recvrefwhich all hang-up.recv().awaitinstead ofrecv_ref().await, there is no hangup, see examplethingbuf_send.x86_64-unknown-linux-gnu, but I initially found it onaarch64-linux-androidso I guess it does not depend on the platform.rustcin version1.76andnightly, so probably not related to the compiler.thingbuf::mpscwithtokio::mpscin one example, which works as expected: It still logs the alive messages and does not hang up.I would have expected the same behavior as I see with
tokio::mpsc, so I guess it is a bug. But please let me know if there is a limitation which I overlooked or if I can provide further info.The text was updated successfully, but these errors were encountered: