Lettuce Redis Connection Thread Priority

[emindeniz99]

Hello 👋🏻,
At work, we are using spring boot for the backend. Our Redis connections are managed by Lettuce.
Sometimes our application suffers from CPU peak, CPU hits 90-100%. When that happens, all Redis commands slow very. On the other side, our mysql commands are not slowing down like Redis commands.
Our mysql configuration is 100 connection thread pool. The Redis connection is a single connection thanks to lettuce.

We wonder if cpu scheduling can cause this case or not. Because MySQL commands(threads) can execute on the CPU 100 times more than Redis 1 connection thread.

Do you have any idea? What do you think about this idea?

Thank you
Have a nice day

[mp911de]

It boils down to what your Redis I/O threads are busy with. If your RedisCodec is expensive, then this might be a clue. In any case, please record a profiling of the Redis I/O threads and inspect what keeps the threads busy for further analysis.

[GChapel]

Agree with @emindeniz99. When the CPU hits 100%, many threads wait for responses to redis commands provided by the single lettuce thread.

The processor does not prioritize the lettuce thread enough to avoid the bottleneck. I tried giving higher priority to the thread, nothing changed

On my side, I hope that virtual threads will increase lettuce throughput in this context

[tishun]

Agree with @emindeniz99. When the CPU hits 100%, many threads wait for responses to redis commands provided by the single lettuce thread.

The processor does not prioritize the lettuce thread enough to avoid the bottleneck. I tried giving higher priority to the thread, nothing changed

On my side, I hope that virtual threads will increase lettuce throughput in this context

Did using virtual threads help?

[tishun]

Recording a profile with the example of the occurrence would help us understand how we can improve the code.
From @GChapel 's comments I gather that the event loop lettuce uses to process commands (specifically responses) is not getting enough CPU time - is that correct? Could you verify that with some profile we can analyse?

[mp911de]

I don't think that thread priority is going to help. In Spring, you can configure whether to use a shared connection or whether to use dedicated (potentially pooled) connections. It would help to get a reproducer that shows usage and bottlenecks. Anything else is just poking at things in the dark.

[tishun]

I agree, there are too many open questions.