Fairly distribute disk I/O

[kylos101]

Bug description

We tried to limit here, in ws-daemon, but it is not currently working.

Steps to reproduce

Run a workload that demands fast read or write, you'll get speeds in excess of 300Mi! This can starve other workspaces (on the same node) of disk I/O.

Workspace affected

n/a

Expected behavior

Your workloads should have disk IO limited, to some extent, so that the node and other workspaces are not starved.

Example repository

No response

Anything else?

We are experimenting to see how nicely this works.

Also, getting the above to work may inform as to why our current IO limiter does not work.

Ultimately though, we need a solution, and it must fit nicely with workspace classes.

CC: @aledbf @Furisto @atduarte

[kylos101]

Initial solve for this was done in #9440, we are currently monitoring.

[kylos101]

@aledbf @Furisto is there anything outstanding for cgroup v1 or v2 that you can think of as it pertains to IO limiting?

• [ws-daemon] Support config reload for IO limits #9440 was the initial solve for cgroup v1.
• For cgroup v2, I see these changes: [ws-daemon] Only limit storage device classes #9309 and Ensure IO limits are written after first workspace is stopped #9404.

Would it make sense to document cgroup support here for self-hosted?

cc: @corneliusludmann @mrsimonemms

[Furisto]

@kylos101 IO limiting with cgroup v2 does not really work yet. The best we could say is that if you want IO limiting for self-hosted then you need a system with cgroup v1.

[kylos101]

Thanks for the heads up, @Furisto ! I've removed this from scheduled work for now, as we've resolved the saas issue with #9440.

I think we need to think about the business value of cgroup V2. Given IOLimit and CPU limit are both not working as expected for cgroup V2, I wonder if we can do a skateboard of workspace classes on cgroup v1, and fix cgroup v2 later - in a follow-on epic. Wdyt?

CC: @csweichel @atduarte for awareness

[atduarte]

@Furisto @kylos101 what is the current situation in SaaS, with cgroupsv2? I was under the impression that we have IO limiting in place.

[aledbf]

Yes, that's right. It's working.