Evaluate Profile-Guided Optimization (PGO) and LLVM BOLT

[zamazan4ik]

Hi!

Recently I did a lot of PGO benchmarks on different software - all available results are here. According to my tests, PGO can help with achieving better performance in many workloads, including compiler-like (compilers, static analysis, code formatters, etc.). Since Ruff cares about performance, I think it would be a good idea to test PGO on it.

I can suggest to do the following things:

• Evaluate PGO on Ruff
• If it benefits Ruff - add a note to the Ruff documentation about building with PGO. In this case, users and maintainers who build their own Ruff packages will be aware of PGO as an additional way to optimize Ruff
• Optimize provided by Ruff team binaries on the CI (like it's already done for other projects like Rustc)
• After that, I suggest trying to apply LLVM BOLT as an additional post-PGO step. Rustc already does it on some platforms

For the Rust projects, I suggest PGO optimizing with cargo-pgo.

I already tried to optimize Ruff with PGO on my local machines but unfortunately met a bug in Rustc on the LTO and PGO boundary. More details about the bug are available at rust-lang/rust#115344 (comment) . So I suggest leaving this issue as-is. And later when the bug in the upstream will be fixed, try to apply PGO to Ruff once again.

[charliermarsh]

Thanks for this. I'm happy to support an evaluation to see if Ruff can benefit from PGO, at which point we can evaluate the tradeoffs of integrating it into our production pipeline. Is there anything we can do on our end to resolve that upstream bug (e.g., changes we can make to Ruff itself)?

[zamazan4ik]

Is there anything we can do on our end to resolve that upstream bug (e.g., changes we can make to Ruff itself)?

Right now disabling LTO is the only way to avoid the bug. But I do not recommend doing it since LTO provides performance benefits too but with less CI tweaking (enabling one flag with LTO vs implementing 2-stage builds with PGO). So I suggest just waiting for the fix in the upstream and then trying to test LTO + PGO once again.