1. Reduce compile time by 22%+ 2. Fix compile linking error on Ubuntu 22.04 gcc/g++ 11.4 with Cuda 12.4 #171
Conversation
…king errors: Ubuntu 22.04 gcc/g++ 11.4 2. Reduce compile time: A) Reduce thread contention due to ninja naively using all cores. Instead, use half cores but more threads. B) Only compile for the base archs. sm80 for ampere, sm89 for ada, and sm90 for hopper. Skip compilation for minor cuda archs such as ampere/86.
The throughput per sm doubles from sm80 to sm86 (https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#arithmetic-instructions), so nvcc might behave a little bit differently because of the difference. I agree with you to only keep |
|
Thank you for your contribution! |
|
@yzh119 See if you can stop and restart the CI runners. I see the current one is already failing with very similar ld/linking errors that the PR may fix. https://github.com/flashinfer-ai/flashinfer/actions/runs/8226582215 |
1. Do not generate prefill kernels for `page_size=8` 2. Build with `-Xfatbin=-compress-all` to reduce binary size. Followup of #171 , @Qubitium the cuda architectures to be compiled could be controlled by environment variable `TORCH_CUDA_ARCH_LIST`, so I removed the gencode/archs specified in compile args.
|
Another Github action is running: https://github.com/flashinfer-ai/flashinfer/actions/runs/8227731974 |
Test Env:
This PR resolves two issues 1) compile linking failed on my env 2) reduce compile time by ~22%. I expect the diff to be larger if underlying io storage is slow spinners/ssd.
-std=c++17based on my monitoring of the compilation processes on main branch.Linking Error Stracktrace
Test was performed on a Zen3 2x9334 system with 48 cores (96 threads) but for test I limited via lxd container to max 80 cores/threads. Ran an earlier version of the PR test with full 96 cores/threads and the diff % is same. Between each test I removed python/build and python/csrc/generated dirs.
Main:
PR:
Diff is ~22% but I expect the value to be even larger if underlying io system is slow spinner/sdd since thread contention would kill small io during compile.
Cause of slow compilation is as follows:
Primary cause is thread contention (resource over-subscription). Ninja using all cores by default is not optimal. Fix: Use half the cores but increase the number of threads nvcc can spawn from 1 to 8.
Also reduce the unnecessary compilation of minor archs. For example, Ampere has sm80 and sm86. The only diff is hardware resources such as sm cores, cache size, etc. I did not find any documentation from Nvidia that nvcc actually compile differently due to same arch but different cache size, sm cores. Just compile for the base archs. sm80 for Ampere, sm89 for Ada, sm90 for Hopper.
ref: https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html
@yzh119