[Performance] 40% performance drop using lora vs no lora

[brucethemoose]

...
Processed prompts: 100% 1321/1321 [09:09<00:00,  2.40it/s]
training  | ic| "Testing '" + lora + "'": "Testing 'models/v3/checkpoint-1000'"
Processed prompts: 100% 1321/1321 [16:01<00:00,  1.37it/s]
training  | ic| "Lora '" + lora + "' testing complete.": "Lora 'models/v3/checkpoint-1000' testing complete."
training  | ic| "Testing '" + lora + "'": "Testing 'models/v3/final_lora'"
Processed prompts: 100% 1321/1321 [15:59<00:00,  1.38it/s]
training  | ic| "Lora '" + lora + "' testing complete.": "Lora 'models/v3/final_lora' testing complete."
training  | ic| "Testing '" + lora + "'": "Testing 'models/v3/checkpoint-2000'"
...

Testing offline batched inference with a big bunch of long context, short reply prompts, feeding the vllm engine all 1321 prompts as a list in a single batch. Tested on an 80GB A100.

This isn't a bug per se, or a major problem, more like a... note. I guess a big performance drop for loras is expected?

[Yard1]

With the current kernels it's expected they will not be able to handle long prompt-short reply case very well. In general around 20% performance drop is expected.

[sfc-gh-ybsat]

@Yard1 when do you expect Punica to ship the SGMV kernels? Looking at their github repo, seems like there is no progress being made in the past month

[Yard1]

I think we can look into whether we can use the existing SGMV kernels in currently released punica!

[chenqianfzh]

#2893

adding the thread as there are some info about SGMV & BGKV is shared in that thread.

[findalexli]

@Yard1 How is the 20% drop calculated?

[mgoin]

This should be lessened by about 2x with the new landed Triton kernels #5036

[colefranks]

@mgoin I see you have closed the issue, can you share your numbers? I am wondering how much the long-prompt/short output case has improved. The numbers in #5036 don't really cover this case as far as I can tell.

[jeejeelee]

@mgoin I see you have closed the issue, can you share your numbers? I am wondering how much the long-prompt/short output case has improved. The numbers in #5036 don't really cover this case as far as I can tell.

@colefranks Could you please provide more detailed information, such as the model and LoRA rank used?