[Performance] Tracking open Issues and PRs (pytorch transformers)

[stas00]

Let's use this Issue to track performance issues and enhancement requests, so it's easier to prioritize the work.

This is for pytorch transformers

Also I will label it as a Good Difficult Issue in case someone is ready for a challenging but rewarding experience of figuring things out. If you do want to take the challenge comment in the corresponding Issue/PR that resonates with you so others would know you're working on it.

If I missed any other relevant open performance-related Issues/PRs that need attention please comment below.

Regression:

• [WIP] FSMT bart-like refactor #11218 Regression after Bart-like refactoring - need to compare the original Bart refactoring PR since most likely the regression happened there.
• [ ]

Odd slowness:

• [trainer] figuring out why eval with --fp16_full_eval is 25% slower #10816 figuring out why eval with --fp16_full_eval is 25% slower
• [ ]

Fused kernels possibilities:

• Megatron fused CUDA kernels to improve Hugging Face model classes' scalability #11368 Megatron fused CUDA kernels to improve Hugging Face model classes' scalability
• research pytorch kernels?
• I know Deepspeed has various kernels that we might be able to use

Faster / leaner startup / module loading

• [performance] module init w/ from_pretrained skip storage allocation #12274 - skip storage allocation which gets dropped for pretrained weights

Faster optimizers

• Memory Efficient FP 16 Training #12084 - a proposal to port MemoryEfficientFP16Optimizer from fairseq
• [trainer] new in pytorch: torch.optim._multi_tensor faster optimizers  #9965 - torch.optim._multi_tensor faster optimizers - having some bottleneck in the test script - need to profile

Scalability

• [Tensor Parallelism] Megatron-LM to transformers #10321 Tensor Parallelism

Deepspeed-specific features

• [DeepSpeed] Features to integrate / Optimizations to add / Experiments to do #9606 a list of features that can be integrated
• [Deepspeed] [performance] inefficient load with from_pretrained w/ zero3 #12273 - make from_pretrained loading faster

Tests

• No issue yet, but we really need to add performance regression tests

[JuheonChu]

@stas00 If I want to work on this issue, should I pick one of those issues to keep track of its performance?
Can you also tell me how I can keep track of the performances? Can you give me some guidance?

[stas00]

Hi @JuheonChu, this is not an Issue to work on. As the title says this is a collection of pointers to track other Issues. It's dated but many issues that it links to are still valid. So you can click on the issue that resonates with you and discuss the details there - not here.

I hope this addresses your question.