Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Kernel] Support Fp8 Checkpoints (Dynamic + Static) #4332

Merged
merged 93 commits into from
Apr 30, 2024
Merged

[Kernel] Support Fp8 Checkpoints (Dynamic + Static) #4332

merged 93 commits into from
Apr 30, 2024

Commits on Apr 23, 2024

  1. fixed fp8 conflict with aqlm

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    79c94a1 View commit details
    Browse the repository at this point in the history

  2. added quantization tests to buildkite

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    f8b57e4 View commit details
    Browse the repository at this point in the history

  3. removed commented out piece

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    7175e5b View commit details
    Browse the repository at this point in the history

  4. model loaded!

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    7a7520d View commit details
    Browse the repository at this point in the history

  5. renamed

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    e0b4d72 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. stash

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    f96428e View commit details
    Browse the repository at this point in the history

  2. added static fp8

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    88ba83b View commit details
    Browse the repository at this point in the history

  3. to try with torch.scaled_mm

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    0848b25 View commit details
    Browse the repository at this point in the history

  4. stash

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    15882ea View commit details
    Browse the repository at this point in the history

  5. added way to do weight quantization

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    7e6b675 View commit details
    Browse the repository at this point in the history

  6. working!

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    cc959ea View commit details
    Browse the repository at this point in the history

  7. fixed llama

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    8d68dbc View commit details
    Browse the repository at this point in the history

  8. fixed llama again

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    881fc65 View commit details
    Browse the repository at this point in the history

  9. updated names

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    e6dd46f View commit details
    Browse the repository at this point in the history

  10. nit

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    7e3933b View commit details
    Browse the repository at this point in the history

  11. cleanup

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    453a236 View commit details
    Browse the repository at this point in the history

  12. cleanup

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    310e0a7 View commit details
    Browse the repository at this point in the history

  13. missed file :)

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    ab4cb02 View commit details
    Browse the repository at this point in the history

  14. Update fp8.py

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Apr 24, 2024
    Configuration menu
    Copy the full SHA
    2edd93a View commit details
    Browse the repository at this point in the history

  15. Implement static scaling for Mixtral

    @pcmoritz
    pcmoritz committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    ccee5d3 View commit details
    Browse the repository at this point in the history

  16. fix

    @pcmoritz
    pcmoritz committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    8f71c79 View commit details
    Browse the repository at this point in the history

  17. update

    @pcmoritz
    pcmoritz committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    6eb01e0 View commit details
    Browse the repository at this point in the history

  18. fix

    @pcmoritz
    pcmoritz committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    dc89cbc View commit details
    Browse the repository at this point in the history

  19. update

    @pcmoritz
    pcmoritz committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    be60845 View commit details
    Browse the repository at this point in the history

  20. update

    @pcmoritz
    pcmoritz committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    4613cb5 View commit details
    Browse the repository at this point in the history

  21. fix

    @pcmoritz
    pcmoritz committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    3d95d86 View commit details
    Browse the repository at this point in the history

  22. move

    @pcmoritz
    pcmoritz committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    642763f View commit details
    Browse the repository at this point in the history

  23. update

    @pcmoritz
    pcmoritz committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    706e931 View commit details
    Browse the repository at this point in the history

  24. lol

    @pcmoritz
    pcmoritz committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    9a3c78c View commit details
    Browse the repository at this point in the history

  25. fix cuda graph

    @pcmoritz
    pcmoritz committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    1b6f020 View commit details
    Browse the repository at this point in the history

  26. fix

    @pcmoritz
    pcmoritz committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    b09bcec View commit details
    Browse the repository at this point in the history

  27. update

    @pcmoritz
    pcmoritz committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    052e2b3 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2024

  1. update

    @pcmoritz
    pcmoritz committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    b33c6d7 View commit details
    Browse the repository at this point in the history

  2. refactor

    @pcmoritz
    pcmoritz committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    475f58d View commit details
    Browse the repository at this point in the history

  3. update

    @pcmoritz
    pcmoritz committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    56b4880 View commit details
    Browse the repository at this point in the history

  4. revert

    @pcmoritz
    pcmoritz committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    be37154 View commit details
    Browse the repository at this point in the history

  5. format

    @pcmoritz
    pcmoritz committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    9c54d19 View commit details
    Browse the repository at this point in the history

  6. Update vllm/_custom_ops.py 

    Co-authored-by: Woosuk Kwon <[email protected]>
    @pcmoritz @WoosukKwon
    pcmoritz and WoosukKwon authored Apr 25, 2024
    Configuration menu
    Copy the full SHA
    c5155ea View commit details
    Browse the repository at this point in the history

  7. Update vllm/model_executor/layers/fused_moe/fused_moe.py 

    Co-authored-by: Woosuk Kwon <[email protected]>
    @pcmoritz @WoosukKwon
    pcmoritz and WoosukKwon authored Apr 25, 2024
    Configuration menu
    Copy the full SHA
    948cca7 View commit details
    Browse the repository at this point in the history

  8. Update vllm/model_executor/models/mixtral.py 

    Co-authored-by: Woosuk Kwon <[email protected]>
    @pcmoritz @WoosukKwon
    pcmoritz and WoosukKwon authored Apr 25, 2024
    Configuration menu
    Copy the full SHA
    3feb887 View commit details
    Browse the repository at this point in the history

  9. format

    @pcmoritz
    pcmoritz committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    df16316 View commit details
    Browse the repository at this point in the history

  10. support static scales

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    7b6b0fa View commit details
    Browse the repository at this point in the history

  11. fixed example

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    1a3b2e1 View commit details
    Browse the repository at this point in the history

  12. Delete quantize.ipynb

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Apr 25, 2024
    Configuration menu
    Copy the full SHA
    63ad2ef View commit details
    Browse the repository at this point in the history

  13. Update vllm/_custom_ops.py 

    Co-authored-by: Woosuk Kwon <[email protected]>
    @pcmoritz @WoosukKwon
    pcmoritz and WoosukKwon authored Apr 25, 2024
    Configuration menu
    Copy the full SHA
    794f1a1 View commit details
    Browse the repository at this point in the history

  14. update

    @pcmoritz
    pcmoritz committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    c13b6a4 View commit details
    Browse the repository at this point in the history

  15. update

    @pcmoritz
    pcmoritz committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    5a230ed View commit details
    Browse the repository at this point in the history

  16. format

    @pcmoritz
    pcmoritz committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    80069c9 View commit details
    Browse the repository at this point in the history

  17. activation_scale -> act_scale

    @pcmoritz
    pcmoritz committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    5ce17d0 View commit details
    Browse the repository at this point in the history

  18. Update scheme->activation_scheme

    @mgoin
    mgoin committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    5fc0335 View commit details
    Browse the repository at this point in the history

  19. fix dynamic scaling -- need init to zero due to atomic update

    @pcmoritz
    pcmoritz committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    92d5162 View commit details
    Browse the repository at this point in the history

  20. Format

    @mgoin
    mgoin committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    e1bfe10 View commit details
    Browse the repository at this point in the history

  21. Fix tuple type

    @mgoin
    mgoin committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    7242600 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2024

  1. Merge remote-tracking branch 'pcmoritz/mixtral-fp8-static' into fp8-s… 

    …tatic
    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    8512513 View commit details
    Browse the repository at this point in the history

  2. stash tyler's state

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    21ddbb4 View commit details
    Browse the repository at this point in the history

  3. stash

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    d27015c View commit details
    Browse the repository at this point in the history

  4. cutlass working, but slow jitting on hotpath

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    1111f87 View commit details
    Browse the repository at this point in the history

  5. first end to end run with mixtral

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    f5d32ae View commit details
    Browse the repository at this point in the history

  6. added missed file

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    924e8ce View commit details
    Browse the repository at this point in the history

  7. Update run_fp8.py 

    Co-authored-by: Tyler Michael Smith <[email protected]>
    @mgoin @tlrmchlsmth
    mgoin and tlrmchlsmth authored Apr 26, 2024
    Configuration menu
    Copy the full SHA
    823a2e7 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2024

  1. Dynamic FP8 works, but static does not (#213) 

    merging michael's work
    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Apr 27, 2024
    Configuration menu
    Copy the full SHA
    81f42be View commit details
    Browse the repository at this point in the history

  2. static correctness

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    1a4fd8a View commit details
    Browse the repository at this point in the history

  3. static fp8 loading

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    e48c981 View commit details
    Browse the repository at this point in the history

  4. working for dense models

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    02f683e View commit details
    Browse the repository at this point in the history

  5. Update weight_utils.py

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Apr 27, 2024
    Configuration menu
    Copy the full SHA
    81b73ef View commit details
    Browse the repository at this point in the history

  6. moving mixtral updates to separate pr

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    58dbe0f View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into fp8-static

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Apr 27, 2024
    Configuration menu
    Copy the full SHA
    6068dc5 View commit details
    Browse the repository at this point in the history

  8. make ./format pass

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    a8d4b33 View commit details
    Browse the repository at this point in the history

  9. better comments in linear.py

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    5be0970 View commit details
    Browse the repository at this point in the history

  10. better comments in linear.py

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    ef7992b View commit details
    Browse the repository at this point in the history

  11. fixed opt-125

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    0667791 View commit details
    Browse the repository at this point in the history

  12. removed run_fp8.py

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    d8adf14 View commit details
    Browse the repository at this point in the history

  13. format

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    9bb1a2b View commit details
    Browse the repository at this point in the history

  14. Cleanup opt.py

    @mgoin
    mgoin committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    169c9ed View commit details
    Browse the repository at this point in the history

  15. added testing

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    8ef9c7d View commit details
    Browse the repository at this point in the history

  16. ./format.sh

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    c7d6dd6 View commit details
    Browse the repository at this point in the history

  17. fixed typing

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    50b5823 View commit details
    Browse the repository at this point in the history

  18. fixed typing

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    4156ca9 View commit details
    Browse the repository at this point in the history

  19. added warning format

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    3148fc9 View commit details
    Browse the repository at this point in the history

  20. Update opt.py 

    re-added opt change, will fix in another PR
    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Apr 27, 2024
    Configuration menu
    Copy the full SHA
    7846d67 View commit details
    Browse the repository at this point in the history

  21. formatted

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    ba408c6 View commit details
    Browse the repository at this point in the history

  22. Update vllm/model_executor/layers/quantization/fp8.py 

    Co-authored-by: Michael Goin <[email protected]>
    @robertgshaw2-neuralmagic @mgoin
    robertgshaw2-neuralmagic and mgoin authored Apr 27, 2024
    Configuration menu
    Copy the full SHA
    04617fd View commit details
    Browse the repository at this point in the history

  23. Update vllm/model_executor/layers/quantization/fp8.py 

    Co-authored-by: Michael Goin <[email protected]>
    @robertgshaw2-neuralmagic @mgoin
    robertgshaw2-neuralmagic and mgoin authored Apr 27, 2024
    Configuration menu
    Copy the full SHA
    cc3d395 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2024

  1. auto detect shared scale (#214)

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Apr 28, 2024
    Configuration menu
    Copy the full SHA
    f556016 View commit details
    Browse the repository at this point in the history

  2. ./format.sh

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    30bfbd8 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. Update vllm/model_executor/layers/quantization/fp8.py 

    Co-authored-by: Cody Yu <[email protected]>
    @robertgshaw2-neuralmagic @comaniac
    robertgshaw2-neuralmagic and comaniac authored Apr 29, 2024
    Configuration menu
    Copy the full SHA
    572107a View commit details
    Browse the repository at this point in the history

  2. ./format.sh

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    41fbde9 View commit details
    Browse the repository at this point in the history

  3. addressed cody's comments + format

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    f2cd561 View commit details
    Browse the repository at this point in the history

  4. make mypy happy

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    125266e View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. Merge remote-tracking branch 'upstream/main' into fp8-static

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    8a566a7 View commit details
    Browse the repository at this point in the history

  2. test

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    280a4d5 View commit details
    Browse the repository at this point in the history

  3. cleaned up

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    8e1ede1 View commit details
    Browse the repository at this point in the history

  4. Update vllm/model_executor/layers/quantization/fp8.py 

    Co-authored-by: Michael Goin <[email protected]>
    @robertgshaw2-neuralmagic @mgoin
    robertgshaw2-neuralmagic and mgoin authored Apr 30, 2024
    Configuration menu
    Copy the full SHA
    d067428 View commit details
    Browse the repository at this point in the history