Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding PiSSA as an optional initialization method of LoRA #1626

Merged
merged 112 commits into from
May 15, 2024
Merged

Adding PiSSA as an optional initialization method of LoRA #1626

merged 112 commits into from
May 15, 2024

Commits on Mar 14, 2024

  1. add pisa_init

    @MFX12138
    MFX12138 committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    a320ff7 View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2024

  1. Update layer.py

    @fxmeng
    fxmeng authored Mar 15, 2024
    Configuration menu
    Copy the full SHA
    13155fe View commit details
    Browse the repository at this point in the history

  2. Update layer.py

    @fxmeng
    fxmeng authored Mar 15, 2024
    Configuration menu
    Copy the full SHA
    89bc355 View commit details
    Browse the repository at this point in the history

  3. Update layer.py

    @fxmeng
    fxmeng authored Mar 15, 2024
    Configuration menu
    Copy the full SHA
    9f00cb9 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2024

  1. svd_lowrank and bitsandbytes

    @fxmeng
    fxmeng committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    05edb1d View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2024

  1. Update layer.py

    @fxmeng
    fxmeng authored Mar 27, 2024
    Configuration menu
    Copy the full SHA
    58b2fc2 View commit details
    Browse the repository at this point in the history

Commits on Apr 2, 2024

  1. Update layer.py 

    rename as pissa
    @fxmeng
    fxmeng authored Apr 2, 2024
    Configuration menu
    Copy the full SHA
    f105336 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'huggingface:main' into main

    @fxmeng
    fxmeng authored Apr 2, 2024
    Configuration menu
    Copy the full SHA
    ec15caf View commit details
    Browse the repository at this point in the history

Commits on Apr 9, 2024

  1. make style

    @fxmeng
    fxmeng committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    3a9e9bd View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. support scaling != 1

    @fxmeng
    fxmeng committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    d1bb5dd View commit details
    Browse the repository at this point in the history

  2. usv^t=w->vsu^t=w^t

    @fxmeng
    fxmeng committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    823c166 View commit details
    Browse the repository at this point in the history

  3. from torch import svd_lowrank

    @fxmeng
    fxmeng committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    65286be View commit details
    Browse the repository at this point in the history

  4. raise init_lora_weights error

    @fxmeng
    fxmeng committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    bae0da9 View commit details
    Browse the repository at this point in the history

  5. dequantize_bnb_weight

    @fxmeng
    fxmeng committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    e912970 View commit details
    Browse the repository at this point in the history

  6. use pissa without changing base model

    @fxmeng
    fxmeng committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    c679a50 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2024

  1. comment on \Delta AB

    @fxmeng
    fxmeng committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    7fabf84 View commit details
    Browse the repository at this point in the history

  2. save_initial_pissa_and_residual_model

    @fxmeng
    fxmeng committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    bc2b566 View commit details
    Browse the repository at this point in the history

  3. finetune in 4 bit

    @fxmeng
    fxmeng committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    0b8aea6 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2024

  1. finetune in 4 bit

    @fxmeng
    fxmeng committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    cf5eac3 View commit details
    Browse the repository at this point in the history

  2. finetune in 4 bit

    @fxmeng
    fxmeng committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    decaea6 View commit details
    Browse the repository at this point in the history

  3. finetune in 4 bit

    @fxmeng
    fxmeng committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    5dd82e3 View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2024

  1. add pissa_utils

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    d2c8aab View commit details
    Browse the repository at this point in the history

  2. add pissa_utils

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    ce2948a View commit details
    Browse the repository at this point in the history

  3. add pissa_utils

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    0e15216 View commit details
    Browse the repository at this point in the history

  4. add pissa_utils

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    3488f69 View commit details
    Browse the repository at this point in the history

  5. add pissa_utils

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    814f1c7 View commit details
    Browse the repository at this point in the history

  6. add pissa_utils

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    db99cb3 View commit details
    Browse the repository at this point in the history

  7. add pissa_utils

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    374e309 View commit details
    Browse the repository at this point in the history

  8. make style

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    fb75ab3 View commit details
    Browse the repository at this point in the history

  9. make style

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    47f8534 View commit details
    Browse the repository at this point in the history

  10. svd first quantization

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    9a5067a View commit details
    Browse the repository at this point in the history

  11. optinal path

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    079d316 View commit details
    Browse the repository at this point in the history

  12. optinal path

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    2edfb07 View commit details
    Browse the repository at this point in the history

  13. optinal path

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    8408959 View commit details
    Browse the repository at this point in the history

  14. optinal path

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    528e49f View commit details
    Browse the repository at this point in the history

  15. optinal path

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    e3458c8 View commit details
    Browse the repository at this point in the history

  16. optinal path

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    f543fc1 View commit details
    Browse the repository at this point in the history

  17. optinal path

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    11d9843 View commit details
    Browse the repository at this point in the history

  18. optinal path

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    1d1e162 View commit details
    Browse the repository at this point in the history

  19. optinal path

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    5565b9b View commit details
    Browse the repository at this point in the history

  20. quantization_config

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    bdfd306 View commit details
    Browse the repository at this point in the history

  21. quantization_config

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    bd7c814 View commit details
    Browse the repository at this point in the history

  22. quantization_config

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    3486fad View commit details
    Browse the repository at this point in the history

  23. pissa_finetuning

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    f7d0453 View commit details
    Browse the repository at this point in the history

  24. pissa_finetuning

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    c2c8ac3 View commit details
    Browse the repository at this point in the history

  25. lora config

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    4d9b74b View commit details
    Browse the repository at this point in the history

  26. lora config

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    ae0c3e6 View commit details
    Browse the repository at this point in the history

  27. lora config

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    606a692 View commit details
    Browse the repository at this point in the history

  28. readme

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    0e961ab View commit details
    Browse the repository at this point in the history

  29. readme

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    3893276 View commit details
    Browse the repository at this point in the history

  30. lora.md

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    57f65ff View commit details
    Browse the repository at this point in the history

  31. make style

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    33e1a8c View commit details
    Browse the repository at this point in the history

  32. make style

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    51161a5 View commit details
    Browse the repository at this point in the history

  33. readme

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    52110a5 View commit details
    Browse the repository at this point in the history

  34. readme

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    30e7673 View commit details
    Browse the repository at this point in the history

  35. readme

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    b76f82b View commit details
    Browse the repository at this point in the history

  36. test_init

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    3917777 View commit details
    Browse the repository at this point in the history

  37. test_init

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    2dc3ddd View commit details
    Browse the repository at this point in the history

  38. test_init

    @fxmeng
    fxmeng committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    f0642c6 View commit details
    Browse the repository at this point in the history

  39. test

    @fxmeng
    fxmeng authored Apr 15, 2024
    Configuration menu
    Copy the full SHA
    7b8af8e View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2024

  1. TrainingArguments

    @fxmeng
    fxmeng committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    f8889ee View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' of https://github.com/fxmeng/peft

    @fxmeng
    fxmeng committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    83deb5a View commit details
    Browse the repository at this point in the history

  3. TrainingArguments

    @fxmeng
    fxmeng committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    41ed2b3 View commit details
    Browse the repository at this point in the history

  4. TrainingArguments

    @fxmeng
    fxmeng committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    c05cdae View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2024

  1. save_init_and_ft

    @fxmeng
    fxmeng committed Apr 20, 2024
    Configuration menu
    Copy the full SHA
    c53bd85 View commit details
    Browse the repository at this point in the history

  2. unload

    @fxmeng
    fxmeng committed Apr 20, 2024
    Configuration menu
    Copy the full SHA
    3cbacb0 View commit details
    Browse the repository at this point in the history

  3. explain W^{res}

    @fxmeng
    fxmeng committed Apr 20, 2024
    Configuration menu
    Copy the full SHA
    c4222ac View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. save_and_load

    @fxmeng
    fxmeng committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    ae69146 View commit details
    Browse the repository at this point in the history

  2. save_and_load

    @fxmeng
    fxmeng committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    082c120 View commit details
    Browse the repository at this point in the history

  3. make style

    @fxmeng
    fxmeng committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    efbd5aa View commit details
    Browse the repository at this point in the history

  4. save_as_lora

    @fxmeng
    fxmeng committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    26feaa2 View commit details
    Browse the repository at this point in the history

  5. save_and_load

    @fxmeng
    fxmeng committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    a3613c5 View commit details
    Browse the repository at this point in the history

  6. save_and_load

    @fxmeng
    fxmeng committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    07eda02 View commit details
    Browse the repository at this point in the history

  7. readme

    @fxmeng
    fxmeng committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    c9b683b View commit details
    Browse the repository at this point in the history

  8. readme

    @fxmeng
    fxmeng committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    694e6de View commit details
    Browse the repository at this point in the history

  9. readme

    @fxmeng
    fxmeng committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    629941d View commit details
    Browse the repository at this point in the history

  10. readme

    @fxmeng
    fxmeng committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    6fbc20f View commit details
    Browse the repository at this point in the history

  11. test quantization error

    @fxmeng
    fxmeng committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    412effb View commit details
    Browse the repository at this point in the history

  12. test

    @fxmeng
    fxmeng committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    bfb8e84 View commit details
    Browse the repository at this point in the history

  13. test

    @fxmeng
    fxmeng committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    964aee2 View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2024

  1. test

    @fxmeng
    fxmeng committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    0b47f9a View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2024

  1. lora_model.subtract_pissa_init

    @fxmeng
    fxmeng committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    1f44f8b View commit details
    Browse the repository at this point in the history

  2. convert_pissa_to_lora

    @fxmeng
    fxmeng committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    8608303 View commit details
    Browse the repository at this point in the history

  3. initial_adapter

    @fxmeng
    fxmeng committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    49d1257 View commit details
    Browse the repository at this point in the history

  4. initial_adapter

    @fxmeng
    fxmeng committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    83ad3bd View commit details
    Browse the repository at this point in the history

  5. test

    @fxmeng
    fxmeng committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    c575450 View commit details
    Browse the repository at this point in the history

  6. make style

    @fxmeng
    fxmeng committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    29f987c View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. Update tests/test_initialization.py 

    Co-authored-by: Benjamin Bossan <[email protected]>
    @fxmeng @BenjaminBossan
    fxmeng and BenjaminBossan authored May 1, 2024
    Configuration menu
    Copy the full SHA
    e7c8fe6 View commit details
    Browse the repository at this point in the history

  2. Update src/peft/tuners/lora/model.py 

    Co-authored-by: Benjamin Bossan <[email protected]>
    @fxmeng @BenjaminBossan
    fxmeng and BenjaminBossan authored May 1, 2024
    Configuration menu
    Copy the full SHA
    3ded9bd View commit details
    Browse the repository at this point in the history

  3. Update src/peft/tuners/lora/model.py 

    Co-authored-by: Benjamin Bossan <[email protected]>
    @fxmeng @BenjaminBossan
    fxmeng and BenjaminBossan authored May 1, 2024
    Configuration menu
    Copy the full SHA
    1ff7d94 View commit details
    Browse the repository at this point in the history

  4. Update docs/source/developer_guides/lora.md 

    Co-authored-by: Benjamin Bossan <[email protected]>
    @fxmeng @BenjaminBossan
    fxmeng and BenjaminBossan authored May 1, 2024
    Configuration menu
    Copy the full SHA
    dfa2179 View commit details
    Browse the repository at this point in the history

  5. Update src/peft/peft_model.py 

    Co-authored-by: Benjamin Bossan <[email protected]>
    @fxmeng @BenjaminBossan
    fxmeng and BenjaminBossan authored May 1, 2024
    Configuration menu
    Copy the full SHA
    0234aee View commit details
    Browse the repository at this point in the history

  6. save_pissa_as_lora as a sub-method

    @fxmeng
    fxmeng committed May 1, 2024
    Configuration menu
    Copy the full SHA
    06505cc View commit details
    Browse the repository at this point in the history

  7. convert_pissa_to_lora

    @fxmeng
    fxmeng committed May 1, 2024
    Configuration menu
    Copy the full SHA
    fd01b79 View commit details
    Browse the repository at this point in the history

  8. test quant

    @fxmeng
    fxmeng committed May 1, 2024
    Configuration menu
    Copy the full SHA
    da49fb2 View commit details
    Browse the repository at this point in the history

  9. warning

    @fxmeng
    fxmeng committed May 1, 2024
    Configuration menu
    Copy the full SHA
    9f25bc1 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2024

  1. preprocess

    @fxmeng
    fxmeng committed May 3, 2024
    Configuration menu
    Copy the full SHA
    95bfbb1 View commit details
    Browse the repository at this point in the history

  2. test_lora_pissa_linear_init_default

    @fxmeng
    fxmeng committed May 3, 2024
    Configuration menu
    Copy the full SHA
    45636e5 View commit details
    Browse the repository at this point in the history

  3. quantized_convert_errors

    @fxmeng
    fxmeng committed May 3, 2024
    Configuration menu
    Copy the full SHA
    bd76657 View commit details
    Browse the repository at this point in the history

  4. solve conflicts

    @fxmeng
    fxmeng committed May 3, 2024
    Configuration menu
    Copy the full SHA
    61efdf7 View commit details
    Browse the repository at this point in the history

Commits on May 4, 2024

  1. deepcopy

    @fxmeng
    fxmeng committed May 4, 2024
    Configuration menu
    Copy the full SHA
    e2a943d View commit details
    Browse the repository at this point in the history

  2. test_lora_pissa_conversion_same_output_after_loading_with_quantization

    @fxmeng
    fxmeng committed May 4, 2024
    Configuration menu
    Copy the full SHA
    005087b View commit details
    Browse the repository at this point in the history

Commits on May 6, 2024

  1. nuclear_norm_error

    @fxmeng
    fxmeng committed May 6, 2024
    Configuration menu
    Copy the full SHA
    52c4620 View commit details
    Browse the repository at this point in the history

  2. nuclear_norm_error

    @fxmeng
    fxmeng committed May 6, 2024
    Configuration menu
    Copy the full SHA
    274001c View commit details
    Browse the repository at this point in the history

  3. nuclear_norm_error

    @fxmeng
    fxmeng committed May 6, 2024
    Configuration menu
    Copy the full SHA
    e0fd841 View commit details
    Browse the repository at this point in the history

Commits on May 7, 2024

  1. fix a bug

    @fxmeng
    fxmeng committed May 7, 2024
    Configuration menu
    Copy the full SHA
    c803300 View commit details
    Browse the repository at this point in the history

  2. fix a bug

    @fxmeng
    fxmeng committed May 7, 2024
    Configuration menu
    Copy the full SHA
    b1f0038 View commit details
    Browse the repository at this point in the history

  3. ruff

    @fxmeng
    fxmeng committed May 7, 2024
    Configuration menu
    Copy the full SHA
    232e3d2 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' of https://github.com/fxmeng/peft

    @fxmeng
    fxmeng committed May 7, 2024
    Configuration menu
    Copy the full SHA
    9482167 View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. doc-builder

    @MFX12138
    MFX12138 committed May 8, 2024
    Configuration menu
    Copy the full SHA
    74f470b View commit details
    Browse the repository at this point in the history

  2. quantization_test

    @MFX12138
    MFX12138 committed May 8, 2024
    Configuration menu
    Copy the full SHA
    8a6e429 View commit details
    Browse the repository at this point in the history

  3. 更新 test_gpu_examples.py 

    Co-authored-by: Benjamin Bossan <[email protected]>
    @fxmeng @BenjaminBossan
    fxmeng and BenjaminBossan authored May 8, 2024
    Configuration menu
    Copy the full SHA
    60596ec View commit details
    Browse the repository at this point in the history