Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v3] Training refactor - MultiGPU, loss logging, bf16, etc. #2449

Merged
merged 90 commits into from
Apr 25, 2024
Merged

[v3] Training refactor - MultiGPU, loss logging, bf16, etc. #2449

merged 90 commits into from
Apr 25, 2024

Commits on Oct 26, 2022

  1. See UKPLab#1638: Adds huggingface trainer for sentence transformers

    @matthewfranglen
    matthewfranglen committed Oct 26, 2022
    Configuration menu
    Copy the full SHA
    d4aea88 View commit details
    Browse the repository at this point in the history

  2. Fix type of tokenizer

    @matthewfranglen
    matthewfranglen committed Oct 26, 2022
    Configuration menu
    Copy the full SHA
    602c0a4 View commit details
    Browse the repository at this point in the history

  3. Get the trainer using the feature collation

    @matthewfranglen
    matthewfranglen committed Oct 26, 2022
    Configuration menu
    Copy the full SHA
    9d984c9 View commit details
    Browse the repository at this point in the history

  4. Update the docstring to reflect changes

    @matthewfranglen
    matthewfranglen committed Oct 26, 2022
    Configuration menu
    Copy the full SHA
    c5f3dde View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2024

  1. Merge branch 'master' of https://github.com/UKPLab/sentence-transformers 

     into pr-1733
    @tomaarsen
    tomaarsen committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    0b179cf View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2024

  1. Initial draft for refactoring training usig the Transformers Trainer

    @tomaarsen
    tomaarsen committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    3ebcc27 View commit details
    Browse the repository at this point in the history

  2. Separate 'fit' functionality (new and old) into a mixin

    @tomaarsen
    tomaarsen committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    09fce93 View commit details
    Browse the repository at this point in the history

  3. Resolve test issues

    @tomaarsen
    tomaarsen committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    cf7d1c7 View commit details
    Browse the repository at this point in the history

  4. Reformat

    @tomaarsen
    tomaarsen committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    14fac0a View commit details
    Browse the repository at this point in the history

  5. Update the imports

    @tomaarsen
    tomaarsen committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    8a8912b View commit details
    Browse the repository at this point in the history

  6. Add TODO regarding custom label columns

    @tomaarsen
    tomaarsen committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    dea5c94 View commit details
    Browse the repository at this point in the history

  7. Remove dead code

    @tomaarsen
    tomaarsen committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    ff8ef4d View commit details
    Browse the repository at this point in the history

  8. Don't provide the trainer to the eval sampler

    @tomaarsen
    tomaarsen committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    636ba4d View commit details
    Browse the repository at this point in the history

  9. Introduce datasets as a dependency

    @tomaarsen
    tomaarsen committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    7840bac View commit details
    Browse the repository at this point in the history

  10. Introduce "accelerate" as a dependency

    @tomaarsen
    tomaarsen committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    c0461e9 View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2024

  1. Avoid use_amp on CPU tests

    @tomaarsen
    tomaarsen committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    eb3ccfe View commit details
    Browse the repository at this point in the history

  2. Specify that SentenceTransformer is a class, not a module

    @tomaarsen
    tomaarsen committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    aaf72f3 View commit details
    Browse the repository at this point in the history

  3. Avoid circular import

    @tomaarsen
    tomaarsen committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    9a3912b View commit details
    Browse the repository at this point in the history

  4. Remove | used as an "or" operator in typing

    @tomaarsen
    tomaarsen committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    036bc40 View commit details
    Browse the repository at this point in the history

Commits on Feb 5, 2024

  1. Use test evaluator after training, as intended

    @tomaarsen
    tomaarsen committed Feb 5, 2024
    Configuration menu
    Copy the full SHA
    37ab364 View commit details
    Browse the repository at this point in the history

Commits on Feb 6, 2024

  1. Use tokenize function instead of tokenizer; 

    Add EvaluatorCallback which calls the evaluator on every epoch (for BC);
Stop saving "do_lower_case" from Transformer;
    @tomaarsen
    tomaarsen committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    de51acd View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into v3/trainer_refactor

    @tomaarsen
    tomaarsen committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    0b3811e View commit details
    Browse the repository at this point in the history

  3. Reformat

    @tomaarsen
    tomaarsen committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    645bea8 View commit details
    Browse the repository at this point in the history

  4. Revert Transformer tokenizer changes

    @tomaarsen
    tomaarsen committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    9cfeb0c View commit details
    Browse the repository at this point in the history

  5. Add support for the tokenizer to return more than just input_ids & at… 

    …tention_masks

Required for LSTM
    @tomaarsen
    tomaarsen committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    69800a3 View commit details
    Browse the repository at this point in the history

  6. Use the test evaluators after training the examples

    @tomaarsen
    tomaarsen committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    b0dce5d View commit details
    Browse the repository at this point in the history

  7. Use pure torch for BoW tokenization

    @tomaarsen
    tomaarsen committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    78efaec View commit details
    Browse the repository at this point in the history

  8. Use dev evaluator for BiLSTM - test fails

    @tomaarsen
    tomaarsen committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    0ec8ef4 View commit details
    Browse the repository at this point in the history

  9. Add Trainer support for BoW-based models

    @tomaarsen
    tomaarsen committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    5e69caf View commit details
    Browse the repository at this point in the history

  10. Pass epoch to evaluator in every-epoch callback 

    For fit backwards compatibility
    @tomaarsen
    tomaarsen committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    8a7c56b View commit details
    Browse the repository at this point in the history

  11. Run formatting

    @tomaarsen
    tomaarsen committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    016983a View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2024

  1. Use steps_per_epoch to set max_steps if possible

    @tomaarsen
    tomaarsen committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    d2d1f1b View commit details
    Browse the repository at this point in the history

  2. Ignore extracting dataloader arguments for now

    @tomaarsen
    tomaarsen committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    66d15a1 View commit details
    Browse the repository at this point in the history

  3. Remove dead code

    @tomaarsen
    tomaarsen committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    170843c View commit details
    Browse the repository at this point in the history

  4. Allow both "label" and "score" columns for labels

    @tomaarsen
    tomaarsen committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    af60a59 View commit details
    Browse the repository at this point in the history

  5. Reformatting

    @tomaarsen
    tomaarsen committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    620003b View commit details
    Browse the repository at this point in the history

  6. Improve errors if datasets don't match with loss dictionary well

    @tomaarsen
    tomaarsen committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    7aed046 View commit details
    Browse the repository at this point in the history

  7. Made tests more consistent; list instead of set

    @tomaarsen
    tomaarsen committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    3f02af2 View commit details
    Browse the repository at this point in the history

  8. Simplify trainer with DatasetDict

    @tomaarsen
    tomaarsen committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    d79bffa View commit details
    Browse the repository at this point in the history

  9. Implement a proportional sampler in addition to round robin

    @tomaarsen
    tomaarsen committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    984b0d6 View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2024

  1. Add CLIP finetuning support to the Trainer

    @tomaarsen
    tomaarsen committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    581b023 View commit details
    Browse the repository at this point in the history

  2. Start updating evaluators to return dictionaries

    @tomaarsen
    tomaarsen committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    2eb0cb3 View commit details
    Browse the repository at this point in the history

  3. Reformat

    @tomaarsen
    tomaarsen committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    96063f9 View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2024

  1. Merge branch 'master' into v3/trainer_refactor

    @tomaarsen
    tomaarsen committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    c736f44 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. Hackishly insert the DataParallel model into the loss function

    @tomaarsen
    tomaarsen committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    424f807 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into v3/trainer_refactor

    @tomaarsen
    tomaarsen committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    5abff20 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2024

  1. Allow for fsdp=["full_shard", "auto_wrap"] 

    with fsdp_config={"transformer_layer_cls_to_wrap": "BertLayer"}
    @tomaarsen
    tomaarsen committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    37d6007 View commit details
    Browse the repository at this point in the history

  2. Re-add support for DataParallel

    @tomaarsen
    tomaarsen committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    732b7a2 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2024

  1. Use 'ParallelMode.NOT_PARALLEL'

    @tomaarsen
    tomaarsen committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    87fac73 View commit details
    Browse the repository at this point in the history

  2. Prevent crash with DDP & an evaluation set

    @tomaarsen
    tomaarsen committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    4982cb7 View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2024

  1. When training with multiple datasets, add "dataset_name" column 

    Rather than relying on some Batch Sampler hacking (which fails with some distributed training approaches)
    @tomaarsen
    tomaarsen committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    bfa29a8 View commit details
    Browse the repository at this point in the history

  2. Update type hints: make loss & evaluator optional 

    Co-authored-by: Wang Bo <[email protected]>
    @tomaarsen @bwanglzu
    tomaarsen and bwanglzu authored Mar 13, 2024
    Configuration menu
    Copy the full SHA
    7351908 View commit details
    Browse the repository at this point in the history

  3. Set correct superclasses for samplers

    @tomaarsen
    tomaarsen committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    d1fa120 View commit details
    Browse the repository at this point in the history

  4. Override 'accelerator.even_batches' as it's incompatible with multi-d… 

    …ataset
    @tomaarsen
    tomaarsen committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    4f97f21 View commit details
    Browse the repository at this point in the history

  5. Throw exception if "return_loss" or "dataset_name" columns are used

    @tomaarsen
    tomaarsen committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    deeb1a8 View commit details
    Browse the repository at this point in the history

  6. Set min. version for accelerate

    @tomaarsen
    tomaarsen committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    a0071a5 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2024

  1. Merge branch 'master' into v3/trainer_refactor

    @tomaarsen
    tomaarsen committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    bf940bc View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. Heavily extend model card generation

    @tomaarsen
    tomaarsen committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    f937325 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into v3/trainer_refactor

    @tomaarsen
    tomaarsen committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    59eed8c View commit details
    Browse the repository at this point in the history

  3. Remove some dead code

    @tomaarsen
    tomaarsen committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    078ac76 View commit details
    Browse the repository at this point in the history

  4. Fix evaluator type hints

    @tomaarsen
    tomaarsen committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    cd924ca View commit details
    Browse the repository at this point in the history

  5. Ensure that 'model_card_template.md' is included in the built package

    @tomaarsen
    tomaarsen committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    518a71f View commit details
    Browse the repository at this point in the history

  6. Rephrase comments slightly

    @tomaarsen
    tomaarsen committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    19eaadb View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2024

  1. Heavily refactor samplers; add no duplicates/group by label samplers

    @tomaarsen
    tomaarsen committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    459ca98 View commit details
    Browse the repository at this point in the history

Commits on Apr 17, 2024

  1. Ensure that data_loader.dataset exists in FitMixin

    @tomaarsen
    tomaarsen committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    6f7816b View commit details
    Browse the repository at this point in the history

  2. Adopt 8 as the default batch

    @tomaarsen
    tomaarsen committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    21ae280 View commit details
    Browse the repository at this point in the history

  3. Fix logging error in example

    @tomaarsen
    tomaarsen committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    9062671 View commit details
    Browse the repository at this point in the history

  4. Remove the deprecated correct_bias

    @tomaarsen
    tomaarsen committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    650c4eb View commit details
    Browse the repository at this point in the history

  5. Simplify with walrus operator

    @tomaarsen
    tomaarsen committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    778480f View commit details
    Browse the repository at this point in the history

  6. Fix some bugs in set_widget_examples with short datasets

    @tomaarsen
    tomaarsen committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    cd660e8 View commit details
    Browse the repository at this point in the history

  7. Improve docstring slightly

    @tomaarsen
    tomaarsen committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    17d99fe View commit details
    Browse the repository at this point in the history

  8. Add edge case in case training data has an unrecognized format

    @tomaarsen
    tomaarsen committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    8804443 View commit details
    Browse the repository at this point in the history

  9. Fix extracting dataset metadata

    @tomaarsen
    tomaarsen committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    d1b0b9d View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2024

  1. Remove moot TYPE_CHECKING

    @tomaarsen
    tomaarsen committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    600bc6f View commit details
    Browse the repository at this point in the history

  2. Set base model when loading a ST model also

    @tomaarsen
    tomaarsen committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    bcff5d3 View commit details
    Browse the repository at this point in the history

  3. Add test_dataloader, add prefetch_factor to dataloaders

    @tomaarsen
    tomaarsen committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    1c4c720 View commit details
    Browse the repository at this point in the history

  4. Resolve predict_example fix; fix newlines in text

    @tomaarsen
    tomaarsen committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    959c19b View commit details
    Browse the repository at this point in the history

  5. Fix bug in compute_dataset_metrics examples

    @tomaarsen
    tomaarsen committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    56321ae View commit details
    Browse the repository at this point in the history

  6. Add call to action in ValueError

    @tomaarsen
    tomaarsen committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    3de8910 View commit details
    Browse the repository at this point in the history

  7. Reuse original model card if no training is done

    @tomaarsen
    tomaarsen committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    e7717b6 View commit details
    Browse the repository at this point in the history

  8. Also collect nested losses (e.g. MatryoshkaLoss) and make losses in tags

    @tomaarsen
    tomaarsen committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    92ac930 View commit details
    Browse the repository at this point in the history

  9. Remove generated tag; keep loss: prefix on tags

    @tomaarsen
    tomaarsen committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    a7688a1 View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2024

  1. Remove unused arguments

    @tomaarsen
    tomaarsen committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    22f9edf View commit details
    Browse the repository at this point in the history

  2. Add support for "best model step" in model card

    @tomaarsen
    tomaarsen committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    406d0d0 View commit details
    Browse the repository at this point in the history

  3. Make hyperparameters code-formatted

    @tomaarsen
    tomaarsen committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    1f233e9 View commit details
    Browse the repository at this point in the history

  4. Fix load_best_model for Transformers models, prevent for non-Transfor… 

    …mers
    @tomaarsen
    tomaarsen committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    7eb077f View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' of https://github.com/UKPLab/sentence-transformers 

     into v3/trainer_refactor
    @tomaarsen
    tomaarsen committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    db025eb View commit details
    Browse the repository at this point in the history

  6. Store base_model_revision in model_card_data

    @tomaarsen
    tomaarsen committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    5a1e2f5 View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2024

  1. Prevent crash when loading a local model

    @tomaarsen
    tomaarsen committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    7b42232 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. Allow for bfloat16 inference

    @tomaarsen
    tomaarsen committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    56922e3 View commit details
    Browse the repository at this point in the history