Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add SophiaG optimizer #844

Open
wants to merge 18 commits into
base: main
Choose a base branch
from
Open

Add SophiaG optimizer #844

wants to merge 18 commits into from

Commits on Oct 28, 2024

  1. Create sophia.py

    @PeanutButterRat
    PeanutButterRat committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    d7feed2 View commit details
    Browse the repository at this point in the history

  2. Register SophiaG as a usable optimizer

    @PeanutButterRat
    PeanutButterRat committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    72e4672 View commit details
    Browse the repository at this point in the history

  3. Add differentiable default

    @PeanutButterRat
    PeanutButterRat committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    daa4e61 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2024

  1. Add finetuning recipe for SophiaG

    @PeanutButterRat
    PeanutButterRat committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    6205ed3 View commit details
    Browse the repository at this point in the history

  2. Add optimizer field to SophiaG recipe

    @PeanutButterRat
    PeanutButterRat committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    5ed973c View commit details
    Browse the repository at this point in the history

  3. Move SophiaG recipe to instruction_finetune.py

    @PeanutButterRat
    PeanutButterRat committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    f8d1eaa View commit details
    Browse the repository at this point in the history

  4. Format and add type hints

    @PeanutButterRat
    PeanutButterRat committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    3f28592 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2024

  1. Join sophiag and _single_tensor_sophiag

    @PeanutButterRat
    PeanutButterRat committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    f006e3b View commit details
    Browse the repository at this point in the history

  2. Add license header

    @PeanutButterRat
    PeanutButterRat committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    c7b5ce2 View commit details
    Browse the repository at this point in the history

  3. Add docstrings to SophiaG and SophiaGConfig

    @PeanutButterRat
    PeanutButterRat committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    e7f3338 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2024

  1. Add hessian update and k hyperparameter

    @PeanutButterRat
    PeanutButterRat committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    e5cdff4 View commit details
    Browse the repository at this point in the history

  2. Fix usage of k hyperparameter

    @PeanutButterRat
    PeanutButterRat committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    e3c81cd View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2024

  1. Add type hints and format SophiaG

    @PeanutButterRat
    PeanutButterRat committed Nov 9, 2024
    Configuration menu
    Copy the full SHA
    d1e6d57 View commit details
    Browse the repository at this point in the history

  2. Remove unused imports

    @PeanutButterRat
    PeanutButterRat committed Nov 9, 2024
    Configuration menu
    Copy the full SHA
    50fd1fb View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2024

  1. Explicitly cast step to an int

    @PeanutButterRat
    PeanutButterRat committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    8e13174 View commit details
    Browse the repository at this point in the history

  2. Create test_sophiag.py

    @PeanutButterRat
    PeanutButterRat committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    5f30548 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2024

  1. Add SophiaG unit test

    @PeanutButterRat
    PeanutButterRat committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    8ae5403 View commit details
    Browse the repository at this point in the history

  2. Change SophiaG test to check weights

    @PeanutButterRat
    PeanutButterRat committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    148f647 View commit details
    Browse the repository at this point in the history