Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor pre tokenization tool #219

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Refactor pre tokenization tool #219

wants to merge 7 commits into from

Commits on Aug 18, 2024

  1. add Shuffle and ParquetReader support

    @eliebak
    eliebak committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    126a90b View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2024

  1. add slurm support and refactor the code

    @eliebak
    eliebak committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    5f5f2f1 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. no email in local pipeline

    @eliebak
    eliebak committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    5a038cb View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. fix: cpus-per-task typo

    @eliebak
    eliebak committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    03f2c5d View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. add mem_per_cpu_gb flag

    @eliebak
    eliebak committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    bb61d8d View commit details
    Browse the repository at this point in the history

  2. fix typo readers => reader

    @eliebak
    eliebak committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    5dc5861 View commit details
    Browse the repository at this point in the history

  3. fix if => elif

    @eliebak
    eliebak committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    7502bae View commit details
    Browse the repository at this point in the history