Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Parallel Data Loading Shufflable Iterable Datasets/DataStreams #100

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Support Parallel Data Loading Shufflable Iterable Datasets/DataStreams #100

wants to merge 6 commits into from

Commits on Jul 25, 2023

  1. Preliminary implementation for distributed shufflable streams 

    Signed-off-by: Alex-Brooks <[email protected]>
    @alex-jw-brooks
    alex-jw-brooks committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    376cef2 View commit details
    Browse the repository at this point in the history

  2. Add stream partition docstrings and type hints 

    Signed-off-by: Alex-Brooks <[email protected]>
    @alex-jw-brooks
    alex-jw-brooks committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    67ee5a3 View commit details
    Browse the repository at this point in the history

  3. Add test for multiworker shuffling 

    Signed-off-by: Alex-Brooks <[email protected]>
    @alex-jw-brooks
    alex-jw-brooks committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    89b2858 View commit details
    Browse the repository at this point in the history

  4. Stream wrapper linter and code formatting 

    Signed-off-by: Alex-Brooks <[email protected]>
    @alex-jw-brooks
    alex-jw-brooks committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    b7bd4cb View commit details
    Browse the repository at this point in the history

  5. Add notice for multiworker shuffling behaviors 

    Signed-off-by: Alex-Brooks <[email protected]>
    @alex-jw-brooks
    alex-jw-brooks committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    b6d1b82 View commit details
    Browse the repository at this point in the history

  6. Cache datastream length 

    Signed-off-by: Alex-Brooks <[email protected]>
    @alex-jw-brooks
    alex-jw-brooks committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    1392991 View commit details
    Browse the repository at this point in the history