[data] Add microbenchmark for reading and transforming images from preprocessed image files #37610
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Stephanie Wang <[email protected]>
Signed-off-by: Stephanie Wang <[email protected]>
Signed-off-by: Stephanie Wang <[email protected]>
|
Running test here: https://buildkite.com/ray-project/release-tests-pr/builds/46005 |
Signed-off-by: Stephanie Wang <[email protected]>
Signed-off-by: Stephanie Wang <[email protected]>
Signed-off-by: Stephanie Wang <[email protected]>
c21
approved these changes
Jul 21, 2023
raulchen
approved these changes
Jul 21, 2023
stephanie-wang
assigned stephanie-wang and unassigned raulchen, c21, scottjlee and stephanie-wang
stephanie-wang
added
the
@author-action-required
Signed-off-by: Stephanie Wang <[email protected]>
Signed-off-by: Stephanie Wang <[email protected]>
Signed-off-by: Stephanie Wang <[email protected]>
Signed-off-by: Stephanie Wang <[email protected]>
Signed-off-by: Stephanie Wang <[email protected]>
Signed-off-by: Stephanie Wang <[email protected]>
Signed-off-by: Stephanie Wang <[email protected]>
stephanie-wang
force-pushed
the
read-preprocssed-micro
branch
from
c5a6e11 to
710e8fa
Compare
Signed-off-by: Stephanie Wang <[email protected]>
stephanie-wang
added a commit
that referenced
this pull request
Jul 28, 2023
Adds a dep required by #37610. Signed-off-by: Stephanie Wang <[email protected]>
NripeshN
pushed a commit
to NripeshN/ray
that referenced
this pull request
Aug 15, 2023
…eprocessed image files (ray-project#37610) MosaicML streaming and tf.data have their own specialized file formats to improve image load time. This adds a microbenchmark similar to read_images_comparison_microbenchmark_single_node, except that it first preprocesses the images into the right formats. --------- Signed-off-by: Stephanie Wang <[email protected]> Signed-off-by: NripeshN <[email protected]>
NripeshN
pushed a commit
to NripeshN/ray
that referenced
this pull request
Aug 15, 2023
Adds a dep required by ray-project#37610. Signed-off-by: Stephanie Wang <[email protected]> Signed-off-by: NripeshN <[email protected]>
arvind-chandra
pushed a commit
to lmco/ray
that referenced
this pull request
Aug 31, 2023
…eprocessed image files (ray-project#37610) MosaicML streaming and tf.data have their own specialized file formats to improve image load time. This adds a microbenchmark similar to read_images_comparison_microbenchmark_single_node, except that it first preprocesses the images into the right formats. --------- Signed-off-by: Stephanie Wang <[email protected]> Signed-off-by: e428265 <[email protected]>
arvind-chandra
pushed a commit
to lmco/ray
that referenced
this pull request
Aug 31, 2023
Adds a dep required by ray-project#37610. Signed-off-by: Stephanie Wang <[email protected]> Signed-off-by: e428265 <[email protected]>
vymao
pushed a commit
to vymao/ray
that referenced
this pull request
Oct 11, 2023
…eprocessed image files (ray-project#37610) MosaicML streaming and tf.data have their own specialized file formats to improve image load time. This adds a microbenchmark similar to read_images_comparison_microbenchmark_single_node, except that it first preprocesses the images into the right formats. --------- Signed-off-by: Stephanie Wang <[email protected]> Signed-off-by: Victor <[email protected]>
vymao
pushed a commit
to vymao/ray
that referenced
this pull request
Oct 11, 2023
Adds a dep required by ray-project#37610. Signed-off-by: Stephanie Wang <[email protected]> Signed-off-by: Victor <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
MosaicML streaming and tf.data have their own specialized file formats to improve image load time. This adds a microbenchmark similar to read_images_comparison_microbenchmark_single_node, except that it first preprocesses the images into the right formats.
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.