Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge load / resize / cache to optimize data loading efficiency for classification #2438

Merged
merged 15 commits into from
Aug 22, 2023

Conversation

goodsong81
Copy link
Contributor

@goodsong81 goodsong81 commented Aug 18, 2023

Summary

[Changes]

  • Implement base LoadResizeDataFromOTXDataset to merge & wrap
    • LoadImageFromOTXDataset
    • (Optional) LoadAnnotationFromOTXDataset
    • (Optional) Resize
  • Enable in-memory cache after resize for memory efficiency
  • Implement ResizeTo to run Resize only if the expected size is different from current size
  • Apply to classification (other tasks will be handled in upcoming PRs)
  • Apply to configurable input size

[Results]

Branch Size Cache Data time Iter time E2E time
develop 224x224 off 0.0997 0.1670 15.94
  224x224 35M/2G 0.0304 0.0970 13.29
  64x64 off 0.0897 0.1349 13.21
  64x64 35M/2G 0.0164 0.0624 10.61
feat/load-resize-cache 224x224 off 0.1044 0.1745 13.07
  224x224 18M/2G 0.0304 0.0983 11.06
  64x64 off 0.0881 0.1348 12.28
  64x64 9M/2G 0.0176 0.0636 10.03

How to test

[Example CMD]

otx train src/otx/algorithms/classification/configs/efficientnet_b0_cls_incr/template.yaml --train-data-roots ./data/CUB_200_2011_64/train --val-data-roots ./data/CUB_200_2011_64/val --workspace outputs/load-resize-cls//effb0_CUB_200_2011_64_64x64_cache2g params --learning_parameters.input_size 64x64 --algo_backend.mem_cache_size 2000000000

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have added e2e tests for validation.
  • I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
  • I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
  • I have linked related issues.

License

  • I submit my code changes under the same Apache License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

@goodsong81 goodsong81 added the ENHANCE Enhancement of existing features label Aug 18, 2023
@github-actions github-actions bot added ALGO Any changes in OTX Algo Tasks implementation TEST Any changes in tests labels Aug 18, 2023
Signed-off-by: Songki Choi <[email protected]>
@goodsong81 goodsong81 marked this pull request as ready for review August 18, 2023 06:38
@goodsong81 goodsong81 requested a review from a team as a code owner August 18, 2023 06:38
@goodsong81 goodsong81 requested a review from eunwoosh August 18, 2023 06:39
eunwoosh
eunwoosh previously approved these changes Aug 18, 2023
Copy link
Contributor

@eunwoosh eunwoosh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your work! LGTM but I have some questions. please take a look.

jaegukhyun
jaegukhyun previously approved these changes Aug 21, 2023
Signed-off-by: Songki Choi <[email protected]>
eunwoosh
eunwoosh previously approved these changes Aug 22, 2023
Signed-off-by: Songki Choi <[email protected]>
@goodsong81 goodsong81 requested a review from eunwoosh August 22, 2023 07:26
@goodsong81 goodsong81 merged commit 59acdcd into openvinotoolkit:develop Aug 22, 2023
@goodsong81 goodsong81 deleted the feat/load-resize-cache branch August 22, 2023 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ALGO Any changes in OTX Algo Tasks implementation ENHANCE Enhancement of existing features TEST Any changes in tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants