Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge load / resize / cache to optimize data loading efficiency for detection & instance segmentation #2453

Merged

Conversation

goodsong81
Copy link
Contributor

@goodsong81 goodsong81 commented Aug 25, 2023

Summary

[Changes]

[Results (sampled)]

  • peanut (1920x1080) / YOLOX (640x640) / default setting (num_workers=2)
Branch Size Cache Data time Iter time E2E time Accuracy
develop 256x256 off 0.3891 0.4666 94.97 1.0
  256x256 168M/2G 0.0767 0.1570 41.31 1.0
feat/load-resize-cache-detiseg 256x256 off 0.3072 0.3847 78.56 1.0
  256x256 64M/2G 0.0253 0.1001 32.05 1.0

How to test

[Example CMD]

otx train src/otx/algorithms/detection/configs/detection/cspdarknet_yolox/template.yaml --train-data-roots ./data/peanut --workspace outputs/load-resize-det/feat/load-resize-cache-detiseg/peanut_cspdarknet_yolox_256x256_cache2g params --learning_parameters.input_size 256x256 --algo_backend.mem_cache_size 2000000000

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have added e2e tests for validation.
  • I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
  • I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
  • I have linked related issues.

License

  • I submit my code changes under the same Apache License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

@goodsong81 goodsong81 added the ENHANCE Enhancement of existing features label Aug 25, 2023
@goodsong81 goodsong81 added this to the 1.5.0 milestone Aug 25, 2023
@github-actions github-actions bot added ALGO Any changes in OTX Algo Tasks implementation TEST Any changes in tests labels Aug 25, 2023
Signed-off-by: Songki Choi <[email protected]>
@goodsong81 goodsong81 marked this pull request as ready for review August 28, 2023 04:31
@goodsong81 goodsong81 requested a review from a team as a code owner August 28, 2023 04:31
eunwoosh
eunwoosh previously approved these changes Aug 28, 2023
Copy link
Contributor

@eunwoosh eunwoosh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I have a minor question. Please take a look.

Copy link
Contributor

@jaegukhyun jaegukhyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason that this changes does not affect to all data pipeline.py for example detr_data_pipeline.py

jaegukhyun
jaegukhyun previously approved these changes Aug 29, 2023
@goodsong81 goodsong81 merged commit 2cb5d2b into openvinotoolkit:develop Aug 29, 2023
@goodsong81 goodsong81 deleted the feat/load-resize-cache-detiseg branch August 29, 2023 04:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ALGO Any changes in OTX Algo Tasks implementation ENHANCE Enhancement of existing features TEST Any changes in tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants