Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Add onnx evaluation tool #279

Merged
merged 12 commits into from
Apr 29, 2021

Conversation

RunningLeon
Copy link
Collaborator

@RunningLeon RunningLeon commented Apr 26, 2021

This PR includes:

  • Add a tool to evaluate ONNX models with ONNXRuntime backend.
  • Align metrics of models between PyTorch and ONNXRuntime.
  • Update the document.

Results and Models

Model Config Dataset Metric PyTorch ONNX Runtime
ESRGAN esrgan_x4c64b23g32_g1_400k_div2k.py Set5 PSNR 28.2700 28.2619
SSIM 0.7778 0.7784
Set14 PSNR 24.6328 24.6290
SSIM 0.6491 0.6494
DIV2K PSNR 26.6531 26.6532
SSIM 0.7340 0.7340
ESRGAN esrgan_psnr_x4c64b23g32_g1_1000k_div2k.py Set5 PSNR 30.6428 30.6307
SSIM 0.8559 0.8565
Set14 PSNR 27.0543 27.0422
SSIM 0.7447 0.7450
DIV2K PSNR 29.3354 29.3354
SSIM 0.8263 0.8263
SRCNN srcnn_x4k915_g1_1000k_div2k.py Set5 PSNR 28.4316 28.4120
SSIM 0.8099 0.8106
Set14 PSNR 25.6486 25.6367
SSIM 0.7014 0.7015
DIV2K PSNR 27.7460 27.7460
SSIM 0.7854 0.78543

Notes:

  • All ONNX models are evaluated with dynamic shape on the datasets and images are preprocessed according to the original config file.
  • This tool is still experimental, and we only support restorer for now.

@RunningLeon RunningLeon requested a review from nbei April 26, 2021 09:23
@RunningLeon RunningLeon self-assigned this Apr 26, 2021
@RunningLeon RunningLeon removed the request for review from nbei April 26, 2021 09:26
@codecov
Copy link

codecov bot commented Apr 26, 2021

Codecov Report

Merging #279 (6b3cf88) into master (293b513) will increase coverage by 0.55%.
The diff coverage is 88.63%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #279      +/-   ##
==========================================
+ Coverage   79.99%   80.54%   +0.55%     
==========================================
  Files         160      162       +2     
  Lines        8003     8091      +88     
  Branches     1185     1192       +7     
==========================================
+ Hits         6402     6517     +115     
+ Misses       1456     1429      -27     
  Partials      145      145              
Flag Coverage Δ
unittests 80.54% <88.63%> (+0.55%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmedit/core/export/wrappers.py 88.37% <88.37%> (ø)
mmedit/core/export/__init__.py 100.00% <100.00%> (ø)
mmedit/models/mattors/dim.py 82.69% <0.00%> (+1.92%) ⬆️
mmedit/models/mattors/base_mattor.py 85.41% <0.00%> (+3.12%) ⬆️
.../models/backbones/encoder_decoders/encoders/vgg.py 98.33% <0.00%> (+5.00%) ⬆️
...ckbones/encoder_decoders/decoders/plain_decoder.py 100.00% <0.00%> (+33.70%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 293b513...6b3cf88. Read the comment docs.

@RunningLeon RunningLeon requested a review from innerlee April 26, 2021 10:23
@RunningLeon RunningLeon changed the title [WIP]: Add onnx evaluation tool [Feature]: Add onnx evaluation tool Apr 27, 2021
@RunningLeon RunningLeon requested a review from nbei April 28, 2021 08:19
Copy link
Collaborator

@Yshuo-Li Yshuo-Li left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine.

@innerlee
Copy link
Contributor

Please add some unit tests for functions and classes in mmedit/core/export/wrappers.py

@grimoire
Copy link
Member

@innerlee can we just ignore these functions and classes in Codecov since it is hard to test without a real ONNX model.

@innerlee
Copy link
Contributor

a real ONNX model.

A dummy model contains a single conv or fc is good enough.

@innerlee
Copy link
Contributor

If the onnx runtime is not available at CI, you can edit the config similar to this line https://github.com/open-mmlab/mmediting/blob/master/.github/workflows/build.yml#L54

@innerlee innerlee merged commit 4d2a23d into open-mmlab:master Apr 29, 2021
Yshuo-Li pushed a commit to Yshuo-Li/mmediting that referenced this pull request Jul 15, 2022
* update doc for onnx eval tool

* add ort eval tool

* add mattor test support

* update document

* update doc

* add format to html table

* add onnx wraper test

* fix lint

* better onnx unit test

* skip if torch==1.3.x

Co-authored-by: q.yao <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants