Add GLEAN #296

ckkelvinchan · 2021-05-09T05:24:53Z

This PR adds GLEAN for image super-resolution. It makes use of StyleGANv2 borrowed from MMGeneration.

Paper: GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution

add GenerateFrameIndicesForRecurrent

…o glean

… glean

… glean Conflicts: mmedit/models/backbones/__init__.py mmedit/models/backbones/sr_backbones/__init__.py

codecov · 2021-05-09T07:25:14Z

Codecov Report

Merging #296 (d1a05e1) into master (98a7ca3) will decrease coverage by 0.48%.
The diff coverage is 74.83%.

@@            Coverage Diff             @@
##           master     #296      +/-   ##
==========================================
- Coverage   80.76%   80.28%   -0.49%     
==========================================
  Files         171      177       +6     
  Lines        8526     9280     +754     
  Branches     1234     1352     +118     
==========================================
+ Hits         6886     7450     +564     
- Misses       1488     1641     +153     
- Partials      152      189      +37

Flag	Coverage Δ
unittests	`80.26% <74.70%> (-0.50%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmedit/models/losses/perceptual_loss.py	`56.75% <0.00%> (-1.58%)`	⬇️
mmedit/models/restorers/glean.py	`63.33% <63.33%> (ø)`
mmedit/models/components/stylegan2/modules.py	`64.40% <64.40%> (ø)`
...ls/components/stylegan2/generator_discriminator.py	`84.93% <84.93%> (ø)`
mmedit/models/components/stylegan2/common.py	`94.73% <94.73%> (ø)`
.../models/backbones/sr_backbones/glean_styleganv2.py	`97.00% <97.00%> (ø)`
mmedit/models/backbones/__init__.py	`100.00% <100.00%> (ø)`
mmedit/models/backbones/sr_backbones/__init__.py	`100.00% <100.00%> (ø)`
mmedit/models/components/__init__.py	`100.00% <100.00%> (ø)`
mmedit/models/components/stylegan2/__init__.py	`100.00% <100.00%> (ø)`
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 98a7ca3...d1a05e1. Read the comment docs.

ckkelvinchan · 2021-05-19T03:52:34Z

Hi @Yshuo-Li, many of the changes are added by @nbei. Here are the main changes related to GLEAN

mmedit/models/backbones/sr_backbones/glean_styleganv2.py
mmedit/models/restorers/glean.py
mmedit/models/losses/perceptual_loss.py

Yshuo-Li · 2021-05-21T06:38:24Z

mmedit/models/backbones/sr_backbones/glean_styleganv2.py

+        self.encoder = nn.ModuleList()
+        self.encoder.append(
+            nn.Sequential(
+                RRDBFeatureExtractor(3, 64, num_blocks=23),


Maybe adding mid_channels(defaults 64) , in_channels and out_channels to the class (GLEANStyleGANv2) will be better.

Yshuo-Li · 2021-05-21T07:07:41Z

mmedit/models/backbones/sr_backbones/glean_styleganv2.py

+                        nn.LeakyReLU(negative_slope=0.2, inplace=True),
+                        nn.Conv2d(64, 3, 3, 1, 1)))
+
+    def forward(self, lr):


Named LQ in other models of MMEditing

Yshuo-Li · 2021-05-21T07:15:10Z

mmedit/models/components/stylegan2/common.py

+
+
+@torch.no_grad()
+def get_mean_latent(generator, num_samples=4096, bs_per_repeat=1024):


Need docstring

@nbei Could you help with that? I have copied the docstring of get_mean_latent from mmgeneration, but there is no docstring for style_mixing in mmgeneration.

Yshuo-Li · 2021-05-21T07:22:12Z

tests/test_glean.py

+        with pytest.raises(AssertionError):  # test with metric but gt is None
+            with torch.no_grad():
+                data_batch.pop('gt')
+                restorer(**data_batch, test_mode=True)


Need unittests of the assertion errors

The model for GLEAN has only this assertion error. Do you mean the backbone? The unittests are in test_glean_net.py.

Such as
mmedit/models/components/stylegan2/common.py#L16
mmedit/models/backbones/sr_backbones/glean_styleganv2.py#L271

For the unittests for glean_styleganv2.py, it is in test_glean_net.py. I will add unittests for common.py.

ckkelvinchan · 2021-05-22T06:55:42Z

Will add docstring for style_mixing.

innerlee · 2021-05-27T01:52:40Z

mmedit/models/backbones/sr_backbones/glean_styleganv2.py

+        channels = self.generator.channels
+
+        # encoder
+        num_styles = int(math.log2(out_size)) * 2 - 2


For consistency, we use np for math ops

math.log2 -> np.log2

innerlee · 2021-05-27T02:06:29Z

Some notes:

Replace all math.xx to np.xx
For the style-gan code, we will refactor it to use mmgen once the infrastructure is ready. Currently we simply do the copy-paste

And please rebase master.

* clone from MMEditing * add GenerateFrameIndicesForRecurrent * add unit test * update format * add stylegan2 with pretrained models * add GLEAN architecture (WIP) * GLEAN architecture * Use build_component to build StyleGANv2 * add GenerateFrameIndicesForRecurrent * add stylegan2 with pretrained models * minor revision of architecture * Minor fix * Add MSELoss for perceptual loss * merge master * merge master * sort * Add GLEAN model * Add unittest * Remove pretrained in test * Replace by test_srgan for verification * Change disc to ModifiedVGG * Remove init_weights * Remove _load_pretrained_model * revert to test_srgan * Change Discriminator * Use original StyleGAN2 discriminator * Use GLEAN as generator * Change to GLEAN model * minor change * remove redundancy in test_glean.py * install * Add unittests * Revert to original StyleGAN2 discriminator * Change math.log2 to np.log2 Co-authored-by: Rui Xu <[email protected]>

ckkelvinchan and others added 25 commits March 26, 2021 10:13

clone from MMEditing

c0dc3d3

add GenerateFrameIndicesForRecurrent

5e107e0

add unit test

ae02181

update format

8cc6c8b

Merge pull request #2 from ckkelvinchan/augmentations

c9781e8

add GenerateFrameIndicesForRecurrent

add stylegan2 with pretrained models

7d3c668

Add StyleGAN2 components

d6094e4

add GLEAN architecture (WIP)

0693e55

GLEAN architecture

d2bf8d3

Use build_component to build StyleGANv2

b5b2efd

add GenerateFrameIndicesForRecurrent

ca39f10

add stylegan2 with pretrained models

a7cb67b

Merge branch 'glean' of https://github.com/ckkelvinchan/mmediting int…

34b1d8f

…o glean

Merge branch 'master' of https://github.com/open-mmlab/mmediting into…

cb735af

… glean

minor revision of architecture

feb7f7c

Minor fix

f9bf419

Add MSELoss for perceptual loss

112e140

merge master

1442b39

merge master

a1fd615

Merge branch 'master' of https://github.com/open-mmlab/mmediting into…

8815e7f

… glean Conflicts: mmedit/models/backbones/__init__.py mmedit/models/backbones/sr_backbones/__init__.py

sort

f257964

Add GLEAN model

ffa651b

Add unittest

326fd3b

Remove pretrained in test

37d0fe9

Replace by test_srgan for verification

a35bbf9

ckkelvinchan added the need test label May 9, 2021

ckkelvinchan added 4 commits May 9, 2021 14:28

Change disc to ModifiedVGG

96ffe92

Remove init_weights

06fb8ce

Remove _load_pretrained_model

e53d097

revert to test_srgan

cb4c4af

Change Discriminator

dbb0212

ckkelvinchan added 4 commits May 9, 2021 15:41

Use original StyleGAN2 discriminator

0de7d9a

Use GLEAN as generator

830ec65

Change to GLEAN model

2ac5e59

minor change

e279361

ckkelvinchan removed the need test label May 9, 2021

remove redundancy in test_glean.py

604c102

innerlee requested a review from Yshuo-Li May 11, 2021 11:51

innerlee mentioned this pull request May 13, 2021

Iteration Plan May 2021 #302

Closed

7 tasks

Yshuo-Li reviewed May 21, 2021

View reviewed changes

install

6c09470

ckkelvinchan added the status/WIP work in progress normally label May 22, 2021

Add unittests

854e14a

Yshuo-Li approved these changes May 24, 2021

View reviewed changes

ckkelvinchan removed the status/WIP work in progress normally label May 25, 2021

innerlee reviewed May 27, 2021

View reviewed changes

ckkelvinchan added 3 commits May 27, 2021 11:34

Revert to original StyleGAN2 discriminator

b36c2bd

Change math.log2 to np.log2

054cf83

rebase master

d1a05e1

innerlee merged commit f0b2ab4 into open-mmlab:master May 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GLEAN #296

Add GLEAN #296

ckkelvinchan commented May 9, 2021

codecov bot commented May 9, 2021 •

edited

Loading

ckkelvinchan commented May 19, 2021 •

edited

Loading

Yshuo-Li May 21, 2021

Yshuo-Li May 21, 2021

Yshuo-Li May 21, 2021

ckkelvinchan May 21, 2021

Yshuo-Li May 21, 2021

ckkelvinchan May 21, 2021 •

edited

Loading

Yshuo-Li May 21, 2021

ckkelvinchan May 22, 2021

ckkelvinchan commented May 22, 2021

innerlee May 27, 2021

innerlee commented May 27, 2021 •

edited

Loading



		@torch.no_grad()
		def get_mean_latent(generator, num_samples=4096, bs_per_repeat=1024):

Add GLEAN #296

Add GLEAN #296

Conversation

ckkelvinchan commented May 9, 2021

codecov bot commented May 9, 2021 • edited Loading

Codecov Report

ckkelvinchan commented May 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ckkelvinchan May 21, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ckkelvinchan commented May 22, 2021

Choose a reason for hiding this comment

innerlee commented May 27, 2021 • edited Loading

codecov bot commented May 9, 2021 •

edited

Loading

ckkelvinchan commented May 19, 2021 •

edited

Loading

ckkelvinchan May 21, 2021 •

edited

Loading

innerlee commented May 27, 2021 •

edited

Loading