[Feature] Add SOD datasets #913

acdart · 2021-09-27T09:56:50Z

Support DUTS dataset, which is the most popular dataset in salient object detection.

DUTS contains 10,553 training images and 5,019 test images. All training images are collected from the ImageNet DET training/val sets, while test images are collected from the ImageNet DET test set and the SUN data set. Both the training and test set contain very challenging scenarios for saliency detection. Accurate pixel-level ground truths are manually annotated by 50 subjects.

DUTS is currently the largest saliency detection benchmark with the explicit training/test evaluation protocol. For fair comparison in the future research, the training set of DUTS serves as a good candidate for learning DNNs, while the test set and other public data sets can be used for evaluation.

Related links:
https://paperswithcode.com/sota/salient-object-detection-on-duts-te
https://github.com/ArcherFMY/sal_eval_toolbox

MengzhangLI · 2021-09-27T10:07:13Z

Hi, thanks for your nice pr, we would review it as soon as possible.

Salient object detection is an important branch in semantic segmentation, could you also join us in supporting other representative datasets, such as MSRA, THUR15K, ECSSD and so on?

Best,

codecov · 2021-09-27T10:07:29Z

Codecov Report

Merging #913 (5581198) into master (36eb2d8) will decrease coverage by 0.05%.
The diff coverage is 81.81%.

❗ Current head 5581198 differs from pull request most recent head b6526af. Consider uploading reports for the commit b6526af to get more accurate results

@@            Coverage Diff             @@
##           master     #913      +/-   ##
==========================================
- Coverage   89.62%   89.56%   -0.06%     
==========================================
  Files         113      117       +4     
  Lines        6263     6307      +44     
  Branches      989      993       +4     
==========================================
+ Hits         5613     5649      +36     
- Misses        452      460       +8     
  Partials      198      198

Flag	Coverage Δ
unittests	`89.56% <81.81%> (-0.06%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmseg/datasets/dut_omron.py	`80.00% <80.00%> (ø)`
mmseg/datasets/duts.py	`80.00% <80.00%> (ø)`
mmseg/datasets/ecssd.py	`80.00% <80.00%> (ø)`
mmseg/datasets/hku_is.py	`80.00% <80.00%> (ø)`
mmseg/datasets/__init__.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 36eb2d8...b6526af. Read the comment docs.

Junjun2016 · 2021-09-28T07:09:13Z

Paper: PiCANet
Code: link1, link2.

Junjun2016 · 2021-09-28T09:50:46Z

Paper: U^2Net.
Code: link1.

Junjun2016 · 2021-09-28T09:56:48Z

U^2Net results:

Junjun2016 · 2021-09-28T10:02:43Z

Paper: BASNet.
Code: link1.

Junjun2016 · 2021-09-28T10:29:41Z

Paper: Cascaded Partial Decoder for Fast and Accurate Salient Object Detection (CPD CVPR'19).
Code: link1.

Junjun2016 · 2021-09-28T10:38:41Z

Paper: Stacked Cross Refinement Network for Edge-Aware Salient Object Detection (SCRN ICCV'19).
Code: link1.

acdart · 2021-09-28T17:58:49Z

abot metrics in SOD

in SOD, metrics are always computed like 'mean(calc_metric(pred[i], gt[i]))', it's different with MMSegmentation
need to add support for some popular metrics: MAE, max-Fmeasure, mean-Fmeasure, Smeasure, Emeasure
score-map output is needed by some metrics like max-Fmeasure

Junjun2016 · 2021-09-29T05:25:30Z

abot metrics in SOD

in SOD, metrics are always computed like 'mean(calc_metric(pred[i], gt[i]))', it's different with MMSegmentation

need to add support for some popular metrics: MAE, max-Fmeasure, mean-Fmeasure, Smeasure, Emeasure

score-map output is needed by some metrics like max-Fmeasure

add a return_logit argument to the forward of segmentor the same as return_loss , pass the argument by kwargs, so we only should modify simple_test, aug_test, and eval_hooks.
modify the calculation logic of pre_eval_to_metrics to support sample average metrics.
integrate this PR to support reducing background class or not for BCE, especially for one channel case.

integrate from #295

Junjun2016 · 2021-10-18T02:54:25Z

Hi @shinya7y
Could you please help us review this PR together?

shinya7y · 2021-10-18T17:06:38Z

Sure. I will only check superficial issues because I'm not familiar with SOD and mmseg.

shinya7y · 2021-10-18T17:18:42Z

docs/dataset_prepare.md

+
+In salient object detection (SOD), HKU-IS is used for evaluation.
+
+First，download [HKU-IS.rar](https://sites.google.com/site/ligb86/mdfsaliency/).


I cannot access https://sites.google.com/site/ligb86/mdfsaliency/
Is it the same as https://i.cs.hku.hk/~gbli/deep_saliency.html ?

shinya7y · 2021-10-18T17:22:08Z

docs/dataset_prepare.md

+
+### DUTS
+
+First，download [DUTS-TR.zip](http://saliencydetection.net/duts/download/DUTS-TR.zip) and [DUTS-TE.zip](http://saliencydetection.net/duts/download/DUTS-TE.zip) .


Three ) . in this file to ).

shinya7y · 2021-10-18T17:28:16Z

mmseg/apis/test.py

@@ -99,7 +101,8 @@ def single_gpu_test(model,
        if pre_eval:
            # TODO: adapt samples_per_gpu > 1.
            # only samples_per_gpu=1 valid now
-            result = dataset.pre_eval(result, indices=batch_indices)
+            result = dataset.pre_eval(
+                result, return_logit, indices=batch_indices)


return_logit=return_logit

shinya7y · 2021-10-18T17:28:50Z

mmseg/apis/test.py

@@ -215,7 +223,8 @@ def multi_gpu_test(model,
        if pre_eval:
            # TODO: adapt samples_per_gpu > 1.
            # only samples_per_gpu=1 valid now
-            result = dataset.pre_eval(result, indices=batch_indices)
+            result = dataset.pre_eval(
+                result, return_logit, indices=batch_indices)


return_logit=return_logit

shinya7y · 2021-10-18T17:29:52Z

mmseg/models/segmentors/encoder_decoder.py

@@ -216,7 +216,7 @@ def whole_inference(self, img, img_meta, rescale):

        return seg_logit

-    def inference(self, img, img_meta, rescale):
+    def inference(self, img, img_meta, rescale, return_logit):


def inference(self, img, img_meta, rescale, return_logit=False):

Docstring for new argument.

shinya7y · 2021-10-18T17:33:52Z

mmseg/models/segmentors/encoder_decoder.py

        """Simple test with single image."""
-        seg_logit = self.inference(img, img_meta, rescale)
-        seg_pred = seg_logit.argmax(dim=1)
+        seg_logit = self.inference(img, img_meta, rescale, return_logit)


return_logit=return_logit in the three calls of self.inference

shinya7y · 2021-10-18T17:39:25Z

mmseg/core/evaluation/__init__.py


 __all__ = [
    'EvalHook', 'DistEvalHook', 'mean_dice', 'mean_iou', 'mean_fscore',
    'eval_metrics', 'get_classes', 'get_palette', 'pre_eval_to_metrics',
-    'intersect_and_union'
+    'intersect_and_union', 'calc_sod_metrics', 'eval_sod_metrics', 'pre_eval_to_sod_metrics'


Code style is different in many files.
Please refer to https://github.com/open-mmlab/mmsegmentation/blob/master/.github/CONTRIBUTING.md

shinya7y · 2021-10-18T17:43:45Z

tools/convert_datasets/dut_omron.py

+    print('Making directories...')
+    mmcv.mkdir_or_exist(out_dir)
+    mmcv.mkdir_or_exist(osp.join(out_dir, 'images'))
+    mmcv.mkdir_or_exist(osp.join(out_dir, 'images', 'validation'))


Many mkdir_or_exist calls in dataset converters seem redundant.

shinya7y · 2021-10-18T17:47:14Z

mmseg/core/evaluation/metrics.py

+    if isinstance(pred_label, str):
+        pred_label = torch.from_numpy(np.load(pred_label))
+    else:
+        pred_label = torch.from_numpy((pred_label))


two parentheses needed?

shinya7y · 2021-10-18T17:53:03Z

tools/convert_datasets/dut_omron.py

+
+        assert len(os.listdir(image_dir)) == DUT_OMRON_LEN \
+               and len(os.listdir(mask_dir)) == \
+               DUT_OMRON_LEN, 'len(DUT-OMRON) != {}'.format(DUT_OMRON_LEN)


assert len(os.listdir(image_dir)) == DUT_OMRON_LEN \ and len(os.listdir(mask_dir)) == DUT_OMRON_LEN, \ f'len(DUT-OMRON) != {DUT_OMRON_LEN}'

Similar modifications of the other converters will improve readability.

jshilong · 2021-10-21T14:37:50Z

mmseg/core/evaluation/eval_hooks.py

@@ -30,9 +30,11 @@ def __init__(self,
                 by_epoch=False,
                 efficient_test=False,
                 pre_eval=False,
+                 return_logit=False,


add docstr for this argument

jshilong · 2021-10-21T14:41:56Z

tools/convert_datasets/hku_is.py

@@ -0,0 +1,67 @@
+# Copyright (c) OpenMMLab. All rights reserved.


Please add comment for this script and give an example， you can refer to
https://github.com/open-mmlab/mmdetection/blob/9874180a015beee874c904357fe4ed4fab4b46a4/tools/analysis_tools/optimize_anchors.py#L2

Do this for other scripts too

Junjun2016 · 2021-10-24T18:57:53Z

Hi @acdart
The CI has been fixed, please merge the master branch.

Junjun2016 · 2021-11-02T22:10:53Z

docs/dataset_prepare.md

+
+```shell
+python tools/convert_datasets/hku_is.py /path/to/HKU-IS.rar
+```


Could modify the Chinese document accordingly.

Junjun2016 · 2021-11-02T22:33:28Z

configs/_base_/datasets/duts.py

+        img_dir='images/training',
+        ann_dir='annotations/training',
+        pipeline=train_pipeline),
+    val=dict(


Can we use concat dataset (#833) and do evaluation seperately.

Junjun2016 · 2021-11-02T22:34:39Z

mmseg/apis/test.py

@@ -38,6 +38,7 @@ def single_gpu_test(model,
                    efficient_test=False,
                    opacity=0.5,
                    pre_eval=False,
+                    return_logit=False,


docstring for this new argument.

Junjun2016 · 2021-11-02T22:39:44Z

mmseg/models/segmentors/encoder_decoder.py

+        if return_logit:
+            output = seg_logit
+        else:
+            if seg_logit.shape[1] >= 2:


It would be better to add some comments for different cases (different shape).

RockeyCoss · 2021-11-15T08:22:38Z

Please merge the master branch into your branch, thank you.

MengzhangLI · 2021-11-24T13:15:47Z

Hi, @acdart . Sorry to bother you.

Do you plan to continue finishing this pr? We can spare more time or human resources in the next month if you need it.

Looking forward to your reply.

Best,

Junjun2016 · 2021-12-23T02:15:59Z

Add class name and palette to mmseg/core/evaluation/class_names.py.

…n-mmlab#913)

* polish localizer code * add deprecated warnings * update warning msg

zhangyajie and others added 2 commits September 27, 2021 17:44

[Feature] init duts dataset

bbfdd9b

Merge branch 'open-mmlab:master' into sod_datasets

7a0bc0b

Junjun2016 added the WIP Work in process label Sep 27, 2021

zhangyajie added 2 commits September 27, 2021 20:32

add DUT-OMRON dataset

836766e

add ECSSD dataset

13e3db9

acdart changed the title ~~[Feature] Add DUTS dataset~~ [Feature] Add SOD datasets Sep 27, 2021

add HKU-IS dataset

5581198

zhangyajie added 5 commits October 4, 2021 16:46

[Fix] Fix expand onehot bug in BCE

ebf8254

integrate from #295

init return_logit

4cc8aeb

[Fix] Fix single channel output

1994c78

[Refactor] Refactor SOD dataset and eval pipeline

a7ae896

[Fix] fix

b6526af

shinya7y suggested changes Oct 18, 2021

View reviewed changes

jshilong reviewed Oct 21, 2021

View reviewed changes

Junjun2016 reviewed Nov 2, 2021

View reviewed changes

MengzhangLI mentioned this pull request Nov 11, 2021

Invitation of merging CTDNet into MMSegmentation iCVTEAM/CTDNet#1

Closed

wjkim81 pushed a commit to wjkim81/mmsegmentation that referenced this pull request Dec 3, 2023

Update the link of interhand annotation. Fix a bug in LoadImage. (ope…

d087385

…n-mmlab#913)

sibozhang pushed a commit to sibozhang/mmsegmentation that referenced this pull request Mar 22, 2024

[Improvement] Polish localizer related code (open-mmlab#913)

95486d6

* polish localizer code * add deprecated warnings * update warning msg

acdart closed this by deleting the head repository May 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Add SOD datasets #913

[Feature] Add SOD datasets #913

acdart commented Sep 27, 2021 •

edited by Junjun2016

Loading

MengzhangLI commented Sep 27, 2021

codecov bot commented Sep 27, 2021 •

edited

Loading

Junjun2016 commented Sep 28, 2021 •

edited

Loading

Junjun2016 commented Sep 28, 2021

Junjun2016 commented Sep 28, 2021

Junjun2016 commented Sep 28, 2021 •

edited

Loading

Junjun2016 commented Sep 28, 2021

Junjun2016 commented Sep 28, 2021

acdart commented Sep 28, 2021

Junjun2016 commented Sep 29, 2021

Junjun2016 commented Oct 18, 2021

shinya7y commented Oct 18, 2021

shinya7y Oct 18, 2021

shinya7y Oct 18, 2021

shinya7y Oct 18, 2021

shinya7y Oct 18, 2021

shinya7y Oct 18, 2021

Junjun2016 Nov 2, 2021

shinya7y Oct 18, 2021

shinya7y Oct 18, 2021

shinya7y Oct 18, 2021

shinya7y Oct 18, 2021

shinya7y Oct 18, 2021

jshilong Oct 21, 2021

jshilong Oct 21, 2021

jshilong Oct 21, 2021

Junjun2016 commented Oct 24, 2021

Junjun2016 Nov 2, 2021

Junjun2016 Nov 2, 2021

Junjun2016 Nov 2, 2021

Junjun2016 Nov 2, 2021 •

edited

Loading

RockeyCoss commented Nov 15, 2021

MengzhangLI commented Nov 24, 2021

Junjun2016 commented Dec 23, 2021


		In salient object detection (SOD), HKU-IS is used for evaluation.

		First，download [HKU-IS.rar](https://sites.google.com/site/ligb86/mdfsaliency/).


		### DUTS

		First，download [DUTS-TR.zip](http://saliencydetection.net/duts/download/DUTS-TR.zip) and [DUTS-TE.zip](http://saliencydetection.net/duts/download/DUTS-TE.zip) .

		@@ -0,0 +1,67 @@
		# Copyright (c) OpenMMLab. All rights reserved.

[Feature] Add SOD datasets #913

[Feature] Add SOD datasets #913

Conversation

acdart commented Sep 27, 2021 • edited by Junjun2016 Loading

MengzhangLI commented Sep 27, 2021

codecov bot commented Sep 27, 2021 • edited Loading

Codecov Report

Junjun2016 commented Sep 28, 2021 • edited Loading

Junjun2016 commented Sep 28, 2021

Junjun2016 commented Sep 28, 2021

Junjun2016 commented Sep 28, 2021 • edited Loading

Junjun2016 commented Sep 28, 2021

Junjun2016 commented Sep 28, 2021

acdart commented Sep 28, 2021

Junjun2016 commented Sep 29, 2021

Junjun2016 commented Oct 18, 2021

shinya7y commented Oct 18, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Junjun2016 commented Oct 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Junjun2016 Nov 2, 2021 • edited Loading

Choose a reason for hiding this comment

RockeyCoss commented Nov 15, 2021

MengzhangLI commented Nov 24, 2021

Junjun2016 commented Dec 23, 2021

acdart commented Sep 27, 2021 •

edited by Junjun2016

Loading

codecov bot commented Sep 27, 2021 •

edited

Loading

Junjun2016 commented Sep 28, 2021 •

edited

Loading

Junjun2016 commented Sep 28, 2021 •

edited

Loading

Junjun2016 Nov 2, 2021 •

edited

Loading