SDNet

This repository is about the work "Transcending Pixels: Boosting Saliency Detection via Scene Understanding From Aerial Imagery" in IEEE TGRS 2023.

Abstract

Existing remote sensing image salient object detection (RSI-SOD) methods widely perform object-level semantic understanding with pixel-level supervision, but ignore the image-level scene information. As a fundamental attribute of RSIs, the scene has a complex intrinsic correlation with salient objects, which may bring hints to improve saliency detection performance. However, existing RSI-SOD datasets lack both pixel- and image-level labels, and it is non-trivial to effectively transfer the scene domain knowledge for more accurate saliency localization. To address these challenges, we first annotate the image-level scene labels of three RSI-SOD datasets inspired by remote sensing scene classification. On top of it, we present a novel scene-guided dual-stream network (SDNet), which can perform cross-task knowledge distillation from the scene classification to facilitate accurate saliency detection. Specifically, a scene knowledge transfer module (SKTM) and a conditional dynamic guidance module (CDGM) are designed for extracting saliency key area as spatial attention from the scene subnet and guiding the saliency subnet to generate scene-enhanced saliency features, respectively. Finally, an object contour awareness module (OCAM) is introduced to enable the model to focus more on irregular spatial details of salient objects from the complicated background. Extensive experiments reveal that our SDNet outperforms over 20 state-of-the-art algorithms on three datasets. Moreover, we show that the proposed framework is model-agnostic, and its extension to six baselines can bring significant performance benefits.

Methodology

1. Define 12 types of scene categories and manually annotate scene labels for three RSI-SOD datasets

Scene Category	ORSSD Dataset	EORSSD Dataset	ORSI-4199 Dataset
Airplane Facilities	136	427	504
Industrial Facilities	49	57	441
Bridges	14	37	429
Ships	145	449	418
Rural Buildings	48	96	634
Transportation Facilities	30	71	317
Highways	49	229	571
Rivers	81	105	70
Lakes	104	192	303
Islands	46	148	5
Sports Facilities	77	116	354
Others	21	73	153
Total	800	2000	4199

2. Propose a multitask learning-based (MTL) scene-guided dual-branch network (SDNet)

3. Demonstrate the scene learning is effective

4. Reveal the SDNet is model-agnostic

How to use

1. Install newest versions of torch and torchdata

thop                      0.0.31
tqdm                      4.59.0
numpy                     1.20.2
timm                      0.4.12
tokenizers                0.12.1
torch                     1.8.1
torchvision               0.9.1

2. Download weights files from Google Drive

https://drive.google.com/drive/folders/1zygarM13gu48gJQ1jPXCAUZEsuDB2Y0l

3. Run getsmaps.py to generate the saliency maps

python getsmaps.py

4. Run compute_metrics.py to calculate the qualititive results

python compute_metrics.py

Citation (If you think this repository could help you, please cite:)

@ARTICLE{SDNet2023,

  author={Liu, Yanfeng and Xiong, Zhitong and Yuan, Yuan and Wang, Qi},
  
  journal={IEEE Transactions on Geoscience and Remote Sensing}, 
  
  title={Transcending Pixels: Boosting Saliency Detection via Scene Understanding From Aerial Imagery}, 
  
  year={2023},
  
  volume={61},
  
  number={},
  
  pages={1-16},

  doi={10.1109/TGRS.2023.3298661}

  }

@ARTICLE{SRAL2023,

  author={Liu, Yanfeng and Xiong, Zhitong and Yuan, Yuan and Wang, Qi},
  
  journal={IEEE Transactions on Geoscience and Remote Sensing}, 
  
  title={Distilling Knowledge From Super-Resolution for Efficient Remote Sensing Salient Object Detection}, 
  
  year={2023},
  
  volume={61},
  
  number={},
  
  pages={1-16},
  
  doi={10.1109/TGRS.2023.3267271}
  
  }

@ARTICLE{UG2L2023,

  author={Liu, Yanfeng and Yuan, Yuan and Wang, Qi},

  journal={IEEE Geoscience and Remote Sensening Letters},

  title={Uncertainty-Aware Graph Reasoning with Global Collaborative Learning for Remote Sensing Salient Object Detection},

  year={2023},

  volume={20},

  number={},

  pages={1-5},

  doi={10.1109/LGRS.2023.3299245}

}

@InProceedings{RSSOD2023,

  author = {Xiong, Zhitong and Liu, Yanfeng and Wang, Qi and Zhu, Xiao Xiang},

  title = {RSSOD-Bench: A Large-Scale Benchmark Dataset for Salient Object Detection in Optical Remote Sensing Imagery},

  booktitle = {Proc. IEEE International Geoscience and Remote Sensing Symposium (IGARSS)},

  pages={6549-6552},

  year = {2023}

}

@ARTICLE{HFANet2022,

  author={Wang, Qi and Liu, Yanfeng and Xiong, Zhitong and Yuan, Yuan},

  journal={IEEE Transactions on Geoscience and Remote Sensing},

  title={Hybrid Feature Aligned Network for Salient Object Detection in Optical Remote Sensing Imagery},

  year={2022},

  volume={60},

  number={},

  pages={1-15},

  doi={10.1109/TGRS.2022.3181062}

}

Acknowledgment and our other works

1. https://github.com/EarthNets/Dataset4EO
2. https://github.com/lyf0801/UG2L
3. https://github.com/lyf0801/HFANet
4. https://github.com/lyf0801/SRAL
5. https://github.com/rmcong/DAFNet_TIP20
6. https://github.com/rmcong/EORSSD-dataset
7. https://github.com/rmcong/ORSSD-dataset