NUS AY20/21 EE4002D Final Year Project: Real-time Detection of Interest Regions in Multiple Weather and Lighting Conditions

Abstract

This is a final year project done by four final year student in NUS: Ai Zhengwei, Guo Haoren, Huang Lixing and Zhao Zhijie.

The project began in August 2020, start of the Semester one of Year 4.

This project aims for developing a high performance and light-weight model with lane detection and traffic sign detection features.

The combined model was built on AdelaiDet which is on top of Detectron2 and modified Ultra-fast architecture to be fitted into FCOS model.

Acknowledgements

We would like to thank the following people who have helped and guided us through the course of my final year project.

First of all, we want to express our deepest gratitude to our final year project supervisor Assistant Professor Feng Jiashi for his patient listening, supportive mentoring and inspiring guidance. Prof. Feng has kindly arranged weekly meetings for us to update the project progress, during which he always provided timely feedback and pointed out new research directions. We cannot finish this project without his support.

We want to thank Dr. Liew Jun Hao, who has given us a lot of suggestions as well as encouragement whenever we had doubts or difficulties in our project. We also want to thank our final year project examiner Associate Professor Robby Tan, who has listened to our CA1 presentation and given advice on how we could improve.

It was our great honour to join the big family of Learning & Vision lab led by Prof. Feng, where all the lab-mates are extremely helpful, and they have given me lots of support.

Contributions

The main contribution of this project can be summarized as follows:

• A combined model that can perform traffic sign detection and lane detection at the same time with good accuracy and efficiency.

• We show how to train a unified model on two disjoint datasets (i.e. lane detection dataset without traffic sign annotations and vice versa).

• CoordConv which incorporates positional information to optimize the based model.

• Apply FCOS model to traffic sign detection branch to improve the efficiency of traffic sign detection but keep similar average precision.

• Ultra-Fast model to lane detection branch and apply Cycle Gan to enhance the performance for night image detection.

• CycleGAN to synthesize realistic night-time images to augment the existing dataset, encouraging the model to perform better even on nighttime.

Combined Model Architecture

Results and Model Comparison

Traffic Sign Detection Results Comparison

Model	AP	AP50	FPS
FCOS R50	62.414	82.264	25
FCOS R50 RT	75.465	89.754	10
Combined Model	66.732	86.063	21.27
Combined Model(Added Coord Conv)	65.505	84.297	20

Lane Detection Results Comparison

	Original Ultra-Fast Model	New Combined Model	New Combined Model (no padding loss)
Normal	0.8862	0.8821	0.9
Crowd	0.7079	0.7866	0.8039
Night	0.6664	0.6304	0.6289
Noline	0.4274	0.5744	0.5873
Shadow	0.5987	0.7967	0.7434
Arrow	0.7948	0.8589	0.8683
Hilight	0.6091	0.7356	0.7334
Curve	0.6697	0.6483	0.6591
Total	0.7135	0.7529	0.7644

Demo

Demo Images

The following are the results we got from the combined model:

Demo Video

• Demo Video 1
• Demo Video 2

AdelaiDet

AdelaiDet is an open source toolbox for multiple instance-level recognition tasks on top of Detectron2. All instance-level recognition works from our group are open-sourced here.

To date, AdelaiDet implements the following algorithms:

• FCOS
• BlendMask
• MEInst
• ABCNet
• CondInst
• SOLO (mmdet version)
• SOLOv2
• BoxInst to be released (video demo)
• DirectPose to be released

Models

COCO Object Detecton Baselines with FCOS

Name	inf. time	box AP	download
FCOS_R_50_1x	16 FPS	38.7	model
FCOS_MS_R_101_2x	12 FPS	43.1	model
FCOS_MS_X_101_32x8d_2x	6.6 FPS	43.9	model
FCOS_MS_X_101_32x8d_dcnv2_2x	4.6 FPS	46.6	model
FCOS_RT_MS_DLA_34_4x_shtw	52 FPS	39.1	model

More models can be found in FCOS README.md.

COCO Instance Segmentation Baselines with BlendMask

Model	Name	inf. time	box AP	mask AP	download
Mask R-CNN	R_101_3x	10 FPS	42.9	38.6
BlendMask	R_101_3x	11 FPS	44.8	39.5	model
BlendMask	R_101_dcni3_5x	10 FPS	46.8	41.1	model

For more models and information, please refer to BlendMask README.md.

COCO Instance Segmentation Baselines with MEInst

Name	inf. time	box AP	mask AP	download
MEInst_R_50_3x	12 FPS	43.6	34.5	model

For more models and information, please refer to MEInst README.md.

Total_Text results with ABCNet

Name	inf. time	e2e-hmean	det-hmean	download
attn_R_50	11 FPS	67.1	86.0	model

For more models and information, please refer to ABCNet README.md.

COCO Instance Segmentation Baselines with CondInst

Name	inf. time	box AP	mask AP	download
CondInst_MS_R_50_1x	14 FPS	39.7	35.7	model
CondInst_MS_R_50_BiFPN_3x_sem	13 FPS	44.7	39.4	model
CondInst_MS_R_101_3x	11 FPS	43.3	38.6	model
CondInst_MS_R_101_BiFPN_3x_sem	10 FPS	45.7	40.2	model

For more models and information, please refer to CondInst README.md.

Note that:

• Inference time for all projects is measured on a NVIDIA 1080Ti with batch size 1.
• APs are evaluated on COCO2017 val split unless specified.

Installation

First install Detectron2 following the official guide: INSTALL.md.

Please use Detectron2 with commit id 9eb4831 if you have any issues related to Detectron2.

Then build AdelaiDet with:

git clone https://github.com/aim-uofa/AdelaiDet.git
cd AdelaiDet
python setup.py build develop

If you are using docker, a pre-built image can be pulled with:

docker pull tianzhi0549/adet:latest

Some projects may require special setup, please follow their own README.md in configs.

Quick Start

Inference with Pre-trained Models

1. Pick a model and its config file, for example, fcos_R_50_1x.yaml.
2. Download the model wget https://cloudstor.aarnet.edu.au/plus/s/glqFc13cCoEyHYy/download -O fcos_R_50_1x.pth
3. Run the demo with

python demo/demo.py \
    --config-file configs/FCOS-Detection/R_50_1x.yaml \
    --input input1.jpg input2.jpg \
    --opts MODEL.WEIGHTS fcos_R_50_1x.pth

Train Your Own Models

To train a model with "train_net.py", first setup the corresponding datasets following datasets/README.md, then run:

OMP_NUM_THREADS=1 python tools/train_net.py \
    --config-file configs/FCOS-Detection/R_50_1x.yaml \
    --num-gpus 8 \
    OUTPUT_DIR training_dir/fcos_R_50_1x

To evaluate the model after training, run:

OMP_NUM_THREADS=1 python tools/train_net.py \
    --config-file configs/FCOS-Detection/R_50_1x.yaml \
    --eval-only \
    --num-gpus 8 \
    OUTPUT_DIR training_dir/fcos_R_50_1x \
    MODEL.WEIGHTS training_dir/fcos_R_50_1x/model_final.pth

Note that:

• The configs are made for 8-GPU training. To train on another number of GPUs, change the --num-gpus.
• If you want to measure the inference time, please change --num-gpus to 1.
• We set OMP_NUM_THREADS=1 by default, which achieves the best speed on our machines, please change it as needed.
• This quick start is made for FCOS. If you are using other projects, please check the projects' own README.md in configs.

Acknowledgements

The authors are grateful to Nvidia, Huawei Noah's Ark Lab, ByteDance, Adobe who generously donated GPU computing in the past a few years.

Citing AdelaiDet

If you use this toolbox in your research or wish to refer to the baseline results published here, please use the following BibTeX entries:

@misc{tian2019adelaidet,
  author =       {Tian, Zhi and Chen, Hao and Wang, Xinlong and Liu, Yuliang and Shen, Chunhua},
  title =        {{AdelaiDet}: A Toolbox for Instance-level Recognition Tasks},
  howpublished = {\url{https://git.io/adelaidet}},
  year =         {2019}
}

@inproceedings{tian2019fcos,
  title     =  {{FCOS}: Fully Convolutional One-Stage Object Detection},
  author    =  {Tian, Zhi and Shen, Chunhua and Chen, Hao and He, Tong},
  booktitle =  {Proc. Int. Conf. Computer Vision (ICCV)},
  year      =  {2019}
}

@inproceedings{chen2020blendmask,
  title     =  {{BlendMask}: Top-Down Meets Bottom-Up for Instance Segmentation},
  author    =  {Chen, Hao and Sun, Kunyang and Tian, Zhi and Shen, Chunhua and Huang, Yongming and Yan, Youliang},
  booktitle =  {Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)},
  year      =  {2020}
}

@inproceedings{zhang2020MEInst,
  title     =  {Mask Encoding for Single Shot Instance Segmentation},
  author    =  {Zhang, Rufeng and Tian, Zhi and Shen, Chunhua and You, Mingyu and Yan, Youliang},
  booktitle =  {Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)},
  year      =  {2020}
}

@inproceedings{liu2020abcnet,
  title     =  {{ABCNet}: Real-time Scene Text Spotting with Adaptive {B}ezier-Curve Network},
  author    =  {Liu, Yuliang and Chen, Hao and Shen, Chunhua and He, Tong and Jin, Lianwen and Wang, Liangwei},
  booktitle =  {Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)},
  year      =  {2020}
}

@inproceedings{wang2020solo,
  title     =  {{SOLO}: Segmenting Objects by Locations},
  author    =  {Wang, Xinlong and Kong, Tao and Shen, Chunhua and Jiang, Yuning and Li, Lei},
  booktitle =  {Proc. Eur. Conf. Computer Vision (ECCV)},
  year      =  {2020}
}

@inproceedings{wang2020solov2,
  title   =  {{SOLOv2}: Dynamic and Fast Instance Segmentation},
  author  =  {Wang, Xinlong and Zhang, Rufeng and Kong, Tao and Li, Lei and Shen, Chunhua},
  booktitle =  {Proc. Advances in Neural Information Processing Systems (NeurIPS)},
  year    =  {2020}
}

@article{tian2019directpose,
  title   =  {{DirectPose}: Direct End-to-End Multi-Person Pose Estimation},
  author  =  {Tian, Zhi and Chen, Hao and Shen, Chunhua},
  journal =  {arXiv preprint arXiv:1911.07451},
  year    =  {2019}
}

@inproceedings{tian2020conditional,
  title     =  {Conditional Convolutions for Instance Segmentation},
  author    =  {Tian, Zhi and Shen, Chunhua and Chen, Hao},
  booktitle =  {Proc. Eur. Conf. Computer Vision (ECCV)},
  year      =  {2020}
}

License

For academic use, this project is licensed under the 2-clause BSD License - see the LICENSE file for details. For commercial use, please contact Chunhua Shen.