Amodal Instance Segmentation with Transformer

Table of Contents

• Introduction
• Usage
• Trained models
• Acknowledgement
• Citation

Introduction

The figure above illustrates our AISFormer architecture. The main implementation of this network can be foundd here.

Usage

1. Installation

1.1. Set up project directory:

• Create a parent project folder

mkdir ~/AmodalSeg
export PROJECT_DIR=~/AmodalSeg 

1.2. Install python environment

• Conda, Pytorch and other dependencies

conda create -n aisformer python=3.8 -y
source activate aisformer 

conda install pytorch==1.10.0 torchvision==0.11.0 cudatoolkit=11.3 -c pytorch
pip install ninja yacs cython matplotlib tqdm
pip install opencv-python==4.4.0.40
pip install scikit-image
pip install timm==0.4.12
pip install setuptools==59.5.0
pip install torch-dct

• Install cocoapi

cd $PROJECT_DIR/
git clone https://github.com/cocodataset/cocoapi.git
cd cocoapi/PythonAPI
python setup.py build_ext install

• Install AISFormer

cd $PROJECT_DIR/
git clone https://github.com/UARK-AICV/AISFormer
cd AISFormer/
python3 setup.py build develop

• Expected Directory Structure

$PROJECT_DIR/
|-- AISFormer/
|-- cocoapi/

2. Data preparation

2.1. KINS dataset

Download the Images from KITTI dataset.

The Amodal Annotations could be found at KINS dataset

2.2. D2SA dataset

The D2S Amodal dataset could be found at mvtec-d2sa.

2.3. COCOA-cls dataset

The COCOA dataset annotation from here (reference from github.com/YihongSun/Bayesian-Amodal) The images of COCOA dataset is the train2014 and val2014 of COCO dataset.

2.4. Expected folder structure for each dataset

AISFormer support datasets as coco format. It can be as follow (not necessarily the same as it depends on register data code)

$PROJECT_DIR/
|-- AISFormer/
|-- cocoapi/
|-- data/
|---- datasets/
|------- KINS/
|---------- train_imgs/
|---------- test_imgs/
|---------- annotations/
|------------- train.json
|------------- test.json
|------- D2SA/
|...

Then, See here for more details on data registration

2.5. Generate occluder mask annotation

After registering, run the preprocessing scripts to generate occluder mask annotation, for example:

python -m detectron2.data.datasets.process_data_amodal \
   /path/to/KINS/train.json \
   /path/to/KINS/train_imgs \
   kins_dataset_train

the expected new annotation can be as follow:

$PROJECT_DIR/
|-- AISFormer/
|-- cocoapi/
|-- data/
|---- datasets/
|------- KINS/
|---------- train_imgs/
|---------- test_imgs/
|---------- annotations/
|------------- train.json
|------------- train_amodal.json
|------------- test.json
|------- D2SA/
|...

3. Training, Testing and Demo

Configuration files for training AISFormer on each datasets are available here. To train, test and run demo, see the example scripts at scripts/:

Trained models

• AISFormer R50 on KINS
• AISFormer R50 on D2SA (TBA)
• AISFormer R50 on COCOA-cls (TBA)

Acknowledgement

This code utilize BCNet for dataset mapping with occluder, VRSP-Net for amodal evalutation, and detectron2 as entire pipeline with Faster RCNN meta arch.

Citation

@article{tran2022aisformer,
  title={AISFormer: Amodal Instance Segmentation with Transformer},
  author={Tran, Minh and Vo, Khoa and Yamazaki, Kashu and Fernandes, Arthur and Kidd, Michael and Le, Ngan},
  journal={arXiv preprint arXiv:2210.06323},
  year={2022}
}