Fast Contextual Scene Graph Generation with Unbiased Context Augmentation

---

This repository includes the code for scene graph generation on three datasets VG, OpenPSG and OIDv6.

Basic operating requirement:

pip install -r requirements.txt

Contents

VG_dataset

Download the VG annotations dataset

Download the evaluation folder evaluation

Download the yolov5 object detection pre-trained model yolol_object_test_28.pt

Then put them into the VG_dataset folder

 cd VG_dataset

VG_dataset
    ├── ckn (contains trained ckn models)
    ├── vdn (contains trained vdn models)
    ├── evaluation 
    ├── models (yolov5 model)
    ├── dataset (annotations from VG datasets)
    ├── utils 
    ├── eval (contains eval files)
    ├── ckn_main.py
    ├── datapath.py
    ├── dataset.py
    ├── yolo_dataset.py
    └── yolol_object_test_28.pt

Evalution

C-SGG PredCls evalution on VG

 python eval/predcls_ckn_val.py 

C-SGG SGGen evalution on VG

 python eval/sggen_ckn_val.py 

CV-SGG PredCls evalution on VG

Download VG images from Scene-Graph-Benchmark

Modify the 'image_file' in datapath.py according to download images'path

 python eval/predcls_vdn_val.py 

CV-SGG SGGen evalution on VG

 python eval/sggen_vdn_val.py 

Train

During the verification process, the loaded annotations will be saved as npy files for subsequent fast loading. Before training, the npy file of the test set needs to be deleted.

 rm *.npy *.pkl 

Start training

train C-SGG

 python ckn_main.py 

train CV-SGG

 python vdn_main.py 

Acknowledgment:

The VG dataset and evalutation are from Scene-Graph-Benchmark

The yolov5 model are trained based on yolov5

---

PSG_dataset

Download the PSG annotations dataset

Download the panoptic segmentation results SegFromer_PVTV for context description.

Download the groudtruth psg_eval_results.pytorch for evaluation.

Then put them into the PSG_dataset folder

 cd PSG_dataset

PSG_dataset
    ├── ckn (contains trained ckn models)
    ├── vdn  (contains trained vdn models)
    ├── dataset (annotations from OpenPSG datasets)
    ├── openpsg (from [OpenPSG] code repository)
    ├── psg_eval_results.pytorch (contains groudtruth for facilitate evaluation)
    ├── SegFromer_PVTV (contains panoptic segmentation results)
    ├── psg_results (previous best results)
    ├── psg_infer.py 
    ├── psg_visual_infer.py
    ├── requirements.txt (OpenPSG depends on mmcv==1.4.3 and mmdet==2.20.0, the available environment is uploaded here)
    └── sgg_eval.py

C-SGG SGGen evalution on OpenPSG

python psg_infer.py

Under the /PSG_dataset folder will generation a new file relation.json

python sgg_eval.py

Evaluate newly generated results. Due to the processed grondtruth from OpenPSG /tools/grade.py, the computer memory is preferably >32GB

CV-SGG SGGen evalution on OpenPSG

Download the images from OpenPSG

Rewrite the 'image_path' in psg_visual_infer.py according to dataset path

python psg_visual_infer.py #infer

python sgg_eval.py #evalute

Acknowledgment:

The annotation, images, and groudtruth are from OpenPSG, any environment configuration issues can also refer to it.

The panoptic segmentation results are from Panoptic SegFormer PVTv2-B5

---

OID_dataset

Download the openimage_v6_test for object detection results and groudtruth.

Then put it into the OID_dataset folder.

 cd OID_dataset

OID_dataset
    ├── ckn   (contains trained ckn models)
    ├── openimage_v6_test (for facilitate evaluation)
    ├── pysgg (from [PySGG] code repository)
    ├── vdn   (contains trained vdn models)
    ├── utils_evaluation.py
    ├── oid_inference.py
    ├── oid_visual_inference.py
    └── oid_evaluation.py

C-SGG SGGen evalution on OIDv6

python oid_inference.py

Under the /openimage_v6_test folder will generation a new file eval_results.pytorch

python oid_evaluation.py

Evaluate newly generated results

CV-SGG SGGen evalution on OIDv6

Download the processed OpenImagev6 dataset from PySGG

Modify the 'yourpath' in oid_visual_inference.py according to dataset path

python oid_visual_inference.py

Under the /openimage_v6_test folder will generation a new file eval_results.pytorch

python oid_evaluation.py

Evaluate newly generated results

Acknowledgment:

The processed datasets and object detection results are from PySGG