no message

README.md

@@ -11,6 +11,7 @@ Our paper [Unbiased Scene Graph Generation from Biased Training](https://arxiv.o
 - [x] 2020.06.23 Add No Graph Constraint Mean Recall@K (ng-mR@K) and No Graph Constraint Zero-Shot Recall@K (ng-zR@K) [\[link\]](METRICS.md#explanation-of-our-metrics)
 - [x] 2020.06.23 Allow Scene Graph Detection (SGDet) on Custom Images [\[link\]](#SGDet-on-custom-images)
 - [x] 2020.07.21 Change Custom Scene Graph Detection Output to Json Files
+- [x] 2020.07.21 Visualize Detected Scene Graphs of Custom Images
 
 ## Contents
 
@@ -25,11 +26,12 @@ Our paper [Unbiased Scene Graph Generation from Biased Training](https://arxiv.o
 6. [Scene Graph Generation as RoI_Head](#scene-graph-generation-as-RoI_Head)
 7. [Training on Scene Graph Generation](#perform-training-on-scene-graph-generation)
 8. [Evaluation on Scene Graph Generation](#Evaluation)
-9. [:star2: Detect Scene Graphs on Your Custum Images](#SGDet-on-custom-images)
-10. [Other Options that May Improve the SGG](#other-options-that-may-improve-the-SGG)
-11. [Tips and Tricks for TDE on any Unbiased Task](#tips-and-Tricks-for-any-unbiased-taskX-from-biased-training)
-12. [Frequently Asked Questions](#frequently-asked-questions)
-13. [Citations](#Citations)
+9. [**Detect Scene Graphs on Your Custom Images** :star1:](#SGDet-on-custom-images)
+10. [**Visualize Detected Scene Graphs of Custom Images** :star1:](#Visualize-Detected-SGs-of-Custom-Images)
+11. [Other Options that May Improve the SGG](#other-options-that-may-improve-the-SGG)
+12. [Tips and Tricks for TDE on any Unbiased Task](#tips-and-Tricks-for-any-unbiased-taskX-from-biased-training)
+13. [Frequently Asked Questions](#frequently-asked-questions)
+14. [Citations](#Citations)
 
 ## Overview
 
@@ -159,7 +161,7 @@ CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --master_port 10028 --
 ```
 
 ### Examples of Pretrained Causal MOTIFS-SUM models
-Examples of Pretrained Causal MOTIFS-SUM models on SGDet/SGCls/PredCls (batch size 12): [SGDet Download](https://onedrive.live.com/embed?cid=22376FFAD72C4B64&resid=22376FFAD72C4B64%21781947&authkey=AF_EM-rkbMyT3gs), [SGCls Download](https://onedrive.live.com/embed?cid=22376FFAD72C4B64&resid=22376FFAD72C4B64%21781938&authkey=AO_ddcgNpVVGE-g), [PredCls Download](https://onedrive.live.com/embed?cid=22376FFAD72C4B64&resid=22376FFAD72C4B64%21781937&authkey=AOzowl5-07RzJz4)
+Examples of Pretrained Causal MOTIFS-SUM models on SGDet/SGCls/PredCls (batch size 12): [(SGDet Download)](https://onedrive.live.com/embed?cid=22376FFAD72C4B64&resid=22376FFAD72C4B64%21781947&authkey=AF_EM-rkbMyT3gs), [(SGCls Download)](https://onedrive.live.com/embed?cid=22376FFAD72C4B64&resid=22376FFAD72C4B64%21781938&authkey=AO_ddcgNpVVGE-g), [(PredCls Download)](https://onedrive.live.com/embed?cid=22376FFAD72C4B64&resid=22376FFAD72C4B64%21781937&authkey=AOzowl5-07RzJz4)
 
 Corresponding Results (The original models used in the paper are lost. These are the fresh ones, so there are some fluctuations on the results. More results can be found in [Reported Results](METRICS.md#reported-results)):
 
@@ -175,13 +177,20 @@ MOTIFS-PredCls-TDE  | 33.38 | 45.88 | 51.25 | 17.85 | 24.75 | 28.70 | 8.28 | 14.
 ## SGDet on Custom Images
 Note that evaluation on custum images is only applicable for SGDet model, because PredCls and SGCls model requires additional ground-truth bounding boxes information. To detect scene graphs into a json file on your own images, you need to turn on the switch TEST.CUSTUM_EVAL and give a folder path that contains the custom images to TEST.CUSTUM_PATH. Only JPG files are allowed. The output will be saved as custom_prediction.json in the given DETECTED_SGG_DIR.
 
-Test Example 1 : (SGDet, Causal, **TDE**, SUM Fusion, MOTIFS Model) [checkpoint](https://onedrive.live.com/embed?cid=22376FFAD72C4B64&resid=22376FFAD72C4B64%21781947&authkey=AF_EM-rkbMyT3gs)
+Test Example 1 : (SGDet, Causal, **TDE**, SUM Fusion, MOTIFS Model) [(checkpoint)](https://onedrive.live.com/embed?cid=22376FFAD72C4B64&resid=22376FFAD72C4B64%21781947&authkey=AF_EM-rkbMyT3gs)
 ```bash
 CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --master_port 10027 --nproc_per_node=1 tools/relation_test_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX False MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False MODEL.ROI_RELATION_HEAD.PREDICTOR CausalAnalysisPredictor MODEL.ROI_RELATION_HEAD.CAUSAL.EFFECT_TYPE TDE MODEL.ROI_RELATION_HEAD.CAUSAL.FUSION_TYPE sum MODEL.ROI_RELATION_HEAD.CAUSAL.CONTEXT_LAYER motifs TEST.IMS_PER_BATCH 1 DTYPE "float16" GLOVE_DIR /home/kaihua/glove MODEL.PRETRAINED_DETECTOR_CKPT /home/kaihua/checkpoints/causal-motifs-sgdet OUTPUT_DIR /home/kaihua/checkpoints/causal-motifs-sgdet TEST.CUSTUM_EVAL True TEST.CUSTUM_PATH /home/kaihua/checkpoints/custom_images DETECTED_SGG_DIR /home/kaihua/checkpoints/your_output_path
 ```
 
-The output is a json file. For each image, the output is a dictionary containing bbox(sorted), bbox_labels(sorted), bbox_scores(sorted), rel_pairs(sorted), rel_labels(sorted), rel_scores(sorted), rel_all_scores(sorted), where the last rel_all_scores give all 51 predicates probability for each pair of objects. Note that there may be some trivial bounding boxes, so you can select top-k bbox for better scene graphs.
+Test Example 2 : (SGDet, Causal, **original**, SUM Fusion, MOTIFS Model) [(same checkpoint)](https://onedrive.live.com/embed?cid=22376FFAD72C4B64&resid=22376FFAD72C4B64%21781947&authkey=AF_EM-rkbMyT3gs)
+```bash
+CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --master_port 10027 --nproc_per_node=1 tools/relation_test_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX False MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False MODEL.ROI_RELATION_HEAD.PREDICTOR CausalAnalysisPredictor MODEL.ROI_RELATION_HEAD.CAUSAL.EFFECT_TYPE none MODEL.ROI_RELATION_HEAD.CAUSAL.FUSION_TYPE sum MODEL.ROI_RELATION_HEAD.CAUSAL.CONTEXT_LAYER motifs TEST.IMS_PER_BATCH 1 DTYPE "float16" GLOVE_DIR /home/kaihua/glove MODEL.PRETRAINED_DETECTOR_CKPT /home/kaihua/checkpoints/causal-motifs-sgdet OUTPUT_DIR /home/kaihua/checkpoints/causal-motifs-sgdet TEST.CUSTUM_EVAL True TEST.CUSTUM_PATH /home/kaihua/checkpoints/custom_images DETECTED_SGG_DIR /home/kaihua/checkpoints/your_output_path
+```
+
+The output is a json file. For each image, the output is a dictionary containing bbox(sorted), bbox_labels(sorted), bbox_scores(sorted), rel_pairs(sorted), rel_labels(sorted), rel_scores(sorted), rel_all_scores(sorted), where the last rel_all_scores give all 51 predicates probability for each pair of objects. 
 
+## Visualize Detected SGs of Custom Images
+To visualize the detected scene graphs of custom images, you can follow the jupyter notes: visualization/3.visualize_custom_SGDet.jpynb. The inputs of our visualization code are custom_prediction.json and custom_data_info.json under DETECTED_SGG_DIR. They will be automatically generated if you run the above instruction successfully. Note that there may be too much trivial bounding boxes and relationships, so you can select top-k bbox and predicates for better scene graphs by change parameters box_topk and rel_topk. 
 
 ## Other Options that May Improve the SGG
 

maskrcnn_benchmark/data/build.py

@@ -239,6 +239,10 @@ def make_data_loader(cfg, mode='train', is_distributed=False, start_iter=0):
             custom_data_info['idx_to_files'] = dataset.custom_files
             custom_data_info['ind_to_classes'] = dataset.ind_to_classes
             custom_data_info['ind_to_predicates'] = dataset.ind_to_predicates
+
+            if not os.path.exists(cfg.DETECTED_SGG_DIR):
+                os.makedirs(cfg.DETECTED_SGG_DIR)
+
             with open(os.path.join(cfg.DETECTED_SGG_DIR, 'custom_data_info.json'), 'w') as outfile:  
                 json.dump(custom_data_info, outfile)
             print('=====> ' + str(os.path.join(cfg.DETECTED_SGG_DIR, 'custom_data_info.json')) + ' SAVED !')

maskrcnn_benchmark/engine/inference.py

@@ -161,6 +161,7 @@ def inference(
 def custom_sgg_post_precessing(predictions):
     output_dict = {}
     for idx, boxlist in enumerate(predictions):
+        xyxy_bbox = boxlist.convert('xyxy').bbox
         # current sgg info
         current_dict = {}
         # sort bbox based on confidence
@@ -170,22 +171,22 @@ def custom_sgg_post_precessing(predictions):
         bbox_labels = []
         bbox_scores = []
         for i in sortedid:
-            bbox.append(boxlist.bbox[i].tolist())
+            bbox.append(xyxy_bbox[i].tolist())
             bbox_labels.append(boxlist.get_field('pred_labels')[i].item())
             bbox_scores.append(boxlist.get_field('pred_scores')[i].item())
         current_dict['bbox'] = bbox
         current_dict['bbox_labels'] = bbox_labels
         current_dict['bbox_scores'] = bbox_scores
         # sorted relationships
-        rel_sortedid, _ = get_sorted_bbox_mapping(boxlist.get_field('pred_rel_scores').max(1)[0].tolist())
+        rel_sortedid, _ = get_sorted_bbox_mapping(boxlist.get_field('pred_rel_scores')[:,1:].max(1)[0].tolist())
         # sorted rel
         rel_pairs = []
         rel_labels = []
         rel_scores = []
         rel_all_scores = []
         for i in rel_sortedid:
-            rel_labels.append(boxlist.get_field('pred_rel_scores')[i].max(0)[1].item())
-            rel_scores.append(boxlist.get_field('pred_rel_scores')[i].max(0)[0].item())
+            rel_labels.append(boxlist.get_field('pred_rel_scores')[i][1:].max(0)[1].item() + 1)
+            rel_scores.append(boxlist.get_field('pred_rel_scores')[i][1:].max(0)[0].item())
             rel_all_scores.append(boxlist.get_field('pred_rel_scores')[i].tolist())
             old_pair = boxlist.get_field('rel_pair_idxs')[i].tolist()
             rel_pairs.append([id2sorted[old_pair[0]], id2sorted[old_pair[1]]])