No results on test dataset

[juanluisdb]

• Operating system: ubuntu 16.04
• CUDA version: 9.0
• cuDNN version: 7.0.5
• NVIDIA driver version: 396.44
• GPU models (for all devices if they are not all the same): 950M 4Gb
• python --version output: 2.7.12

I trained a Faster R-CNN with ResNet50 model on my custom train dataset, and everything seems ok. But when I test it on my custom test dataset (test of train_net.py), I get this results:

INFO test_engine.py: 320: Wrote detections to: home/juan/detectron/test/tools_test/generalized_rcnn/detections.pkl
INFO test_engine.py: 162: Total inference time: 101.063s
INFO task_evaluation.py: 76: Evaluating detections
INFO json_dataset_evaluator.py: 162: Writing bbox results json to: home/juan/detectron/test/tools_test/generalized_rcnn/bbox_tools_test_results.json
INFO task_evaluation.py: 62: Evaluating bounding boxes is done!
INFO task_evaluation.py: 181: copypaste: Dataset: tools_test
INFO task_evaluation.py: 183: copypaste: Task: box
INFO task_evaluation.py: 186: copypaste: AP,AP50,AP75,APs,APm,APl
INFO task_evaluation.py: 187: copypaste: -1.0000,-1.0000,-1.0000,-1.0000,-1.0000,-1.0000

But If I run infer_simple.py with the test images, I can see that model is relatively good:

If I test it on my train dataset (I know that this isn't correct), I get this:

INFO test_engine.py: 320: Wrote detections to: /home/juan/detectron/test/tools_train/generalized_rcnn/detections.pkl
INFO test_engine.py: 162: Total inference time: 180.051s
INFO task_evaluation.py: 76: Evaluating detections
INFO json_dataset_evaluator.py: 162: Writing bbox results json to: /home/juan/detectron/test/tools_train/generalized_rcnn/bbox_tools_train_results.json
Loading and preparing results...
DONE (t=0.01s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=0.38s).
Accumulating evaluation results...
DONE (t=0.05s).
INFO json_dataset_evaluator.py: 222: ~~~~ Mean and per-category AP @ IoU=[0.50,0.95] ~~~~
INFO json_dataset_evaluator.py: 223: 93.0
INFO json_dataset_evaluator.py: 231: 95.5
INFO json_dataset_evaluator.py: 231: 91.5
INFO json_dataset_evaluator.py: 231: 91.9
INFO json_dataset_evaluator.py: 232: ~~~~ Summary metrics ~~~~
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.930
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.997
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.993
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.908
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.938
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.449
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.949
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.951
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.927
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.958
INFO json_dataset_evaluator.py: 199: Wrote json eval results to: ./test/tools_train/generalized_rcnn/detection_results.pkl
INFO task_evaluation.py: 62: Evaluating bounding boxes is done!
INFO task_evaluation.py: 181: copypaste: Dataset: tools_train
INFO task_evaluation.py: 183: copypaste: Task: box
INFO task_evaluation.py: 186: copypaste: AP,AP50,AP75,APs,APm,APl
INFO task_evaluation.py: 187: copypaste: 0.9298,0.9967,0.9932,-1.0000,0.9079,0.9380

I don't know why if model can detect object on test dataset, I don't get summary metrics... and only I get -1.0000 in all metrics.

My original dataset's labels was in xml, and I convert it to json with this file, but the generated json to test and train are slightly differents, because this script assign a id number to a name in order in which script find it in dataset. I testes withouth changes, change the json with find+replace to leave category_id same in both json and finally, I changed script, I replaced line 22, and I assign category_item_id to harcoded name:

if name == "plier":
    category_item_id = 1
elif name == "screwdriver":
    category_item_id = 2
elif name == "wrench":
    category_item_id = 3

But in all cases, I always get same results if I test on my test dataset.

[gadcam]

Looks like a duplicate of #293.
#293 (comment) could be the solution

(#588 could be relevant too)

[juanluisdb]

Yes, I changed name of test dataset and everything is ok! thanks!

[shenghsiaowong]

your AP is very high ,my AP is low, how to improve it?

[juanluisdb]

your AP is very high ,my AP is low, how to improve it?

This AP is evaluating with the train dataset

[derekhsu]

Yes, I changed name of test dataset and everything is ok! thanks!

I change voc_2007_val to voc_2007_test but it didn't work as well

[juanluisdb]

The folder can't contain "test"