per_class_accuracy = -nan in many classes

[trminh89]

Hi @alexgkendall

Thanks to your guide! I'm able to train my data and can run the test script to visualize the prediction results. However, there are 2 problems I hope you can give your expert opinion:

1. When do the training, many classes show "per_class_accuracy = -nan", I guess it's normal since my data usually have only 2 or 3 classes in each input image. But does this problem affect the prediction result of our net?
2. Do we have a script/way to summarize the per_class_accuracy when testing? By now test_segmentation_camvid.py file only shows the input/ground truth/prediction images but no statistic over test data available.

Thanks so much!

I1126 12:45:14.784065 13474 solver.cpp:486] Iteration 100, lr = 0.1
I1126 12:45:20.671650 13474 solver.cpp:214] Iteration 120, loss = 0.0285711
I1126 12:45:20.671739 13474 solver.cpp:229] Train net output #0: accuracy = 0.993626
I1126 12:45:20.671751 13474 solver.cpp:229] Train net output #1: loss = 0.0285711 (* 1 = 0.0285711 loss)
I1126 12:45:20.671769 13474 solver.cpp:229] Train net output #2: per_class_accuracy = 0.999469
I1126 12:45:20.671774 13474 solver.cpp:229] Train net output #3: per_class_accuracy = 0.0605096
I1126 12:45:20.671780 13474 solver.cpp:229] Train net output #4: per_class_accuracy = 0
I1126 12:45:20.671785 13474 solver.cpp:229] Train net output #5: per_class_accuracy = -nan
I1126 12:45:20.671790 13474 solver.cpp:229] Train net output #6: per_class_accuracy = -nan
I1126 12:45:20.671795 13474 solver.cpp:229] Train net output #7: per_class_accuracy = -nan
I1126 12:45:20.671810 13474 solver.cpp:229] Train net output #8: per_class_accuracy = -nan
I1126 12:45:20.671815 13474 solver.cpp:229] Train net output #9: per_class_accuracy = -nan
I1126 12:45:20.671823 13474 solver.cpp:486] Iteration 120, lr = 0.1
I1126 12:45:26.587383 13474 solver.cpp:214] Iteration 140, loss = 0.0888415
I1126 12:45:26.587419 13474 solver.cpp:229] Train net output #0: accuracy = 0.976074
I1126 12:45:26.587431 13474 solver.cpp:229] Train net output #1: loss = 0.0888415 (* 1 = 0.0888415 loss)
I1126 12:45:26.587437 13474 solver.cpp:229] Train net output #2: per_class_accuracy = 0.999803
I1126 12:45:26.587443 13474 solver.cpp:229] Train net output #3: per_class_accuracy = 0.12532
I1126 12:45:26.587448 13474 solver.cpp:229] Train net output #4: per_class_accuracy = -nan
I1126 12:45:26.587453 13474 solver.cpp:229] Train net output #5: per_class_accuracy = -nan
I1126 12:45:26.587458 13474 solver.cpp:229] Train net output #6: per_class_accuracy = 0
I1126 12:45:26.587463 13474 solver.cpp:229] Train net output #7: per_class_accuracy = -nan
I1126 12:45:26.587468 13474 solver.cpp:229] Train net output #8: per_class_accuracy = -nan
I1126 12:45:26.587473 13474 solver.cpp:229] Train net output #9: per_class_accuracy = 0.0593645
I1126 12:45:26.587481 13474 solver.cpp:486] Iteration 140, lr = 0.1
I1126 12:45:32.485353 13474 solver.cpp:214] Iteration 160, loss = 0.0526263
I1126 12:45:32.485385 13474 solver.cpp:229] Train net output #0: accuracy = 0.983314
I1126 12:45:32.485396 13474 solver.cpp:229] Train net output #1: loss = 0.0526263 (* 1 = 0.0526263 loss)
I1126 12:45:32.485402 13474 solver.cpp:229] Train net output #2: per_class_accuracy = 0.999622
I1126 12:45:32.485409 13474 solver.cpp:229] Train net output #3: per_class_accuracy = 0.528261
I1126 12:45:32.485414 13474 solver.cpp:229] Train net output #4: per_class_accuracy = -nan
I1126 12:45:32.485419 13474 solver.cpp:229] Train net output #5: per_class_accuracy = -nan
I1126 12:45:32.485424 13474 solver.cpp:229] Train net output #6: per_class_accuracy = 0.0106952
I1126 12:45:32.485429 13474 solver.cpp:229] Train net output #7: per_class_accuracy = -nan
I1126 12:45:32.485433 13474 solver.cpp:229] Train net output #8: per_class_accuracy = -nan
I1126 12:45:32.485438 13474 solver.cpp:229] Train net output #9: per_class_accuracy = 0.0363024
I1126 12:45:32.485445 13474 solver.cpp:486] Iteration 160, lr = 0.1
I1126 12:45:38.385970 13474 solver.cpp:214] Iteration 180, loss = 0.041313
I1126 12:45:38.386006 13474 solver.cpp:229] Train net output #0: accuracy = 0.988395
I1126 12:45:38.386018 13474 solver.cpp:229] Train net output #1: loss = 0.041313 (* 1 = 0.041313 loss)
I1126 12:45:38.386024 13474 solver.cpp:229] Train net output #2: per_class_accuracy = 0.995309
I1126 12:45:38.386030 13474 solver.cpp:229] Train net output #3: per_class_accuracy = 0.00348028
I1126 12:45:38.386036 13474 solver.cpp:229] Train net output #4: per_class_accuracy = 0
I1126 12:45:38.386041 13474 solver.cpp:229] Train net output #5: per_class_accuracy = -nan
I1126 12:45:38.386047 13474 solver.cpp:229] Train net output #6: per_class_accuracy = -nan
I1126 12:45:38.386051 13474 solver.cpp:229] Train net output #7: per_class_accuracy = -nan
I1126 12:45:38.386080 13474 solver.cpp:229] Train net output #8: per_class_accuracy = -nan
I1126 12:45:38.386086 13474 solver.cpp:229] Train net output #9: per_class_accuracy = -nan

[alexgkendall]

Hi @trminh89 ,

1. You can ignore the per class accuracy nans, it is just dividing by zero to get the class accuracy as that class is not present in the training mini-batch. It has no effect on the system and can be ignored (or treated as "class not present").
2. I will be releasing a testing and eval script next week.

Cheers.

[trminh89]

Thanks @alexgkendall !

I'm waiting for your scripts.

Best,

[trminh89]

Hi @alexgkendall,
Is the evaluation script ready?
Cheers,

[alexgkendall]

Hey sorry for the delay. I've added the test script which I use in Matlab - you will need to modify it to iterate through your results. Hope this helps!

https://github.com/alexgkendall/SegNet-Tutorial/blob/master/Scripts/compute_test_results.m