mAPs of some networks are much lower than expected.

[chjej202]

Hello,

I tested multiple pre-trained weights provided by tkDNN with "map_demo."
However, mAP of some networks are lower than expected.
(mobilenetv2ssd512, yolo3tiny, dla34_cnet, resnet101_cnet)

mAP of yolo3tiny is too small. (mAP 0.5 : 0.12)
mAP of mobilenetv2ssd512 is too small even though input image is larger (512x512) than yolo (416x416) variants.
mAP of dla34_cnet and resnet101_cnet is not competitive to yolov3 which is an old network compared to cnet.

Do you have mAP results of all the networks which can run with "map_demo"?

I ran the program on Nvidia Jetson AGX Xavier with Jetpack 4.3
I used coco 2017 valid set with 4952 images to check the accuracy of each network.
All models are built with FP16 mode.

The below results shows the mAP of each network.

cspresnext50-panet-spp
Classes: 80 mAP 0.5: 0.638923
Classes: 80 mAP 0.55: 0.620606
Classes: 80 mAP 0.6: 0.598838
Classes: 80 mAP 0.65: 0.567956
Classes: 80 mAP 0.7: 0.524836
Classes: 80 mAP 0.75: 0.47366
Classes: 80 mAP 0.8: 0.393327
Classes: 80 mAP 0.85: 0.270965
Classes: 80 mAP 0.9: 0.128609
Classes: 80 mAP 0.95: 0.0145363
mAP 0.5:0.95 = 0.423226
avg precision: 0.80101 avg recall: 0.669992 avg f1 score:0.729667

yolo4tiny
Classes: 80 mAP 0.5: 0.276675
Classes: 80 mAP 0.55: 0.262956
Classes: 80 mAP 0.6: 0.245394
Classes: 80 mAP 0.65: 0.220419
Classes: 80 mAP 0.7: 0.188917
Classes: 80 mAP 0.75: 0.145544
Classes: 80 mAP 0.8: 0.0963829
Classes: 80 mAP 0.85: 0.0461462
Classes: 80 mAP 0.9: 0.0122035
Classes: 80 mAP 0.95: 0.000870022
mAP 0.5:0.95 = 0.149551
avg precision: 0.661997 avg recall: 0.307922 avg f1 score:0.420331

yolov4
Classes: 80 mAP 0.5: 0.59428
Classes: 80 mAP 0.55: 0.576454
Classes: 80 mAP 0.6: 0.556307
Classes: 80 mAP 0.65: 0.524986
Classes: 80 mAP 0.7: 0.487346
Classes: 80 mAP 0.75: 0.434977
Classes: 80 mAP 0.8: 0.364772
Classes: 80 mAP 0.85: 0.262967
Classes: 80 mAP 0.9: 0.139866
Classes: 80 mAP 0.95: 0.0178965
mAP 0.5:0.95 = 0.395985
avg precision: 0.757215 avg recall: 0.633702 avg f1 score:0.689974

yolo3tiny
Classes: 80 mAP 0.5: 0.120749
Classes: 80 mAP 0.55: 0.115145
Classes: 80 mAP 0.6: 0.108731
Classes: 80 mAP 0.65: 0.0975029
Classes: 80 mAP 0.7: 0.0819545
Classes: 80 mAP 0.75: 0.0617638
Classes: 80 mAP 0.8: 0.0400224
Classes: 80 mAP 0.85: 0.0181904
Classes: 80 mAP 0.9: 0.00465814
Classes: 80 mAP 0.95: 0.0005046
mAP 0.5:0.95 = 0.0649222
avg precision: 0.438526 avg recall: 0.168005 avg f1 score:0.242937

yolo3
Classes: 80 mAP 0.5: 0.53915
Classes: 80 mAP 0.55: 0.518971
Classes: 80 mAP 0.6: 0.49056
Classes: 80 mAP 0.65: 0.450069
Classes: 80 mAP 0.7: 0.400232
Classes: 80 mAP 0.75: 0.324907
Classes: 80 mAP 0.8: 0.227801
Classes: 80 mAP 0.85: 0.118608
Classes: 80 mAP 0.9: 0.0299095
Classes: 80 mAP 0.95: 0.00156115
mAP 0.5:0.95 = 0.310177
avg precision: 0.734336 avg recall: 0.573578 avg f1 score:0.644077

mobilentv2ssd512
Classes: 80 mAP 0.5: 0.298511
Classes: 80 mAP 0.55: 0.289307
Classes: 80 mAP 0.6: 0.274653
Classes: 80 mAP 0.65: 0.256511
Classes: 80 mAP 0.7: 0.232621
Classes: 80 mAP 0.75: 0.204093
Classes: 80 mAP 0.8: 0.164697
Classes: 80 mAP 0.85: 0.112635
Classes: 80 mAP 0.9: 0.0504613
Classes: 80 mAP 0.95: 0.00523403
mAP 0.5:0.95 = 0.188872
avg precision: 0.640366 avg recall: 0.334324 avg f1 score:0.439298

resnet101_cnet
Classes: 80 mAP 0.5: 0.438912
Classes: 80 mAP 0.55: 0.424819
Classes: 80 mAP 0.6: 0.406554
Classes: 80 mAP 0.65: 0.384212
Classes: 80 mAP 0.7: 0.354354
Classes: 80 mAP 0.75: 0.312488
Classes: 80 mAP 0.8: 0.262612
Classes: 80 mAP 0.85: 0.199722
Classes: 80 mAP 0.9: 0.116854
Classes: 80 mAP 0.95: 0.0265936
mAP 0.5:0.95 = 0.292712
avg precision: 0.606124 avg recall: 0.494776 avg f1 score:0.544819

dla34_cnet
Classes: 80 mAP 0.5: 0.463488
Classes: 80 mAP 0.55: 0.449222
Classes: 80 mAP 0.6: 0.42911
Classes: 80 mAP 0.65: 0.408078
Classes: 80 mAP 0.7: 0.379758
Classes: 80 mAP 0.75: 0.341952
Classes: 80 mAP 0.8: 0.293338
Classes: 80 mAP 0.85: 0.226469
Classes: 80 mAP 0.9: 0.138922
Classes: 80 mAP 0.95: 0.032453
mAP 0.5:0.95 = 0.316279
avg precision: 0.569139 avg recall: 0.536718 avg f1 score:0.552453

[mive93]

Hi @chjej202
yes, it is true. The problem is that (I'm sure) you used confidence threshold = 0.3.
It changes a lot the results indeed. Try using 0.05 instead and you will get the correct values.

[chjej202]

Hi, I changed a confidence threshold (conf_threshold) in DetectionNN.h from 0.3 to 0.05.
It increases mAP of all networks, but I still don't know some networks (dla34_cnet, resnet101_cnet, mobilenetv2ssd, yolov3tiny) work poorly compared to other networks.

Do you have any reference results tested by map_demo to check my results are correct or not?

===========================================================================
Below results are my test results with conf_threshold = 0.05

yolov3tiny (416x416, conf_threshold = 0.05)
Classes: 80 mAP 0.5: 0.180512
Classes: 80 mAP 0.55: 0.166291
Classes: 80 mAP 0.6: 0.150633
Classes: 80 mAP 0.65: 0.129194
Classes: 80 mAP 0.7: 0.103001
Classes: 80 mAP 0.75: 0.0742617
Classes: 80 mAP 0.8: 0.0453536
Classes: 80 mAP 0.85: 0.0195794
Classes: 80 mAP 0.9: 0.00479772
Classes: 80 mAP 0.95: 0.000505335
mAP 0.5:0.95 = 0.0874129
avg precision: 0.184815 avg recall: 0.327286 avg f1 score:0.236232

mobilenetv2ssd512 (512x512, conf_threshold = 0.05)
Classes: 80 mAP 0.5: 0.376689
Classes: 80 mAP 0.55: 0.359028
Classes: 80 mAP 0.6: 0.334558
Classes: 80 mAP 0.65: 0.304146
Classes: 80 mAP 0.7: 0.267701
Classes: 80 mAP 0.75: 0.228689
Classes: 80 mAP 0.8: 0.180935
Classes: 80 mAP 0.85: 0.120037
Classes: 80 mAP 0.9: 0.0523691
Classes: 80 mAP 0.95: 0.00519432
mAP 0.5:0.95 = 0.222935

resnet101_cnet (512x512, conf_threshold = 0.05)
Classes: 80 mAP 0.5: 0.502354
Classes: 80 mAP 0.55: 0.481269
Classes: 80 mAP 0.6: 0.454985
Classes: 80 mAP 0.65: 0.425261
Classes: 80 mAP 0.7: 0.387514
Classes: 80 mAP 0.75: 0.337049
Classes: 80 mAP 0.8: 0.279151
Classes: 80 mAP 0.85: 0.209493
Classes: 80 mAP 0.9: 0.120783
Classes: 80 mAP 0.95: 0.0270623
mAP 0.5:0.95 = 0.322492
avg precision: 0.0872499 avg recall: 0.683761 avg f1 score:0.154753

dla34_cnet (512x512, conf_threshold = 0.05)
Classes: 80 mAP 0.5: 0.525197
Classes: 80 mAP 0.55: 0.506484
Classes: 80 mAP 0.6: 0.480806
Classes: 80 mAP 0.65: 0.453351
Classes: 80 mAP 0.7: 0.41684
Classes: 80 mAP 0.75: 0.369797
Classes: 80 mAP 0.8: 0.313306
Classes: 80 mAP 0.85: 0.238509
Classes: 80 mAP 0.9: 0.14341
Classes: 80 mAP 0.95: 0.0331084
mAP 0.5:0.95 = 0.348081
avg precision: 0.0990177 avg recall: 0.740642 avg f1 score:0.174682

yolov4tiny (416x416, conf_threshold = 0.05)
Classes: 80 mAP 0.5: 0.371197
Classes: 80 mAP 0.55: 0.345488
Classes: 80 mAP 0.6: 0.314449
Classes: 80 mAP 0.65: 0.274678
Classes: 80 mAP 0.7: 0.223176
Classes: 80 mAP 0.75: 0.164865
Classes: 80 mAP 0.8: 0.10477
Classes: 80 mAP 0.85: 0.0487025
Classes: 80 mAP 0.9: 0.012689
Classes: 80 mAP 0.95: 0.000883334
mAP 0.5:0.95 = 0.18609
avg precision: 0.289453 avg recall: 0.501469 avg f1 score:0.367044

yolov4 (416x416, conf_threshold = 0.05)
Classes: 80 mAP 0.5: 0.672525
Classes: 80 mAP 0.55: 0.645826
Classes: 80 mAP 0.6: 0.615812
Classes: 80 mAP 0.65: 0.574007
Classes: 80 mAP 0.7: 0.52595
Classes: 80 mAP 0.75: 0.463363
Classes: 80 mAP 0.8: 0.381599
Classes: 80 mAP 0.85: 0.271517
Classes: 80 mAP 0.9: 0.142526
Classes: 80 mAP 0.95: 0.0179826
mAP 0.5:0.95 = 0.431111
avg precision: 0.426695 avg recall: 0.771391 avg f1 score:0.549458

yolov3 (416x416, conf_threshold = 0.05)
Classes: 80 mAP 0.5: 0.623093
Classes: 80 mAP 0.55: 0.592053
Classes: 80 mAP 0.6: 0.550943
Classes: 80 mAP 0.65: 0.499786
Classes: 80 mAP 0.7: 0.436613
Classes: 80 mAP 0.75: 0.34891
Classes: 80 mAP 0.8: 0.238972
Classes: 80 mAP 0.85: 0.122449
Classes: 80 mAP 0.9: 0.0305109
Classes: 80 mAP 0.95: 0.00156678
mAP 0.5:0.95 = 0.34449
avg precision: 0.42965 avg recall: 0.721075 avg f1 score:0.53846

csresnext50-panet-spp (416x416, conf_threshold = 0.05)
Classes: 80 mAP 0.5: 0.722289
Classes: 80 mAP 0.55: 0.69365
Classes: 80 mAP 0.6: 0.658735
Classes: 80 mAP 0.65: 0.616528
Classes: 80 mAP 0.7: 0.559819
Classes: 80 mAP 0.75: 0.497859
Classes: 80 mAP 0.8: 0.406925
Classes: 80 mAP 0.85: 0.276912
Classes: 80 mAP 0.9: 0.129772
Classes: 80 mAP 0.95: 0.0145543
mAP 0.5:0.95 = 0.457704
avg precision: 0.495352 avg recall: 0.802046 avg f1 score:0.61245

[mive93]

HI @chjej202,

No idea why. I will check all of them tomorrow and will give you a proper answer

[chjej202]

Thank you. I will wait for your answer.

[mive93]

Hi @chjej202,

sorry for the wait. It actually took me till today to find out and solve the issue.
Here you can find what I did : AlexeyAB/darknet#5354 (comment).
Moreover, here's the current mAP results of tkDNN https://github.com/ceccocats/tkDNN#map-results.

The results are now coherent to darknet :)
Notice that the values of the map differ from the ones in the paper: this is due to the different testset, which is not test-dev, but COCO val 2017 instead.

Also, tkDNN mAP computation is a bit pessimistic wrt the one by CodaLab, but I still don't know why.

[chjej202]

Thank you for your reply. :)
My mAP is similar to your results, so I will close the issue.