Training graph dives to Zero

[hnwynters]

I have been trying to train a custom class. I am working with the cats images from the Google Image Set., a set of 3,500 images. I put about 10% of those images into the val directory and the rest in the train directory. I wrote my own program to convert the annotations and the numbers check out with the convert_voc_to yolo.py program. No problem there. I have tried training with 6 decimal places and as many as 16 decimal places and there is no change.

I set the yolov3-custom.cfg to 4000 with 3200 and 3600. 18 filters and 1 class; Cat. I have my train.txt and val.txt files pointing to the images and the annotation (txt) files are in the same directory as the images. I have tried the image size beginning at 416 going all the way up to 608. The images are all fairly large so I don't thing size is an issue. I've also set batch to 64 and subdivisions to 8 and 16 and 32 and 64. The only change it is takes much longer to dive to zero. With a mAP value of zero as well.

I'm using Windows 10 with a Ryzen 9 5950x processor and a RTX3090 and based on what I see on my screen the GPU is working as hard as it can go with 16 processors doing all they can do.. It normally takes less than 2 hours to train those images. That seems fast to me but since I haven't been able to get it to work I don't even know that for sure.

I can train the image set koala and get a mAP value of between 90 and 100. detection for koalas works very well. I also tried training it with the dragonfly image set and even though it also dived down to zero pretty quickly, and the mAP stayed at zero, it was able to detect dragonflies fairly well; I just don't trust it yet. It needs more samples but for now I just want to see it work as it should.

I'm using the AlexeyAB Darknet version as of 6/26/2022 compiled with Visual Studio 2019.

I even made sure the backslash in the train.txt, val.txt, and detector.data files were downslope, , rather than the upslope, /. I don't know why that is backwards, at least to me, but it's easy enough to do it. I've also tried it both ways with similar results.

Can anyone point me in the right direction? I'm still fairly new to Darknet and it seems powerful. I'd love to see this PC do something besides waste my time.

I have seen a few discussions about low mAP values but their solution did nothing for my issue.

[stephanecharette]

What does "dives to zero" mean? You've explained a bunch of things, but not what isn't working!

[hnwynters]

Sorry, thought that would be obvious. Guess not.

The graph stays pegged at the top for perhaps 250 iterations and then drops below 20 as it begins it's journey down to zero. There is some curve to the graph but very slight. By the time it reaches the 400th iteration the average loss is very near zero and after passing 400 it simply drops slowly down to absolute zero before the 600th iteration.

I would say it is similar to dropping off a cliff with perhaps a slight wind.

I've checked the images using yolo mark and the bounding boxes are right where they should be.

Not sure if there is a way to tell if the images are actually being opened. Last night when I tried the show_imgs parameter the program opened perhaps six images and seemed to lock up. I did not just let it run all night to see if I was wrong but there was no change to the command box and the images refused to close when I tried to do so.

[hnwynters]

If I may add a bit more information. Once reaching zero it stays at zero until the 4000 iterations have been completed. The mAP value is always reported as zero.

This does not seem to be training. The idea that I have a perfect data set is nonsensical.

[stephanecharette]

The idea that I have a perfect data set is nonsensical.

Who said it should be?

Why not simply paste the chart.png so we can see it? At the same time, post a sample image and the annotation you have. My prediction is your image size+object size don't work with the network dimensions you've chosen to use. Thus, zero predictions are made, and the mAP stays at zero the entire way.

[hnwynters]

Here is a chart from my last run. The bottom line continues on as seen.

Here is one of the images and it's associated txt file.

00b26c158f79580a.txt

I have tried a width and height of up to 608.

I'll try running one at twice that size and see what happens.

[stephanecharette]

From this chart, I can tell you don't have mAP enabled. Did you train using the recommended command: https://www.ccoderun.ca/programming/darknet_faq/#training_command

I assume the cat is what you are trying to find? Do you have the entire cat as an annotation? Or just the head? If you have the entire cat, then your network size can be 224x224 without any problem:

From this image, you can see the cat is still much larger than the 16x16 minimum. So if you want you could probably reduce the network even more without any problems.

[hnwynters]

Yes, I'm looking for the entire cat. I'll try reducing the size. Anything to speed things up always helps.

I do have the -map parameter in my darknet command but I did not allow the graph to run all the way to the end. Here is what the full training session looks like.

I was not aware one could train without using the weights file. I'll reduce the size and try running without the weights and see what I wind up with.

[hnwynters]

I changed the width and height to 224. I added the -map parameter so you can see what I'm describing is accurate. I played around the the learning_rate setting. At one point I had decreased it to 0.0000005 and all it did was shift the cliff drop toward the right. Still dropped down to zero and a mAP value of 0. I also tried several intermediate values with similar results.

I have over 3000 positive training cat images and at least that many negative images with empty *.txt files. I have about 500 positive val images with about the same number of negative images; again with empty *.txt files.

It does run faster and that is nice but I don't think any training is going on. I know it's opening the images because when I set -show_imgs I can see pictures of cats.

I'm including the chart I ran earlier today with a max_batch of 6000 with all the positive and negative images for both train and val. I'm fairly certain I have set the computer up properly as I do get positive hits on new images of koalas that I trained on Saturday or Sunday as well as with the dragonfly class, though I distrust the results as the graph was similar to that as with the cats class. Only about 350 images for dragonflies.

I am thoroughly stuck at this point.

[stephanecharette]

Turn your learning rate down. You have an over-simplistic network. #6730

Also: how many negative samples do you have? 50% of your images should be negative samples. https://www.ccoderun.ca/programming/darknet_faq/#tldr

[stephanecharette]

Join the discord? Share your files and I'll try it out and see what I get?

[hnwynters]

Here are a few of the important files. Perhaps you can take a look and see where I'm going wrong.
yolov3-custom .cfg.txt
Train.bat.txt
Detector.data.txt
classes.txt
Train.txt
Val.txt
00a1fff876518970.txt
00a1fff876518970.xml.txt

As I mentioned earlier, I wrote a program to convert to/from voc and yolo.txt files. I'm compared the values and you are free to do so as well. I have included one xml and my version of the yolo.txt. I can easily modify my program to limit the number of decimal places should that wind up being the problem.

I"m not familiar with Discord. I don't mind loading files to my dropbox if you are interested.

It may also be that I made an error when using CMake to compile Darknet. I can also load images of my CMake settings used to compile and build Darknet.

[hnwynters]

Now here is something weird and I think it's important.

I wrote a program to convert the xml files to yolo txt files. I can change the decimal places as I wish. But here is the interesting thing.

The top line is the results of the program that I wrote and the second line is the line converted by convert_voc_to_yolo.py.
0 0.6811523437500000, 0.5957031250000000, 0.2177734375000000, 0.8033854166666666
0 0.68115234375 0.595703125 0.2177734375 0.8033854166666666

The numbers are correct but there is a significant difference. All the trailing zeros are eliminated from the convert_voc_to_yolo.py program. I am running a sample against these files and the training appears to be working. Here is the chart of the training result.

So I went in and modified my program to remove all trailing zeros during conversion and I wound up this these values.
0 0.68115234375 0.595703125 0.2177734375 0.8033854166666666

Notice how they are exactly like the convert_voc_to_yolo.py values. Same images and same values.

And here is the chart with the results of the new values.

I assume the difference in the graphs and mAP values are due to training being not an exact science.
But here is the important question...
Why do trailing zeros cause training errors? Aren't they just nothings and shouldn't nothings affect nothing? But somehow there is a difference.

[mr-mainak]

I'm having the same problem as yours. However there are no trailing zeros in my case. This is the output while calculating mAP:

calculation mAP (mean average precision)...
 Detection layer: 30 - type = 28 
 Detection layer: 37 - type = 28 

 detections_count = 0, unique_truth_count = 406  
class_id = 0, name = fire, ap = 0.00%   	 (TP = 0, FP = 0) 

 for conf_thresh = 0.25, precision = -nan, recall = 0.00, F1-score = -nan 
 for conf_thresh = 0.25, TP = 0, FP = 0, FN = 406, average IoU = 0.00 % 

 IoU threshold = 50 %, used Area-Under-Curve for each unique Recall 
 mean average precision ([email protected]) = 0.000000, or 0.00 % 

Any sugestions? @stephanecharette @hnwynters @AlexeyAB
Regards