Use the original image to get landmarks and descriptors.

[matiasdelellis]

Well,
I already insisted many times for the degradation of the images when resizing, etc. 😅

Note that when dlib compute the descriptor, it already generates a cut of the face resizing it to 150px. This is yet another degradation, and therefore is better to pass the original image, and let dlib do its magic. 😃

Well, a little test, with our The Big Bang Theory test suite.. 😅

• The first image are the clusters that have an error, with the master branch and the default parameters. (Model 1, 0.4 and 0.99)
• The second image is with the proposed code, and of course the same parameters.

It is not a substantial improvement, but an improvement in the end. 😅
I test it several times, and I always get the same difference.

[stalker314314]

Huh, I did this as I though it will be slower a lot because image will be bigger? To be honest, I never tested it:) If there is no perf impact, of course this is better approach!

[matiasdelellis]

HI @stalker314314

Huh, I did this as I though it will be slower a lot because image will be bigger?

The slow part is the discovery of faces. So, analysis time and memory usage over the area remains absolutely linear.

Maybe I should repeat https://github.com/matiasdelellis/facerecognition/wiki/Comparative-analysis-of-the-models to confirm memory usage, but about times definitely does not change.

GIT MASTER

• real 140m45,593s
• 1422 faces found for clustering
• 756 persons found after clustering

PROPOSED ORIGINAL LANDMARKING

• real 132m14,386s
• 1422 faces found for clustering
• 763 persons found after clustering

As a curiosity, the 100 that i added, was to try to use jitters, which supposedly improves the face descriptor. Last night changed it to test the jitters again, but the result is much worse. 😞
It certainly results in new good face clusters which has not previously discovered, but the errors also increased dramatically. 😞
Reading a little, it seems that this option just change the face image, and average the resulting descriptors. So, again, I prefer to disable it, and work with the original image. 😅

To be honest, I never tested it:)

Out of curiosity, are you still using the application? 😅

If there is no perf impact, of course this is better approach!

My idea is to add another command to migrate between models. Reuse the faces of one of the model, and just calculate the landmarking and descriptors for the new model. It should be relatively fast ..

[matiasdelellis]

Taking advantage of the latest code, here a new little experiment ..

• The first image is the result of analyzing the images with the new code, Model 1 and default parameters. Practically the same image, but this time made absolutely from scratch.
• The second image is the result of migrating the faces from Model 1 to Model 2, and also clustering with the default values.

As you can see, the result of model 2 are much worse, and therefore will discourage their use. 😞
Note that to ensure this is consistent, then delete the Model 1 data, and migrate from Model 2 and I got the same results.

The use is something like that ..

sudo -u apache php occ face:setup --model 1
time sudo -u apache php occ face:background_job -vvv -u user
#real    144m43,629s
#1422 faces found for clustering
#780 persons found after clustering
sudo -u apache php occ face:setup --model 2
time sudo -u apache php occ face:migrate -u user -m 1
293/433 [==================>---------]  67%
#real    17m3,950s
#1422 faces found for clustering
#768 persons found after clustering

As you can see, the first analysis took 2.4 hours and the migration just 17 minutes. This is only 11% of the time. 😬

p.s.: I have my suspicions that the best model to analyze would be HOG since it will not return so many profile faces that are the problems. 🙈

[stalker314314]

Out of curiosity, are you still using the application? sweat_smile

Not at the moment. I changed how I use Nextcloud (from ordinary app to QNAP package...don't ask...), so didn't yet compiled it yet (ABI compat is PITA for QNAP, so you need to nail exact OS on which you are going to cross-compile for it to nail both CXXABI and GLIBCXX to be able to compile dlib)

My idea is to add another command to migrate between models. Reuse the faces of one of the model, and just calculate the landmarking and descriptors for the new model. It should be relatively fast ..

Oh, good idea! Obviosuly, will not work in generic case, both it should work nicely for the models you currently have, I guess

[matiasdelellis]

Houston, we have a problem!. 🙈 haha

I deleted absolutely everything, and I did the same analysis, with all my photos. 😄
Obviously I was very confident, and after 18 hours of analysis, 😬
...resulted in a first cluster with 15% of faces, completely garbage. 😞 😢 😭

Well, after 5 hours thinking, and reviewing the code, I discovered that most of the photos are rotated by exif. 😞
Basically our TempImage rotates the images, and returns the rotated Rect, and when it calculates the landmarks, and the descriptor in original images(WItout rotate..) does it on garbage. 😞

[stalker314314]

I am sorry for your "loss", but this is super funny, I know I shouldn't, but I can't stop laughing:D this is super cool "bug":)

[matiasdelellis]

I am sorry for your "loss", but this is super funny, I know I shouldn't, but I can't stop laughing:D this is super cool "bug":)

haha.. Don't worry, finally this a "nice experiment". 😅

As a curiosity, this was one of the responses of the little survey I made, and it certainly is true. We are still defining many things, and we try to improve others. 😉
But despite all reported many clustering errors, the responses were quite positive. 😄

Not at the moment. I changed how I use Nextcloud.

It's okay... For my part I continue to use with few images, I still have pending test the app with all the family photos. 😬

[matiasdelellis]

Well .. Definitely we have to discard this whole branch. I've been thinking about it, and it's impossible to do it right.

• The CNN face detector can detect the rotated face, although with low confidence. HOG most likely won't detect it.
• The landmark detector is also trained with vertical faces. Therefore it fails in the rotated images.
• The alignment with de landmarks is essential for the correct calculation of the descriptor. So the descriptors are bad.
• Assuming that we can fix these errors, there are so many calculations that need to be done to rotate the rectangles and landmarks. I wrote them, but the main problem is the number of times that have to used throughout the program.

In resume, it is better to analyze the image with the rotation applied as the user would see, which is supposed to be how the faces are correctly aligned. and simply work with the points relative to that image.

Dlib does not have support to open the rotated image, davisking/dlib#1706
Therefore we have to create the temporary file, just to rotate the image.

It is a pity, but today I think there is nothing better to do. 😞

[matiasdelellis]

🙈

[matiasdelellis]

Well.. I think with this we come to the best we can offer. 😃

Finally add 10,000 photos of family, vacations and friends events to test.

So after 4 days of analysis (With the latest commits, this is 6% slower.), with this branch (with the face size restriction too.) and default parameters i get:

	10463 faces found for clustering
	5602 persons found after clustering

Looking for the wrong groups, they are mainly profile faces, badly rotated images, and some babies. At this point, the only way to improve it is by training at least one new landmark model.

For now, it's out of any plan. 😅

Well, doing math, this is just an 2.13% error. I think it is acceptable.

[matiasdelellis]

Rebased on master to test the minimum face size option.

So, my latest conclusions:

1: The minimum face size seems to be quite useless. 😅

Deactivating the option (min_size = 0), i just get an 3.7% error with model 1, and using the default value (min_size = 125) it only improves to 3.23% but increases the number of clusters by 10%. The small benefit does not seem to be justified.. 😞

Min. Size	Clusters	% Clusters	Bad Faces	% Error
0	5054	0,00 %	387	3,70 %
45	5056	0,04 %	387	3,70 %
60	5084	0,59 %	385	3,68 %
70	5109	1,09 %	380	3,63 %
75	5124	1,39 %	382	3,65 %
80	5202	2,93 %	367	3,51 %
90	5214	3,17 %	370	3,54 %
100	5350	5,86 %	357	3,41 %
125	5602	10,84 %	338	3,23 %

On the other hand, wrote some documentation, and already there began to doubt , because the smaller images look great.

• https://github.com/matiasdelellis/facerecognition/wiki/Mininimum-face-size

However I still think the smallest faces should be ignored. At least the smallest of 60px, so I will remove the frontend option, and left it as an advanced option.

2: It is evident that model 2 should be discouraged.

After 4 days I finished analyzing my most complete collection of photos. Set up model 2, and migrate faces from model 1.
It was quite fast (2Hrs 45 Mins for migrate 10682 photos and 10463 faces), and the result was very bad.

Min. Size	Clusters	% Clusters	Bad Faces	% Error
0	4928	0,00 %	1527	14,59 %
80	5072	2,92 %	818	7,82 %
125	5519	11,99 %	742	7,09 %

14.59% error, and reduces to 7.09% if use the default minimum face size. Very disappointing. 😞
First I thought that face migration was still poorly implemented, but set up model 3, and migrated back from model 1, and the result was exactly the same as model 1. 😬
So migration works well, and Model 2 (more precisely the 68-points landmarks model) has many problems, and should not be recommended.

Side note. If the dlib author recommends the 5-point model, use the 5-point model !! 😅

[matiasdelellis]

Ok. More conclusions:

3: This branch also seems quite useless.. haha.

I am doing more tests:

1. I reset my entire collection. (How many times have I done this already? 😅 )
2. Analyzed the entire collection on the master branch over model 1. (I do it with small images to try to do it fast.)
3. Switch to this branch and configure model 3. (This uses the same 5-point model and the same renet)
4. Migrate faces from model 1 to model 3.

Well, theoretically, the result of model 3, which was migrated analyzing the original images, should be the same or better.
So, I do the same analysis (Count the faces that were bad clustered), and calculate the error percentage.
And.. it seems to be a little worse. WTF?. 😞

I have many theories, but I have to do more tests. 🤦

p.s: I am reluctant due of the excessive cost, but I think that I should buy a graphic accelerator with CUDA.. 😞

[matiasdelellis]

Yet another little experiment .. 😅

As I said in the last comment, when analyzing the faces in master and migrating the faces to model 3, it turned out a little worse. Just a little, and inconclusive, but considering that we hoped to improve the result, it contradicts the whole objective of this PR.

The experiment is this... Change the detected face rectangle by randomly adding or subtracting a pixel.

[matias@nube Tasks]$ git diff
diff --git a/lib/BackgroundJob/Tasks/ImageProcessingTask.php b/lib/BackgroundJob/Tasks/ImageProcessingTask.php
index a866b6a..43c001c 100644
--- a/lib/BackgroundJob/Tasks/ImageProcessingTask.php
+++ b/lib/BackgroundJob/Tasks/ImageProcessingTask.php
@@ -144,7 +144,7 @@ class ImageProcessingTask extends FaceRecognitionBackgroundTask {
                                $faces = array();
                                foreach ($rawFaces as $rawFace) {
                                        // Get landmarks of face from model
-                                       $rawLandmarks = $this->model->detectLandmarks($tempImage->getTempPath(), $rawFace);
+                                       $rawLandmarks = $this->model->detectLandmarks($tempImage->getTempPath(), $this->getChangedFaceRect($rawFace));
                                        // Get descriptor of face from model
                                        $descriptor = $this->model->computeDescriptor($tempImage->getTempPath(), $rawLandmarks);
 
@@ -246,6 +246,20 @@ class ImageProcessingTask extends FaceRecognitionBackgroundTask {
                return $this->maxImageAreaCached;
        }
 
+       private function getChangedFaceRect(array $rawFace): array {
+               $face = [];
+               $face['left'] = $rawFace['left'] + random_int(-1, 1);
+               $face['right'] = $rawFace['right'] + random_int(-1, 1);
+               $face['top'] = $rawFace['top'] ;//+ random_int(-1, 1);
+               $face['bottom'] = $rawFace['bottom'];// + random_int(-1, 1);
+               $face['detection_confidence'] = $rawFace['detection_confidence'];
+               return $face;
+       }
+
        /**
         * Helper method, to normalize face sizes back to original dimensions, based on ratio
         *
[matias@nube Tasks]$ 

...and the result is this ..

Face Rect	Bad Faces	Error
Raw (From resized)	6	0,50 %
Raw +-1 Top	8	0,67 %
Raw +-1 Left	11	0,92 %
Raw +-1 Right	12	1,00 %
Raw +-1 Top/Right	12	1,00 %
Raw +-1 Left/Right	13	1,09 %
Raw +-1 Top/Bottom	20	1,67 %
Raw +-1 All	16	1,34 %

You can see that just by changing a pixel on one of the sides, the clustering error increases, and almost tripled when changing all 4 sides.

Well, the first clue to this was this issue of dlib (Having difficulties on selecting bound box for shape_predictor. davisking/dlib#2093) but in particular this comment.. davisking/dlib#2093 (comment)

In summary, the landmark model is very sensitive to training data, and in particular to the size of the face box. So, changing a single pixel can already result in an bad prediction of the landmarks.

In this branch, we obtain the rectangle of the resized image, and we try to get the landmarks from the normalized rectangle. Since the rectangle is in pixels, we round the numbers to get integers, and here we come to the point that we cannot control and it fails.
Well, rounding can result in changing the rectangle by adding or removing a pixel, and something as simple as this makes the results worse.

So, I'm going to cancel this PR again .. 😓 😅

[matiasdelellis]

Close in favour of #309