Face Recognition vs Recognize
Since Nextcloud Hub 3 (Aka Nextcloud 25), Recognize was officially integrated as a facial recognition tool for Nextcloud. The reasons for the new development are due to the belief that this is a very complicated application to set up. However, we still defend the design decisions that we applied in this tool.
Let's dismiss the arguments, and I'm just going to do a little experiment to compare both tools.
So, The idea was simple... Install Nextcloud 25 on a small computer, add our (super?) test suite of 2162 photos of The Big Bang Theory, and try both apps. This allowed me to check our installation guides, that of course I found a few things to fix, but in general, it works perfectly.
My new "server" (After five years?):
- Intel(R) Celeron(R) CPU N3350 @ 1.10GHz
- 2 GB RAM and 1 GB Swap.
- eMMC disk 8 GB for system.
- HDD disk of 500GB for storage.
I didn't lie when I said it was a small machine, right? 🙈 but for other things it is a great machine for testing it. 😬
First install our application. Set them with the default model (Just to follow the docs), and as the face:setup command suggests me a maximum of 1.2 GB of ram, just assigned 1.1 GB for caution. Then, in the settings page, also select the maximum image size that allowed me that was 1240x930px (About 1.1 megapixels), and finally just run the face:background_job command in a single execution until finished.
At the end, speaking of raw numbers, what interests us is how many faces detect, in how many clusters I group them, and maybe how long it takes.
SELECT COUNT(1) FROM `oc_facerecog_faces`
4853
SELECT COUNT(1) FROM `oc_facerecog_persons`
1773
SELECT SUM(`processing_duration`)
FROM `oc_facerecog_images`
32335337
So, raw resume...? It detect 4853 faces, group them into 2443 people and it takes almost 9 hours (32335337ms/1000/60/60 = 8,982 038 056 Hrs.)
Then install Recognize, and without configuring anything at all (Just enabling as administrator), the process started. I have to be humble and admit that it's nice that it works so easily. 🎉 🎩 It should be noted that practically from the beginning I could see the results in the photo application, but to make a comparison, first I prefer to talk about the hard numbers
SELECT COUNT(1) FROM `oc_recognize_face_detections`
4493
SELECT COUNT(1) FROM `oc_recognize_face_clusters`
675
SELECT COUNT(1) FROM `oc_recognize_face_detections`
WHERE `cluster_id` IS NULL
869
Wow, Recognize detect 4493 faces, (practically the same as our application), generated 675 clusters of similar faces as persons, and left 869 faces that he couldn't group.
- Our application needs certain settings, which scare users. Recognize works automatically. 🎉
- Both applications detected practically the same number of faces. In our application, this result can be improved maximizing the size of temporary files (But need a bit more of RAM), and Recognize use always the original images, preventing detecting faces in small images.
- To compare the results of number of persons detected (Initially without assigning names) we must add the clusters and the faces without grouping in Recognize (675 + 869 = 1544 persons) since our application considers the individual faces as people as well, having a similar clustering result as well.
- Regarding the processing time, our application took 14.94 seconds (Of course with this particular configuration and hardware) for each photo (or 8,98 Hrs), and I was able to accurately track when the process would finish. As for Recognize, there is no way to know the progress of the process (I can't take it from the database). It only informs the number of remaining images, which in principle does nothing more than increase (and scares), and then decreases until it is 0. I can say that it took about 27 hours, but at this point it's unfair to compare since Face Recognize ran only once, and Recognize ran every 5 minutes with cron.
- Our app offers something like a gallery in the user settings, and Recognize has official integration with the Photos, and the new Memories application. Honestly I can never compete with it. 😞
- Our application has a detailed view of the faces clustering, which allows us to validate them. Recognize by not showing you the faces it's clustering, you never know if the cluster/person is correct.
- Our app has a side panel in Files and Photos, where you can clearly see the faces and give them names. This allows you to better know who you tag, but mainly you can assign names to faces that could not be grouped. In Recognize through photos, the image is tagged (Instead of the face) which IMHO is very confusing and error prone. In particular, there are 869 faces that are impossible to tag in recognize.. 😕
I in particular give a lot of importance to raw numbers, clusters, individual faces and errors, however these do not reflect a real value to compare the quality of the results. By definition of both applications, a person is a grouping of similar faces, which are presumed to be the same person, and the quality of the application must be defined by the quality of these clusters of faces.
As I mentioned before, our application emphasizes in shows the faces to name them. This lets you know if you correctly tag the faces of a single person, or if there are faces mixed up. Recognize however, initially shows a list of faces, where when clicking shows you a list of photos, which are presumably of the same person.
In our test suite, it is likely that all 5 actors are in the same photo, and clicking on any of the faces brings up a list that may initially seem correct. But doing a finer review, it seems that there are some errors. You think that the photographs are of a person, and then you find several photos where this person is not found.
Well, Having a list of faces, it is much easier to evaluate. Face Recognition has them, (and sincerely always was an evaluation tool), and Recognize doesn't. So, If I want to compare both applications, I have to generate something to compare what really matters, and insist again only the quality of the faces clustering matters... Let's do it.? 😬
Finally, after thinking around a lot, I solved it by adding a new experimental command that extracts the faces of both applications, and organizes them in folders indicating the size of the cluster, and all the ids to verify them.
[matias@ecap:/var/www/html/nextcloud]$ sudo -u www-data php occ face:compare -o /tmp/FRTest
[matias@ecap:/var/www/html/nextcloud]$
As a very basic check, I check that the number of faces match in both apps.
[matias@ecap:/var/www/html/nextcloud]$ cd /tmp/FRTest
[matias@ecap FRTest]$ find FaceRecognition/ -type f | wc -l
4853
[matias@ecap FRTest]$ find Recognize/ -type f | wc -l
4493
Great, seems correct.. 😄
Here you can see the result.
From here on, manual work is needed to separate the good clusters (absolute 100% correct faces), from those with an error.
[matias@bangho-mdl FaceRecognition]$ find . -type f | wc -l
4853
[matias@bangho-mdl FaceRecognition]$ find Good\ clusters/ -type f | wc -l
2504
[matias@bangho-mdl FaceRecognition]$ find Bad\ Clusters/ -type f | wc -l
423
[matias@bangho-mdl FaceRecognition]$ find Individual\ Clusters/ -type f | wc -l
1926
Calculating some percentages...
| Total | Good | Bad | Individual |
|---|---|---|---|
| 4853 | 2504 | 423 | 1926 |
| 100,00 % | 51,60 % | 8,72 % | 39,69 % |
We have almost 9 percent of clustering errors, distributed in 23 wrong people. 😞 This is consistent with previous research work, and using model 4 would have had much better results.. 😅
D'Oh!. This reminds me that I never did this comparison of the 4 models. 😅 Today I start with it. 😬
... and recognize.? 🤔
[matias@bangho-mdl Recognize]$ find . -type f | wc -l
4493
[matias@bangho-mdl Recognize]$ find Good\ Clusters/ -type f | wc -l
1861
[matias@bangho-mdl Recognize]$ find Bad\ Clusters/ -type f | wc -l
1717
[matias@bangho-mdl Recognize]$ find Individual\ Clusters/ -type f | wc -l
46
[matias@bangho-mdl Recognize]$ find Individual\ -type f | wc -l
| Total | Good | Bad | Individual | Don't Clustered |
|---|---|---|---|---|
| 4493 | 1861 | 1717 | 46 | 869 |
| 100,00 % | 41,42 % | 38,22 % | 1,02 % | 19,34 % |
This is a 38 percent clustering error, distributed in 81 wrong people, 😞 and furthermore, there is a 19 percent of faces that were not clustered. 🤔 (I honestly don't know how to consider these, but in short they are faces that cannot be tagged). Of course can add some tools to fix them, but honestly, it seems like a lot of errors.
NOTE: Honestly, I did all this evaluation seriously, but knowing that Recognize would have a greater number of errors, however I am surprised by such a bad result.. 😞 If I only told them that the process was going to give bad results just because they chose the wrong tools, but did not offer real data, it would remain an unfortunate comment.
To be continue... 😉
