Thousands of Clusters that Include Many Different People as Same Person

[bsaggy]

Describe the bug
I'm running Recognize against ~35k images. It's creating way too many clusters, currently above 7k and growing.

MariaDB [nextcloud]> select count(*) from oc_recognize_face_clusters;
+----------+
| count(*) |
+----------+
|     7392 |
+----------+
1 row in set (0.005 sec)

The craziest part is that I'll click on a cluster in the Memories App with Mark Person in Preview enabled, and see multiple different people with the green bound box around them across all the pictures in that same cluster.

For example, one cluster has myself, my wife, my grandmother, my mother in law, my sister in law, my brother in law, a friend of a different skin color - all as the person of interest in this cluster. Another cluster has my 2 month old son, myself, my wife, my sister in law, my grandfather, etc. all as the person of interest in the cluster.

While I understand there is a margin for error in facial recognition, I have to believe something is wrong here. With over 7,000 clusters and every cluster containing all kinds of people of interest as indicated by the green bounding box, this is pretty much useless to me at this point. ~92k queued files still yet to go.

To Reproduce
Steps to reproduce the behavior:
I can't say that this is necessarily "reproducible", but this is the evolution of how I have used Recognize thus far.

1. Installed Recognize v2.x on Nextcloud v24.x and let it do its thing.
2. Upgraded to NextCloud 25.0.0 and then 25.0.1 while also upgrading to Recognize v3.1.2.
3. At some point, I did a Reset Tags for Classified Files and a Reset Faces for Classified Files since I didn't seem to have made that much progress yet coming from v2.x and wanted to start fresh on v3.x. I also issued a recrawl. Otherwise I've been letting it just run for about a week now.

Expected behavior
Facial Recognition to work more accurately, and not identity my whole family including friends as the same person. Create less clusters but with more accuracy.

Recognize (please complete the following information):

• JS-only mode: WASM Mode
• Enabled modes: face recognition only

Server (please complete the following information):
System Configuration

Ubuntu 20.04 VM
32 GB RAM (I increased memory & vCPU in hopes of speeding things up to no avail)
8 vCPU
NC v25.0.1
Recognize v3.1.2

Recognize Configuration:

Face Recognition is enabled
6 Cores allowed for use
WASM mode is enabled

Additional context
If there's anything else I can check or do, please let me know.

[marcelklehr]

Hi @bdevy

Thanks for your feedback, I'll try to help troubleshoot. There could be multiple things wrong here. One thing is the number of clusters which is likely due to the recently increased sensitivity in clustering which will create multiple clusters for the same person if the person looks a bit different on some days (think, new haircut, new glasses, etc.). I'm thinking about exposing the sensitivity here as an admin setting to let people play with it themselves. The second thing that could go wrong is the attribution of who this cluster is about. I take it you're using the "memories" app as the photos app doesn't display the person the cluster is about. When looking at the photos of one cluster or another, can you make out a person that is in all photos of a cluster? This is likely the actual person this cluster is about? If there is no person that is in all photos of a cluster, then the actual clustering is at fault, which is quite weird. Perhaps it has to do with WASM mode not being as accurate as normal tensorflow mode. I'll have to investigate that.

[bsaggy]

Hi @marcelklehr,

I think the sensitivity is definitely a factor - there are some pictures where the same person looks a bit different being sent to different unique clusters, but then there are are also pictures of the same person where the pictures were taken just moments apart - the picture looks almost identical but those pictures are being sent to different unique clusters as well. Very confusing. A knob to tune sensitivity might be helpful in this scenario.

I am using the "Memories" app and it works very well - the Photos app had some severe performance issues as the number of clusters grew (One issue is that the merge person dialog would stop appearing when the number of clusters got to maybe around 100 - the second issue is that the photos page would hang while loading when clicking on People, almost as if it was reclassifying all of the recognized faces in real-time - there would even be node.js process running on the server during this time but it got to the point where the page was just unusable. I digress).

When looking at the photos of one cluster or another, can you make out a person that is in all photos of a cluster?

Anyway, no, I cannot make out a person that is in all photos of a given cluster. I do have the "Mark person in preview" enabled so I see the green box around the person - but this box is around different people in the same cluster. Some pictures still have the person I think the cluster is about, but many pictures don't. So my answer to your question has to be "no", and as you say the clustering may be at fault. I'm checking to see if can enable AVX to utilize normal tensorflow mode instead of WASM mode if you think that might help. Over 10,200 clusters found now (~3k increase from 24 hours ago) and 88k files still in queue.

Thank you for your time in helping to understand these issues!

Brian

[marcelklehr]

the Photos app had some severe performance issues as the number of clusters grew

Yeah, we're aware of that now and I've provided a fix that will be released soon. :)

Anyway, no, I cannot make out a person that is in all photos of a given cluster.

Mmh, that's not good :/

[adrianog91]

I have the same problem with the same configuration in WASM mode

[khlschrnk]

Anyway, no, I cannot make out a person that is in all photos of a given cluster.

Mmh, that's not good :/

Same here, the people in several clusters seem to be completely random. I also see mixed faces between babies and elderly and also different colored hair.

[marcelklehr]

I tried but I cannot reproduce this. I enabled WASM mode on my dev instance and let it sift through my personal photo collection. The result is a well sorted collection of photos categorized by face. No false positives at all apart from one rubbish cluster with all the outliers.

[rhatguy]

Same issue here. I'm running in native tensorflow (not WASM). Mine has been running for 1-2 weeks now and is 53k to go. I see on my dual 14 core server (28 real cores) that recognize keeps only one CPU thread spiked and isn't leveraging all available cores, so the initial run through my 41k picture collection is taking forever. I have cores set to 16 in the recognize settings, but it doesn't seem to do anything. Now Nextcloud is complaining that my cron isn't running for over 4 hours because each recognize run is taking so long.

Below you can see an example of one of the clusters that is showing inside of Memories. If clustering is going to be this bad its probably less than useful.

[derekakelly]

I am also experiencing this problem.

MariaDB [nextcloud]> select count(*) from oc_recognize_face_clusters;
+----------+
| count(*) |
+----------+
|    20020 |
+----------+
1 row in set (0.007 sec)

[bsaggy]

I tried but I cannot reproduce this. I enabled WASM mode on my dev instance and let it sift through my personal photo collection. The result is a well sorted collection of photos categorized by face. No false positives at all apart from one rubbish cluster with all the outliers.

Hi @marcelklehr,

I enabled CPU pass through on my VM and WASM mode was no longer necessary so I disabled it. However, the issue persisted. I finally disabled Facial Recognition at 20k clusters. I don't see any point in letting it continue in its current state.

I had installed and just started using Recognition v2 on NC 24 shortly before upgrading to NC 25 and Recognition v3. I don't know if that factors into the behavior at all, but thought it was worth mentioning.

@rhatguy's example is pretty spot on to what I've observed as well - clusters with many different people of interest. Is there any more info I can provide you to aid in the troubleshooting of this issue? Is there a way to "reset" Recognize back to its defaults as if it were a fresh install, and if so is it worth attempting that to see if it then behaves as expected?

Thanks,
Brian

[illnesse]

fwiw, it happens without WASM too

[marcelklehr]

@illnesse Do you experience it, too? Didn't it work for you before?