remove duplicate entries in photos

[elpraga]

I have noticed that many of the photo entries are listed twice

Reason 1:
there are two identical copies of the photo - two files got scanned, fair enough, but it still would be nice to have only one photo shown on the map if the files are identical (sometimes there are reasons to have two copies of the same photo)

Reason 2:
Some photos do not have a duplicate file on my system as I am aware, but still I can see duplicate entries.. Could the reason be that I ran the occ scan command twice?

I assumed that the second run will just update the information. Is that the case or does the information get added every time? If the information gets added, is there a way to erase all the information and then run the scan again?

[tacruc]

Could the reason be that I ran the occ scan command twice?

No, occ delets all DB entries before scanning pictures

[jancborchardt]

there are two identical copies of the photo - two files got scanned, fair enough, but it still would be nice to have only one photo shown on the map if the files are identical (sometimes there are reasons to have two copies of the same photo)

Makes sense, you could have copied photos over to other folders etc. We could combine duplicate photos into the same entry indeed.

[tacruc]

I like the Idea, but I still have problems how to detect duplicated photos efficiently.
By Name? By hash? By Size?
I think it has to be on the data level as comparing the actual image (on a visual level) might be to difficult. On the other hand we cloud just set an option to configure the folder, where to look for pictures.

[jancborchardt]

Yeah, I’d assume by hash or so. That’s why I set it to a milestone further in the future. And we can still think about if it makes sense to include. :)

[elpraga]

What about checking names first (faster) and if name matches, could hashes as well?

That would exclude identical photos with different names, but it would be significantly faster...

[julien-nc]

Tough decision to make. Is it worth it to loose performance to avoid showing identical pictures? 🤔

I don't think using hashes is that problematic performance-wise. It would slow down the scan a bit but that's it. Let's assume there would be an index on the "hash" field in the DB which would allow us to group results by hash when asking for all user's pictures, efficiently avoiding duplicates.

@elpraga File name does not seem a good indicator to me. First for the reason you mention and also I think there are good chances to have many files with the same name but a different content, specially when manipulating photos which are often automatically named "DSC000XX.jpg" for example.

[tacruc]

I would suggest a simple solution: A possibility to configure the path in which we look for pictures, and leave it to the user to not have duplicates in there.

Otherwise, before we go to implement something, we should thing about, why there are duplicates. Then we can check which cases we can solve with which solution.

[SignalWhisperer]

Since the goal (AFAIK) is to place them on a map and timeline, I doubt two non-identical photos will have the exact same location and timestamp. This could be worth looking into. The names can change and resizing the image while keeping EXIF data the same will make it seem like they are duplicate (but with a different modification date).

[justicenode]

I have the same problem but some of my images don't just show up twice but many times. Those are just 38 images but each of them gets displayed 4 times. So a total of 152 images gets shown on the map even though its actually 1/4 of that.

[adegans]

To scan for duplicates you can look at different meta/exif values -
Since it's a map, location is important and relevant. If 2 files pop up with the same original creation date and geo graphic location (GPS, long/lat) flag it for review.

A hash may work, but if the file name is different it already doesn't match...