[Feature] Duplicate Rules

[johnny2678]

I have searched the existing feature requests to make sure this is not a duplicate request.

• Yes

The feature

Great job with the duplicate feature.

I have 75,000 dupes! So while I click away, I have had some time to think about how the "recommendations" might be more finely tuned.

ex. If photo resolution is the same and a HEIC and JPG show as dupes, immich usually recommends the JPG as the file size as bigger. However, I prefer the HEIC because of the compression and the live photo

another one - if an item exists in an external library and in the immich library and the resolutions are the same, I prefer the external library version.

I'm sure others have different opinions, but that's why it would be nice to be able to define our own rules.

Thanks for the project!

Platform

• Server
• Web
• Mobile

[marciopamplona]

Another very useful feature is enable/disable video dedupe. I have a lot of false positives due to the AI model.
It could be very useful to have a different configuration for detecting videos with a more strict rule (different maximum distance).

[radh21301]

Here are some more points to think about. It would be great if we could set a list of rules and prioritize them. And each user can have a different priority - basically, an option to reorder them as needed.

The following are some rules I could think of:

1. GPS Available (Yes/No)
2. Exif Tag Available (Yes/No).
   • DateTimeOriginal:
     • I have this criterion because there could be pictures with the metadata stripped, say a WhatsApp image. These images that don't have the metadata will most certainly show up in an odd location in the timeline. I always prefer images with the right metadata.
   • CreateDate - This is for videos
   • Starred
   • The user could choose any tag from a certain pool of most commonly used exif tags
3. Image/Video size (Small/Large)
4. Prefer external library OR immich internal library
5. Prefer Immich Favorited Asset

The list can be refined based on what people think.

[premiumcola]

This would be awesome :)! thanks for your detailled thoughts on this topic

[Rundervinkje]

I'd like to add-on to your list with:

1. Duplicates (same name). Like D001_img2.jpg and D001_img2.jpg in a different folder.
2. Duplicates (partly same name). Meaning, it could even account for partial matches, like recognizing D001_img2.jpg and attachment_D001_img2.jpg as the same image despite slight differences in the name.
3. Duplicates per Maximum Detection Distance (MDD). Meaning, run the duplicate search job with different MDD's and sift through them at a later date.

[cookie-monster1649]

I'd also love to add in

• Is in album (yes/no)

[tairasu]

this is fairly easy to implement since the suggestDuplicateByFileSize object is just sorting each iteration by file size and selects the last object (biggest file). instead you could first check if there is one or more HEIC file and then ignore JPGs.

obviously this should be optional. and it would need a UI for user preferences.

[Ccheese4]

Also been thinking about this as I go through my 13,000 duplicates. Would at least reduce my clicks so I don't have to switch the heic and the jpg auto selection.

[Ccheese4]

also the ability to skip a duplicate pairing would be nice and come back to it later.

[maicss]

I really need this. Do you think we should begin a new conversation about this?

[maicss]

I created a new discussion about this. #17566

[chrissi5120]

I effed up big time and uploaded from my wifes "recent" folder which seems to be populated by google photos?

We now got like 8k photos and video as duplicates with/without GPS.

being able to select by GPS would save me tears.

[githubsean]

What I would like is if some information from the deleted duplicate is copied over to the kept duplicate. A good example is folders.

Say there are two photos. One is low resolution (from Google Takeout), one is high resolution (original manually copied off phone). The low resolution version is in an album, the high resolution one is not in any album. The duplicate checker will correctly suggest to keep the high resolution version, but the effect is to remove this photo from the album the low resolution version was in.

Similar issues might happen with EXIF dates or GPS coordinates - the photo being discarded might have this detail, but the kept photo might not (though given the example above, the "original" will likely have all this). My main issue is with keeping album information.

[njspix]

Wanted to bump this - file size is a decent criteria but might end up selecting for poor compression algorithms (JPEG instead of HEIC, for example) rather than actually more data. For a preliminary go, how about sorting by (decompressed) pixel count rather than size? I believe it's possible to get the 'real' pixel dimensions of HEIF/HEIC and JPEG images without decoding them.

[JasonMAnderson]

A dilemma I run into is camera backup from my phone. I want my photos from my phone immediately available, so I turn-on camera backup. But I also manually copy my camera photos to my external library every month because my external library is my "single source of truth" as it's accessible outside of Immich. I need a rule to always delete the duplicate in the camera backup if a dupe exists in external library.

[premiumcola]

Yeaaah thanks for the comment on this aspect:
I would also like to set my "external libary source" as first priority - and keep all fotos once in there but delete the duplicates on all other sources.

[jcisio]

You can also move your uploaded photos (you should use a time based template to make it easy) to the external library, instead of moving your camera photos?

[rwlove]

Prefer to keep photos already in an album.

[nicolas-albert]

Hi. I'm a Google Photo use from years. I have my home Immich server and I push to it my 450 Go of media from Google Takeout.
Each photos edited for cropping or color are duplicated (+10000).
Also currently saved from my phone (+4000 to save) already here with the Takeout import and bigger.
But I prefer smaller (cropped and better compression) and already in album.
I hope this feature will be selected 👍

[jcisio]

For external library we should be able to mark some directories as "preferred source". For uploaded photos: we should be able to mark some albums as "preferred source". If this is implemented then it would solve your problem.

[andredatele]

Another one bumping this feature. I'm using immich-go to import from Google Photos, but the dupe detection is not perfect, leaving most of the work to be done in Immich itself. Would be great to have the ability to prefer assets already in albums.

[nicolas-albert]

immichDuplicate.html.zip
I created a bookmarklet to help me choose simple cases.
It searches for French words and needs to be adapted.
It auto-validates every second and stops on ambiguous cases (until pressed again).
It's better to understand it before running it.

[killforby]

Bonjour comment exécuter le bookmarket ?

[nicolas-albert]

Bonjour,
De souvenir, il faut l'ajouter dans la barre de favoris. Une fois sur la page de recherche de duplication sur Immich, on clique dessus et ça va choisir une image suivant des critères (dont je ne me souviens plus) et virer les autres, puis passer à la suivante jusqu'à tomber sur un cas que le script ne gère pas.

[killforby]

D'accord je vais voir ça, vraiment dommage qu'il intègre pas la suppression par étiquette dans immich directement

[killforby]

Et merci pour ta réponse

[shanep2300]

I want the option to be able to 'skip' and 'ignore'. Currently I have multiple albums that are of different resolutions but same file name. IE: WebRes, FullRes, Raw. I don't want any of those removed.

Maybe be able to see which albums the dupes are in and then uncheck it from looking in there anymore?

[PANICBUTT0N]

An "undo" button and the ability to not classify assets as dupes if they have different filenames (e.g. IMG_1234 and IMG_1235) would be nice as well.

[GiorsAltF4]

yeah this is literally all I'd need, Ctrl-Z to undo and a way to filter the duplicates type found (so for example if they have different names they don't appear in the filtered view)!

I wonder if there's any way to bump this suggestion

[punqbund]

As mentioned initially, regarding de-duplication: Select one or more preferred libraries for resolving duplicates. Images from these libraries will be selected to keep (or will be automatically kept in case of resolving all duplicates), and images from all other libraries will be deleted unless you define additional criteria.

[chrissi5120]

seconded. The focus should be exactly here.

The simplest situations like checking duplicates between multiple libraries OR albums should be implemented first.
Next priority could be quick-wins like whatever above is also straightforward to implement.

Another big focus should be in UI/UX, the more you can fix here, the better.

[hday0]

As someone who is having to go through 8000+ duplicates found by the utility manually, this would seriously be a god send. I want to add a rule to prioritize auto selection of a date timestamp instead of a generic "img_2492.jpg"

[plsander]

I would find ability to review dups associated with images in a particular album useful.

[mmomjian]

#10537

[sebastiencastermans]

Hey everyone!
I made a Python script to help clean up duplicate photos and videos on Immich servers. It smartly prioritizes Apple’s HEIC files and offers a dry-run mode so you can preview before deleting anything.
If you’re looking to tidy your Immich library safely and efficiently, feel free to check it out! Happy to get feedback or ideas for improvements.
I hope it will be usefull to you :)

[joekueng]

[bo0tzz]

@joekueng mangling things through an LLM isn't very useful, it's made up a bunch of things that aren't implemented so the basis that your proposed PR would build on doesn't exist.

[joekueng]

Hello everyone, I have implemented this to prioritize photos from the external library.

Obviously, I still need to test it thoroughly, but does it make sense as a first approach? Also in terms of the UI.

What do you think of this approach?

[jjmonester]

Tbh, I have the opposite problem. I want to prioritise the internal library. Due to re-uploads through the app, and adding to albums, internal photos are the ones I want to maintain.

It'd solve the problem for me if there was a toggle to prefer internal or external libraries. Having this toggle between the two would be a godsend.

[joekueng]

@jjmonester I can implement it, thanks for the idea. Do you have any ideas on how to do it for the UI?

[jjmonester]

I haven't thought deeply about the intention of this whole tool, but when I think of the users of Immich, generally everyone has slightly varied setups and are technical power users (at least those using the deduplication tool). I think we'd need to account for this.

I'd assume then this is probably one of many filters that users would want to set to change how the algorithm works for default deduping.

For the simplicity of this implementation, I'd create a settings dropdown with defaults set to how Immich is now as 'Default'. This setting would then effectively override the logic set by the OG devs, so I'd want to know explicitly if my settings are changed when using the tool.

So some rough requirements

• A Settings dropdown button for the dedupe utility
• A dropdown for this setting. Could be something like 'Prefer assets in: Default, Internal library, External Library'
• Create some kind of visible prompt, a colour change perhaps on the settings dropdown button, if any settings deviate from the Immich defaults. This is to signal to users that a change from defaults is currently in effect.
• Create a confirmation prompt (modal) on pressing 'Deduplicate All' confirming that the settings have been altered from the defaults.

TL;dr

• Maintain clear defaults for standard Immich behaviour
• Allow for future customisation and detailed utility changes
• Warn users for changes in place away from default.

I'm not a designer so can't help much with that sorry! I'm a product weirdo myself 😊

[jjmonester]

To just add on, there's so much discussion above about other variations and extra metadata to add to help sort. I think we start simple as above and go from there.

If a dedupe tool is effectively a list of ordered rules like @radh21301 , that could be a good future state to move towards. A rule builder of sorts where one sets the logic of the underlying system so you can make it work for your specific use case. But we'd need the metadata exposed and usable first.

Personally, I mainly care about maintaining any albums / favourites and additional metadata added to images. If there was a way to ensure that info is maintained regardless of asset, that would be great too.

[joekueng]

Looking at the other utilities, I don't think they are mature tools. There should be a section in the settings to change the parameters.

However, I believe that covering the three cases:

• prefer external
• default
• prefer internal

is necessary in order to use the deduplication tool.

In the future, it will certainly be necessary to change the logic and implement new settings and functions as discussed in #10537.

[joekueng]

I changed the way here you select the preferred libbrary, now you can choose between internal external or leave default.

The logic prior to preference has not been changed.

What do you think of this implementation? @jjmonester @bo0tzz

[zurkiyeh]

Looks great thanks a lot!

[soylentseth]

that is perfect for my needs. thank you sir

[The-Croz]

Could this be supplemented with an option to prefer HEIC images over JPEG? Have been looking for this option but have not found it anywhere.

[joekueng]

@The-Croz
I'm waiting for the pull request to be approved, and then I'll definitely try implementing the HEIC preference.

[The-Croz]

@joekueng that would be fantastic. Appreciate your work on this. Hopefully they merge your PR soon!

[agentgodzilla]

The critical duplication "rule" for me that is missing is the ability to exclude certain albums (id take an option to exclude all of them) from comparison. For example I have my wedding photos in my immich which due to the nature of them I have ~1000 items that need deduplicating, but 99% of it is photos I definitely want to keep.

[cosmickylenny]

any news on when this will be available?

[bo0tzz]

https://justinmayer.com/posts/any-updates/

[dhlavaty]

Duplicate Tool could not only delete photos, but also merge META information.

I have a lot of duplicates in my library, and it often happens that a photo I want to keep because of its image, does not have all the EXIF data that the photo I am deleting has. This would add the EXIF information from the deleted photo to the one I want to keep.

It would be great if we could consider the Duplicate tool as a merge tool — something like Merge Contacts in your Contacts app.

[punqbund]

In my opinion, if there is a need to store and combine metadata from a deleted duplicate, this should only be done in the database. The link between the metadata of the deleted duplicate and the file you are keeping should also only exist in the database. Writing metadata to an image would be file editing. In my opinion, editing files goes beyond the scope of a photo library, at least in the short term. This is my experience based on the development of DAM systems. If you change the original, the question of undoing the changes naturally arises, which in turn means version control, and so on. Such things can be planned for the long term, but for now, you can think about storing and merging metadata from a deleted duplicate with a link to the preserved file at the database level.

[punqbund]

And keep in mind, that removing duplicates doesn't always mean deleting them, but rather removing the database entries.

[dhlavaty]

Agree of course. No image file manipulation in any way.

[antonio-ivanovski]

Similar to the proposal from @dhlavaty, often times there is duplicate with lower quality that has ended up in an album but the better one is not in album. This is often the case when someone has performed Google Photos upload and also managed to get the original photos uploaded too.

Proposing option when a deleted photo was in a album, there to be option to be replaced with some of the photos that were kept.

[Tim-Dooley]

I hope this feature gets implemented soon with some options.
I have many duplicate files where the lower quality one is in the album.
Personally I would like the following order of operations for duplicates

1. highest resolution first
2. heif/heic over JPG
3. Smaller files over larger files
   As an add on, merge album links.

This would mean if the higher resolution file is jpg then discard the heic
If resolution is equal between a jpg and heic discard the jpg.
If resolution is equal & file extensions match, keep the smaller.

[MacPaKKo]

I am removing duplicates.
I'd like to have options to sort proposed duplicates by date, so photos will follow a logical timeline, or by size (movies for sure come first), so I'll quickly clean space on my disk (following delete rules).
Thanks