fix(server): don't reimport files more than once #16375

etnoy · 2025-02-27T09:17:50Z

Whenever a library asset is changed it will be reimported during the next scan. However, the metadata extractor puts the incorrect date in fileModifiedAt, so that the file will be reimported on the next scan, and the next, and the next, no matter if it was changed or not.

This fix puts the correct date in fileModifiedAt so that this won't happen. fileCreatedAt should be derived from the file, and not exif data.

Yes, this is a bug from my previous PR #16225...hope this is correct now.

mertalev · 2025-02-27T09:35:08Z

server/src/services/metadata.service.ts

      localDateTime,
      fileCreatedAt: exifData.dateTimeOriginal ?? undefined,
-      fileModifiedAt: exifData.modifyDate ?? undefined,
+      fileModifiedAt: asset.fileModifiedAt,


I'd prefer if you used a fileModifiedAt variable instead of assigning to asset.

I'd prefer if you used a fileModifiedAt variable instead of assigning to asset.

This is much less trivial than might first seen. It must be assigned to asset in order for getDates to work, and assigning a new variable only makes it more complex.

Rewriting the metadata extraction feature is on my list as it's getting rather messy, but that's not in scope for this PR.

mertalev · 2025-02-27T10:43:48Z

server/src/services/metadata.service.ts

      asset.fileModifiedAt = stats.mtime;
    }

+    const fileModifiedAt = asset.fileModifiedAt;


No, I mean that instead of asset.fileModifiedAt = stats.mtime; followed by const fileModifiedAt = asset.fileModifiedAt;, I'd prefer const fileModifiedAt = stats.mtime; (or inlining to fileModifiedAt: stats.mtime)

There is no other example in the metadata service from what I can see where the asset is modified rather than creating a variable.

Ah, I see. I inlined it as per your suggestion, let me know what you think.

alextran1502 · 2025-02-27T15:30:51Z

Is this something we can use Medium Tests for?

etnoy · 2025-02-27T15:44:01Z

Is this something we can use Medium Tests for?

I haven't played with them enough to know yet, but it's high on my todo list to migrate as many library tests as possible to that.

Let's get this fix in 1.127.1 first.

alextran1502 · 2025-02-27T16:17:13Z

Can you help fix the failed tests?

etnoy · 2025-02-27T16:26:28Z

Can you help fix the failed tests?

Sure, I didn't know they failed before I left home. I'll do that in a few hours when I'm back at the computer. If you are short on time the fix should be easy, just modify the mocks and asserts a little

* fix(server) don't reimport files more than once * fix: test --------- Co-authored-by: Alex Tran <[email protected]>

scrapix · 2025-03-05T18:45:29Z

I think the adaption is not sufficient... I'm currently on v1.128

I've changed access rights and owner with chmod / chown. (File Inode Change Date/Time)
Now all assets (+100.000) are being re-importet including usage of all services

thumbnail creation
face tagging
what not...

although nothing about the picture itself changed, each asset is being re-imported.

Solution proposal
If no other option is going to be found, an image hash comparison should be done prior to starting all service (like thumbnai re-generation, face tagging etc.). To prevent unnecessary processing.


immich_server            | [Nest] 7  - 03/05/2025, 6:30:37 PM   DEBUG [Microservices:LibraryService] Asset was offline or modified, updating asset record /usr/src/app/photos/etomy/Mali/2020/2020-04-13/20200413_170744.jpg
immich_server            | [Nest] 7  - 03/05/2025, 6:30:37 PM   DEBUG [Microservices:LibraryService] Asset was offline or modified, updating asset record /usr/src/app/photos/etomy/Mali/2020/2020-04-13/20200413_170748.jpg
immich_server            | [Nest] 7  - 03/05/2025, 6:30:37 PM   DEBUG [Microservices:LibraryService] Asset was modified, queuing 
...
... 200.000 more ...
...

❯ exiftool Mali/2020/2020-04-13/20200413_170744.jpg

ExifTool Version Number         : 12.41
File Name                       : 20200413_170744.jpg
Directory                       : Mali/2020/2020-04-13
File Size                       : 4.6 MiB
File Modification Date/Time     : 2020:10:27 21:01:33+00:00
File Access Date/Time           : 2023:10:17 00:57:17+00:00
File Inode Change Date/Time     : 2025:03:02 09:20:26+00:00
File Permissions                : -rwxrwxrwx
File Type                       : JPEG
File Type Extension             : jpg
MIME Type                       : image/jpeg

etnoy · 2025-03-05T18:49:47Z

I think the adaption is not sufficient... I'm currently on v1.128

I've changed access rights and owner with chmod / chown. (File Inode Change Date/Time) Now all assets (+100.000) are being re-importet including usage of all services

thumbnail creation

face tagging

what not...

although nothing about the picture itself changed, each asset is being re-imported.

Solution proposal If no other option is going to be found, an image hash comparison should be done prior to starting all service (like thumbnai re-generation, face tagging etc.). To prevent unnecessary processing.
immich_server            | [Nest] 7  - 03/05/2025, 6:30:37 PM   DEBUG [Microservices:LibraryService] Asset was offline or modified, updating asset record /usr/src/app/photos/etomy/Mali/2020/2020-04-13/20200413_170744.jpg
immich_server            | [Nest] 7  - 03/05/2025, 6:30:37 PM   DEBUG [Microservices:LibraryService] Asset was offline or modified, updating asset record /usr/src/app/photos/etomy/Mali/2020/2020-04-13/20200413_170748.jpg
immich_server            | [Nest] 7  - 03/05/2025, 6:30:37 PM   DEBUG [Microservices:LibraryService] Asset was modified, queuing 
...
... 200.000 more ...
...
❯ exiftool Mali/2020/2020-04-13/20200413_170744.jpg

ExifTool Version Number         : 12.41
File Name                       : 20200413_170744.jpg
Directory                       : Mali/2020/2020-04-13
File Size                       : 4.6 MiB
File Modification Date/Time     : 2020:10:27 21:01:33+00:00
File Access Date/Time           : 2023:10:17 00:57:17+00:00
File Inode Change Date/Time     : 2025:03:02 09:20:26+00:00
File Permissions                : -rwxrwxrwx
File Type                       : JPEG
File Type Extension             : jpg
MIME Type                       : image/jpeg

This is by design. You changed the files, and the mtime was updated. Then it will be reimported.

scrapix · 2025-03-05T20:53:20Z

@etnoy @alextran1502 do you think it's worth opening a Feature Request for my solution proposal to make re-importing more efficent?

Solution proposal

On image import create min 2 hashes and save them to the database:
1. of the complete image file
2. just of the image / video part of the file

If a file change is being identified compare if image / video part hash has changed compared to the changed file
- if no change in image part, just run the metadata job
- if change also run jobs like thumb re-creation, face identification etc..

Goal: prevent unnecessary processing. if no changes happen to content

etnoy · 2025-03-05T22:12:59Z

@etnoy @alextran1502 do you think it's worth opening a Feature Request for my solution proposal to make re-importing more efficent?

Solution proposal

On image import create min 2 hashes and save them to the database:

of the complete image file

just of the image / video part of the file

If a file change is being identified compare if image / video part hash has changed compared to the changed file - if no change in image part, just run the metadata job - if change also run jobs like thumb re-creation, face identification etc..

Goal: prevent unnecessary processing. if no changes happen to content

No, I don't think so. By design, we want to allow multiple files with identical hashes with different paths within an external library

etnoy added the changelog:bugfix label Feb 27, 2025

etnoy requested a review from danieldietzler as a code owner February 27, 2025 09:17

github-actions bot added the 🗄️server label Feb 27, 2025

etnoy changed the title ~~fix(server) don't reimport files more than once~~ fix(server): don't reimport files more than once Feb 27, 2025

etnoy force-pushed the fix/dont-reimport-again branch from 23b6e31 to 4334c2a Compare February 27, 2025 09:24

mertalev reviewed Feb 27, 2025

View reviewed changes

etnoy force-pushed the fix/dont-reimport-again branch from 4334c2a to bf53e5a Compare February 27, 2025 10:32

mertalev reviewed Feb 27, 2025

View reviewed changes

fix(server) don't reimport files more than once

5fc4318

etnoy force-pushed the fix/dont-reimport-again branch from bf53e5a to 5fc4318 Compare February 27, 2025 11:20

fix: test

14db60e

alextran1502 approved these changes Feb 27, 2025

View reviewed changes

alextran1502 enabled auto-merge (squash) February 27, 2025 16:42

alextran1502 merged commit d20e2e2 into main Feb 27, 2025
39 checks passed

alextran1502 deleted the fix/dont-reimport-again branch February 27, 2025 16:45

renovate bot mentioned this pull request Apr 23, 2025

feat(container): update immich-app raypappa/homelab#485

Merged

1 task

Uh oh!

fix(server): don't reimport files more than once #16375

fix(server): don't reimport files more than once #16375

Uh oh!

Conversation

etnoy commented Feb 27, 2025

Uh oh!

mertalev Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

etnoy Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

mertalev Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mertalev Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

etnoy Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

alextran1502 commented Feb 27, 2025

Uh oh!

etnoy commented Feb 27, 2025

Uh oh!

alextran1502 commented Feb 27, 2025

Uh oh!

etnoy commented Feb 27, 2025

Uh oh!

Uh oh!

scrapix commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

etnoy commented Mar 5, 2025

Uh oh!

scrapix commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

etnoy commented Mar 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mertalev Feb 27, 2025 •

edited

Loading

scrapix commented Mar 5, 2025 •

edited

Loading

scrapix commented Mar 5, 2025 •

edited

Loading