Deduplication from external library does not resolve

[cleoold]

The bug

After scanning images from my external library, the deduplication tool prompted me to review duplicate images. I stacked all of them. But when I refreshed the page, the same set of images appeared again for me to review.

Later on I found clicking "keep all" had the same issue.

Before:

Then I clicked keep all:

Refresh the page:

Reran the dedup job and the same issue persisted.

If I trashed photos then they would disappear.

The OS that Immich Server is running on

Ubuntu 24.04

Version of Immich Server

v1.125.7

Version of Immich Mobile App

NA

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

Identical to https://github.com/immich-app/immich/blob/v1.125.7/docker/docker-compose.yml with external library and userns_mode: "keep-id"

Your .env content

Identical to https://github.com/immich-app/immich/blob/v1.125.7/docker/example.env

Reproduction steps

1. Scan from external library
2. Resolve duplicates in one go
3. Refresh and same photos show up
   ...

Relevant log output

Additional information

At one time I held Shift+S to stack all duplicates and because it was too fast, the server ran into an error

ERROR [Api:ErrorInterceptor~xa7l1wf4] Unknown error: PostgresError: duplicate key value violates unique constraint "REL_91704e101438fd0653f582426d"
PostgresError: duplicate key value violates unique constraint "REL_91704e101438fd0653f582426d"
    at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:788:26)
    at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:474:6)
    at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
    at Socket.emit (node:events:524:28)
    at addChunk (node:internal/streams/readable:561:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
    at Readable.push (node:internal/streams/readable:392:5)
    at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)

I wonder if this could be related, but probably not. And if this was a user error, how should one remedy it?

[bo0tzz]

Please post your compose file anyways. Is your external library mounted readonly?

[masx200]

Query failed : {
  durationMs: 7.784904000000097,
  error: PostgresError: duplicate key value violates unique constraint "UQ_assets_owner_checksum"
      at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:788:26)
      at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:474:6)
      at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
      at Socket.emit (node:events:524:28)
      at addChunk (node:internal/streams/readable:561:12)
      at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
      at Readable.push (node:internal/streams/readable:392:5)
      at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
      at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
    severity_local: 'ERROR',
    severity: 'ERROR',
    code: '23505',
    detail: 'Key ("ownerId", checksum)=(ccf17505-b855-4d20-987b-7458906e5e95, \\xf12414cd43dfebfc2779200692a8b832f048ee69) already exists.',
    schema_name: 'public',
    table_name: 'assets',
    constraint_name: 'UQ_assets_owner_checksum',
    file: 'nbtinsert.c',
    line: '663',
    routine: '_bt_check_unique'
  },
  sql: 'insert into "assets" ("ownerId", "libraryId", "checksum", "originalPath", "deviceAssetId", "deviceId", "fileCreatedAt", "fileModifiedAt", "localDateTime", "type", "isFavorite", "isArchived", "duration", "isVisible", "originalFileName") values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15) returning *',
  params: [
    'ccf17505-b855-4d20-987b-7458906e5e95',
    null,
    <Buffer f1 24 14 cd 43 df eb fc 27 79 20 06 92 a8 b8 32 f0 48 ee 69>,
    'upload/upload/ccf17505-b855-4d20-987b-7458906e5e95/c4/f7/c4f7b1ed-cf6b-4ecf-bc35-e4a2f79621b9.jpg',
    '1000138900',
    '138b46c29a4df081f8c1e85835c84e5230ae1b11472467504de606fc9a9b561b',
    2025-02-01T06:03:53.000Z,
    2025-02-01T06:03:53.000Z,
    2025-02-01T06:03:53.000Z,
    'IMAGE',
    false,
    false,
    '0:00:00.000000',
    true,
    'Screenshot_2025-02-01-14-03-53-242_com.pwrd.hotta.laohu.jpg'
  ]
}
[Nest] 7  - 02/04/2025, 12:32:12 PM    WARN [Microservices:MetadataService] Asset d58ce7a7-596c-4db0-a860-ae8ca4e65dad has no time zone information
[Nest] 7  - 02/04/2025, 12:32:12 PM   ERROR [Microservices:JobService] Unable to run job handler (metadataExtraction/metadata-extraction): Error: ENOENT: no such file or directory, stat 'upload/upload/ccf17505-b855-4d20-987b-7458906e5e95/8b/69/8b6956e3-4841-42a5-a89e-680db2dafbac.jpg'
[Nest] 7  - 02/04/2025, 12:32:12 PM   ERROR [Microservices:JobService] Error: ENOENT: no such file or directory, stat 'upload/upload/ccf17505-b855-4d20-987b-7458906e5e95/8b/69/8b6956e3-4841-42a5-a89e-680db2dafbac.jpg'
    at async Object.stat (node:internal/fs/promises:1036:18)
    at async MetadataService.handleMetadataExtraction (/usr/src/app/dist/services/metadata.service.js:126:23)
    at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:148:28)
    at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:130:13)
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7  - 02/04/2025, 12:32:12 PM   ERROR [Microservices:JobService] Object:
{
  "id": "a66dd3e6-1fc7-41fe-86a2-c7eba080a317",
  "source": "upload"
}

Query failed : {
  durationMs: 26.027733000009903,
  error: PostgresError: duplicate key value violates unique constraint "UQ_assets_owner_checksum"
      at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:788:26)
      at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:474:6)
      at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
      at Socket.emit (node:events:524:28)
      at addChunk (node:internal/streams/readable:561:12)
      at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
      at Readable.push (node:internal/streams/readable:392:5)
      at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
      at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
    severity_local: 'ERROR',
    severity: 'ERROR',
    code: '23505',
    detail: 'Key ("ownerId", checksum)=(ccf17505-b855-4d20-987b-7458906e5e95, \\xaf4abee3006ce05a6edb4693d073a359b163efca) already exists.',
    schema_name: 'public',
    table_name: 'assets',
    constraint_name: 'UQ_assets_owner_checksum',
    file: 'nbtinsert.c',
    line: '663',
    routine: '_bt_check_unique'
  },
  sql: 'insert into "assets" ("ownerId", "libraryId", "checksum", "originalPath", "deviceAssetId", "deviceId", "fileCreatedAt", "fileModifiedAt", "localDateTime", "type", "isFavorite", "isArchived", "duration", "isVisible", "originalFileName") values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15) returning *',
  params: [
    'ccf17505-b855-4d20-987b-7458906e5e95',
    null,
    <Buffer af 4a be e3 00 6c e0 5a 6e db 46 93 d0 73 a3 59 b1 63 ef ca>,
    'upload/upload/ccf17505-b855-4d20-987b-7458906e5e95/aa/66/aa66ed16-7c18-4e53-a3e2-1a7d9e1b2033.jpg',
    '1000138899',
    '138b46c29a4df081f8c1e85835c84e5230ae1b11472467504de606fc9a9b561b',
    2025-02-01T06:03:49.000Z,
    2025-02-01T06:03:49.000Z,
    2025-02-01T06:03:49.000Z,
    'IMAGE',
    false,
    false,
    '0:00:00.000000',
    true,
    'Screenshot_2025-02-01-14-03-49-106_com.pwrd.hotta.laohu.jpg'
  ]
}
[Nest] 7  - 02/04/2025, 12:32:12 PM    WARN [Microservices:MetadataService] Asset 3beffa71-1e4f-40a7-97a4-5ab572d0228c has no time zone information

[TheSander562]

Same issue here : v1.125.7
But i have no external library and everything is working like removing is indeed removing them. But keeping them both then it will re-appear in the duplicate page after refresh.

Let me know what is needed to debug the issue.
Because no issue appears in the Immich log of the container.

[kjack9]

Add me to this as well. Stacking duplicates often doesn't work. Doing "Keep All" means that when I refresh the page, the same duplicate shows up again. This was working okay before, so the break appears to be relatively recent. Maybe around the time of the migration to Kysely?

EDIT: No external libraries invloved here either.

[Emil-G]

I see the same on [v1.125.7]. If i try to stack images down to zero, they all return when I refresh the page.

EDIT: No external libraries involved here either.

[elcobenoz]

I am also experiencing the same issue on v1.126.1

I am using external libraries, and I am able to delete them manually. When using the duplicate tool and then deleting the picture, as long as it was only 2 and I have deleted 1, it will not come back. However anything more than 2 or when I click keep all they will come back after a refresh.

[stevestrike]

I have the same issue as well on v1.125.7
Additionally, there are several formerly stacked items that I cannot delete at all. Immich will show you the image, but the delete permanently option fails, and the item is not visible in the trash.

No external libraries here

[SerkanBe]

Same here, with v1.126.1

docker-compose.txt

The two external libraries are mounted read only. Everything worked until my last update... which was from 1.123.1 to 1.126.1 I think, I'm not sure anymore.

I delete duplicates manually, that works fine. After refreshing the library they're gone. But stacking and keeping all duplicates doesn't seem to stick, they reappear after reloading the page.

[frazei]

version 1.126.1, same problem on internal library.
Stacking duplicates do not resolve the state of duplicated anymore.
To resolve it I have to manually empty the "duplicateId" field in the public.assets table in SQL.

[atsza661]

Same problem 1.126.1
I have only external library pictures. Stacking the duplicates, keeping 1, keeping all does nothing. I go to the main homepage, then dedup again, back to square 1

[kjack9]

Just confirmed that this still exists in v1.127.0