Successfully sync an iCloud Library to self hosted Immich in an airtight env.

[smileBeda]

How I set up my docker to avoid opening ports

Prerequisites:

• apart of what immich needs anyway,
• a docker network (in my case called npm_proxy)
• an instance of NGINX Proxy Manager (NPM)
• in my case, an instance of Technitium to handle local DNS but there are other ways to do that

note that I am on a Lattepanda with intel chip, some settings might differ on your machine specially related to ML and Accel

docker-compose.yml:

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
      file: hwaccel.transcoding.yml
      service: vaapi # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/data
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    #ports:
     # - '2283:2283'
    depends_on:
      - redis
      - database
    restart: unless-stopped
    healthcheck:
      disable: false
    networks:
      npm_proxy:
        ipv4_address: 192.168.97.19
      immich_internal:

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-openvino
    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
      file: hwaccel.ml.yml
      service: openvino # set to one of [armnn, cuda, rocm, openvino, openvino-wsl, rknn] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: unless-stopped
    healthcheck:
      disable: false
    networks:
      - immich_internal

  redis:
    container_name: immich_redis
    image: docker.io/valkey/valkey:8-bookworm@sha256:facc1d2c3462975c34e10fccb167bfa92b0e0dbd992fc282c29a61c3243afb11
    healthcheck:
      test: redis-cli ping || exit 1
    restart: unless-stopped
    networks:
      - immich_internal

  database:
    container_name: immich_postgres
    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:32324a2f41df5de9efe1af166b7008c3f55646f8d0e00d9550c16c9822366b4a
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
      # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs
      # DB_STORAGE_TYPE: 'HDD'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    shm_size: 128mb
    restart: unless-stopped
    networks:
      - immich_internal

volumes:
  model-cache:

networks:
  npm_proxy:
    external: true
  immich_internal:
    driver: bridge

.env

UPLOAD_LOCATION=/path-to/library
DB_DATA_LOCATION=/path-to/postgres
IMMICH_VERSION=v1.139.3
DB_PASSWORD=db_password
DB_USERNAME=db_username
DB_DATABASE_NAME=immich

hwaccel.ml.yml

services:

  openvino:
    devices:
      - /dev/dri:/dev/dri

hwaccel.transcoding.yml

services:
  cpu: {}

  nvenc:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu
                - compute
                - video

  quicksync:
    devices:
      - /dev/dri:/dev/dri

  rkmpp:
    security_opt: # enables full access to /sys and /proc, still far better than privileged: true
      - systempaths=unconfined
      - apparmor=unconfined
    group_add:
      - video
    devices:
      - /dev/rga:/dev/rga
      - /dev/dri:/dev/dri
      - /dev/dma_heap:/dev/dma_heap
      - /dev/mpp_service:/dev/mpp_service
      #- /dev/mali0:/dev/mali0 # only required to enable OpenCL-accelerated HDR -> SDR tonemapping
    volumes:
      #- /etc/OpenCL:/etc/OpenCL:ro # only required to enable OpenCL-accelerated HDR -> SDR tonemapping
      #- /usr/lib/aarch64-linux-gnu/libmali.so.1:/usr/lib/aarch64-linux-gnu/libmali.so.1:ro # only required to enable OpenCL-accelerated HDR -> SDR tonemapping

  vaapi:
    devices:
      - /dev/dri:/dev/dri

  vaapi-wsl: # use this for VAAPI if you're running Immich in WSL2
    devices:
      - /dev/dri:/dev/dri
    volumes:
      - /usr/lib/wsl:/usr/lib/wsl
    environment:
      - LIBVA_DRIVER_NAME=d3d12

Spin up with docker compose up -d and setup your instance (admin user and other users). Make sure to set VAAPI in the video transcoding settings on the admin side. Make sure Enable face import is enabled.

In your NPM entry for the URL under which you run immich, make sure to add something like this to the advanced tab so to allow larger uploads:

client_max_body_size 20G;
proxy_request_buffering off;
proxy_max_temp_file_size 0;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;

Point the host to the IP you set in docker for it, and port 2283

---

Actual Import process

• make sure all your iCloud assets are on your machine (desktop). This is done by open the Photos app, go to Photos > Settings (or Preferences in older macOS versions), select the iCloud tab, and then choose the Download originals to this Mac. If you never did this before, let it first download all things, before you proceed.

• Install (ideally in a venv) osxphotos python library and run the following command (adapt as per the manpage, but be sure to understand each option!): osxphotos export "/path-to/icloud-backup" --skip-bursts --not-shared --not-syndicated --not-shared-moment --not-shared-library --filename "{id}" --ignore-date-modified -V --sidecar xmp . NOTE, this skips bursts because I had almost none, but if you have and want them, remove that flag. It also skips shared folders and shared albums because A) if you shared items they are in the main timeline anyway and B) anything shared to you you likely do not want in the library. Finally, this does sync hidden folder of iCloud (the mobile app is not able to see that one album so be prepared for some number mismatch). To dry run, add --dry-run.
• Install immich CLI and connect to your server. If you have self signed certificate - before log in, do export NODE_EXTRA_CA_CERTS=$HOME/your-cert.pem
• run immich upload /path-to/icloud-backup. For dry run, do --dry-run. This takes about 2 hours on 14k assets using LAN. Let it finish.
• Once done, make sure all jobs are run at least once on the admin side of immich
• Done that, connect your mobile app, switch to beta timeline, do NOT sync any album yet. Let the hashing process finish first. Restart the app several times even if it says it is done, somehow it does not finish all hashing in the first attempt. Once you are sure all assets are hashed proceed.
• Choose to sync recents (all) in the app, and enable backup. If you have NO outliers and NO items left in your main timeline that where also in hidden folder and otherwise did export with osxphotos exactly what the app sees too (that includes bursts etc) then it will work just fine. It might be that you get a few outliers that are re-uploaded, for example if you added --skip-edited to the export, or did not export bursts but have bursts in your library, those will be re-uploaded and basically be duplicates. Chances are if that happens, those will appear in immich without meta data and thus easily findable in the "most recent" items. Or, use the utility to de-duplicate.

This way, I synced 14k photos approx (some videos being 20gb large) in about 2 days.
The entire process also works without osxphotos/cli, but it takes ages, approximately on a LAN it syncs on my end 1000 pics of medium size per day.
Note that if you do NOT use osxphotos chances are a HIGH number of your assets will miss metadata. Somehow, iCloud Photos does a strange thing with locations and other metadata that the app cannot understand - however osxphotos, when exporting to the sidecar files, does understand and put it in a immicc-compatible format. Don't ask me the technical details on it, I do not understand them, but I know there where hundreds of dislocated images in my lib after a app-only import and that was not the case with the osxphotos sidecar/cli import.

Hope this helps anyone.

[the80hz]

Hi, thanks for a great job. I want to clarify a couple of points.

First, you said that

if you do NOT use osxphotos chances are a HIGH number of your assets will miss metadata

but when I export files unchanged from the macOS Photos application itself, xmp sidecars are also created there. Or they really may not store some important information (adjusted datetime, or manual geotag).

Secondly, I have quite a lot of photos taken on the iPhone, but there was incorrect information about the location, and I manually corrected it through macOS Photos. Does this affect the creation of duplicates in Immich when syncing already via an iOS device?
I remember when I synced the photo just through the iOS app, the corrected location was not uploaded.

And third, I have ~800 GB of photos and videos, that is, 42,000 items in iCloud. Let's say osxphotos is a mandatory part of the library migration, is it possible to split the synchronization anyway, at least for each year? I don't have a drive big enough to hold two copies of the library.

[smileBeda]

but when I export files unchanged from the macOS Photos application itself, xmp sidecars are also created there. Or they really may not store some important information (adjusted datetime, or manual geotag).

Problem was on my end that exporting numbers of items from Photos app using Mac's export feature... it crashed, even if I exported a thousands a time. I suspect it was due to some file too large in the assets, but can't tell.
I never managed to actually try a set of "Mac" exported items due to that. It might work!

Does this affect the creation of duplicates in Immich when syncing already via an iOS device?
I remember when I synced the photo just through the iOS app, the corrected location was not uploaded.

Probably, but I had no such items so you have to try.
"the correct locations was not uploaded" is exactly what I saw too and it is def solved with osxphotos sidecars. Something that tool does different than immich, and it does it right. As long you export the sidecar and not write to file the metadata you should be fine. Keep in mind that edited assets in OSX are stored along the original so make sure to not add --skip-edited to your export. You'll unfortunately end up with original asset without metadata (I suspect) but at least you've your edit correct. I think osxphotos even has a --skip-original but you've to check the man page.

And third, I have ~800 GB of photos and videos, that is, 42,000 items in iCloud.

That is a lot lol - I had 105GB, 14k assets. Yes, splitting sync is possible both ways:

• osxphotos has an --update flag so you do not actually need to export all at once but can "update" existing exports with new bulks. --limit LIMIT might help you, or --added-after DATE and --added-before DATE. Note thou that the --update flag probably only works when you keep the media in the drive, not delete them, so the update flag in this case wouldn't help (but the date filters certainly will)
• immich CLI has no "limit" as far I know but it does manage to detect duplicates before upload so you can just abort and restart it if you want, although I do not see an issue with 800GB, it just will take much longer (approx 16 hours probably), and of course, re-run it many times on an updated bulk, it will smartly recognize if anything was already uploaded.

you wrote the --filename "{id}" parameter

I had some issues with immich recognizing the sidecar coupled to the files and suspected it due to the many (n) I had in filename (n being a number) so I needed something unique. But honestly, that might be due to me having done it wrong, it is worth trying without this flag on a small subset to see if it works anyway (It was a bit of a rollercoaster until I got things working so I did not want to make new experiments the 10th Time I reset the whole immich database lol)

[the80hz]

After thinking about how osxphotos works, I came up with an export script that seems right for my needs:

1. Export by year
   Instead of running one big export for the entire library, I split it into chunks by year using:
   --from-date 2015-01-01 --to-date 2015-12-31

2. Incremental retries for missing files
   On the first run, some files go into missing.
   Then I re-run the same command with --update → gradually all missing files get downloaded.

3. Sidecar XMP
   I use --sidecar xmp --sidecar-drop-ext, this works well with Live Photos.

4. Reports
   I added --report export_{something}.csv --append to keep track of what was exported and what failed.

5. Directory structure
   For me, the most critical part: files are organized as {id}/{original_name} which keeps everything unique and tidy.

6. Preserve capture date with --touch-file
   By default, exported files will have the filesystem modification date set to the time of export.
   With --touch-file, osxphotos sets the file’s modification time (mtime) to the actual photo capture date.
   This ensures that Immich recognize the correct creation date, even if they fall back to file timestamps.

osxphotos export "/Volumes/PhotosBackup/icloud-2015" \
  --from-date 2015-01-01 --to-date 2015-12-31 \
  --directory "{id}" --filename "{original_name}" \
  --ignore-date-modified \
  --sidecar xmp --sidecar-drop-ext \
  --touch-file \
  --download-missing --use-photokit --retry 3 \
  --exportdb "$HOME/osxphotos-export-2015.db" --update \
  --report export_2015.csv --append -V

[smileBeda]

I think with sidecar it's not needed to add touch-file.
On the opposite be careful: as I understood touch-file it alters meta data INSIDE THE MEDIA. As such hashing won't be identical anymore later when immich hashes medias and you could end up with 1:1 duplicates

[the80hz]

--touch-file doesn’t mess with EXIF at all, it just tweaks the file’s timestamps. So the bytes (and the hash) stay the same. Only --exiftool actually changes the media.

[skatsubo]

I had some issues with immich recognizing the sidecar coupled to the files

Maybe related: there is currently an edge case with sidecar handling, will be fixed in #21312