Skip to content

Match Python cache fallback when symlinks are unavailable#172

Open
i386 wants to merge 1 commit into
huggingface:mainfrom
Mesh-LLM:mesh/windows-symlink-fallback
Open

Match Python cache fallback when symlinks are unavailable#172
i386 wants to merge 1 commit into
huggingface:mainfrom
Mesh-LLM:mesh/windows-symlink-fallback

Conversation

@i386

@i386 i386 commented May 27, 2026

Copy link
Copy Markdown

Summary

Mirror Python huggingface_hub cache behavior for platforms where snapshot symlinks cannot be created, especially Windows without Developer Mode/admin symlink privileges.

  • Try to create the snapshot pointer as a symlink on Windows instead of unconditionally copying.
  • If symlink creation fails, move a freshly downloaded blob into the snapshot path so a new download does not leave both blobs/<etag> and snapshots/<commit>/<filename> as full-size files.
  • Continue copying existing blobs into snapshots when symlinks are unavailable, because an existing blob may already be referenced by other snapshots.
  • Reuse an existing regular snapshot file after metadata lookup when the cache is already in degraded no-symlink mode and the blob path is absent.

Python reference

This follows the Python huggingface_hub cache fallback behavior:

Why

The Python implementation avoids keeping both the extensionless blob and the named snapshot file for a fresh download when symlinks are unavailable. The current Rust implementation always copies on Windows, which can double disk usage for large model files.

Tests

Added focused regression coverage for the degraded no-symlink path:

  • pointer_fallback_moves_new_blob verifies fresh blobs are moved to the snapshot path and the blob path is removed.
  • pointer_fallback_copies_existing_blob verifies existing blobs are copied so other snapshot references remain valid.
  • cache_download_reuses_regular_snapshot_when_blob_is_absent verifies a cache already in degraded no-symlink mode reuses the regular snapshot file instead of downloading again.

Validation

  • cargo fmt --check with nightly rustfmt
  • cargo test -p hf-hub --lib (167 passed)
  • cargo clippy -p hf-hub --all-features -- -D warnings
  • git diff --check

@i386 i386 marked this pull request as ready for review May 27, 2026 10:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant