Skip to content

Slow post-compilation file copy steps #3111

@orlp

Description

@orlp

A colleague of mine uses a Linux machine running on an Ext4 disk, and is noticing long (~8 seconds) post-compilation steps (with the latest version of maturin). Using profiling I've identified that the full multi-gigabyte artifact (Polars with debug symbols) is copied multiple times.

One copy occurs here:

// 2a. Install the artifact
debug!("Installing {} from {}", target.display(), source.display());
fs::copy(&source, &target).with_context(|| {

Another copy occurs here:

// Rename succeeded — we now own the only copy. Put a copy
// back at the original location for users who expect the
// artifact at the standard cargo output path. Skip if a
// new file already appeared (cargo / rust-analyzer rebuilt).
if artifact_path.exists() {
tracing::debug!(
"Skipping copy-back: {} was recreated by another process",
artifact_path.display()
);
} else if let Err(err) = reflink_or_copy(&new_artifact_path, artifact_path) {

Can these copies be replaced with renames / softlinks, or avoided altogether? Reflink is not enough, his filesystem (Ext4, which is really common) does not support them.

For example to avoid the second copy, I believe the "copy back" can just be a "rename back" step after the modification/reading process is complete. A complete file copy just to then be able to 'safely' mmap the file to read it seems rather silly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions