A colleague of mine uses a Linux machine running on an Ext4 disk, and is noticing long (~8 seconds) post-compilation steps (with the latest version of maturin). Using profiling I've identified that the full multi-gigabyte artifact (Polars with debug symbols) is copied multiple times.
One copy occurs here:
|
// 2a. Install the artifact |
|
debug!("Installing {} from {}", target.display(), source.display()); |
|
fs::copy(&source, &target).with_context(|| { |
Another copy occurs here:
|
// Rename succeeded — we now own the only copy. Put a copy |
|
// back at the original location for users who expect the |
|
// artifact at the standard cargo output path. Skip if a |
|
// new file already appeared (cargo / rust-analyzer rebuilt). |
|
if artifact_path.exists() { |
|
tracing::debug!( |
|
"Skipping copy-back: {} was recreated by another process", |
|
artifact_path.display() |
|
); |
|
} else if let Err(err) = reflink_or_copy(&new_artifact_path, artifact_path) { |
Can these copies be replaced with renames / softlinks, or avoided altogether? Reflink is not enough, his filesystem (Ext4, which is really common) does not support them.
For example to avoid the second copy, I believe the "copy back" can just be a "rename back" step after the modification/reading process is complete. A complete file copy just to then be able to 'safely' mmap the file to read it seems rather silly.
A colleague of mine uses a Linux machine running on an Ext4 disk, and is noticing long (~8 seconds) post-compilation steps (with the latest version of maturin). Using profiling I've identified that the full multi-gigabyte artifact (Polars with debug symbols) is copied multiple times.
One copy occurs here:
maturin/src/binding_generator/mod.rs
Lines 190 to 192 in da71d84
Another copy occurs here:
maturin/src/build_context/repair.rs
Lines 335 to 344 in da71d84
Can these copies be replaced with renames / softlinks, or avoided altogether? Reflink is not enough, his filesystem (Ext4, which is really common) does not support them.
For example to avoid the second copy, I believe the "copy back" can just be a "rename back" step after the modification/reading process is complete. A complete file copy just to then be able to 'safely'
mmapthe file to read it seems rather silly.