Performance with large git repos

[sbidoul]

Hi,

I found a situation where uv is significantly slower than pip: installing from a large git repo.

Reproducer: uv -v pip install "odoo @ git+https://github.com/odoo/odoo@17.0" (warning: this requires ~6GB disk space and a ~4GB download).

After the download (observed by monitoring network activity), there seems to be a pause of several minutes where it is unclear what uv is doing, even with -v or RUST_LOG=trace.

[charliermarsh]

Thanks!

[charliermarsh]

@zanieb - This may be solved by changing the default to use the Git CLI.

[serozhenka]

I also experience a strange pause for around 7 seconds (out of 17 for the overall install) when trying to install a private git package via a git tag and not a commit ref. As @sbidoul mentioned nothing indicates what's going on even when enabling additional verbosity.

[zanieb]

@serozhenka is this on the latest version?

[serozhenka]

@zanieb it was 0.1.6 at the time of writing, but I have just checked on 0.1.8 and it's the same. Installation time dropped down by 3s on average, but that lag still exists.

upd. my bad on that, missed a single log statement, that's actually the repo subdirectory fetch itself that takes 8s. But overall it's still 1-2s slower that pip with cold cache.

[sbidoul]

From my part, I have noticed a solid improvement with the switch to using the git CLI.

In pip we use git clone --filter=blob:none to improve performance with large repos, but the pip git clones are transient (we only keep the resulting wheel).

So I think (I've not mesured rigorously) that with a cold cache, uv is slower than pip with large git repos, but when the git db cache is pre-populated, subsequent operations will be faster.

[flying-sheep]

but when the git db cache is pre-populated, subsequent operations will be faster.

which doesn‘t apply here, right? we just care about a repo’s

1. working directory contents (i.e. all the files)
2. git history metadata (in case setuptools_scm or so wants to know how many commits it has been since the last tag)

both of these are being downloaded with --filter=blob:none, then we build, then we throw the repo away and don’t do any subsequent operations on it.

So uv should definitely do that, unless I’m missing something.

[charliermarsh]

Yeah, we tried it here: #3950. It just hasn't been done.

[flying-sheep]

I think what we could e.g. do is treat each repo as database of sorts:

1. if
   1. … we haven’t seen the repo before, clone the git repo into the cache using --filter=blob:none and the requested ref
   2. … the repo is in the cache and the ref is a tag or branch or unknown commit hash, fetch the requested ref into that repo
2. create a work tree from that repo and ref and use that to build stuff

or something similar.

that would help with retaining info as opposed to re-cloning.

[charliermarsh]

I think that's roughly what we do already? We maintain a Git database. The way it works is that we have a single clone of each repo, and when we need to use a commit, we fetch it into the Git database then create a local clone of that ref to build from. (But I don't believe it's blobless.)

[flying-sheep]

I see, then with a warm cache, subsequent VCS installs of the same repo should be fast-ish already (only new commits will be fetched on top of what was there last time), but we can still improve that initial clone to speed up the cold cache case by making the initial clone blobless. (I think creating a work tree instead of a local clone doesn’t make a huge difference, but could be a tiny bit faster and take less space)

I think the only use case that would be slower would be to

1. install a package from VCS
2. then install a package from an older commit of that same VCS

because the way you currently doing it, all old commits are already fetched, so this would be an almost entirely local operation

[zanieb]

We've tried to make the initial clone cheaper, e.g.,

• Fetch git trees on demand #3950
• Load Git trees lazily on checkout #8190

If you want to dig into that, feel free! It seems hard.