tree view is unusably slow on a large monorepo

[bradzacher]

Check for existing issues

• Completed

Describe the bug / provide steps to reproduce it

Screen.Recording.2024-02-23.at.14.16.31.mov

Sorry about the blurring - internal company monorepo so can't really share the names out.

When opening zed it takes around 20s for zed to show the 1st-level contents of the folder.

Problem number one here is that there is no progress bar or activity indicator to show that something is happening.
The first time I added the folder to the IDE I thought I'd done something wrong because nothing showed up. I switched away to read the docs and when I went back to zed it showed the folders.

I then try to expand a folder. The folder I clicked on has JUST ONE file in it. The video cuts off before the folder contents show up - I let it go for about a minute before I got bored and stopped recording so I don't know how long it took overall (I trimmed the video down).

Sadly this makes the IDE unusable as a daily driver for me.

Environment

Zed: v0.123.3 (Zed)
OS: macOS 14.3.1
Memory: 64 GiB
Architecture: aarch64

If applicable, add mockups / screenshots to help explain present your vision of the feature

No response

If applicable, attach your ~/Library/Logs/Zed/Zed.log file to this issue.

If you only need the most recent lines, you can run the zed: open log command palette action to see the last 1000.

The last 7000 lines of the log are just this repeated:

2024-02-23T14:29:06+10:30 [ERROR] crates/fs/src/repository.rs:147: Error { code: -1, klass: 10, message: "invalid data in index - calculated checksum does not match expected" }
2024-02-23T14:29:06+10:30 [ERROR] crates/fs/src/repository.rs:215: Error { code: -1, klass: 10, message: "invalid data in index - calculated checksum does not match expected" }

[jianghoy]

It looks like the repo is in some state that causes the underlying libgit2 rs crate (used to manipulate git) error.
I tried to replicate by:

1. cloning a fairly large repo onto my machine (in this case I used k8s)
2. open using zed.
   The result is here: loom recording of opening up k8s in zed, which looks way faster than your experience.

Could you share what the git repo looks like when you started? e.g. is it stuck in rebasing? etc etc.

[jbkkd]

I suffer from the same issue on a large monolith, with the same error in the log.

This may or may not be helpful, but in my case 99% of the files are Python files.

[bradzacher]

Could you share what the git repo looks like when you started? e.g. is it stuck in rebasing? etc etc.

Nothing weird - the repo was in a clean state.

libgit2 rs crate

Oof - this is likely the problem.

From my experience libgit2 is missing many performance features implemented in the git CLI which makes it unusable on our monorepo.

I recently tried to use it in an internal CLI I was building. I built a command similar to git status and it took around ~40s to run - compared with ~5-10s for the native git cli.

I also found that some of the newer features of git aren't supported in libgit2 - which caused crashes in libgit2.

[bradzacher]

When I get to my desk I'll do a file count to illustrate the size of the monorepo so you can understand the scale.

[bradzacher]

$ cd company_repo
$ git ls-files | wc -l
 311564
$ du -sh .
 47G    .

$ cd kubernetes
$ git ls-files | wc -l
 24149
$ du -sh .
 1.3G	.

just to illustrate - our company's monorepo is over 10x larger than the k8 repo in terms of number of files, and many times larger in terms of disk weight.

Another illustration of the weight difference:

$ cd company_repo
$ time git status
 0.70s user 0.06s system 97% cpu 0.777 total
$ time git status
 0.67s user 0.05s system 96% cpu 0.755 total

$ cd kubernetes
$ time git status
 0.02s user 0.04s system 74% cpu 0.086 total
$ time git status
 0.02s user 0.03s system 61% cpu 0.089 total

Note that this is with both folders in a clean, no changes, not rebasing, all pushed state after already having run status a few times.

[jianghoy]

There're plenty of issues regarding libgit2's performance being a couple *magnitude* slower than git cli for most of the operations. One thing that could potentially be a fix is to bump the libgit2 version (`0.14.2+1.5.1` the former is libgit2 rs binding's version number, the latter is libgit2 c code's version number), see if some fixes are shipped. Brad Zacher ***@***.***> 于2024年2月25日周日 20:15写道：

…

$ cd company_repo $ git ls-files | wc -l 311564 $ du -sh . 47G . $ cd kubernetes $ git ls-files | wc -l 24149 $ du -sh . 1.3G . @jianghoy <https://github.com/jianghoy> just to illustrate - our company's monorepo is over 10x larger than the k8 repo in terms of number of files, and many times larger in terms of disk weight. Another illustration of the weight difference: (Note that this is with both folders in a clean, no changes, not rebasing, all pushed state.) $ cd company_repo $ time git status 0.70s user 0.06s system 97% cpu 0.777 total $ time git status 0.67s user 0.05s system 96% cpu 0.755 total $ cd kubernetes $ time git status 0.02s user 0.04s system 74% cpu 0.086 total $ time git status 0.02s user 0.03s system 61% cpu 0.089 total — Reply to this email directly, view it on GitHub <#8242 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIIWBJIMBKDT5Q3EGSENNEDYVPOZ3AVCNFSM6AAAAABDWCHVWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRTGE2TGNRTGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jianghoy]

OK created a feature branch that updated libgit2 to 0.16.2+1.7.2, which seems to be the current stable version of libgit2 rust binding:https://github.com/jianghoy/zed/tree/update-libgit2, the build passes on my test machine. Maybe you could give it a spin and see if it works better?
If you follow this instruction and build the code: https://github.com/jianghoy/zed/blob/update-libgit2/docs/src/developing_zed__building_zed_macos.md, at the end by running cargo run it will launch a debug build of Zed. If everything works out I'll create PR.