Extremely slow with large repositories.

[AnalogFeelings]

This program is very slow when running it on large repositories, for example ReactOS.

[spenserblack]

@o2sh Should we revisit this?

[o2sh]

Sure, but after #211 and #309, I am running out of ideas. 😞

ping @CephalonRho @yoichi @HallerPatrick

[shuni64]

A lot of time still appears to be spent reading commit information in Repo::get_logs, most of which is only used by Repo::get_authors.

Reading commits in parallel seems like a way to speed this up a bit, but I'm not sure if this is possible with libgit2.
git-repository from gitoxide seems like a promising alternative.

Otherwise some form of caching could work, but I'm not sure if this is a good idea.

[o2sh]

gitoxide is indeed a promising alternative.
It might be an idea worth exploring even it means losing some features like mailmap.

[spenserblack]

mailmap is on their radar, at least. I guess we can open an issue to show them that there's interest for it 😆

If we do drop .mailmap support, reopening #447, we should probably make a release before we drop support. That way users that have used the .mailmap feature (#596) will have a release that's as up-to-date as possible before the breaking change.

[o2sh]

I created the issue GitoxideLabs/gitoxide#363.

If we do drop .mailmap support, reopening #447, we should probably make a release before we drop support. That way users that have used the .mailmap feature (#596) will have a release that's as up-to-date as possible before the breaking change.

That's a perfect plan 💯

[Byron]

Reading commits in parallel seems like a way to speed this up a bit, but I'm not sure if this is possible with libgit2.
git-repository from gitoxide seems like a promising alternative.

It looks like major speedups are possible even on a single thread for commit graph traversal, for example I'd expect onefetch to go from ~19s on the v5.16 linux kernel checkout to something like 11s.

If more/all operations are done in parallel, like the syntax analysis, it should become as fast as the slowest of these operations, the commit graph traversal which clocks in at about ~7s.

I wouldn't know how to parallelize the commit graph traversal though - the only way I can imagine this to work is to traverse different branches on multiple threads. This usually comes at the overhead of avoiding them to do duplicate work which requires a parallel hashset (like dashmap) which will limit the amount of threads that are effective there. That also depends on the graph, a linear history with a single trunk can't be sped up at all. Anyway, it's sounds like an interesting task to implement high-performance parallel traversal, maybe gitoxide can provide one once all other options are exhausted here.

The above is my profiling run on the linux kernel v5.16. Most of it is the commit graph traversal, the spike towards the end is tokei, and there is about ~1s of releasing memory (which can and should probably be avoided with process::abort() or by leaking the values with mem::forget()).

All in all, I think switching to gitoxide and running the syntax analysis alone in parallel, along with process::abort() to avoid memory deallocation at the end, one should be able to get roughly 2x the speed on the linux kernel.

[o2sh]

Thanks a lot for your input @Byron, I've created two issues as a follow up #628 and #629.

[Byron]

I couldn't resist to do a quick measurement on reactos which seems tame compared to the linux kernel. The numbers, however, are even more promising.

It seems that ~1.5s are spent on the commit graph traversal, a task which could be accomplished in ~0.4s with gitoxide.

Interestingly tokei finished in about ~0.5s so it appears that if both would run in parallel, one should finish the onefetch invocation in about 0.5s, down from ~1.5s . Can't wait to see this happen!

[o2sh]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.