git pull takes over a minute and 20% of CPU

[EmpireJones]

When there is a new commit available to pull on a branch, git pull seems to be taking over a minute and is using a consistent ~20% CPU on all cores of my 4 core (8 virtual core) machine, for the duration. This same behavior is occurring for another user/computer.

[dabura667]

a new commit

Are you sure it's only pulling ONE commit?

This repo has like 45687465873465 commits per second... so I wouldn't be surprised if it's just taking a while to parse all the deltas.

[strib]

@EmpireJones If you do a keybase log send, and post the resulting ID here, I can take a look.

[EmpireJones]

@dabura667 To clarify, I'm referring to a team repo. But yes, only one commit.

@strib be070d5ea27fde18b31af71c

Thanks!

[strib]

Is this the fetch you did around 18:05 your local time? The logs are a bit truncated unfortunately, so I can't see how long it took. It seems to be quite a big repo though, from what I see in the log it's looking at a ton of objects.

Do you still have the terminal output for what git said while you were fetching? I'm interested to see how much data was transferred.

Also, if you're willing to share more logs, that might be useful. To do so, you can try:

tar -czf /keybase/private/strib@github,empirejones/client-12730.tgz ~/.cache/keybase/keybase.git*

[EmpireJones]

Yeah, that should be the one.

Log sent.

Initializing Keybase... done.
Syncing with Keybase... done.
Counting and decrypting: 16 objects... done.
Reading and decrypting metadata: (100.00%) 16/16 objects... done.
Fetching and decrypting objects: (100.00%) 16/16 objects... done.

[strib]

Thanks! I don't see anything obviously unusual going on, other than all the data being read. A few more questions:

• Has this happened for more than one commit, or did it happen just the once (on multiple devices) with the same commit?
• If it was just one commit, did the commit contain an unusual amount of data? A few big binary files checked in or anything? The output above says only 16 objects, so I guess it wasn't a lot of files.
• After you fetch the commit, if you pull a second time, is it fast?
• Do you know anything about the origin of the repo? Was it imported from a pre-existing repo with a big pre-existing history, or was it empty initially and has grown only via keybase git commits?

[EmpireJones]

• More than one commit. I've noticed it during 4 pulls so far (which is every time for this repo), but didn't before that because I was working on a branch alone (nobody else commiting).
• N/A as it was more than one commit, but it was just a handful of lines of code in 4 files
• Yep, pulling again happens quickly.
• It's a somewhat big repo, but the majority of it is text/code; I try to avoid checking in any large files (e.g. binary), although it might have some graphics. It was previously stored on BitBucket and then AWS CodeCommit. I didn't truncate the history when switching to keybase, which was recently.

The repo has ~1500 commits.

[strib]

Ok thanks. I wonder if the repo just has too many pack files and that's slowing everything down. One thing you can try is forcing garbage-collection on the repo. We do have a keybase git gc command that you can try, but I don't think we've turned on the ability to re-pack objects yet into a single pack file.

However, if you can be sure that no one will try to push to the repo for a few minutes, you can try running regular git gc. To do that, try cd /keybase/team/<teamname>/.kbfs_git/<reponame>. Then ls objects/packs/*.pack to see if you have a lot of packfiles. If so, you can try git gc. But if someone tries to push during that time, it might leave data in a weird state.

[EmpireJones]

I wasn't able to do a git gc inside of the .kbfs_git/<reponame> folder. I am getting these errors:
First Run:

error: unable to create temporary file: Interrupted system call
fatal: unable to force loose object
error: failed to run repack

Additional runs:

Counting objects: 24304, done.
error: pack-objects died of signal 7 
error: failed to run repack

The ./objects/pack/ folder currently has 79 files.

git count-objects -v reports this info:

warning: garbage found: ./objects/05/tmp_obj_7wObmh
warning: garbage found: ./objects/pack/tmp_pack_odRnG4
warning: garbage found: ./objects/pack/tmp_pack_8qMNAb
count: 14
size: 9
in-pack: 24360
packs: 38
size-pack: 100460
prune-packable: 14
garbage: 3
size-garbage: 3584

Running git gc in my local repo folder did not result in any performance improvements.

[strib]

Ah sorry, try git gc -quiet, then you can probably avoid those system call errors.

[strib]

If that doesn't work, I can try to get you a build with a working keybase-based gc next week.

[EmpireJones]

Does that only need to be run from one machine? If so, it didn't seem to impact performance. It ran without errors though.

Thanks for continuing to assist with this!

[EmpireJones]

I discovered a large file (~90mb) which was committed accidently by a team member. I'm working on getting that removed from history. It appears that just using a branch with the commit removed does not fix the performance issue (perhaps because it's still in the pack files from the other branches).

• Does this sound accurate -- that a large file would impact the checkout performance of future commits?
• Or is the slow performance more likely due to the number of commits/changes total?