Git pull failing with "git-remote-keybase error: (1) packfile not found"

[telotortium]

This has been happening on one of my private git repos since around the beginning of this week (only on macOS 10.13.3):

$ git pull
Initializing Keybase... done.
Syncing with Keybase... done.
git-remote-keybase error: (1) packfile not found

The first time this happened, I upgraded Keybase to Version 1.0.47-20180410052738+f705a9510f. However, I'm continuing to see this happen. I saw #11358 and tried to run keybase git delete $REPO, since I have the repo cloned locally, but that command hangs.

[strib]

@telotortium please run keybase log send and we can take a look. Having some downtime right now that might be affecting your ability to delete the repo.

[telotortium]

Log ID: d0b6431d457ef26c9c6b771c

[strib]

It's dying looking for pack-0000000000000000000000000000000000000000.idx, which seems like a problem to me, since that's not a likely packfile name (usually it's named by the hash).

@telotortium how did you create and initialize this repo? Did you push --all an existing repo, or have you built it up commit-by-commit?

(Our downtime is over now, in case you want to retry deleting and recreating the repo.)

[telotortium]

I started a new repo on my workstation, created the repo in Keybase, manually set the remote in my local repo to point to Keybase, and then ran git push to push changes to it. Usually I use git push --force-with-lease. I have only the master branch, so using git push --all would make no difference.

[strib]

I'm just wondering if git-remote-keybase initialized the repo on the first push from a repo with a long history already, in which case the bug might have already existed in that initial repo (unrelated to KBFS). But if you started with an empty repo and built it up commit by commit, this isn't likely.

Are you willing to share the repo with me for debugging purposes? If so, and it's not too big, please do this:

cp -r /keybase/private/telotortium/.kbfs_git/<repo-name> /keybase/private/telotortium,strib/

(substituting your repo name for <repo-name>, I don't want to expose it publicly here.) Then I can dig into it locally.

[telotortium]

I've shared the repo with you.

[strib]

Thanks. I play around with it and am not having any issues cloning, pulling or pushing. Hrm. Are you still getting the packfile failure on a git pull? If so, in your local repo where you're doing the pull from, what's the HEAD commit ID?

[telotortium]

Locally the head commit is ec55898. On KBFS HEAD is at be300d2 and has the local HEAD as a direct ancestor. Should I just try to clone the repo elsewhere locally?

[strib]

Yeah a full clone from what is in KBFS worked for me, so you might want to try that. But that's not very satisfying, I'll try to repro your issue on my own too.

[telotortium]

I performed a full clone 2 days ago. Now I'm seeing the issue again. I know that I recently updated the repo from a Linux workstation, so maybe that has something to do with it. The version on the workstation is 1.0.45-20180313160419.f0de728311 - I'll try upgrading to the most recent version (currently 1.0.47-20180403160041.242bcf96eb).

[strib]

Hrm, odd. I don't think we've done any major git-related changes since that last release, but yeah it's always good to try upgrading.

I was not able to reproduce the pull issue with the repo you shared with me, even when making sure my local checkout only included up to commit ec55898.

It might be worth doing a log send from your Linux workstation, so we can check if any obvious errors happened during the push. But right now I'm still stumped.

[telotortium]

Log ID from Linux: 446816aef643c4996ebee11c. However, I restarted Keybase since the push that seems to have caused the problems on macOS (due to upgrading on Linux), so I'm not sure how useful it is. Also, I ran git push from Linux just now, but I'm still seeing the same issues on macOS.

[telotortium]

Another thought: pack-0000000000000000000000000000000000000000.idx looks a lot like it's a filename that would result if the code had forgotten to set a variable or field that should hold a Git object ref, because Go zero initializes all fields and variables.

[strib]

Oh, I might see the problem. It looks like, at the KBFS level, you once had two clients (or maybe two simultaneous git commands on the same client) updating the repo at the same time. This led KBFS to do conflict resolution, which led to special "conflict" files in your repo/objects/pack directory. It seems like the git library we're using isn't correctly ignoring these files, and is instead trying to treat them like regular files, leading to the zero hash.

If I'm right, this should clean it up for you (where $REPONAME is set to your repo's name):

rm /keybase/private/telotorium/.kbfs_git/$REPONAME/objects/pack/*conflicted*

Can you give that a try and see if it helps? (Though I'm still not sure why I wasn't able to repro it myself.)