Performance degradation

[icambridge]

So if I have a long running process I end up with what I would consider serious performance degradation. The following code when the process originally starts takes ms however as the process goes on the takes longer and longer no matter the size of the repository. I've seen it literally taking 10s of minutes on really small git repositories.

func (gc *GitRepository) FetchFiles() (*FileList, error) {

	if len(gc.files) > 0 {
		return &FileList{gc.files}, nil
	}

	log.Infof("Fetching files for %s", gc.name)
	ref, err := gc.repository.Head()
	if err != nil {
		return nil, err
	}

	// ... retrieving the commit object
	commit, err := gc.repository.Commit(ref.Hash())

	if err != nil {
		return nil, err
	}
	tree, err := commit.Tree()

	if err != nil {
		return nil, err
	}
	fileIt := tree.Files()

	defer fileIt.Close()
	repoFiles := map[string]File{}
	for {
		fo, err := fileIt.Next()

		if err != nil {
			break
		}
		repoFile := GitFile{
			fo:  fo,
		}
		repoFiles[fo.Name] = repoFile
	}
	gc.files = repoFiles
	log.Infof("Finished fetching files for %s", gc.name)
	return &FileList{repoFiles}, nil
}

Also may as well use this issue for the memory performance issues with this library and commits as can be seen on https://github.com/icambridge/go-git-example with symfony/symfony taking up a GB of RAM just to get the commits.

Overall I have a 8 core 16GB server processing 140 repositories an hour running out of memory.

[smola]

@icambridge Thank you for the report!

• Which version of go-git are you using?
• How are you instantiating repositories? (NewMemoryRepository or NewFilesystemRepository)
• Are you cleaning up references to repositories after you process them?

[icambridge]

@smola

• I am (or was before the last merge) using the latest master.
• For the memory usage of 1GB in https://github.com/icambridge/go-git-example I am using NewMemoryRepository. In my actual app I'm using NewFilesystemRepository.
• I think so yes. My memory usage goes up and down instead of constantly climbing. So I assume no memory leak. However there doesn't seem to be a close method so I'm just relying on garbage collection.

[icambridge]

I've finally got round to adding profiling to the process. These were taken while i had 15 goroutines all taking a while in the fetch files method.

CPU snapshot

Memory snapshot

isostat output

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.04     2.31    2.48    1.28    78.55   298.43   200.81     0.23   60.78    0.93  177.05   1.08   0.40

[smola]

@icambridge We recently merged #218 which makes full packfile decoding (e.g. clones) both faster and with less allocations. We're planning further improvements on this front for other use cases.

[icambridge]

Thanks, I was going to mention I've built the with latest master and performance difference is 100% improvement.

[smola]

Ok. I'm closing this as fixed.
@icambridge Feel free to reopen if the problem persists.

[icambridge]

@smola There still appears to be degrading performance, as in it slows down over time. I'm solving this myself by constantly restarting the process every 30 minutes.