This repository was archived by the owner on Sep 11, 2020. It is now read-only.
Improve packfile reading performance #906
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Miguel Molina <[email protected]>
plumbing/format/idxfile: add new Index and MemoryIndex
Signed-off-by: Javi Fontan <[email protected]>
In one case it disables the cache and the other disables lookup when the scanner is not seekable. Could be added back later. Signed-off-by: Javi Fontan <[email protected]>
It's still not complete: * 64 bit offsets * IdxChecksum Signed-off-by: Javi Fontan <[email protected]>
Signed-off-by: Javi Fontan <[email protected]>
Signed-off-by: Javi Fontan <[email protected]>
Signed-off-by: Javi Fontan <[email protected]>
Signed-off-by: Javi Fontan <[email protected]>
This functionality may be moved elsewhere in the future but is needed now to fit filesystem.ObjectStorage and the new index. Signed-off-by: Javi Fontan <[email protected]>
Index is also automatically generated when OnFooter is called. Signed-off-by: Javi Fontan <[email protected]>
Now dotgit.PackWriter uses the new packfile.Parser and index. Signed-off-by: Javi Fontan <[email protected]>
Feature/new packfile parser
Signed-off-by: Miguel Molina <[email protected]>
Signed-off-by: Javi Fontan <[email protected]>
Signed-off-by: Javi Fontan <[email protected]>
Signed-off-by: Javi Fontan <[email protected]>
Bugfixes and IndexStorage
Signed-off-by: Miguel Molina <[email protected]>
plumbing: packfile, new Packfile representation
Signed-off-by: Javi Fontan <[email protected]>
Signed-off-by: Javi Fontan <[email protected]>
Signed-off-by: Javi Fontan <[email protected]>
Tests and indexes in packfile decoder
Signed-off-by: Miguel Molina <[email protected]>
ajnavarro
reviewed
Jul 31, 2018
plumbing/format/packfile/decoder.go
Outdated
| } | ||
|
|
||
| d.offsetToHash[h.Offset] = obj.Hash() |
There was a problem hiding this comment.
why do we need that having the index?
There was a problem hiding this comment.
You may not have the index built
ajnavarro
reviewed
Jul 31, 2018
| @@ -47,6 +46,7 @@ func (s *ReaderSuite) TestDecode(c *C) { | |||
| }) | |||
| } | |||
|
|
|||
| /* | |||
There was a problem hiding this comment.
should that be uncommented?
ajnavarro
reviewed
Jul 31, 2018
| var base plumbing.EncodedObject | ||
| var ok bool | ||
| hash, err := p.FindHash(offset) | ||
| if err == nil { |
There was a problem hiding this comment.
yeah, that's intentional
ajnavarro
reviewed
Jul 31, 2018
|
|
||
| func (s *PackfileSuite) TestContent(c *C) { | ||
| storer := memory.NewObjectStorage() | ||
| decoder, err := NewDecoder(NewScanner(s.f.Packfile()), storer) |
There was a problem hiding this comment.
why we are using decoder here?
There was a problem hiding this comment.
To fill the memory storage and check if Packfile decodes objects correctly as decoder does
|
Windows tests seem to be failing because of an access denied removing fixtures 🤷♀️ |
Signed-off-by: Miguel Molina <[email protected]>
|
Just added a benchmark for Before, using Now, using As we can see, Parser is slower. Like, way slower. |
|
New results after some optimizations, now it's faster and uses less memory than decoder. |
Signed-off-by: Miguel Molina <[email protected]>
|
We might want to rename |
|
@erizocosmico Windows issue is likely to be related to a packfile not being closed before the test ends. |
Signed-off-by: Miguel Molina <[email protected]>
|
Added benchmarks for PackfileIter. Before: Now: Now, reading object content: |
Signed-off-by: Miguel Molina <[email protected]>
erizocosmico
force-pushed
the
perf/packfile-reads
branch
from
fa697fa to
65dc4f9
Compare
ajnavarro
reviewed
Aug 9, 2018
plumbing/format/packfile/parser.go
Outdated
| @@ -460,6 +460,8 @@ type objectInfo struct { | |||
| Parent *objectInfo | |||
| Children []*objectInfo | |||
| SHA1 plumbing.Hash | |||
|
|
|||
| Content []byte | |||
There was a problem hiding this comment.
I think with an LRU cache for content by offset we can avoid using a lot of memory.
There was a problem hiding this comment.
We'd still need to read objects more times, which is why it was slower before
Signed-off-by: Miguel Molina <[email protected]>
|
Renamed DiskObject to FSObject and I'm fixing the not closed packfile thing |
Signed-off-by: Miguel Molina <[email protected]>
erizocosmico
force-pushed
the
perf/packfile-reads
branch
from
24dc247 to
d93b386
Compare
Signed-off-by: Miguel Molina <[email protected]>
|
Tests should be passing now. |
Signed-off-by: Miguel Molina <[email protected]>
|
I think it can be reviewed now |
mcuadros
suggested changes
Aug 10, 2018
| @@ -17,6 +17,11 @@ var ( | |||
| ErrMalformedIdxFile = errors.New("Malformed IDX file") | |||
| ) | |||
|
|
|||
| const ( | |||
| fanout = 256 | |||
There was a problem hiding this comment.
the previous fanout was 255, was an error?
There was a problem hiding this comment.
Real fanout size is 256, we were using 255 because the last element is the object count. JGit uses 256 as well https://github.com/eclipse/jgit/blob/6d370d837c5faa7caff2e6e3e4723b887f2fbdca/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/PackIndexV2.java#L67
plumbing/format/idxfile/idxfile.go
Outdated
| Version uint32 | ||
| Fanout [256]uint32 | ||
| // FanoutMapping maps the position in the fanout table to the position | ||
| // in the Names, Offset32 and Crc32 slices. This improves the memory |
plumbing/format/idxfile/idxfile.go
Outdated
| FanoutMapping [256]int | ||
| Names [][]byte | ||
| Offset32 [][]byte | ||
| Crc32 [][]byte |
plumbing/format/idxfile/idxfile.go
Outdated
| Hash plumbing.Hash | ||
| CRC32 uint32 | ||
| Offset uint64 | ||
| func (idx *MemoryIndex) findHashIndex(h plumbing.Hash) int { |
There was a problem hiding this comment.
In go is recommended return a second boolean value
findHashIndex(h plumbing.Hash) (int, bool)
plumbing/format/idxfile/idxfile.go
Outdated
| return idx.getCrc32(k, i) | ||
| } | ||
|
|
||
| func (idx *MemoryIndex) getCrc32(firstLevel, secondLevel int) (uint32, error) { |
| firstLevel, secondLevel int | ||
| } | ||
|
|
||
| func (i *idxfileEntryIter) Next() (*Entry, error) { |
There was a problem hiding this comment.
this function can be split in two
There was a problem hiding this comment.
Which part do you mean? I don't see any way to split this
Signed-off-by: Miguel Molina <[email protected]>
jfontan
suggested changes
Aug 13, 2018
| } | ||
| } | ||
|
|
||
| if delta && !p.scanner.IsSeekable { |
There was a problem hiding this comment.
Shouldn't we use the storer to retrieve objects instead of storing all deltas in memory?
There was a problem hiding this comment.
Some storers, such as the MemoryStorage, do not allow deltas stored inside them
jfontan
approved these changes
Aug 13, 2018
mcuadros
approved these changes
Aug 14, 2018
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
More info at: https://docs.google.com/document/d/1I2Vte4LhMdpx0kvlrg-k6hNA-0kUGtUkC1-kfwbCmeI