Skip to content

panic: runtime error: invalid memory address or nil pointer dereference #705

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
smacker opened this issue Feb 22, 2019 · 16 comments
Closed
Assignees
Labels
bug Something isn't working

Comments

@smacker
Copy link

smacker commented Feb 22, 2019

gitbase v0.19.0-beta4

query:

SELECT 
r.repository_id, SUM(ARRAY_LENGTH(SPLIT(b.blob_content, '\n'))) as lines_count
FROM refs r
NATURAL JOIN commit_blobs ct
NATURAL JOIN blobs b
WHERE r.ref_name = 'HEAD'
GROUP BY r.repository_id
ORDER BY lines_count DESC

Traceback:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x87fe52]

goroutine 6076 [running]:
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-git.v4/plumbing/cache.(*ObjectLRU).Put(0xc000aa4ff0, 0x14440c0, 0xc01391da40)
	/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-git.v4/plumbing/cache/object_lru.go:64 +0x352
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem.(*ObjectStorage).getFromUnpacked(0xc017bec868, 0xa9112fc6650d62b5, 0x83f2209191640cdc, 0xa7d9180f, 0x14440c0, 0xc01391da40, 0x0, 0x0)
	/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem/object.go:344 +0x39a
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem.(*ObjectStorage).EncodedObject(0xc017bec868, 0x112fc6650d62b503, 0xf2209191640cdca9, 0xa7d9180f83, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem/object.go:254 +0x3eb
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.GetBlob(0x1441f00, 0xc017bec850, 0xa9112fc6650d62b5, 0x83f2209191640cdc, 0xa7d9180f, 0x650d62b5000081a4, 0x91640cdca9112fc6, 0xa7d9180f83f22091)
	/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/blob.go:23 +0x4e
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*FileIter).Next(0xc01e6b4b40, 0x4211e8, 0xc00498dd10, 0x7efccf3a1cb3)
	/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/file.go:100 +0x136
github.com/src-d/gitbase.(*squashCommitBlobsIter).Advance(0xc017becbd0, 0xc, 0xc00113fb01)
	/go/src/github.com/src-d/gitbase/squash_iterator.go:2754 +0x7c
github.com/src-d/gitbase.(*squashCommitBlobBlobsIter).Advance(0xc02245c5a0, 0x5c6fc722, 0x27c2150a)
	/go/src/github.com/src-d/gitbase/squash_iterator.go:3051 +0x49
github.com/src-d/gitbase.(*chainableRowIter).Next(0xc00b3d0370, 0x5777dccfb2, 0x2118820, 0x20, 0x25, 0xc00113fc80)
	/go/src/github.com/src-d/gitbase/squash.go:150 +0x37
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql.(*spanIter).Next(0xc02245c5f0, 0xc000063290, 0xc000062000, 0xc00113fcb0, 0x414d10, 0xc0044c4b90)
	/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/session.go:346 +0x5d
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan.(*trackedRowIter).Next(0xc0227dad20, 0x50, 0x44b4f8, 0x52307915bd55c, 0x27c2138b, 0x27c2138b0113fd68)
	/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan/process.go:145 +0x37
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan.(*FilterIter).Next(0xc00bb17e00, 0x5777dcce26, 0x2118820, 0x3, 0x3, 0xc00113feb2)
	/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan/filter.go:105 +0x38
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql.(*spanIter).Next(0xc02245c780, 0xc00113fef8, 0x44b4f8, 0x52307915bd4b6, 0xc027c212db, 0x27c212db0113fe20)
	/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/session.go:346 +0x5d
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan.(*iter).Next(0xc0227dad40, 0x5777dccd80, 0x2118820, 0x4dd96c, 0xc02226c440, 0xc026896120)
	/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan/project.go:129 +0x38
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql.(*spanIter).Next(0xc02245c7d0, 0xc00113feac, 0x3, 0x2, 0x0, 0x0)
	/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/session.go:346 +0x5d
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan.(*exchangeRowIter).iterPartition(0xc01b52a5a0, 0x142d620, 0xc00b3d01b0)
	/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan/exchange.go:245 +0x251
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan.(*exchangeRowIter).start.func1(0xc01b52a5a0, 0xc0217396b0, 0x142d620, 0xc00b3d01b0)
	/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan/exchange.go:170 +0x3f
created by github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan.(*exchangeRowIter).start
	/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan/exchange.go:169 +0x10d
@erizocosmico
Copy link
Contributor

It might be mine

SELECT uast_extract(
    uast(blob_content, 'csharp', "(//csharp:BinaryExpression_AddExpression/Left/uast:String | //csharp:InterpolatedStringExpression//csharp:InterpolatedStringTextToken[1])[starts-with(normalize-space(@Value), 'SELECT') or starts-with(normalize-space(@Value), 'select') or starts-with(normalize-space(@Value), 'UPDATE') or starts-with(normalize-space(@Value), 'update') or starts-with(normalize-space(@Value), 'DELETE') or starts-with(normalize-space(@Value), 'delete') or starts-with(normalize-space(@Value), 'INSERT') or starts-with(normalize-space(@Value), 'insert') or starts-with(normalize-space(@Value), 'CREATE') or starts-with(normalize-space(@Value), 'create') or starts-with(normalize-space(@Value), 'ALTER') or starts-with(normalize-space(@Value), 'alter') or starts-with(normalize-space(@Value), 'DROP') or starts-with(normalize-space(@Value), 'drop')]"),
    '@pos') AS positions,
    repository_id,
    file_path
FROM (
    SELECT f.repository_id,
        f.file_path,
        b.blob_content
    FROM (
        SELECT *
        FROM refs r
        NATURAL JOIN commit_blobs cb
        NATURAL JOIN blobs
        WHERE r.ref_name = 'HEAD'
            AND NOT IS_BINARY(blob_content)
    ) b
    INNER JOIN (
        SELECT repository_id, file_path, blob_hash
        FROM refs r
        NATURAL JOIN commit_files cf
        WHERE r.ref_name = 'HEAD'
    ) f
    ON b.blob_hash = f.blob_hash
        AND b.repository_id = f.repository_id
    WHERE language(f.file_path, b.blob_content) = 'C#'
) t
WHERE positions IS NOT NULL

@ajnavarro ajnavarro added the bug Something isn't working label Feb 22, 2019
@ajnavarro
Copy link
Contributor

Appears to be a race condition on go-git lru cache.

@erizocosmico
Copy link
Contributor

It does not happen in v0.19.0-inmemjoin (and probably earlier versions), have we updated go-git since then?

@ajnavarro
Copy link
Contributor

we did: 73c1730

pinging @jfontan

@ajnavarro
Copy link
Contributor

Other option can be that we are hitting more the cache walking commits using the history instead of the packfile.

@kuba--
Copy link
Contributor

kuba-- commented Feb 22, 2019

I've run go-git test (master): go test -v -race ./...
Here is what I got:

PANIC: object_test.go:227: FsSuite.TestPackfileReindex

... Panic: runtime error: index out of range (PC=0x10935CA)

/usr/local/opt/go/libexec/src/runtime/panic.go:513
  in gopanic
/usr/local/opt/go/libexec/src/runtime/panic.go:44
  in panicindex
go-git-fixtures.v3/fixtures.go:270
  in FsSuite.TestPackfileReindex
object_test.go:230
  in FsSuite.TestPackfileReindex
/usr/local/opt/go/libexec/src/reflect/value.go:308
  in Value.Call
/usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1333
  in goexit
OOPS: 55 passed, 8 skipped, 1 PANICKED
--- FAIL: Test (61.02s)
----------------------------------------------------------------------
PANIC: parser_test.go:80: ParserSuite.TestThinPack

... Panic: runtime error: index out of range (PC=0x10946DA)

/usr/local/opt/go/libexec/src/runtime/panic.go:513
  in gopanic
/usr/local/opt/go/libexec/src/runtime/panic.go:44
  in panicindex
go-git-fixtures.v3/fixtures.go:270
  in ParserSuite.TestThinPack
parser_test.go:88
  in ParserSuite.TestThinPack
/usr/local/opt/go/libexec/src/reflect/value.go:308
  in Value.Call
/usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1333
  in goexit
OOPS: 38 passed, 1 PANICKED
--- FAIL: Test (181.16s)
----------------------------------------------------------------------
PANIC: decoder_test.go:206: IndexSuite.TestDecodeEndOfIndexEntry

... Panic: runtime error: index out of range (PC=0x1092FEA)

/usr/local/opt/go/libexec/src/runtime/panic.go:513
  in gopanic
/usr/local/opt/go/libexec/src/runtime/panic.go:44
  in panicindex
go-git-fixtures.v3/fixtures.go:270
  in IndexSuite.TestDecodeEndOfIndexEntry
decoder_test.go:207
  in IndexSuite.TestDecodeEndOfIndexEntry
/usr/local/opt/go/libexec/src/reflect/value.go:308
  in Value.Call
/usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1333
  in goexit
OOPS: 15 passed, 1 PANICKED
--- FAIL: Test (0.31s)
----------------------------------------------------------------------
FAIL: config_test.go:12: ConfigSuite.TestUnmarshall
config_test.go:40:
    c.Assert(err, IsNil)
... value scanner.ErrorList = scanner.ErrorList{(*scanner.Error)(0xc000112a80)} ("16:9: unquoted '\\' must be followed by new line")

OOPS: 30 passed, 1 FAILED
--- FAIL: Test (0.02s)

@jfontan
Copy link
Contributor

jfontan commented Feb 22, 2019

It seems that it's trying to insert in the cache an object bigger than the max size. This should not happen but it's the only explanation I have.

@smacker can you check if there's a blob bigger than 512 Mb?

SELECT b.repository_id, b.blob_hash, b.blob_size
FROM refs r
NATURAL JOIN commit_blobs ct
NATURAL JOIN blobs b
WHERE r.ref_name = 'HEAD' AND b.blob_size > 531628032;

@jfontan
Copy link
Contributor

jfontan commented Feb 22, 2019

Also, can yo do an fsck to the repositories? Some of them may contain a broken object that is causing this.

@smacker
Copy link
Author

smacker commented Feb 22, 2019

@jfontan do you need it to be run on the version that panics or an older one that works?

on v0.19.0-inmemjoin it produces zero results:
screenshot 2019-02-22 at 18 33 16

@smacker
Copy link
Author

smacker commented Feb 22, 2019

Result of fsck (with removed names):

$ for d in `ls`; do (echo "repo: $d"; cd "$d"; git fsck --no-progress); done
notice: HEAD points to an unborn branch (master)
notice: No default references
notice: HEAD points to an unborn branch (master)
notice: No default references
notice: HEAD points to an unborn branch (master)
notice: No default references
notice: HEAD points to an unborn branch (master)
notice: No default references
notice: HEAD points to an unborn branch (master)
notice: No default references

@ajnavarro
Copy link
Contributor

Related issue: src-d/go-git#1071

@jfontan
Copy link
Contributor

jfontan commented Feb 25, 2019

I've not been able to reproduce the error with git objects. Adding objects with bogus data to the cache I could make it crash in the same place. Hopefully the check I've added is enough.

PR: src-d/go-git#1076

@kuba--
Copy link
Contributor

kuba-- commented Feb 25, 2019

It's not about data race (because LRU cache is thread-safe).
It's also not about one single big object, because it won't be added to cache.
Maybe I'm wrong but the mystery looks that we exceeded the cache size, and we try to delete an old object which became nil.

	last := c.ll.Back()
	lastObj := last.Value.(plumbing.EncodedObject) <--- panic
	lastSize := FileSize(lastObj.Size())

in a meantime someone changed the reference of that object.

@jfontan
Copy link
Contributor

jfontan commented Feb 25, 2019

@kuba-- yes, the test does just that. Change the size of one object in cache:

	o := NewObjectLRU(9 * Byte)

 	a1 := newObject(s.aObject.Hash().String(), 9*Byte)
	a2 := newObject(s.aObject.Hash().String(), 1*Byte)
	b := newObject(s.bObject.Hash().String(), 1*Byte)

 	o.Put(a1)
	a1.SetSize(-5)
	o.Put(a2)
	o.Put(b)

@smola
Copy link
Contributor

smola commented Feb 26, 2019

@ajnavarro Is this a regression since v0.18.0?

@ajnavarro
Copy link
Contributor

@smola yes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

6 participants