Gitbase doesn't work on windows with mounted directory for indexes

[smacker]

Gitbase 0.19.0.

Screenshots because it's hard to copy-past from remote windows console.

Client:

Server:

The error:

unable to save the index
open file: open ...: invalid argument

But the file is actually created on host file system:

I tried to read the source code and see where the error can come from. I found this line: https://github.com/pilosa/pilosa/blob/f2994736585a8aafc2f2c47c3698b7acd3b95373/fragment.go#L199 which looks like the right place.

I tried to reproduce it by creating simple script and running it inside docker with mounted directory:

package main

import (
	"fmt"
	"os"
)

func main() {
	asPilosa := "/mounted/asPilosa"

	_, err := os.OpenFile(asPilosa, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
	if err != nil {
		fmt.Printf("open file as pilosa: %s\n", err)
	}
}

but no luck. It creates the file without error.

It's super inconvinient to debug inside remote desktop. So I didn't go further by rebuilding gitbase with extra debug messages.

[kuba--]

It can be a silly thing with .dir, but so far hard to say. Btw. I'm still wondering how it's gonna work on windows, because pilosa uses Flocks for locking.
I'll take a closer look.

[kuba--]

Following code can reproduce this issue:

package main

import "github.com/pilosa/pilosa"

func main() {
        idx, err := pilosa.NewIndex("/mounted/pilosa", "idx-win")
        if err != nil {
                panic(err)
        }

        fld, err := idx.CreateField("fld-win", pilosa.OptFieldTypeDefault())
        if err != nil {
                panic(err)
        }
        rows := []uint64{1, 2, 1, 2, 1, 2, 1, 2, 1}
        cols := []uint64{1, 2, 3, 4, 5, 6, 7, 8, 9}

        err = fld.Import(rows,cols, nil)
        if err != nil {
                panic(err)
        }
}

I tested on windows inside a docker container. It happens only for Import (SetBit works fine).

[kuba--]

This is pilosa's issue. I'm not 100% sure where the problem is, but the way how pilosa opens/closes hierarchy of files (/index/field/view/fragment/storage,cache) doesn't work for mounted volumes. I assume one of the syscalls (Mmap, Flock, Munmap, ...) is a root cause.
Basically pilosa's fragment tries to open a file:

r, e = syscall.Open(name, flag|syscall.O_CLOEXEC, syscallMode(perm))
		if e == nil {
			break
		}

which returns an error invalid argument. After deep dive into go's std. lib., it happens only when File pointer is nil. Calling directly syscall.Open works fine. So, definitively pilosa previous calls hide some attributes.

Definitively we don't want to replace field.Import by SetBit, because of performance reasons. So I've submitted an issue to pilosa project:
FeatureBaseDB/featurebase#1914

[erizocosmico]

Waiting for src-d/go-mysql-server#679 to be merged before the fix for this

[ajnavarro]

@erizocosmico already merged.

[kuba--]

Setting syswrap.SetMaxMapCount(0) in pilosa driver init function should solve the windows issue.

[erizocosmico]

PIlosa version bump in #804. When it's merged, we will add the fix.