x/vgo: vendor writes misleading vgo.list file

[iand]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go 1.10

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/iand/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/iand"
GORACE=""
GOROOT="/opt/go"
GOTMPDIR=""
GOTOOLDIR="/opt/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build364187917=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Added a dependency on a package not referred to in an import statement. In this sample it's a tool required for go generate but in the codebase where I first encountered this issue it is several tools used as part of the continuous build process that we have previously vendored in the past using govendor and dep.

Example that reproduces it.

• Checked out https://github.com/iand/vgo-vendor
• Added require golang.org/x/tools v0.0.0-20180525024113-a5b4c53f6e8b to go.mod
• Ran vgo build
• Ran vgo vendor

What did you expect to see?

The source for golang.org/x/tools to be copied into vendor directory

What did you see instead?

vendor/vgo.list updated to

golang.org/x/tools          v0.0.0-20180525024113-a5b4c53f6e8b

but golang.org/x/tools is not copied to the vendor directory.

I note from a quick look at the vgo source that the vendor command uses the import list to build the copy list but uses a different list when writing vgo.list

[Piszmog]

I am experience a similar thing when using go-micro in my project. vgo.list has go-micro listed but will not be copied to the vendor directory.

Even stranger if I delete the vendor directory and run vgo vendor, the directory will not be created as long as my project is using go-micro (if I remove all references to go-micro, then the vendor directory will be created and populated)

[gopherbot]

Change https://golang.org/cl/116759 mentions this issue: cmd/go/internal/vgo: fix handling of vendor dir in package patterns

[rsc]

Probably the vgo.list file should be refined to make clear that certain packages from certain modules are copied. For the specific case of iand/vgo-vendor, there's no path from the packages in that repo to anything that imports golang.org/x/tools, so x/tools is (correctly) not copied. It should also not appear in vgo.list.

@Piszmog, please file a separate issue about the problem you are seeing, with a bit more detail. I sent a CL 116759 that might help or might not - I don't fully understand what you were reporting.

[iand]

It was my expectation that adding require golang.org/x/tools v0.0.0-20180525024113-a5b4c53f6e8b to go.mod would declare golang.org/x/tools as a dependency that needs to be vendored by vgo -vendor.

Enabling this behaviour would address many use cases where tools supporting development or build are to be versioned along with direct dependencies. Examples include tools referenced by go:generate, asset packing utilities (go-bindata etc.), protobuf compilers and linter/validators.

[rsc]

@iand that's not what vendor is defined to do. Vendor only copies the specific packages needed by the module, not everything recursively reachable from go.mod (a far larger set). To force vendoring of some set of additional packages, I would suggest putting a file tools.go into the root of the module that says:

// Imports of tools we want "vgo vendor" to include.

// +build tools

package tools

import (
    _ "rsc.io/2fa"
)

You wouldn't put _ "golang.org/x/tools" though, since that's not a command. You want to specify exactly the pieces that you need to be able to build/run.

It's important that we have some indication that these are not just unused go.mod entries, since we also want to be able to diagnose unused go.mod entries. :-)

[gopherbot]

Change https://golang.org/cl/117258 mentions this issue: cmd/go/internal/vgo: process packages for all build tags in vendor