cmd/go: add go mod download to download modules

[sgnn7]

What version of Go are you using (go version)?

go1.11beta2

Does this issue reproduce with the latest release?

Yes (1.11beta2)

What operating system and processor architecture are you using (go env)?

amd64 - linux

What did you do?

Copying just go.mod into an empty directory should allow go mod -vendor (or maybe even go mod -sync) to download the dependencies regardless of having any other files within the directory. Without this, you cannot use something like Docker to cache the dependencies independent of the codebase itself.

$ docker run -it --rm golang:1.11beta2-alpine /bin/sh
/go # apk add -u git
/go # mkdir /foo
/go # cd /foo

/foo # echo "module github.com/sgnn7/test
> 
> require (
>         cloud.google.com/go v0.25.0 // indirect
> )
> " > go.mod

/foo # cat go.mod
module github.com/sgnn7/test

require (
        cloud.google.com/go v0.25.0 // indirect
)

/foo # go mod -vendor
go: finding cloud.google.com/go v0.25.0
go: no dependencies to vendor

/foo # go mod -sync
/foo # # Nothing

What did you expect to see?

go mod should have downloaded go.mod dependencies based on go.mod content regardless if there's code in the repo or not.

What did you see instead?

Neither go mod -vendor nor go mod -sync downloaded anything.

[bcmills]

Without this, you cannot use something like Docker to cache the dependencies independent of the codebase itself.

Can you include some more detail about your use-case? (Why is that a thing you want to do?)

[sgnn7]

@bcmills Sure.

Let's say that you have your code build a Docker image on every git change (most pipelines do this and is probably the biggest use case).

• With dep you could create a Docker layer of cached dependencies by just copying Gopkg.* and fetching the dependencies after which you then copy the full repo and build the project (see here for the original issue for dep on this). This meant that unless you changed the Gopkg files, no dependency re-download was ever needed, making builds with non-dependency changes quick and efficient.
• With current go mod, you cannot just copy go.mod and go.sum and then run go mod -vendor to get this feature parity since it just doesn't download anything without Go files in the repo, forcing the Docker layer to be cache-busted on the smallest of changes in the repo (e.g. README), causing re-download of all dependencies every time which is both slow and extremely inefficient.

[rsc]

Vendor is tailored for the dependencies of a specific module,
not warming a cache.

Go modules have an actual cache, separate from the vendor directory.
If you want to warm that cache in a base docker image, that would be:

go list -e $(go list -m all)

That will trigger a download of every module from go.mod
(and their dependencies) into the GOPATH/src/mod cache,
which should make those downloads available to higher
docker layers, all without using the vendor directory.

[sgnn7]

@rsc Hmm.. I'll give that a try. Is there a reason why this isn't a base go mod command?

[sgnn7]

@rsc: Your cache warming suggestion doesn't work out of the box. The fixed version is below:

go list $(go list -m all 2>/dev/null)

Edit: Nope that doesn't work either fully. This should be working though (may need escaping for Dockerfile):

 go list -e $(go list -m all 2>/dev/null | awk '{print $1}')

^ this is really not something that should be found in Dockerfiles

Edit 2 Third iteration that is the least messy so far:

go list -e $(go list -f '{{.Path}}' -m all 2>/dev/null)

IMO, If this seems like something that should be used often, it probably needs to get into the main CLI. As a sidenote to this hack, go list has no distinction in docs as to why it is the only one that can download modules vs -sync and -vendor.

[rsc]

All the commands downloads modules as needed.
sync and vendor are defined to chase down only the package-granularity dependencies of the packages in the main module.
If there are no packages in the main module, they chase down nothing.
List on the other hand can be asked directly about dependency packages,
and answering requires downloading them to look at the code.
It's certainly not a direct answer, but it should work.

That said, it seems to me that go get -d -m asks for exactly this
operation and fails to deliver it (the -d is ignored with -m).
We should fix that.