dep cannot vendor dependencies that do not have Go source code

[davecheney]

What version of dep are you using (dep version)?

What dep command did you run?

I am trying to convert a project that uses glide to vendor two repositories containing protocol buffer definitions. The abstract from glide.yaml is

hash: 2f5e7adae8a174aebf478c2aac823837001c82d40f647fd9da15be270739057c
updated: 2017-10-23T16:03:08.007589083-04:00
imports:
- name: github.com/envoyproxy/data-plane-api
  version: 5b29f002b44f3f6e3921bd3b4535aeb91f89e84a
- name: github.com/googleapis/googleapis
  version: 5c6df0cd18c6a429eab739fb711c27f6e1393366
- name: github.com/golang/protobuf
  version: ab9f9a6dab164b7d1246e0e688b0ab7b94d8553e
  subpackages:
  - jsonpb
  - proto
  - protoc-gen-go/descriptor
  - ptypes
  - ptypes/any
  - ptypes/duration
  - ptypes/struct
  - ptypes/timestamp
  - ptypes/wrappers

The two repositories in question are github.com/envoyproxy/data-plane-api and github.com/googleapis/googleapis. When I try to add

+
+[[constraint]]
+  name="github.com/envoyproxy/data-plane-api"
+  revision="5b29f002b44f3f6e3921bd3b4535aeb91f89e84a"
+
+[[constraint]]
+  name="github.com/googleapis/googleapis"
+  revision="5c6df0cd18c6a429eab739fb711c27f6e1393366"

dep ensure refuses to vendor the repositories

What did you expect to see?

those two repositories vendored into vendor/

What did you see instead?

% dep ensure
ensure Solve(): No versions of github.com/envoyproxy/data-plane-api met constraints:
        5b29f002b44f3f6e3921bd3b4535aeb91f89e84a: Could not introduce github.com/envoyproxy/data-plane-api@5b29f002b44f3f6e3921bd3b4535aeb91f89e84a, as its subpackage github.com/envoyproxy/data-plane-api does not contain usable Go code (*build.NoGoError).. (Package is required by (root).)
        master: Could not introduce github.com/envoyproxy/data-plane-api@master, as its subpackage github.com/envoyproxy/data-plane-api does not contain usable Go code (*build.NoGoError).. (Package is required by (root).)
        typo: Could not introduce github.com/envoyproxy/data-plane-api@typo, as it is not allowed by constraint 5b29f002b44f3f6e3921bd3b4535aeb91f89e84a from project github.com/heptio/zzz.

[markcampanelli]

I hit same issue when trying to use dep to manage some non-Go build tools.

[markuswustenberg]

This also happens when fetching e.g. github.com/docker/docker, which doesn't have Go code in the root (since v0.9.1 at least):

$ dep ensure -v -add github.com/docker/docker       
Fetching sources...
(1/1) github.com/docker/docker

Root project is "code.uber.internal/infra/dep-no-go-code"
 1 transitively valid internal packages
 1 external packages imported from 1 projects
(0)   ✓ select (root)
(1)	? attempt github.com/docker/docker with 1 pkgs; 206 versions to try
(1)	    try github.com/docker/docker@v1.13.1
(2)	✗   github.com/docker/docker at v1.13.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.13.0
(2)	✗   github.com/docker/docker at v1.13.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.12.6
(2)	✗   github.com/docker/docker at v1.12.6 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.12.5
(2)	✗   github.com/docker/docker at v1.12.5 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.12.4
(2)	✗   github.com/docker/docker at v1.12.4 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.12.3
(2)	✗   github.com/docker/docker at v1.12.3 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.12.2
(2)	✗   github.com/docker/docker at v1.12.2 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.12.1
(2)	✗   github.com/docker/docker at v1.12.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.12.0
(2)	✗   github.com/docker/docker at v1.12.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.11.2
(2)	✗   github.com/docker/docker at v1.11.2 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.11.1
(2)	✗   github.com/docker/docker at v1.11.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.11.0
(2)	✗   github.com/docker/docker at v1.11.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.10.3
(2)	✗   github.com/docker/docker at v1.10.3 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.10.2
(2)	✗   github.com/docker/docker at v1.10.2 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.10.1
(2)	✗   github.com/docker/docker at v1.10.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.10.0
(2)	✗   github.com/docker/docker at v1.10.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.9.1
(2)	✗   github.com/docker/docker at v1.9.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.9.0
(2)	✗   github.com/docker/docker at v1.9.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.8.3
(2)	✗   github.com/docker/docker at v1.8.3 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.8.2
(2)	✗   github.com/docker/docker at v1.8.2 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.8.1
(2)	✗   github.com/docker/docker at v1.8.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.8.0
(2)	✗   github.com/docker/docker at v1.8.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.7.1
(2)	✗   github.com/docker/docker at v1.7.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.7.0
(2)	✗   github.com/docker/docker at v1.7.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.6.2
(2)	✗   github.com/docker/docker at v1.6.2 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.6.1
(2)	✗   github.com/docker/docker at v1.6.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.6.0
(2)	✗   github.com/docker/docker at v1.6.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.5.0
(2)	✗   github.com/docker/docker at v1.5.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.4.1
(2)	✗   github.com/docker/docker at v1.4.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.4.0
(2)	✗   github.com/docker/docker at v1.4.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.3.3
(2)	✗   github.com/docker/docker at v1.3.3 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.3.2
(2)	✗   github.com/docker/docker at v1.3.2 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.3.1
(2)	✗   github.com/docker/docker at v1.3.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.3.0
(2)	✗   github.com/docker/docker at v1.3.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.2.0
(2)	✗   github.com/docker/docker at v1.2.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.1.2
(2)	✗   github.com/docker/docker at v1.1.2 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.1.1
(2)	✗   github.com/docker/docker at v1.1.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.1.0
(2)	✗   github.com/docker/docker at v1.1.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.0.1
(2)	✗   github.com/docker/docker at v1.0.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.0.0
(2)	✗   github.com/docker/docker at v1.0.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v0.12.0
(2)	✗   github.com/docker/docker at v0.12.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v0.11.1
(2)	✗   github.com/docker/docker at v0.11.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v0.11.0
(2)	✗   github.com/docker/docker at v0.11.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v0.10.0
(2)	✗   github.com/docker/docker at v0.10.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v0.9.1
(1)	✓ select github.com/docker/docker@v0.9.1 w/1 pkgs

Version:

$ dep version
dep:
 version     : devel
 build date  : 
 git hash    : 
 go version  : go1.9.1
 go compiler : gc
 platform    : darwin/amd64

[sdboyer]

yeah...this is tricky. i'm considering the possibility that we change the rules on required to not complain if the named packages lack Go code. alternatively, we could relax the "no go code" rule in general and consider any files that may possibly be usable to Go good enough to make a package "valid" for import. that would open up the possibility of false positives with real imports...but that seems a lot less likely to be a real problem than the false negatives being described here.

the only other alternative i can think of right now is something like #269, which is a whole other can of worms.

@markuswustenberg

This also happens when fetching e.g. github.com/docker/docker, which doesn't have Go code in the root (since v0.9.1 at least):

yeah, the design here implicitly expects that you give -add a package - not necessarily a project root - that is actually importable. this is one of those confounded expectations arising from bumping up against the model, again - folks are thinking, "i want to grab this down and just have it there so i can reference stuff from it," but dep's model is "you don't actually use these, so we're gonna get rid of it."

this is gonna be a point of friction for a while, unfortunately. the easiest way to make e.g. your editor's autocomplete work is to just trick it by also having that project on GOPATH - and by the time that we get around to adding the new storage area, loosely outlined here, i think most of this pain should go away. but it's gonna be a while.

[markuswustenberg]

Thanks for your answer @sdboyer.

alternatively, we could relax the "no go code" rule in general and consider any files that may possibly be usable to Go good enough to make a package "valid" for import.

In my view, any arbitrary binary is potentially usable by Go code, so that's seems a bit of an odd restriction. Maybe there could just be an option to override this behaviour?

yeah, the design here implicitly expects that you give -add a package - not necessarily a project root - that is actually importable

In this case, I'm trying to grab the docker client which is under github.com/docker/docker/client, but dep won't let me import that directly. Is there any way to get this working with the current tools?

[sdboyer]

In my view, any arbitrary binary is potentially usable by Go code, so that's seems a bit of an odd restriction.

the key here is the definition of "usable" - dep is, for now, primarily concerned with what is necessary to compile Go code. not generate; not run. it's nice when we can also capture those things, but for now, they're not our primary objective. that puts arbitrary binaries out of scope, unless i've missed something quite big about extensible compiler capabilities. (entirely possible!)

In this case, I'm trying to grab the docker client which is under github.com/docker/docker/client, but dep won't let me import that directly. Is there any way to get this working with the current tools?

dep ensure -add github.com/docker/docker/client

[discordianfish]

I have a related issue. Not sure if I should open a new issue for that though:
I'm building a Dockerfile and I want to add Gopkg.* first, then run dep ensure before I add the code in a next step. That's a common pattern which allows Docker to cache the dependency-install step, so that I don't have to run this every time I change the code. Due to the 'no go code' rule this isn't possible when using dep.

[sdboyer]

@discordianfish this issue is specifically about the case where there's no Go code in a dependency, which is different from having no Go code in the current project. dep ensure -vendor-only should cover your case.

[discordianfish]

Oh perfect, thanks! Sorry for the noise.

[davecheney]

Have a look at github.com/heptio/contour. I’m using the pattern you describe and it is working well.

…

On 11 Nov 2017, at 01:40, discordianfish ***@***.***> wrote: Oh perfect, thanks! Sorry for the noise. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[jhayotte]

Hi @sdboyer,
I tried out to migrate from Glide to Dep and dependencies including a list of protofile like https://github.com/googleapis/googleapis/tree/master/google/api cannot be vendored.

Would be nice to have a workaround in the meantime. Many people are trying to use dep and are blocked by this issue :/

[sdboyer]

yeah, i don't see any problems with expanding the definition of required to cover this case. so we have a path to a solution - just, need the elbow grease to get it done.

[krisnova]

Happy to lend some commits while I am enjoying my time of being unemployed if you want to assign this or any other issues to me

[jhayotte]

Hi @sdboyer,

I can contribute to this issue as well. We just need to clarify your expectations.

Do you wish that we remove the constraint looking for *.go files? (IMO that's what we should do) or do we handle it as a warning?

dep/gps/pkgtree/pkgtree.go

Lines 219 to 226 in 13df556

	gofiles, err := filepath.Glob(filepath.Join(p.Dir, "*.go"))
	if err != nil {
	return err
	}
	if len(gofiles) == 0 {
	return &build.NoGoError{Dir: p.Dir}
	}