Verify correctness of service discovery for `metadata', `metadata.google.internal`, etc. on GKE and AWS

[briansmith]

See kubernetes/kubernetes#8512 (comment):

The metadata service is the de facto standard for distributing short-lived creds to apps running on EC2 (IAM roles) or GCE (scoped compute service accounts), and SDKs from both support this very well.

If that is still true, then we can't safely do DNS resolution for these hostnames from the controller's Destination service's pods, because the metadata returned would be the metadata intended for the Destination service's node, not the node that the proxied pod is running on.

See also kubernetes/kubernetes#8867.

/cc @olix0r @adleong
.

[briansmith]

This, #62, #366, and #384 all make me think that we should do DNS resolution within the proxy, and not within the Destination service.

Specifically, I think we need to do something like this: When we see a partially-qualified name like "metadata" or "metadata.$namespace", we need to use DNS to resolve both that name and "metadata.$namespace.svc.$zone". If and only if they resolve to the same IP address then subscribe to the name using the control plane's Destination service. Otherwise, use the IP addresses in the DNS response received (and probably skip our internal load balancing).

[briansmith]

The GCE documentation for the metadata service is https://cloud.google.com/compute/docs/storing-retrieving-metadata#querying.

The documentation for AWS is at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-data-retrieval. The stack overflow answer at https://stackoverflow.com/a/42315582 is very helpful to understand that our service discovery logic probably needs to consider link-local addresses specially.

[briansmith]

In Conduit 0.3, "metadata" won't resolve to the right thing (it will resolve as if it were "metadata.$namespace.svc"), but "metadata.google.internal" should work correctly.

I'll tentatively propose that we make "metadata" (IIUC, the deprecated legacy name for "metadata.google.internal") work in 0.4.