envoy per gateway #404

arkodg · 2022-09-21T23:20:45Z

create a xdsIR per gateway
create a infraIR per gateway
use the gateway namespace-name as the key for above IRs
populate the envoy bootstrap config with a service cluster val that matches the IR key
populate the xds server snapshot for each xds request based on the service cluster

Fixes: #349

Signed-off-by: Arko Dasgupta [email protected]

internal/gatewayapi/translator.go

arkodg · 2022-09-23T02:12:04Z

attaching some test logs done with 2 gateways

2 Gateway resources

$ kubectl get gateway/eg -o yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"Gateway","metadata":{"annotations":{},"name":"eg","namespace":"default"},"spec":{"gatewayClassName":"eg","listeners":[{"name":"http","port":8080,"protocol":"HTTP"}]}}
  creationTimestamp: "2022-09-23T02:00:18Z"
  generation: 1
  name: eg
  namespace: default
  resourceVersion: "1007"
  uid: c1b84856-c1f8-4126-892d-f8762ed9da44
spec:
  gatewayClassName: eg
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: http
    port: 8080
    protocol: HTTP
status:
  conditions:
  - lastTransitionTime: "2022-09-23T02:00:18Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 1
    reason: Scheduled
    status: "True"
    type: Scheduled
  - lastTransitionTime: "2022-09-23T02:00:18Z"
    message: No addresses have been assigned to the Gateway
    observedGeneration: 1
    reason: AddressNotAssigned
    status: "False"
    type: Ready
  listeners:
  - attachedRoutes: 1
    conditions:
    - lastTransitionTime: "2022-09-23T02:00:22Z"
      message: Listener is ready
      observedGeneration: 1
      reason: Ready
      status: "True"
      type: Ready
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute

$ kubectl get gateway/eg-2 -o yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"Gateway","metadata":{"annotations":{},"name":"eg-2","namespace":"default"},"spec":{"gatewayClassName":"eg","listeners":[{"hostname":"www.example-2.com","name":"http-2","port":8080,"protocol":"HTTP"}]}}
  creationTimestamp: "2022-09-23T02:00:53Z"
  generation: 1
  name: eg-2
  namespace: default
  resourceVersion: "1008"
  uid: c6eb611a-09cc-4880-8b33-e8535344e344
spec:
  gatewayClassName: eg
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    hostname: www.example-2.com
    name: http-2
    port: 8080
    protocol: HTTP
status:
  conditions:
  - lastTransitionTime: "2022-09-23T02:00:53Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 1
    reason: Scheduled
    status: "True"
    type: Scheduled
  - lastTransitionTime: "2022-09-23T02:00:53Z"
    message: No addresses have been assigned to the Gateway
    observedGeneration: 1
    reason: AddressNotAssigned
    status: "False"
    type: Ready
  listeners:
  - attachedRoutes: 1
    conditions:
    - lastTransitionTime: "2022-09-23T02:00:53Z"
      message: Listener is ready
      observedGeneration: 1
      reason: Ready
      status: "True"
      type: Ready
    name: http-2
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute

2 HTTPRoute resources

$ kubectl get httproute/httpbin -o yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"HTTPRoute","metadata":{"annotations":{},"name":"httpbin","namespace":"default"},"spec":{"hostnames":["www.example.com"],"parentRefs":[{"name":"eg"}],"rules":[{"backendRefs":[{"group":"","kind":"Service","name":"httpbin","port":80,"weight":1}],"matches":[{"path":{"type":"PathPrefix","value":"/"}}]}]}}
  creationTimestamp: "2022-09-23T02:00:22Z"
  generation: 3
  name: httpbin
  namespace: default
  resourceVersion: "1006"
  uid: aafe12cc-9ff7-489c-a9e8-8129aa2c4d80
spec:
  hostnames:
  - www.example.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: httpbin
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /
status:
  parents:
  - conditions:
    - lastTransitionTime: "2022-09-23T02:00:22Z"
      message: Route is accepted
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: eg
  - conditions:
    - lastTransitionTime: "2022-09-23T02:01:00Z"
      message: Route is accepted
      observedGeneration: 2
      reason: Accepted
      status: "True"
      type: Accepted
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: eg-2      
$ kubectl get httproute/httpbin-2 -o yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"HTTPRoute","metadata":{"annotations":{},"name":"httpbin-2","namespace":"default"},"spec":{"hostnames":["www.example-2.com"],"parentRefs":[{"name":"eg-2"}],"rules":[{"backendRefs":[{"group":"","kind":"Service","name":"httpbin","port":80,"weight":1}],"matches":[{"path":{"type":"PathPrefix","value":"/"}}]}]}}
  creationTimestamp: "2022-09-23T02:01:22Z"
  generation: 1
  name: httpbin-2
  namespace: default
  resourceVersion: "993"
  uid: 47ff8327-7f01-45b7-8f9b-6236fe2b8a9a
spec:
  hostnames:
  - www.example-2.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg-2
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: httpbin
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /
status:
  parents:
  - conditions:
    - lastTransitionTime: "2022-09-23T02:01:22Z"
      message: Route is accepted
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: eg-2

Port forward to the 2 envoy services

$ kubectl -n envoy-gateway-system port-forward service/envoy-default-eg 8888:8080 &
$ kubectl -n envoy-gateway-system port-forward service/envoy-default-eg-2 8889:8080 &

Can reach www.example.com from service/envoy-default-eg but not www.example-2.com

$ curl --verbose --header "Host: www.example.com" http://localhost:8888/get
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8888 (#0)
> GET /get HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
Handling connection for 8888
< HTTP/1.1 200 OK
< server: envoy
< date: Fri, 23 Sep 2022 02:06:09 GMT
< content-type: application/json
< content-length: 238
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 1
< 
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "www.example.com", 
    "User-Agent": "curl/7.64.1", 
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  }, 
  "origin": "10.1.0.12", 
  "url": "http://www.example.com/get"
}
* Connection #0 to host localhost left intact
* Closing connection 0

$ curl --verbose --header "Host: www.example-2.com" http://localhost:8888/get
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8888 (#0)
> GET /get HTTP/1.1
> Host: www.example-2.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
Handling connection for 8888
< HTTP/1.1 404 Not Found
< date: Fri, 23 Sep 2022 02:06:28 GMT
< server: envoy
< content-length: 0
< 
* Connection #0 to host localhost left intact
* Closing connection 0

Can reach www.example-2.com from service/envoy-default-eg-2 but not www.example.com

$ curl --verbose --header "Host: www.example.com" http://localhost:8889/get
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8889 (#0)
> GET /get HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
Handling connection for 8889
< HTTP/1.1 404 Not Found
< date: Fri, 23 Sep 2022 02:06:35 GMT
< server: envoy
< content-length: 0
< 
* Connection #0 to host localhost left intact
* Closing connection 0
$ curl --verbose --header "Host: www.example-2.com" http://localhost:8889/get
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8889 (#0)
> GET /get HTTP/1.1
> Host: www.example-2.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
Handling connection for 8889
< HTTP/1.1 200 OK
< server: envoy
< date: Fri, 23 Sep 2022 02:06:40 GMT
< content-type: application/json
< content-length: 242
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 1
< 
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "www.example-2.com", 
    "User-Agent": "curl/7.64.1", 
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  }, 
  "origin": "10.1.0.13", 
  "url": "http://www.example-2.com/get"
}
* Connection #0 to host localhost left intact
* Closing connection 0

arkodg · 2022-09-23T23:21:39Z

reviewers please note that this PR doesnt handle delete of Gateway resources, will raise an issue to track it once this PR is in, and will fix be fixed in a follow up commit.

codecov-commenter · 2022-09-26T23:14:41Z

Codecov Report

Merging #404 (5f5f281) into main (96e7f54) will decrease coverage by 0.31%.
The diff coverage is 70.22%.

@@            Coverage Diff             @@
##             main     #404      +/-   ##
==========================================
- Coverage   60.46%   60.15%   -0.32%     
==========================================
  Files          40       40              
  Lines        4323     4327       +4     
==========================================
- Hits         2614     2603      -11     
- Misses       1560     1574      +14     
- Partials      149      150       +1

Impacted Files	Coverage Δ
internal/cmd/xdstest.go	`3.65% <0.00%> (ø)`
internal/envoygateway/config/config.go	`0.00% <ø> (ø)`
internal/gatewayapi/runner/runner.go	`55.78% <0.00%> (-5.92%)`	⬇️
internal/infrastructure/kubernetes/infra.go	`68.49% <0.00%> (ø)`
internal/message/types.go	`92.45% <0.00%> (-7.55%)`	⬇️
internal/xds/server/runner/runner.go	`24.71% <0.00%> (ø)`
internal/xds/translator/runner/runner.go	`72.00% <45.45%> (-13.19%)`	⬇️
internal/infrastructure/kubernetes/service.go	`72.15% <75.00%> (+0.72%)`	⬆️
internal/crypto/certgen.go	`77.21% <100.00%> (ø)`
internal/gatewayapi/translator.go	`86.07% <100.00%> (+0.14%)`	⬆️
... and 8 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

danehans · 2022-09-27T18:32:25Z

xref: #380

danehans · 2022-09-27T18:34:31Z

xref: #382

arkodg · 2022-09-27T19:58:14Z

@danehans I cherry-picked #435 on top of this PR and CI now passes

* create a xdsIR per gateway * create a infraIR per gateway * use the gateway namespace-name as the key for above IRs * populate the envoy bootstrap config with a node id that matches the IR key * populate the xds server snapshot for each xds request based on the node id Fixes: envoyproxy#349 Signed-off-by: Arko Dasgupta <[email protected]>

Signed-off-by: Arko Dasgupta <[email protected]>

Signed-off-by: danehans <[email protected]>

Signed-off-by: Arko Dasgupta <[email protected]>

danehans · 2022-09-28T00:45:08Z

I tested this PR with #438 and the additional conformance tests pass.

danehans · 2022-09-28T00:55:57Z

internal/crypto/certgen.go

@@ -21,7 +21,7 @@ const (
 	DefaultEnvoyGatewayDNSPrefix = config.EnvoyGatewayServiceName

 	// DefaultEnvoyDNSPrefix defines the default Envoy DNS prefix.
-	DefaultEnvoyDNSPrefix = config.EnvoyServiceName
+	DefaultEnvoyDNSPrefix = "*"


Created #440 to develop a more secure control plane auth solution.

danehans · 2022-09-28T01:05:43Z

internal/infrastructure/kubernetes/deployment.go

+func expectedDeploymentName(proxyName string) string {
+	return fmt.Sprintf("%s-%s", config.EnvoyDeploymentPrefix, proxyName)
+}
+


This reverts commit 231e4fb.

This reverts commit 231e4fb. Signed-off-by: danehans <[email protected]>

arkodg requested a review from a team as a code owner September 21, 2022 23:20

arkodg marked this pull request as draft September 21, 2022 23:21

arkodg added this to the 0.2.0-rc2 milestone Sep 21, 2022

skriss reviewed Sep 22, 2022

View reviewed changes

internal/gatewayapi/translator.go Outdated Show resolved Hide resolved

danehans mentioned this pull request Sep 22, 2022

Adds Support for Control Plane TLS #208

Merged

arkodg force-pushed the ir-per-gateway branch from 5e8ec0e to 3435da1 Compare September 22, 2022 22:34

arkodg requested a review from skriss September 23, 2022 02:12

arkodg marked this pull request as ready for review September 23, 2022 02:13

arkodg requested a review from a team September 23, 2022 19:33

arkodg force-pushed the ir-per-gateway branch from 1a0d520 to 15af025 Compare September 23, 2022 23:34

arkodg mentioned this pull request Sep 26, 2022

Handle IR deletes #423

Closed

arkodg force-pushed the ir-per-gateway branch from 15af025 to c171031 Compare September 26, 2022 22:38

arkodg force-pushed the ir-per-gateway branch 2 times, most recently from 5f5f281 to f79369d Compare September 27, 2022 00:46

This was referenced Sep 27, 2022

Updates Infra IR to Support Multiple Envoy Services #389

Closed

HTTPRouteListenerHostnameMatching Conformance Test Fails #434

Closed

arkodg force-pushed the ir-per-gateway branch from f79369d to 5bd432b Compare September 27, 2022 19:47

arkodg and others added 10 commits September 27, 2022 14:48

append proxy name to infra resources

7374ff7

Signed-off-by: Arko Dasgupta <[email protected]>

fix provider infra service and deployment names

214b980

Signed-off-by: Arko Dasgupta <[email protected]>

1:1 b/w service-cluster and irKey (gwNs-gwName)

20c1697

Signed-off-by: Arko Dasgupta <[email protected]>

detect conflicts per gateway instead of across all

e9a52f4

Signed-off-by: Arko Dasgupta <[email protected]>

infra found ports per gateway

1e25b90

Signed-off-by: Arko Dasgupta <[email protected]>

rm ()

eeb00c8

Signed-off-by: Arko Dasgupta <[email protected]>

Rebased and Fixed Test Failures

e9a56e8

Signed-off-by: danehans <[email protected]>

Updates Envoy Cert as Wildcard Hostname

96b4693

Signed-off-by: danehans <[email protected]>

sort to add stablity in translated o/p

886fba3

Signed-off-by: Arko Dasgupta <[email protected]>

lint

41dc3c1

Signed-off-by: Arko Dasgupta <[email protected]>

arkodg force-pushed the ir-per-gateway branch from 5bd432b to 41dc3c1 Compare September 27, 2022 21:48

This was referenced Sep 28, 2022

Refine Control Plane Auth #440

Closed

Hash the Names of Managed Infra Resources #441

Closed

danehans approved these changes Sep 28, 2022

View reviewed changes

arkodg merged commit 231e4fb into envoyproxy:main Sep 28, 2022

This was referenced Sep 28, 2022

Adds Gateway Watch to HTTPRoute Controller #403

Merged

Adds GatewayClass Watch to Gateway Controller #412

Closed

This was referenced Sep 28, 2022

Add support for multiple gateways #380

Closed

Xds Translator Incorrectly Creates Listener #382

Closed

danehans added a commit that referenced this pull request Sep 28, 2022

Revert "envoy per gateway (#404)"

886ab4b

This reverts commit 231e4fb.

danehans mentioned this pull request Sep 28, 2022

Revert "envoy per gateway" #445

Closed

danehans added a commit that referenced this pull request Sep 28, 2022

Revert "envoy per gateway (#404)"

fffe65a

This reverts commit 231e4fb. Signed-off-by: danehans <[email protected]>

arkodg mentioned this pull request Mar 27, 2023

Merge Gateways on to a single EnvoyProxy fleet #1230

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

envoy per gateway #404

envoy per gateway #404

Uh oh!

arkodg commented Sep 21, 2022 •

edited

Loading

Uh oh!

Uh oh!

arkodg commented Sep 23, 2022

Uh oh!

arkodg commented Sep 23, 2022

Uh oh!

codecov-commenter commented Sep 26, 2022 •

edited

Loading

Uh oh!

danehans commented Sep 27, 2022

Uh oh!

danehans commented Sep 27, 2022

Uh oh!

arkodg commented Sep 27, 2022

Uh oh!

danehans commented Sep 28, 2022

Uh oh!

danehans Sep 28, 2022

Uh oh!

danehans Sep 28, 2022

Uh oh!

Uh oh!

envoy per gateway #404

envoy per gateway #404

Uh oh!

Conversation

arkodg commented Sep 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

arkodg commented Sep 23, 2022

Uh oh!

arkodg commented Sep 23, 2022

Uh oh!

codecov-commenter commented Sep 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

danehans commented Sep 27, 2022

Uh oh!

danehans commented Sep 27, 2022

Uh oh!

arkodg commented Sep 27, 2022

Uh oh!

danehans commented Sep 28, 2022

Uh oh!

danehans Sep 28, 2022

Choose a reason for hiding this comment

Uh oh!

danehans Sep 28, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

arkodg commented Sep 21, 2022 •

edited

Loading

codecov-commenter commented Sep 26, 2022 •

edited

Loading