Evolve AntreaProxy with framework and feature updates #7371

hongliangl · 2025-08-14T07:36:16Z

For #6940

This commit brings AntreaProxy:

Add serviceProxyHealthy field to Service health check response in AntreaProxy.
Add healthz server serving on port 10256 to AntreaProxy, which is a replacement
of kube-proxy health server serving on 10256.
Add support of feature gate PreferSameTrafficDistribution in AntreaProxy. Refer
to this https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/3015-prefer-same-node.
Remove Endpoints API support in AntreaProxy.
Align the code third_party/proxy with K8s 1.34.2.

Dependencies:

Refine AntreaProxy framework to eliminate duplicated processing in dual-stack #7380

antoninbas

some small initial comments, I will review #7380 more in details first

build/charts/antrea/values.yaml

cmd/antrea-agent/agent.go

pkg/agent/proxy/proxier.go

antoninbas · 2025-10-24T22:12:39Z

docs/feature-gates.md

-| Feature Name                  | Component          | Default | Stage | Alpha Release | Beta Release | GA Release | Extra Requirements | Notes                                         |
-| ----------------------------- | ------------------ | ------- | ----- | ------------- | ------------ | ---------- | ------------------ | --------------------------------------------- |
-| `AntreaProxy`                 | Agent              | `true`  | GA    | v0.8          | v0.11        | v1.14      | Yes                | Must be enabled for Windows.                  |
-| `EndpointSlice`               | Agent              | `true`  | GA    | v0.13.0       | v1.11        | v1.14      | Yes                |                                               |


I think that one thing that is worth pointing out is that even though we are keeping this FeatureGate around, setting it to False will have no effect any more (unlike the AntreaProxy FeatureGate for example). So I think the current warning is not sufficient for EndpointSlice. If the FeatureGate is explicitly set to False in the config, we should log an error stating that the configuration will be ignored.

How about adding this in NewProxyServer:

if !features.DefaultFeatureGate.Enabled(features.EndpointSlice) { klog.ErrorS(nil, "EndpointSlice cannot be disabled, ignoring feature gateway EndpointSlice setting to false") }

Can you add "Cannot be disabled" in the Notes (last column of this table)?

Since we can use LockToDefault, we don't need this note anymore?

We should still mention it, as otherwise there is no way for a user to tell simply by looking at the documentation

ping for this

Added Locked to true (cannot be disabled). to column Notes.

third_party/proxy/metaproxier/meta_proxier.go

hack/verify-spelling.sh

pkg/apis/ports.go

cmd/antrea-agent/options.go

pkg/agent/proxy/proxier.go

docs/feature-gates.md

hack/.spelling_failures

This commit brings AntreaProxy: - Add `serviceProxyHealthy` field to Service health check response in AntreaProxy. - Add healthz server serving on port 10256 to AntreaProxy, which is a replacement of kube-proxy health server serving on 10256. - Add support of feature gate PreferSameTrafficDistribution in AntreaProxy. Refer to this https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/3015-prefer-same-node. - Remove Endpoints API support in AntreaProxy. - Align the code `third_party/proxy` with K8s 1.34.2. Signed-off-by: Hongliang Liu <[email protected]>

antoninbas · 2025-12-01T22:13:44Z

pkg/agent/nodemanager/nodemanager.go

+// The upstream NodeManager starts its own informerFactory, which is undesirable for Antrea as we rely on a single
+// shared informerFactory for all controllers. To avoid the overhead and duplication of running an additional
+// informerFactory, Antrea provides a lightweight replacement NodeManager implementation instead.
+type NodeManager struct {


Actually we have a couple of examples of standalone filtered informers:

antrea/pkg/agent/client/endpoint_resolver.go

Lines 93 to 102 in c0f59a7

serviceInformer := corev1informers.NewFilteredServiceInformer(

kubeClient,

namespace,

informerDefaultResync,

cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc},

func(listOptions *metav1.ListOptions) {

listOptions.FieldSelector = fields.OneTermEqualSelector("metadata.name", serviceName).String()

},

)

endpointSliceInformer := discoveryv1informers.NewFilteredEndpointSliceInformer(

If you only want to watch one Node, I think it's acceptable to use a dedicated filtered informer, instead of using the existing shared informer factory. So if this was the only issue, you could use the upstream NodeManager code. However, there seems to be a key difference, which is that the upstream code panics / exits on NodeIP change. I am not sure this is the behavior we want for Antrea, or at least it requires more discussion. That behavior doesn't seem to be customizable in the upstream code.

Actually we have a couple of examples of standalone filtered informers:

Thanks for pointing out the existing standalone filtered informers. My bad, I didn't found that we have existing standalone filter informers. That's my only concern. If we have the existing usages, I'm happy to use that.

Additionally, after discussing with @tnqn yesterday, an alternative NodeManager might be a good choice.

The upstream NodeManager starts its own informerFactory, which is undesirable for Antrea as we should try to rely on a single shared informerFactory for all controllers.

At least in Antrea, only ProxyHealthServer consumes NodeManager. We may extend it in the future or replace it with upstream NodeManager one day.

I am not sure this is the behavior we want for Antrea, or at least it requires more discussion. That behavior doesn't seem to be customizable in the upstream code.

I saw that the corresponding functions are commented in upstream NodeManager for now. At least it should be safe for Antrea to introduce the upstream NodeManager.

// We exit whenever there is a change in NodeIPs detected initially, and NodeIPs received // on node watch event. if !reflect.DeepEqual(n.nodeIPs, nodeIPs) { klog.InfoS("NodeIPs changed for the node", "node", klog.KObj(node), "newNodeIPs", nodeIPs, "oldNodeIPs", n.nodeIPs) // FIXME: exit // klog.Flush() // n.exitFunc(1) }

In sum:

Upstream NodeManager: less code to maintain, but has its own informerFactory and non-configurable NodeIP-change semantics.

Antrea NodeManager: consistent with shared informerFactory and fully controllable, but requires local maintenance.

cc @antoninbas @tnqn

@antoninbas A small clarification: the upstream Kubernetes NodeManager creates its own dedicated SharedInformerFactory, as shown in L75–83 of the upstream code https://github.com/kubernetes/kubernetes/blob/ec1bf8a4f3a5f054065225dc8275c66b93310d17/pkg/proxy/node.go#L75.

It then builds a filtered Node informer from that factory, rather than using a shared factory.

Additionally, I checked all locations where SharedInformerFactory objects are used in Antrea. Currently, we only create informer factories per top-level component, such as:

./cmd/flow-aggregator/flow-aggregator.go: informerFactory := informers.NewSharedInformerFactoryWithOptions(k8sClient, informerDefaultResync, informers.WithTransform(k8s.NewTrimmer(k8s.TrimPod, k8s.TrimNode))) ./cmd/antrea-controller/controller.go: informerFactory := informers.NewSharedInformerFactoryWithOptions(client, informerDefaultResync, informers.WithTransform(k8s.NewTrimmer(k8s.TrimPod))) ./cmd/antrea-controller/controller.go: crdInformerFactory := crdinformers.NewSharedInformerFactoryWithOptions(crdClient, informerDefaultResync, crdinformers.WithTransform(k8s.NewTrimmer())) ./cmd/antrea-agent/agent.go: informerFactory := informers.NewSharedInformerFactoryWithOptions(k8sClient, informerDefaultResync, informers.WithTransform(k8s.NewTrimmer(k8s.TrimNode))) ./cmd/antrea-agent/agent.go: crdInformerFactory := crdinformers.NewSharedInformerFactoryWithOptions(crdClient, informerDefaultResync, crdinformers.WithTransform(k8s.NewTrimmer())) ./cmd/antrea-agent/agent.go: mcInformerFactoryWithNamespaceOption = mcinformers.NewSharedInformerFactoryWithOptions(mcClient, ./cmd/antrea-agent/agent.go: mcInformerFactory = mcinformers.NewSharedInformerFactoryWithOptions(mcClient, informerDefaultResync, mcinformers.WithTransform(k8s.NewTrimmer()))

antoninbas · 2025-12-01T22:21:44Z

third_party/proxy/runner/bounded_frequency_runner.go

+
+Modifies:
+
+- Remove `defer close(bfr.run)` from `BoundedFrequencyRunner.Loop()`. AntreaProxy may still call `BoundedFrequencyRunner.Run()`


Hmm, the upstream code doesn't make sense to me... The reader goroutine should never be responsible for closing the channel...
Do you know why they don't have the same panic problem upstream? How is the architecture different?

The dedicated reason is that they pass wait.NeverStop that will never exit to Loop: https://github.com/kubernetes/kubernetes/blob/9998041e0ffe0dd3f2abab3b9f95505c4402bf14/pkg/proxy/nftables/proxier.go#L760.

hongliangl · 2025-12-02T02:57:23Z

I pushed another commit using the K8s upstream NodeManager.

luolanzone

two nits.
btw, I noticed there are race condition failures in the unit tests, please check and fix it, and there are also following error and warning, can you double check?

 I1202 02:59:53.842500   31953 shared_informer.go:648] "Warning: the specified resyncPeriod is invalid because this shared informer doesn't support resyncing" desired="1m0s"
I1202 02:59:53.842567   31953 shared_informer.go:648] "Warning: the specified resyncPeriod is invalid because this shared informer doesn't support resyncing" desired="1m0s"

E1202 02:59:53.944341   31953 proxier.go:1645] "Healthz server failed" err="failed to start proxy healthz on 0.0.0.0:10256: listen tcp 0.0.0.0:10256: bind: address already in use"

pkg/agent/proxy/proxier.go

pkg/config/agent/config.go

antoninbas

Only a few nits, otherwise LGTM

cmd/antrea-agent/agent.go

cmd/antrea-agent/options.go

antoninbas · 2025-12-02T22:14:13Z

docs/feature-gates.md

-| Feature Name                  | Component          | Default | Stage | Alpha Release | Beta Release | GA Release | Extra Requirements | Notes                                         |
-| ----------------------------- | ------------------ | ------- | ----- | ------------- | ------------ | ---------- | ------------------ | --------------------------------------------- |
-| `AntreaProxy`                 | Agent              | `true`  | GA    | v0.8          | v0.11        | v1.14      | Yes                | Must be enabled for Windows.                  |
-| `EndpointSlice`               | Agent              | `true`  | GA    | v0.13.0       | v1.11        | v1.14      | Yes                |                                               |


ping for this

antoninbas · 2025-12-02T22:15:06Z

docs/network-requirements.md

 [3] _Antrea automatically adds the firewall rules to allow the WireGuard packets
 (starting from v2.4), so the manual configuration on the host is not needed._
+
+[4] _The default value is 10256, but it can be overridden in the antrea-agent


Also indicate that it can be completely disabled with antreaProxy.disableServiceHealthCheckServer

Updated to

[4] _The default value is 10256, but it can be overridden in the antrea-agent configuration `antreaProxy.serviceHealthCheckServerBindAddress`. It is used only for external load balancer health checks. If `antreaProxy.disableServiceHealthCheckServer` is set `true`, the health check server listening on the port will be disabled._

luolanzone · 2025-12-03T02:35:46Z

/test-all

luolanzone

LGTM

Signed-off-by: Hongliang Liu <[email protected]>

hongliangl · 2025-12-03T03:25:58Z

/test-all

hongliangl · 2025-12-03T03:26:57Z

@antoninbas @tnqn @luolanzone Thanks for reviewing this PR!

* Evolve AntreaProxy with framework and feature updates This commit brings AntreaProxy: - Add `serviceProxyHealthy` field to Service health check response in AntreaProxy. - Add healthz server serving on port 10256 to AntreaProxy, which is a replacement of kube-proxy health server serving on 10256. - Add support of feature gate PreferSameTrafficDistribution in AntreaProxy. Refer to this https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/3015-prefer-same-node. - Remove Endpoints API support in AntreaProxy. - Align the code `third_party/proxy` with K8s 1.34.2. Signed-off-by: Hongliang Liu <[email protected]>

hongliangl force-pushed the 20250802-update-3-party branch 10 times, most recently from 8302c67 to a83213c Compare August 20, 2025 07:23

hongliangl changed the title ~~[WIP] Update code in third_party/proxy~~ Evolve AntreaProxy with framework and feature updates Aug 20, 2025

hongliangl requested review from antoninbas, jianjuns, luolanzone and tnqn August 20, 2025 07:24

hongliangl added the area/proxy Issues or PRs related to proxy functions in Antrea label Aug 20, 2025

hongliangl added this to the Antrea v2.5 release milestone Aug 20, 2025

hongliangl marked this pull request as ready for review August 20, 2025 07:24

hongliangl force-pushed the 20250802-update-3-party branch from a83213c to 1ba593f Compare August 20, 2025 09:59

hongliangl mentioned this pull request Oct 21, 2025

Remove feature gates AntreaProxy and NodePortLocal from chart / standard manifests #7505

Merged

hongliangl force-pushed the 20250802-update-3-party branch 2 times, most recently from 88eceb7 to b9dc7dd Compare October 23, 2025 08:35

antoninbas reviewed Oct 24, 2025

View reviewed changes

hongliangl force-pushed the 20250802-update-3-party branch 4 times, most recently from 75a7bea to 32a4415 Compare October 31, 2025 15:42

luolanzone reviewed Nov 3, 2025

View reviewed changes

hongliangl force-pushed the 20250802-update-3-party branch 2 times, most recently from 6e17422 to 069839c Compare November 6, 2025 07:34

hongliangl removed the request for review from jianjuns December 1, 2025 08:28

hongliangl force-pushed the 20250802-update-3-party branch from 434ec6b to 19e8fa9 Compare December 1, 2025 11:52

hongliangl requested review from antoninbas and luolanzone December 1, 2025 11:52

antoninbas reviewed Dec 1, 2025

View reviewed changes

hongliangl force-pushed the 20250802-update-3-party branch from 95a0efd to 3055a78 Compare December 2, 2025 02:53

hongliangl requested a review from antoninbas December 2, 2025 02:57

luolanzone reviewed Dec 2, 2025

View reviewed changes

pkg/agent/proxy/proxier.go Outdated Show resolved Hide resolved

pkg/config/agent/config.go Outdated Show resolved Hide resolved

hongliangl force-pushed the 20250802-update-3-party branch from 3055a78 to 2cd15c3 Compare December 2, 2025 12:51

hongliangl requested a review from luolanzone December 2, 2025 12:52

hongliangl force-pushed the 20250802-update-3-party branch from 2cd15c3 to 0a3aa29 Compare December 2, 2025 13:00

antoninbas reviewed Dec 2, 2025

View reviewed changes

hongliangl force-pushed the 20250802-update-3-party branch 2 times, most recently from 0f4ace4 to c11e942 Compare December 3, 2025 02:32

luolanzone previously approved these changes Dec 3, 2025

View reviewed changes

hongliangl dismissed luolanzone’s stale review via c37ec22 December 3, 2025 02:48

hongliangl force-pushed the 20250802-update-3-party branch from c11e942 to c37ec22 Compare December 3, 2025 02:48

Use K8s upstream NodeManager

fb0f547

Signed-off-by: Hongliang Liu <[email protected]>

hongliangl force-pushed the 20250802-update-3-party branch from c37ec22 to fb0f547 Compare December 3, 2025 02:50

antoninbas approved these changes Dec 3, 2025

View reviewed changes

luolanzone approved these changes Dec 3, 2025

View reviewed changes

luolanzone merged commit 7c59949 into antrea-io:main Dec 3, 2025
61 checks passed

hongliangl deleted the 20250802-update-3-party branch December 3, 2025 06:02

luolanzone mentioned this pull request Dec 24, 2025

Install iptables rules to allow AntreaProxy health check packets #7605

Open

	serviceInformer := corev1informers.NewFilteredServiceInformer(
	kubeClient,
	namespace,
	informerDefaultResync,
	cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc},
	func(listOptions *metav1.ListOptions) {
	listOptions.FieldSelector = fields.OneTermEqualSelector("metadata.name", serviceName).String()
	},
	)
	endpointSliceInformer := discoveryv1informers.NewFilteredEndpointSliceInformer(


		Modifies:

		- Remove `defer close(bfr.run)` from `BoundedFrequencyRunner.Loop()`. AntreaProxy may still call `BoundedFrequencyRunner.Run()`

Evolve AntreaProxy with framework and feature updates #7371

Evolve AntreaProxy with framework and feature updates #7371

Uh oh!

Conversation

hongliangl commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antoninbas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hongliangl Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hongliangl Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hongliangl commented Dec 2, 2025

Uh oh!

luolanzone left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

antoninbas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luolanzone commented Dec 3, 2025

Uh oh!

luolanzone left a comment

Choose a reason for hiding this comment

Uh oh!

hongliangl commented Dec 3, 2025

Uh oh!

hongliangl commented Dec 3, 2025

Uh oh!

Uh oh!

hongliangl commented Aug 14, 2025 •

edited

Loading

hongliangl Oct 31, 2025 •

edited

Loading

hongliangl Dec 2, 2025 •

edited

Loading