[NodePortLocal] Add IPv6 and Dual-Stack support #7594

antoninbas · 2025-11-25T20:57:43Z

Add IPv6 and dual-stack support to NodePortLocal on Linux Nodes using iptables/ip6tables. NPL previously only supported IPv4.

Key changes:

Separate NPL mappings per IP family: For dual-stack Services, NPL now creates independent mappings for IPv4 and IPv6, each with its own Node IP and Node port. The required IP families are determined from the Service's .spec.ipFamilies field.
IPv4 and IPv6 port space: Node ports for IPv4 and IPv6 are allocated independently from the same port range, as iptables and ip6tables rules operate independently and do not interfere.
Node IP address selection: The NPL controller now watches the Node object to obtain Node IP addresses for both IP families, prioritizing external IPs over internal IPs. When Node IPs change, all local Pods are automatically reconciled with updated NPL annotations. Note that prior to this change, internal Node IPs had precedence, but external IPs make more sense for a feature such as this one.
IP family tracking: Introduced an efficient ipFamilies bitmask type to track sets of IP families with no memory allocations.
Annotation format: NPL annotations now include an ipFamily field to distinguish between IPv4 and IPv6 mappings. For example:

[
  {"podPort":8080,"nodeIP":"10.10.10.10","nodePort":61002, "protocol":"tcp","ipFamily":"IPv4"},
  {"podPort":8080,"nodeIP":"fd12:3456:789a:1::1", "nodePort":61003,"protocol":"tcp","ipFamily":"IPv6"}
]

PortTable and iptables rule management: The NPL controller gets one PortTable per IP family and all rules are managed independently. The AddAllRules function now performs separate restore operations for each IP family.
Port allocation: Modified LocalPortOpener.OpenLocalPort to accept an isIPv6 parameter and bind to the appropriate network type (tcp4/tcp6, udp4/udp6) based on the Pod IP family.

Implementation notes:

Assumes Kubernetes version >= 1.23, which guarantees the availability of .spec.ipFamilies on Services and .status.podIPs on Pods.
Windows support remains IPv4-only and is not affected by this change.
The implementation maintains backward compatibility by treating existing annotations without the ipFamily field as IPv4.
All existing unit tests have been updated, and new tests have been added to verify Node IP updates and dual-stack functionality.
E2E tests have been updated to support IPv6 and dual-stack testing. All NPL Services are created with the PreferDualStack policy.

Fixes #7513

antoninbas · 2025-11-25T20:58:09Z

/test-ipv6-e2e
/test-ipv6-only-e2e

antoninbas · 2025-11-25T22:29:28Z

/test-kind-ipv6-e2e
/test-kind-ipv6-only-e2e

antoninbas · 2025-11-25T23:30:30Z

/test-kind-ipv6-e2e
/test-kind-ipv6-only-e2e

antoninbas · 2025-11-26T17:51:02Z

/test-kind-ipv6-e2e
/test-kind-ipv6-only-e2e

antoninbas · 2025-11-26T21:31:09Z

/test-kind-ipv6-e2e
/test-kind-ipv6-only-e2e

antoninbas · 2025-11-26T23:34:51Z

/test-kind-ipv6-e2e
/test-kind-ipv6-only-e2e

Add IPv6 and dual-stack support to NodePortLocal on Linux Nodes using iptables/ip6tables. NPL previously only supported IPv4. Key changes: 1. **Separate NPL mappings per IP family**: For dual-stack Services, NPL now creates independent mappings for IPv4 and IPv6, each with its own Node IP and Node port. The required IP families are determined from the Service's `.spec.ipFamilies` field. 2. **IPv4 and IPv6 port space**: Node ports for IPv4 and IPv6 are allocated independently from the same port range, as iptables and ip6tables rules operate independently and do not interfere. 3. **Node IP address selection**: The NPL controller now watches the Node object to obtain Node IP addresses for both IP families, prioritizing external IPs over internal IPs. When Node IPs change, all local Pods are automatically reconciled with updated NPL annotations. Note that prior to this change, internal Node IPs had precedence, but external IPs make more sense for a feature such as this one. 4. **IP family tracking**: Introduced an efficient `ipFamilies` bitmask type to track sets of IP families with no memory allocations. 5. **Annotation format**: NPL annotations now include an `ipFamily` field to distinguish between IPv4 and IPv6 mappings. For example: ```json [ {"podPort":8080,"nodeIP":"10.10.10.10","nodePort":61002, "protocol":"tcp","ipFamily":"IPv4"}, {"podPort":8080,"nodeIP":"fd12:3456:789a:1::1", "nodePort":61003,"protocol":"tcp","ipFamily":"IPv6"} ] ``` 6. **PortTable and iptables rule management**: The NPL controller gets one PortTable per IP family and all rules are managed independently. The `AddAllRules` function now performs separate restore operations for each IP family. 7. **Port allocation**: Modified `LocalPortOpener.OpenLocalPort` to accept an `isIPv6` parameter and bind to the appropriate network type (tcp4/tcp6, udp4/udp6) based on the Pod IP family. Implementation notes: - Assumes Kubernetes version >= 1.23, which guarantees the availability of `.spec.ipFamilies` on Services and `.status.podIPs` on Pods. - Windows support remains IPv4-only and is not affected by this change. - The implementation maintains backward compatibility by treating existing annotations without the `ipFamily` field as IPv4. - All existing unit tests have been updated, and new tests have been added to verify Node IP updates and dual-stack functionality. - E2E tests have been updated to support IPv6 and dual-stack testing. All NPL Services are created with the PreferDualStack policy. Fixes antrea-io#7513 Signed-off-by: Antonin Bas <[email protected]>

Signed-off-by: Antonin Bas <[email protected]>

antoninbas · 2025-11-27T01:36:23Z

/test-kind-ipv6-e2e
/test-kind-ipv6-only-e2e

antoninbas · 2025-12-01T19:29:59Z

/test-kind-ipv6-e2e
/test-kind-ipv6-only-e2e

Signed-off-by: Antonin Bas <[email protected]>

antoninbas · 2025-12-02T21:22:18Z

/test-kind-ipv6-e2e
/test-kind-ipv6-only-e2e

antoninbas · 2025-12-02T22:55:40Z

/test-kind-ipv6-e2e

luolanzone · 2025-12-09T06:59:46Z

pkg/agent/nodeportlocal/portcache/port_table_others.go

+				// This will be handled gracefully by the NPL controller: if there is an
+				// annotation using this port, it will be removed and replaced with a new
+				// one with a valid port mapping.
+				klog.ErrorS(err, "Cannot bind to local port, skipping it", "port", nplPort.NodePort)


Suggested change

klog.ErrorS(err, "Cannot bind to local port, skipping it", "port", nplPort.NodePort)

klog.ErrorS(err, "Cannot bind to local port, skipping it", "port", nplPort.NodePort, "ipv6", pt.IsIPv6)

luolanzone · 2025-12-09T07:04:37Z

pkg/agent/nodeportlocal/portcache/port_table_others_test.go

+				)
+
+				// Bubble time will automatically advance when goroutines are blocked.
+				portTable.RestoreRules(t.Context(), allNPLPorts)


I suppose we can have some ipv6 NPL ports in allNPLPorts to verify the ipv6?

luolanzone · 2025-12-09T07:24:09Z

pkg/agent/nodeportlocal/testing/annotations.go

-		expectedAnnotation := a.find(nplAnnotation.PodPort, nplAnnotation.Protocol)
-		if !assert.NotNilf(t, expectedAnnotation, "Unexpected annotation with PodPort %d", nplAnnotation.PodPort) {
+		expectedAnnotation := a.find(nplAnnotation.PodPort, nplAnnotation.Protocol, nplAnnotation.IPFamily)
+		if !assert.NotNilf(t, expectedAnnotation, "Unexpected annotation with PodPort %d, Protocol %s, IPFamily %s", nplAnnotation.PodPort, nplAnnotation.Protocol, nplAnnotation.IPFamily) {


Suggested change

if !assert.NotNilf(t, expectedAnnotation, "Unexpected annotation with PodPort %d, Protocol %s, IPFamily %s", nplAnnotation.PodPort, nplAnnotation.Protocol, nplAnnotation.IPFamily) {

if !assert.NotNilf(t, expectedAnnotation, "Unexpected annotation with Pod port %d, Protocol %s, IPFamily %s", nplAnnotation.PodPort, nplAnnotation.Protocol, nplAnnotation.IPFamily) {

luolanzone · 2025-12-09T07:24:53Z

pkg/agent/nodeportlocal/k8s/npl_controller.go

+			if portData != nil && portData.Defunct() {
+				klog.InfoS("Deleting defunct NodePortLocal rule for Pod to prevent re-use", "pod", klog.KObj(pod), "podIP", podIP, "port", port, "protocol", protocol)
+				if err := portTable.DeleteRule(key, port, protocol); err != nil {
+					return fmt.Errorf("failed to delete defunct rule for Pod %s, Pod Port %d, Protocol %s: %w", key, port, protocol, err)


Suggested change

return fmt.Errorf("failed to delete defunct rule for Pod %s, Pod Port %d, Protocol %s: %w", key, port, protocol, err)

return fmt.Errorf("failed to delete defunct rule for Pod %s, Pod port %d, Protocol %s: %w", key, port, protocol, err)

luolanzone · 2025-12-09T07:25:21Z

pkg/agent/nodeportlocal/k8s/npl_controller.go

+			if portData != nil && portData.PodIP != podIP {
+				klog.InfoS("Deleting NodePortLocal rule for Pod because of IP change", "pod", klog.KObj(pod), "podIP", podIP, "prevPodIP", portData.PodIP)
+				if err := portTable.DeleteRule(key, port, protocol); err != nil {
+					return fmt.Errorf("failed to delete rule for Pod %s, Pod Port %d, Protocol %s: %w", key, port, protocol, err)


luolanzone · 2025-12-09T07:26:57Z

pkg/agent/nodeportlocal/types/types.go

+const (
+	IPFamilyIPv4    IPFamilyType = "IPv4"
+	IPFamilyIPv6    IPFamilyType = "IPv6"
+	IPFamilyUnknown IPFamilyType = ""


I am not seeing it's used anywhere.

luolanzone · 2025-12-09T07:35:24Z

pkg/agent/nodeportlocal/npl_agent_init.go

 	}
-
-	return nplk8s.NewNPLController(kubeClient, podInformer, serviceInformer.Informer(), portTable, nodeName), nil
+	if ipv6Enabled {


ipv6 is not supported on Windows, I suppose we should skip this on Windows platform?

XinShuYang · 2025-12-09T13:59:34Z

pkg/agent/nodeportlocal/types/types.go

+	NodeIP   string       `json:"nodeIP"`
+	NodePort int          `json:"nodePort"`
+	Protocol string       `json:"protocol"`
+	IPFamily IPFamilyType `json:"ipFamily"`


Should we update annotation examples in the document?

XinShuYang · 2025-12-09T15:03:11Z

pkg/agent/nodeportlocal/k8s/npl_controller.go

-					return fmt.Errorf("failed to add rule for Pod %s: %v", key, err)
+
+			portTable := c.getPortTableForFamily(ipFamily)
+			portData := portTable.GetEntry(key, port, protocol)


Is it safe to call portTable.GetEntry without portTable nil check?

XinShuYang · 2025-12-09T15:06:30Z

pkg/agent/nodeportlocal/k8s/npl_controller.go

+	}
+
+	if c.updateNodeIPs(newNode) {
+		klog.InfoS("Node IPs changed, reconciling all Pods", "node", klog.KObj(newNode), "IPv4", c.nodeIPv4, "IPv6", c.nodeIPv6)


Is it expected to print unprotected c.nodeIPv4 and c.nodeIPv6 here?

antoninbas added area/proxy/nodeportlocal Issues or PRs related to the NodePortLocal feature area/transit/ipv6 Issues or PRs related to IPv6. labels Nov 25, 2025

antoninbas force-pushed the npl-ipv6-support branch from a0273a3 to e7298ac Compare November 25, 2025 22:27

antoninbas force-pushed the npl-ipv6-support branch from e7298ac to 9569d84 Compare November 25, 2025 23:29

antoninbas force-pushed the npl-ipv6-support branch from 503b873 to 4577fb6 Compare November 26, 2025 21:30

antoninbas added 2 commits November 26, 2025 17:35

Fixing e2e tests for IPv6 / dual-stack

9560ed0

Signed-off-by: Antonin Bas <[email protected]>

antoninbas force-pushed the npl-ipv6-support branch from 59dcfc5 to 9bc251e Compare November 27, 2025 01:36

antoninbas force-pushed the npl-ipv6-support branch from 9bc251e to 9e5bc6c Compare December 1, 2025 19:29

antoninbas marked this pull request as ready for review December 1, 2025 22:35

antoninbas requested review from luolanzone and tnqn December 1, 2025 22:35

Use correct destination format for IPv6 DNAT rules

49df002

Signed-off-by: Antonin Bas <[email protected]>

antoninbas force-pushed the npl-ipv6-support branch from 9e5bc6c to 49df002 Compare December 2, 2025 21:22

luolanzone added the action/release-note Indicates a PR that should be included in release notes. label Dec 9, 2025

luolanzone reviewed Dec 9, 2025

View reviewed changes

luolanzone requested a review from XinShuYang December 9, 2025 07:45

XinShuYang reviewed Dec 9, 2025

View reviewed changes

	klog.ErrorS(err, "Cannot bind to local port, skipping it", "port", nplPort.NodePort)
	klog.ErrorS(err, "Cannot bind to local port, skipping it", "port", nplPort.NodePort, "ipv6", pt.IsIPv6)

	if !assert.NotNilf(t, expectedAnnotation, "Unexpected annotation with PodPort %d, Protocol %s, IPFamily %s", nplAnnotation.PodPort, nplAnnotation.Protocol, nplAnnotation.IPFamily) {
	if !assert.NotNilf(t, expectedAnnotation, "Unexpected annotation with Pod port %d, Protocol %s, IPFamily %s", nplAnnotation.PodPort, nplAnnotation.Protocol, nplAnnotation.IPFamily) {

	return fmt.Errorf("failed to delete defunct rule for Pod %s, Pod Port %d, Protocol %s: %w", key, port, protocol, err)
	return fmt.Errorf("failed to delete defunct rule for Pod %s, Pod port %d, Protocol %s: %w", key, port, protocol, err)

[NodePortLocal] Add IPv6 and Dual-Stack support #7594

Are you sure you want to change the base?

[NodePortLocal] Add IPv6 and Dual-Stack support #7594

Uh oh!

Conversation

antoninbas commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antoninbas commented Nov 25, 2025

Uh oh!

antoninbas commented Nov 25, 2025

Uh oh!

antoninbas commented Nov 25, 2025

Uh oh!

antoninbas commented Nov 26, 2025

Uh oh!

antoninbas commented Nov 26, 2025

Uh oh!

antoninbas commented Nov 26, 2025

Uh oh!

antoninbas commented Nov 27, 2025

Uh oh!

antoninbas commented Dec 1, 2025

Uh oh!

antoninbas commented Dec 2, 2025

Uh oh!

antoninbas commented Dec 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

antoninbas commented Nov 25, 2025 •

edited

Loading