Service CIDR must be manually configured when using kube-proxy in nftables mode with VPC CNI iptables-nft

[shaikatzz]

What happened?

When running kube-proxy in native nftables mode while VPC CNI uses iptables-nft, pods on secondary ENIs cannot reach Kubernetes services. Traffic gets incorrectly marked with 0x80 and routed via eth0 instead of the correct secondary interface.

Root Cause Analysis

Using nft monitor trace, I identified the issue:

1. MANGLE PREROUTING (priority -150) runs before kube-proxy's NAT DNAT
2. VPC CNI's CONNMARK --restore-mark rule in mangle restores mark 0x80 from conntrack
3. At this point, destination is still the ClusterIP (not yet DNAT'd to pod IP)
4. The routing decision uses fwmark 0x80 → routes via main table → eth0
5. kube-proxy DNAT happens after the routing decision

Trace evidence:

ip mangle PREROUTING rule iifname "eni*" xt match "comment" counter packets 3325625 bytes 644878056 xt target "CONNMARK" (verdict continue)
ip mangle PREROUTING policy accept meta mark 0x00000080   ← MARK SET BEFORE DNAT
...
ip mangle FORWARD packet: iif "eni1caf450b99a" oif "eth0" ... ip daddr 10.71.20.123  ← WRONG INTERFACE

The packet enters with dst=172.20.96.189 (ClusterIP), gets marked 0x80 in mangle PREROUTING, routing decides to use eth0, then DNAT changes destination to pod IP.

Why this doesn't happen with iptables mode kube-proxy

In iptables mode, both VPC CNI and kube-proxy use the same ip nat table. The rule ordering within a single table is deterministic - kube-proxy's DNAT rules in PREROUTING run before VPC CNI's CONNMARK chain checks the destination.

In nftables mode, kube-proxy creates rules in the ip kube-proxy table while VPC CNI (via iptables-nft) creates rules in the ip nat table. The mangle table (priority -150) runs before both, and the CONNMARK restore happens there.

Expected behavior

• Traffic from pods on secondary ENIs to ClusterIP services should be routed via the correct interface (eth1, eth2, etc.)
• Service CIDR should be automatically excluded from CONNMARK, or documentation should clearly state this requirement for nftables mode

Workaround

Manually set the service CIDR in environment variables:

kubectl set env daemonset aws-node -n kube-system
AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS="172.20.0.0/16"

Suggested improvements

1. Documentation: Add a note in the README and troubleshooting guide that when using kube-proxy in nftables mode, AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS must be configured with the cluster's service CIDR

2. Consider auto-detection: The CNI could query the Kubernetes API for the service CIDR.

Environment

• VPC CNI version: 1.21.1
• Kubernetes version: 1.34.2
• kube-proxy version: 1.34.2
• OS: Bottlerocket OS 1.51.0

[jayanthvn]

@shaikatzz thanks for sharing this. We did an investigation last week and have a similar root cause.

In iptables kube-proxy mode, Kubernetes Service DNAT (KUBE-SERVICES) and AWS VPC CNI connmark rules are evaluated within the same nat/PREROUTING chain , resulting in deterministic, linear ordering: Service DNAT runs first, translating the ClusterIP to a backend Pod IP, and the subsequent VPC CIDR exclusion prevents the AWS connmark from being applied to Service traffic. As a result, no connmark is set for ClusterIP flows, the early mangle/PREROUTING restore has nothing to restore, and source-based policy routing correctly sends pod traffic out the corresponding ENI.

The theory is, In nftables kube-proxy mode, kube-proxy programs Service DNAT into a separate ip kube-proxy table, while AWS VPC CNI programs connmark rules into the ip nat table. Both attach base chains to hook prerouting at the same priority (dstnat), and nftables does not guarantee execution order across tables with equal hook and priority. This allows the AWS connmark rule to run before kube-proxy DNAT, causing the connection to be marked while the destination is still the ClusterIP (not in VPC CIDR). The connmark is then restored to a packet mark in mangle/PREROUTING, triggering a policy routing override (fwmark 0x80) that forces egress via the primary ENI and breaks symmetric routing for Service traffic from pods on secondary ENIs. We noticed if kube-proxy starts after AWS VPC CNI then routing was working as expected which is yet to be debugged and maybe when hook and priority are identical, the kernel’s execution order becomes dependent on how hooks are inserted into the per-hook list (i.e., module/service start order / registration order)...

If the packet destination matches an excluded CIDR (recommended workaround), the connection is never marked (ctmark is never set). Once that invariant holds, everything downstream becomes safe, including mangle restore.

When you set exclude SNAT CIDRs = Service CIDR, that CIDR is added to:

amazon-vpc-cni-k8s/pkg/networkutils/network.go

Line 601 in cdfc4c5

allCIDRs = append(allCIDRs, n.excludeSNATCIDRs...)

and then programmed into AWS-CONNMARK-CHAIN-0 as early RETURN rules:

-A AWS-CONNMARK-CHAIN-0 -d <CIDR> -j RETURN

amazon-vpc-cni-k8s/pkg/networkutils/network.go

Line 641 in cdfc4c5

comment = "AWS CONNMARK CHAIN, EXCLUDED CIDR"

To validate this we changed the priority and why changing priority fixes the Issue -

By installing the AWS connmark logic at a numerically later priority than dstnat, kube-proxy Service DNAT is guaranteed to execute first. This ensures that the destination address is already translated to the backend Pod IP before the AWS connmark logic runs, allowing the VPC CIDR exclusion to match correctly and preventing the connmark from being applied to Service traffic. With no connmark set, the subsequent restore step does not apply the fwmark, and source-based policy routing continues to route pod traffic out the appropriate ENI, restoring symmetric routing behavior.

We are evaluating other options for long term fix..

[cdirubbio]

Adding on top of that:

We noticed if kube-proxy starts after AWS VPC CNI then routing was working as expected

This is a result of netfilter's behavior when hook and priority are the same as @jayanthvn suggested. If we look at the netfilter source code, the below diagram represents the last-registered, first-executed behavior of the code:

Original hooks: [PREROUTING(priority=dstnat)]
Inserting:      nat-prerouting(priority=dstnat)
Result:         [nat-prerouting(priority=dstnat), PREROUTING(priority=dstnat)]
                  ↑ new hook inserted before existing one with same priority

This is backed up by the nftrace's on my two nodes.

# On problematic nodes, see nat PREROUTING followed by kube-proxy nat-prerouting
trace id 0de1ce09 ip nat PREROUTING packet: iif "enia0e661b6e52" ether saddr 6a:37:e2:77:af:54 ether daddr c2:1e:c2:27:a5:43 ip saddr 172.31.204.7 ip daddr 10.100.0.1 ip dscp cs0 ip ecn not-ect ip ttl 127 ip id 14150 ip length 60tcp sport 35760 tcp dport 443 tcp flags == syn tcp window 62727
trace id 0de1ce09 ip nat PREROUTING rule ip daddr 10.100.0.1 meta nftrace set 1 (verdict continue)
trace id 0de1ce09 ip nat PREROUTING policy accept meta mark 0x00000080
trace id 0de1ce09 ip kube-proxy nat-prerouting packet: iif "enia0e661b6e52" ether saddr 6a:37:e2:77:af:54 ether daddr c2:1e:c2:27:a5:43 ip saddr 172.31.204.7 ip daddr 10.100.0.1 ip dscp cs0 ip ecn not-ect ip ttl 127 ip id 14150 ip length 60 tcp sport 35760 tcp dport 443 tcp flags == syn tcp window 62727
trace id 0de1ce09 ip kube-proxy nat-prerouting rule jump services (verdict jump services)

1024:   from all fwmark 0x80/0x80 lookup main <--- we are hitting here, leading to ENI dropping packets on src/dst check
...
1536:   from 172.31.204.7 lookup 2

Versus when I forced kube-proxy to start after aws-node, since KP chains are being executed first, we see DNAT happening with a verdict accept, leading to further prerouting hook chains to not be executed.

trace id 1fa75ff6 ip kube-proxy nat-prerouting packet: iif "enid30d5bf6636" ether saddr fe:d9:bb:21:9d:0c ether daddr 36:ca:6d:55:f0:39 ip saddr 172.31.100.244 ip daddr 10.100.0.1 ip dscp cs0 ip ecn not-ect ip ttl 127 ip id 21185 ip length 60 tcp sport 36278 tcp dport 443 tcp flags == syn tcp window 62727 
trace id 1fa75ff6 ip kube-proxy nat-prerouting rule jump services (verdict jump services)
trace id 1fa75ff6 ip kube-proxy services rule ip daddr . meta l4proto . th dport vmap @service-ips (verdict goto service-2QRHZV4L-default/kubernetes/tcp/https)
trace id 1fa75ff6 ip kube-proxy service-2QRHZV4L-default/kubernetes/tcp/https rule numgen random mod 2 vmap { 0 : goto endpoint-XQLFGMP6-default/kubernetes/tcp/https__172.31.107.171/443, 1 : goto endpoint-R2TBHSAE-default/kubernetes/tcp/https__172.31.255.113/443 } (verdict goto endpoint-R2TBHSAE-default/kubernetes/tcp/https__172.31.255.113/443)
trace id 1fa75ff6 ip kube-proxy endpoint-R2TBHSAE-default/kubernetes/tcp/https__172.31.255.113/443 rule meta l4proto tcp dnat to 172.31.255.113:443 (verdict accept)

[elliottmoos]

Just wanted to say thanks for this issue. I've been struggling with this for a few weeks now.

[mxk]

Is the workaround needed when using AWS_VPC_K8S_CNI_EXTERNALSNAT="true"? Trying to determine whether this applies to clusters deployed in a private subnet with NAT Gateway.

[cdirubbio]

@mxk if you have AWS_VPC_K8S_CNI_EXTERNALSNAT set to true, the PREROUTING chain is empty in the nat table, so everything will work as expected and no need for the workaround.

       chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
        }