Skip to content

aws-node fails to initialize with trunk ENI in dual-stack subnet #3572

@jfernandez

Description

@jfernandez

What happened:

The aws-node IPAM daemon fails to initialize when a trunk ENI is present on the node in IPv6 prefix delegation mode with Security Groups for Pods enabled. The daemon crashes during nodeInit() when attempting to set up the trunk ENI's network configuration.

Error logs:

{"level":"info","ts":"2026-01-13T20:37:16.488Z","msg":"eni-<redacted ID> is of type: trunk"}
{"level":"info","ts":"2026-01-13T20:37:16.603Z","msg":"Setting up network for an ENI with IP address , MAC address 0a:ff:fd:4c:b5:cb, CIDR 2a03:5640:f13c::/64, route table 4 and Network Card 0"}
{"level":"debug","ts":"2026-01-13T20:37:16.603Z","msg":"Adding IP address <nil>"}
{"level":"warn","ts":"2026-01-13T20:37:16.603Z","msg":"Error trying to set up ENI eni-<redacted ID>: failed to set up ENI eni-<redacted ID> network: setupENINetwork: failed to add IP addr to ENI: numerical result out of range"}

The aws-node pod enters CrashLoopBackOff as the readiness probe times out waiting for IPAM initialization.

Attach logs

Will provide via email to [email protected]

What you expected to happen:

The aws-node IPAM daemon should successfully initialize with the trunk ENI present, recognizing that trunk ENIs in IPv6 prefix delegation mode don't have primary IP addresses and should skip network setup (similar to how IP allocation is already skipped for trunk ENIs).

How to reproduce it (as minimally and precisely as possible):

  1. Configure an EKS cluster with IPv6 prefix delegation mode:
    • ENABLE_IPv6=true
    • ENABLE_IPv4=false
    • ENABLE_PREFIX_DELEGATION=true
  2. Enable Security Groups for Pods by setting ENABLE_POD_ENI=true in the aws-node DaemonSet
  3. Deploy vpc-resource-controller which creates and attaches a trunk ENI to the node
  4. Restart the aws-node pod (or wait for a new node to join with trunk ENI already attached)

Anything else we need to know?:

Root Cause Analysis

In pkg/ipamd/ipamd.go, the setupENI() function calls SetupENINetwork() for all non-primary ENIs, including trunk ENIs (lines 1279-1285):

// For other ENIs, set up the network
if eni != primaryENI {
    subnetCidr := eniMetadata.SubnetIPv6CIDR
    err := c.networkClient.SetupENINetwork(c.primaryIP[eni], eniMetadata.MAC, ...)
    // c.primaryIP[eni] is empty string for trunk ENIs!
}

For trunk ENIs in IPv6 prefix delegation mode:

  1. eniMetadata.PrimaryIPv6Address() returns empty string (trunk ENIs have no IPv6 addresses, only the trunk interface type)
  2. This empty string is stored in c.primaryIP[eni]
  3. SetupENINetwork() is called with empty string for eniIP

In pkg/networkutils/network.go, the setupENINetwork() function unconditionally tries to add the primary IP (lines 1067-1096):

eniAddr := &net.IPNet{
    IP:   net.ParseIP(eniIP),  // net.ParseIP("") returns nil
    Mask: eniSubnetIPNet.Mask,
}
// ...
log.Debugf("Adding IP address %s", eniAddr.String())  // Prints "<nil>"
if err = netLink.AddrAdd(link, &netlink.Addr{IPNet: eniAddr}); err != nil {
    return errors.Wrap(err, "setupENINetwork: failed to add IP addr to ENI")  // Fails here
}

Trunk ENIs are logical constructs created by vpc-resource-controller for branch ENI attachment. They intentionally have no IPv4/IPv6 addresses - only the InterfaceType: trunk attribute. Branch ENIs (created for pods with SecurityGroupPolicy) are associated to the trunk ENI via VLAN tagging.

Existing Partial Fix

The code already has logic to skip IP allocation for trunk ENIs (line 1269):

if c.enableIPv6 {
    if !isTrunkENI {
        err := c.assignIPv6Prefix(eni, eniMetadata.NetworkCard)
    }
}

And skips IP rules for trunk ENIs (line 1166 in network.go):

if !isTrunkENI && !isRuleConfigured {
    // Add IP rule for Primary IP of the ENI
}

However, the check is missing for the SetupENINetwork() call itself.

Additional Notes:

  • IPv6 Security Groups for Pods support was added in PR IPv6 Security Groups for Pods Support #2639 (v1.15.x)
  • Commit 315d56a ("skip adding IPs to trunk ENI") partially addressed this for IP allocation but not for SetupENINetwork()
  • The bug only manifests when aws-node starts/restarts AFTER the trunk ENI is already attached. If aws-node starts before trunk ENI attachment, initialization succeeds (trunk ENI arrives later via reconciliation)
  • Currently, there is no workaround. The aws-node pod will remain in CrashLoopBackOff if restarted while a trunk ENI is present

Environment:

  • Kubernetes version (use kubectl version): v1.30.6-eks
  • CNI Version: v1.21.1
  • OS (e.g: cat /etc/os-release): Ubuntu 22.04.1 LTS (custom AMI, not standard EKS-optimized AMI or EKS Auto Mode)
  • Kernel (e.g. uname -a): Linux 6.5.x
  • Instance Type: r7a.2xlarge
  • Configuration:
    • ENABLE_IPv6=true
    • ENABLE_IPv4=false
    • ENABLE_PREFIX_DELEGATION=true
    • ENABLE_POD_ENI=true

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions