flowexporter: hostNetwork Pod to Pod flows may be misclassified as INTER_NODE and are untested

[antoninbas]

Summary

Following up on #7884 (external-to-Pod flow support): there is no test coverage for flows whose source is a local hostNetwork Pod (i.e. a real process in the host network namespace using the Node's own addressing), as opposed to a simulated external client. Reading the code, I believe these flows do not take the FROM_EXTERNAL path that #7884 added, and are instead classified as INTER_NODE records that can never be correlated by the Flow Aggregator.

I have not confirmed this on a live cluster — see "What is not verified" below. Filing this mainly so the behavior gets pinned down by tests one way or the other.

All code references are to main @ 50e3fe8469a481fb205f6bceb3c31fd3707a2602.

What #7884 added

findFlowType in pkg/agent/flowexporter/destination.go:318-363 gained two branches:

if srcIsGw {
    if conn.DestinationPodNamespace == "" {
        return utils.FlowTypeUnsupported
    }
    // The source IP is the remote node's gateway: this is the destination-node half of an
    // inter-node FROM_EXTERNAL connection. Export it as INTER_NODE [...]
    return utils.FlowTypeInterNode
}
if !srcIsPod {
    if dstIsPod {
        // Any source that is not a Pod IP and not a gateway IP is treated as FromExternal.
        // This covers external-to-Pod traffic (e.g. a host-network process on the node
        // connecting to a local Pod via the node's transport IP rather than antrea-gw0).
        return utils.FlowTypeFromExternal
    }
    return utils.FlowTypeUnsupported
}

Before #7884 both of these returned FlowTypeUnsupported, so neither shape was exported at all.

The comment on the second branch explicitly mentions the host-network case, but is conditioned on the source being the Node's transport IP rather than antrea-gw0. That condition looks load-bearing, and I do not think it holds for the common hostNetwork-Pod-to-Pod cases.

The gap

srcIsGw is true for the local gateway, not just remote ones. LookupIPInPodSubnets (pkg/agent/controller/noderoute/node_route_controller.go:822-846) finds the Pod subnet containing the IP and compares it against util.GetGatewayIPForPodPrefix(prefix). podSubnets is seeded with this Node's own PodIPv4CIDR/PodIPv6CIDR at controller construction (node_route_controller.go:145-154), so the local antrea-gw0 IP yields srcIsGw == true exactly like a remote gateway IP does.

antrea-gw0 carries the gateway IP with the Pod CIDR mask. allocateGatewayAddresses builds gwIP := &net.IPNet{IP: ip.NextIP(subnetID), Mask: localSubnet.Mask} (pkg/agent/agent.go:1273) and assigns it to the gateway link, so there is a connected route for the local Pod CIDR on antrea-gw0. For remote Pod CIDRs in encap mode, AddRoutes installs the route with LinkIndex = gateway and Gw = peer gateway IP and no explicit Src (pkg/agent/route/route_linux.go:2028-2031); in WireGuard mode Src is set explicitly to the local gateway IP (route_linux.go:2008-2016).

If the kernel therefore selects the antrea-gw0 IP as the source address for host-originated traffic to a Pod IP (this is the part I have not verified empirically), then:

• hostNetwork Pod → Pod, same Node: srcIsGw is true and the destination Pod is local, so DestinationPodNamespace != "" → INTER_NODE. The !srcIsPod / FROM_EXTERNAL branch is never reached.
• hostNetwork Pod → Pod, different Node: on the source Node the destination Pod is remote, so DestinationPodNamespace == "" → FlowTypeUnsupported, nothing exported. On the destination Node the source is the remote gateway IP, so again → INTER_NODE.

In both cases the exported record has DestinationPodName set and SourcePodName empty. In the Flow Aggregator that record is isRecordFromDst (pkg/flowaggregator/intermediate/aggregate.go:760-762) and isCorrelationRequired returns true for INTER_NODE (aggregate.go:806-813), so it waits for a source-Node half that is never produced.

The different-Node case is the more interesting one: on the destination Node such a record is indistinguishable from the destination half of a genuine inter-Node external-to-Pod flow — which is precisely the record shape the srcIsGw branch was added to produce — but it has no peer to merge with.

The source Pod is unidentifiable regardless of flow type. The Pod store filters out hostNetwork Pods (pkg/util/objectstore/podstore.go:49):

return !pod.Spec.HostNetwork && !k8s.IsPodTerminated(pod)

This is the same store used by the agent (fillPodInfo) and by the Flow Aggregator (fillK8sMetadata), so sourcePodName/sourcePodNamespace will be empty for these flows no matter which branch they take. Any test must assert on flow type and destination metadata, not on source Pod identity.

Existing coverage

e2e: testExternalToPodFlows has four subtests — "Connection to source node", "Connection to destination node", "NodePortExternalTrafficPolicyLocal", "NodePortLocal". All four obtain their client through createExternalToPodConnection / createNPLConnection (test/e2e/flowaggregator_test.go:2069 and :2124), which create a privileged hostNetwork Pod only as a vehicle for getCommandInFakeExternalNetwork, then curl from inside that fake netns. The source IP is a randomly generated off-cluster address (randExternalSubnet) and the destination is always a NodePort on the Node IP. Nothing exercises a host-network process using the Node's or the gateway's own IP as the actual source toward a Pod.

unit: TestDestination_findFlowType in pkg/agent/flowexporter/destination_test.go does cover both branches as table entries — "Non-Pod source (e.g. Node host IP) to Pod - FromExternal" (:445) and "Source is gateway with destination Pod namespace - InterNode (destination-node from-external)" (:415). But these feed findFlowType a hand-built Connection, so they assert the branch logic and not which branch a real hostNetwork Pod flow actually lands in.

Proposed tests

Add subtests under testExternalToPodFlows (or a sibling function) using a plain hostNetwork Pod — NewPodBuilder(...).InHostNetwork() without the fake-netns wrapper:

1. hostNetwork Pod → regular Pod, same Node, direct Pod IP.
2. hostNetwork Pod → regular Pod, different Node, direct Pod IP.
3. hostNetwork Pod → Service (ClusterIP and/or NodePort) → regular Pod. This is the only one of the three that reaches FromExternalCorrelator, and the only one where the source might plausibly be the transport IP rather than the gateway IP.

Rather than assuming the source address, the test should discover it — e.g. run ip route get <podIP> inside the hostNetwork Pod and parse the src field, then use that when filtering collector output. That keeps the test correct across encap/noEncap/WireGuard instead of baking one routing assumption into an assertion.

What is not verified

• I have not run any of this on a live cluster. The source-address claim is inferred from the route/address configuration cited above, not observed.
• Only the encap and WireGuard paths in AddRoutes were examined. noEncap/hybrid, and Windows, may select the transport IP instead and therefore may genuinely land on the FROM_EXTERNAL branch.
• I did not trace what the Flow Aggregator ultimately emits for the uncorrelated INTER_NODE record after the correlation timeout, or with which fields populated.

Open question

If the analysis holds, what should these flows be? FROM_EXTERNAL matches the fact that the source is not a Pod the exporter can identify, but the traffic is not external to the cluster either, and the source Node is known even though the source Pod is not. Worth deciding before writing the assertions, since the tests will either pin current behavior or fail until findFlowType is changed.

[Dyanngg]

I ran this on a live cluster, on both IP families. The analysis holds, with one correction, one confirmation, and one extra finding. Every observation below reproduced identically on IPv4 and IPv6.

Testbed: 3-node kind cluster, encap mode, AntreaProxy enabled, FlowExporter + FlowAggregator + IPFIX collector. Run twice, once IPv4-only (Pod CIDR 10.244.0.0/16) and once IPv6-only (Pod CIDR fd00:10:244::/56). I added temporary instrumentation to findFlowType and ran three new e2e subtests using a plain InHostNetwork() Pod (no fake netns), discovering the source address from curl's %{local_ip} rather than assuming it.

The source-address claim is confirmed

The kernel selects the antrea-gw0 address, not the transport IP, for host-originated traffic to a Pod IP — both for a local and a remote Pod, on both stacks:

Family	From	To	Source selected	antrea-gw0	Transport IP
IPv4	kind-worker	local Pod 10.244.1.6	10.244.1.1	10.244.1.1	172.18.0.3
IPv4	kind-worker2	remote Pod 10.244.1.6	10.244.2.1	10.244.2.1	172.18.0.4
IPv6	kind-worker	local Pod fd00:10:244:1::18	fd00:10:244:1::1	fd00:10:244:1::1	fd91:8054:ef1f::3
IPv6	kind-worker2	remote Pod fd00:10:244:1::18	fd00:10:244:2::1	fd00:10:244:2::1	fd91:8054:ef1f::4

ip route get and curl %{local_ip} agree. And srcIsGw is indeed true for the Node's own gateway:

IPv4: findFlowType inputs srcIP="10.244.1.1" dstIP="10.244.1.6" srcPort=47968 dstPort=80
  srcIsPod=true srcIsGw=true dstIsPod=true dstIsGw=false srcPod=""
  dstPod="…/host-net-to-pod-flows6izql3gv" isFromExternal=false
→ srcIsGw with destination Pod: exporting as InterNode

IPv6: findFlowType inputs srcIP="fd00:10:244:1::1" dstIP="fd00:10:244:1::18" srcPort=57380 dstPort=80
  srcIsPod=true srcIsGw=true dstIsPod=true dstIsGw=false srcPod=""
  dstPod="…/host-net-to-pod-flows9bppydkh" isFromExternal=false
→ srcIsGw with destination Pod: exporting as InterNode

Test results

Subtest	IPv4	IPv6
hostNetwork Pod → Pod, same Node, direct Pod IP	fails	fails
hostNetwork Pod → Pod, different Node, direct Pod IP	fails	fails
hostNetwork Pod → ClusterIP Service → Pod	passes (flowType 4)	passes (flowType 4)

Both failing subtests, on both families, are INTER_NODE records discarded by the Aggregator. The Aggregator side is confirmed too — all four failing flows appear by exact 5-tuple in its log:

IPv4: Deleting the record after waiting for ready to send with key: &{10.244.1.1 10.244.1.6 6 47968 80}
IPv4: Deleting the record after waiting for ready to send with key: &{10.244.2.1 10.244.1.6 6 37148 80}
IPv6: Deleting the record after waiting for ready to send with key: &{fd00:10:244:1::1 fd00:10:244:1::18 6 57380 80}
IPv6: Deleting the record after waiting for ready to send with key: &{fd00:10:244:2::1 fd00:10:244:1::18 6 43348 80}

All four flow_type:FLOW_TYPE_INTER_NODE with stats_from_source:{} empty. So the answer to "what does the Aggregator ultimately emit for the uncorrelated INTER_NODE record" is: nothing. It is deleted after MaxRetries and never reaches the collector.

Correction: the source Node emits nothing at all

For the different-Node case I expected findFlowType to be reached on the source Node and return Unsupported. It is never reached, on either family: there is not a single instrumentation line on the source Node (kind-worker2) for that destination Pod IP, in either run. The connection does not enter the source Node's Antrea-zone conntrack store in the first place. Same outcome, earlier cause — and it means the destination-Node record can never be correlated, no matter what the source Node would have classified it as.

Confirmation: the Service case behaves exactly as speculated

Host-originated ClusterIP traffic is masqueraded to the Node's transport IP, which is in no Pod subnet, so srcIsPod=false, srcIsGw=false → FROM_EXTERNAL, correlated against the zone-0 entry:

IPv4: findFlowType inputs srcIP="172.18.0.3" dstIP="10.244.1.6" srcPort=40950
  srcIsPod=false srcIsGw=false dstIsPod=true isFromExternal=true
  origDstIP="10.96.26.133" proxySnatIP="10.244.1.1" proxySnatPort=12254
→ non-Pod source to Pod destination: exporting as FromExternal

IPv6: findFlowType inputs srcIP="fd91:8054:ef1f::3" dstIP="fd00:10:244:1::18" srcPort=55876
  srcIsPod=false srcIsGw=false dstIsPod=true isFromExternal=true
  origDstIP="fd00:10:96::5edc" proxySnatIP="fd00:10:244:1::1" proxySnatPort=40153
→ non-Pod source to Pod destination: exporting as FromExternal

In each single ~2 minute run the Aggregator discarded a large number of uncorrelated INTER_NODE records, every one gateway-sourced:

IPv6: 99 dropped —  66 fd00:10:244::1 (control-plane gw), 26 fd00:10:244:1::1 (kind-worker gw), 7 fd00:10:244:2::1 (kind-worker2 gw)
      by destination: coredns (55), clickhouse (30), ipfix-collector (11), test flows (2), flow-aggregator (1)

IPv4: 154 dropped — 137 10.244.0.1 (control-plane gw), 16 10.244.2.1 (kind-worker2 gw), 1 10.244.1.1 (kind-worker gw)
      by destination: coredns (128, across 2 replicas), clickhouse (15), ipfix-collector (9), test flows (2)

These are kubelet probes and other ordinary node-originated traffic, not hostNetwork Pods. So all host-network→Pod traffic is currently invisible in flow visibility, not just traffic from hostNetwork Pods. (The IPv4/IPv6 counts aren't directly comparable — coredns ran 2 replicas in the IPv4 run vs. 2 spread differently across zones in IPv6 — but the shape is the same both times: overwhelmingly coredns and clickhouse liveness/readiness traffic.)

On the open question — a complication for any findFlowType-only fix

The obvious fix is to distinguish the local gateway from a remote one and classify local-gateway→local-Pod as FROM_EXTERNAL, which needs no correlation. But the two cases below are indistinguishable at findFlowType, on both families:

IPv4: srcIP="10.244.1.1" dstIP="10.244.1.6" srcPort=47968 dstPort=80 srcIsPod=true srcIsGw=true dstIsPod=true dstIsGw=false srcPod="" dstPod="…" isFromExternal=false origDstIP="10.244.1.6" origDstPort=80 proxySnatIP="invalid IP" proxySnatPort=0
IPv4: srcIP="10.244.1.1" dstIP="10.244.1.6" srcPort=12254 dstPort=80 srcIsPod=true srcIsGw=true dstIsPod=true dstIsGw=false srcPod="" dstPod="…" isFromExternal=false origDstIP="10.244.1.6" origDstPort=80 proxySnatIP="invalid IP" proxySnatPort=0

IPv6: srcIP="fd00:10:244:1::1" dstIP="…::18" srcPort=57380 dstPort=80 srcIsPod=true srcIsGw=true dstIsPod=true dstIsGw=false srcPod="" dstPod="…" isFromExternal=false origDstIP="…::18" origDstPort=80 proxySnatIP="invalid IP" proxySnatPort=0
IPv6: srcIP="fd00:10:244:1::1" dstIP="…::18" srcPort=40153 dstPort=80 srcIsPod=true srcIsGw=true dstIsPod=true dstIsGw=false srcPod="" dstPod="…" isFromExternal=false origDstIP="…::18" origDstPort=80 proxySnatIP="invalid IP" proxySnatPort=0

In each family, the two lines are identical but for the ephemeral port. The first line is the genuine host→Pod flow that should be exported. The second is the post-SNAT view of the host→ClusterIP connection above — note the port matches that connection's proxySnatPort exactly (12254 on IPv4, 40153 on IPv6) — whose correlated FROM_EXTERNAL record was already exported with the real source (172.18.0.3:40950 / fd91:8054:ef1f::3:55876). Classifying local-gateway sources as FROM_EXTERNAL would therefore start emitting duplicate records for flows that are already reported correctly, with a less useful source address.

That second record exists because correlation is lost mid-flow: the same connection was correlated at one poll and uncorrelated at the next (confirmed on both families — e.g. IPv6 at 17:30:04 isFromExternal=true vs. 17:30:08 isFromExternal=false for the same 5-tuple). The zone-0 conntrack entry expired while the Antrea-zone entry lived on. That looks like a separate defect worth its own issue; fixing it would make an uncorrelated gateway-sourced connection unambiguously host-originated and make the classification fix side-effect-free.

The different-Node case may not be fixable in findFlowType at all: on the destination Node a remote hostNetwork→Pod flow is byte-identical to the destination half of a genuine inter-Node from-external flow, and per the Correction above the source Node produces nothing to disambiguate it.

---

This analysis and the test/log data above were generated by Claude and reviewed by @Dyanngg before posting.