Add support for Antrea Egress in hybrid mode #7239

hongliangl · 2025-06-19T12:52:11Z

This commit adds support of Antrea Egress in hybrid mode. The key changes include:

IPTables rules added in table mangle, chains ANTREA-PREROUTING and ANTREA-POSTROUTING

# Set connmark for the first packets of connections originated remote Pods.
-A ANTREA-PREROUTING ! -s 10.244.0.0/24 -i antrea-gw0 -m comment --comment "Antrea: set connmark for the first Egress request packet" -m conntrack --ctstate NEW -m mark ! --mark 0x0/0xff -j CONNMARK --set-xmark 0x40000000/0x40000000

# Restore fwmark from connmark for reply packets of connections originated from remote Pods.
-A ANTREA-PREROUTING -m comment --comment "Antrea: restore fwmark from connmark for reply packets sourced from remote Pods" -m conntrack --ctstate ESTABLISHED -m conntrack --ctdir REPLY -m connmark --mark 0x40000000/0x40000000 -j CONNMARK --restore-mark --nfmask 0x40000000 --ctmask 0x40000000

# Clear fwmark from reply packets of connections originated from remote Pods.
-A ANTREA-POSTROUTING -m comment --comment "Antrea: clear fwmark from reply Egress packets to remote Pods" -m conntrack --ctstate ESTABLISHED -m conntrack --ctdir REPLY -j MARK --set-xmark 0x0/0x40000000

IP rule

root@kind-control-plane:/# ip rule | grep 141
32765:  from all fwmark 0x40000000/0x40000000 lookup 141

A default route in the policy-routing table 141 and its corresponding static ip neighbor
are used to forward reply Egress packets back to OVS pipeline.

 root@kind-control-plane:/# ip route list table 141
 default via 169.254.0.251 dev antrea-gw0 onlink

 root@kind-control-plane:/# ip neigh | grep 169.254.0.251
 169.254.0.251 dev antrea-gw0 lladdr aa:bb:cc:dd:ee:ff PERMANENT

OVS flows

# Encapsulate packets sourced from antrea-gw0 and destined to remote Pod CIDRs via tunnel.
table=L3Forwarding, priority=201,ct_mark=0x1/0xf,ip,in_port=antrea-gw0, nw_dst=10.244.1.0/24 actions=set_field:ba:5e:d1:55:aa:c0->eth_src,set_field:aa:bb:cc:dd:ee:ff->eth_dst,set_field:172.18.0.3->tun_dst,set_field:0x10/0xf0->reg0,goto_table:L3DecTTL
table=L3Forwarding, priority=201,ct_mark=0x1/0xf,ip,in_port=antrea-gw0, nw_dst=10.244.2.0/24 actions=set_field:ba:5e:d1:55:aa:c0->eth_src,set_field:aa:bb:cc:dd:ee:ff->eth_dst,set_field:172.18.0.4->tun_dst,set_field:0x10/0xf0->reg0,goto_table:L3DecTTL
table=L3Forwarding, priority=201,ct_mark=0x1/0xf,ip,in_port=antrea-gw0, nw_dst=10.244.3.0/24 actions=set_field:ba:5e:d1:55:aa:c0->eth_src,set_field:aa:bb:cc:dd:ee:ff->eth_dst,set_field:172.18.0.5->tun_dst,set_field:0x10/0xf0->reg0,goto_table:L3DecTTL

The benchmark of hybrid vs encap of Egress traffic:

Pod	Payload	Encap	Hybrid	Hybrid/Encap
remotepodipv4-cluster	TCP_STREAM	16724.83	15910.85	0.92
remotepodipv4-cluster	TCP_RR	24005.69	24099.39	0.97
remotepodipv4-cluster	TCP_CRR	3254.50	3254.88 1	1
localpodipv4-cluster	TCP_STREAM	23728.62	22551.3	0.95
localpodipv4-cluster	TCP_RR	29130.42	29629.38	1.01
localpodipv4-cluster	TCP_CRR	3255.32	3254.93 1	1

Dependencies:

luolanzone · 2025-08-04T03:00:58Z

@hongliangl please check if patch ut coverage can be improved.

hongliangl · 2025-08-04T03:24:18Z

@hongliangl please check if patch ut coverage can be improved.

Will check.

cmd/antrea-agent/options.go

pkg/agent/route/route_linux.go

test/integration/agent/route_test.go

pkg/agent/openflow/pipeline.go

pkg/agent/route/route_linux.go

jianjuns · 2025-08-11T23:09:25Z

pkg/agent/openflow/pipeline.go

+		Cookie(f.cookieAllocator.Request(f.category).Raw()).
+		MatchProtocol(ipProtocol).
+		MatchRegMark(FromGatewayRegMark, FromExternalRegMark).
+		MatchPktMark(types.RemotePodSourceMark, &types.RemotePodSourceMark).


Why we need to matach a remote Pod mark here? Could not we just match egress traffic, since destination is already matched by MatchDstIPNet(peerSubnet)?

This is to exclude the request packets of external-to-Service (backend is a remote Pod) connections. I will add comment to this line to explain.

antoninbas

some initial comments

cmd/antrea-agent/options.go

pkg/agent/openflow/client.go

pkg/agent/openflow/pipeline.go

pkg/agent/route/route_linux.go

cmd/antrea-agent/options.go

docs/egress.md

pkg/agent/route/route_linux.go

pkg/agent/types/net.go

antoninbas

Change looks good to me, I think @tnqn should review as well

pkg/agent/route/route_linux_test.go

pkg/agent/types/net.go

pkg/agent/route/route_linux.go

pkg/agent/types/net.go

This commit adds support of Antrea Egress in hybrid mode. The key changes include: - IPTables rules added in table `mangle`, chains `ANTREA-PREROUTING` and `ANTREA-POSTROUTING` ```shell # Set connmark for the first packets of connections originated remote Pods. -A ANTREA-PREROUTING ! -s 10.244.0.0/24 -i antrea-gw0 -m comment --comment "Antrea: set connmark for the first Egress request packet" -m conntrack --ctstate NEW -m mark ! --mark 0x0/0xff -j CONNMARK --set-xmark 0x40000000/0x40000000 # Restore fwmark from connmark for reply packets of connections originated from remote Pods. -A ANTREA-PREROUTING -m comment --comment "Antrea: restore fwmark from connmark for reply packets sourced from remote Pods" -m conntrack --ctstate ESTABLISHED -m conntrack --ctdir REPLY -m connmark --mark 0x40000000/0x40000000 -j CONNMARK --restore-mark --nfmask 0x40000000 --ctmask 0x40000000 # Clear fwmark from reply packets of connections originated from remote Pods. -A ANTREA-POSTROUTING -m comment --comment "Antrea: clear fwmark from reply Egress packets to remote Pods" -m conntrack --ctstate ESTABLISHED -m conntrack --ctdir REPLY -j MARK --set-xmark 0x0/0x40000000 ``` - IP rule ```shell root@kind-control-plane:/# ip rule | grep 141 32765: from all fwmark 0x40000000/0x40000000 lookup 141 ``` - A default route in the policy-routing table 141 and its corresponding static ip neighbor are used to forward reply Egress packets back to OVS pipeline. ```shell root@kind-control-plane:/# ip route list table 141 default via 169.254.0.251 dev antrea-gw0 onlink root@kind-control-plane:/# ip neigh | grep 169.254.0.251 169.254.0.251 dev antrea-gw0 lladdr aa:bb:cc:dd:ee:ff PERMANENT ``` - OVS flows ```shell # Encapsulate packets sourced from antrea-gw0 and destined to remote Pod CIDRs via tunnel. table=L3Forwarding, priority=201,ct_mark=0x1/0xf,ip,in_port=antrea-gw0, nw_dst=10.244.1.0/24 actions=set_field:ba:5e:d1:55:aa:c0->eth_src,set_field:aa:bb:cc:dd:ee:ff->eth_dst,set_field:172.18.0.3->tun_dst,set_field:0x10/0xf0->reg0,goto_table:L3DecTTL table=L3Forwarding, priority=201,ct_mark=0x1/0xf,ip,in_port=antrea-gw0, nw_dst=10.244.2.0/24 actions=set_field:ba:5e:d1:55:aa:c0->eth_src,set_field:aa:bb:cc:dd:ee:ff->eth_dst,set_field:172.18.0.4->tun_dst,set_field:0x10/0xf0->reg0,goto_table:L3DecTTL table=L3Forwarding, priority=201,ct_mark=0x1/0xf,ip,in_port=antrea-gw0, nw_dst=10.244.3.0/24 actions=set_field:ba:5e:d1:55:aa:c0->eth_src,set_field:aa:bb:cc:dd:ee:ff->eth_dst,set_field:172.18.0.5->tun_dst,set_field:0x10/0xf0->reg0,goto_table:L3DecTTL ``` Signed-off-by: Hongliang Liu <[email protected]>

Signed-off-by: Hongliang Liu <[email protected]>

tnqn

LGTM

hongliangl · 2025-11-07T04:25:07Z

/test-all

antoninbas

LGTM - but I did ask for a clarification in https://github.com/antrea-io/antrea/pull/7239/files#r2505192467, PTAL

hongliangl · 2025-11-11T06:52:04Z

LGTM - but I did ask for a clarification in https://github.com/antrea-io/antrea/pull/7239/files#r2505192467, PTAL

@antoninbas Replied in #7239 (comment). If the performance loss is acceptable using a unified pkt_mark regardless of traffic modes, we can refine the implementation in a subsequent PR.

Created an issue #7554 to track that.

hongliangl force-pushed the egress-hybrid branch 3 times, most recently from 5bcb6c8 to 2f0eae9 Compare June 26, 2025 17:25

hongliangl marked this pull request as ready for review June 26, 2025 17:26

hongliangl added action/release-note Indicates a PR that should be included in release notes. area/transit/egress Issues or PRs related to Egress (SNAT for traffic egressing the cluster). labels Jun 26, 2025

hongliangl requested review from antoninbas and jianjuns June 26, 2025 17:27

hongliangl force-pushed the egress-hybrid branch 5 times, most recently from 61f8d42 to c36407b Compare June 30, 2025 09:22

hongliangl force-pushed the egress-hybrid branch from c36407b to 15b17e8 Compare July 10, 2025 02:59

hongliangl added this to the Antrea v2.5 release milestone Jul 11, 2025

hongliangl mentioned this pull request Jul 17, 2025

Add supoort for Egress and AntreaProxy DSR when wireguard is enabled #7321

Draft

4 tasks

hongliangl force-pushed the egress-hybrid branch 2 times, most recently from 15b17e8 to afbbf6d Compare July 24, 2025 15:43

luolanzone reviewed Aug 8, 2025

View reviewed changes

jianjuns reviewed Aug 11, 2025

View reviewed changes

hongliangl force-pushed the egress-hybrid branch 2 times, most recently from acf1edf to ca06d56 Compare August 12, 2025 03:07

hongliangl requested review from jianjuns and luolanzone August 12, 2025 03:11

hongliangl force-pushed the egress-hybrid branch 3 times, most recently from b5210d6 to 507b3fe Compare August 12, 2025 09:29

antoninbas reviewed Aug 12, 2025

View reviewed changes

antoninbas reviewed Oct 14, 2025

View reviewed changes

pkg/agent/types/net.go Outdated Show resolved Hide resolved

hongliangl dismissed luolanzone’s stale review via 1bb82b7 October 15, 2025 06:57

hongliangl force-pushed the egress-hybrid branch 2 times, most recently from 1bb82b7 to efb5191 Compare October 15, 2025 06:59

hongliangl requested review from antoninbas and luolanzone October 16, 2025 04:13

antoninbas reviewed Oct 16, 2025

View reviewed changes

pkg/agent/route/route_linux_test.go Outdated Show resolved Hide resolved

pkg/agent/types/net.go Outdated Show resolved Hide resolved

hongliangl force-pushed the egress-hybrid branch 2 times, most recently from 24013fb to 9e0e31b Compare October 20, 2025 05:43

hongliangl requested a review from antoninbas October 20, 2025 05:53

hongliangl force-pushed the egress-hybrid branch 2 times, most recently from 383dfe9 to 64cb0d9 Compare October 24, 2025 07:07

tnqn reviewed Oct 24, 2025

View reviewed changes

hongliangl force-pushed the egress-hybrid branch from d51733a to eacb369 Compare October 29, 2025 02:28

hongliangl requested a review from tnqn October 29, 2025 02:28

hongliangl force-pushed the egress-hybrid branch 3 times, most recently from 46ec86a to 01e4486 Compare November 3, 2025 02:24

hongliangl added 2 commits November 3, 2025 10:25

For review comments

9f4ef0e

Signed-off-by: Hongliang Liu <[email protected]>

hongliangl force-pushed the egress-hybrid branch from 01e4486 to 9f4ef0e Compare November 3, 2025 02:25

tnqn approved these changes Nov 7, 2025

View reviewed changes

antoninbas approved these changes Nov 7, 2025

View reviewed changes

hongliangl mentioned this pull request Nov 11, 2025

Use a generic pkt_mark to mark Egress traffic #7554

Open

antoninbas merged commit 6a1997f into antrea-io:main Nov 12, 2025
69 of 72 checks passed

xliuxu mentioned this pull request Dec 5, 2025

Egress in hybrid mode does not work when accessing local IPs on the Egress Node #7612

Closed

Add support for Antrea Egress in hybrid mode #7239

Add support for Antrea Egress in hybrid mode #7239

Uh oh!

Conversation

hongliangl commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

luolanzone commented Aug 4, 2025

Uh oh!

hongliangl commented Aug 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jianjuns Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

hongliangl Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

antoninbas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

antoninbas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tnqn left a comment

Choose a reason for hiding this comment

Uh oh!

hongliangl commented Nov 7, 2025

Uh oh!

antoninbas left a comment

Choose a reason for hiding this comment

Uh oh!

hongliangl commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hongliangl commented Jun 19, 2025 •

edited

Loading

hongliangl commented Nov 11, 2025 •

edited

Loading