Skip to content

We have noticed an instance of Reconciliation Defects in your repository #8178

Open
#8187
Open
We have noticed an instance of Reconciliation Defects in your repository#8178
#8187
Labels
kind/bugCategorizes issue or PR as related to a bug.

Description

@zyue110026
zyue110026
opened on Jul 12, 2026

Describe the bug

When deploying Antrea and modifying the proxy layer during initialization, a transient race condition may leave newly created Pods in an inconsistent network state.

Specifically, an affected Pod can lose access to Kubernetes ClusterIP Services (e.g., the Kubernetes API Service at 10.96.0.1:443) even though the Antrea agent reports a healthy status and other Pods on the same node continue to communicate normally.

The issue appears to be limited to the affected Pod's network datapath state. Antrea does not appear to automatically reconcile or repair the connectivity of the affected Pod after the network components become healthy. The connectivity failure persists indefinitely until the Pod is deleted and recreated.

This suggests a potential reconciliation gap during concurrent Service proxy initialization and endpoint/network state updates, where an existing Pod network state is not restored after the transient failure condition disappears.

To Reproduce

  1. Create a standard local cluster using kind.

  2. Simulate a delayed core networking start by scaling CoreDNS down to 0: kubectl scale deployment coredns -n kube-system --replicas=0

  3. Install Antrea: kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/main/build/yamls/antrea.yml

  4. Before Antrea fully completes initialization, repeatedly delete the kube-proxy Pod:
    kubectl get pods -n kube-system -l k8s-app=kube-proxy -o jsonpath='{.items[*].metadata.name}'

  5. Repeat the deletion 10 times.

  6. Wait approximately 20-60 seconds.

  7. Restore CoreDNS:
    kubectl scale deployment coredns -n kube-system --replicas=2

  8. Repeat the kube-proxy deletion 10 times

  9. Observe the CoreDNS status.

The issue is intermittent. In 12 repeated executions of the reproduction procedure, the failure was observed 2 times. The exact timing appears to influence whether the issue occurs.

Expected

Once antrea-agent, antrea-controller, and kube-proxy settle into a Running state, the CNI network rules should completely reconcile. CoreDNS pods should successfully resolve the path to the Kubernetes API Server cluster IP and transition to a 2/2 READY state.

Actual behavior

One CoreDNS replica may enter an unhealthy state:

NAMESPACE      NAME                         READY   STATUS
kube-system    coredns-66bc5c9577-ph45f     0/1     Running
kube-system    coredns-66bc5c9577-smtjj     1/1     Running

After waiting for several minutes, the affected Pod does not recover automatically and eventually enters CrashLoopBackOff:

NAMESPACE      NAME                         READY   STATUS
kube-system    coredns-66bc5c9577-ph45f     0/1     CrashLoopBackOff
kube-system    coredns-66bc5c9577-smtjj     1/1     Running

The CoreDNS container continuously fails because it cannot access the Kubernetes API Service:

[ERROR] plugin/kubernetes: Unhandled Error

[INFO] plugin/kubernetes:
failed to list *v1.Namespace:
Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0":
dial tcp 10.96.0.1:443: connect: no route to host

However, Antrea components remain healthy:

antrea-agent Running
antrea-controller Running

Antrea agent logs:

% kubectl -n kube-system logs ds/antrea-agent --tail=100
I0712 19:00:29.946775       1 node_route_controller.go:331] Reconciliation for AntreaAgentNodeRouteController
I0712 19:00:29.946669       1 shared_informer.go:388] "Caches are synced" controller="ExternalIPPoolController"
I0712 19:00:29.946880       1 cluster.go:528] "Processed Node event" eventType="Join" nodeName="kind-control-plane" affectedExternalIPPoolNum=1
E0712 19:00:29.946967       1 endpoint_resolver.go:239] "Cannot resolve endpoint because Service is unavailable" err="no endpoints available for service \"antrea\"" service="kube-system/antrea"
I0712 19:00:29.947974       1 shared_informer.go:388] "Caches are synced" controller="ServiceExternalIPController"
I0712 19:00:29.947987       1 shared_informer.go:388] "Caches are synced" controller="AntreaAgentTraceflowController"
I0712 19:00:29.948305       1 shared_informer.go:388] "Caches are synced" controller="service config"
I0712 19:00:29.948350       1 shared_informer.go:388] "Caches are synced" controller="endpoint slice config"
I0712 19:00:29.948523       1 shared_informer.go:409] "Caches are synced"
I0712 19:00:29.948675       1 client.go:156] "Didn't get Endpoint URL for Antrea Service, skip updating Antrea Client"
I0712 19:00:30.048884       1 shared_informer.go:388] "Caches are synced" controller="AntreaAgentEgressController"
I0712 19:00:30.049086       1 route_linux.go:1928] "Restoring IP routes and rules for Egress"
I0712 19:00:30.049256       1 egress_controller.go:1214] Starting watch for EgressGroup
E0712 19:00:30.049276       1 egress_controller.go:1217] "Failed to get antrea client" err="Antrea client is not ready"
I0712 19:00:31.847969       1 networkpolicy_controller.go:629] Waiting for Antrea client to be ready
I0712 19:00:33.847551       1 networkpolicy_controller.go:629] Waiting for Antrea client to be ready
I0712 19:00:34.848401       1 networkpolicy_controller.go:637] Stopped waiting for Antrea client
I0712 19:00:34.849013       1 networkpolicy_controller.go:655] Waiting for all watchers to complete full sync
I0712 19:00:34.849545       1 networkpolicy_controller.go:989] "Starting watch" objectType="AppliedToGroup"
I0712 19:00:34.849941       1 networkpolicy_controller.go:989] "Starting watch" objectType="AddressGroup"
I0712 19:00:34.850495       1 networkpolicy_controller.go:989] "Starting watch" objectType="NetworkPolicy"
E0712 19:00:34.850560       1 networkpolicy_controller.go:992] "Failed to start watch" err="Antrea client is not ready" objectType="AddressGroup"
I0712 19:00:34.850673       1 networkpolicy_controller.go:967] "Getting init events from fallback" objectType="AddressGroup"
E0712 19:00:34.850753       1 networkpolicy_controller.go:992] "Failed to start watch" err="Antrea client is not ready" objectType="AppliedToGroup"
I0712 19:00:34.850868       1 networkpolicy_controller.go:967] "Getting init events from fallback" objectType="AppliedToGroup"
E0712 19:00:34.850577       1 networkpolicy_controller.go:992] "Failed to start watch" err="Antrea client is not ready" objectType="NetworkPolicy"
I0712 19:00:34.851446       1 networkpolicy_controller.go:967] "Getting init events from fallback" objectType="NetworkPolicy"
I0712 19:00:34.864610       1 networkpolicy_controller.go:657] All watchers have completed full sync, installing flows for init events
I0712 19:00:34.869108       1 networkpolicy_controller.go:662] Starting NetworkPolicy workers now
I0712 19:00:34.869150       1 networkpolicy_controller.go:668] Starting IDAllocator worker to maintain the async rule cache
I0712 19:00:34.869259       1 status_controller.go:207] Starting NetworkPolicy StatusController
I0712 19:00:34.870134       1 agent.go:675] Cleaning up flow-restore-wait config
I0712 19:00:34.873270       1 agent.go:688] Cleaned up flow-restore-wait config
I0712 19:00:34.873360       1 agent.go:957] "Flow restoration has completed"
I0712 19:00:34.874177       1 collector.go:93] "Waiting for stale flows from previous agent round to be deleted"
I0712 19:00:35.036675       1 serving.go:411] Generated self-signed cert in-memory
I0712 19:00:35.049846       1 egress_controller.go:1214] Starting watch for EgressGroup
E0712 19:00:35.049877       1 egress_controller.go:1217] "Failed to get antrea client" err="Antrea client is not ready"
I0712 19:00:35.270639       1 handler.go:304] Adding GroupVersion system.antrea.io v1beta1 to ResourceManager
I0712 19:00:35.271383       1 agent.go:53] Starting Antrea Agent Monitor
I0712 19:00:35.271528       1 pipeline.go:467] Start collecting OVS meter stats
I0712 19:00:35.274389       1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0712 19:00:35.274514       1 shared_informer.go:402] "Waiting for caches to sync"
I0712 19:00:35.274412       1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController
I0712 19:00:35.274581       1 shared_informer.go:402] "Waiting for caches to sync"
I0712 19:00:35.274425       1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0712 19:00:35.274829       1 shared_informer.go:402] "Waiting for caches to sync"
I0712 19:00:35.278091       1 secure_serving.go:214] Serving securely on [::]:10350
I0712 19:00:35.278137       1 tlsconfig.go:243] "Starting DynamicServingCertificateController"
I0712 19:00:35.375733       1 shared_informer.go:409] "Caches are synced"
I0712 19:00:35.375757       1 shared_informer.go:409] "Caches are synced"
I0712 19:00:35.375771       1 shared_informer.go:409] "Caches are synced"
I0712 19:00:39.703554       1 agent.go:586] Deleting stale flows from previous round if any
I0712 19:00:39.703616       1 agent.go:498] Persisting round number 1 to OVSDB
I0712 19:00:39.705814       1 agent.go:503] Round number 1 was persisted to OVSDB
I0712 19:00:39.705927       1 collector.go:99] Start collecting metrics
I0712 19:00:39.850241       1 networkpolicy_controller.go:989] "Starting watch" objectType="AppliedToGroup"
I0712 19:00:39.850343       1 networkpolicy_controller.go:989] "Starting watch" objectType="NetworkPolicy"
E0712 19:00:39.850356       1 networkpolicy_controller.go:992] "Failed to start watch" err="Antrea client is not ready" objectType="AppliedToGroup"
I0712 19:00:39.850242       1 networkpolicy_controller.go:989] "Starting watch" objectType="AddressGroup"
E0712 19:00:39.850405       1 networkpolicy_controller.go:992] "Failed to start watch" err="Antrea client is not ready" objectType="AddressGroup"
E0712 19:00:39.850355       1 networkpolicy_controller.go:992] "Failed to start watch" err="Antrea client is not ready" objectType="NetworkPolicy"
I0712 19:00:40.050003       1 egress_controller.go:1214] Starting watch for EgressGroup
E0712 19:00:40.050061       1 egress_controller.go:1217] "Failed to get antrea client" err="Antrea client is not ready"
I0712 19:00:41.336737       1 endpoint_resolver.go:275] "Selected a new Endpoint for Service, notifying listeners" service="kube-system/antrea" url="https://172.18.0.2:10349"
I0712 19:00:41.337512       1 client.go:178] Updating Antrea client with the new CA bundle
I0712 19:00:44.850687       1 networkpolicy_controller.go:989] "Starting watch" objectType="NetworkPolicy"
I0712 19:00:44.850686       1 networkpolicy_controller.go:989] "Starting watch" objectType="AddressGroup"
I0712 19:00:44.850696       1 networkpolicy_controller.go:989] "Starting watch" objectType="AppliedToGroup"
I0712 19:00:44.865549       1 networkpolicy_controller.go:1004] Started watch for AppliedToGroup
I0712 19:00:44.865549       1 networkpolicy_controller.go:1004] Started watch for NetworkPolicy
I0712 19:00:44.865594       1 networkpolicy_controller.go:1004] Started watch for AddressGroup
I0712 19:00:44.865781       1 networkpolicy_controller.go:1032] Received 0 init events for AddressGroup
I0712 19:00:44.865840       1 networkpolicy_controller.go:1032] Received 0 init events for AppliedToGroup
I0712 19:00:44.865846       1 networkpolicy_controller.go:1032] Received 0 init events for NetworkPolicy
I0712 19:00:45.050248       1 egress_controller.go:1214] Starting watch for EgressGroup
I0712 19:00:45.053285       1 egress_controller.go:1235] Started watch for EgressGroup
I0712 19:00:45.053348       1 egress_controller.go:1261] Received 0 init events for EgressGroup
I0712 19:02:17.183693       1 server.go:434] "Received CmdAdd request" request="cni_args:{container_id:\"e823dadc3218fff6df5c42b52f80ce76f8923a97fdbcc6d2b65ab8cb23ee8c7f\" netns:\"/var/run/netns/cni-3bba130b-bdb9-cbe1-4318-e9c8c258e47d\" ifname:\"eth0\" args:\"K8S_POD_INFRA_CONTAINER_ID=e823dadc3218fff6df5c42b52f80ce76f8923a97fdbcc6d2b65ab8cb23ee8c7f;K8S_POD_UID=b08abafb-52c9-452c-b6b3-f11c0d3039a6;IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=coredns-66bc5c9577-vrht4\" path:\"/opt/cni/bin\" network_configuration:\"{\\\"cniVersion\\\":\\\"0.3.0\\\",\\\"ipam\\\":{\\\"type\\\":\\\"host-local\\\"},\\\"name\\\":\\\"antrea\\\",\\\"type\\\":\\\"antrea\\\"}\"}"
I0712 19:02:17.184783       1 server.go:434] "Received CmdAdd request" request="cni_args:{container_id:\"7ea0f8e14d23bf1a6c7b1d047222e79b3762d22a147c08a8b378b4f07f13fe8f\" netns:\"/var/run/netns/cni-dd548ad6-b858-3d02-0b32-76cdd1922004\" ifname:\"eth0\" args:\"K8S_POD_UID=584dce39-7b41-4e2a-89f7-1b512db9e309;IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=coredns-66bc5c9577-np4hf;K8S_POD_INFRA_CONTAINER_ID=7ea0f8e14d23bf1a6c7b1d047222e79b3762d22a147c08a8b378b4f07f13fe8f\" path:\"/opt/cni/bin\" network_configuration:\"{\\\"cniVersion\\\":\\\"0.3.0\\\",\\\"ipam\\\":{\\\"type\\\":\\\"host-local\\\"},\\\"name\\\":\\\"antrea\\\",\\\"type\\\":\\\"antrea\\\"}\"}"
I0712 19:02:17.192874       1 server.go:504] "Allocated IP addresses" container="e823dadc3218fff6df5c42b52f80ce76f8923a97fdbcc6d2b65ab8cb23ee8c7f" result={"cniVersion":"1.1.0","ips":[{"address":"10.244.0.2/24","gateway":"10.244.0.1"}]}
I0712 19:02:17.192969       1 server.go:504] "Allocated IP addresses" container="7ea0f8e14d23bf1a6c7b1d047222e79b3762d22a147c08a8b378b4f07f13fe8f" result={"cniVersion":"1.1.0","ips":[{"address":"10.244.0.3/24","gateway":"10.244.0.1"}]}
I0712 19:02:17.214124       1 pod_configuration.go:292] "Configured container interface" Pod="kube-system/coredns-66bc5c9577-np4hf" container="7ea0f8e14d23bf1a6c7b1d047222e79b3762d22a147c08a8b378b4f07f13fe8f" interface="eth0" hostInterface="coredns--4218ff"
I0712 19:02:17.214162       1 server.go:534] "CmdAdd for container succeeded" container="7ea0f8e14d23bf1a6c7b1d047222e79b3762d22a147c08a8b378b4f07f13fe8f"
I0712 19:02:17.216804       1 pod_configuration.go:292] "Configured container interface" Pod="kube-system/coredns-66bc5c9577-vrht4" container="e823dadc3218fff6df5c42b52f80ce76f8923a97fdbcc6d2b65ab8cb23ee8c7f" interface="eth0" hostInterface="coredns--3ec0b6"
I0712 19:02:17.216824       1 server.go:534] "CmdAdd for container succeeded" container="e823dadc3218fff6df5c42b52f80ce76f8923a97fdbcc6d2b65ab8cb23ee8c7f"
I0712 19:13:03.704946       1 server.go:434] "Received CmdAdd request" request="cni_args:{container_id:\"565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb\" netns:\"/var/run/netns/cni-a6cb76fd-3da9-77a7-8bb0-0c4a678b445b\" ifname:\"eth0\" args:\"K8S_POD_UID=a1f628e9-fb3a-466f-9d15-dbad95438394;IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=debug;K8S_POD_INFRA_CONTAINER_ID=565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb\" path:\"/opt/cni/bin\" network_configuration:\"{\\\"cniVersion\\\":\\\"0.3.0\\\",\\\"ipam\\\":{\\\"type\\\":\\\"host-local\\\"},\\\"name\\\":\\\"antrea\\\",\\\"type\\\":\\\"antrea\\\"}\"}"
I0712 19:13:03.720163       1 server.go:504] "Allocated IP addresses" container="565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb" result={"cniVersion":"1.1.0","ips":[{"address":"10.244.0.4/24","gateway":"10.244.0.1"}]}
I0712 19:13:03.739977       1 pod_configuration.go:292] "Configured container interface" Pod="default/debug" container="565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb" interface="eth0" hostInterface="debug-77837f"
I0712 19:13:03.740004       1 server.go:534] "CmdAdd for container succeeded" container="565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb"
I0712 19:13:21.957460       1 server.go:589] "Received CmdDel request" request="cni_args:{container_id:\"565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb\" netns:\"/var/run/netns/cni-a6cb76fd-3da9-77a7-8bb0-0c4a678b445b\" ifname:\"eth0\" args:\"K8S_POD_NAMESPACE=default;K8S_POD_NAME=debug;K8S_POD_INFRA_CONTAINER_ID=565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb;K8S_POD_UID=a1f628e9-fb3a-466f-9d15-dbad95438394;IgnoreUnknown=1\" path:\"/opt/cni/bin\" network_configuration:\"{\\\"cniVersion\\\":\\\"0.3.0\\\",\\\"ipam\\\":{\\\"type\\\":\\\"host-local\\\"},\\\"name\\\":\\\"antrea\\\",\\\"type\\\":\\\"antrea\\\"}\"}"
I0712 19:13:21.959267       1 pod_configuration.go:568] "Deleted container OVS port" container="565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb" interface="debug-77837f"
I0712 19:13:21.965629       1 server.go:560] "Deleted interfaces for container" container="565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb"
I0712 19:13:21.968312       1 server.go:583] "CmdDel for container succeeded" container="565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb"
I0712 19:13:23.734573       1 server.go:589] "Received CmdDel request" request="cni_args:{container_id:\"565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb\" ifname:\"eth0\" args:\"K8S_POD_UID=a1f628e9-fb3a-466f-9d15-dbad95438394;IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=debug;K8S_POD_INFRA_CONTAINER_ID=565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb\" path:\"/opt/cni/bin\" network_configuration:\"{\\\"cniVersion\\\":\\\"0.3.0\\\",\\\"ipam\\\":{\\\"type\\\":\\\"host-local\\\"},\\\"name\\\":\\\"antrea\\\",\\\"type\\\":\\\"antrea\\\"}\"}"
I0712 19:13:23.735291       1 server.go:560] "Deleted interfaces for container" container="565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb"
I0712 19:13:23.741212       1 server.go:583] "CmdDel for container succeeded" container="565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb"
I0712 19:13:23.756774       1 server.go:589] "Received CmdDel request" request="cni_args:{container_id:\"565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb\" ifname:\"eth0\" args:\"K8S_POD_INFRA_CONTAINER_ID=565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb;K8S_POD_UID=a1f628e9-fb3a-466f-9d15-dbad95438394;IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=debug\" path:\"/opt/cni/bin\" network_configuration:\"{\\\"cniVersion\\\":\\\"0.3.0\\\",\\\"ipam\\\":{\\\"type\\\":\\\"host-local\\\"},\\\"name\\\":\\\"antrea\\\",\\\"type\\\":\\\"antrea\\\"}\"}"
I0712 19:13:23.756898       1 server.go:560] "Deleted interfaces for container" container="565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb"
I0712 19:13:23.760455       1 server.go:583] "CmdDel for container succeeded" container="565ae0732bea46c02912148cc382182158ed9973af230a6fbbcda9f5171cbacb"

Versions:

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions