CNI initialization sometimes fails when using "conf_template"

[lbernail]

Hello

We’ve just started seeing a new issue with containerd, and I’m wondering if you could have some ideas:

• we use the conf_template option because we rely on CloudAllocator (GCP)
• sometimes (maybe 10% of the new nodes), the conf is successfully generated but containerd fails to create the sandbox with failed to find network info for sandbox
• restarting containerd fixes the issue

After some initial digging, it looks like containerd generates the (valid) CNI config file and updates its status to Ready (so the kubelet starts to create pods) but doesn’t load it successfully and afterwards does not call the CNI plugin when creating the sandbox (no logs for the CNI plugin and no directory allocation for the ipam host-local plugin).

I look at the code for the CRI Status API and I don't see how it could give a Ready response without a successful though:

cri/pkg/server/status.go

Line 47 in 0e42438

if err := c.netPlugin.Load(cni.WithLoNetwork, cni.WithDefaultConf); err != nil {

No big change on our side except an update to kubelet 1.10.11 (we have been running containerd 1.1.2 for more than 6 monts and had this issue for the first time 3 weeks ago.

Would you have an idea what could be the root cause?
I can of course provide more details and since the issue is happening almost daily I will be able to run additional debug commands

[Random-Liu]

If you see "failed to find network info for sandbox", it means the code executed to: https://github.com/containerd/cri/blob/release/1.0/pkg/server/sandbox_run.go#L548

That will only happen if this if check doesn't go through: https://github.com/containerd/cri/blob/release/1.0/pkg/server/sandbox_run.go#L541

It means that either CNI result doesn't have eth0, or doesn't have an IP. In either case, this seems to be a CNI issue.

[lbernail]

@Random-Liu this was my first idea but we use this configuration:

{
    "cniVersion": "0.3.1",
    "name": "pod-network",
    "plugins": [
        {
            "cniVersion": "0.3.1",
            "type": "ptp",
            "mtu": 1460,
            "ipam": {
                "type": "host-local",
                "subnet": "{{.PodCIDR}}",
                "routes": [
                    {
                        "dst": "0.0.0.0/0"
                    }
                ]
            }
        }
    ]
}

And one of the first thing performed by the host-local plugin is to create the /var/lib/cni/networks/pod-network/ directory:
https://github.com/containernetworking/plugins/blob/a326f9d3f81b47bac27573d9129e488b29bdddf0/plugins/ipam/host-local/main.go#L57

When the issue happens this directory is never created and there is no CNI error in the containerd logs. It's hard to use a custom containerd build in our case (because the issue is a bit rare and would involve large scale updates) but I could verify this with a custom host-local build (or by simply removing the binary when the issue happens).

Could the the CNI configuration somehow be only partially parsed? (loopback only for instance)

[lbernail]

I just verified this: if I remove the CNI plugin binaries (ptp and host-local), the error stays the same when containerd is in this "broken" state.
However once I restart containerd I get:

error="failed to setup network for sandbox "2d707a0bd9ad8705d8798204abf95cb75bb75ce79a186658a1022158a7e61414": failed to find plugin "ptp" in path [/opt/cni/bin]"

So the CNI plugin is not called but containerd is in NetworkReady state and the CNI file has been generated

[lbernail]

I did an additional test: I removed the loopback plugin during the issue. In this case, I get the expected "failed to find plugin "loopback" in path [/opt/cni/bin]"

This seems to indicate that containerd only load the loopback interface configuration and not the rest of it (despite generating a valid cni configuration that works after a containerd restart)

[Random-Liu]

Could the the CNI configuration somehow be only partially parsed? (loopback only for instance)

It is possible, and containerd only periodically reload cni config in 1.2.x+, see #825.

Do you have a cni config file before you start containerd? If you want to use conf_template, you should make sure your cni config directory is empty. Can you check crictl info to see whether your cni config path is what you expect?

And we need #979 to better debug this kind of issue in the future.

[lbernail]

The /etc/cni/net.d directory is empty when the node starts
I tried to reproduce by putting an empty file in there (in case the file was created in the FS before the template was rendered) without success (of course, the empty file is not valid json)

#825 should solve our issue because the next time the kubelet calls the status API, it will reload the valid configuration (do you know how often this call is made?). We plan to migrate to 1.2.2 soon so may be good there (I'd still like to understand the root cause for the issue)

#979 is definitely helpful for debug purposes. I think it could make sense to display this information in debug mode