Is restarting containerd after configuring container networking required?

[tkellen]

First, thank you for kube-router. I've been using it a lot lately and it works wonderfully!

I'm experiencing an issue related to bootstrapping a cluster that I am hoping you all can help with.

If I don't restart containerd after deploying my kube-router pods, any pod not connected to the host network fails to start.

I have a full example process, including the containerd restart documented here:
https://github.com/tkellen/kwm#first-time-user-guide

If you skip the containerd restart step you'll see that kube-dns (or any other pod not connected to the host network) fails to start with the error "Failed create pod sandbox". with. As soon as you restart containerd, the issue resolves itself. Is this expected? If not, can you help me understand what the correct process for this would be?

Thanks so much!

[tkellen]

I've posted this over on the cri-containerd repo because I'm unclear if this is an issue with kube-router or cri-containerd (or neither, maybe it's just me!). Here's the link:
containerd/cri#545

[murali-reddy]

Not sure of exact reason. But many users seems faced "Failed create pod sandbox" with various CNI plugins

kubernetes/kubernetes#56902
kubernetes/kubeadm#578
kubernetes/kubernetes#55094

[tkellen]

Thanks @murali-reddy! It looks like we've gotten some traction over on containerd/cri#545 so I am going to close this.

[t3hmrman]

tl;dr Just coming to say that I just ran into this, and restarting cri-containerd/containerd is required.

I wasn't getting "Failed to create pod sandbox" errors, but the network was just silently failing. kube-router logs looked good and it wrote configuration like you'd expect, automatically detecting a cidr to use for my cluster (10.244.1.0/24), but I noticed that all the nodes actually had IPs like 10.244.0.41. I thought maybe this wasn't a problem (I also was using canal before switching to kube-router), but it was.

After much head scratching, checking configuration, creating tutum/curl, alpine, and nginx pods trying to access one from the other, I had the thought to systemctl restart cri-containerd containerd and that's what did it. I would have saved myself some time if I'd seen/read containerd/cri#545 sooner

[murali-reddy]

Thanks @t3hmrman for sharing.

[murali-reddy]

@tkellen looks like you are still running into issues as noted in containerd/cri#545 I am reopening here to track and work for resolution.

[Random-Liu]

@murali-reddy @tkellen We think that we should not support dynamically loading and changing cni config, because it will only break the system.

I'm not quite familiar with kube-router, but based on what @tkellen told me, it is doing multi-stage cni config initialization. Given that, I think it is safer to initialize the cni config in a different directory, and move the config to actual cni directly at the end. Or else, node may become ready during the initialization, and any pod created during initialization will be in a bad state.

[wilmardo]

Isn't this fixed by containerd/cri/#825?

[tkellen]

My testing showed that it was not but I have since moved on from bare-metal installations to using EKS. I appreciate all the attention and effort deeply but I'm not presently able to validate :/.

[aauren]

Closing as stale given the status of the containerd ticket. Feel free to re-open with current specifics.