1.33 kubelite crash loop

[AlexGustafsson]

Summary

After an upgrade to 1.33 the cluster starts, but quickly stops working again. Looking through the logs, kubelite panics.

Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: E0426 16:05:23.053843   71239 runtime.go:142] "Observed a panic" panic="runtime error: invalid memory address or nil pointer dereference" panicGoValue="\"invalid memory address or nil pointer dereference\"" stacktrace=<
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         goroutine 7114 [running]:
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/apimachinery/pkg/util/runtime.logPanic({0x4f07580, 0x77ed780}, {0x3f9f0c0, 0x77584b0})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/apimachinery/pkg/util/runtime/runtime.go:132 +0xbc
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x4f07580, 0x77ed780}, {0x3f9f0c0, 0x77584b0}, {0x77ed780, 0x0, 0x10000c00d7cb740?})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/apimachinery/pkg/util/runtime/runtime.go:107 +0x116
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x4eb08a0?})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/apimachinery/pkg/util/runtime/runtime.go:64 +0x105
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         panic({0x3f9f0c0?, 0x77584b0?})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 runtime/panic.go:792 +0x132
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).doPodResizeAction(0xc0088d2340, 0xc008d6bb08, {0x0, 0x0, {0xc00b72ec40, 0x40}, 0x2, 0x0, {0x0, 0x0, ...}, ...})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:820 +0x94f
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).SyncPod(0xc0088d2340, {0x4f079c0, 0xc00db24dd0}, 0xc008d6bb08, 0xc00a77e580, {0x77ed780, 0x0, 0x0}, 0xc00881bb00)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:1409 +0x3de8
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet.(*Kubelet).SyncPod(0xc0086f1808, {0x4f07aa0, 0xc0078ad4f0}, 0x2, 0xc008d6bb08, 0x0, 0xc00a77e580)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/kubernetes/pkg/kubelet/kubelet.go:2068 +0x1973
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet.(*podWorkers).podWorkerLoop.func1(...)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/kubernetes/pkg/kubelet/pod_workers.go:1286
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet.(*podWorkers).podWorkerLoop(0xc0088dd040, {0xc008291140, 0x24}, 0xc00a88d180)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/kubernetes/pkg/kubelet/pod_workers.go:1291 +0x57e
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet.(*podWorkers).UpdatePod.func1()
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/kubernetes/pkg/kubelet/pod_workers.go:950 +0x118
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         created by k8s.io/kubernetes/pkg/kubelet.(*podWorkers).UpdatePod in goroutine 5419
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/kubernetes/pkg/kubelet/pod_workers.go:945 +0x20db
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:  >
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: panic: runtime error: invalid memory address or nil pointer dereference [recovered]
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         panic: runtime error: invalid memory address or nil pointer dereference
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x398a9cf]
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: goroutine 7114 [running]:
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x4f07580, 0x77ed780}, {0x3f9f0c0, 0x77584b0}, {0x77ed780, 0x0, 0x10000c00d7cb740?})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/apimachinery/pkg/util/runtime/runtime.go:114 +0x1a9
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x4eb08a0?})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/apimachinery/pkg/util/runtime/runtime.go:64 +0x105
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: panic({0x3f9f0c0?, 0x77584b0?})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         runtime/panic.go:792 +0x132
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).doPodResizeAction(0xc0088d2340, 0xc008d6bb08, {0x0, 0x0, {0xc00b72ec40, 0x40}, 0x2, 0x0, {0x0, 0x0, ...}, ...})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:820 +0x94f
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).SyncPod(0xc0088d2340, {0x4f079c0, 0xc00db24dd0}, 0xc008d6bb08, 0xc00a77e580, {0x77ed780, 0x0, 0x0}, 0xc00881bb00)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:1409 +0x3de8
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).SyncPod(0xc0086f1808, {0x4f07aa0, 0xc0078ad4f0}, 0x2, 0xc008d6bb08, 0x0, 0xc00a77e580)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/kubelet.go:2068 +0x1973
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/kubernetes/pkg/kubelet.(*podWorkers).podWorkerLoop.func1(...)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:1286
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/kubernetes/pkg/kubelet.(*podWorkers).podWorkerLoop(0xc0088dd040, {0xc008291140, 0x24}, 0xc00a88d180)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:1291 +0x57e
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/kubernetes/pkg/kubelet.(*podWorkers).UpdatePod.func1()
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:950 +0x118
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: created by k8s.io/kubernetes/pkg/kubelet.(*podWorkers).UpdatePod in goroutine 5419
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:945 +0x20db

What Should Happen Instead?

Upgrading from 1.32 to 1.33 should work.

Reproduction Steps

1. Run 1.32(.3)/stable
2. Upgrade to 1.33/stable using sudo snap refresh microk8s --channel=1.33/stable

Introspection Report

inspection-report-20250426_161309.tar.gz

Can you suggest a fix?

No.

Are you interested in contributing with a fix?

No (I don't know enough about the issue).

[AlexGustafsson]

Rolling back to 1.32 allows the cluster to start again.

[berkayoz]

Hey @AlexGustafsson,

Thank you for reporting this issue. Sadly I could not reproduce the issue locally. My initial impression is that this is an upstream regression. We'll try to look into what might cause this and update this issue with more information.

[AlexGustafsson]

1.33 on a fresh cluster worked, upgrading to 1.34 and subsequently 1.35 worked as well. I'm ok with closing this as I won't be going back to 1.33 and I'm guessing others have upgraded as well.