Skip to content
This repository was archived by the owner on Mar 9, 2022. It is now read-only.

Conversation

@Random-Liu
Copy link
Member

@Random-Liu Random-Liu commented Oct 5, 2017

Currently, containerd returns stats with empty contents for stopped task (See https://github.com/containerd/containerd/blob/master/linux/task.go#L256).

If we access fields of the stats directly, it will cause a panic, and I did see it in my test:

I1004 23:26:45.253172     370 instrumented_service.go:290] ImageFsInfo returns filesystem info [&FilesystemUsage{Timestamp:1507159602508004076,StorageId:&StorageIdentifier{Uuid:70045b4a-dd4c-4e37-889c-f1992da26c82,},UsedBytes:&UInt64Value{Value:861907840,},InodesUsed:&UInt64Value{Value:27594,},}]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x100 pc=0x1782c1f]

goroutine 770 [running]:
github.com/kubernetes-incubator/cri-containerd/pkg/server.(*criContainerdService).getContainerMetrics(0xc4200fa700, 0xc420177e80, 0x40, 0xc4203a0840, 0x54, 0xc420177ec0, 0x40, 0xc4202a64e0, 0xc42037cd70, 0x47, ...)
	/home/lantaol/workspace/src/github.com/kubernetes-incubator/cri-containerd/pkg/server/container_stats_list.go:111 +0x58f
github.com/kubernetes-incubator/cri-containerd/pkg/server.(*criContainerdService).toCRIContainerStats(0xc4200fa700, 0xc420732d00, 0x7, 0x8, 0xc420814400, 0x7, 0x8, 0xc4201738e0, 0x0, 0x0)
	/home/lantaol/workspace/src/github.com/kubernetes-incubator/cri-containerd/pkg/server/container_stats_list.go:62 +0x289
github.com/kubernetes-incubator/cri-containerd/pkg/server.(*criContainerdService).ListContainerStats(0xc4200fa700, 0x7fd1e08f6640, 0xc4205b5e00, 0xc4205d4370, 0x1, 0x1, 0x19057c0)
	/home/lantaol/workspace/src/github.com/kubernetes-incubator/cri-containerd/pkg/server/container_stats_list.go:45 +0x2ce
github.com/kubernetes-incubator/cri-containerd/pkg/server.(*instrumentedService).ListContainerStats(0xc42000e078, 0x7fd1e08f6640, 0xc4205b5e00, 0xc4205d4370, 0x0, 0x0, 0x0)
	/home/lantaol/workspace/src/github.com/kubernetes-incubator/cri-containerd/pkg/server/instrumented_service.go:317 +0x12e
github.com/kubernetes-incubator/cri-containerd/vendor/k8s.io/kubernetes/pkg/kubelet/apis/cri/v1alpha1/runtime._RuntimeService_ListContainerStats_Handler(0x1a1b7a0, 0xc42000e078, 0x7fd1e08f6640, 0xc4205b5e00, 0xc42068c500, 0x0, 0x0, 0x0, 0xc42033659a, 0x0)
	/home/lantaol/workspace/src/github.com/kubernetes-incubator/cri-containerd/vendor/k8s.io/kubernetes/pkg/kubelet/apis/cri/v1alpha1/runtime/api.pb.go:4079 +0x28d
github.com/kubernetes-incubator/cri-containerd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc4201b6240, 0x23c6fe0, 0xc4203eaf20, 0xc4201ea5a0, 0xc420430ab0, 0x23a44f0, 0xc4205b5ec0, 0x0, 0x0)
	/home/lantaol/workspace/src/github.com/kubernetes-incubator/cri-containerd/vendor/google.golang.org/grpc/server.go:753 +0xcd0
github.com/kubernetes-incubator/cri-containerd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc4201b6240, 0x23c6fe0, 0xc4203eaf20, 0xc4201ea5a0, 0xc4205b5ec0)
	/home/lantaol/workspace/src/github.com/kubernetes-incubator/cri-containerd/vendor/google.golang.org/grpc/server.go:955 +0x15a0
github.com/kubernetes-incubator/cri-containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc42025a810, 0xc4201b6240, 0x23c6fe0, 0xc4203eaf20, 0xc4201ea5a0)
	/home/lantaol/workspace/src/github.com/kubernetes-incubator/cri-containerd/vendor/google.golang.org/grpc/server.go:517 +0xa9
created by github.com/kubernetes-incubator/cri-containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/home/lantaol/workspace/src/github.com/kubernetes-incubator/cri-containerd/vendor/google.golang.org/grpc/server.go:518 +0xa1

This PR fixes it.

@abhi /cc @crosbymichael
Signed-off-by: Lantao Liu [email protected]

Signed-off-by: Lantao Liu <[email protected]>
Copy link
Member

@abhi abhi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Random-Liu Random-Liu added the lgtm label Oct 5, 2017
@Random-Liu
Copy link
Member Author

Apply LGTM based on #328 (review).

@Random-Liu Random-Liu merged commit 3a5ec1c into containerd:master Oct 5, 2017
@Random-Liu Random-Liu deleted the fix-container-stats-panic branch October 5, 2017 04:45
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants