Checkpoint and restart recovery

[Random-Liu]

There are several restart recovery problems with current cri-containerd:

1. cri-containerd restart. Because cri-containerd maintains all internal state in-memory, including sandbox list, container list and image list, once restarted all state will be lost.
2. containerd restart. When containerd restart and reconnect, there may be state mismatch between containerd and cri-containerd, e.g. a container dies during containerd is down.

To fix this, we should recover/reconcile state during cri-containerd start or after containerd restart and reconnect.

There are 3 kinds of internal state:

1. Image list. Containerd has all the information we need, we just need to list images from containerd and recover the image list.
2. Sandbox/container metadata: Most of the metadata is not provided by containerd, we need to checkpoint them for restart recovery. However, because metadata is constant, we could save it into containerd container label so as to leverage containerd metadata store to save it for us.
3. Container status: Container status is not persisted by containerd, we need to persist it ourselves. And because it's constantly changing, we may not want to abuse containerd container label to save it. So we need to maintain its checkpoint ourselves.

/cc @kubernetes-incubator/maintainers-cri-containerd

[Random-Liu]

@yanxuean Are you interested in this? We could start from a simple PR, just put container/sandbox metadata into container label when create container.

[yanxuean]

@Random-Liu Ok.

[Random-Liu]

Just FYI. We could leverage containerd/containerd#1378 to store our metadata.

[abhi]

@yanxuean thanks for offering help with this. Let us know if you got a chance to take a look at it and if you need any help from us.

[yanxuean]

OK. I'm studying the RunPodSandbox function now. If there is a need I will.
@Random-Liu Do you means that we should put sandboxstore.Sandbox into containers.Container.extensions by json string?

[Random-Liu]

@yanxuean sandboxstore.Sandbox.Metadata and sandbox.Container.Metadata.

And if we use the new extensions, we don't need to convert it to json. Just put the type into Any direclty.

[Random-Liu]

@yanxuean How's going? The first step is relatively straight forward, but this feature itself is important to this release. :) If you haven't got a chance to look at this, we have to start working on this now. :)

[yanxuean]

@Random-Liu I modified it, and I'm testing it, but it always is fail.
because containerd/containerd#1378 has not commited, I merge it on local(api/next.pb.txt, containers.proto, container.go), and add withextension function.
I'm analyzing the reason.

[Random-Liu]

@yanxuean Could you add them to container labels for the first version? So that we could start consuming that information.

After containerd/containerd#1378 is merged, we could leverage it, but the restart recovery logic won't be affected. :)

[yanxuean]

@Random-Liu OK, I modify it today

[Random-Liu]

@yanxuean Thanks!