Using runtime-tools to test containerd

[alban]

Following discussion on opencontainers/oci-conformance#36, I am trying to see how to use runtime-tools to test containerd.

I am exploring how to write a wrapper around containerd implementing the OCI Runtime Command Line Interface that runtime-tools would use as a $RUNTIME to talk to containerd.

containerd's ctr CLI tool can run (create+start) a container or start it. There is no CLI for creating one without starting it but it can be done both at the gRPC level or using the Golang client library.

CLI command cdr task start does the following:

• calls the NewTask() method from the client library. This calls the gRPC command c.client.TaskService().Create()
• calls the task.Start() method from the client library. This calls the gRPC command t.client.TaskService().Start

``

runtime-tools tests don't download images from internet but instead they create the config.json and the rootfs themselves with the specific configuration that need to be tested. However, the containerd interface mostly deals with images. It is possible to call create in the gRPC protocol and include config.json on the wire but the root.path will unfortunately not be respected in the last version. A fix is in progress (containerd/containerd#2412, containerd/containerd#2418, thanks @flx42 and @crosbymichael!).

Since runtime-tools tests have no control over which directory containerd will write the config.json received by gRPC, the rootfs prepared by runtime-tools tests will not be in the same directory. This contradicts the OCI specs (see @flx42's comment) but containerd might allow absolute paths in root.path outside of the bundle anyway, so that might be ok for the containerd wrapper to rely on that.

[alban]

There is no CLI for creating one without starting it but it can be done both at the gRPC level or using the Golang client library.

Oh, create exists now. Sorry, I was looking at an old version.

[alban]

My tests so far:

• checkout this branch of containerd: Handle abs path for rootfs in oci hook containerd/containerd#2418
• Compile and start containerd:

$ make
$ sudo systemd-run -p Delegate=yes -p KillMode=process $PWD/bin/containerd

• run a test:

$ sudo RUNTIME=$PWD/ctroci.sh validation/process_rlimits/process_rlimits.t

With the following ctroci.sh script:

#!/bin/bash

CTR=/home/alban/go/src/github.com/containerd/containerd/bin/ctr

echo "$@" >> /tmp/ctroci.log

verb=$1
shift

docreate () {
	if [ "$1" == "--pid-file" ] ; then
		pidfileparam="--pid-file $2"
		shift 2
	fi
	if [ "$1" == "--bundle" ] ; then
		bundle=$2
		configfile=$bundle/config.json
		rootfs=`jq -r '.root.path' < $configfile`
		if ! [[ "$rootfs" =~ ^/ ]]; then
			rootfs=`readlink -f $bundle/$rootfs`
		fi
		shift 2
	fi

	echo $CTR -n oci-tests containers create $pidfileparam --config $configfile --rootfs $rootfs $@ >> /tmp/ctroci.log
	$CTR -n oci-tests containers create $pidfileparam --config $configfile --rootfs $rootfs $@

	echo $CTR -n oci-tests tasks start $@ >> /tmp/ctroci.log
	$CTR -n oci-tests tasks start $@
}

dostart () {
	echo no start >> /tmp/ctroci.log
}

dostate () {
	echo $CTR -n oci-tests containers info $@ >> /tmp/ctroci.log
	$CTR -n oci-tests containers info $@
}

dodelete () {
	echo $CTR -n oci-tests containers delete $@ >> /tmp/ctroci.log
	$CTR -n oci-tests containers delete $@
}


case $verb in
create)
	docreate "$@"
	;;
start)
	dostart "$@"
	;;
state)
	dostate "$@"
	;;
delete)
	dodelete "$@"
	;;
*)
	echo "unknown command $verb" >> /tmp/ctroci.log
	;;
esac

The test fails with timeout in waiting for the container status because $RUNTIME state is not done yet and cdr task start needs to be split up as mentioned above.

[crosbymichael]

Ok, I think we need to add a ctr task create and you should be good.

[alban]

@crosbymichael I started on containerd/containerd#2486

I could get it working with e.g. test validation/linux_cgroups_memory/linux_cgroups_memory.t that use a "create" without a "start".

I'll try to get the "state" working now.

[alban]

I could get some tests to pass using:

• this containerd branch: https://github.com/kinvolk/containerd/commits/alban/ocitests
• this ctroci.sh script https://gist.github.com/alban/793274bd3fc063c3ff51a2c795b15669

The difficulties are:

• adding ctr task create. WIP: Implement ctr task create containerd/containerd#2486
• get the OCI state of a container. For now, I use runc state as a workaround. Get the OCI state of a container containerd/containerd#2491
• ctr task start deletes the container when it terminates. Work around in kinvolk/containerd@798ef1b. But it should be fine once we use the gRPC interface directly.
• getting the stdout. runc prints the stdout on the fd of the create phase but containerd prints the output on the fd of the start phase. I used a pipe in ctroci.sh as a workaround.

Then, I can compare containerd and runc. I used runtime-tools with #658.

sudo make TAP=$(which tap) RUNTIME=/home/alban/go/src/github.com/containerd/ctroci/ctroci.sh localvalidation 2>&1 | tee /tmp/containerd.logs

sudo make TAP=$(which tap) RUNTIME=runc localvalidation 2>&1 | tee /tmp/runc.logs

Outputs:

• runc.logs
• containerd.logs

I'll look at the details tomorrow.