iso/kicbase: bump to containerd/nerdctl 2.x #21409

nirs · 2025-08-24T12:06:05Z

This is #21368 with the following changes:

Update kubernetes versions to be compatible with containerd v2
Rebase on master
Fix whitespace in hash files
Unify containerd-bin makefiles
Update contained-bin Config.in
Remove containerd.conf.d directory
Add missing _AARCH64 to variable names
Bump nerdctl to 2.1.3 (it must be upgraded with containerd)

Fixes #20497

Based on #21490 for testing

k8s-ci-robot · 2025-08-24T12:06:07Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

k8s-ci-robot · 2025-08-24T12:06:11Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nirs
Once this PR has been reviewed and has the lgtm label, please assign spowelljr for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

nirs · 2025-08-24T12:06:26Z

/ok-to-build-iso

nirs · 2025-08-24T13:42:28Z

/ok-to-build-iso

nirs · 2025-08-24T17:10:13Z

/ok-to-build-iso

deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.mk

nirs · 2025-08-24T17:26:42Z

/ok-to-build-iso

medyagh · 2025-08-25T18:28:18Z

See the logs at:
https://storage.googleapis.com/minikube-builds/logs/21409/5ca83e9/iso_build.txt

for example for this PR
https://storage.googleapis.com/minikube-builds/logs/20669/22ed44e/iso_build.txt

5ca83e9

medyagh · 2025-08-25T18:37:54Z

/ok-to-build-iso

minikube-bot · 2025-08-25T23:05:41Z

Hi @nirs, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

nirs

The config.toml does not help with #21408 and may break code configuring containerd using sed(!?). We need to replace the configuration code with proper toml parsing but for now we can keep the existing config.

deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.mk

medyagh · 2025-08-28T18:39:13Z

/ok-to-test

nirs · 2025-08-31T21:52:34Z

/ok-to-build-iso

medyagh · 2025-09-03T17:22:41Z

deploy/iso/minikube-iso/arch/x86_64/package/containerd-bin/containerd-bin.mk

 	$(INSTALL) -Dm755 \
 		$(@D)/bin/containerd-shim-runc-v2 \
 		$(TARGET_DIR)/usr/bin
 	$(INSTALL) -Dm755 \
 		$(@D)/bin/ctr \
 		$(TARGET_DIR)/usr/bin
+	$(INSTALL) -Dm755 \


whats this ?

Part of the release, looks like a stress test. Maybe we can remove it but I'm not sure it worth the effort.

medyagh

nice job now that did this nice surgery how about adding a tiny Test to the functional test,
that systemctl status the containerd and checks for No Errors or Weird Failures?
minikube ssh --- systemctl... or journalctl ... time to add some Tiny Tests for Container Runtimes
or maybe checking if containerd loaded the Config without any parsing issues ? so in future when someone breaks the config we can see in a small subtest

medyagh

~~-lets make a follow up PR to bump for KIC as well https://github.com/kubernetes/minikube/blob/af31c001ad72c82983c1cc59a80689fef4c0de94/deploy/kicbase/Dockerfile#L51-~~

medyagh · 2025-09-03T17:31:17Z

/ok-to-build-image

medyagh · 2025-09-03T17:31:33Z

title of PR should be iso/kicbase:

nirs · 2025-09-03T19:48:39Z

nice job now that did this nice surgery how about adding a tiny Test to the functional test, that systemctl status the containerd and checks for No Errors or Weird Failures? minikube ssh --- systemctl... or journalctl ... time to add some Tiny Tests for Container Runtimes or maybe checking if containerd loaded the Config without any parsing issues ? so in future when someone breaks the config we can see in a small subtest

Good idea for future work, but this should not be a test but a self-test that minkube does after configuring containerd (or other components).

For the implementation we should use only command returning machine readable format.

minikube ssh -- sudo nerdctl info --format json

{
  "ID": "e49f45dc-65fc-4ecd-99d5-469a39db14ee",
  "Driver": "overlayfs",
  "Plugins": {
    "Log": [
      "fluentd",
      "journald",
      "json-file",
      "none",
      "syslog"
    ],
    "Storage": [
      "native",
      "overlayfs"
    ]
  },
  "MemoryLimit": true,
  "SwapLimit": true,
  "CpuCfsPeriod": true,
  "CpuCfsQuota": true,
  "CPUShares": true,
  "CPUSet": true,
  "CPURealtime": false,
  "PidsLimit": true,
  "IPv4Forwarding": true,
  "BridgeNfIptables": true,
  "BridgeNfIp6tables": true,
  "OomKillDisable": false,
  "SystemTime": "2025-09-03T19:43:48.800224305Z",
  "LoggingDriver": "json-file",
  "CgroupDriver": "systemd",
  "CgroupVersion": "2",
  "KernelVersion": "6.6.95",
  "OperatingSystem": "Buildroot 2025.02",
  "OSType": "linux",
  "Architecture": "aarch64",
  "NCPU": 2,
  "MemTotal": 6222721024,
  "Name": "minikube",
  "ServerVersion": "v2.1.4",
  "SecurityOptions": [
    "name=seccomp,profile=builtin",
    "name=cgroupns"
  ],
  "Warnings": null
}

minikube ssh -- systemctl show containerd --no-pager -p LoadState,ActiveState,MainPID,ControlPID,SubState,StateChangeTimestamp

MainPID=780
ControlPID=0
LoadState=loaded
ActiveState=active
SubState=running
StateChangeTimestamp=Wed 2025-09-03 18:56:29 UTC

minikube-bot · 2025-09-03T22:10:28Z

Hi @nirs, we have updated your PR with the reference to newly built kicbase image. Pull the changes locally if you want to test with them or update your PR further.

medyagh · 2025-09-04T03:56:12Z

/retest-this-please

medyagh · 2025-09-04T05:12:37Z

@nirs please take a look at Docker Linux conainerd test https://storage.googleapis.com/minikube-builds/logs/21409/41274/Docker_Linux_containerd.html#fail_TestDockerEnvContainerd

medyagh · 2025-09-04T05:24:13Z

/restest-this-please

minikube-pr-bot · 2025-09-04T06:39:28Z

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21409 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 49.5s    │ 49.9s                  │
│ enable ingress │ 15.8s    │ 16.1s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 51.3s 49.6s 48.9s 48.6s 49.0s
Times for minikube (PR 21409) start: 51.0s 48.3s 53.2s 48.9s 47.9s

Times for minikube ingress: 15.8s 15.8s 15.8s 15.8s 15.8s
Times for minikube (PR 21409) ingress: 16.3s 15.8s 16.3s 16.3s 15.8s

docker driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21409 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 24.9s    │ 25.1s                  │
│ enable ingress │ 13.6s    │ 13.6s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 24.9s 22.9s 26.3s 25.1s 25.1s
Times for minikube (PR 21409) start: 24.0s 26.4s 25.8s 23.8s 25.3s

Times for minikube ingress: 13.6s 13.6s 13.6s 13.6s 13.6s
Times for minikube (PR 21409) ingress: 13.6s 13.6s 13.6s 13.6s 13.6s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21409 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 21.8s    │ 22.4s                  │
│ enable ingress │ 27.2s    │ 23.7s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 21.9s 22.0s 22.3s 20.9s 22.1s
Times for minikube (PR 21409) start: 22.6s 21.9s 22.8s 22.8s 21.7s

Times for minikube ingress: 23.6s 24.1s 24.1s 24.1s 40.1s
Times for minikube (PR 21409) ingress: 24.1s 24.1s 24.1s 22.6s 23.6s

minikube-pr-bot · 2025-09-04T07:36:25Z

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment	Test Name	Flake Rate
Docker_Linux_crio_arm64 (6 failed)	TestFunctional/parallel/ServiceCmdConnect(gopogh)	0.00% (chart)
Docker_Linux_crio_arm64 (6 failed)	TestFunctional/parallel/ServiceCmd/DeployApp(gopogh)	0.00% (chart)
Docker_Linux_crio_arm64 (6 failed)	TestFunctional/parallel/ServiceCmd/HTTPS(gopogh)	0.00% (chart)
Docker_Linux_crio_arm64 (6 failed)	TestFunctional/parallel/ServiceCmd/Format(gopogh)	0.00% (chart)
Docker_Linux_crio_arm64 (6 failed)	TestFunctional/parallel/ServiceCmd/URL(gopogh)	0.00% (chart)
Docker_Linux_containerd_arm64 (1 failed)	TestDockerEnvContainerd(gopogh)	0.00% (chart)
Docker_Windows (1 failed)	TestErrorSpam/setup(gopogh)	Unknown
Docker_Linux_containerd (1 failed)	TestDockerEnvContainerd(gopogh)	0.00% (chart)
KVM_Linux_crio (4 failed)	TestNoKubernetes/serial/StartNoArgs(gopogh)	2.13% (chart)

Besides the following environments also have failed tests:

Docker_Linux_crio: 16 failed (gopogh)

To see the flake rates of all tests by environment, click here.

nirs · 2025-09-04T16:35:44Z

@nirs please take a look at Docker Linux conainerd test https://storage.googleapis.com/minikube-builds/logs/21409/41274/Docker_Linux_containerd.html#fail_TestDockerEnvContainerd

I did not see this failure before and have no idea bout the dockerenv feature and also cannot test it (no docker here).

Lets do a retest.

nirs · 2025-09-04T16:35:57Z

/retest

nirs · 2025-09-04T17:15:57Z

We are missing containerd update for kicbase:

$ minikube start --driver docker --container-runtime containerd
😄  minikube v1.36.0 on Fedora 42 (kvm/amd64)
✨  Using the docker driver based on user configuration
📌  Using Docker driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.47-1756936034-21409 ...
🔥  Creating docker container (CPUs=2, Memory=12000MB) ...
📦  Preparing Kubernetes v1.34.0 on containerd 1.7.27 ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass

❗  /usr/local/bin/kubectl is version 1.32.1, which may have incompatibilities with Kubernetes 1.34.0.
    ▪ Want kubectl v1.34.0? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

nirs · 2025-09-04T17:21:21Z

also make update-containers-version is broken with this change:

$ make update-containerd-version 
cd hack && go run update/containerd_version/containerd_version.go
I0904 20:19:56.988592   24785 update.go:91] The Plan:
{
  "deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.mk": {
    "Content": null,
    "Replace": {
      "CONTAINERD_BIN_AARCH64_COMMIT = .*": "CONTAINERD_BIN_AARCH64_COMMIT = 75cb2b7193e4e490e9fbdc236c0e811ccaba3376",
      "CONTAINERD_BIN_AARCH64_VERSION = .*": "CONTAINERD_BIN_AARCH64_VERSION = v2.1.4"
    }
  },
  "deploy/iso/minikube-iso/arch/x86_64/package/containerd-bin/containerd-bin.mk": {
    "Content": null,
    "Replace": {
      "CONTAINERD_BIN_COMMIT = .*": "CONTAINERD_BIN_COMMIT = 75cb2b7193e4e490e9fbdc236c0e811ccaba3376",
      "CONTAINERD_BIN_VERSION = .*": "CONTAINERD_BIN_VERSION = v2.1.4"
    }
  }
}
I0904 20:19:56.990394   24785 update.go:100] Local repo successfully updated

$ git diff
diff --git a/deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.hash b/deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.hash
index 2496b9d51..22c57a285 100644
--- a/deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.hash
+++ b/deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.hash
@@ -1 +1,2 @@
 sha256 846d13bc2bf1c01ae2f20d13beb9b3a1e50b52c86e955b4ac7d658f5847f2b0e containerd-2.1.4-linux-arm64.tar.gz
+sha256 8c409f8a0ba6015cb365a95e039a0fc9f3663c891a26eb52c0fb8cd291ba75d4 v2.1.4.tar.gz
diff --git a/deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.mk b/deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.mk
index 9b44802e2..85591afe5 100644
--- a/deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.mk
+++ b/deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.mk
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-CONTAINERD_BIN_AARCH64_VERSION = 2.1.4
+CONTAINERD_BIN_AARCH64_VERSION = v2.1.4
 CONTAINERD_BIN_AARCH64_SITE = https://github.com/containerd/containerd/releases/download/v$(CONTAINERD_BIN_AARCH64_VERSION)
 CONTAINERD_BIN_AARCH64_SOURCE = containerd-$(CONTAINERD_BIN_AARCH64_VERSION)-linux-arm64.tar.gz
 CONTAINERD_BIN_AARCH64_STRIP_COMPONENTS = 0
diff --git a/deploy/iso/minikube-iso/arch/x86_64/package/containerd-bin/containerd-bin.hash b/deploy/iso/minikube-iso/arch/x86_64/package/containerd-bin/containerd-bin.hash
index 45abbd6ee..dd3ae3bcd 100644
--- a/deploy/iso/minikube-iso/arch/x86_64/package/containerd-bin/containerd-bin.hash
+++ b/deploy/iso/minikube-iso/arch/x86_64/package/containerd-bin/containerd-bin.hash
@@ -1 +1,2 @@
 sha256 316d510a0428276d931023f72c09fdff1a6ba81d6cc36f31805fea6a3c88f515 containerd-2.1.4-linux-amd64.tar.gz
+sha256 8c409f8a0ba6015cb365a95e039a0fc9f3663c891a26eb52c0fb8cd291ba75d4 v2.1.4.tar.gz
diff --git a/deploy/iso/minikube-iso/arch/x86_64/package/containerd-bin/containerd-bin.mk b/deploy/iso/minikube-iso/arch/x86_64/package/containerd-bin/containerd-bin.mk
index 40c3338c2..434d8b493 100644
--- a/deploy/iso/minikube-iso/arch/x86_64/package/containerd-bin/containerd-bin.mk
+++ b/deploy/iso/minikube-iso/arch/x86_64/package/containerd-bin/containerd-bin.mk
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-CONTAINERD_BIN_VERSION = 2.1.4
+CONTAINERD_BIN_VERSION = v2.1.4
 CONTAINERD_BIN_SITE = https://github.com/containerd/containerd/releases/download/v$(CONTAINERD_BIN_VERSION)
 CONTAINERD_BIN_SOURCE = containerd-$(CONTAINERD_BIN_VERSION)-linux-amd64.tar.gz
 CONTAINERD_BIN_STRIP_COMPONENTS = 0

Issues:

The update breaks the makefile
containerd kicbase version is not updated

medyagh · 2025-09-04T17:34:09Z

@nirs regarding the test TestDockerEnvContainerd it is a legit failure I did 2 retests,

if you wanna know what this test is

it evals minikube docker-env in to the bash and it expects the "Docker client" talks to the Containerd inside minikube
so the user does "docker build" or "docker image ls" it should build against containerd inside minikbue or list images inside minikube

this is done by some network technics we do I forgot is it port forwarding or what

medyagh · 2025-09-04T17:41:46Z

take a look at these two PRs #15452 #16761 of how we implemened docker-env for containerd

k8s-ci-robot · 2025-09-06T06:03:41Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 24, 2025

k8s-ci-robot requested a review from ComradeProgrammer August 24, 2025 12:06

k8s-ci-robot requested a review from prezha August 24, 2025 12:06

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 24, 2025

nirs commented Aug 24, 2025

View reviewed changes

deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.mk Outdated Show resolved Hide resolved

nirs force-pushed the containerd2 branch from b1b7860 to 5ca83e9 Compare August 24, 2025 17:25

nirs marked this pull request as ready for review August 24, 2025 19:06

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 24, 2025

k8s-ci-robot requested a review from medyagh August 24, 2025 19:06

nirs mentioned this pull request Aug 24, 2025

containerd: preloaded images fail to save #21408

Open

nirs marked this pull request as draft August 26, 2025 21:56

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 26, 2025

nirs commented Aug 26, 2025

View reviewed changes

deploy/iso/minikube-iso/arch/aarch64/package/containerd-bin-aarch64/containerd-bin.mk Show resolved Hide resolved

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Aug 28, 2025

This comment has been minimized.

Sign in to view

nirs force-pushed the containerd2 branch 2 times, most recently from d56d7f2 to 0d79fb4 Compare August 31, 2025 21:49

medyagh reviewed Sep 3, 2025

View reviewed changes

nirs changed the title ~~ISO: bump to containerd 2~~ iso/kicbase: bump to containerd 2 Sep 3, 2025

nirs changed the title ~~iso/kicbase: bump to containerd 2~~ iso/kicbase: bump to containerd/nerdctl 2.x Sep 3, 2025

nirs requested a review from medyagh September 3, 2025 18:01

This comment has been minimized.

Sign in to view

Updating kicbase image to v0.0.47-1756936034-21409

0aa34a4

nirs mentioned this pull request Sep 3, 2025

crictl rmi removes all images with the same image id instead of the specified tag kubernetes-sigs/cri-tools#1911

Open

This comment has been minimized.

Sign in to view

nirs marked this pull request as draft September 4, 2025 17:15

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 4, 2025

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 6, 2025

iso/kicbase: bump to containerd/nerdctl 2.x #21409

Are you sure you want to change the base?

iso/kicbase: bump to containerd/nerdctl 2.x #21409

Uh oh!

Conversation

nirs commented Aug 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Aug 24, 2025

Uh oh!

k8s-ci-robot commented Aug 24, 2025

Uh oh!

nirs commented Aug 24, 2025

Uh oh!

nirs commented Aug 24, 2025

Uh oh!

nirs commented Aug 24, 2025

Uh oh!

Uh oh!

nirs commented Aug 24, 2025

Uh oh!

medyagh commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

medyagh commented Aug 25, 2025

Uh oh!

minikube-bot commented Aug 25, 2025

Uh oh!

nirs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

medyagh commented Aug 28, 2025

Uh oh!

This comment has been minimized.

nirs commented Aug 31, 2025

Uh oh!

medyagh Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

nirs Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

medyagh left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

medyagh left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

medyagh commented Sep 3, 2025

Uh oh!

medyagh commented Sep 3, 2025

Uh oh!

nirs commented Sep 3, 2025

Uh oh!

This comment has been minimized.

minikube-bot commented Sep 3, 2025

Uh oh!

medyagh commented Sep 4, 2025

Uh oh!

This comment has been minimized.

medyagh commented Sep 4, 2025

Uh oh!

medyagh commented Sep 4, 2025

Uh oh!

This comment has been minimized.

minikube-pr-bot commented Sep 4, 2025

Uh oh!

minikube-pr-bot commented Sep 4, 2025

Uh oh!

nirs commented Sep 4, 2025

Uh oh!

nirs commented Sep 4, 2025

Uh oh!

nirs commented Sep 4, 2025

Uh oh!

nirs commented Sep 4, 2025

Uh oh!

nirs commented Aug 24, 2025 •

edited

Loading

medyagh commented Aug 25, 2025 •

edited

Loading

medyagh left a comment •

edited

Loading

medyagh left a comment •

edited

Loading

medyagh commented Sep 4, 2025 •

edited

Loading