Skip to content
This repository was archived by the owner on Jun 3, 2025. It is now read-only.

Commit 89564d4

Browse files
committed
Add guide on creating multi-arch manifests
While kaniko itself does not natively support building multi-arch container manifests, it may be used in combination with tools such as manifest-tool to create and merge seperate arch builds into a single manifest. Fixes #1102 Fixes #786
1 parent 971b9f4 commit 89564d4

2 files changed

Lines changed: 249 additions & 84 deletions

File tree

README.md

Lines changed: 245 additions & 84 deletions
Original file line numberDiff line numberDiff line change
@@ -35,89 +35,92 @@ _If you are interested in contributing to kaniko, see
3535
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
3636
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
3737

38-
**Table of Contents** _generated with
39-
[DocToc](https://github.com/thlorenz/doctoc)_
40-
41-
- [kaniko - Build Images In Kubernetes](#kaniko---build-images-in-kubernetes)
42-
- [🚨NOTE: kaniko is not an officially supported Google product🚨](#note-kaniko-is-not-an-officially-supported-google-product)
43-
- [Community](#community)
44-
- [How does kaniko work?](#how-does-kaniko-work)
45-
- [Known Issues](#known-issues)
46-
- [Demo](#demo)
47-
- [Tutorial](#tutorial)
48-
- [Using kaniko](#using-kaniko)
49-
- [kaniko Build Contexts](#kaniko-build-contexts)
50-
- [Using Azure Blob Storage](#using-azure-blob-storage)
51-
- [Using Private Git Repository](#using-private-git-repository)
52-
- [Using Standard Input](#using-standard-input)
53-
- [Running kaniko](#running-kaniko)
54-
- [Running kaniko in a Kubernetes cluster](#running-kaniko-in-a-kubernetes-cluster)
55-
- [Kubernetes secret](#kubernetes-secret)
56-
- [Running kaniko in gVisor](#running-kaniko-in-gvisor)
57-
- [Running kaniko in Google Cloud Build](#running-kaniko-in-google-cloud-build)
58-
- [Running kaniko in Docker](#running-kaniko-in-docker)
59-
- [Caching](#caching)
60-
- [Caching Layers](#caching-layers)
61-
- [Caching Base Images](#caching-base-images)
62-
- [Pushing to Different Registries](#pushing-to-different-registries)
63-
- [Pushing to Docker Hub](#pushing-to-docker-hub)
64-
- [Pushing to Google GCR](#pushing-to-google-gcr)
65-
- [Pushing to GCR using Workload Identity](#pushing-to-gcr-using-workload-identity)
66-
- [Pushing to Amazon ECR](#pushing-to-amazon-ecr)
67-
- [Pushing to Azure Container Registry](#pushing-to-azure-container-registry)
68-
- [Pushing to JFrog Container Registry or to JFrog Artifactory](#pushing-to-jfrog-container-registry-or-to-jfrog-artifactory)
69-
- [Additional Flags](#additional-flags)
70-
- [Flag `--build-arg`](#flag---build-arg)
71-
- [Flag `--cache`](#flag---cache)
72-
- [Flag `--cache-dir`](#flag---cache-dir)
73-
- [Flag `--cache-repo`](#flag---cache-repo)
74-
- [Flag `--cache-copy-layers`](#flag---cache-copy-layers)
75-
- [Flag `--cache-run-layers`](#flag---cache-run-layers)
76-
- [Flag `--cache-ttl duration`](#flag---cache-ttl-duration)
77-
- [Flag `--cleanup`](#flag---cleanup)
78-
- [Flag `--compressed-caching`](#flag---compressed-caching)
79-
- [Flag `--context-sub-path`](#flag---context-sub-path)
80-
- [Flag `--custom-platform`](#flag---custom-platform)
81-
- [Flag `--digest-file`](#flag---digest-file)
82-
- [Flag `--dockerfile`](#flag---dockerfile)
83-
- [Flag `--force`](#flag---force)
84-
- [Flag `--git`](#flag---git)
85-
- [Flag `--image-name-with-digest-file`](#flag---image-name-with-digest-file)
86-
- [Flag `--image-name-tag-with-digest-file`](#flag---image-name-tag-with-digest-file)
87-
- [Flag `--insecure`](#flag---insecure)
88-
- [Flag `--insecure-pull`](#flag---insecure-pull)
89-
- [Flag `--insecure-registry`](#flag---insecure-registry)
90-
- [Flag `--label`](#flag---label)
91-
- [Flag `--log-format`](#flag---log-format)
92-
- [Flag `--log-timestamp`](#flag---log-timestamp)
93-
- [Flag `--no-push`](#flag---no-push)
94-
- [Flag `--oci-layout-path`](#flag---oci-layout-path)
95-
- [Flag `--push-retry`](#flag---push-retry)
96-
- [Flag `--registry-certificate`](#flag---registry-certificate)
97-
- [Flag `--registry-mirror`](#flag---registry-mirror)
98-
- [Flag `--reproducible`](#flag---reproducible)
99-
- [Flag `--single-snapshot`](#flag---single-snapshot)
100-
- [Flag `--skip-tls-verify`](#flag---skip-tls-verify)
101-
- [Flag `--skip-tls-verify-pull`](#flag---skip-tls-verify-pull)
102-
- [Flag `--skip-tls-verify-registry`](#flag---skip-tls-verify-registry)
103-
- [Flag `--skip-unused-stages`](#flag---skip-unused-stages)
104-
- [Flag `--snapshot-mode`](#flag---snapshot-mode)
105-
- [Flag `--tar-path`](#flag---tar-path)
106-
- [Flag `--target`](#flag---target)
107-
- [Flag `--use-new-run`](#flag---use-new-run)
108-
- [Flag `--verbosity`](#flag---verbosity)
109-
- [Flag `--ignore-var-run`](#flag---ignore-var-run)
110-
- [Flag `--ignore-path`](#flag---ignore-path)
111-
- [Flag `--image-fs-extract-retry`](#flag---image-fs-extract-retry)
112-
- [Debug Image](#debug-image)
113-
- [Security](#security)
114-
- [Verifying Signed Kaniko Images](#verifying-signed-kaniko-images)
115-
- [Kaniko Builds - Profiling](#kaniko-builds---profiling)
116-
- [Comparison with Other Tools](#comparison-with-other-tools)
117-
- [Community](#community-1)
118-
- [Limitations](#limitations)
119-
- [mtime and snapshotting](#mtime-and-snapshotting)
120-
- [References](#references)
38+
- [Community](#community)
39+
- [How does kaniko work?](#how-does-kaniko-work)
40+
- [Known Issues](#known-issues)
41+
- [Demo](#demo)
42+
- [Tutorial](#tutorial)
43+
- [Using kaniko](#using-kaniko)
44+
- [kaniko Build Contexts](#kaniko-build-contexts)
45+
- [Using Azure Blob Storage](#using-azure-blob-storage)
46+
- [Using Private Git Repository](#using-private-git-repository)
47+
- [Using Standard Input](#using-standard-input)
48+
- [Running kaniko](#running-kaniko)
49+
- [Running kaniko in a Kubernetes cluster](#running-kaniko-in-a-kubernetes-cluster)
50+
- [Kubernetes secret](#kubernetes-secret)
51+
- [Running kaniko in gVisor](#running-kaniko-in-gvisor)
52+
- [Running kaniko in Google Cloud Build](#running-kaniko-in-google-cloud-build)
53+
- [Running kaniko in Docker](#running-kaniko-in-docker)
54+
- [Caching](#caching)
55+
- [Caching Layers](#caching-layers)
56+
- [Caching Base Images](#caching-base-images)
57+
- [Pushing to Different Registries](#pushing-to-different-registries)
58+
- [Pushing to Docker Hub](#pushing-to-docker-hub)
59+
- [Pushing to Google GCR](#pushing-to-google-gcr)
60+
- [Pushing to GCR using Workload Identity](#pushing-to-gcr-using-workload-identity)
61+
- [Pushing to Amazon ECR](#pushing-to-amazon-ecr)
62+
- [Pushing to Azure Container Registry](#pushing-to-azure-container-registry)
63+
- [Pushing to JFrog Container Registry or to JFrog Artifactory](#pushing-to-jfrog-container-registry-or-to-jfrog-artifactory)
64+
- [Additional Flags](#additional-flags)
65+
- [Flag `--build-arg`](#flag---build-arg)
66+
- [Flag `--cache`](#flag---cache)
67+
- [Flag `--cache-dir`](#flag---cache-dir)
68+
- [Flag `--cache-repo`](#flag---cache-repo)
69+
- [Flag `--cache-copy-layers`](#flag---cache-copy-layers)
70+
- [Flag `--cache-run-layers`](#flag---cache-run-layers)
71+
- [Flag `--cache-ttl duration`](#flag---cache-ttl-duration)
72+
- [Flag `--cleanup`](#flag---cleanup)
73+
- [Flag `--compressed-caching`](#flag---compressed-caching)
74+
- [Flag `--context-sub-path`](#flag---context-sub-path)
75+
- [Flag `--custom-platform`](#flag---custom-platform)
76+
- [Flag `--digest-file`](#flag---digest-file)
77+
- [Flag `--dockerfile`](#flag---dockerfile)
78+
- [Flag `--force`](#flag---force)
79+
- [Flag `--git`](#flag---git)
80+
- [Flag `--image-name-with-digest-file`](#flag---image-name-with-digest-file)
81+
- [Flag `--image-name-tag-with-digest-file`](#flag---image-name-tag-with-digest-file)
82+
- [Flag `--insecure`](#flag---insecure)
83+
- [Flag `--insecure-pull`](#flag---insecure-pull)
84+
- [Flag `--insecure-registry`](#flag---insecure-registry)
85+
- [Flag `--label`](#flag---label)
86+
- [Flag `--log-format`](#flag---log-format)
87+
- [Flag `--log-timestamp`](#flag---log-timestamp)
88+
- [Flag `--no-push`](#flag---no-push)
89+
- [Flag `--oci-layout-path`](#flag---oci-layout-path)
90+
- [Flag `--push-retry`](#flag---push-retry)
91+
- [Flag `--registry-certificate`](#flag---registry-certificate)
92+
- [Flag `--registry-client-cert`](#flag---registry-client-cert)
93+
- [Flag `--registry-mirror`](#flag---registry-mirror)
94+
- [Flag `--reproducible`](#flag---reproducible)
95+
- [Flag `--single-snapshot`](#flag---single-snapshot)
96+
- [Flag `--skip-tls-verify`](#flag---skip-tls-verify)
97+
- [Flag `--skip-tls-verify-pull`](#flag---skip-tls-verify-pull)
98+
- [Flag `--skip-tls-verify-registry`](#flag---skip-tls-verify-registry)
99+
- [Flag `--skip-unused-stages`](#flag---skip-unused-stages)
100+
- [Flag `--snapshot-mode`](#flag---snapshot-mode)
101+
- [Flag `--tar-path`](#flag---tar-path)
102+
- [Flag `--target`](#flag---target)
103+
- [Flag `--use-new-run`](#flag---use-new-run)
104+
- [Flag `--verbosity`](#flag---verbosity)
105+
- [Flag `--ignore-var-run`](#flag---ignore-var-run)
106+
- [Flag `--ignore-path`](#flag---ignore-path)
107+
- [Flag `--image-fs-extract-retry`](#flag---image-fs-extract-retry)
108+
- [Debug Image](#debug-image)
109+
- [Security](#security)
110+
- [Verifying Signed Kaniko Images](#verifying-signed-kaniko-images)
111+
- [Kaniko Builds - Profiling](#kaniko-builds---profiling)
112+
- [Creating Multi-arch Container Manifests Using Kaniko and Manifest-tool](#creating-multi-arch-container-manifests-using-kaniko-and-manifest-tool)
113+
- [General Workflow](#general-workflow)
114+
- [Limitations and Pitfalls](#limitations-and-pitfalls)
115+
- [Example CI Pipeline (GitLab)](#example-ci-pipeline-gitlab)
116+
- [Building the Separate Container Images](#building-the-separate-container-images)
117+
- [Merging the Container Manifests](#merging-the-container-manifests)
118+
- [On the Note of Adding Versioned Tags](#on-the-note-of-adding-versioned-tags)
119+
- [Comparison with Other Tools](#comparison-with-other-tools)
120+
- [Community](#community-1)
121+
- [Limitations](#limitations)
122+
- [mtime and snapshotting](#mtime-and-snapshotting)
123+
- [References](#references)
121124

122125
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
123126

@@ -1135,6 +1138,163 @@ profiling,
11351138
2. If you are using the kaniko `debug` image, you can copy the file in the
11361139
`pre-stop` container lifecycle hook.
11371140

1141+
## Creating Multi-arch Container Manifests Using Kaniko and Manifest-tool
1142+
1143+
While Kaniko itself currently does not support creating multi-arch manifests
1144+
(contributions welcome), one can use tools such as [manifest-tool](https://github.com/estesp/manifest-tool)
1145+
to stitch multiple separate builds together into a single container manifest.
1146+
1147+
### General Workflow
1148+
1149+
The general workflow for creating multi-arch manifests is as follows:
1150+
1151+
1. Build separate container images using Kaniko on build hosts matching your
1152+
target architecture and tag them with the appropriate ARCH tag.
1153+
2. Push the separate images to your container registry.
1154+
3. Manifest-tool identifies the separate manifests in your container registry,
1155+
according to a given template.
1156+
4. Manifest-tool pushes a combined manifest referencing the separate manifests.
1157+
1158+
![Workflow Multi-arch](docs/images/multi-arch.drawio.svg)
1159+
1160+
### Limitations and Pitfalls
1161+
1162+
The following conditions must be met:
1163+
1164+
1. You need access to build-machines running the desired architectures
1165+
(running Kaniko in an emulator, e.g. QEMU should also be possible but goes
1166+
beyond the scope of this documentation). This is something to keep in mind
1167+
when using SaaS build tools such as github.com or gitlab.com, of which at the
1168+
time of writing neither supports any non-x86_64 SaaS runners ([GitHub](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources),[GitLab](https://docs.gitlab.com/ee/ci/runners/saas/linux_saas_runner.html#machine-types-available-for-private-projects-x86-64)),
1169+
so be prepared to bring your own machines ([GitHub](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners),[GitLab](https://docs.gitlab.com/runner/register/).
1170+
2. Kaniko needs to be able to run on the desired architectures. At the time of
1171+
writing, the official Kaniko container supports [linux/amd64, linux/arm64,
1172+
linux/s390x and linux/ppc64le (not on *-debug images)](https://github.com/GoogleContainerTools/kaniko/blob/main/.github/workflows/images.yaml).
1173+
3. The container registry of your choice must be OCIv1 or Docker v2.2
1174+
compatible.
1175+
1176+
### Example CI Pipeline (GitLab)
1177+
1178+
It is up to you to find an automation tool that suits your needs best.
1179+
We recommend using a modern CI/CD system such as GitHub workflows or GitLab CI.
1180+
As we (the authors) happen to use GitLab CI, the following examples are
1181+
tailored to this specific platform but the underlying principles should apply
1182+
anywhere else and the examples are kept simple enough, so that you should be
1183+
able to follow along, even without any previous experiences with this specific
1184+
platform. When in doubt, visit the [gitlab-ci.yml reference page](https://docs.gitlab.com/ee/ci/yaml/index.html)
1185+
for a comprehensive overview of the GitLab CI keywords.
1186+
1187+
#### Building the Separate Container Images
1188+
1189+
gitlab-ci.yml:
1190+
1191+
```yaml
1192+
# define a job for building the containers
1193+
build-container:
1194+
stage: container-build
1195+
# run parallel builds for the desired architectures
1196+
parallel:
1197+
matrix:
1198+
- ARCH: amd64
1199+
- ARCH: arm64
1200+
tags:
1201+
# run each build on a suitable, preconfigured runner (must match the target architecture)
1202+
- runner-${ARCH}
1203+
image:
1204+
name: gcr.io/kaniko-project/executor:debug
1205+
entrypoint: [""]
1206+
script:
1207+
# build the container image for the current arch using kaniko
1208+
- >-
1209+
/kaniko/executor
1210+
--context "${CI_PROJECT_DIR}"
1211+
--dockerfile "${CI_PROJECT_DIR}/Dockerfile"
1212+
# push the image to the GitLab container registry, add the current arch as tag.
1213+
--destination "${CI_REGISTRY_IMAGE}:${ARCH}"
1214+
```
1215+
1216+
#### Merging the Container Manifests
1217+
1218+
gitlab-ci.yml:
1219+
1220+
```yaml
1221+
# define a job for creating and pushing a merged manifest
1222+
merge-manifests:
1223+
stage: container-build
1224+
# all containers must be build before merging them
1225+
# alternatively the job may be configured to run in a later stage
1226+
needs:
1227+
- container-build
1228+
artifacts: false
1229+
tags:
1230+
# may run on any architecture supported by manifest-tool image
1231+
- runner-xyz
1232+
image:
1233+
name: mplatform/manifest-tool:alpine
1234+
script:
1235+
- >-
1236+
manifest-tool
1237+
# authorize against your container registry
1238+
--username=${CI_REGISTRY_USER}
1239+
--password=${CI_REGISTRY_PASSWORD}
1240+
push from-args
1241+
# define the architectures you want to merge
1242+
--platforms linux/amd64,linux/arm64
1243+
# "ARCH" will be automatically replaced by manifest-tool
1244+
# with the appropriate arch from the platform definitions
1245+
--template ${CI_REGISTRY_IMAGE}:ARCH
1246+
# The name of the final, combined image which will be pushed to your registry
1247+
--target ${CI_REGISTRY_IMAGE}
1248+
```
1249+
1250+
#### On the Note of Adding Versioned Tags
1251+
1252+
For simplicity's sake we deliberately refrained from using versioned
1253+
tagged images (all builds will be tagged as "latest") in the
1254+
previous examples, as we feel like this adds to much platform and workflow
1255+
specific code.
1256+
1257+
Nethertheless, for anyone interested in how we handle (dynamic) versioning in
1258+
GitLab, here is a short rundown:
1259+
1260+
- If you are only interested in building tagged releases, you can simply
1261+
use the [GitLab predefined](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html) `CI_COMMIT_TAG` variable when running a tag pipeline.
1262+
- When you (like us) want to additionally build container images outside of
1263+
releases, things get a bit messier. In our case, we added a additional job
1264+
which runs before the build and merge jobs (don't forget to extend the `needs`
1265+
section of the build and merge jobs accordingly), which will set the tag to
1266+
`latest` when running on the default branch, to the commit hash when run on
1267+
other branches and to the release tag when run on a tag pipeline.
1268+
1269+
gitlab-ci.yml:
1270+
1271+
```yaml
1272+
container-get-tag:
1273+
stage: pre-container-build-stage
1274+
tags:
1275+
- runner-xyz
1276+
image: busybox
1277+
script:
1278+
# All other branches are tagged with the currently built commit SHA hash
1279+
- |
1280+
# If pipeline runs on the default branch: Set tag to "latest"
1281+
if test "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH"; then
1282+
tag="latest"
1283+
# If pipeline is a tag pipeline, set tag to the git commit tag
1284+
elif test -n "$CI_COMMIT_TAG"; then
1285+
tag="$CI_COMMIT_TAG"
1286+
# Else set the tag to the git commit sha
1287+
else
1288+
tag="$CI_COMMIT_SHA"
1289+
fi
1290+
- echo "tag=$tag" > build.env
1291+
# parse tag to the build and merge jobs.
1292+
# See: https://docs.gitlab.com/ee/ci/variables/#pass-an-environment-variable-to-another-job
1293+
artifacts:
1294+
reports:
1295+
dotenv: build.env
1296+
```
1297+
11381298
## Comparison with Other Tools
11391299

11401300
Similar tools include:
@@ -1152,7 +1312,8 @@ All of these tools build container images with different approaches.
11521312
BuildKit (and `img`) can perform as a non-root user from within a container but
11531313
requires seccomp and AppArmor to be disabled to create nested containers.
11541314
`kaniko` does not actually create nested containers, so it does not require
1155-
seccomp and AppArmor to be disabled.
1315+
seccomp and AppArmor to be disabled. BuildKit supports "cross-building"
1316+
multi-arch containers by leveraging QEMU.
11561317

11571318
`orca-build` depends on `runc` to build images from Dockerfiles, which can not
11581319
run inside a container (for similar reasons to `img` above). `kaniko` doesn't

0 commit comments

Comments
 (0)