Implementation of the multi-platform support for builders and buildpack packages RFC 0128

[jjbustamante]

Summary

The purpose of this PR is to implement the RFC 0128.

It adds the capability to the pack buildpack package and pack builder create to generate multi-platform OCI images and create an image index to combine them.

A draft version of this PR was demo during KubeCon EU 24. See the recording here

Output

Before

• We were not able to create multi-platform buildpacks and builders

After

• Follow the RFC documentation to specify Targets entries in the buildpack.toml or package.toml files.

Check the document mentioned above, it has a detailed output

Documentation

• Should this change be documented?
  • Yes, see
  • No

Related

Resolves #1459
Resolves #2079

[codecov]

Codecov Report

Attention: Patch coverage is 21.31336% with 683 lines in your changes are missing coverage. Please review.

Project coverage is 75.98%. Comparing base (1e41857) to head (c7b1521).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2086      +/-   ##
==========================================
- Coverage   79.69%   75.98%   -3.71%     
==========================================
  Files         176      192      +16     
  Lines       13254    14038     +784     
==========================================
+ Hits        10562    10665     +103     
- Misses       2022     2676     +654     
- Partials      670      697      +27     

Flag	Coverage Δ
os_linux	74.93% <21.32%> (-3.69%)	⬇️
os_macos	72.72% <20.28%> (-3.58%)	⬇️
os_windows	75.40% <21.32%> (-3.68%)	⬇️
unit	75.98% <21.32%> (-3.71%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

[anthonydahanne]

Hello 👋
We have been testing the latest version

./pack --version
0.0.0+git-3cb6546.build-5619

from
https://github.com/buildpacks/pack/actions/runs/8118576298

We had success running, from the ~/buildpack directory, using:

pack buildpack package anthonydahanne/syft:dual6 --publish
Successfully published package anthonydahanne/syft:dual6 and saved to registry

with a layout similar to this:

tree ~/buildpack
/home/ubuntu/buildpack
├── LICENSE
├── NOTICE
├── README.md
├── buildpack.toml
└── linux
    ├── amd64
    │   └── bin
    │       ├── build -> main
    │       ├── detect -> main
    │       └── main
    └── arm64
        └── bin
            ├── build -> main
            ├── detect -> main
            └── main

5 directories, 10 files

and we successfully obtained a multi arch image:

docker manifest inspect anthonydahanne/syft:dual6
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 428,
         "digest": "sha256:91f6b87823cfb8b0df8d415da9b40774790f655bc7c576e4b2d4abde8cf0816a",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 428,
         "digest": "sha256:01da98c691e2912e8f32c8b28c93ec26968d6c5857fb4e19313d718d3156f121",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

Yay 🎉

But then, we tried running it from elsewhere, using such a package.toml:

cat ~/package.toml
[buildpack]
uri = "/home/ubuntu/buildpack"

./pack buildpack package anthonydahanne/syft:dual5 --config "${HOME}"/package.toml --publish
Warning: A new '--target' flag is available to set the platform for the buildpack package, using 'linux' as default
Successfully published package anthonydahanne/syft:dual5 and saved to registry

and this did not produce an expected dual arch image:

docker manifest inspect anthonydahanne/syft:dual5
{
	"schemaVersion": 2,
	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
	"config": {
		"mediaType": "application/vnd.docker.container.image.v1+json",
		"size": 1172,
		"digest": "sha256:0d68e7e4493ba8637d296368acd932332611dd9633bf977c77f8f5339982e0aa"
	},
	"layers": [
		{
			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
			"size": 5662387,
			"digest": "sha256:0f3a885b04767878fefc27a532e71911888fbba96708b632c0a5f95ca9a2c1a8"
		}
	]
}

Is it either a bug or do we do something wrong?
Thank you 🙏

[jjbustamante]

Hi @anthonydahanne

Is it either a bug or do we do something wrong?

Yeah, it is a bug, I noticed that when preparing my script for the KubeCon demo, I haven't took a look yet, sorry! keep testing it running from the buildpack/builder folder

[sophiewigmore]

We're just going to document all of our issues as we encounter them. Using the same pack version as @anthonydahanne's comment above, we are running into a duplicate manifest issue.

1. We successfully run pack buildpack package <image>:<tag> --publish from the /buildpack directory per your instruction
2. We re-run pack buildpack package <image>:<tag> --publish, as we will likely do in our CI pipelines everntually to update the latest tag for an image.
3. The image manifest contains duplicate manifest entries, instead of being updated:

docker manifest inspect <image>:<tag>
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 428,
         "digest": "sha256:63fa2e8bbdaead0b5e9f38f968eeb0c1beebdb7c94b62729ee7132da078ec17c",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 428,
         "digest": "sha256:ac56bf570204550814118621a098a5182ba732919454869711a9b4c51abf3e28",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 428,
         "digest": "sha256:63fa2e8bbdaead0b5e9f38f968eeb0c1beebdb7c94b62729ee7132da078ec17c",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 428,
         "digest": "sha256:ac56bf570204550814118621a098a5182ba732919454869711a9b4c51abf3e28",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

We'd expect the manifest list to only contain the latest entries

[jjbustamante]

Thank you @sophiewigmore ! Yes I noticed the duplication too! that's part of the manifest code Husni was working on.

[anthonydahanne]

hello @jjbustamante ! any updates since Kubecon?
on the Paketo sides of things, we use the exp. pack CLI from a good number of our buildpacks workflows to support dual arch!
Please let us know! Thanks!

[jjbustamante]

hello @jjbustamante ! any updates since Kubecon? on the Paketo sides of things, we use the exp. pack CLI from a good number of our buildpacks workflows to support dual arch! Please let us know! Thanks!

Hi @anthonydahanne, some updates:

• I am working on trying to merge the branch from Husni and Sai with the implementation of the manifest commands RFC, We need these changes first because that's the code that is handling the Image Index creation logic.
• I was off last Thursday/Friday but the multi-plaftorm RFC is on voting state, I think we should be merging that one this week.
• Once the manifest command logic is merged, I still need to fix the current bugs and polish a little bit this logic to un-draft it, I hope at the end of this week or next one I can have more updates on this!

[natalieparellano]

Still going through it, but leaving a few comments in the meantime :)

[natalieparellano]

@jjbustamante alrighty I've finally looked through all the files. I left a few comments. It mostly looks sensible, though we'll definitely want more tests and probably an acceptance test or two. LMK if I can be of any help at all.

[natalieparellano]

Looks good to me. Thanks for all of your hard work @jjbustamante ❤️