pack build command to export to OCI layout format on disk

[jjbustamante]

Summary

This PR expose the functionality to end users to being able to export the application image to disk in OCI layout format. The final application image can be configured to contain all the blobs from the run-image (default behavior) or using the -sparse flag the blobs from the run-image will be omitted.

Output

full image saved on disk

Let's suppose the application was built using the following command

> pack build oci:my-java-app --builder cnbs/jbustamantevmware-builder:bionic --verbose --path /Users/jbustamante/workspace/buildpack.io/samples/apps/java-maven --trust-builder

lifecycle output ...

Saving /layout-repo/index.docker.io/library/my-java-app/latest...
*** Images (/layout-repo/index.docker.io/library/my-java-app/latest@sha256:b20d9a4baac8198e2f2621a4142376a934ec9308b7d2b82dd21b4edf92629176):
      /layout-repo/index.docker.io/library/my-java-app/latest

*** Manifest Size: 1660
Reading buildpack directory: /layers/samples_java-maven
Reading buildpack directory item: jdk
Reading buildpack directory item: jdk.toml
Reading buildpack directory item: launch.toml
Reading buildpack directory item: maven_m2
Reading buildpack directory item: maven_m2.toml
Reusing tarball for layer "samples/java-maven:jdk" with SHA: sha256:a940b3489712c1f762caef8b37675f15364a65f10fb7d3f2299109f7a637afef
Adding cache layer 'samples/java-maven:jdk'
Layer 'samples/java-maven:jdk' SHA: sha256:a940b3489712c1f762caef8b37675f15364a65f10fb7d3f2299109f7a637afef
Adding cache layer 'samples/java-maven:maven_m2'
Layer 'samples/java-maven:maven_m2' SHA: sha256:8bc9932ebe39f940fc9f7ca4a608bd0598b26fc7cd2adee5728638d81d43e6c0
Successfully built image my-java-app

We can see the application was saved on disk in OCI layout format at folder my-java-app

> tree my-java-app
my-java-app
├── blobs
│   └── sha256
│       ├── 1251fdfda1211eae488b85b9d6878da34885eaae539a5db22021a2a6e0d8ddc6
│       ├── 2fa192256ce255c6ea6c1296eadfe2feba8094f40e6aa85e699645caca2e85d8
│       ├── 5a44e4f7b58d74fe6f92dd7028075c91191128d1e2e7f39846fe061a9a98836e
│       ├── 630eb8c42eff26fe99796f8ac95d4cbf68c9840911b904c8f70cfd08bc88aee7
│       ├── 6898e29a423a34b0482cf77292814e4009aaae456e461301e4997af1e27084be
│       ├── 6c5215613ce3919ce4eebeb6497aa402cd171a1f41f6f812363c7180e2238972
│       ├── 72d9f18d70f395ff9bfae4d193077ccea3ca583e3da3dd66f5c84520c0100727
│       ├── 9aff7610b7790d95c66d6216d976a90452f92058a45343958a4b19d9d277d493
│       ├── a99cbc83e1aaf58380e98988983afbf463b055852462b24c8b609100b74bde75
│       ├── b20d9a4baac8198e2f2621a4142376a934ec9308b7d2b82dd21b4edf92629176
│       └── f9d6350d0c44c0e7165a522155f53181ce8c163a6b8ead1f6baea22d1a8d8a78
├── index.json
└── oci-layout

2 directories, 13 files

image saved on disk avoiding run-image blobs

On the other hand, if we used the --sparse flag to export the image

> pack build oci:my-java-app --builder cnbs/jbustamantevmware-builder:bionic --verbose --path /Users/jbustamante/workspace/buildpack.io/samples/apps/java-maven --trust-builder --sparse

The output do not contain the run-image blobs

> tree my-java-app
my-java-app
├── blobs
│   └── sha256
│       ├── 2fa192256ce255c6ea6c1296eadfe2feba8094f40e6aa85e699645caca2e85d8
│       ├── 5a44e4f7b58d74fe6f92dd7028075c91191128d1e2e7f39846fe061a9a98836e
│       ├── 6622426666a7b61c086c84203082d5f64495be1f8b085137e53b0554cfcdb50a
│       ├── 6c5215613ce3919ce4eebeb6497aa402cd171a1f41f6f812363c7180e2238972
│       ├── a99cbc83e1aaf58380e98988983afbf463b055852462b24c8b609100b74bde75
│       ├── d51dbef8a63d4ca36ef0a358e2e7e77cfffc2687644f5cf9d1844d3b8b11c412
│       ├── f53c5083455f65f6e262b356e15a889e0f351f6c725aa3031103c27e3b15190a
│       └── f9d6350d0c44c0e7165a522155f53181ce8c163a6b8ead1f6baea22d1a8d8a78
├── index.json
└── oci-layout

2 directories, 10 files

Before

The feature doesn't exist in previous version of pack

After

With this new PR, pack will have the following new behavior:

• OCI layout feature is experimental, in order to be used, it must be enabled by pack users running pack config experimental true
• Once the experimental mode is enabled, the user can enable the feature using oci:<image-path> when pack build is invoked
• If the user wants to prevent run-image blobs to be written on disk, they can passthrough the flag --sparse
• For now, pack will created a folder at /$HOME/.pack/layout-repo to save the run-image in OCI layout, this path will be mounted during lifecycle build execution.

Documentation

• Should this change be documented?
  • Yes, see #529
  • No

Related

Resolves #1548

[codecov]

Codecov Report

Merging #1596 (005aed5) into main (a803a52) will decrease coverage by 0.47%.
The diff coverage is 26.93%.

❗ Current head 005aed5 differs from pull request most recent head 256b0dc. Consider uploading reports for the commit 256b0dc to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1596      +/-   ##
==========================================
- Coverage   79.72%   79.25%   -0.47%     
==========================================
  Files         165      147      -18     
  Lines       10885     9076    -1809     
==========================================
- Hits         8677     7192    -1485     
+ Misses       1683     1463     -220     
+ Partials      525      421     -104     

Flag	Coverage Δ
os_linux	79.09% <26.93%> (-0.41%)	⬇️
os_macos	76.83% <26.93%> (-0.79%)	⬇️
os_windows	79.15% <26.93%> (-0.46%)	⬇️
unit	79.25% <26.93%> (-0.47%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

[jkutner]

This feels a little odd to me. Are we sure we want the default layout dir in the config dir?

Also, it looks like this isn't actually configurable right? It's either the default or nothing.

[jjbustamante]

Yeah, for now, because the feature is experimental, I didn't want to do a lot of work giving the user a configuration option. basically, the idea is:

• We need a place to save the run-image on disk, maybe the user doesn't care, that's why I am saving them in the pack home directory
• I am planning to document this behavior in the docs so users can be aware of it.

The other alternative is, I could work on the user experience for configuring and customizing this kind of thing, but, I was planing to do it after shipping at least something the users can play with

[jjbustamante]

Actually, I can see pack already save some data in the same folder, for example.

➜  > tree ~/.pack -L 1
/Users/jbustamante/.pack
├── completion.zsh
├── config.toml
├── download-cache
├── layout-repo
├── registry-a932275bd19c2d9e1b88fa06698fd2f5427a363d25bf87fa500691c373089381
├── registry2908354347
└── registry57772754

6 directories, 2 files

Those registry-* folders are already saving data there

[jkutner]

Is it just metadata that goes in this dir? Or does it actually store large files?

[jjbustamante]

it will save the run-image in OCI layout format, it could have all the blobs depending on whether the user set the sparse flag or not. But it could have large files, for sure