Regression: processes is not parseable in metadata.toml

[ipmb]

Summary

I'm using pack to build images based on Heroku's buildpacks which just updated to lifecycle 0.15.0. processes in metadata.toml lost its formatting in the upgrade. Originally filed at heroku/cnb-builder-images#295

---

Reproduction

Steps

$ mkdir testcase && cd $_
$ touch requirements.txt
$ echo 'web: foo' > Procfile
$ pack build --builder heroku/builder-classic:22 testapp

Current behavior

processes is an array of strings

$ docker run --rm -it --platform linux/amd64 --entrypoint launcher testapp cat /layers/config/metadata.toml
processes = ["type = \"web\"\ncommand = \"foo\"\nargs = []\ndirect = false\nbuildpack-id = \"heroku/procfile\"\n"]
buildpack-default-process-type = "web"

[[buildpacks]]
  id = "heroku/python"
  version = "0.0.0"
  api = "0.4"

[[buildpacks]]
  id = "heroku/procfile"
  version = "2.0.0"
  api = "0.8"
  homepage = "https://github.com/heroku/procfile-cnb"

Expected

processes is an array of tables

$ docker run --rm -it --platform linux/amd64 --entrypoint launcher testapp-previous cat /layers/config/metadata.toml
buildpack-default-process-type = "web"

[[buildpacks]]
  api = "0.4"
  id = "heroku/python"
  version = "0.0.0"

[[buildpacks]]
  api = "0.8"
  homepage = "https://github.com/heroku/procfile-cnb"
  id = "heroku/procfile"
  version = "2.0.0"

[[processes]]
  type = "web"
  command = "foo"
  direct = false
  buildpack-id = "heroku/procfile"

---

Context

lifecycle version

0.15.0 (not present in 0.14.1

platform version(s)

Pack:
  Version:  0.27.0+git-f4f5be1.build-3382
  OS/Arch:  linux/amd64

Default Lifecycle Version:  0.14.1

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 33
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 0197261a30bf81f1ee8e6a4dd2dea0ef95d67ccb
 runc version: v1.1.3-0-g6724737
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-126-generic
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.23GiB
 Name: ll-server
 ID: S4IG:47J6:2P4Z:YWAT:TFGO:DSH7:UGNH:IB6N:UMEO:NIIW:ST3Q:LUC3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

[edmorley]

I also encountered a secondary issue that seems to have the same root cause.

When the builder is untrusted, and --lifecycle-image used, and that image is an older lifecycle version -- then the export phase fails since it's unable to parse the /layers/config/metadata.toml written by the newer lifecycle that shipped with the builder itself (that ran for the build phase).

eg:

$ pack config trusted-builders remove heroku/builder-classic:22
$ pack build --builder heroku/builder-classic:22 --lifecycle-image buildpacksio/lifecycle:0.14.3 --clear-cache testapp
...
===> EXPORTING
[exporter] ERROR: failed to export: read build metadata: toml: type mismatch for launch.Process: expected table but found string
ERROR: failed to build: executing lifecycle. This may be the result of using an untrusted builder: failed with status code: 62

[edmorley]

The lifecycle version in heroku/builder-classic:22 has been downgraded for now, and as such to reproduce one now needs to specify the old builder image explicitly:

pack build --builder heroku/builder-classic@sha256:435f3aea7709bc984bcb6a1538412c001b579453aee26144304dee14fad1b3eb testapp

[ipmb]

@edmorley when I hit this issue, I wanted to downgrade the builder, but only saw a single tag on Docker Hub. Is there a way I can see the list of previous images so I can grab it by the sha256 hash in the future?

[edmorley]

I retrieved the SHA256 from Docker Hub's "Advanced Image Management" page, however that's presumably only shown to those with write access to the docker repository. It may still be possible to use Docker Hub's API to fetch this (eg https://docs.docker.com/docker-hub/api/latest/#tag/images). Otherwise I would look in heroku/builder CI logs to find the upload of the previous image.