Skip to content

Commit 34468fc

Browse files
AkihiroSudaAkihiroSuda
authored
Merge pull request #6 from AkihiroSuda/dev
Add an example of `golang:1.21-alpine3.18 `
2 parents 623c36d + 53428c8 commit 34468fc

File tree

4 files changed

+464
-156
lines changed

4 files changed

+464
-156
lines changed

‎README.md

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,3 +110,129 @@ See the `docker://` example above, and read `docker` as `podman`.
110110

111111
### Accessing private images
112112
To access private images, create a credential file as `~/.docker/config.json` using `docker login`.
113+
114+
115+
- - -
116+
# Examples
117+
## Non-reproducible Docker Hub images
118+
119+
### `golang:1.21-alpine3.18`
120+
The sources of the official Docker Hub images are available at <https://github.com/docker-library>.
121+
122+
For example, the source of [`golang:1.21-alpine3.18`](https://hub.docker.com/layers/library/golang/1.21-alpine3.18/images/sha256-dd8888bb7f1b0b05e1e625aa29483f50f38a9b64073a4db00b04076cec52b71c?context=explore)
123+
can be found at <https://github.com/docker-library/golang/blob/d1ff31b86b23fe721dc65806cd2bd79a4c71b039/1.21/alpine3.18/Dockerfile>.
124+
125+
The source can be built as follows:
126+
127+
```console
128+
$ DOCKER_BUILDKIT=0 docker build -t my-golang-1.21-alpine3.18 'https://github.com/docker-library/golang.git#d1ff31b86b23fe721dc65806cd2bd79a4c71b039:1.21/alpine3.18'
129+
...
130+
Successfully tagged my-golang-1.21-alpine3.18:latest
131+
```
132+
133+
> **Note**
134+
>
135+
> `DOCKER_BUILDKIT=0` is specified here because the official `golang:1.21-alpine3.18` image is currently built with the legacy builder.
136+
> A future revision of the official image may be built with BuildKit, and in such a case, `DOCKER_BUILDKIT=1` will rather need to be specified here.
137+
138+
The resulting image binary (`my-golang-1.21-alpine3.18`) can be compared with the official image binary (`golang:1.21-alpine3.18`) as follows:
139+
140+
```console
141+
$ diffoci diff docker://golang:1.21-alpine3.18 docker://my-golang-1.21-alpine3.18 --semantic --report-dir=~/diff
142+
INFO[0000] Loading image "docker.io/library/golang:1.21-alpine3.18" from "docker"
143+
docker.io/library/golang:1.21 alpine3.18 saved
144+
Importing elapsed: 2.6 s total: 0.0 B (0.0 B/s)
145+
INFO[0004] Loading image "docker.io/library/my-golang-1.21-alpine3.18:latest" from "docker"
146+
docker.io/library/my golang 1.21 alpine3 saved
147+
Importing elapsed: 2.6 s total: 0.0 B (0.0 B/s)
148+
TYPE NAME INPUT-0 INPUT-1
149+
Layer ctx:/layers-1/layer length mismatch (457 vs 454)
150+
File lib/apk/db/scripts.tar eef110e559acb7aa00ea23ee7b8bddb52c4526cd394749261aa244ef9c6024a4 342eaa013375398497bfc21dff7dd017a647032ec5c486011142c576b7ccc989
151+
Layer ctx:/layers-1/layer name "usr/local/share/ca-certificates/.wh..wh..opq" only appears in input 0
152+
Layer ctx:/layers-1/layer name "usr/share/ca-certificates/.wh..wh..opq" only appears in input 0
153+
Layer ctx:/layers-1/layer name "etc/ca-certificates/.wh..wh..opq" only appears in input 0
154+
Layer ctx:/layers-2/layer length mismatch (13927 vs 13926)
155+
Layer ctx:/layers-2/layer name "usr/local/go/.wh..wh..opq" only appears in input 0
156+
File lib/apk/db/scripts.tar 073bb5094fc5bba800f06661dc7f1325c5cb4250b13209fb9e3eaf4e60e4bfc4 1369581b62bd60304c59556ea85f585bd498040c8fa223243622bb7990833063
157+
Layer ctx:/layers-3/layer length mismatch (4 vs 3)
158+
Layer ctx:/layers-3/layer name "go/.wh..wh..opq" only appears in input 0
159+
```
160+
161+
> **Note**
162+
> The `--semantic` flag is specified to ignore differences of timestamps, image names, and other "boring" attributes.
163+
> Without this flag, the `diffoci` command may print an enourmous amount of output.
164+
165+
In the `my-golang-1.21-alpine3.18` image, special files called ["Opaque whiteouts"](https://github.com/opencontainers/image-spec/blob/v1.0.2/layer.md#whiteouts) (`.wh..wh..opq`)
166+
are missing due to filesystem difference between Docker Hub's build machine and the local machine.
167+
168+
Also, the `lib/apk/db/scripts.tar` file in the layer 1 is not reproducible due to the timestamps of the tar entries inside it.
169+
The differences can be inspected by running the [`diffoscope`](https://diffoscope.org/) command for `~/diff/input-{0,1}/layers-1/lib/apk/db/scripts.tar`:
170+
```console
171+
$ sudo apt-get install -y diffoscope
172+
173+
$ diffoscope ~/diff/input-0/layers-1/lib/apk/db/scripts.tar ~/diff/input-1/layers-1/lib/apk/db/scripts.tar
174+
--- /home/suda/diff/input-0/layers-1/lib/apk/db/scripts.tar
175+
+++ /home/suda/diff/input-1/layers-1/lib/apk/db/scripts.tar
176+
├── file list
177+
│ @@ -1,9 +1,9 @@
178+
│ --rwxr-xr-x 0 root (0) root (0) 56 2023-08-09 03:36:47.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-install
179+
│ --rwxr-xr-x 0 root (0) root (0) 983 2023-08-09 03:36:47.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-install
180+
│ --rwxr-xr-x 0 root (0) root (0) 755 2023-08-09 03:36:47.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-upgrade
181+
│ --rwxr-xr-x 0 root (0) root (0) 983 2023-08-09 03:36:47.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-upgrade
182+
│ --rwxr-xr-x 0 root (0) root (0) 139 2023-08-09 03:36:47.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-install
183+
│ --rwxr-xr-x 0 root (0) root (0) 1239 2023-08-09 03:36:47.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-upgrade
184+
│ --rwxr-xr-x 0 root (0) root (0) 546 2023-08-09 03:36:47.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.trigger
185+
│ --rwxr-xr-x 0 root (0) root (0) 137 2023-08-09 03:36:47.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.post-deinstall
186+
│ --rwxr-xr-x 0 root (0) root (0) 63 2023-08-09 03:36:47.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.trigger
187+
│ +-rwxr-xr-x 0 root (0) root (0) 56 2023-08-24 07:50:41.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-install
188+
│ +-rwxr-xr-x 0 root (0) root (0) 983 2023-08-24 07:50:41.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-install
189+
│ +-rwxr-xr-x 0 root (0) root (0) 755 2023-08-24 07:50:41.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-upgrade
190+
│ +-rwxr-xr-x 0 root (0) root (0) 983 2023-08-24 07:50:41.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-upgrade
191+
│ +-rwxr-xr-x 0 root (0) root (0) 139 2023-08-24 07:50:41.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-install
192+
│ +-rwxr-xr-x 0 root (0) root (0) 1239 2023-08-24 07:50:41.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-upgrade
193+
│ +-rwxr-xr-x 0 root (0) root (0) 546 2023-08-24 07:50:41.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.trigger
194+
│ +-rwxr-xr-x 0 root (0) root (0) 137 2023-08-24 07:50:41.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.post-deinstall
195+
│ +-rwxr-xr-x 0 root (0) root (0) 63 2023-08-24 07:50:41.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.trigger
196+
```
197+
198+
These differences are boring, but not filtered out by the `--semantic` flag of the `diffoci` command, because `diffoci` is not aware of the formats of the files inside the image layers.
199+
200+
The `lib/apk/db/scripts.tar` file in the layer 2 has the same issue:
201+
<details>
202+
<p>
203+
204+
```console
205+
$ diffoscope ~/diff/input-0/layers-2/lib/apk/db/scripts.tar ~/diff/input-1/layers-2/lib/apk/db/scripts.tar
206+
--- /home/suda/diff/input-0/layers-2/lib/apk/db/scripts.tar
207+
+++ /home/suda/diff/input-1/layers-2/lib/apk/db/scripts.tar
208+
├── file list
209+
│ @@ -1,9 +1,9 @@
210+
│ --rwxr-xr-x 0 root (0) root (0) 56 2023-08-09 04:41:27.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-install
211+
│ --rwxr-xr-x 0 root (0) root (0) 983 2023-08-09 04:41:27.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-install
212+
│ --rwxr-xr-x 0 root (0) root (0) 755 2023-08-09 04:41:27.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-upgrade
213+
│ --rwxr-xr-x 0 root (0) root (0) 983 2023-08-09 04:41:27.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-upgrade
214+
│ --rwxr-xr-x 0 root (0) root (0) 139 2023-08-09 04:41:27.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-install
215+
│ --rwxr-xr-x 0 root (0) root (0) 1239 2023-08-09 04:41:27.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-upgrade
216+
│ --rwxr-xr-x 0 root (0) root (0) 546 2023-08-09 04:41:27.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.trigger
217+
│ --rwxr-xr-x 0 root (0) root (0) 137 2023-08-09 04:41:27.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.post-deinstall
218+
│ --rwxr-xr-x 0 root (0) root (0) 63 2023-08-09 04:41:27.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.trigger
219+
│ +-rwxr-xr-x 0 root (0) root (0) 56 2023-08-24 07:50:52.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-install
220+
│ +-rwxr-xr-x 0 root (0) root (0) 983 2023-08-24 07:50:52.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-install
221+
│ +-rwxr-xr-x 0 root (0) root (0) 755 2023-08-24 07:50:52.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-upgrade
222+
│ +-rwxr-xr-x 0 root (0) root (0) 983 2023-08-24 07:50:52.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-upgrade
223+
│ +-rwxr-xr-x 0 root (0) root (0) 139 2023-08-24 07:50:52.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-install
224+
│ +-rwxr-xr-x 0 root (0) root (0) 1239 2023-08-24 07:50:52.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-upgrade
225+
│ +-rwxr-xr-x 0 root (0) root (0) 546 2023-08-24 07:50:52.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.trigger
226+
│ +-rwxr-xr-x 0 root (0) root (0) 137 2023-08-24 07:50:52.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.post-deinstall
227+
│ +-rwxr-xr-x 0 root (0) root (0) 63 2023-08-24 07:50:52.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.trigger
228+
```
229+
230+
</p>
231+
</details>
232+
233+
Depending on the time to build the image, more differences may happen, especially when the Alpine packages on the internet are bumped up.
234+
235+
#### Conclusion
236+
This example indicates that although the official `golang:1.21-alpine3.18` image binary is not fully reproducible, its non-reproducibility is practically negligible, and
237+
this image binary can be assured to be certainly built from with the [published source](https://github.com/docker-library/golang/blob/d1ff31b86b23fe721dc65806cd2bd79a4c71b039/1.21/alpine3.18/Dockerfile).
238+
**If the published source is trustable**, this image binary can be trusted too.

‎cmd/diffoci/commands/diff/diff.go

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ func NewCommand() *cobra.Command {
3939
"ignore-timestamps",
4040
"ignore-history",
4141
"ignore-file-order",
42+
"ignore-file-mode-redundant-bits",
4243
"ignore-image-name",
4344
}
4445
for _, f := range flagNames {
@@ -60,6 +61,7 @@ func NewCommand() *cobra.Command {
6061
flags.Bool("ignore-timestamps", false, "Ignore timestamps")
6162
flags.Bool("ignore-history", false, "Ignore history")
6263
flags.Bool("ignore-file-order", false, "Ignore file order in tar layers")
64+
flags.Bool("ignore-file-mode-redundant-bits", false, "Ignore redundant bits of file mode")
6365
flags.Bool("ignore-image-name", false, "Ignore image name annotation")
6466
flags.Bool("semantic", false, "[Recommended] Alias for --ignore-*=true")
6567

@@ -96,6 +98,10 @@ func action(cmd *cobra.Command, args []string) error {
9698
if err != nil {
9799
return err
98100
}
101+
options.IgnoreFileModeRedundantBits, err = flags.GetBool("ignore-file-mode-redundant-bits")
102+
if err != nil {
103+
return err
104+
}
99105
options.IgnoreImageName, err = flags.GetBool("ignore-image-name")
100106
if err != nil {
101107
return err

‎cmd/diffoci/imagegetter/imagegetter.go

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,18 @@ func (g *ImageGetter) getPodman(ctx context.Context, rawRef string, plats []ocis
137137
return g.loadDocker(ctx, podman, name, plats)
138138
}
139139

140+
type readerWithEOF struct {
141+
io.Reader
142+
}
143+
144+
func (r *readerWithEOF) Read(p []byte) (int, error) {
145+
n, err := r.Reader.Read(p)
146+
if errors.Is(err, os.ErrClosed) {
147+
err = io.EOF
148+
}
149+
return n, err
150+
}
151+
140152
// loadDocker runs `docker save` and loads the result
141153
func (g *ImageGetter) loadDocker(ctx context.Context, docker, name string, plats []ocispec.Platform) (*images.Image, error) {
142154
log.G(ctx).Infof("Loading image %q from %q", name, docker)
@@ -151,7 +163,7 @@ func (g *ImageGetter) loadDocker(ctx context.Context, docker, name string, plats
151163
if err = dockerCmd.Start(); err != nil {
152164
return nil, fmt.Errorf("failed to run %v: %w", dockerCmd.Args, err)
153165
}
154-
if err = Load(ctx, g.progressWriter, g.transferrer, r, plats, name); err != nil {
166+
if err = Load(ctx, g.progressWriter, g.transferrer, &readerWithEOF{r}, plats, name); err != nil {
155167
return nil, fmt.Errorf("failed to load an archive (from %v): %w", dockerCmd.Args, err)
156168
}
157169
if err = r.Close(); err != nil {

0 commit comments

Comments
 (0)