error removing var/run to make way for new symlink: unlinkat

[hadrien-toma]

Actual behavior
Can not build the image, was working and suddenly was not.

Expected behavior
Be able to build the image.

To Reproduce
File system:

issue
├── context
│   ├── builder.ash
│   └── Dockerfile
└── index.bash

1 directory, 3 files

• Create an issue directory
• Create the following index.bash file under the issue directory:

#!/bin/bash

hereDir=`dirname $0 | while read a; do cd $a && pwd && break; done `

pids=""
rc=0

trap 'sigintTrap' 2

sigintTrap() {
	if [ "${pids}X" != "X" ]; then
		kill -9 ${pids}
	fi
	exit 2
}

minikube delete
minikube start

minikube mount ${hereDir}:/issue &
pids="${pids} $!"

minikube dashboard &
pids="${pids} $!"

cat <<EOF | kubectl apply --force -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: kaniko-job
spec:
  activeDeadlineSeconds: 1000
  template:
    metadata:
      name: kaniko-job
    spec:
      containers:
      - args:
        - -c
        - |
          /busybox/sh <<'EOF'

          /kaniko/executor \
            --context "/issue/context" \
            --no-push \
            --dockerfile "/issue/context/Dockerfile" \
            --insecure

          EOF
        command: [ "/busybox/sh" ]
        name: kaniko-job
        image: gcr.io/kaniko-project/executor:debug
        volumeMounts:
        - name: issue
          mountPath: /issue
      restartPolicy: OnFailure
      volumes:
      - name: issue
        hostPath:
          path: /issue
EOF

for pid in ${pids}; do
	wait ${pid} || let "rc=1"
done

if [ "${rc}" == "1" ]; then
	exit 1
fi

• Create a context directory in the issue directory
• Create the following Dockerfile file under the context directory:

FROM alpine:latest

COPY ./builder.ash /builder.ash

RUN chmod +x /builder.ash && /builder.ash ; rm -rf /builder.ash

• Create the following builder.ash file under the context directory:

#!/bin/ash

echo "Building..."

• Give execution permission to the index.bash script: chmod +x issue/index.bash
• Run the index.bash script: ./issue/index.bash
• I get the following in the pod:

�[36mINFO�[0m[0000] Resolved base name alpine:latest to alpine:latest 
�[36mINFO�[0m[0000] Resolved base name alpine:latest to alpine:latest 
�[36mINFO�[0m[0000] Retrieving image manifest alpine:latest      
�[36mINFO�[0m[0001] Retrieving image manifest alpine:latest      
�[36mINFO�[0m[0002] Built cross stage deps: map[]                
�[36mINFO�[0m[0002] Retrieving image manifest alpine:latest      
�[36mINFO�[0m[0003] Retrieving image manifest alpine:latest      
�[36mINFO�[0m[0004] Unpacking rootfs as cmd COPY ./builder.ash /builder.ash requires it. 
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_01_25_19_26_04.767922246: read-only file system

Additional Information

> docker version && minikube version && kubectl version && uname -a

Client: Docker Engine - Community
Version:           19.03.5
API version:       1.40
Go version:        go1.12.12
Git commit:        633a0ea838
Built:             Wed Nov 13 07:29:52 2019
OS/Arch:           linux/amd64
Experimental:      false

Server: Docker Engine - Community
Engine:
 Version:          19.03.5
 API version:      1.40 (minimum version 1.12)
 Go version:       go1.12.12
 Git commit:       633a0ea838
 Built:            Wed Nov 13 07:28:22 2019
 OS/Arch:          linux/amd64
 Experimental:     false
containerd:
 Version:          1.2.10
 GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
 Version:          1.0.0-rc8+dev
 GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
 Version:          0.18.0
 GitCommit:        fec3683
minikube version: v1.6.2
commit: 54f28ac5d3a815d1196cd5d57d707439ee4bb392
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Linux mewtwo-ubuntu 5.3.0-26-generic #28~18.04.1-Ubuntu SMP Wed Dec 18 16:40:14 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[eduscrakozabrus]

I am have this error in gitlab ci

.gitlab-ci.yaml
build:
stage: build
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
script:
- echo "{"auths":{"$CI_REGISTRY":{"username":"$CI_REGISTRY_USER","password":"$CI_REGISTRY_PASSWORD"}}}" > /kaniko/.docker/config.json
- /kaniko/executor --cache=true --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME-$CI_COMMIT_SHA

INFO[0004] Unpacking rootfs as cmd ADD Gemfile* /app/ requires it.
60 error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_01_26_11_37_35.908221403: read-only file system

[eduscrakozabrus]

solution worked for me
change version of kaniko: gcr.io/kaniko-project/executor:debug-v0.15.0

[hadrien-toma]

Thank you @eduscrakozabrus, worked for me too 🙏. So is this a bug in the newest version?

[followsound]

I've also been having this issue - this solved it

[axyjo]

Looks like a duplicate of #1001

[bozzo]

I had the same issue, i've fixed it by adding --additional-whitelist /var/run/secrets with the gcr.io/kaniko-project/executor:debug image.

[cvgw]

Strong recommendation to everyone; use an explicitly tagged release of the kaniko image. Don't use latest in your critical systems. This change has not been released and is only affecting people using latest I was incorrect, latest is only built on release. This issue is only affecting people who are using the debug tag which is built on every commit to master. But still, use an explicit version!

[rokerkony]

@cvgw thx for the suggestion, it makes totally sense. It would be great to mention it in the readme as it always advises to use :latest

[tejal29]

closing this as dupe of #1001

[fischerman]

@tejal29 What is the solution to this problem? #1001 seems to be related to CloudBuild (which I have no knowledge about). This issue here seems to be related to Kubernetes. AFAIK whitelisting /var/run exposes the service account token in the image, which is a security thread. Not whitelisting breaks the build.

[bnouvelbmll]

Bug still present in debug-v0.17.0 - reverting to earlier version fix the build but not ideal.

'''
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_02_04_10_04_34.525459306: read-only file system
'''

[consideRatio]

I had the same issue, i've fixed it by adding --additional-whitelist /var/run/secrets with the gcr.io/kaniko-project/executor:debug image.

@bozzo, where is this parameter passed? To kaniko executor? It didn't recognize that flag. I'm confused about that.