small perf optimizing. Only remove whiteout path if it needs to be included in base image

[tejal29]

For large base images, we end up calling os.Remove for every whiteout file.

Relates to #1087, #1073
Fixes: #1073 (comment)

In this pr, in case whiteout files are to be included, skip removing them from the File System because

1. These whiteout paths are not going to be created during the untar process.
2. These whiteout paths may be symlinks which point to stdout or stderr and deleting them is not allowed. See

Performance optimization:
1 mins speed up when unpacking base image gcr.io/deeplearning-platform-release/tf2-cpu.2-1 of size 2.3GB

Testing notes

Building with gcr.io/kaniko-project:executor:debug the time spent is 4 mins.

/busybox/time kaniko/executor -f dockerfiles/Dockerfile1 --context=dir://workspace --destination=gcr.io/tejal-test/test-ml-latest-master
tejaldesai@@skaffold (prototype)$ docker run -it --entrypoint /busybox/sh -v /usr/local/google/home/tejaldesai/.config/gcloud:/root/.config/gcloud -v /usr/local/google/home/tejaldesai/workspace/kaniko/integration:/workspace gcr.io/tejal-test/executor:debug
/ # /busybox/time kaniko/executor -f dockerfiles/Dockerfile1 --context=dir://workspace --destination=gcr.io/tejal-test/test-ml-latest-master
INFO[0000] Resolved base name gcr.io/deeplearning-platform-release/tf2-cpu.2-1 to gcr.io/deeplearning-platform-release/tf2-cpu.2-1 
INFO[0000] Using dockerignore file: /workspace/.dockerignore 
INFO[0000] Resolved base name gcr.io/deeplearning-platform-release/tf2-cpu.2-1 to gcr.io/deeplearning-platform-release/tf2-cpu.2-1 
INFO[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-cpu.2-1 
INFO[0001] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-cpu.2-1 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-cpu.2-1 
INFO[0002] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-cpu.2-1 
INFO[0002] Unpacking rootfs as cmd RUN mkdir /tpu requires it. 
INFO[0102] Taking snapshot of full filesystem...        
INFO[0105] Resolving paths                              
INFO[0186] RUN mkdir /tpu                               
INFO[0186] cmd: /bin/sh                                 
INFO[0186] args: [-c mkdir /tpu]                        
INFO[0186] Taking snapshot of full filesystem...        
INFO[0188] Resolving paths                              
INFO[0236] COPY test.py /tpu/                           
INFO[0236] Resolving paths                              
INFO[0236] Taking snapshot of files...                  
real	4m 0.38s
user	2m 11.51s
sys	2m 10.03s

Building using latest master

busybox/time kaniko/executor -f dockerfiles/Dockerfile1 --context=dir://workspace --destination=gcr.io/tejal-test/test-ml

/ # /busybox/time kaniko/executor -f dockerfiles/Dockerfile1 --context=dir://workspace --destination=gcr.io/tejal-test/test-ml
INFO[0000] Resolved base name gcr.io/deeplearning-platform-release/tf2-cpu.2-1 to gcr.io/deeplearning-platform-release/tf2-cpu.2-1 
INFO[0000] Using dockerignore file: /workspace/.dockerignore 
INFO[0000] Resolved base name gcr.io/deeplearning-platform-release/tf2-cpu.2-1 to gcr.io/deeplearning-platform-release/tf2-cpu.2-1 
INFO[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-cpu.2-1 
INFO[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-cpu.2-1 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-cpu.2-1 
INFO[0001] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-cpu.2-1 
INFO[0002] Unpacking rootfs as cmd RUN mkdir /tpu requires it. 
INFO[0105] Taking snapshot of full filesystem...        
INFO[0109] Resolving paths                              
INFO[0188] RUN mkdir /tpu                               
INFO[0188] cmd: /bin/sh                                 
INFO[0188] args: [-c mkdir /tpu]                        
INFO[0188] Taking snapshot of full filesystem...        
INFO[0189] Resolving paths                              
INFO[0236] COPY test.py /tpu/                           
INFO[0236] Resolving paths                              
INFO[0236] Taking snapshot of files...                  
real	3m 59.81s
user	2m 13.99s
sys	2m 10.16s

Both the images are same and not difference in size

tejaldesai@@skaffold (prototype)$ container-diff diff gcr.io/tejal-test/test-ml-latest-master gcr.io/tejal-test/test-ml

-----Size-----

Image size difference between gcr.io/tejal-test/test-ml-latest-master and gcr.io/tejal-test/test-ml: None