podman 5 stuck on sending tarball

[Fydon]

Issue Description

podman gets stuck on sending tarball for a ARM64 build on a x64 Windows PC. I left sending tarball running for over 10.5 hours after all other steps completed and had still not progressed.

Steps to reproduce the issue

Steps to reproduce the issue

1. git clone --branch podman-5-stalls git@github.com:Fydon/opentelemetry-cpp-contrib.git .
2. cd instrumentation/otel-webserver-module
3. docker buildx build -t apache_centos7_arm64 -f docker/centos7/Dockerfile --load . --platform linux/arm64 --build-arg AUTOCONF_BUILD_TYPE='aarch64-unknown-linux-gnu' --build-arg BUILD_ARCH='arm64' --build-arg CMAKE_ARCH='aarch64' --build-arg GOSU_ARCH='arm64' --build-arg JDK_ARCH='aarch64' --build-arg NGINX_ARCH='aarch64'

Describe the results you received

Stuck on sending tarball

Describe the results you expected

I expected the layers to be exported and an image to be created.

podman info output

> podman info
OS: windows/amd64
provider: wsl
version: 5.0.1

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to connect: dial tcp 127.0.0.1:4161: connectex: No connection could be made because the target machine actively refused it.

> podman --version
podman.exe version 5.0.1

Windows 11 Version 23H2 Build 22631.3296

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[djgraff209]

I have also encountered this problem on Windows 10 & 11 with podman 4.8+ (including 5.0.0 and 5.0.1)

When this scenario occurs the podman machine becomes unresponsive and the only recourse is to terminate and restart the machine. However, this does not always work and the same scenario arises after restart.

[Fydon]

Yes it consumes a lot of resources. I have 64 GB RAM and so applications like Edge become slow to respond and then stopped, but otherwise applications continued to function. Therefore I expect that the export would have been able to complete if it was progressing. It would be nice to have a progress bar on the "sending tarball" process if it was possible.

[Fydon]

I originally thought the problem was to do with complex builds but testing even Docker's getting started app won't load.

1. git clone git@github.com:docker/getting-started-app.git .
2. create the Dockerfile below
3. docker build -t getting-started --load .

FROM node:18-alpine
WORKDIR /app
COPY . .
RUN yarn install --production
CMD ["node", "src/index.js"]
EXPOSE 3000

Sorry I misunderstood @djgraff209 comment. Yes once this fails, I need to stop and start the Podman Machine to do anything more. My PC as a whole also becomes less responsive.

[djgraff209]

I have to see if I can locate the error message I was seeing in the machine logs - something to the effect of deadline exceeded.

I have vague recollection of this having to do something with WSL networking in half-duplex mode but my memory is suspect on that.

[djgraff209]

This is an issue I saw microsoft/WSL#10688

As well as the referenced issue #20379

[Luap99]

I am going to close this as duplicate of #20379 as there is enough information there and I have no reason to think it is a different issue here.

[djgraff209]

followup note - I just ran wsl --update to update to version 2.1.5 a few minutes ago on Windows 11 and it appears (on the face of it) running with podman 5.0.1 that docker buildx build ... --load now is working and very snappy

I cannot say definitively if wsl 2.1.5 resolved this or not but it certainly seems to have resolved for me.

[Fydon]

Thank you for the suggestion. I was already on 2.1.5. Tried docker build -t getting-started --load . again just to be sure and it is still stuck on sending tarball. The build takes a few seconds but the exporting of layers has been running for minutes and still not completed.

>  wsl --update
Checking for updates.
The most recent version of Windows Subsystem for Linux is already installed.

> wsl --version
WSL version: 2.1.5.0
Kernel version: 5.15.146.1-2
WSLg version: 1.0.60
MSRDC version: 1.2.5105
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22631.3447

[djgraff209]

I also had a docker CLI update from chocolatey (failed to mention that) prior to updating wsl

Docker (cli) v26.0.1 build d260a54

[Fydon]

I also tried updating to 2.2.2, but still the same.

> wsl --update --pre-release
Checking for updates.
Updating Windows Subsystem for Linux to version: 2.2.2.

> wsl --version
WSL version: 2.2.2.0
Kernel version: 5.15.150.1-2
WSLg version: 1.0.61
MSRDC version: 1.2.5105
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22631.3447

I don't have chocolatey, but downloading the CLI from their website instead didn't resolve it for me.

> ~\Downloads\docker.exe version  
Client:
 Version:           26.0.1
 API version:       1.41 (downgraded from 1.45)
 Go version:        go1.21.9
 Git commit:        d260a54
 Built:             Thu Apr 11 10:54:24 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: linux/amd64/fedora-39
 Podman Engine:
  Version:          5.1.0-dev-4452d307e
  APIVersion:       5.1.0-dev-4452d307e
  Arch:             amd64
  BuildTime:        2024-04-09T01:00:00+01:00
  Experimental:     false
  GitCommit:
  GoVersion:        go1.21.9
  KernelVersion:    5.15.150.1-microsoft-standard-WSL2
  MinAPIVersion:    4.0.0
  Os:               linux
 Conmon:
  Version:          conmon version 2.1.10, commit:
  Package:          conmon-2.1.10-1.20240313132120223048.main.19.gaffab49.fc39.x86_64
 OCI Runtime (crun):
  Version:          crun version UNKNOWN
commit: 655e6e977722e2ed31de33877547ee60e091a5a7
rundir: /run/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  Package:          crun-1.14.4-1.20240402154251979914.main.23.ga32cc45.fc39.x86_64
 Engine:
  Version:          5.1.0-dev-4452d307e
  API version:      1.41 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:
  Built:            Tue Apr  9 01:00:00 2024
  OS/Arch:          linux/amd64
  Experimental:     false