TLS Handshake timeout pulling images starting in v1.31.8+k3s1 (works in v1.31.7)

[bmorris53]

Bug Report: TLS Handshake Timeout on Image Pulls in K3s v1.31.8+

Summary

When upgrading from K3s v1.31.7+k3s1 to K3s v1.31.8+k3s1 (and all later versions in the 1.31.x and 1.32.x lines), all container image pulls fail with TLS handshake timeout errors. This issue occurs consistently across multiple registries (docker.io, quay.io, etc.) and prevents workloads from being deployed.

The problem does not occur on v1.31.7+k3s1 and earlier.

Environment

• OS: Rocky Linux 9.6 (Blue Onyx)
• Kernel: 5.14.0-570.39.1.el9_6.x86_64
• Arch: x86_64
• Cluster: Multi-master K3s cluster, on Openstack VMs
• Proxy: None (no proxy configured in environment)
• Firewall: Outbound 443 is open and verified with openssl and curl
• CNI: Cilium v1.7.6 consistent across k3s versions

k3s config.yaml for server nodes

cluster-init: true
token: "<redacted>"
node-ip: 172.40.0.119,<ipv6-redacted>
cluster-cidr: 10.42.0.0/16,fd01::/48
service-cidr: 10.43.0.0/16,fd02::/112
flannel-backend: none
disable-network-policy: true
disable-kube-proxy: true
selinux: true
secrets-encryption: true
write-kubeconfig-mode: "0644"
tls-san: 172.40.0.250
disable: metrics-server

Versions Tested

K3s Version	Go Version	containerd Version	Status
v1.31.7+k3s1	go1.23.6	2.0.4	✅ Works
v1.31.8+k3s1	go1.23.6	2.0.4	❌ Fails
v1.31.12+k3s1	go1.23.11	2.0.5	❌ Fails
v1.32.4+k3s1	go1.23.6	2.0.5	❌ Fails
v1.32.7+k3s1	go1.23.10	2.0.5	❌ Fails

Symptoms

Pods enter ImagePullBackOff with messages like:

Failed to pull image "rancher/mirrored-coredns-coredns:1.12.1": failed to resolve reference: failed to do request: Head "https://registry-1.docker.io/v2/...": net/http: TLS handshake timeout

Manually testing from host works:

curl -v https://registry-1.docker.io/v2/
openssl s_client -connect registry-1.docker.io:443

Both succeed.

But using containerd directly fails:

sudo /usr/local/bin/k3s ctr images pull docker.io/library/busybox:1.36
# error: TLS handshake timeout

Enabling containerd debug logging shows:

DEBU[0000] resolving
DEBU[0000] do request
DEBU[0000] fetch response received
DEBU[0000] ... timeout while fetching

Notes

• Issue starts between v1.31.7 (works) and v1.31.8 (fails).
• Both use Go 1.23.6 and containerd 2.0.4, suggesting a change in vendored dependencies or TLS handling in that release.
• Not specific to docker.io; also affects quay.io and other registries.
• Not specific to registries.yaml or mirrors — even direct public pulls fail.

Steps to Reproduce

1. Install K3s v1.31.8+k3s1 or later on Rocky Linux 9.6 (x86_64).
2. Attempt to pull an image:

   sudo /usr/local/bin/k3s ctr images pull docker.io/library/busybox:1.36

3. Observe TLS handshake timeout.

Test Script used:

# versions to try, newest→oldest between .12 and .7
#VERS="v1.31.12+k3s1 v1.31.11+k3s1 v1.31.10+k3s1 v1.31.9+k3s1 v1.31.8+k3s1"
VERS="v1.32.4+k3s1 v1.32.7+k3s1"

for V in $VERS; do
  echo "===== TEST $V =====" | tee -a bisect.log
  curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=$V sh -s - || { echo "install $V failed"; break; }
  sleep 6
  sudo /usr/local/bin/k3s ctr version | tee -a bisect.log

  # tiny images from multiple registries
  for IMG in docker.io/library/busybox:1.36 quay.io/prometheus/node-exporter:v1.8.1; do
    echo "Pull $IMG" | tee -a bisect.log
    # capture very chatty trace per attempt
    sudo /usr/local/bin/k3s ctr --debug images pull --http-trace --local "$IMG" \
      &> pull-$V-$(echo "$IMG"|tr '/:' '__').log
    echo "exit=$?" | tee -a bisect.log
  done
done

Logs from tests (exit=1 == FAIL | exit=0 == PASS):

===== TEST v1.31.12+k3s1 =====
Client:
  Version:  v2.0.5-k3s2.32
  Revision:
  Go version: go1.23.11

Server:
  Version:  v2.0.5-k3s2.32
  Revision:
  UUID: 6b8b8b92-e6ee-477b-abca-729e283e5ddd
Pull docker.io/library/busybox:1.36
exit=1
Pull quay.io/prometheus/node-exporter:v1.8.1
exit=1
===== TEST v1.31.11+k3s1 =====
Client:
  Version:  v2.0.5-k3s2.32
  Revision:
  Go version: go1.23.10

Server:
  Version:  v2.0.5-k3s2.32
  Revision:
  UUID: 6b8b8b92-e6ee-477b-abca-729e283e5ddd
Pull docker.io/library/busybox:1.36
exit=1
Pull quay.io/prometheus/node-exporter:v1.8.1
exit=1
===== TEST v1.31.10+k3s1 =====
Client:
  Version:  v2.0.5-k3s1.32
  Revision:
  Go version: go1.23.10

Server:
  Version:  v2.0.5-k3s1.32
  Revision:
  UUID: 6b8b8b92-e6ee-477b-abca-729e283e5ddd
Pull docker.io/library/busybox:1.36
exit=1
Pull quay.io/prometheus/node-exporter:v1.8.1
exit=1
===== TEST v1.31.9+k3s1 =====
Client:
  Version:  v2.0.5-k3s1.32
  Revision:
  Go version: go1.23.8

Server:
  Version:  v2.0.5-k3s1.32
  Revision:
  UUID: 6b8b8b92-e6ee-477b-abca-729e283e5ddd
Pull docker.io/library/busybox:1.36
exit=1
Pull quay.io/prometheus/node-exporter:v1.8.1
exit=1
===== TEST v1.31.8+k3s1 =====
Client:
  Version:  v2.0.4-k3s2
  Revision:
  Go version: go1.23.6

Server:
  Version:  v2.0.4-k3s2
  Revision:
  UUID: 6b8b8b92-e6ee-477b-abca-729e283e5ddd
Pull docker.io/library/busybox:1.36
exit=1
Pull quay.io/prometheus/node-exporter:v1.8.1
exit=1
===== TEST v1.31.7+k3s1 =====
Client:
  Version:  v2.0.4-k3s2
  Revision:
  Go version: go1.23.6

Server:
  Version:  v2.0.4-k3s2
  Revision:
  UUID: 6b8b8b92-e6ee-477b-abca-729e283e5ddd
Pull docker.io/library/busybox:1.36
exit=0
Pull quay.io/prometheus/node-exporter:v1.8.1
exit=0
===== TEST v1.32.4+k3s1 =====
Client:
  Version:  v2.0.4-k3s2
  Revision:
  Go version: go1.23.6

Server:
  Version:  v2.0.4-k3s2
  Revision:
  UUID: 6b8b8b92-e6ee-477b-abca-729e283e5ddd
Pull docker.io/library/busybox:1.36
exit=1
Pull quay.io/prometheus/node-exporter:v1.8.1
exit=1
===== TEST v1.32.7+k3s1 =====
Client:
  Version:  v2.0.5-k3s2.32
  Revision:
  Go version: go1.23.10

Server:
  Version:  v2.0.5-k3s2.32
  Revision:
  UUID: 6b8b8b92-e6ee-477b-abca-729e283e5ddd
Pull docker.io/library/busybox:1.36
exit=1
Pull quay.io/prometheus/node-exporter:v1.8.1
exit=1

Downgrade to v1.31.7+k3s1 and repeat — the pull succeeds.

Expected Behavior

K3s should pull images successfully as in v1.31.7 and earlier.

Actual Behavior

All image pulls fail with TLS handshake timeouts on v1.31.8 and later.

Impact

• Prevents workloads from starting (all image pulls fail).
• Impacts both system components (CoreDNS, Traefik, etc.) and user workloads.
• Blocks upgrades beyond v1.31.7.

Request

Please investigate changes introduced in v1.31.8+k3s1 that affect containerd's TLS handling.
This appears unrelated to containerd itself (still 2.0.4 at that point) and may be tied to Go TLS libraries or other vendored changes.

---

Contact

Environment available for reproducing/testing if needed. Logs, debug traces, and configs can be provided on request.

[brandond]

Yeah... I really can't reproduce this. We've received no other reports of this from other users either, so I suspect this is going to be something unique to your environment, rather than a defect in k3s.

You said that 443 outbound is open, but do you have a firewall or something doing packet inspection that might be shutting down connections based on something it doesn't like about the TLS handshake? Are you using a proxy? Do you have a security agent running on your host? What else is different for you?

One of the changes between 1.31.7 and 1.31.8 was to switch from go 1.22 to 1.23 in go.mod, along with minor bumps to x/net and x/crypto.

https://github.com/k3s-io/k3s/compare/v1.31.7+k3s1..v1.31.8+k3s1#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6

I am not aware of any changes in go1.23 that would cause the behavior you're reporting: https://go.dev/doc/go1.23#cryptotlspkgcryptotls

[bmorris53]

Literally the only change made is upgrading the k3s binary and its inclusions. I Run the install script with the exact same config on the exact same nodes, no changes in CNI, or other workloads running in the cluster and it breaks on anything newer than 1.31.7. Downgrading, using the install script, back to 1.31.7 restores functionality with no other changes.

I do not have proxies in this environment.

I will be able to check the firewall logs tomorrow to see if it is killing SSL handshakes, but like you said, if there weren't significant crypto changes between go versions, I don't know why my firewalls would block TLS traffic on 1.31.8+ vs 1.31.7.

I do find it interesting that when running k3s -version with 1.31.7 installed, go version shows 1.23.6, but it would make sense if it were actually 1.22 and that something in the bump to 1.23 is not agreeing with my environment.

I do have a cloud protection suite running from CrowdStrike with an Admission Controller, falcon-sensor, and an image analyzer. I do not see anything in their respective logs blocking. The fact that I get the TLS Handshake timeout when using ctr to pull the image directly I think means that those protections are bypassed anyway.

[brandond]

wrt to the go version that is used to build, vs the version in go.mod, I will reference the upstream docs:

https://go.dev/doc/modules/gomod-ref#go

The go directive affects use of new language features:

For packages within the module, the compiler rejects use of language features introduced after the version specified by the go directive. For example, if a module has the directive go 1.12, its packages may not use numeric literals like 1_000_000, which were introduced in Go 1.13.

Again I don't think that's related.

I will be able to check the firewall logs tomorrow to see if it is killing SSL handshakes
I do have a cloud protection suite running from CrowdStrike with an Admission Controller, falcon-sensor, and an image analyzer.

Please disable as much of this as you can, and confirm that your firewall isn't blocking anything. We've received literally zero other reports of any similar problems with these versions, so it's highly likely to be one of these.

[bmorris53]

I will still have our firewall team look at the SSL handshakes tomorrow.

Tonight I reset back to v1.31.7, Crowdstrike still in place, no Firewall changes, no underlying node changes, all works.

I removed the Crowdstrike workloads completely by deleting their namespaces and confirming all workloads gone.

I upgraded to v1.32.7 with the install script, again making no other changes aside from removal of CS.

ctr image pulls still fail with TLS Handshake Timeouts

I installed nerdctl and pointed it to the k3s containerd socket and am able to successfully pull images into the k3s containerd k8s.io namespace then use ctr to list them.

This makes it very difficult to think it is some middlebox if nerdctl, with the k3s containerd daemon and socket is able to pull just fine.

I did the same test with v1.31.8 and got the same result. Dropped back to v1.31.7 and all started working again.

v1.32.7 test

v1.31.8 test

v1.31.7 test with successful ctr and nerdctl pulls

v1.33.4 test for completeness, still fails

[brandond]

Im not sure how nerdctl works - if it does the pulls itself and writes to the image store, or if it merely asks containerd to pull.

I do know that containerd mirror support requires pulling via CRI API, instead of the native containerd API. Are you using mirrors? Do your mirrors have specific TLS version requirements? Can you reproduce this with upstream containerd v2.0.4, not the one bundled with k3s?

Unless you can come up with some way to reproduce this on an arbitrary node outside your specific network, I'm really not sure how I can help, or what else I might do with any of the information you're providing. I believe that it's happening to you, but that's about all I can offer.

[bmorris53]

No I do not have any registries or mirrors configured, ie no registries.yml in play. Everything is direct to the public registries. From what I can read, nerdctl is handing off the instruction to containerd to pull the image.  It is a substitute for ctr. Sent from my iPhoneOn Sep 17, 2025, at 10:41 PM, Brad Davidson ***@***.***> wrote:brandond left a comment (k3s-io/k3s#12940) Im not sure how nerdctl works - if it does the pulls itself and writes to the image store, or if it merely asks containerd to pull. I do know that containerd mirror support requires pulling via CRI API, instead of the native containerd API. Are you using mirrors? Do your mirrors have specific TLS version requirements? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[bmorris53]

I verified with my security team that they are not blocking anything on the FW.

I did some extra testing today and found that if I disable the IPv6 stack on my interface, everything works. If I re-enable the IPv6 stack, ctr goes back to trying IPv6 and it all fails.

Keep in mind, nerdctl and curl all work over IPv6 when communicating with the external registries indicating general IPv6 networking is fine. And IPv6 works fine on v1.31.7.

So seems to be something specific to ctr's interaction with IPv6 in these later builds.

I also tried several containerd config changes that I could find online to try and force the cri to use IPv4 for dns but they all seem to be ignored. And also tried pinning docker.io to one of its IPv4 addresses using regisitries.yaml, which also seemed to be ignored as ctr pulls still attempted IPv6.

config.toml.tmpl

version = 3
root = "/var/lib/rancher/k3s/agent/containerd"
state = "/run/k3s/containerd"

[grpc]
  address = "/run/k3s/containerd/containerd.sock"

[plugins.'io.containerd.internal.v1.opt']
  path = "/var/lib/rancher/k3s/agent/containerd"

[plugins.'io.containerd.grpc.v1.cri']
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"

[plugins.'io.containerd.cri.v1.runtime']
  enable_selinux = true
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  device_ownership_from_security_context = false

[plugins.'io.containerd.cri.v1.images']
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true

[plugins.'io.containerd.cri.v1.images'.pinned_images]
  sandbox = "rancher/mirrored-pause:3.6"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runhcs-wcow-process]
  runtime_type = "io.containerd.runhcs.v1"

[plugins.'io.containerd.cri.v1.images'.registry]
  config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"

[plugins."io.containerd.grpc.v1.cri".registry.configs."registry-1.docker.io".resolver]
  dialer = "tcp4"

registries.yaml

mirrors:
  "docker.io":
    endpoint:
      - "https://54.82.82.253"
configs:
  "54.82.82.253":
    tls:
      insecure_skip_verify: false

Resulted in:

# File generated by k3s. DO NOT EDIT.
server = "https://registry-1.docker.io/v2"
capabilities = ["pull", "resolve", "push"]

[host]

[host."https://54.82.82.253/v2"]
  capabilities = ["pull", "resolve"]

And pulls with ctr still attempted IPv6:

$ sudo /usr/local/bin/k3s ctr --debug images pull --http-trace --local docker.io/library/busybox:1.36
DEBU[0000] Asset dir /var/lib/rancher/k3s/data/2d3d7fcf0abecfb50febe0c967319924342b145aefffc1fa5b38d792a5c1dcfc
DEBU[0000] Running /var/lib/rancher/k3s/data/2d3d7fcf0abecfb50febe0c967319924342b145aefffc1fa5b38d792a5c1dcfc/bin/ctr [ctr --debug images pull --http-trace --local docker.io/library/busybox:1.36]
DEBU[0000] fetching                                      image="docker.io/library/busybox:1.36"
DEBU[0000] resolving                                     host=registry-1.docker.io
DEBU[0000] do request                                    host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v2.0.4-k3s2 request.method=HEAD url="https://registry-1.docker.io/v2/library/busybox/manifests/1.36"
DEBU[0000] DNS lookup                                    host=registry-1.docker.io
DEBU[0000] DNS lookup complete                           coalesced=false result="2600:1f18:2148:bc01:b3c6:6a5a:1c56:6411"
INFO[0010] fetch failed                                  error="failed to do request: Head \"https://registry-1.docker.io/v2/library/busybox/manifests/1.36\": net/http: TLS handshake timeout" host=registry-1.docker.io
ctr: failed to resolve reference "docker.io/library/busybox:1.36": failed to do request: Head "https://registry-1.docker.io/v2/library/busybox/manifests/1.36": net/http: TLS handshake timeout

If I add the following to /etc/hosts, it works, but would have to do this for every registry:

54.82.82.253    registry-1.docker.io
54.175.184.239  auth.docker.io

Results in:

$ sudo /usr/local/bin/k3s ctr --debug images pull --http-trace --local docker.io/library/busybox:1.36
DEBU[0000] Asset dir /var/lib/rancher/k3s/data/2d3d7fcf0abecfb50febe0c967319924342b145aefffc1fa5b38d792a5c1dcfc
DEBU[0000] Running /var/lib/rancher/k3s/data/2d3d7fcf0abecfb50febe0c967319924342b145aefffc1fa5b38d792a5c1dcfc/bin/ctr [ctr --debug images pull --http-trace --local docker.io/library/busybox:1.36]
DEBU[0000] fetching                                      image="docker.io/library/busybox:1.36"
DEBU[0000] resolving                                     host=registry-1.docker.io
DEBU[0000] do request                                    host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v2.0.4-k3s2 request.method=HEAD url="https://registry-1.docker.io/v2/library/busybox/manifests/1.36"
DEBU[0000] DNS lookup                                    host=registry-1.docker.io
DEBU[0000] DNS lookup complete                           coalesced=false result=54.82.82.253
DEBU[0000] Connection successful                         remote_addr="54.82.82.253:443" reused=false
DEBU[0000] fetch response received                       host=registry-1.docker.io response.header.connection=keep-alive response.header.content-length=158 response.header.content-type=application/json response.header.date="Fri, 19 Sep 2025 17:53:22 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=76.81.131.46 response.header.strict-transport-security="max-age=31536000" response.header.www-authenticate="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/busybox:pull\"" response.status="401 Unauthorized" url="https://registry-1.docker.io/v2/library/busybox/manifests/1.36"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/busybox:pull\"" host=registry-1.docker.io
DEBU[0000] do request                                    host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v2.0.4-k3s2 request.method=HEAD url="https://registry-1.docker.io/v2/library/busybox/manifests/1.36"
DEBU[0000] DNS lookup                                    host=auth.docker.io
DEBU[0000] DNS lookup complete                           coalesced=false result=54.175.184.239
DEBU[0000] Connection successful                         remote_addr="54.175.184.239:443" reused=false
DEBU[0000] Connection successful                         remote_addr="54.82.82.253:443" reused=true
DEBU[0000] fetch response received                       host=registry-1.docker.io response.header.connection=keep-alive response.header.content-length=9535 response.header.content-type=application/vnd.oci.image.index.v1+json response.header.date="Fri, 19 Sep 2025 17:53:22 GMT" response.header.docker-content-digest="sha256:4b8407fadd8100c61b097d63efe992b2c033e7d371c9117f7a9462fe87e31176" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source= response.header.etag="\"sha256:4b8407fadd8100c61b097d63efe992b2c033e7d371c9117f7a9462fe87e31176\"" response.header.ratelimit-limit="100;w=21600" response.header.ratelimit-remaining="100;w=21600" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" url="https://registry-1.docker.io/v2/library/busybox/manifests/1.36"
DEBU[0000] resolved                                      desc.digest="sha256:4b8407fadd8100c61b097d63efe992b2c033e7d371c9117f7a9462fe87e31176" host=registry-1.docker.io
DEBU[0000] fetch                                         digest="sha256:4b8407fadd8100c61b097d63efe992b2c033e7d371c9117f7a9462fe87e31176" mediatype=application/vnd.oci.image.index.v1+json size=9535
DEBU[0000] fetch                                         digest="sha256:0816a07fed2a47ec15cf0885f8ea43b68d2ee5ac091c5a8f644fe06fa500888f" mediatype=application/vnd.oci.image.manifest.v1+json size=610
DEBU[0000] fetch                                         digest="sha256:bed300b774c1a5211a31611c5f77d0ecfeab663475171ff051dfc9d67d996faf" mediatype=application/vnd.oci.image.manifest.v1+json size=559
DEBU[0000] fetch                                         digest="sha256:aefc3a378c4cf11a6d85071438d3bf7634633a34c6a68d4c5f928516d556c366" mediatype=application/vnd.oci.image.manifest.v1+json size=610
DEBU[0000] fetch                                         digest="sha256:9fcc7e4b4446ed7531372a9cd397349750dcee99f9f666b53dded9b8052eafcc" mediatype=application/vnd.oci.image.manifest.v1+json size=610
DEBU[0000] fetch                                         digest="sha256:4bd82624fb43378360d603cd7cf356d2d15fad8776b9fd48d1e45af6bc97952c" mediatype=application/vnd.oci.image.manifest.v1+json size=610
DEBU[0000] fetch                                         digest="sha256:4a35a7836fe08f340a42e25c4ac5eef4439585bbbb817b7bd28b2cd87c742642" mediatype=application/vnd.oci.image.manifest.v1+json size=610
DEBU[0000] fetch                                         digest="sha256:11461e20209daaea60978a4cdd15be4ce0479187c5c1cf8c9af343309b8ba5c5" mediatype=application/vnd.oci.image.manifest.v1+json size=559
DEBU[0000] fetch                                         digest="sha256:843b9fe470fe0774dadfc16b9a79c2eb857cb78b58faa39e74e15a1deccf52bb" mediatype=application/vnd.oci.image.manifest.v1+json size=610
DEBU[0000] fetch                                         digest="sha256:b51ff6ead549c0bef24fa658b9fa6add1af4a5edcb158c1bdc751a07f86d9eec" mediatype=application/vnd.oci.image.manifest.v1+json size=608
DEBU[0000] fetch                                         digest="sha256:ee5f9eb2a2029537e90b1ccdb98d8bbc8b90247b77a5459f340c9078f3fddb01" mediatype=application/vnd.oci.image.manifest.v1+json size=559
DEBU[0000] fetch                                         digest="sha256:e0daf5ced82aa2fe4ee9979675e3dfe1c307007feef00768c1ebcda79dd1e33c" mediatype=application/vnd.oci.image.manifest.v1+json size=559
DEBU[0000] fetch                                         digest="sha256:bdea9c898ef557ae93ea71350fc13ce324e660c3ca294f8cfe81bf475a0ce694" mediatype=application/vnd.oci.image.manifest.v1+json size=610
DEBU[0000] fetch                                         digest="sha256:2a9ac789d2b8df031c378d95148524ef69943c987a8f46481c45b160e2a4fd7b" mediatype=application/vnd.oci.image.manifest.v1+json size=559
DEBU[0000] fetch                                         digest="sha256:917e000af13902d736c7b9b5bc821584163459a85929243ef7007de97d970511" mediatype=application/vnd.oci.image.manifest.v1+json size=610
DEBU[0000] fetch                                         digest="sha256:feb787454b8b10de6d0ce3645c5ffd5f2b7ebc2414e43fba56a1164281289707" mediatype=application/vnd.oci.image.manifest.v1+json size=559
DEBU[0000] fetch                                         digest="sha256:f1da175102356cbf088e1ddb226717874e36778e99ace6d4ff71edddfcc2e420" mediatype=application/vnd.oci.image.manifest.v1+json size=559
DEBU[0000] fetch                                         digest="sha256:d28ecf4e700dbecd41a28d4ce2705374e5e438e6674e868f43b1fddae03cd0db" mediatype=application/vnd.oci.image.manifest.v1+json size=559
DEBU[0001] fetch                                         digest="sha256:45280de0603b5c21747e728a14cda49b80a801dc492ddd2038def71268ec09a0" mediatype=application/vnd.oci.image.config.v1+json size=475
DEBU[0001] fetch                                         digest="sha256:04baeb1a890e5b1aa829cab2cb6bc0d2d6938bb17248256e2ff5a4c956abcbde" mediatype=application/vnd.oci.image.config.v1+json size=167
DEBU[0001] fetch                                         digest="sha256:a0aa8a5596529ff6969e508485d4a3cdab23b54888c01312a0b6bac2cc03d369" mediatype=application/vnd.oci.image.config.v1+json size=477
DEBU[0001] fetch                                         digest="sha256:a9739a817d63534f3c8ceef491b880e3b562e746a6645feb0dc8b3333a4973eb" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=2209397
DEBU[0001] fetch                                         digest="sha256:1827167fde90df99d9341a27fbce2b445550eb2b18105e03f98102f00c0ec35e" mediatype=application/vnd.oci.image.config.v1+json size=459
DEBU[0001] fetch                                         digest="sha256:b9162ba8d2fd73929191ff142c576928d7b8833f2a68b52599a24bbd49c6afcb" mediatype=application/vnd.oci.image.config.v1+json size=457
DEBU[0001] fetch                                         digest="sha256:95393b3e768eee6f6d3650c1dff2778910a82c43f23ca5cf5d62e256ac0c8c0e" mediatype=application/vnd.oci.image.config.v1+json size=461
DEBU[0001] fetch                                         digest="sha256:65ced27e7f90cbe7b13be20241889a3a7c6432f3bab54702afc8e538d9a5c7e1" mediatype=application/vnd.oci.image.config.v1+json size=167
DEBU[0001] fetch                                         digest="sha256:1bc7aef50b8ee9d1bf2aa6268d3cd176e8dae3e48a435b39bce1c4b0834fad4f" mediatype=application/vnd.oci.image.config.v1+json size=461
DEBU[0001] fetch                                         digest="sha256:0c0b536e292a45959f32a1ced3d74b0b8d209f44d57af993bd2dce7d6d008b7b" mediatype=application/vnd.oci.image.config.v1+json size=167
DEBU[0001] fetch                                         digest="sha256:f1e9ff73e59d237fafa0fea8109952ea406f7b57a12788ec196e3b5786385bd6" mediatype=application/vnd.oci.image.config.v1+json size=478
DEBU[0001] fetch                                         digest="sha256:445572e662e1b3a205829ed4638651716ceb76ce150925f845299abe2685a39e" mediatype=application/vnd.oci.image.config.v1+json size=167
DEBU[0001] fetch                                         digest="sha256:17d9e583ed21c07771b022ea28762551fada538978f6b3e9964ef49bd0abd462" mediatype=application/vnd.oci.image.config.v1+json size=475
DEBU[0001] fetch                                         digest="sha256:9cbe799fcfa07f1cda526c684b8e3da9ef1cf3610525bcc7b7414afe428cd9d3" mediatype=application/vnd.oci.image.config.v1+json size=459
DEBU[0001] fetch                                         digest="sha256:76bb7b2181836ea6b92d83b2ecd3712c87b679da662ac767be6d348a2fdf1c1b" mediatype=application/vnd.oci.image.config.v1+json size=167
DEBU[0001] fetch                                         digest="sha256:126ef9a7597edfb01658952aa18539c55a45446153c3339e9b18ccbbe59198d0" mediatype=application/vnd.oci.image.config.v1+json size=167
DEBU[0001] fetch                                         digest="sha256:fd6b57cbdbd1bd18b55fa257dcf643981d5a0f38367a016ff91aab10a6a4942f" mediatype=application/vnd.oci.image.config.v1+json size=167
DEBU[0001] fetch                                         digest="sha256:41060c8e176a7783f7f6420acb9ad6b5a8333e6c6b21d039592bd5b37d367ea4" mediatype=application/vnd.oci.image.config.v1+json size=167
DEBU[0001] unpacking                                     image="docker.io/library/busybox:1.36"
unpacking linux/amd64 sha256:4b8407fadd8100c61b097d63efe992b2c033e7d371c9117f7a9462fe87e31176...
done: 23.383933ms

[brandond]

Are you sure that other things are successfully using IPv6? As far as I can see the logs don't actually show what IP it is actually connecting to, it could just be falling back to IPv4 for some reason.

Maybe grab a packet capture, and see what's actually going on? Is is possible that you've allowed access on port 443 for IPv4 but not v6?

See also the discussion at:

• net: ipv4 chosen first when ipv6 is available golang/go#68795