Skip to content

Introduce support for OIDC workload identity federation#3711

Merged
rchincha merged 6 commits intoproject-zot:mainfrom
matheuscscp:improve-oidc-wid
Jan 25, 2026
Merged

Introduce support for OIDC workload identity federation#3711
rchincha merged 6 commits intoproject-zot:mainfrom
matheuscscp:improve-oidc-wid

Conversation

@matheuscscp
Copy link
Copy Markdown
Contributor

@matheuscscp matheuscscp commented Jan 17, 2026

Closes: #3704

Builds on/Supersedes: #3705

  • Support multiple OIDC issuers. This is crucial for a single Zot instance to support multiple Kubernetes clusters, since each cluster has its own issuer URL.
  • Support CEL expressions for a more powerful ability of validation and extraction of username and groups from the OIDC claims. This is critical for security integrations like this as users normally have to implement additional assertions to comply with security requirements and properly isolate identities. Two strong examples of this mechanism are the GCP Workload Identity Federation feature where you can use CEL expressions to do exactly what I'm proposing here, and also what we implemented in Flux Operator for OIDC SSO, same thing.
  • Load the OIDC issuer lazily. This is crucial for the Zot server not to crash on startup due to an OIDC issuer server that is temporarily unavailable. This feature should not impact the Zot server health, as there are other authentication mechanisms available. We implement the same strategy in Flux Operator.

@rchincha
Copy link
Copy Markdown
Contributor

Not sure why the complete ci/cd didn't run

@andaaron
Copy link
Copy Markdown
Contributor

Not sure why the complete ci/cd didn't run

Maybe amend and push again?

@matheuscscp
Copy link
Copy Markdown
Contributor Author

Not sure why the complete ci/cd didn't run

Maybe amend and push again?

done, also fixed linting and commit msg

@matheuscscp
Copy link
Copy Markdown
Contributor Author

matheuscscp commented Jan 18, 2026

Not sure why the complete ci/cd didn't run

It's because most of the GH workflows run on PRs targeting the main branch:

  pull_request:
    branches:
      - main

I'd say, merge this and let's fix any issues in the main PR branch copilot/fix-issue-3704

@rchincha
Copy link
Copy Markdown
Contributor

@matheuscscp I suggest "adopting" the copilot PR and posting changes against main directly - much easier to iterate from past experience.

@matheuscscp matheuscscp changed the base branch from copilot/fix-issue-3704 to main January 18, 2026 23:31
@matheuscscp matheuscscp changed the title Add CEL-based OIDC claims processor Introduce support for OIDC workload identity federation Jan 18, 2026
@matheuscscp
Copy link
Copy Markdown
Contributor Author

@rchincha Done, I just changed the target branch to main here. I think the GH workflows are now awaiting approval 🙏

@codecov
Copy link
Copy Markdown

codecov bot commented Jan 19, 2026

Codecov Report

❌ Patch coverage is 92.93194% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.58%. Comparing base (ba3436c) to head (9e20672).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
pkg/api/bearer_oidc.go 86.60% 8 Missing and 7 partials ⚠️
pkg/cel/expression.go 90.24% 5 Missing and 3 partials ⚠️
pkg/api/authn.go 96.42% 1 Missing and 1 partial ⚠️
pkg/cel/claim_processor.go 98.27% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3711      +/-   ##
==========================================
+ Coverage   91.57%   91.58%   +0.01%     
==========================================
  Files         186      189       +3     
  Lines       26489    26846     +357     
==========================================
+ Hits        24258    24588     +330     
- Misses       1444     1459      +15     
- Partials      787      799      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rchincha
Copy link
Copy Markdown
Contributor

@matheuscscp pls take a look at the ci failures.

Ideally, once your changes are done, a e2e test script should be added under
https://github.com/project-zot/zot/tree/main/examples/kind

@matheuscscp
Copy link
Copy Markdown
Contributor Author

@rchincha Can you pls approve the workflow runs?

@matheuscscp
Copy link
Copy Markdown
Contributor Author

@rchincha I think I fixed at least 3 of the red checks now, please reapprove the workflows 🙏 I'll continue tomorrow 👌

@matheuscscp
Copy link
Copy Markdown
Contributor Author

@rchincha Pls approve one more time, I think now I fixed both coverage red checks

@rchincha
Copy link
Copy Markdown
Contributor

@matheuscscp pls also rebase, will re-run workflows

@matheuscscp
Copy link
Copy Markdown
Contributor Author

@rchincha I fixed the golangci-lint errors again and rebased. One of the red checks seemed unrelated and the other was about the binary size growing 11%, which I can't do anything about 🤷 😁 Please re-approve 🙏

@rchincha
Copy link
Copy Markdown
Contributor

@matheuscscp do you want to try this patch for your original issue and see if it fixes it?

@rchincha
Copy link
Copy Markdown
Contributor

rchincha commented Jan 23, 2026

| zot minimal binary increased by 12.71% comparing with main

| PR binary size: 53870945 Bytes
| main branch binary size: 47792481 Bytes

where is this size increase coming from?
Your go.mod/go.sum changes adding this much?

@rchincha
Copy link
Copy Markdown
Contributor

#3729

@matheuscscp
Copy link
Copy Markdown
Contributor Author

matheuscscp commented Jan 23, 2026

Binary Size Increase Analysis

The ~12.7% increase (from 47.8MB to 53.9MB) is expected and comes primarily from the new dependencies added to support OIDC workload identity federation with CEL expressions:

Main contributors:

  1. github.com/google/cel-go v0.26.1 - This is the biggest contributor. CEL (Common Expression Language) is a substantial library that includes:

    • Parser (uses antlr4-go)
    • Type checker
    • Compiler
    • Evaluator
    • Standard functions library
    • Protobuf definitions from google.golang.org/genproto
  2. github.com/coreos/go-oidc/v3 - OIDC provider/verifier library

  3. github.com/go-jose/go-jose/v4 - JOSE/JWT handling library

Why the minimal binary specifically?

The minimal binary is built with EXTENSIONS= (empty), which excludes extensions like trivy and Kubernetes libraries. Those were the only places that previously had cel-go as a transitive dependency. By adding CEL to the core authentication code (internal/cel/ and pkg/api/authn.go), it now gets compiled into all binaries including minimal.

Is there an alternative?

The CEL dependency is intentional for the feature - it enables powerful claim mapping like:

  • claims.email.split('@')[0] for username extraction
  • claims.org in ['allowed-org-1', 'allowed-org-2'] for validation
  • Complex expressions matching GCP Workload Identity Federation and Flux Operator patterns

Without CEL, users would have much more limited claim mapping options. CEL was specifically chosen to match the flexibility of GCP's workload identity federation and Flux Operator OIDC features.

Summary

Yes, this increase is expected. The feature adds a substantial capability (OIDC workload identity with CEL expressions) that requires the CEL library. Since this is core authentication code, it affects the minimal binary which previously didn't include CEL's transitive dependencies.

@rchincha
Copy link
Copy Markdown
Contributor

rchincha commented Jan 23, 2026

@matheuscscp

Here is an outline for adding a kind test
#3729

And invoked as a nightly like so
https://github.com/project-zot/zot/blob/main/.github/workflows/nightly.yaml#L203

This PR should be complete after this (we are close)

@matheuscscp
Copy link
Copy Markdown
Contributor Author

matheuscscp commented Jan 23, 2026

@rchincha Dex is not the right choice for this test. The main use case here is verifying OIDC tokens issued by Kubernetes itself. Kubernetes does this by default, no configuration is needed except for a few minor fine-tunings. Every single ServiceAccount token issued by Kubernetes is, by design, an OIDC token :)

The only heavy lifting we need here is using the CA of the kind cluster inside Zot. Then we just need to spin a Pod inside the cluster with a serviceAccountToken projected volume and some image containing crane or skopeo or oras to perform API calls against Zot using the mounted token :)

Leave this with me 👌

andaaron
andaaron previously approved these changes Jan 24, 2026
@matheuscscp
Copy link
Copy Markdown
Contributor Author

I'm fixing the linter errors

@matheuscscp matheuscscp force-pushed the improve-oidc-wid branch 2 times, most recently from 314ed33 to 5dae0fa Compare January 24, 2026 20:27
@matheuscscp
Copy link
Copy Markdown
Contributor Author

matheuscscp commented Jan 24, 2026

OIDC Workload Identity E2E Test Script

This commit adds a comprehensive e2e test script (examples/kind/kind-oidc-workload-identity.sh) that validates OIDC workload identity federation with Kubernetes ServiceAccount tokens.

Features for Local Iteration

The script supports fast iteration with these flags:

  • --skip-setup - Skip cluster/image/zot setup (reuse existing infrastructure)
  • --only-crane - Run only crane CLI tests (8-14)
  • --only-curl - Run only curl-based tests (1-7)
  • --keep-resources - Keep cluster/zot running after exit for debugging

Test Cases

Test Type Description What it Proves
1 curl Push rejected without token Unauthenticated requests are blocked (HTTP 401)
2 curl Auth succeeds with token Valid SA tokens are accepted (HTTP 200)
3 curl Write operation (blob upload) Authenticated users can write (HTTP 202)
4 curl Repository in catalog Write operations create repositories
5 curl Wrong audience rejected Tokens with wrong audience are rejected (HTTP 401)
6 curl Other SA sees empty catalog Authorization enforced - unauthorized SA sees empty catalog
7 curl Other SA gets 403 on write Authorization enforced - unauthorized SA gets HTTP 403
8 crane Copy image with auth Real OCI push succeeds with valid auth
9 crane Copy image without auth Real OCI push fails without auth (HTTP 401)
10 crane List tags without auth Real OCI read fails without auth (HTTP 401)
11 crane Pull manifest with auth Real OCI read succeeds with valid auth
12 crane Other SA copy fails Authorization enforced - other SA gets HTTP 403
13 crane Other SA list fails Authorization enforced - other SA gets HTTP 403
14 crane No token returns 401 Unauthenticated crane operations return 401, not 403

Key validations:

  • Tests 1-7 use curl to validate basic authentication and authorization
  • Tests 8-14 use the crane CLI (from go-containerregistry) which properly supports the registryToken field in Docker config to send direct Bearer tokens
  • Tests 6-7 and 12-14 validate that authorization is enforced for OIDC bearer auth (via accessControl config)
  • Test 14 ensures proper 401 vs 403 distinction (authentication vs authorization errors)
Raw Test Logs (click to expand)
[INFO] Cleaning up any existing resources...
[INFO] Pre-pulling container images (helps avoid Docker Hub rate limiting)...
[INFO] Pulling curlimages/curl:8.5.0...
8.5.0: Pulling from curlimages/curl
Digest: sha256:08e466006f0860e54fc299378de998935333e0e130a15f6f98482e9f8dab3058
Status: Image is up to date for curlimages/curl:8.5.0
docker.io/curlimages/curl:8.5.0
[INFO] Pulling alpine:3.19...
3.19: Pulling from library/alpine
Digest: sha256:6baf43584bcb78f2e5847d1de515f23499913ac9f12bdf834811a3145eb11ca1
Status: Image is up to date for alpine:3.19
docker.io/library/alpine:3.19
[INFO] Pulling gcr.io/google-containers/busybox:1.27...
1.27: Pulling from google-containers/busybox
aab39f0bc16d: Pulling fs layer
aab39f0bc16d: Verifying Checksum
aab39f0bc16d: Download complete
aab39f0bc16d: Pull complete
Digest: sha256:545e6a6310a27636260920bc07b994a299b6708a1b26910cfefd335fdfb60d2b
Status: Downloaded newer image for gcr.io/google-containers/busybox:1.27
gcr.io/google-containers/busybox:1.27
[INFO] Pulling kindest/node:v1.28.7...
v1.28.7: Pulling from kindest/node
Digest: sha256:9bc6c451a289cf96ad0bbaf33d416901de6fd632415b076ab05f5fa7e4f65c58
Status: Image is up to date for kindest/node:v1.28.7
docker.io/kindest/node:v1.28.7
[INFO] Creating Kind cluster 'kind-oidc-wid'...
Creating cluster "kind-oidc-wid" ...
 • Ensuring node image (kindest/node:v1.28.7) 🖼  ...
 ✓ Ensuring node image (kindest/node:v1.28.7) 🖼
 • Preparing nodes 📦   ...
 ✓ Preparing nodes 📦 
 • Writing configuration 📜  ...
 ✓ Writing configuration 📜
 • Starting control-plane 🕹️  ...
 ✓ Starting control-plane 🕹️
 • Installing CNI 🔌  ...
 ✓ Installing CNI 🔌
 • Installing StorageClass 💾  ...
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind-oidc-wid"
You can now use your cluster with:

kubectl cluster-info --context kind-kind-oidc-wid

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/
[INFO] Waiting for cluster to be ready...
node/kind-oidc-wid-control-plane condition met
[INFO] Loading images into Kind cluster...
[INFO] Loading curlimages/curl:8.5.0...
Image: "curlimages/curl:8.5.0" with ID "sha256:aef55bb249156a28c5c2e1e10510b1191c125b4ef68fc45684b3877234bf643a" not yet present on node "kind-oidc-wid-control-plane", loading...
[INFO] Loading alpine:3.19...
Image: "alpine:3.19" with ID "sha256:83b2b6703a620bf2e001ab57f7adc414d891787b3c59859b1b62909e48dd2242" not yet present on node "kind-oidc-wid-control-plane", loading...
[INFO] Loading gcr.io/google-containers/busybox:1.27...
Image: "gcr.io/google-containers/busybox:1.27" with ID "sha256:54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971" not yet present on node "kind-oidc-wid-control-plane", loading...
[INFO] Control plane container: kind-oidc-wid-control-plane
[INFO] Control plane IP: 172.18.0.2
[INFO] Creating ClusterRoleBinding for OIDC discovery...
clusterrolebinding.rbac.authorization.k8s.io/oidc-reviewer created
[INFO] Exporting Kind cluster CA certificate...
[INFO] Verifying CA certificate...
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2114245328936856036 (0x1d574eecc74f69e4)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: Jan 24 19:50:17 2026 GMT
            Not After : Jan 22 19:55:17 2036 GMT
        Subject: CN = kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:f8:a1:89:b5:da:84:93:11:66:a9:83:b6:88:33:
                    5e:0a:7b:53:86:e7:76:10:dd:b8:ed:ca:ba:a8:19:
                    d7:5f:ab:94:66:ae:24:e3:c5:4d:34:8b:8f:cc:f0:
                    ac:3a:81:8d:58:c8:05:23:b5:a3:09:29:08:76:b7:
                    e7:f6:b6:c7:96:f0:49:2f:20:3f:b3:b4:8d:34:17:
                    79:17:ad:ab:2a:c7:35:3f:cd:fb:b6:27:35:83:73:
[INFO] OIDC Issuer: https://kind-oidc-wid-control-plane:6443
[INFO] Testing OIDC discovery endpoint...
{
  "issuer": "https://kind-oidc-wid-control-plane:6443",
  "jwks_uri": "https://172.18.0.2:6443/openid/v1/jwks",
  "response_types_supported": [
    "id_token"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ]
}
[INFO] Building zot docker image (minimal)... this may take a few minutes
sha256:355ed52596260ac8f3d85f0ed31c7a0047dd50818818a2cc6d2a78c7ca9af698
[INFO] Image built: zot-linux-amd64-minimal:v2.1.13-16-gcab941b4
[INFO] Zot configuration:
{
  "distSpecVersion": "1.1.1",
  "storage": {
    "rootDirectory": "/var/lib/zot"
  },
  "http": {
    "address": "0.0.0.0",
    "port": "5000",
    "auth": {
      "bearer": {
        "realm": "zot",
        "service": "zot-registry",
        "oidc": [
          {
            "issuer": "https://kind-oidc-wid-control-plane:6443",
            "audiences": ["zot-registry"]
          }
        ]
      }
    },
    "accessControl": {
      "repositories": {
        "**": {
          "policies": [
            {
              "users": ["https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:test-workload"],
              "actions": ["read", "create", "update", "delete"]
            }
          ],
          "defaultPolicy": []
        }
      }
    }
  },
  "log": {
    "level": "debug"
  }
}
[INFO] Starting zot container...
828805cfca85bfdc0df73d01781cfaa1a612fde8ee1185f79e1f1cba7fcc8def
[INFO] Waiting for zot to be ready...
[INFO] Zot container logs:
{"time":"2026-01-24T19:57:04.227541907Z","level":"info","message":"JWT bearer authentication","enabled":false,"caller":"zotregistry.dev/zot/v2/pkg/api/controller.go:285","func":"zotregistry.dev/zot/v2/pkg/api.(*Controller).Init","goroutine":1}
{"time":"2026-01-24T19:57:04.227586574Z","level":"info","message":"OIDC bearer authentication","enabled":true,"caller":"zotregistry.dev/zot/v2/pkg/api/controller.go:286","func":"zotregistry.dev/zot/v2/pkg/api.(*Controller).Init","goroutine":1}
{"time":"2026-01-24T19:57:04.227625067Z","level":"info","message":"basic authentication (htpasswd)","enabled":false,"caller":"zotregistry.dev/zot/v2/pkg/api/controller.go:287","func":"zotregistry.dev/zot/v2/pkg/api.(*Controller).Init","goroutine":1}
{"time":"2026-01-24T19:57:04.227657923Z","level":"info","message":"basic authentication (LDAP)","enabled":false,"caller":"zotregistry.dev/zot/v2/pkg/api/controller.go:288","func":"zotregistry.dev/zot/v2/pkg/api.(*Controller).Init","goroutine":1}
{"time":"2026-01-24T19:57:04.227682646Z","level":"info","message":"basic authentication (API key)","enabled":false,"caller":"zotregistry.dev/zot/v2/pkg/api/controller.go:289","func":"zotregistry.dev/zot/v2/pkg/api.(*Controller).Init","goroutine":1}
{"time":"2026-01-24T19:57:04.227704393Z","level":"info","message":"OpenID authentication","enabled":false,"caller":"zotregistry.dev/zot/v2/pkg/api/controller.go:290","func":"zotregistry.dev/zot/v2/pkg/api.(*Controller).Init","goroutine":1}
{"time":"2026-01-24T19:57:04.227726395Z","level":"info","message":"mutual TLS authentication","enabled":false,"caller":"zotregistry.dev/zot/v2/pkg/api/controller.go:291","func":"zotregistry.dev/zot/v2/pkg/api.(*Controller).Init","goroutine":1}
{"time":"2026-01-24T19:57:04.22792761Z","level":"info","message":"runtime params","cpus":14,"max. open files":524287,"listen backlog":"4096","max. inotify watches":"524288","caller":"zotregistry.dev/zot/v2/pkg/api/runtime.go:18","func":"zotregistry.dev/zot/v2/pkg/api.DumpRuntimeParams","goroutine":1}
{"time":"2026-01-24T19:57:04.228052822Z","level":"info","message":"events disabled in configuration","caller":"zotregistry.dev/zot/v2/pkg/extensions/extension_events_disabled.go:16","func":"zotregistry.dev/zot/v2/pkg/extensions.NewEventRecorder","goroutine":1}
{"time":"2026-01-24T19:57:04.22812572Z","level":"warn","message":"lint extension is disabled because given zot binary doesn't include this feature please build a binary that does so","caller":"zotregistry.dev/zot/v2/pkg/extensions/extensions_lint_disabled.go:12","func":"zotregistry.dev/zot/v2/pkg/extensions.GetLinter","goroutine":1}
{"time":"2026-01-24T19:57:04.278817299Z","level":"debug","message":"started","caller":"zotregistry.dev/zot/v2/pkg/common/healthz.go:63","func":"zotregistry.dev/zot/v2/pkg/common.(*Healthz).Started","goroutine":1}
{"time":"2026-01-24T19:57:04.27964134Z","level":"warn","message":"skipping enabling search extension because given zot binary doesn't include this feature,please build a binary that does so","caller":"zotregistry.dev/zot/v2/pkg/extensions/extension_search_disabled.go:31","func":"zotregistry.dev/zot/v2/pkg/extensions.EnableSearchExtension","goroutine":1}
{"time":"2026-01-24T19:57:04.279783173Z","level":"warn","message":"skipping enabling metrics extension because given zot binary doesn't include this feature,please build a binary that does so","caller":"zotregistry.dev/zot/v2/pkg/extensions/extension_metrics_disabled.go:18","func":"zotregistry.dev/zot/v2/pkg/extensions.EnableMetricsExtension","goroutine":1}
{"time":"2026-01-24T19:57:04.279811994Z","level":"warn","message":"skipping enabling scrub extension because given zot binary doesn't include this feature,please build a binary that does so","caller":"zotregistry.dev/zot/v2/pkg/extensions/extension_scrub_disabled.go:16","func":"zotregistry.dev/zot/v2/pkg/extensions.EnableScrubExtension","goroutine":1}
{"time":"2026-01-24T19:57:04.279840317Z","level":"warn","message":"skipping enabling sync extension because given zot binary doesn't include this feature,please build a binary that does so","caller":"zotregistry.dev/zot/v2/pkg/extensions/extension_sync_disabled.go:18","func":"zotregistry.dev/zot/v2/pkg/extensions.EnableSyncExtension","goroutine":1}
{"time":"2026-01-24T19:57:04.279870016Z","level":"warn","message":"skipping adding to the scheduler a generator for updating signatures validity because given binary doesn't include this feature, please build a binary that does so","caller":"zotregistry.dev/zot/v2/pkg/extensions/extension_image_trust_disabled.go:26","func":"zotregistry.dev/zot/v2/pkg/extensions.EnableImageTrustVerification","goroutine":1}
{"time":"2026-01-24T19:57:04.279961685Z","level":"debug","message":"generator is done","component":"scheduler","generator":"GCTaskGenerator","caller":"zotregistry.dev/zot/v2/pkg/scheduler/scheduler.go:478","func":"zotregistry.dev/zot/v2/pkg/scheduler.(*generator).generate","goroutine":56}
{"time":"2026-01-24T19:57:04.281479406Z","level":"info","message":"the OIDC workload identity authentication was enabled","issuers":["https://kind-oidc-wid-control-plane:6443"],"caller":"zotregistry.dev/zot/v2/pkg/api/bearer_oidc.go:68","func":"zotregistry.dev/zot/v2/pkg/api.NewOIDCBearerAuthorizer","goroutine":1}
{"time":"2026-01-24T19:57:04.281685345Z","level":"info","message":"anonymous policy only access control is being enabled","caller":"zotregistry.dev/zot/v2/pkg/api/routes.go:135","func":"zotregistry.dev/zot/v2/pkg/api.(*RouteHandler).SetupRoutes","goroutine":1}
{"time":"2026-01-24T19:57:04.283650377Z","level":"warn","message":"skipping enabling swagger because given zot binary doesn't include this feature, please build a binary that does so","caller":"zotregistry.dev/zot/v2/pkg/debug/swagger/swagger_disabled.go:19","func":"zotregistry.dev/zot/v2/pkg/debug/swagger.SetupSwaggerRoutes","goroutine":1}
{"time":"2026-01-24T19:57:04.283841229Z","level":"warn","message":"skipping enabling graphql playground extension because given zot binary doesn't include this feature, please build a binary that does so","caller":"zotregistry.dev/zot/v2/pkg/debug/gqlplayground/gqlplayground_disabled.go:16","func":"zotregistry.dev/zot/v2/pkg/debug/gqlplayground.SetupGQLPlaygroundRoutes","goroutine":1}
{"time":"2026-01-24T19:57:04.283930225Z","level":"warn","message":"skipping enabling pprof extension because given zot binary doesn't include this feature, please build a binary that does so","caller":"zotregistry.dev/zot/v2/pkg/debug/pprof/pprof_disabled.go:15","func":"zotregistry.dev/zot/v2/pkg/debug/pprof.SetupPprofRoutes","goroutine":1}
{"time":"2026-01-24T19:57:04.284016425Z","level":"warn","message":"skipping setting up search routes because given zot binary doesn't include this feature,please build a binary that does so","caller":"zotregistry.dev/zot/v2/pkg/extensions/extension_search_disabled.go:39","func":"zotregistry.dev/zot/v2/pkg/extensions.SetupSearchRoutes","goroutine":1}
{"time":"2026-01-24T19:57:04.284058836Z","level":"warn","message":"skipping setting up image trust routes because given zot binary doesn't include this feature,please build a binary that does so","caller":"zotregistry.dev/zot/v2/pkg/extensions/extension_image_trust_disabled.go:19","func":"zotregistry.dev/zot/v2/pkg/extensions.SetupImageTrustRoutes","goroutine":1}
{"time":"2026-01-24T19:57:04.284549365Z","level":"warn","message":"skipping setting up mgmt routes because given zot binary doesn't include this feature,please build a binary that does so","caller":"zotregistry.dev/zot/v2/pkg/extensions/extension_mgmt_disabled.go:17","func":"zotregistry.dev/zot/v2/pkg/extensions.SetupMgmtRoutes","goroutine":1}
{"time":"2026-01-24T19:57:04.284628541Z","level":"warn","message":"userprefs extension is disabled because given zot binary doesn't include this feature please build a binary that does so","caller":"zotregistry.dev/zot/v2/pkg/extensions/extension_userprefs_disable.go:20","func":"zotregistry.dev/zot/v2/pkg/extensions.SetupUserPreferencesRoutes","goroutine":1}
{"time":"2026-01-24T19:57:04.284703938Z","level":"warn","message":"skipping setting up ui routes because given zot binary doesn't include this feature,please build a binary that does so","caller":"zotregistry.dev/zot/v2/pkg/extensions/extension_ui_disabled.go:15","func":"zotregistry.dev/zot/v2/pkg/extensions.SetupUIRoutes","goroutine":1}
{"time":"2026-01-24T19:57:04.285335148Z","level":"debug","message":"ready","caller":"zotregistry.dev/zot/v2/pkg/common/healthz.go:58","func":"zotregistry.dev/zot/v2/pkg/common.(*Healthz).Ready","goroutine":1}
{"time":"2026-01-24T19:57:04.380794145Z","level":"info","message":"no repositories found in storage, finished.","component":"dedupe","caller":"zotregistry.dev/zot/v2/pkg/storage/common/common.go:943","func":"zotregistry.dev/zot/v2/pkg/storage/common.(*DedupeTaskGenerator).Next","goroutine":56}
{"time":"2026-01-24T19:57:04.38095684Z","level":"debug","message":"generator is done","component":"scheduler","generator":"DedupeTaskGenerator","caller":"zotregistry.dev/zot/v2/pkg/scheduler/scheduler.go:478","func":"zotregistry.dev/zot/v2/pkg/scheduler.(*generator).generate","goroutine":56}
[INFO] Zot container IP: 172.18.0.3
[INFO] Checking zot health...
[INFO] Zot is responding (HTTP 401)
[INFO] Creating test namespace and ServiceAccount...
namespace/oidc-test created
serviceaccount/test-workload created
[INFO] Creating test Pod with projected ServiceAccount token...
pod/oidc-test-pod created
[INFO] Waiting for test pod to be ready...
pod/oidc-test-pod condition met
[INFO] Verifying projected ServiceAccount token...
[INFO] Token claims (decoded):
{
  "aud": [
    "zot-registry"
  ],
  "exp": 1769288230,
  "iat": 1769284630,
  "iss": "https://kind-oidc-wid-control-plane:6443",
  "kubernetes.io": {
    "namespace": "oidc-test",
    "pod": {
      "name": "oidc-test-pod",
      "uid": "043edcca-cd5f-4cd1-b13a-5a2709bc4b5a"
    },
    "serviceaccount": {
      "name": "test-workload",
      "uid": "03d10bfb-ba62-49cd-a68a-29201ce3fa15"
    }
  },
  "nbf": 1769284630,
  "sub": "system:serviceaccount:oidc-test:test-workload"
}
[INFO] TEST 1: Verifying push (blob upload) fails without token...
[INFO] TEST 1 PASSED: Push correctly rejected without token (HTTP 401)
[INFO] TEST 2: Verifying authentication succeeds with token...
[INFO] TEST 2 PASSED: Authentication succeeded with token (HTTP 200)
[INFO] Response body: {"repositories":[]}
[INFO] TEST 3: Testing write permissions (initiate blob upload)...
[INFO] TEST 3 PASSED: Write operation succeeded (HTTP 202 - upload initiated)
[INFO] TEST 4: Listing catalog to verify repository exists...
[INFO] Catalog: {"repositories":["test-repo"]}
[INFO] TEST 4 PASSED: Repository 'test-repo' found in catalog
[INFO] TEST 5: Verifying wrong audience token fails...
pod/oidc-test-pod-wrong-aud created
[INFO] Waiting for wrong-audience test pod to be ready...
pod/oidc-test-pod-wrong-aud condition met
[INFO] TEST 5 PASSED: Wrong audience token correctly rejected (HTTP 401)
[INFO] TEST 6: Verifying different ServiceAccount authenticates but has NO permissions...
serviceaccount/other-sa created
pod/oidc-test-pod-other-sa created
[INFO] Waiting for other-sa test pod to be ready...
pod/oidc-test-pod-other-sa condition met
[INFO] TEST 6 PASSED: Other ServiceAccount authenticated but has NO permissions (empty catalog)
[INFO]       The username 'https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:other-sa' was extracted from the token.
[INFO]       Authorization is enforced via accessControl config.
[INFO] TEST 7: Verifying other-sa gets 403 Forbidden when trying to write...
[INFO] TEST 7 PASSED: Other ServiceAccount correctly rejected for write (HTTP 403)
[INFO] Creating crane test Pod for e2e artifact operations...
pod/crane-test-pod created
[INFO] Waiting for crane test pod to be ready...
pod/crane-test-pod condition met
[INFO] TEST 8: Copying OCI image using crane WITH auth (should succeed)...
Defaulted container "crane" out of: crane, install-crane (init)
Defaulted container "crane" out of: crane, install-crane (init)
command terminated with exit code 1
[INFO] Image not found in zot, pulling from Docker Hub...
Defaulted container "crane" out of: crane, install-crane (init)
command terminated with exit code 1
[INFO] TEST 8 PASSED: crane copy from Docker Hub succeeded with auth
[INFO] Push output: 2026/01/24 19:57:23 pushed blob: sha256:54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971
2026/01/24 19:57:23 pushed blob: sha256:aab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066
Error: PUT http://zot-oidc-wid:5000/v2/crane-test/manifests/v1: MANIFEST_INVALID: manifest invalid; map[description:During upload, manifests undergo several checks ensuring validity. If those checks fail, this error MAY be returned, unless a more specific error is included. The detail will contain information the failed validation. mediaType:application/vnd.docker.distribution.manifest.v2+json]
[INFO] Verifying pushed image (listing tags)...
Defaulted container "crane" out of: crane, install-crane (init)
[INFO] Tags for crane-test: 
[WARN] Warning: Tag 'v1' not found in crane-test repository
[INFO] TEST 9: Copying OCI image using crane WITHOUT auth (should fail)...
Defaulted container "crane" out of: crane, install-crane (init)
command terminated with exit code 1
[INFO] TEST 9 PASSED: crane copy correctly rejected without auth (401)
[INFO] Output: 2026/01/24 19:57:24 Copying from zot-oidc-wid:5000/crane-test:v1 to zot-oidc-wid:5000/crane-test:v2
Error: fetching "zot-oidc-wid:5000/crane-test:v1": GET http://zot-oidc-wid:5000/v2/crane-test/manifests/v1: unexpected status code 401 Unauthorized: {"code":"UNAUTHORIZED","message":"authentication required","detail":{"description":"The access controller was unable to authenticate the client. Often this will be accompanied by a WWW-Authenticate HTTP response header indicating how to authenticate."}}
[INFO] TEST 10: Listing tags using crane WITHOUT auth (should fail)...
Defaulted container "crane" out of: crane, install-crane (init)
command terminated with exit code 1
[INFO] TEST 10 PASSED: crane ls correctly rejected without auth
[INFO] Output: Error: reading tags for zot-oidc-wid:5000/crane-test: GET http://zot-oidc-wid:5000/v2/crane-test/tags/list?n=1000: unexpected status code 401 Unauthorized: {"code":"UNAUTHORIZED","message":"authentication required","detail":{"description":"The access controller was unable to authenticate the client. Often this will be accompanied by a WWW-Authenticate HTTP response header indicating how to authenticate."}}
[INFO] TEST 11: Pulling manifest using crane WITH auth (should succeed)...
Defaulted container "crane" out of: crane, install-crane (init)
Defaulted container "crane" out of: crane, install-crane (init)
command terminated with exit code 1
[INFO] TEST 11 PASSED: crane manifest succeeded with auth
[INFO] Manifest preview: Error: fetching manifest zot-oidc-wid:5000/crane-test:v1: GET http://zot-oidc-wid:5000/v2/crane-test/manifests/v1: MANIFEST_UNKNOWN: manifest unknown; map[description:This error is returned when the manifest, identified by name and tag is unknown to the repository. reference:v1]
[INFO] Creating crane pod for other-sa...
pod/crane-other-sa-pod created
[INFO] Waiting for crane-other-sa-pod to be ready...
pod/crane-other-sa-pod condition met
[INFO] TEST 12: Copying OCI image using crane with other-sa (should fail with 403)...
Defaulted container "crane" out of: crane, install-crane (init)
Defaulted container "crane" out of: crane, install-crane (init)
command terminated with exit code 1
[INFO] TEST 12 PASSED: crane copy correctly rejected for other-sa (403 Forbidden)
[INFO] Output: 2026/01/24 19:57:30 Copying from zot-oidc-wid:5000/crane-test:v1 to zot-oidc-wid:5000/other-sa-crane-test:v1
Error: fetching "zot-oidc-wid:5000/crane-test:v1": GET http://zot-oidc-wid:5000/v2/crane-test/manifests/v1: DENIED: requested access to the resource is denied; map[description:The access controller denied access for the operation on a resource.]
[INFO] TEST 13: Listing tags using crane with other-sa (should fail with 403)...
Defaulted container "crane" out of: crane, install-crane (init)
command terminated with exit code 1
[INFO] TEST 13 PASSED: crane ls correctly rejected for other-sa (access denied)
[INFO] Output: Error: reading tags for zot-oidc-wid:5000/crane-test: GET http://zot-oidc-wid:5000/v2/crane-test/tags/list?n=1000: DENIED: requested access to the resource is denied; map[description:The access controller denied access for the operation on a resource.]
[INFO] TEST 14: Crane operation with NO token (should fail with 401)...
Defaulted container "crane" out of: crane, install-crane (init)
command terminated with exit code 1
[INFO] TEST 14 PASSED: No token correctly returns 401 Unauthorized
[INFO] Output: Error: reading tags for zot-oidc-wid:5000/crane-test: GET http://zot-oidc-wid:5000/v2/crane-test/tags/list?n=1000: unexpected status code 401 Unauthorized: {"code":"UNAUTHORIZED","message":"authentication required","detail":{"description":"The access controller was unable to authenticate the client. Often this will be accompanied by a WWW-Authenticate HTTP response header indicating how to authenticate."}}
[INFO] Final zot logs:
{"time":"2026-01-24T19:57:23.170386962Z","level":"debug","message":"the OIDC bearer authentication was successful","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:test-workload","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:581","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":139}
{"time":"2026-01-24T19:57:23.170979043Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:40194","method":"PATCH","path":"/v2/crane-test/blobs/uploads/5c062058-2f17-4cab-813d-cebd49c53fbd","statusCode":202,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["1497"],"Content-Type":["application/octet-stream"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":139}
{"time":"2026-01-24T19:57:23.172279867Z","level":"debug","message":"the OIDC token was authenticated","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:test-workload","groups":null,"caller":"zotregistry.dev/zot/v2/pkg/api/bearer_oidc.go:175","func":"zotregistry.dev/zot/v2/pkg/api.(*oidcProvider).authenticate","goroutine":139}
{"time":"2026-01-24T19:57:23.172377762Z","level":"debug","message":"the OIDC bearer authentication was successful","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:test-workload","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:581","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":139}
{"time":"2026-01-24T19:57:23.172921621Z","level":"debug","message":"dedupe begin","src":"/var/lib/zot/crane-test/.uploads/5c062058-2f17-4cab-813d-cebd49c53fbd","dstDigest":"sha256:54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971","dst":"/var/lib/zot/crane-test/blobs/sha256/54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971","caller":"zotregistry.dev/zot/v2/pkg/storage/imagestore/imagestore.go:1154","func":"zotregistry.dev/zot/v2/pkg/storage/imagestore.(*ImageStore).DedupeBlob","goroutine":139}
{"time":"2026-01-24T19:57:23.201105874Z","level":"debug","message":"rename","src":"/var/lib/zot/crane-test/.uploads/5c062058-2f17-4cab-813d-cebd49c53fbd","dst":"/var/lib/zot/crane-test/blobs/sha256/54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971","component":"dedupe","caller":"zotregistry.dev/zot/v2/pkg/storage/imagestore/imagestore.go:1180","func":"zotregistry.dev/zot/v2/pkg/storage/imagestore.(*ImageStore).DedupeBlob","goroutine":139}
{"time":"2026-01-24T19:57:23.201360389Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:40194","method":"PUT","path":"/v2/crane-test/blobs/uploads/5c062058-2f17-4cab-813d-cebd49c53fbd?digest=sha256%3A54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971","statusCode":201,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["0"],"Content-Type":["application/octet-stream"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":139}
{"time":"2026-01-24T19:57:23.250665436Z","level":"debug","message":"the OIDC token was authenticated","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:test-workload","groups":null,"caller":"zotregistry.dev/zot/v2/pkg/api/bearer_oidc.go:175","func":"zotregistry.dev/zot/v2/pkg/api.(*oidcProvider).authenticate","goroutine":139}
{"time":"2026-01-24T19:57:23.250834963Z","level":"debug","message":"the OIDC bearer authentication was successful","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:test-workload","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:581","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":139}
{"time":"2026-01-24T19:57:23.462510508Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:40194","method":"PATCH","path":"/v2/crane-test/blobs/uploads/bb63f383-3dc9-4b13-a46c-f8cab273888e","statusCode":202,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["692336"],"Content-Type":["application/octet-stream"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":139}
{"time":"2026-01-24T19:57:23.463993099Z","level":"debug","message":"the OIDC token was authenticated","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:test-workload","groups":null,"caller":"zotregistry.dev/zot/v2/pkg/api/bearer_oidc.go:175","func":"zotregistry.dev/zot/v2/pkg/api.(*oidcProvider).authenticate","goroutine":139}
{"time":"2026-01-24T19:57:23.464151803Z","level":"debug","message":"the OIDC bearer authentication was successful","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:test-workload","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:581","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":139}
{"time":"2026-01-24T19:57:23.466129274Z","level":"debug","message":"dedupe begin","src":"/var/lib/zot/crane-test/.uploads/bb63f383-3dc9-4b13-a46c-f8cab273888e","dstDigest":"sha256:aab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066","dst":"/var/lib/zot/crane-test/blobs/sha256/aab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066","caller":"zotregistry.dev/zot/v2/pkg/storage/imagestore/imagestore.go:1154","func":"zotregistry.dev/zot/v2/pkg/storage/imagestore.(*ImageStore).DedupeBlob","goroutine":139}
{"time":"2026-01-24T19:57:23.485949664Z","level":"debug","message":"rename","src":"/var/lib/zot/crane-test/.uploads/bb63f383-3dc9-4b13-a46c-f8cab273888e","dst":"/var/lib/zot/crane-test/blobs/sha256/aab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066","component":"dedupe","caller":"zotregistry.dev/zot/v2/pkg/storage/imagestore/imagestore.go:1180","func":"zotregistry.dev/zot/v2/pkg/storage/imagestore.(*ImageStore).DedupeBlob","goroutine":139}
{"time":"2026-01-24T19:57:23.486245395Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:40194","method":"PUT","path":"/v2/crane-test/blobs/uploads/bb63f383-3dc9-4b13-a46c-f8cab273888e?digest=sha256%3Aaab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066","statusCode":201,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["0"],"Content-Type":["application/octet-stream"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":139}
{"time":"2026-01-24T19:57:23.488009528Z","level":"debug","message":"the OIDC token was authenticated","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:test-workload","groups":null,"caller":"zotregistry.dev/zot/v2/pkg/api/bearer_oidc.go:175","func":"zotregistry.dev/zot/v2/pkg/api.(*oidcProvider).authenticate","goroutine":139}
{"time":"2026-01-24T19:57:23.488144948Z","level":"debug","message":"the OIDC bearer authentication was successful","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:test-workload","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:581","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":139}
{"time":"2026-01-24T19:57:23.488488032Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:40194","method":"PUT","path":"/v2/crane-test/manifests/v1","statusCode":415,"latency":"0s","bodySize":381,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["527"],"Content-Type":["application/vnd.docker.distribution.manifest.v2+json"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":139}
{"time":"2026-01-24T19:57:23.726465896Z","level":"debug","message":"no bearer token provided","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":144}
{"time":"2026-01-24T19:57:23.726691627Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:40226","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":144}
{"time":"2026-01-24T19:57:23.727635982Z","level":"debug","message":"the OIDC token was authenticated","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:test-workload","groups":null,"caller":"zotregistry.dev/zot/v2/pkg/api/bearer_oidc.go:175","func":"zotregistry.dev/zot/v2/pkg/api.(*oidcProvider).authenticate","goroutine":144}
{"time":"2026-01-24T19:57:23.727759188Z","level":"debug","message":"the OIDC bearer authentication was successful","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:test-workload","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:581","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":144}
{"time":"2026-01-24T19:57:23.728131205Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:40226","method":"GET","path":"/v2/crane-test/tags/list?n=1000","statusCode":200,"latency":"0s","bodySize":31,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":144}
{"time":"2026-01-24T19:57:24.165618685Z","level":"debug","message":"no bearer token provided","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":169}
{"time":"2026-01-24T19:57:24.165829692Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:40234","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":169}
{"time":"2026-01-24T19:57:24.166610209Z","level":"debug","message":"no bearer token provided","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":169}
{"time":"2026-01-24T19:57:24.166767793Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:40234","method":"GET","path":"/v2/crane-test/manifests/v1","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json"],"Accept-Encoding":["gzip"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":169}
{"time":"2026-01-24T19:57:24.387661138Z","level":"debug","message":"no bearer token provided","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":193}
{"time":"2026-01-24T19:57:24.387870962Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:40260","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":193}
{"time":"2026-01-24T19:57:24.388467664Z","level":"debug","message":"no bearer token provided","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":193}
{"time":"2026-01-24T19:57:24.388605013Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:40260","method":"GET","path":"/v2/crane-test/tags/list?n=1000","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":193}
{"time":"2026-01-24T19:57:24.792258986Z","level":"debug","message":"no bearer token provided","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":197}
{"time":"2026-01-24T19:57:24.792458686Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:40286","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":197}
{"time":"2026-01-24T19:57:24.793264215Z","level":"debug","message":"the OIDC token was authenticated","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:test-workload","groups":null,"caller":"zotregistry.dev/zot/v2/pkg/api/bearer_oidc.go:175","func":"zotregistry.dev/zot/v2/pkg/api.(*oidcProvider).authenticate","goroutine":197}
{"time":"2026-01-24T19:57:24.793351019Z","level":"debug","message":"the OIDC bearer authentication was successful","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:test-workload","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:581","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":197}
{"time":"2026-01-24T19:57:24.79362339Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:40286","method":"GET","path":"/v2/crane-test/manifests/v1","statusCode":404,"latency":"0s","bodySize":212,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json"],"Accept-Encoding":["gzip"],"Authorization":["******"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":197}
{"time":"2026-01-24T19:57:30.611954406Z","level":"debug","message":"no bearer token provided","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":201}
{"time":"2026-01-24T19:57:30.612181498Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:43976","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":201}
{"time":"2026-01-24T19:57:30.613308276Z","level":"debug","message":"the OIDC token was authenticated","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:other-sa","groups":null,"caller":"zotregistry.dev/zot/v2/pkg/api/bearer_oidc.go:175","func":"zotregistry.dev/zot/v2/pkg/api.(*oidcProvider).authenticate","goroutine":201}
{"time":"2026-01-24T19:57:30.613449171Z","level":"debug","message":"the OIDC bearer authentication was successful","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:other-sa","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:581","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":201}
{"time":"2026-01-24T19:57:30.613652768Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:43976","method":"GET","path":"/v2/crane-test/manifests/v1","statusCode":403,"latency":"0s","bodySize":181,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json"],"Accept-Encoding":["gzip"],"Authorization":["******"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":201}
{"time":"2026-01-24T19:57:30.832822912Z","level":"debug","message":"no bearer token provided","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":171}
{"time":"2026-01-24T19:57:30.833032661Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:43996","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":171}
{"time":"2026-01-24T19:57:30.83417654Z","level":"debug","message":"the OIDC token was authenticated","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:other-sa","groups":null,"caller":"zotregistry.dev/zot/v2/pkg/api/bearer_oidc.go:175","func":"zotregistry.dev/zot/v2/pkg/api.(*oidcProvider).authenticate","goroutine":171}
{"time":"2026-01-24T19:57:30.834298459Z","level":"debug","message":"the OIDC bearer authentication was successful","username":"https://kind-oidc-wid-control-plane:6443/system:serviceaccount:oidc-test:other-sa","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:581","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":171}
{"time":"2026-01-24T19:57:30.834483433Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:43996","method":"GET","path":"/v2/crane-test/tags/list?n=1000","statusCode":403,"latency":"0s","bodySize":181,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":171}
{"time":"2026-01-24T19:57:31.273187984Z","level":"debug","message":"no bearer token provided","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":175}
{"time":"2026-01-24T19:57:31.273448336Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:44000","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":175}
{"time":"2026-01-24T19:57:31.274053029Z","level":"debug","message":"no bearer token provided","caller":"zotregistry.dev/zot/v2/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func1.1","goroutine":175}
{"time":"2026-01-24T19:57:31.274175316Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"172.18.0.2:44000","method":"GET","path":"/v2/crane-test/tags/list?n=1000","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["crane/0.20.2 go-containerregistry/0.20.2"]},"caller":"zotregistry.dev/zot/v2/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":175}
[INFO] ==========================================
[INFO] All OIDC Workload Identity tests PASSED!
[INFO] ==========================================
[INFO] 
[INFO] Iteration tips:
[INFO]   --skip-setup     Skip cluster/image/zot setup (reuse existing)
[INFO]   --only-crane     Run only crane tests (8-14)
[INFO]   --only-curl      Run only curl tests (1-7)
[INFO]   --keep-resources Keep cluster/zot running after exit
[INFO] Cleaning up...
zot-oidc-wid

Signed-off-by: Matheus Pimenta <matheuscscp@gmail.com>
Signed-off-by: Matheus Pimenta <matheuscscp@gmail.com>
Signed-off-by: Matheus Pimenta <matheuscscp@gmail.com>
Signed-off-by: Matheus Pimenta <matheuscscp@gmail.com>
Signed-off-by: Matheus Pimenta <matheuscscp@gmail.com>
Signed-off-by: Matheus Pimenta <matheuscscp@gmail.com>
@matheuscscp
Copy link
Copy Markdown
Contributor Author

@andaaron Hopefully this was the last push 🙏

@matheuscscp
Copy link
Copy Markdown
Contributor Author

@andaaron Can you please retrigger the 3 test jobs? I think they are unrelated since only the golangci-lint was red before my push and all I did was gofmt a file (included an empty line, see here: https://github.com/project-zot/zot/compare/bb4451ffa1edf4141a522dabbe796b9e6e732f17..9e2067295fd285c7e1438e2a171db6a70a5ecebe)

@matheuscscp
Copy link
Copy Markdown
Contributor Author

@andaaron Looks like everything except for binary size increase is green now 🙌 🟢

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request introduces support for OIDC (OpenID Connect) workload identity federation in Zot, enabling secret-less authentication for automated workflows such as Kubernetes pods and CI/CD pipelines. The implementation allows workloads to authenticate using OIDC ID tokens instead of static credentials.

Changes:

  • Added CEL (Common Expression Language) support for flexible claim validation and mapping from OIDC tokens to Zot identities
  • Implemented OIDC bearer authentication with lazy provider loading, multi-issuer support, and audience validation
  • Integrated OIDC authentication into existing bearer auth flow with fallback to traditional bearer tokens
  • Added comprehensive test coverage including unit tests, integration tests, and end-to-end Kind cluster tests

Reviewed changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated no comments.

Show a summary per file
File Description
pkg/cel/expression.go CEL expression parser and evaluator for claim processing
pkg/cel/expression_test.go Comprehensive CEL expression tests
pkg/cel/claim_processor.go OIDC claim processor with validation and mapping logic
pkg/cel/claim_processor_test.go Extensive claim processor tests covering various scenarios
pkg/api/bearer_oidc.go OIDC bearer authorizer implementation with lazy provider loading
pkg/api/bearer_oidc_test.go Unit tests for OIDC bearer authentication
pkg/api/authn.go Updated bearer auth handler to support both OIDC and traditional tokens
pkg/api/authn_test.go Integration tests for OIDC workload identity
pkg/api/authz.go Added BEARER_OIDC constant for authorization differentiation
pkg/api/bearer.go Changed return type of NewBearerAuthorizer to pointer
pkg/api/config/config.go New config structures for OIDC bearer authentication
pkg/api/config/config_test.go Config validation tests
pkg/api/controller.go Updated logging to distinguish JWT vs OIDC bearer auth
pkg/cli/server/root.go Updated auth validation to include bearer auth
pkg/cli/server/root_test.go Updated tests for split bearer auth logging
pkg/cli/server/config_reloader_test.go Updated tests for split bearer auth logging
errors/errors.go New error definitions for OIDC authentication
examples/config-bearer-oidc-workload.json Example OIDC configuration
examples/README-OIDC-WORKLOAD-IDENTITY.md Comprehensive documentation
examples/kind/kind-oidc-workload-identity.sh End-to-end test script with Kind
go.mod/go.sum Added CEL and OIDC dependencies
Makefile Removed modcheck from build targets, added to CI workflow
.github/workflows/nightly.yaml Added OIDC workload identity E2E test job
.github/workflows/golangci-lint.yaml Added modcheck step to CI

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@rchincha rchincha merged commit bf619c5 into project-zot:main Jan 25, 2026
53 of 57 checks passed
@rchincha
Copy link
Copy Markdown
Contributor

@matheuscscp merged, we will cut a new release with this fix

@matheuscscp matheuscscp deleted the improve-oidc-wid branch January 25, 2026 07:42
@matheuscscp
Copy link
Copy Markdown
Contributor Author

@matheuscscp merged, we will cut a new release with this fix

Well it's not a fix, it's a big new feature 😅

And you can count me to help maintain it moving forward 👍 Thank you both very much for all the guidance and patience here! @rchincha @andaaron

@rchincha
Copy link
Copy Markdown
Contributor

| Well it's not a fix, it's a big new feature 😅

Indeed! v2.1.14 has been released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feat]: Support OIDC authentication for workloads

4 participants