Skip to content

fix: resolve Git Anonymous Resolver excessive memory usage #8677

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tekton-robot merged 2 commits into from
Apr 28, 2025
Merged

fix: resolve Git Anonymous Resolver excessive memory usage #8677

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cf5c211
fix: resolve Git Anonymous Resolver excessive memory usage
aThorp96 Mar 27, 2025
1b28bc8
chore: remove go-git dependency
aThorp96 Apr 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 2 additions & 0 deletions .ko.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
baseImageOverrides:
github.com/tektoncd/pipeline/cmd/resolvers: cgr.dev/chainguard/git@sha256:566235a8ef752f37d285042ee05fc37dbb04293e50f116a231984080fb835693
Copy link
Member

@chmouel chmouel Apr 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this would make all resolvers image having to use the git image... isnt that a bit much for example the http-resolver would def not need it...

Copy link
Member Author

@aThorp96 aThorp96 Apr 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly all of the resolvers run as plugins together in the same pod(s). So at least while the some resolvers will have unnecessary access to the git binary, all of the resolver pods will need access to the binary if the git resolver is enabled. Not sure if that's that much better though

Copy link
Member

@vdemeester vdemeester Apr 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there might be some changes to be done in the tekton/publish.yaml task as well.

Copy link
Member Author

@aThorp96 aThorp96 Apr 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to modifying the task's inline .koconfig like is already done?

7 changes: 7 additions & 0 deletions config/resolvers/resolvers-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,9 @@ spec:
value: "https://artifacthub.io/"
- name: TEKTON_HUB_API
value: "https://api.hub.tekton.dev/"
volumeMounts:
- name: tmp-clone-volume
mountPath: "/tmp"
Comment on lines +109 to +111
Copy link
Member Author

@aThorp96 aThorp96 Apr 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mounting a directory at /tmp is necessary because the security context sets the root filesystem as read-only, and since we're using git clone we have to clone into the filesystem.

Additionally, using an empty-dir volume as the /tmp directory allows us to configure the size of the directory in alignment with the pod's memory requests.

securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
Expand All @@ -115,3 +118,7 @@ spec:
- "ALL"
seccompProfile:
type: RuntimeDefault
volumes:
- name: tmp-clone-volume
emptyDir:
sizeLimit: 4Gi
9 changes: 9 additions & 0 deletions docs/git-resolver.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,15 @@ The differences between the two modes are:
### Git Clone with git clone

Git clone with `git clone` is supported for anonymous and authenticated cloning.
This mode shallow clones the git repo before fetching and checking out the
provided revision.

**Note**: if the revision is a commit SHA which is not pointed-at by a Branch
or Tag ref, the revision might not be able to be fetched, depending on the
git provider's [uploadpack.allowReachableSHA1InWant](https://git-scm.com/docs/protocol-capabilities#_allow_reachable_sha1_in_want)
setting. This is not an issue for major git providers such as Github and
Gitlab, but may be of note for smaller or self-hosted providers such as
Gitea.

#### Task Resolution

Expand Down
18 changes: 0 additions & 18 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ require (
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/ahmetb/gen-crd-api-reference-docs v0.3.1-0.20220720053627-e327d0730470 // Waiting for https://github.com/ahmetb/gen-crd-api-reference-docs/pull/43/files to merge
github.com/cloudevents/sdk-go/v2 v2.15.2
github.com/go-git/go-git/v5 v5.13.2
github.com/google/go-cmp v0.7.0
github.com/google/go-containerregistry v0.20.2
github.com/google/uuid v1.6.0
Expand Down Expand Up @@ -54,18 +53,6 @@ require (
k8s.io/utils v0.0.0-20241210054802-24370beab758
)

require (
github.com/ProtonMail/go-crypto v1.1.5 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.6.2
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
github.com/kevinburke/ssh_config v1.2.0 // indirect
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
)

require (
cel.dev/expr v0.19.1 // indirect
cloud.google.com/go v0.116.0 // indirect
Expand All @@ -75,7 +62,6 @@ require (
cloud.google.com/go/iam v1.2.2 // indirect
cloud.google.com/go/kms v1.20.5 // indirect
cloud.google.com/go/longrunning v0.6.2 // indirect
dario.cat/mergo v1.0.0 // indirect
fortio.org/safecast v1.0.0 // indirect
github.com/42wim/httpsig v1.2.1 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.17.0 // indirect
Expand All @@ -89,8 +75,6 @@ require (
github.com/aws/aws-sdk-go-v2/service/kms v1.37.13 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.10 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cloudflare/circl v1.3.7 // indirect
github.com/cyphar/filepath-securejoin v0.3.6 // indirect
github.com/davidmz/go-pageant v1.0.2 // indirect
github.com/emicklei/go-restful/v3 v3.12.1 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
Expand Down Expand Up @@ -120,12 +104,10 @@ require (
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f // indirect
github.com/pjbgf/sha1cd v0.3.2 // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/ryanuber/go-glob v1.0.0 // indirect
github.com/secure-systems-lab/go-securesystemslib v0.9.0 // indirect
github.com/sigstore/protobuf-specs v0.4.0 // indirect
github.com/skeema/knownhosts v1.3.0 // indirect
github.com/stoewer/go-strcase v1.3.0 // indirect
github.com/titanous/rocacheck v0.0.0-20171023193734-afe73141d399 // indirect
github.com/x448/float16 v0.8.4 // indirect
Expand Down
38 changes: 0 additions & 38 deletions go.sum
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading