fix: complex symlink performance issue #411

kzantow · 2025-06-03T14:11:36Z

This PR adjust how symlinks are matched against glob patterns such that we will now only return the first matching path for an index entry. This often requires no symlink resolution at all, for example when glob patterns match the real path. This solves a problem that is beginning to be apparent with complex webs of symlinks such as large pnpm projects and snaps that would result in an explosion of paths returned to be tested against a glob pattern causing excessive memory usage and processing time, which effectively would cause syft to hang.

A HUGE THANK YOU to @rezmoss for his work to identify this issue and provide a potential fix! You have my deepest gratitude, sir! ❤️

Signed-off-by: Keith Zantow <[email protected]>

github-actions · 2025-06-03T14:13:35Z

Benchmark Test Results

Benchmark results from the latest changes vs base branch

make .tool/task
make[1]: Entering directory '/home/runner/work/stereoscope/stereoscope'
make[1]: Leaving directory '/home/runner/work/stereoscope/stereoscope'
.tool/task show-benchstat
?   	github.com/anchore/stereoscope	[no test files]
?   	github.com/anchore/stereoscope/examples	[no test files]
PASS
ok  	github.com/anchore/stereoscope/internal	0.003s
?   	github.com/anchore/stereoscope/internal/bus	[no test files]
PASS
ok  	github.com/anchore/stereoscope/internal/containerd	0.007s
PASS
ok  	github.com/anchore/stereoscope/internal/docker	0.004s
?   	github.com/anchore/stereoscope/internal/log	[no test files]
PASS
ok  	github.com/anchore/stereoscope/internal/podman	0.004s
?   	github.com/anchore/stereoscope/pkg/event	[no test files]
?   	github.com/anchore/stereoscope/pkg/event/parsers	[no test files]
goos: linux
goarch: amd64
pkg: github.com/anchore/stereoscope/pkg/file
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkTarIndex-4   	   35634	     33235 ns/op	    5699 B/op	      93 allocs/op
BenchmarkTarIndex-4   	   35823	     33626 ns/op	    5698 B/op	      93 allocs/op
BenchmarkTarIndex-4   	   35853	     33402 ns/op	    5699 B/op	      93 allocs/op
BenchmarkTarIndex-4   	   35865	     34355 ns/op	    5696 B/op	      93 allocs/op
BenchmarkTarIndex-4   	   35893	     33370 ns/op	    5699 B/op	      93 allocs/op
BenchmarkTarIndex-4   	   35616	     33386 ns/op	    5698 B/op	      93 allocs/op
BenchmarkTarIndex-4   	   35868	     33499 ns/op	    5698 B/op	      93 allocs/op
PASS
ok  	github.com/anchore/stereoscope/pkg/file	10.796s
PASS
ok  	github.com/anchore/stereoscope/pkg/filetree	0.005s
?   	github.com/anchore/stereoscope/pkg/filetree/filenode	[no test files]
PASS
ok  	github.com/anchore/stereoscope/pkg/image	0.005s
PASS
ok  	github.com/anchore/stereoscope/pkg/image/containerd	0.007s
PASS
ok  	github.com/anchore/stereoscope/pkg/image/docker	0.004s
PASS
ok  	github.com/anchore/stereoscope/pkg/image/oci	0.005s
PASS
ok  	github.com/anchore/stereoscope/pkg/image/oci/credhelpers	0.005s
?   	github.com/anchore/stereoscope/pkg/image/podman	[no test files]
PASS
ok  	github.com/anchore/stereoscope/pkg/image/sif	0.003s
?   	github.com/anchore/stereoscope/pkg/imagetest	[no test files]
PASS
ok  	github.com/anchore/stereoscope/pkg/tree	0.003s
PASS
ok  	github.com/anchore/stereoscope/pkg/tree/node	0.003s
goos: linux
goarch: amd64
pkg: github.com/anchore/stereoscope/test/integration
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkSimpleImage_GetImage/docker-archive-4 	    1158	   1027990 ns/op	  275707 B/op	    2249 allocs/op
BenchmarkSimpleImage_GetImage/docker-archive-4 	    1155	   1040647 ns/op	  275371 B/op	    2248 allocs/op
BenchmarkSimpleImage_GetImage/docker-archive-4 	    1167	   1016196 ns/op	  275260 B/op	    2248 allocs/op
BenchmarkSimpleImage_GetImage/docker-archive-4 	    1152	   1100004 ns/op	  275128 B/op	    2248 allocs/op
BenchmarkSimpleImage_GetImage/docker-archive-4 	    1102	   1049888 ns/op	  275005 B/op	    2248 allocs/op
BenchmarkSimpleImage_GetImage/docker-archive-4 	    1076	   1015610 ns/op	  274975 B/op	    2248 allocs/op
BenchmarkSimpleImage_GetImage/docker-archive-4 	    1166	   1013405 ns/op	  274832 B/op	    2248 allocs/op
--- FAIL: BenchmarkSimpleImage_GetImage/podman
    fixture_image_simple_test.go:175: could not get fixture image: unable to detect input for 'stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7', errs: podman not available: no host address
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 345B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load build context
#3 transferring context: 209B done
#3 DONE 0.0s

#4 [2/3] ADD file-2.txt /somefile-2.txt
#4 CACHED

#5 [1/3] ADD file-1.txt /somefile-1.txt
#5 CACHED

#6 [3/3] ADD target /
#6 CACHED

#7 exporting to image
#7 exporting layers done
#7 writing image sha256:75b202a79b9c32700250c2b35e6cbc118b96133e28706f2e204b2d516977cd82 done
#7 naming to docker.io/library/stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 done
#7 naming to docker.io/library/stereoscope-fixture-image-simple:latest done
#7 DONE 0.0s
ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied"
--- FAIL: BenchmarkSimpleImage_GetImage
    image_fixtures.go:193: using existing image tar: 'test-fixtures/cache/stereoscope-fixture-image-simple-04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7.tar' (size: 22528, modified: 2025-06-03 23:47:15.05687097 +0000 UTC, mode: -rw-r--r--)
    image_fixtures.go:241: Build docker image: name="stereoscope-fixture-image-simple" tag="04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7"
    image_fixtures.go:291: saveImage running: docker image save stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7
    image_fixtures.go:286: 
        	Error Trace:	/home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:286
        	            				/home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:162
        	            				/home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:152
        	            				/home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:33
        	            				/home/runner/work/stereoscope/stereoscope/test/integration/fixture_image_simple_test.go:163
        	Error:      	Received unexpected error:
        	            	exit status 1
        	Test:       	BenchmarkSimpleImage_GetImage
        	Messages:   	could not import docker image to containerd (shell out)
BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4         	   59950	     19950 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4         	   59680	     19935 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4         	   59325	     19954 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4         	   59970	     20009 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4         	   60058	     20146 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4         	   59522	     19998 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4         	   59396	     19895 ns/op	    2712 B/op	      21 allocs/op
--- FAIL: BenchmarkSimpleImage_FetchSquashedContents
    image_fixtures.go:193: using existing image tar: 'test-fixtures/cache/stereoscope-fixture-image-simple-04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7.tar' (size: 22528, modified: 2025-06-03 23:47:15.05687097 +0000 UTC, mode: -rw-r--r--)
    image_fixtures.go:75: error getting fixture image: 'podman' 'image-simple' with request 'podman:stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7': unable to detect input for 'stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7', errs: podman not available: no host address
FAIL
exit status 1
FAIL	github.com/anchore/stereoscope/test/integration	19.518s
?   	github.com/anchore/stereoscope/test/integration/test-fixtures/registry	[no test files]
FAIL
goos: linux
goarch: amd64
pkg: github.com/anchore/stereoscope/pkg/file
cpu: AMD EPYC 7763 64-Core Processor                
ctr: 
           │ .tmp/benchmark-ffe2d2f.txt │
           │           sec/op           │
TarIndex-4                  33.40µ ± 3%

           │ .tmp/benchmark-ffe2d2f.txt │
           │            B/op            │
TarIndex-4                 5.564Ki ± 0%

           │ .tmp/benchmark-ffe2d2f.txt │
           │         allocs/op          │
TarIndex-4                   93.00 ± 0%

pkg: github.com/anchore/stereoscope/test/integration
                                      │ .tmp/benchmark-ffe2d2f.txt │
                                      │           sec/op           │
SimpleImage_GetImage/docker-archive-4                  1.028m ± 7%

                                      │ .tmp/benchmark-ffe2d2f.txt │
                                      │            B/op            │
SimpleImage_GetImage/docker-archive-4                 268.7Ki ± 0%

                                      │ .tmp/benchmark-ffe2d2f.txt │
                                      │         allocs/op          │
SimpleImage_GetImage/docker-archive-4                  2.248k ± 0%

ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied"
                                                   │ .tmp/benchmark-ffe2d2f.txt │
                                                   │           sec/op           │
SimpleImage_FetchSquashedContents/docker-archive-4                  19.95µ ± 1%

                                                   │ .tmp/benchmark-ffe2d2f.txt │
                                                   │            B/op            │
SimpleImage_FetchSquashedContents/docker-archive-4                 2.648Ki ± 0%

                                                   │ .tmp/benchmark-ffe2d2f.txt │
                                                   │         allocs/op          │
SimpleImage_FetchSquashedContents/docker-archive-4                   21.00 ± 0%
goos: linux
goarch: amd64
pkg: github.com/anchore/stereoscope/pkg/file
cpu: AMD EPYC 7763 64-Core Processor                
ctr: 
           │ .tmp/benchmark-ffe2d2f.txt │
           │           sec/op           │
TarIndex-4                  33.40µ ± 3%

           │ .tmp/benchmark-ffe2d2f.txt │
           │            B/op            │
TarIndex-4                 5.564Ki ± 0%

           │ .tmp/benchmark-ffe2d2f.txt │
           │         allocs/op          │
TarIndex-4                   93.00 ± 0%

pkg: github.com/anchore/stereoscope/test/integration
                                      │ .tmp/benchmark-ffe2d2f.txt │
                                      │           sec/op           │
SimpleImage_GetImage/docker-archive-4                  1.028m ± 7%

                                      │ .tmp/benchmark-ffe2d2f.txt │
                                      │            B/op            │
SimpleImage_GetImage/docker-archive-4                 268.7Ki ± 0%

                                      │ .tmp/benchmark-ffe2d2f.txt │
                                      │         allocs/op          │
SimpleImage_GetImage/docker-archive-4                  2.248k ± 0%

ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied"
                                                   │ .tmp/benchmark-ffe2d2f.txt │
                                                   │           sec/op           │
SimpleImage_FetchSquashedContents/docker-archive-4                  19.95µ ± 1%

                                                   │ .tmp/benchmark-ffe2d2f.txt │
                                                   │            B/op            │
SimpleImage_FetchSquashedContents/docker-archive-4                 2.648Ki ± 0%

                                                   │ .tmp/benchmark-ffe2d2f.txt │
                                                   │         allocs/op          │
SimpleImage_FetchSquashedContents/docker-archive-4                   21.00 ± 0%

Signed-off-by: Keith Zantow <[email protected]>

kzantow · 2025-06-03T23:38:00Z

pkg/filetree/search.go

 	}

-	refs, err := sc.referencesInTree(fileEntries)
+	refs, err := sc.firstMatchingReferences("**/*", fileEntries)


There was a problem with the alternate change -- it was returning too many results for different layers. I'm happy to figure that out, but this change also solves the performance issue, albeit with some unnecessary glob matching.

that's alright, the string glob match is fairly cheap when compared to other operations happening here

Signed-off-by: Keith Zantow <[email protected]>

fix: complex symlink performance issue

e4a262c

Signed-off-by: Keith Zantow <[email protected]>

kzantow added 4 commits June 3, 2025 10:24

fix: skip nil nodeAccess results

b81729f

Signed-off-by: Keith Zantow <[email protected]>

chore: tweak glob normalization to match abs paths

2bca555

Signed-off-by: Keith Zantow <[email protected]>

fix: SearchByMIMEType further optimization of firstMatching lookups

59ba487

Signed-off-by: Keith Zantow <[email protected]>

chore: stray commented line

99c778b

Signed-off-by: Keith Zantow <[email protected]>

kzantow commented Jun 3, 2025

View reviewed changes

chore: update nextSegment naming/tests

caa21ad

Signed-off-by: Keith Zantow <[email protected]>

wagoodman approved these changes Jun 4, 2025

View reviewed changes

kzantow mentioned this pull request Jun 4, 2025

Prevent path explosion in file traversal with complex symlinks #354

Closed

kzantow merged commit 344e29f into main Jun 4, 2025
7 checks passed

kzantow deleted the fix/symlink-performance branch June 4, 2025 13:23

kzantow added the bug Something isn't working label Jun 4, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: complex symlink performance issue #411

fix: complex symlink performance issue #411

Uh oh!

kzantow commented Jun 3, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jun 3, 2025 •

edited

Loading

Uh oh!

kzantow Jun 3, 2025

Uh oh!

wagoodman Jun 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fix: complex symlink performance issue #411

fix: complex symlink performance issue #411

Uh oh!

Conversation

kzantow commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark Test Results

Uh oh!

kzantow Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

wagoodman Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kzantow commented Jun 3, 2025 •

edited

Loading

github-actions bot commented Jun 3, 2025 •

edited

Loading