[Custom Agent Deployer] Segmentation Fault

[agithomas]

/home/agikthomas/go/src/github.com/elastic/elastic-package/elastic-package test system -v
2022/10/14 14:04:03 DEBUG Enable verbose logging
2022/10/14 14:04:03 DEBUG Distribution built without a version tag, can't determine release chronology. Please consider using official releases at https://github.com/elastic/elastic-package/releases
Run system tests for the package
2022/10/14 14:04:03 DEBUG running test with configuration 'memory'
2022/10/14 14:04:03 DEBUG setting up service...
2022/10/14 14:04:03 DEBUG setting up service using Docker Compose service deployer
2022/10/14 14:04:03 DEBUG GET http://127.0.0.1:5601/api/status
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x2eda9fc]

goroutine 1 [running]:
github.com/elastic/elastic-package/internal/testrunner/runners/system.(*runner).runTest(0xc000272cf0, 0xc0007ea780, {{0xc000064045, 0x6}, {0x0, 0x0}, {0x0, 0x0, 0x0}, 0x0, ...}, ...)
        /home/agikthomas/go/src/github.com/elastic/elastic-package/internal/testrunner/runners/system/runner.go:311 +0x3dc
github.com/elastic/elastic-package/internal/testrunner/runners/system.(*runner).runTestPerVariant(0xc000272cf0, 0xc0000aa120?, 0x396ddc0?, {0xc00014021c, 0x16}, {0xc000067110?, 0x0?}, {0x0, 0x0})
        /home/agikthomas/go/src/github.com/elastic/elastic-package/internal/testrunner/runners/system/runner.go:206 +0x55c
github.com/elastic/elastic-package/internal/testrunner/runners/system.(*runner).run(0xc000272cf0)
        /home/agikthomas/go/src/github.com/elastic/elastic-package/internal/testrunner/runners/system/runner.go:176 +0x553
github.com/elastic/elastic-package/internal/testrunner/runners/system.(*runner).Run(0x0?, {{{0xc000067110, 0x6b}, {0xc000064045, 0x6}, {0xc000067164, 0x6}}, {0xc000064004, 0x47}, 0x0, ...})
        /home/agikthomas/go/src/github.com/elastic/elastic-package/internal/testrunner/runners/system/runner.go:87 +0x65
github.com/elastic/elastic-package/internal/testrunner.Run({0x3584af2, 0x6}, {{{0xc000067110, 0x6b}, {0xc000064045, 0x6}, {0xc000067164, 0x6}}, {0xc000064004, 0x47}, ...})
        /home/agikthomas/go/src/github.com/elastic/elastic-package/internal/testrunner/testrunner.go:250 +0x99
github.com/elastic/elastic-package/cmd.testTypeCommandActionFactory.func1(0xc0007e4300?, {0xc0007d0820?, 0x0?, 0x1?})
        /home/agikthomas/go/src/github.com/elastic/elastic-package/cmd/testrunner.go:195 +0xf08
github.com/spf13/cobra.(*Command).execute(0xc0007e4300, {0xc0007d0800, 0x1, 0x1})
        /home/agikthomas/.gvm/pkgsets/go1.19/global/pkg/mod/github.com/spf13/cobra@v1.6.0/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc0007e5500)
        /home/agikthomas/.gvm/pkgsets/go1.19/global/pkg/mod/github.com/spf13/cobra@v1.6.0/command.go:1040 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        /home/agikthomas/.gvm/pkgsets/go1.19/global/pkg/mod/github.com/spf13/cobra@v1.6.0/command.go:968
main.main()
        /home/agikthomas/go/src/github.com/elastic/elastic-package/main.go:29 +0xe5

How to reproduce the issue

• Checkout https://github.com/elastic/integrations
• gh pr checkout 4461
• go to ~/go/src/github.com/elastic/integrations/packages/oracle
• Download the latest version of elastic-package
• run elastic-package test system -d memory -v

Free feel to delete the file oracle/data_stream/performance/_dev/deploy/agent/Dockerfile if needed.

[agithomas]

I attempted to find out the root cause of the above mentioned problem
Looks like there run goes into segmentation fault when
LOCAL_CA_CERT or ELASTIC_PACKAGE_CA_CERT is not set
https://github.com/elastic/elastic-package/blob/main/internal/install/_static/docker-custom-agent-base.yml
https://github.com/elastic/elastic-package/blob/47808579c755757ef63800b36202ff390a[…]ernal/testrunner/runners/system/servicedeployer/custom_agent.go
I assume the above key is needed for establishing the communication with the agent.
How to get this pem file? Will the value of LOCAL_CA_CERT / ELASTIC_PACKAGE_CA_CERT will be taken care in CI-CD pipeline similar to SERVICE_LOGS_DIR

[agithomas]

LOCAL_CA_CERT was introduced after elastic-package v0.51.2 and thats why no similar errors could be found in elastic-package v0.51.2

[agithomas]

I believe, i didn't do elastic-package stack shellinit

Let me retry

[jsoriano]

I think this is another case of corrupted or old profiles, such as the ones in #589, #881 or #962.

We should work on something like #557 to solve this.

[agithomas]

Checked out the latest code from main branch, ran the stack init and retested. The issue do not appear any more.