Add mechanism for cleaning up the Elastic stack

[ishleenk17]

Many a times, we observe an issue where the elastic stack doesn't come up when we run elastic-package stack up -v.
It usually gets stuck at "Kibana not ready state".

Below are the Kibana logs when the stack is stuck at this point:

[2022-05-25T11:07:11.635+00:00][INFO ][savedobjects-service] [.kibana] INIT -> CREATE_NEW_TARGET. took: 19ms.
[2022-05-25T11:07:11.637+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> CREATE_NEW_TARGET. took: 14ms.
[2022-05-25T11:07:12.408+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_arm64/headless_shell
[2022-05-25T11:07:41.639+00:00][ERROR][savedobjects-service] [.kibana] Action failed with ‘Request timed out’. Retrying attempt 1 in 2 seconds.
[2022-05-25T11:07:41.640+00:00][INFO ][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 30004ms.

As part of this issue, would like to record the steps one can follow to overcome this issue.

[jsoriano]

Hey @ishleenk17, thanks or reporting this. Do you have more information about how to reproduce the issue?
Does it happen when starting a stack from scratch? Or when re-starting a stopped stack?
On what operating system do you observe these problems?

Do you have a workaround at the moment to solve this issue?

[ishleenk17]

@jsoriano : I see this issue after couple or restart of the stack. I am using MAC OS.

I have figured out couple of ways to resolve this issue. I thought documenting this might help the community.

Cleaning the docker images, docker containers, rebuilding the elastic package binary: After doing these things if we restart the stack it usually come up.
But at times I see that even this doesn't work.

An additional thing that I used as workaround is prune the build cache of docker. I think some Kibana objects get stored in the build cache and are not removed when the stack is brought down.

I was thinking we can document this in the "howto" unde elastic packages

[jsoriano]

If this happens so frequently I think that we should consider this a bug, and try to address it instead of documenting workarounds.

I'd prefer if elastic-package stack could overcome these issues or help with the workarounds. We could try to make elastic-package stack down able to clean everything that can be problematic. Or add a cleanup subcommand for this.

If we find something that cannot be reliably solved directly in elastic-package, then we could add this to some troubleshooting section of the docs.

[ishleenk17]

Sure, that also sound good.
I think best would be to have an elastic-package stack cleanup command which would actually take care of all these dangling kibana object cleanup as well. @jsoriano

[jsoriano]

I managed to reproduce the problem described in this issue by running out of disk space for docker, I have opened an issue to improve handling of this: #856