podman wait condition and communication with systemd service unit

[l4rsV]

Feature request description

The interaction of a systemd unit with podman wait is particularly unsatisfactory when using podman-compose if you want to start, control and monitor more than one container per service.

following option in the systemd unit:

[Service]
Type=simple
ExecStart=podman wait --condition=unhealthy container1 container2


This function causes the systemd unit to fail only when both containers are unhealthy. This means that they are logically linked with "and" instead of "or" as expected.

$podman wait --help
Description: Block until one or more containers stop and then print their exit codes.

It is therefore currently not possible to check the status of several containers per service unit without a workaround:

while true; do podman ps -a --format {{.Status}} --filter label=io.podman.compose.project= | grep unhealthy && exit 1; done && exit 1

I think this could be a great feature to improve the communication between service and container and to implement some kind of check/watchdog.

Regards,
Lars

[Luap99]

The and behavior is expected, see the actual man page for the command which describes it properly.

Waits on one or more containers to stop. The container can be referred to by its name or ID. In the case of multiple containers, Podman waits on each consecutively. After all conditions are satisfied, the containers' return
codes are printed separated by newline in the same order as they were given to the command. An exit code of -1 is emitted for all conditions other than "stopped" and "exited".

So in fact it doesn't even wait in parallel it is literately a simple loop over the arguments and waits one by one.

I don't understand the bigger picture of what you are trying to do. Why would you want to run podman wait as main process in a systemd unit? Would it not be better track the actual main process?
Also I don't know what you are doing if it turns unhealthy but podman already supports --health-on-failure which you can set to stop the container for example and then if you track the actual main container process then the unit will be failed automatically.

This function causes the systemd unit to fail only when both containers are unhealthy.

How would this ever fail? f both containers are unhealthy then I still would expect podman to exit 0.

[l4rsV]

Hi,

Fist of all podman-compose -f docker-compose.yaml systemd -a create-unit creates the following:

[Service]
Type=simple
EnvironmentFile=%h/.config/containers/compose/projects/%i.env
ExecStartPre=-/usr/bin/podman-compose up --no-start
ExecStartPre=/usr/bin/podman pod start pod_%i
ExecStart=/usr/bin/podman-compose wait
ExecStop=/usr/bin/podman pod stop pod_%i

So it is using podman-compose wait as default/main process and the service unit thinks all compose containers are running as long as podman wait returns nothing different. First when all compose containers are crashed, the service unit registers the exit state.
That is something we don't want...

So i had the idea to change the behaviour of podman wait, to use it as some kind of watchdog via systemd.

Yes it would be better to track the actual process in a systemd unit, but this means we will have to create a lot of linked service units and compose has multiple main processes, for each container one. How should the service unit know about that?
it is difficult to build in pipelines and we have no support with podman-compose -a create

--health-on-failure
I tried millions of combinations in podman-compose.yml files, I'm not able to overwrite HealthcheckOnFailureAction.
Only this workaround is possible containers/podman-compose#747 (comment)

podman inspect | grep -i health
"Health": {
"Status": "healthy",
"Healthcheck": {
"/healthcheck.sh"
"HealthcheckOnFailureAction": "none",

This requires that all services have a healthcheck defined, either with a test command, or be explicitly disabled:

this is the point at which it no longer works for us and we would have to convert a large number of yaml's, edit every single container and deviate from the standards.

I know it from docker that the daemon monitors the compose container and restarts it. But we don't have this under podman.
Either I understand something fundamentally wrong here, or podman, especially podman-compose, is severely restricted with the use of systemd services?

My goal is to achieve that the containers restart in the event of an error, i.e. if they are unhealthy. either with a systemd unit or using HealthcheckOnFailureAction.

In order to be able to continue working, my solution looks like this:

[Service]
Type=simple
EnvironmentFile=%h/.config/containers/compose/projects/%i.env
ExecStartPre=-/usr/bin/podman-compose up --no-start
ExecStartPre=/usr/bin/podman start --all
ExecStart=/bin/bash -c "while sleep 5; do podman ps -a --format {{.Status}} --filter label=io.podman.compose.project=%i | grep -iE 'unhealthy|exited' && exit 1; done"
ExecStop=/usr/bin/podman-compose down

Do you have any better idea??

I know it is better to use kube yaml and pods but we are not there yet and we need more docker compatibility

Regards,
Lars

See this related issue:
#17777
containers/podman-compose#747
containers/podman-compose#955

[Luap99]

I know it from docker that the daemon monitors the compose container and restarts it. But we don't have this under podman.

It would be news to me that docker actually does anything when a container hits unhealthy state. There is --restart always which restarts the container on exit but that should work on podman as well.

Note podman-compose is not supported by us the podman maintainers.

The systemd unit that compose uses is really an ugly hack IMO and doiesn't really make use of proper systemd server tracking. The podman team recommends to the use of quadlets instead of compose for service management, https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html. This should integrate much better with systemd and exposes basically all podman cli options so it would be trivial to use HealthOnFailure.

---

In general I am fine to consider this is a feature request for podman wait. I see no issue with adding a new option that actually waits for all containers in parallel and exits on the first match instead of waiting for all.

[arsenalzp]

Hello,
Can I work on it?

[Luap99]

@arsenalzp Sure, just to be clear the requirement would be to add a new podman wait option (--exit-first maybe, not sure on the naming?) that waits for all containers in parallel and then exits once the first container wait completes.

[l4rsV]

perhaps it would also be possible to add a delay option in seconds? Some containers start as unhealthy until their dependencies are available. Start checking for exit codes after x-seconds?
Otherwise the systemd unit gets stuck in the loop.

[Luap99]

perhaps it would also be possible to add a delay option in seconds? Some containers start as unhealthy until their dependencies are available. Start checking for exit codes after x-seconds? Otherwise the systemd unit gets stuck in the loop.

You should be using --health-start-period for that when creating the container. It will ignore all failed healthchecks until the time is reached and shows starting instead.