failed daemon leaves workflow in bad state

[rwong2888]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

possibly related to #14544

i have transient retries on things such as pod deletion

that staging app server errored out with status 143
the other app server deleted
not sure why
then it looks like it spun back up due to transient errors and retries we have with pod deletions
then it looks like it tried to do 2 more retries with the git clone steps again
and workflows are just stuck in a weird state now
the workflow looks failed, but still has pods running

Version(s)

3.7.0

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

n/a

Logs from the workflow controller

running into character limit. linking to logs in slack snippet
https://cloud-native.slack.com/archives/C01QW9QSSSK/p1753901691851319?thread_ts=1753901385.723079&cid=C01QW9QSSSK

Logs from in your workflow's wait container

garbage collection ran i think


kubectl logs -n frontend -c wait -l workflows.argoproj.io/workflow=playwright-simple-site-dg7wr,workflow.argoproj.io/phase!=Succeeded
No resources found in frontend namespace.

[ArmineAzizyan]

Adding a minimal workflow that reproduces this issue

Steps to reproduce

• delete monitor pod

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: daemon-example-
spec:
  entrypoint: main
  templates:
    - name: main
      steps:
        - - name: print
            template: print

        - - name: monitor
            template: monitor

        - - name: fail
            template: fail 

    - name: monitor
      daemon: true
      retryStrategy:
        limit: 3
      script:
        image: python:alpine3.6
        command: [ python ]
        source: |
          import time
          print("Daemon running successfully", flush=True)
          while True:
              print("Daemon updating monitoring status", flush=True)
              time.sleep(5)
    
    - name: print
      script:
        image: busybox:1.35
        command: [sh]
        source: |
           echo "testing print step"

    - name: fail
      script:
        image: python:alpine3.6
        command: [ python ]
        source: |
          import time
          import sys
          time.sleep(60)
          sys.exit(1)

run result

[rwong2888]

terminate in the UI or cli does not actually terminate. the workflow still reports as running

❯ argo terminate daemon-example-s585s
workflow daemon-example-s585s terminated
❯ argo terminate daemon-example-z9pdx
workflow daemon-example-z9pdx terminated

[mbouillaud]

Same here, Argo Workflows 3.7.1 on GKE.

Daemon container with daemon: true marked failed but container showed as completed with exit code 0 :

[rwong2888]

upvote the issue for visibility @mbouillaud !

[eduardodbr]

@rwong2888 @mbouillaud @ArmineAzizyan can you please check if setting:

retryStrategy:
  retryPolicy: Always

works as a workaround? (this will only work if max retry is not achieved, otherwise bug still applies)

[ArmineAzizyan]

@rwong2888 @mbouillaud @ArmineAzizyan can you please check if setting:

retryStrategy:
  retryPolicy: Always

works as a workaround? (this will only work if max retry is not achieved, otherwise bug still applies)

Tested with retryPolicy: "Always"

1. If the last step of the workflow succeeds, it works

2. Can confirm that the issue occurs once the max number of retries is reached

3. If the last step of the workflow fails, the last retry pod is initially marked as succeeded, but eventually it shows as failed ( Using the previously mentioned workflow )

[rwong2888]

Thanks @eduardodbr , I defer to @ArmineAzizyan above