act_runner v2.0.0: runner goes silent after `Worker finished`, server still assigns jobs to it (in_progress + all steps queued)

[SAMYUBA]

Description

A single act_runner (v2.0.0, capacity: 1) periodically stops picking up work
without any error and without exiting. The server considers the runner healthy and
assigns jobs to it — the job flips to in_progress and its task row is created —
but the runner never logs receiving that task, so every step stays queued forever
until the runner process is restarted.

This is not the "job stays waiting / never gets assigned" behaviour described in
#33492. Here the job is assigned; it is the runner that never acts on it.

Environment

Gitea server	1.26.4
act_runner	v2.0.0
Runner count	1 runner with the relevant label (a second runner exists but its labels do not overlap)
runner.capacity	1
log.level	debug
OS	NixOS (runner runs as a systemd service on the same LAN as the server)

⚠️ Note this is explicitly the configuration that #33492 states is not affected
("single runner setups does not seem to be affected"). We reproduce it with one
runner and no parallelism.

Observed behaviour

Normal cycle (from debug logs)

17:44:19  level=info  msg="runner: <name>, with version: v2.0.0, with labels: [<label>], declare successfully"  daemon.go:148
17:44:27  level=info  msg="task 1371 repo is <org>/<repo> ..."                                                  runner.go:333
17:44:27  level=debug msg="Worker 0 started"                                                                    executor.go:101
17:44:27  level=debug msg="Worker 0 executing task 1"                                                           executor.go:105
   ... job runs ...
          level=debug msg="Worker 0 finished (1 tasks executed)"                                                executor.go:119

After Worker 0 finished, the next task normally arrives within seconds to a few minutes
(observed gaps on the same day: 19s, 6m18s, etc.), each time re-entering
task N repo is ... → Worker 0 started.

Failure cycle

17:36:31  level=debug msg="Worker 0 finished (1 tasks executed)"   executor.go:119   <-- last log line
17:38:13  server marks job 1250 as in_progress (task row created)
          (no "task 1250 repo is ..." line)
          (no "Worker 0 started" line)
          journalctl for the runner unit: -- No entries --
17:44:17  external watchdog restarts the runner (job had been stuck 5m46s)
17:44:19  declare successfully
17:44:27  task 1371 ... -> Worker 0 started -> executing        <-- recovers immediately

Job state as seen through the API while stuck:

job 1250: in_progress, started_at 2026-08-01T09:38:13Z
   step: queued  Run actions/checkout@v7
   step: queued  (6 more steps, all queued)

Process state while stuck

The runner is not crashed, hung on CPU, or disconnected:

• systemctl is-active → active
• /proc/<pid>/status → State: S (sleeping), Threads: 13
• 4 ESTABLISHED TCP connections to the Gitea instance on :443
• Total CPU consumed over ~2h wall clock: ~11s (i.e. idle polling, not spinning)

So the process is alive and still connected — it simply never surfaces the assigned task.

Frequency

Measured on one day with 44 workflow runs: 8 occurrences ≈ 18% of runs.

⚠️ Rates must be normalised per-run; raw counts are dominated by daily activity
(the same instance showed 1/4 = 25% and 1/2 = 50% on two quieter days).

What we ruled out

Hypothesis	How it was excluded
tasksVersion staleness (#31074 family)	We carry a local patch forcing TasksVersion: 0 in internal/app/poll/poller.go (verified present in the built derivation via --replace-fail). Rate stayed ~18%.
Concurrency / capacity exhaustion	One failure occurred with zero concurrent runs (previous run started 71 min earlier, next one 40 min later).
Hardware / memory	Host had RAM replaced and rebooted mid-day; failures occurred both before and after the reboot.
Network loss	4 ESTABLISHED connections to the server during the stuck window.
Runner crash / restart loop	Process alive, same PID throughout the stuck window, State: S.

Relation to existing issues

• Gitea Actions FetchTask not reliable assigning queued jobs to idle runners as long no new jobs are queued #33492 — describes jobs never being assigned; explicitly says single-runner setups
  are not affected. Our jobs are assigned; our setup is single-runner. Different failure mode,
  though possibly the same underlying FetchTask path that PR Improve FetchTask reliability / performance #35960 touches.
• Actions Runner processes just one task, then waits forever. (Possible bug in ActionTasksVersion) #31074 (closed) — its title, "Actions Runner processes just one task, then waits forever",
  matches our behaviour closely. The difference is server-side presentation: there the job stayed
  waiting; here it becomes in_progress. If the server now creates the task row before the runner
  acts on it, the same runner-side defect could surface exactly as we observe. Worth checking whether
  the Actions Runner processes just one task, then waits forever. (Possible bug in ActionTasksVersion) #31074 fix regressed or was incomplete for v2.0.0.

What would help narrow it down

debug level does not log the FetchTask round-trip itself, so we cannot see whether the runner
received a task payload and dropped it, or never received one despite the server having assigned it.
A log line on the fetch response path (even at trace) would make this directly diagnosable.

Happy to run instrumented builds against this instance — it reproduces roughly every 5–6 runs.

[bircni]

Does this still happen in 1.27.1?
There were a lot of fixes to actions in 1.27

[SAMYUBA]

Does this still happen in 1.27.1?

Honest answer: I don't know yet — we are still on 1.26.4, so I can't confirm or rule
out 1.27.x. Upgrading this instance needs a maintenance window on my side; I'll report
back once we're on 1.27.x and have enough runs to compare. If the actions fixes in 1.27
cover this, that would be the best outcome.

What I can add meanwhile is a fresh occurrence on 1.26.4 today, with one detail I
haven't seen described anywhere: restarting the runner is enough to recover the job
itself — the server re-delivers the same task to the restarted runner, no rerun needed.

Timeline (same environment as the original report: 1.26.4, act_runner v2.0.0, host executor, single runner, capacity: 1)

19:07:50  job assigned; job status "in_progress"; all 8 steps "queued"
19:12:51  observed: still 8/8 queued, no step ever transitioned
19:14:18  our watchdog flags it and restarts the runner
19:22:31  ** the same job ** starts executing — same job id, same run_attempt,
          and no rerun API call was made (our automation explicitly declined to rerun)
~19:2x    all 8 steps green, run concludes success

Why that detail may narrow it down

The task appears to stay intact server-side for the entire stuck window. Nothing about
the task's own state had to change — a new runner connection was sufficient to make it
deliverable again.

Combined with the earlier finding that PickTask succeeds (the job flips to
in_progress) while the runner never surfaces the task at all, this suggests the loss is
per-connection rather than per-task, somewhere on the FetchTask response path.

If that's right, a stuck job should always be recoverable by bouncing the runner, which
matches everything we've observed so far.

Two traps for anyone automating detection around this

Both cost me a misdiagnosis today, so they may be worth writing down:

1. "All steps queued with zero timestamps" is not by itself a stuck job. A job
   assigned seconds ago looks identical. A detector needs a time component — ours uses
   "hasn't picked up work in the last ~8 minutes".

2. The reap signature is started_at == completed_at (zero-length steps), not
   "timestamp is epoch". When the server reaps the run it fills in real timestamps,
   so an epoch-based check silently stops matching exactly when you need it. Getting this
   backwards makes a reaped zombie look like a genuine failure — and an auto-rerun built
   on it will either never fire, or paper over real failures.

Happy to run an instrumented build or drop our local TasksVersion: 0 patch and retest,
if that would help isolate it — reproduction rate here is roughly one in five or six runs.

[bircni]

I think this might solve it
#38200

It is hard to debug - also we do not support 1.26 actively anymore and recommend upgrading...