toasterson/Anima

Watch

Fork 0

WIP: anima-runner leaks its executor child on the cancel path: a defunct opencode survives as a zombie because tini is PID 1 and the parent never wait()s #609

Draft

toasterson wants to merge 2 commits from claude/wi-01a00463-anima-runner-leaks-its-executor-child-on into main

toasterson commented

2026-08-16 21:56:34 +00:00

Owner

Anima work item 01a00463-15bf-7fb2-8c17-d86a09b60834.

Symptom

Inside anima-runner-opencode-1, hours after the turn had ended:

    PID     ELAPSED STAT COMMAND
      1  2-18:45:15 Ss   docker-init  /sbin/docker-init -- /usr/bin/tini -- anima-runner
      7  2-18:45:15 S    tini         /usr/bin/tini -- anima-runner
      8  2-18:45:15 Sl   anima-runner anima-runner
    266  2-18:43:59 Z    opencode     [opencode] <defunct>

PID 266 is a zombie: opencode exited, but nobody reaped it. It survived every subsequent turn — the runner went on to provision a new worktree five minutes later and kept working normally.

Why tini does not save us here

Dockerfile.runner installs tini for exactly this reason (see the comment at Dockerfile.runner:52-59, "tini: PID 1 that reaps. anima-runner spawns executors (opencode, ...)"). But tini only reaps processes re-parented to it — i.e. orphans, whose parent has already died.

Here the parent is alive: opencode's parent is anima-runner (PID 8), which is running fine. A child whose living parent never calls wait() stays a zombie indefinitely, and PID 1 never gets the chance to collect it. tini is the wrong layer for this failure; the fix has to be in the runner.

Note also the comment at Dockerfile.runner:107-113 — under compose there are two tinis (docker-init at PID 1, ours at PID 7), which is fine but irrelevant to this bug for the same reason.

Trigger path

The leak is on the timeout/cancel path, not the happy path:

15:30:55  acp prompt turn timed out — sending session/cancel  max_secs=3600
15:31:05  cancelled turn did not settle within the grace window; proceeding secs=10

When the executor does not settle inside the grace window the runner proceeds without it. The child is presumably killed or left to die, but the Child handle is dropped without being awaited, so no waitpid ever happens. Rust's std::process::Child explicitly does not reap on drop.

Fix

On the give-up path, keep the Child and reap it — child.kill() then child.wait() (or tokio's Child::wait), ideally with kill_on_drop(true) set at spawn so an early return cannot leak. If the runner intentionally detaches long-lived executors, it needs an explicit reaper task that wait()s on the pids it has spawned.

Why it matters beyond tidiness

Each leaked zombie holds a PID and its exit status. Runners are long-lived (this container had been up 3 days), and every turn that hits the 3600s cap adds one. It also makes "is the executor still alive?" unanswerable by process inspection, which is precisely what an operator reaches for when a session looks wedged.

Test

Spawn a fake executor that ignores session/cancel, drive a turn past the cap, and assert no defunct child remains after the grace window lapses.

The stuck session row this was found alongside: the rebind reaper cannot see status='blocked' (anima-db agent.rs:977), filed separately.
The misclassification of the turn timeout as a provider-quota event is WI-492 / PR #525.

Anima work item `01a00463-15bf-7fb2-8c17-d86a09b60834`. ## Symptom Inside `anima-runner-opencode-1`, hours after the turn had ended: ``` PID ELAPSED STAT COMMAND 1 2-18:45:15 Ss docker-init /sbin/docker-init -- /usr/bin/tini -- anima-runner 7 2-18:45:15 S tini /usr/bin/tini -- anima-runner 8 2-18:45:15 Sl anima-runner anima-runner 266 2-18:43:59 Z opencode [opencode] <defunct> ``` PID 266 is a zombie: opencode exited, but nobody reaped it. It survived every subsequent turn — the runner went on to provision a new worktree five minutes later and kept working normally. ## Why tini does not save us here `Dockerfile.runner` installs tini for exactly this reason (see the comment at Dockerfile.runner:52-59, "tini: PID 1 that reaps. anima-runner spawns executors (opencode, ...)"). But tini only reaps processes **re-parented to it** — i.e. orphans, whose parent has already died. Here the parent is alive: opencode's parent is `anima-runner` (PID 8), which is running fine. A child whose living parent never calls `wait()` stays a zombie indefinitely, and PID 1 never gets the chance to collect it. tini is the wrong layer for this failure; the fix has to be in the runner. Note also the comment at Dockerfile.runner:107-113 — under compose there are two tinis (docker-init at PID 1, ours at PID 7), which is fine but irrelevant to this bug for the same reason. ## Trigger path The leak is on the timeout/cancel path, not the happy path: ``` 15:30:55 acp prompt turn timed out — sending session/cancel max_secs=3600 15:31:05 cancelled turn did not settle within the grace window; proceeding secs=10 ``` When the executor does not settle inside the grace window the runner proceeds without it. The child is presumably killed or left to die, but the `Child` handle is dropped without being awaited, so no `waitpid` ever happens. Rust's `std::process::Child` explicitly does **not** reap on drop. ## Fix On the give-up path, keep the `Child` and reap it — `child.kill()` then `child.wait()` (or tokio's `Child::wait`), ideally with `kill_on_drop(true)` set at spawn so an early return cannot leak. If the runner intentionally detaches long-lived executors, it needs an explicit reaper task that `wait()`s on the pids it has spawned. ## Why it matters beyond tidiness Each leaked zombie holds a PID and its exit status. Runners are long-lived (this container had been up 3 days), and every turn that hits the 3600s cap adds one. It also makes "is the executor still alive?" unanswerable by process inspection, which is precisely what an operator reaches for when a session looks wedged. ## Test Spawn a fake executor that ignores `session/cancel`, drive a turn past the cap, and assert no defunct child remains after the grace window lapses. ## Related - The stuck session row this was found alongside: the rebind reaper cannot see `status='blocked'` (anima-db agent.rs:977), filed separately. - The misclassification of the turn timeout as a provider-quota event is WI-492 / PR #525.

toasterson added 1 commit

2026-08-16 21:56:42 +00:00

anima-runner leaks its executor child on the cancel path: a defunct opencode survives as a zombie because tini is PID 1 and the parent never wait()s fa0482fc29

Anima implementation session.

toasterson added 1 commit

2026-08-16 23:04:50 +00:00

anima-runner leaks its executor child on the cancel path: a defunct opencode survives as a zombie because tini is PID 1 and the parent never wait()s 792805cdfb

Anima implementation session.

This pull request is marked as a work in progress.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin claude/wi-01a00463-anima-runner-leaks-its-executor-child-on:claude/wi-01a00463-anima-runner-leaks-its-executor-child-on

git switch claude/wi-01a00463-anima-runner-leaks-its-executor-child-on

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff claude/wi-01a00463-anima-runner-leaks-its-executor-child-on

git switch claude/wi-01a00463-anima-runner-leaks-its-executor-child-on

git rebase main

git switch main

git merge --ff-only claude/wi-01a00463-anima-runner-leaks-its-executor-child-on

git switch claude/wi-01a00463-anima-runner-leaks-its-executor-child-on

git rebase main

git switch main

git merge --no-ff claude/wi-01a00463-anima-runner-leaks-its-executor-child-on

git switch main

git merge --squash claude/wi-01a00463-anima-runner-leaks-its-executor-child-on