toasterson/Anima

Watch

Fork 0

WIP: A runner-blocked session is unreapable: the rebind reaper matches only ('provisioning','running'), so status='blocked' pins a lane slot forever #610

Draft

toasterson wants to merge 1 commit from claude/wi-01a00462-a-runner-blocked-session-is-unreapable-t into main

toasterson commented

2026-08-17 01:28:29 +00:00

Owner

Anima work item 01a00462-c415-7fa3-9478-b18dbe8cccc1.

Symptom

Session 01a000ae-bb0d-7661-8bde-a30d2667785a (WI-472, opencode) sat at status=blocked with abandoned_at IS NULL for 1h40m and would have sat there forever. It pinned a lane slot against ANIMA_SCHED_MAX_ACTIVE, and it was the sole reason a fleet drain could not reach zero — an operator had to cancel_session it by hand.

Root cause

AgentSessionRepo::abandon_stale_bound (crates/anima-db/src/repo/agent.rs:969-993) is the rebind-window reaper — the thing whose stated job is "what finally retires a zombie whose runner lost it". Its predicate:

UPDATE agent_sessions
   SET status = 'failed', abandoned_at = now(), updated_at = now()
 WHERE runner_id IS NOT NULL
   AND status IN ('provisioning', 'running')   -- <-- 'blocked' is absent
   AND abandoned_at IS NULL
   AND updated_at < now() - make_interval(secs => $1)

blocked is a live, non-terminal status — a blocked session still holds its slot, and lane_occupancy counts it deliberately ("Includes blocked sessions because a blocked session is still pinning a slot"). But no reaper, timeout, or server restart can move it. It is outside every sweep.

How the session got there

Reproduced end-to-end from the runner log:

15:30:55  acp prompt turn timed out — sending session/cancel  max_secs=3600
15:31:05  cancelled turn did not settle within the grace window; proceeding secs=10
15:31:05  lane health changed  executor="opencode" state=NoQuota
15:31:06  skipping commit/push/PR: turn did not end cleanly  stop_reason="cancelled"
15:31:06  no changes to push this turn  session_id=01a000ae-...
15:36:20  provisioning workspace  worktree=.../01a000ea-...    <- runner moved on

The runner behaved correctly: the cap fired, the executor did not acknowledge session/cancel inside the 10s grace, so the runner abandoned the turn and took new work. It left the row at:

status         | blocked
blocked_reason | provider lane blocked (turn timed out after 3600s)
abandoned_at   | (null)

So there is a class of sessions the runner deliberately parks as blocked and never revisits, and nothing server-side collects them.

Fix

Add blocked to the predicate. It satisfies the same staleness argument as running: runner_id IS NOT NULL, no activity for the whole window, and reopen_work_items_for_failed_sessions is exactly the right follow-up.

Consider whether the window should differ for blocked — a session blocked on a legitimately slow provider may want a longer grace than one that simply went silent. If so, take the window as a per-status value rather than a single window_secs.

Test

An integration test that inserts a bound session at status='blocked' with updated_at older than the window, runs the reaper, and asserts it is failed/abandoned_at IS NOT NULL and its work item is dispatchable again. The existing running/provisioning cases should be asserted in the same test so the status list cannot silently narrow again.

The blocked_reason here reads "provider lane blocked" for what was a plain turn timeout, and the lane flipped to NoQuota. That misclassification is WI-492 / PR #525.
anima-runner leaks the executor child on this path: PID 266 Z [opencode] <defunct> survived in the container. tini is PID 1 and only reaps orphans; the parent here is anima-runner (PID 8), which never wait()s.

Anima work item `01a00462-c415-7fa3-9478-b18dbe8cccc1`. ## Symptom Session `01a000ae-bb0d-7661-8bde-a30d2667785a` (WI-472, opencode) sat at `status=blocked` with `abandoned_at IS NULL` for 1h40m and would have sat there forever. It pinned a lane slot against `ANIMA_SCHED_MAX_ACTIVE`, and it was the sole reason a fleet drain could not reach zero — an operator had to `cancel_session` it by hand. ## Root cause `AgentSessionRepo::abandon_stale_bound` (`crates/anima-db/src/repo/agent.rs:969-993`) is the rebind-window reaper — the thing whose stated job is "what finally retires a zombie whose runner lost it". Its predicate: ```sql UPDATE agent_sessions SET status = 'failed', abandoned_at = now(), updated_at = now() WHERE runner_id IS NOT NULL AND status IN ('provisioning', 'running') -- <-- 'blocked' is absent AND abandoned_at IS NULL AND updated_at < now() - make_interval(secs => $1) ``` `blocked` is a **live, non-terminal** status — a blocked session still holds its slot, and `lane_occupancy` counts it deliberately ("Includes blocked sessions because a blocked session is still pinning a slot"). But no reaper, timeout, or server restart can move it. It is outside every sweep. ## How the session got there Reproduced end-to-end from the runner log: ``` 15:30:55 acp prompt turn timed out — sending session/cancel max_secs=3600 15:31:05 cancelled turn did not settle within the grace window; proceeding secs=10 15:31:05 lane health changed executor="opencode" state=NoQuota 15:31:06 skipping commit/push/PR: turn did not end cleanly stop_reason="cancelled" 15:31:06 no changes to push this turn session_id=01a000ae-... 15:36:20 provisioning workspace worktree=.../01a000ea-... <- runner moved on ``` The runner behaved correctly: the cap fired, the executor did not acknowledge `session/cancel` inside the 10s grace, so the runner abandoned the turn and took new work. It left the row at: ``` status | blocked blocked_reason | provider lane blocked (turn timed out after 3600s) abandoned_at | (null) ``` So there is a class of sessions the runner deliberately parks as `blocked` and never revisits, and nothing server-side collects them. ## Fix Add `blocked` to the predicate. It satisfies the same staleness argument as `running`: `runner_id IS NOT NULL`, no activity for the whole window, and `reopen_work_items_for_failed_sessions` is exactly the right follow-up. Consider whether the window should differ for `blocked` — a session blocked on a legitimately slow provider may want a longer grace than one that simply went silent. If so, take the window as a per-status value rather than a single `window_secs`. ## Test An integration test that inserts a bound session at `status='blocked'` with `updated_at` older than the window, runs the reaper, and asserts it is `failed`/`abandoned_at IS NOT NULL` and its work item is dispatchable again. The existing running/provisioning cases should be asserted in the same test so the status list cannot silently narrow again. ## Related, filed separately - The `blocked_reason` here reads "provider lane blocked" for what was a plain turn timeout, and the lane flipped to `NoQuota`. That misclassification is WI-492 / PR #525. - `anima-runner` leaks the executor child on this path: `PID 266 Z [opencode] <defunct>` survived in the container. tini is PID 1 and only reaps orphans; the parent here is `anima-runner` (PID 8), which never `wait()`s.

toasterson added 1 commit

2026-08-17 01:28:29 +00:00

A runner-blocked session is unreapable: the rebind reaper matches only ('provisioning','running'), so status='blocked' pins a lane slot forever 1e7238b5a6

Anima implementation session.

This pull request has changes conflicting with the target branch.

crates/anima-server/src/mcp/handlers.rs

View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Merge commit title

Merge pull request 'WIP: A runner-blocked session is unreapable: the rebind reaper matches only ('provisioning','running'), so status='blocked' pins a lane slot forever' (#610) from claude/wi-01a00462-a-runner-blocked-session-is-unreapable-t into main

Merge commit body

Reviewed-on: https://code.aopc.cloud/toasterson/Anima/pulls/610

Merge pull request 'WIP: A runner-blocked session is unreapable: the rebind reaper matches only ('provisioning','running'), so status='blocked' pins a lane slot forever' (#610) from claude/wi-01a00462-a-runner-blocked-session-is-unreapable-t into main

Reviewed-on: https://code.aopc.cloud/toasterson/Anima/pulls/610

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin claude/wi-01a00462-a-runner-blocked-session-is-unreapable-t:claude/wi-01a00462-a-runner-blocked-session-is-unreapable-t

git switch claude/wi-01a00462-a-runner-blocked-session-is-unreapable-t

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff claude/wi-01a00462-a-runner-blocked-session-is-unreapable-t

git switch claude/wi-01a00462-a-runner-blocked-session-is-unreapable-t

git rebase main

git switch main

git merge --ff-only claude/wi-01a00462-a-runner-blocked-session-is-unreapable-t

git switch claude/wi-01a00462-a-runner-blocked-session-is-unreapable-t

git rebase main

git switch main

git merge --no-ff claude/wi-01a00462-a-runner-blocked-session-is-unreapable-t

git switch main

git merge --squash claude/wi-01a00462-a-runner-blocked-session-is-unreapable-t