toasterson/Anima

Watch

Fork 0

A failed turn is indistinguishable from an empty one: server ignores stop_reason, WI re-dispatches forever in `working` #366

Merged

toasterson merged 2 commits from claude/wi-019f98b5-a-failed-turn-is-indistinguishable-from into main

2026-07-28 10:10:53 +00:00

toasterson commented

2026-07-25 13:03:03 +00:00

Owner

Anima work item 019f98b5-2ccd-7d80-9842-ec3e395b0d12.

Problem

The runner reports why a turn ended; the server throws it away. A turn that died — expired auth, EMFILE, a blown turn budget — lands on the server as the same nothing as a turn where the agent genuinely had no changes. The work item stays in working, the scheduler re-dispatches it on the next tick, the turn dies the same way, and this repeats indefinitely with no failure recorded anywhere a human or the loop guard can see.

This is why the fleet can burn a day and a half at zero throughput while every surface reads "working".

Observed (2026-07-24 → 07-25)

Nothing has merged to main since 2026-07-23 21:22. In that window all three lanes failed silently:

claude lane (anima-runner-1): CLAUDE_CODE_OAUTH_TOKEN expired. Every turn since at least 03:19Z: acp prompt turn failed error=… API Error: 401 {"type":"authentication_error","message":"OAuth access token has expired"} immediately followed by no changes to push this turn. A fresh session id every ~33 min, ~20 consecutive dead turns.
opencode lane (anima-runner-opencode): every turn hangs and is killed at the 3600s cap — acp prompt turn timed out — sending session/cancel, then cancelled turn did not settle within the grace window, then no changes to push this turn. One WI-hour burned per turn, ~1 turn/90 min, zero output.
Also seen on runner-1: EMFILE: too many open files, watch …/.claude thrown out of SettingsManager.setupWatchers during session/new — the session came up crippled and the turn failed, again reported as nothing.

Root cause (code)

crates/anima-runner/src/acp.rs (~L927) already classifies the outcome correctly:

let stop_reason = match waited {
    Some(Ok(Ok(resp))) => stop_reason_str(resp.stop_reason),
    Some(Ok(Err(e)))   => { tracing::warn!(error = %e, "acp prompt turn failed"); "error".to_string() }
    Some(Err(_))       => { /* turn budget blown → cancel */ }

crates/anima-server/src/services/runner.rs (~L1076–1120), handling TurnDone, does exactly one thing with it: writes payload: {"stop_reason": td.stop_reason} onto an acp.turn_completed activity event. There is no branch on the value. The only phase-advancing path below it is gated on !td.pr_ref.is_empty() — and a failed turn has no pr_ref, so it falls through to the same no-op as a legitimately empty turn.

Note the runner does distinguish a dead session (SessionClosed.reason = "failed: …", hardened by #337). The gap is one level down: a turn that fails inside a session that stays alive.

Proposed fix

Branch on td.stop_reason in the TurnDone handler. error and cancelled (budget blown) are failures; end_turn / max_tokens are not.
A failed turn must (a) increment the WI's failed_dispatch_count against the existing park cap, and (b) post a visible note on the WI's anchored thread carrying the runner's error text — the 401 string above is self-diagnosing and a human seeing it once would have fixed the fleet in a minute.
Consecutive failures with an identical cause must not be retried at the normal cadence: back off, and at the cap park the WI needs-human with the cause as the park reason, distinct from a code-failure park.
An expired/invalid executor credential is a lane fault, not a work-item fault — N consecutive auth failures on one runner should mark that runner degraded and stop the scheduler assigning to it, rather than charging each WI a strike for the lane being broken.

Acceptance criteria

A turn ending stop_reason=error no longer reads as an empty turn: the WI's failure counter moves and the error text appears on the thread.
20 consecutive auth failures cannot happen silently — the lane is marked degraded and drains long before that.
A turn killed by the budget cap is recorded as a failure, not as "no changes".
A genuinely empty turn (agent had nothing to push) is still a non-event and still does not count as a failure.
Regression tests: TurnDone{stop_reason:"error", pr_ref:""} → counter incremented + thread note emitted; TurnDone{stop_reason:"end_turn", pr_ref:""} → unchanged.

Files

crates/anima-server/src/services/runner.rs (TurnDone handling, ~L1076)
crates/anima-server/src/scheduler.rs (back-off, lane degradation, park reason)
crates/anima-runner/src/acp.rs (carry the error text on TurnDone, not just the "error" label)
proto/anima/v1/runner.proto (TurnDone needs an error field)

019f8602-ed6b-7421-8a12-a4303f6b745a reopen WIs when orphaned sessions fail — same family, session level rather than turn level.
019f9698-6f38-70d3-a1f8-468e966dabc2 CI-red loop can't tell infra failure from job failure — the same "we threw away the reason" mistake on the CI side.
019f903c-eeb1-7e71-928b-e248db024384 fleet observability metrics + alerts — would have alarmed on this.

Anima work item `019f98b5-2ccd-7d80-9842-ec3e395b0d12`. ## Problem The runner reports *why* a turn ended; the server throws it away. A turn that died — expired auth, EMFILE, a blown turn budget — lands on the server as the same nothing as a turn where the agent genuinely had no changes. The work item stays in `working`, the scheduler re-dispatches it on the next tick, the turn dies the same way, and this repeats indefinitely with no failure recorded anywhere a human or the loop guard can see. This is why the fleet can burn a day and a half at zero throughput while every surface reads "working". ## Observed (2026-07-24 → 07-25) Nothing has merged to `main` since 2026-07-23 21:22. In that window all three lanes failed *silently*: - **claude lane** (`anima-runner-1`): `CLAUDE_CODE_OAUTH_TOKEN` expired. Every turn since at least 03:19Z: `acp prompt turn failed error=… API Error: 401 {"type":"authentication_error","message":"OAuth access token has expired"}` immediately followed by `no changes to push this turn`. A fresh session id every ~33 min, ~20 consecutive dead turns. - **opencode lane** (`anima-runner-opencode`): every turn hangs and is killed at the 3600s cap — `acp prompt turn timed out — sending session/cancel`, then `cancelled turn did not settle within the grace window`, then `no changes to push this turn`. One WI-hour burned per turn, ~1 turn/90 min, zero output. - Also seen on runner-1: `EMFILE: too many open files, watch …/.claude` thrown out of `SettingsManager.setupWatchers` during `session/new` — the session came up crippled and the turn failed, again reported as nothing. ## Root cause (code) `crates/anima-runner/src/acp.rs` (~L927) already classifies the outcome correctly: ```rust let stop_reason = match waited { Some(Ok(Ok(resp))) => stop_reason_str(resp.stop_reason), Some(Ok(Err(e))) => { tracing::warn!(error = %e, "acp prompt turn failed"); "error".to_string() } Some(Err(_)) => { /* turn budget blown → cancel */ } ``` `crates/anima-server/src/services/runner.rs` (~L1076–1120), handling `TurnDone`, does exactly one thing with it: writes `payload: {"stop_reason": td.stop_reason}` onto an `acp.turn_completed` activity event. There is **no branch on the value**. The only phase-advancing path below it is gated on `!td.pr_ref.is_empty()` — and a failed turn has no `pr_ref`, so it falls through to the same no-op as a legitimately empty turn. Note the runner *does* distinguish a dead **session** (`SessionClosed.reason = "failed: …"`, hardened by #337). The gap is one level down: a **turn** that fails inside a session that stays alive. ## Proposed fix 1. Branch on `td.stop_reason` in the `TurnDone` handler. `error` and `cancelled` (budget blown) are failures; `end_turn` / `max_tokens` are not. 2. A failed turn must (a) increment the WI's `failed_dispatch_count` against the existing park cap, and (b) post a visible note on the WI's anchored thread carrying the runner's error text — the 401 string above is self-diagnosing and a human seeing it once would have fixed the fleet in a minute. 3. Consecutive failures with an *identical* cause must not be retried at the normal cadence: back off, and at the cap park the WI `needs-human` with the cause as the park reason, distinct from a code-failure park. 4. An expired/invalid executor credential is a **lane** fault, not a work-item fault — N consecutive auth failures on one runner should mark that runner degraded and stop the scheduler assigning to it, rather than charging each WI a strike for the lane being broken. ## Acceptance criteria - A turn ending `stop_reason=error` no longer reads as an empty turn: the WI's failure counter moves and the error text appears on the thread. - 20 consecutive auth failures cannot happen silently — the lane is marked degraded and drains long before that. - A turn killed by the budget cap is recorded as a failure, not as "no changes". - A genuinely empty turn (agent had nothing to push) is still a non-event and still does not count as a failure. - Regression tests: `TurnDone{stop_reason:"error", pr_ref:""}` → counter incremented + thread note emitted; `TurnDone{stop_reason:"end_turn", pr_ref:""}` → unchanged. ## Files - `crates/anima-server/src/services/runner.rs` (`TurnDone` handling, ~L1076) - `crates/anima-server/src/scheduler.rs` (back-off, lane degradation, park reason) - `crates/anima-runner/src/acp.rs` (carry the error *text* on TurnDone, not just the `"error"` label) - `proto/anima/v1/runner.proto` (`TurnDone` needs an error field) ## Related - `019f8602-ed6b-7421-8a12-a4303f6b745a` reopen WIs when orphaned sessions fail — same family, session level rather than turn level. - `019f9698-6f38-70d3-a1f8-468e966dabc2` CI-red loop can't tell infra failure from job failure — the same "we threw away the reason" mistake on the CI side. - `019f903c-eeb1-7e71-928b-e248db024384` fleet observability metrics + alerts — would have alarmed on this.

toasterson added 1 commit

2026-07-25 13:03:05 +00:00

A failed turn is indistinguishable from an empty one: server ignores stop_reason, WI re-dispatches forever in working 49536b8345

Anima implementation session.

toasterson force-pushed claude/wi-019f98b5-a-failed-turn-is-indistinguishable-from from 49536b8345 to 4aef3b1f32

2026-07-25 13:46:08 +00:00

Compare

toasterson force-pushed claude/wi-019f98b5-a-failed-turn-is-indistinguishable-from from 4aef3b1f32 to 09a83243d4

2026-07-25 14:31:08 +00:00

Compare

toasterson force-pushed claude/wi-019f98b5-a-failed-turn-is-indistinguishable-from from 09a83243d4 to 815c9daff6

2026-07-25 15:16:17 +00:00

Compare

toasterson force-pushed claude/wi-019f98b5-a-failed-turn-is-indistinguishable-from from 815c9daff6 to 3e34c8df00

2026-07-25 21:45:44 +00:00

Compare

toasterson force-pushed claude/wi-019f98b5-a-failed-turn-is-indistinguishable-from from 3e34c8df00 to c914c46149

2026-07-25 23:14:39 +00:00

Compare

toasterson force-pushed claude/wi-019f98b5-a-failed-turn-is-indistinguishable-from from c914c46149 to 971c7d59a1

2026-07-28 08:42:31 +00:00

Compare

toasterson changed title from ~~WIP: A failed turn is indistinguishable from an empty one: server ignores stop_reason, WI re-dispatches forever in working~~ to A failed turn is indistinguishable from an empty one: server ignores stop_reason, WI re-dispatches forever in working

2026-07-28 09:07:36 +00:00

toasterson merged commit 28f7237309 into main

2026-07-28 10:10:53 +00:00

toasterson referenced this pull request from a commit

2026-07-28 10:10:55 +00:00

Merge pull request 'A failed turn is indistinguishable from an empty one: server ignores stop_reason, WI re-dispatches forever in `working`' (#366) from claude/wi-019f98b5-a-failed-turn-is-indistinguishable-from into main

toasterson referenced this pull request

2026-07-28 12:03:10 +00:00

Fleet unwedge: Opus 5 on the Claude lane, opencode un-stalled, agent commits no longer dropped #362