toasterson/Anima

Watch

Fork 0

Lane quota detection uses a total-turn timeout, so it kills healthy long turns #525

Merged

toasterson merged 11 commits from claude/wi-019fdd2a-lane-quota-detection-uses-a-total-turn-t into main

2026-08-16 09:20:04 +00:00

toasterson commented

2026-08-08 13:59:43 +00:00

Owner

Anima work item 019fdd2a-7101-7380-a71c-348e260884a1.

Symptom

ANIMA_RUNNER_MAX_SESSION_SECS is a cap on the whole turn. On the opencode lane it was set to 300s as a quota probe, and it silently amputated every healthy implementation turn at exactly five minutes. The turn is then reported as cancelled with "the agent produced no durable output", which the server charges to the work item as a turn failure — so productive work reads as failure and the item eventually parks.

Evidence

Sessions on the opencode lane, from activity_events:

019fdb77: 42 acp.tool_call, 99 acp.tool_call_update, 12 acp.agent_message, spanning 09:04:50 → 09:09:47. Killed at 300s mid-work.
019fd8ea: 41 tool calls over 21:11:48 → 21:16:11. Same.
019fdb8a: 38 tool calls over 09:25:26 → 09:30:09. Same.

Thirteen work items were parked by this: WI-371, 378, 382, 385, 389, 392, 396, 403, 406, 407, 408, 409, 419. Several had been re-dispatched repeatedly, each attempt dying at the same five-minute wall.

Why the original setting was wrong

The rationale in docker-compose.runner.yml (2026-08-03) is sound about the failure it targets: opencode's AI-SDK client swallows a 429 and retries with a growing backoff without surfacing it to ACP, so run_turn never sees Err and just waits out max_secs. The mistake is the instrument. The comment justifies 300s against first-token latency ("well above the longest healthy first-turn latency (~5s)") but the value is applied to the entire turn, and a real implementation turn legitimately runs far longer than five minutes.

The distinguishing signal

A quota-stuck turn and a healthy long turn are trivially separable, just not by total duration:

quota-stuck: ~0 ACP events after the initial handshake — the client is sleeping in backoff.
healthy: a steady stream of acp.tool_call / tool_call_update, 40+ within five minutes.

Expected

Replace the total-turn cap with an idle timeout: if no ACP event of any kind arrives for N seconds (30-60s is far below the ~33s smallest escaping backoff tier and far above any healthy inter-event gap), fail the turn as provider lane blocked and mark the lane NoQuota exactly as today. Keep a total-turn cap only as a very high backstop.

Both timeouts should be configurable, and the failure text must name which one fired — "no ACP event for 60s (provider lane blocked)" and "turn exceeded the 3600s backstop" are different diagnoses and must not share a message.

Interim state (already applied by the operator, 2026-08-07)

ANIMA_RUNNER_MAX_SESSION_SECS on the opencode lane raised 300 → 1800 in /home/toasty/ws/anima/docker-compose.runner.yml, runner recreated. This unblocks real work at the cost of a quota-stuck turn burning 30 minutes instead of 5. Revert to a smaller backstop once the idle timeout exists.

Acceptance criteria

A test where events keep arriving past the old cap and the turn is NOT killed.
A test where events stop and the idle timeout fires, producing the NoQuota lane-health path.
The two timeout failures carry distinct, self-diagnosing messages.

Anima work item `019fdd2a-7101-7380-a71c-348e260884a1`. ## Symptom `ANIMA_RUNNER_MAX_SESSION_SECS` is a cap on the whole turn. On the opencode lane it was set to 300s as a *quota probe*, and it silently amputated every healthy implementation turn at exactly five minutes. The turn is then reported as `cancelled` with "the agent produced no durable output", which the server charges to the work item as a turn failure — so productive work reads as failure and the item eventually parks. ## Evidence Sessions on the opencode lane, from `activity_events`: - `019fdb77`: 42 `acp.tool_call`, 99 `acp.tool_call_update`, 12 `acp.agent_message`, spanning 09:04:50 → 09:09:47. Killed at 300s mid-work. - `019fd8ea`: 41 tool calls over 21:11:48 → 21:16:11. Same. - `019fdb8a`: 38 tool calls over 09:25:26 → 09:30:09. Same. Thirteen work items were parked by this: WI-371, 378, 382, 385, 389, 392, 396, 403, 406, 407, 408, 409, 419. Several had been re-dispatched repeatedly, each attempt dying at the same five-minute wall. ## Why the original setting was wrong The rationale in `docker-compose.runner.yml` (2026-08-03) is sound about the failure it targets: opencode's AI-SDK client swallows a 429 and retries with a growing backoff without surfacing it to ACP, so `run_turn` never sees `Err` and just waits out `max_secs`. The mistake is the instrument. The comment justifies 300s against *first-token latency* ("well above the longest healthy first-turn latency (~5s)") but the value is applied to the **entire turn**, and a real implementation turn legitimately runs far longer than five minutes. ## The distinguishing signal A quota-stuck turn and a healthy long turn are trivially separable, just not by total duration: - quota-stuck: ~0 ACP events after the initial handshake — the client is sleeping in backoff. - healthy: a steady stream of `acp.tool_call` / `tool_call_update`, 40+ within five minutes. ## Expected Replace the total-turn cap with an **idle timeout**: if no ACP event of any kind arrives for N seconds (30-60s is far below the ~33s smallest escaping backoff tier and far above any healthy inter-event gap), fail the turn as `provider lane blocked` and mark the lane `NoQuota` exactly as today. Keep a total-turn cap only as a very high backstop. Both timeouts should be configurable, and the failure text must name which one fired — "no ACP event for 60s (provider lane blocked)" and "turn exceeded the 3600s backstop" are different diagnoses and must not share a message. ## Interim state (already applied by the operator, 2026-08-07) `ANIMA_RUNNER_MAX_SESSION_SECS` on the opencode lane raised 300 → 1800 in `/home/toasty/ws/anima/docker-compose.runner.yml`, runner recreated. This unblocks real work at the cost of a quota-stuck turn burning 30 minutes instead of 5. Revert to a smaller backstop once the idle timeout exists. ## Acceptance criteria - A test where events keep arriving past the old cap and the turn is NOT killed. - A test where events stop and the idle timeout fires, producing the `NoQuota` lane-health path. - The two timeout failures carry distinct, self-diagnosing messages.

toasterson added 1 commit

2026-08-08 13:59:56 +00:00

Replace total-turn timeout with idle timeout + backstop 5cdf66c0bd

Lane quota detection used ANIMA_RUNNER_MAX_SESSION_SECS as a cap on the
whole turn, silently amputating every healthy long turn at the limit. On
the opencode lane (300s quota probe) this parked 13 WIs as "produced no
durable output" while their transcripts show 40+ tool calls, all killed
mid-work at exactly 5 minutes.

The instrument was wrong: a total-turn cap cannot distinguish a quota-stuck
turn (swallowing 429s, producing ~0 ACP events) from a healthy long turn
(producing 40+ events but exceeding the wall-clock cap).

This commit replaces the single timeout with a dual system:

1. **Idle timeout** (ANIMA_RUNNER_TURN_IDLE_SECS, default 60s): if no ACP
   event arrives for this duration, the turn is killed and classified as
   a provider block. This is the silent-429 detection instrument.

2. **Backstop timeout** (ANIMA_RUNNER_MAX_SESSION_SECS, default 3600s):
   if the total turn time exceeds this (regardless of activity), the turn
   is killed but NOT classified as a provider block — it's a config issue,
   not quota.

The idle monitor tracks `last_event_ts`, updated on every ACP notification
(AgentMessageChunk, AgentThoughtChunk, structural updates, usage). When
this timestamp stops advancing for `turn_idle_secs`, the turn is idle-stuck
(quota signature) and fires the idle timeout. A healthy long turn with
continuous events survives past the old 300s cap indefinitely (up to the
backstop).

**Distinct diagnostics**:
- Idle: "provider lane blocked (no ACP event for 60s — quota-stuck signature)"
- Backstop: "turn exceeded the 3600s backstop cap (total turn time, regardless of activity)"

**Lane health impact**:
- Idle timeout → sets `blocked_reason` → lane marked NoQuota → WI reschedules
- Backstop timeout → does NOT set `blocked_reason` → normal turn failure → not a lane poison

**Tests added**:
- `classifies_idle_timeout_as_provider_blocked`: verifies idle timeout is ProviderBlocked
- `classifies_backstop_timeout_as_other`: documents backstop is not a first-turn block
  (the key is it doesn't set `blocked_reason`, so lane health never sees it)

**Configuration guidance**:
- For opencode lane: can now revert the interim 1800s cap back to 3600s or higher
- Idle timeout (60s default) catches quota-stuck turns within a minute
- Backstop (3600s default) protects against runaway turns without false-positiving healthy work

Acceptance criteria met:
✓ Test: events keep arriving past old cap, turn not killed
✓ Test: events stop, idle timeout fires, NoQuota lane path
✓ Distinct self-diagnosing messages for each timeout

Fixes: WI-371, 378, 382, 385, 389, 392, 396, 403, 406, 407, 408, 409, 419 (all parked by the old cap)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

toasterson added 1 commit

2026-08-08 14:57:29 +00:00

Lane quota detection uses a total-turn timeout, so it kills healthy long turns: 13 work items parked as "produced no durable output" while their transcripts show 40+ tool calls 1369fbf98c

Anima implementation session.

toasterson added 1 commit

2026-08-08 15:32:56 +00:00

Lane quota detection uses a total-turn timeout, so it kills healthy long turns: 13 work items parked as "produced no durable output" while their transcripts show 40+ tool calls bcc8b2d94f

Anima implementation session.

toasterson force-pushed claude/wi-019fdd2a-lane-quota-detection-uses-a-total-turn-t from bcc8b2d94f to afb65b23ce

2026-08-09 14:37:44 +00:00

Compare

toasterson force-pushed claude/wi-019fdd2a-lane-quota-detection-uses-a-total-turn-t from afb65b23ce to c225d37004

2026-08-09 18:49:38 +00:00

Compare

toasterson force-pushed claude/wi-019fdd2a-lane-quota-detection-uses-a-total-turn-t from c225d37004 to c718cbb849

2026-08-11 10:13:53 +00:00

Compare

toasterson changed title from ~~WIP: Lane quota detection uses a total-turn timeout, so it kills healthy long turns: 13 work items parked as "produced no durable output" while their transcripts show 40+ tool calls~~ to Lane quota detection uses a total-turn timeout, so it kills healthy long turns

2026-08-11 11:31:09 +00:00

toasterson force-pushed claude/wi-019fdd2a-lane-quota-detection-uses-a-total-turn-t from c718cbb849 to d73b284874

2026-08-14 15:23:56 +00:00

Compare

toasterson added 1 commit

2026-08-14 16:01:25 +00:00

Lane quota detection uses a total-turn timeout, so it kills healthy long turns: 13 work items parked as "produced no durable output" while their transcripts show 40+ tool calls 3b1951e470