toasterson/Anima

Watch

Fork 0

A model-provider 429 is indistinguishable from work: the fleet ran 90 minutes on two dead lanes with no signal #384

Merged

toasterson merged 1 commit from claude/wi-019f9aa3-a-model-provider-429-is-indistinguishabl into main

2026-07-28 10:19:25 +00:00

toasterson commented

2026-07-26 06:53:47 +00:00

Owner

Anima work item 019f9aa3-1e67-7362-8a65-2b135a9b6992.

What happened (2026-07-25, ~17:00–19:00)

Throughput went to zero. Nothing in Anima said so. Every surface — runner table, session table, delivery board — reported a healthy, busy fleet.

The actual cause was not in Anima's code. Both model providers were out of quota:

opencode    → ollama-cloud/glm-5.2 → https://ollama.com/v1/chat/completions
             HTTP 429  "you (toasterson) have reached your session usage limit"
claude-code → api.anthropic.com    → HTTP/2 429  rate_limit_error

Probed directly from inside each runner container. GET /v1/models returns 200 in 0.18s — connectivity is fine, entitlement is not, so every naive health check passes.

Why nothing caught it

The ACP handshake does not touch the model. A session therefore reaches running and emits its first three events (session_capabilities, config_options, available_commands_update) normally, then the first prompt turn hits the 429 and the agent swallows it. Observed shape:

WI-11   18:38:21 started → 4 events by 18:38:26 → silence → reaped
WI-248  17:59:14 started → 4 events by 17:59:19 → silence 39m → reaped
WI-337  18:38:32 started → 4 events by 18:38:36 → silence → reaped

Four events and no acp.turn_completed, forever. Compare a healthy session (WI-248 at 17:28): acp.tool_call_update ×N then acp.turn_completed {"stop_reason":"end_turn"}.

Then the reaper abandons the session, the WI returns to dispatchable, and the scheduler sends it straight back into the same wall. This is the re-dispatch half of the fleet silent-failure class (PR #362) — but the trigger is new and is not code: it is a billing state.

A dead lane and an idle lane are the same picture. That is the defect.

Why it took the whole fleet down

ANIMA_SCHED_DEFAULT_EXECUTOR=opencode, and scheduler.rs:241 routes every WI whose executor is empty to the default:

let executor = if wi.executor.is_empty() { default_exec.clone() } else { wi.executor.clone() };

All 14 dispatchable items had executor = ''. So 100% of the backlog funnelled into the one lane with no quota, while runner-1 (4 claude-code slots) and akh-gateway-thoth (6 tecton slots) sat idle — 10 of 13 slots unreachable by data, not by capacity. Pinning one item (Akh-Medu WI-250) to tecton moved it ready → working within one scheduler tick, confirming the slots were always there.

Note the compounding: single default executor + no priority ordering (WI-347) + no quota signal means one exhausted API key silently halts everything.

Verification that this is real, not inferred

12-hour session history by lane:

claude-code  07:00→16:00 sessions completing, then nothing
opencode     →16:00 fine (3/3); 17:00 → 0/2; 18:00 → 0/4
tecton       09:00→15:00 (4/4, 5/5, 5/5) — different backend (akhomed), unaffected

tecton stayed healthy because it does not use either exhausted provider — it reaches akhomed on tcp://127.0.0.1:8201. It pushed real PRs today (akh-medu #240, #253 from WI-220 and WI-249). That is the control group: the outage tracks the provider, not Anima.

What to build

Classify the failure. A turn that ends with no acp.turn_completed must not be recorded the same as one that ends with stop_reason=end_turn. Ties into PR #362 — this is the concrete case that makes it urgent.
Lane health must include entitlement, not just reachability. A per-executor preflight that actually spends one token against the configured model, with the result surfaced as lane state (live / no-quota / unreachable). A 200 on /v1/models must never be allowed to read as green.
Do not re-dispatch into a lane known to be out of quota. Hold the WI as dispatchable and leave the lane marked, rather than burning slots and worktrees in a loop. Today's loop swept 20 stranded worktrees on one runner restart.
Surface it. "All lanes out of quota" is the single most important thing the dashboard can say, and today it said nothing. Relates to the Attention Inbox (ADR 0015).
Reconsider the single default executor. A default that points at one provider makes that provider a single point of failure for the entire backlog.

Acceptance

With a provider key deliberately exhausted, the lane shows no-quota within one scheduler interval.
No WI is dispatched into that lane while it is so marked.
A hung post-handshake session is distinguishable from a completed one in the DB without reading logs.

Evidence trail

Reproduce the probe (from inside the runner container, key via env, never inline):

curl -sS -H "Authorization: Bearer $OLLAMA_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"glm-5.2","messages":[{"role":"user","content":"say ok"}],"max_tokens":10}' \
  https://ollama.com/v1/chat/completions

Operator action needed today (outside this WI): top up ollama.com usage, or wait out the Anthropic subscription window, or repoint OPENCODE_MODEL at a provider with headroom.

REFINEMENT 2026-07-25 19:15 — opencode does not fail on 429, it retries silently and holds the slot

The ollama session limit was a rolling window that reset on its own at ~19:12. Decisive detail: the three stranded sessions (WI-11, WI-248, WI-337) resumed without re-dispatch — same session ids, event counts jumping 4 -> 32 / 4 -> 41 / 4 -> 8 with tool calls flowing.

So opencode was internally retrying the 429 for ~34 minutes, emitting nothing. That is worse than failing:

the session holds a runner slot for the entire outage
it is indistinguishable from a long-running turn at every layer
no backoff is visible, so there is no way to tell a 30-second retry from a 30-minute one
reported_free stays low, so the scheduler believes the fleet is busy rather than blocked

This changes the fix. Recommendation 3 (do not re-dispatch into a no-quota lane) is secondary — nothing was being re-dispatched. The primary fix is to make the retry state observable: the runner must surface provider backoff/retry as session state (e.g. a session.blocked{reason=no_quota,retry_after} event), so a slot held on a 429 is visibly different from a slot doing work.

Corollary for detection: a 429 outage self-heals, so post-hoc log forensics will find nothing. The signal has to be captured live, at the moment the provider returns it.

Probe result at 19:15Z: ollama=200 (recovered), anthropic=429 (still capped).

Anima work item `019f9aa3-1e67-7362-8a65-2b135a9b6992`. ## What happened (2026-07-25, ~17:00–19:00) Throughput went to zero. Nothing in Anima said so. Every surface — runner table, session table, delivery board — reported a healthy, busy fleet. The actual cause was **not** in Anima's code. Both model providers were out of quota: ``` opencode → ollama-cloud/glm-5.2 → https://ollama.com/v1/chat/completions HTTP 429 "you (toasterson) have reached your session usage limit" claude-code → api.anthropic.com → HTTP/2 429 rate_limit_error ``` Probed directly from inside each runner container. `GET /v1/models` returns 200 in 0.18s — **connectivity is fine, entitlement is not**, so every naive health check passes. ## Why nothing caught it The ACP handshake does not touch the model. A session therefore reaches `running` and emits its first three events (`session_capabilities`, `config_options`, `available_commands_update`) *normally*, then the first prompt turn hits the 429 and the agent swallows it. Observed shape: ``` WI-11 18:38:21 started → 4 events by 18:38:26 → silence → reaped WI-248 17:59:14 started → 4 events by 17:59:19 → silence 39m → reaped WI-337 18:38:32 started → 4 events by 18:38:36 → silence → reaped ``` Four events and no `acp.turn_completed`, forever. Compare a healthy session (WI-248 at 17:28): `acp.tool_call_update` ×N then `acp.turn_completed {"stop_reason":"end_turn"}`. Then the reaper abandons the session, the WI returns to dispatchable, and the scheduler sends it straight back into the same wall. This is the re-dispatch half of the fleet silent-failure class (PR #362) — but the *trigger* is new and is not code: it is a billing state. **A dead lane and an idle lane are the same picture.** That is the defect. ## Why it took the whole fleet down `ANIMA_SCHED_DEFAULT_EXECUTOR=opencode`, and `scheduler.rs:241` routes every WI whose `executor` is empty to the default: ```rust let executor = if wi.executor.is_empty() { default_exec.clone() } else { wi.executor.clone() }; ``` All 14 dispatchable items had `executor = ''`. So 100% of the backlog funnelled into the one lane with no quota, while `runner-1` (4 claude-code slots) and `akh-gateway-thoth` (6 tecton slots) sat idle — 10 of 13 slots unreachable **by data, not by capacity**. Pinning one item (`Akh-Medu WI-250`) to `tecton` moved it `ready → working` within one scheduler tick, confirming the slots were always there. Note the compounding: single default executor + no priority ordering (WI-347) + no quota signal means one exhausted API key silently halts everything. ## Verification that this is real, not inferred 12-hour session history by lane: ``` claude-code 07:00→16:00 sessions completing, then nothing opencode →16:00 fine (3/3); 17:00 → 0/2; 18:00 → 0/4 tecton 09:00→15:00 (4/4, 5/5, 5/5) — different backend (akhomed), unaffected ``` tecton stayed healthy because it does not use either exhausted provider — it reaches `akhomed` on `tcp://127.0.0.1:8201`. It pushed real PRs today (akh-medu #240, #253 from WI-220 and WI-249). That is the control group: the outage tracks the *provider*, not Anima. ## What to build 1. **Classify the failure.** A turn that ends with no `acp.turn_completed` must not be recorded the same as one that ends with `stop_reason=end_turn`. Ties into PR #362 — this is the concrete case that makes it urgent. 2. **Lane health must include entitlement, not just reachability.** A per-executor preflight that actually spends one token against the configured model, with the result surfaced as lane state (`live` / `no-quota` / `unreachable`). A 200 on `/v1/models` must never be allowed to read as green. 3. **Do not re-dispatch into a lane known to be out of quota.** Hold the WI as dispatchable and leave the lane marked, rather than burning slots and worktrees in a loop. Today's loop swept 20 stranded worktrees on one runner restart. 4. **Surface it.** "All lanes out of quota" is the single most important thing the dashboard can say, and today it said nothing. Relates to the Attention Inbox (ADR 0015). 5. **Reconsider the single default executor.** A default that points at one provider makes that provider a single point of failure for the entire backlog. ## Acceptance - With a provider key deliberately exhausted, the lane shows `no-quota` within one scheduler interval. - No WI is dispatched into that lane while it is so marked. - A hung post-handshake session is distinguishable from a completed one in the DB without reading logs. ## Evidence trail Reproduce the probe (from inside the runner container, key via env, never inline): ``` curl -sS -H "Authorization: Bearer $OLLAMA_API_KEY" -H "Content-Type: application/json" \ -d '{"model":"glm-5.2","messages":[{"role":"user","content":"say ok"}],"max_tokens":10}' \ https://ollama.com/v1/chat/completions ``` Operator action needed today (outside this WI): top up ollama.com usage, or wait out the Anthropic subscription window, or repoint `OPENCODE_MODEL` at a provider with headroom. --- ## REFINEMENT 2026-07-25 19:15 — opencode does not fail on 429, it retries silently and holds the slot The ollama session limit was a **rolling window that reset on its own** at ~19:12. Decisive detail: the three stranded sessions (WI-11, WI-248, WI-337) **resumed without re-dispatch** — same session ids, event counts jumping 4 -> 32 / 4 -> 41 / 4 -> 8 with tool calls flowing. So opencode was internally retrying the 429 for ~34 minutes, emitting nothing. That is worse than failing: - the session holds a runner slot for the entire outage - it is indistinguishable from a long-running turn at every layer - no backoff is visible, so there is no way to tell a 30-second retry from a 30-minute one - reported_free stays low, so the scheduler believes the fleet is busy rather than blocked **This changes the fix.** Recommendation 3 (do not re-dispatch into a no-quota lane) is secondary — nothing was being re-dispatched. The primary fix is to make the retry state observable: the runner must surface provider backoff/retry as session state (e.g. a session.blocked{reason=no_quota,retry_after} event), so a slot held on a 429 is visibly different from a slot doing work. Corollary for detection: **a 429 outage self-heals**, so post-hoc log forensics will find nothing. The signal has to be captured live, at the moment the provider returns it. Probe result at 19:15Z: ollama=200 (recovered), anthropic=429 (still capped).

toasterson added 1 commit

2026-07-26 06:53:48 +00:00

A model-provider 429 is indistinguishable from work: the fleet ran 90 minutes on two dead lanes with no signal 9b9a1cd4b8

Anima implementation session.

toasterson force-pushed claude/wi-019f9aa3-a-model-provider-429-is-indistinguishabl from 9b9a1cd4b8 to 7ef5d85293

2026-07-28 08:52:01 +00:00

Compare

toasterson commented

2026-07-28 08:52:18 +00:00

Author

Owner

Rebased onto main. The blocker was a wire collision, not a textual conflict, so recording the call here.

The collision

Both sides claimed field 5 on Heartbeat:

main: repeated ExecutorQuota executor_quotas = 5;
this branch: repeated LaneStatus lane_status = 5;

Resolution: keep both, `lane_status` moves to field 6

executor_quotas keeps 5 — it is merged and deployed, so renumbering it would break every attached runner mid-flight. lane_status takes 6.

They are complementary, not alternatives, and neither derives the other:

executor_quotas is a slow-timer measurement of account usage (the provider's usage endpoint, polled off the heartbeat path).
lane_status is a state derived from real turn outcomes.
A 429 can land well before reported usage reaches its ceiling — a per-minute rate limit is not weekly quota exhaustion.
UNREACHABLE has no quota analogue at all.
A runner with no probe configured reports empty quotas while still reporting lane state.

The proto diff against main is now purely additive — no field renumbered, no message removed.

Merge notes

Heartbeat carries both; update_heartbeat takes both (quotas: Vec<ExecutorQuota>, lane_status: &[LaneStatus]).
Dispatch predicates keep main's per-executor slot accounting and gain the branch's lane gate: claim_slot and claim_slot_for_review now require executor_effective_free_for(...) > 0 && !h.lane_health.is_sick(executor).
RunnerSnapshot carries executor_quotas and lane_health side by side.
slot.rs: a blocked: close takes precedence over the failed-no-attempt: tagging from WI 019fa2fc — both comments retained on the merged match.
PWA bindings regenerated (agent_pb/dashboard_pb/runner_pb); verified main's committed gen was already in sync first, so the diff here is only this branch's own proto change.

Verification: cargo check -p anima-runner -p anima-server --all-targets clean; cargo test -p anima-server --lib 165 passed (incl. 5 new lane_health tests); cargo test -p anima-runner 73 passed; npm run build in anima-pwa clean. The server suite needing a Postgres container per test was not run.

Context: 2026-07-28 sweep of the 11 open PRs that could no longer rebase onto main.

Rebased onto `main`. The blocker was a **wire collision**, not a textual conflict, so recording the call here. ### The collision Both sides claimed **field 5 on `Heartbeat`**: - `main`: `repeated ExecutorQuota executor_quotas = 5;` - this branch: `repeated LaneStatus lane_status = 5;` ### Resolution: keep both, `lane_status` moves to field 6 `executor_quotas` keeps 5 — it is merged *and deployed*, so renumbering it would break every attached runner mid-flight. `lane_status` takes 6. They are complementary, not alternatives, and neither derives the other: - `executor_quotas` is a slow-timer **measurement** of account usage (the provider's usage endpoint, polled off the heartbeat path). - `lane_status` is a **state** derived from real turn outcomes. - A 429 can land well before reported usage reaches its ceiling — a per-minute rate limit is not weekly quota exhaustion. - `UNREACHABLE` has no quota analogue at all. - A runner with no probe configured reports empty quotas while still reporting lane state. The proto diff against `main` is now **purely additive** — no field renumbered, no message removed. ### Merge notes - `Heartbeat` carries both; `update_heartbeat` takes both (`quotas: Vec<ExecutorQuota>`, `lane_status: &[LaneStatus]`). - Dispatch predicates keep `main`'s per-executor slot accounting **and** gain the branch's lane gate: `claim_slot` and `claim_slot_for_review` now require `executor_effective_free_for(...) > 0 && !h.lane_health.is_sick(executor)`. - `RunnerSnapshot` carries `executor_quotas` and `lane_health` side by side. - `slot.rs`: a `blocked:` close takes precedence over the `failed-no-attempt:` tagging from WI 019fa2fc — both comments retained on the merged match. - PWA bindings regenerated (`agent_pb`/`dashboard_pb`/`runner_pb`); verified `main`'s committed gen was already in sync first, so the diff here is only this branch's own proto change. **Verification:** `cargo check -p anima-runner -p anima-server --all-targets` clean; `cargo test -p anima-server --lib` 165 passed (incl. 5 new `lane_health` tests); `cargo test -p anima-runner` 73 passed; `npm run build` in `anima-pwa` clean. The server suite needing a Postgres container per test was not run. Context: 2026-07-28 sweep of the 11 open PRs that could no longer rebase onto `main`.

toasterson force-pushed claude/wi-019f9aa3-a-model-provider-429-is-indistinguishabl from 7ef5d85293 to 6e5662b902

2026-07-28 10:10:23 +00:00

Compare

toasterson changed title from ~~WIP: A model-provider 429 is indistinguishable from work: the fleet ran 90 minutes on two dead lanes with no signal~~ to A model-provider 429 is indistinguishable from work: the fleet ran 90 minutes on two dead lanes with no signal

2026-07-28 10:13:59 +00:00

toasterson force-pushed claude/wi-019f9aa3-a-model-provider-429-is-indistinguishabl from 6e5662b902 to 78ec05adf7

2026-07-28 10:17:29 +00:00

Compare

toasterson force-pushed claude/wi-019f9aa3-a-model-provider-429-is-indistinguishabl from 78ec05adf7 to 246cbf0fb0

2026-07-28 10:18:53 +00:00

Compare

toasterson merged commit 189f9e380e into main

2026-07-28 10:19:25 +00:00

toasterson referenced this pull request from a commit

2026-07-28 10:19:27 +00:00

Merge pull request 'A model-provider 429 is indistinguishable from work: the fleet ran 90 minutes on two dead lanes with no signal' (#384) from claude/wi-019f9aa3-a-model-provider-429-is-indistinguishabl into main