The account-quota probe polls the provider every 15 min per executor forever, earning 429s and then 403s — cache it, back it off, and stop probing an idle lane #491

Closed

toasterson wants to merge 4 commits from claude/wi-019fc826-the-account-quota-probe-polls-the-provid into main

toasterson commented

2026-08-03 17:14:50 +00:00

Owner

Anima work item 019fc826-cf80-74d0-afd5-81760aea35f8.

Symptom

runner-1 probes account usage for claude-code every 15 minutes, indefinitely,
and the provider has started refusing:

2026-08-03T08:16:54  WARN anima_runner::quota: account quota unavailable executor=claude-code
                     provider=anthropic-oauth note=provider rate-limited the usage probe (429)
2026-08-03T09:16:54  ... note=usage endpoint returned HTTP 403 Forbidden
2026-08-03T09:31:54  ... note=usage endpoint returned HTTP 403 Forbidden
2026-08-03T09:46:55  ... note=usage endpoint returned HTTP 403 Forbidden
2026-08-03T10:16:55  ... note=provider rate-limited the usage probe (429)
2026-08-03T12:16:55  ... note=usage endpoint returned HTTP 403 Forbidden
2026-08-03T13:16:56  ... note=provider rate-limited the usage probe (429)

Alternating 429 (rate limited) and 403 (Forbidden), all day. The same
pattern, 429-only, appeared on 2026-08-01.

Scope — read this before starting

Dispatch is NOT broken. An earlier draft of this work item claimed the lane had
gone dark; that was an operator error reading a quiet window. runner-1 dispatched
and pushed Anima/pulls/487 at 15:02 the same day. This is consistent with
crates/anima-runner/src/quota.rs:631, where Outcome::Throttled | Outcome::Failed
only log — only Outcome::Unsupported retires an executor. The design is already
correct on that point.

So this is a self-inflicted-traffic and observability defect, not a dispatch
defect. Do not go looking for a scheduler bug.

The actual problems

We are rate-limiting ourselves. Every runner probes every advertised executor
every 15 minutes whether or not it is dispatching anything. Nothing caches the last
good answer; nothing backs off when the provider pushes back. Sustained 429s
plausibly escalated into the 403s.
403 is not 429. They are handled identically and differ only in a note string.
429 means "slow down"; 403 means the credential is wrong, expired, or under-scoped
— a condition that will never resolve by retrying on the same cadence.
The log becomes noise. For long stretches the quota WARN is the only line the
runner emits, which makes a genuinely quiet lane and a broken one look identical.
That is precisely what caused the misreading above.

Required

Cache the last successful reading and serve it with an age. A probe is a
refresh, not a precondition.
Exponential backoff with jitter on failure, capped (e.g. 15m → 30m → 1h → 4h).
Only probe when it matters — on attach, and around actual dispatch. An idle
lane should generate no provider traffic.
Handle 403 distinctly: surface it once as an operator-visible executor state
(credential problem), not as a recurring WARN.
Keep dispatch independent, and prove it — a test asserting a lane whose probe
fails still receives assignments, so this stays true under future edits.

Acceptance

Probe traffic to the provider drops by at least an order of magnitude on an idle
lane.
A 403 produces one clear state, not a WARN every 15 minutes.
Repeated probe failure demonstrably does not affect dispatch (covered by test).

Context

Raised by Till 2026-08-03: "probably a rate-limit against the anima runners frequent
usage questions." Related: WI-430 (lane circuit-breaker). Separately worth having —
and NOT part of this item — a signal for "connected lane, eligible work queued, zero
dispatches for N minutes", which would have prevented the misreading that produced the
first draft of this work item.

Anima work item `019fc826-cf80-74d0-afd5-81760aea35f8`. ## Symptom `runner-1` probes account usage for `claude-code` every 15 minutes, indefinitely, and the provider has started refusing: ``` 2026-08-03T08:16:54 WARN anima_runner::quota: account quota unavailable executor=claude-code provider=anthropic-oauth note=provider rate-limited the usage probe (429) 2026-08-03T09:16:54 ... note=usage endpoint returned HTTP 403 Forbidden 2026-08-03T09:31:54 ... note=usage endpoint returned HTTP 403 Forbidden 2026-08-03T09:46:55 ... note=usage endpoint returned HTTP 403 Forbidden 2026-08-03T10:16:55 ... note=provider rate-limited the usage probe (429) 2026-08-03T12:16:55 ... note=usage endpoint returned HTTP 403 Forbidden 2026-08-03T13:16:56 ... note=provider rate-limited the usage probe (429) ``` Alternating **429** (rate limited) and **403** (Forbidden), all day. The same pattern, 429-only, appeared on 2026-08-01. ## Scope — read this before starting **Dispatch is NOT broken.** An earlier draft of this work item claimed the lane had gone dark; that was an operator error reading a quiet window. `runner-1` dispatched and pushed `Anima/pulls/487` at 15:02 the same day. This is consistent with `crates/anima-runner/src/quota.rs:631`, where `Outcome::Throttled | Outcome::Failed` only log — **only `Outcome::Unsupported` retires an executor.** The design is already correct on that point. So this is a **self-inflicted-traffic and observability** defect, not a dispatch defect. Do not go looking for a scheduler bug. ## The actual problems 1. **We are rate-limiting ourselves.** Every runner probes every advertised executor every 15 minutes whether or not it is dispatching anything. Nothing caches the last good answer; nothing backs off when the provider pushes back. Sustained 429s plausibly escalated into the 403s. 2. **403 is not 429.** They are handled identically and differ only in a note string. 429 means "slow down"; 403 means the credential is wrong, expired, or under-scoped — a condition that will never resolve by retrying on the same cadence. 3. **The log becomes noise.** For long stretches the quota WARN is the *only* line the runner emits, which makes a genuinely quiet lane and a broken one look identical. That is precisely what caused the misreading above. ## Required 1. **Cache the last successful reading** and serve it with an age. A probe is a refresh, not a precondition. 2. **Exponential backoff with jitter on failure**, capped (e.g. 15m → 30m → 1h → 4h). 3. **Only probe when it matters** — on attach, and around actual dispatch. An idle lane should generate no provider traffic. 4. **Handle 403 distinctly**: surface it once as an operator-visible executor state (credential problem), not as a recurring WARN. 5. **Keep dispatch independent, and prove it** — a test asserting a lane whose probe fails still receives assignments, so this stays true under future edits. ## Acceptance - Probe traffic to the provider drops by at least an order of magnitude on an idle lane. - A 403 produces one clear state, not a WARN every 15 minutes. - Repeated probe failure demonstrably does not affect dispatch (covered by test). ## Context Raised by Till 2026-08-03: *"probably a rate-limit against the anima runners frequent usage questions."* Related: WI-430 (lane circuit-breaker). Separately worth having — and NOT part of this item — a signal for "connected lane, eligible work queued, zero dispatches for N minutes", which would have prevented the misreading that produced the first draft of this work item.

toasterson added 2 commits

2026-08-03 17:14:50 +00:00

The account-quota probe hammers the provider every 15 min per executor, gets 429/403, and the lane goes dark — cache, back off, and never let a probe failure affect dispatch 1e9afeb028

Anima implementation session.

quota: fix idle backoff multiplier doc to match the value (2x, not 1.5x) f4daaec0f5

The IDLE_BACKOFF_MULTIPLIER constant is 2, but its doc comment said
1.5x. The wait_for_stretches_when_idle test asserts wi >= wa, which holds
for both, but the doc should describe the actual behaviour. No behaviour
change.

Re-pushes WI 019fc826 after attempt 1's CI failure (exit_code=1 with no
retrievable log — SOLSTICE_LOGS_URL unset; reproduced locally as a
shared cargo-target-dir fingerprint collision with a sibling worktree's
anima-core, not a code defect).

toasterson changed title from ~~WIP: The account-quota probe polls the provider every 15 min per executor forever, earning 429s and then 403s — cache it, back it off, and stop probing an idle lane~~ to The account-quota probe polls the provider every 15 min per executor forever, earning 429s and then 403s — cache it, back it off, and stop probing an idle lane

2026-08-04 05:29:37 +00:00

toasterson force-pushed claude/wi-019fc826-the-account-quota-probe-polls-the-provid from 8285841109

CI / conflict-check (push) Successful in 2m30s

Details

CI / docker (push) Successful in 31m43s

Details

to de07363aaf

2026-08-05 18:32:32 +00:00

Compare

toasterson force-pushed claude/wi-019fc826-the-account-quota-probe-polls-the-provid from de07363aaf to 132111626f

2026-08-05 20:45:50 +00:00

Compare