toasterson/Anima

Watch

Fork 0

CI-red corrective loop must distinguish infra failures from job failures (auto-retry CI on infra flake, don't park) #377

Merged

toasterson merged 2 commits from claude/wi-019f9698-ci-red-corrective-loop-must-distinguish into main

2026-07-28 15:25:15 +00:00

toasterson commented

2026-07-26 02:46:22 +00:00

Owner

Anima work item 019f9698-6f38-70d3-a1f8-468e966dabc2.

CORRECTION 2026-07-25 14:50 — read before the rest

The closing note of the earlier revision claimed that, with default_ci_steps fixed, "a fresh 101 is no longer plausible and the one WI-328 saw could only have been historical."

That was wrong. Fresh 101s are not merely plausible; they are essentially the entire failure population. Measured on solstice_prod for the last 24 hours:

Anima native (runs_on=rust) failures     = 24
of which are the protoc/proto3-optional  = 23
of which are a VM boot failure           =  1
of which were caused by the code         =  0

Root cause of the 23: .solstice/job.sh installed protoc from jammy (3.12), which rejects the optional string project_id = 1; added to thread.proto by 5c1dd37 on 2026-07-11. The lane's last green run was the same day. Fixed in PR #369.

The 1 remaining failure (9beabe2f, commit 333fcbaf) was a VM that failed to boot — Dependency failed for /boot/efi, GPT errors — i.e. the second infra class this work item already describes.

So the sharpest statement of the problem is: not one native CI failure in the last 24 hours was caused by the code under test, and every implementer handed "CI red, fix your build" was told something false. That is a stronger mandate for the classification work below than the original two anecdotes.

Confirmed cost

Seven work items auto-parked in 24 hours via auto-parked: CI-red failed dispatch cap reached, five of them Anima:

parked	work item
11:06	WI-B: Executor token scope stripping
11:13	WI-C: Runner interceptor + checkpoint
12:19	WI-A: Backend escalation core
13:22	Reopen work items when their orphaned sessions…
14:40	WI-D: Frontend permission card

WI-A through WI-D is the entire ACP escalation slice, parked in one day for reasons unconnected to its code. Sessions were burned first — one ran 2200s chasing the phantom.

Note this also revises the WI-328 story below: that exit_code=101 was most likely a real, current protoc failure rather than a replayed three-week-old verdict. The stale-verdict defect (fix 5) is still worth implementing — commit-scoping is correct regardless — but it should no longer be presented as the confirmed explanation of what WI-328 saw. The wrong conclusion came from checking the Forgejo Actions lane, which was green throughout, instead of the native lane, which was 100% red.

Problem

When a native-dispatch Solstice CI job fails at the harness/infrastructure layer (the build never ran), the CI-red corrective loop treats it identically to a real code failure. It (1) counts the failure toward the park cap, and (2) re-dispatches the implementer with an infra error as "corrective context" — which the implementer cannot act on. If the implementer then produces no new commit, no new CI run ever fires and the WI silently stalls in working indefinitely.

The protoc evidence above generalises this: a toolchain failure baked into the runner image is neither an infra flake nor a code failure, and it is invisible to both classifications. It must land in the same "not the implementer's problem" bucket.

Observed (2026-07-24)

WI-A 019f83ce-6c98-7113-a424-1be8d78685fb (escalation backend). Both CI attempts died at VM SSH bring-up under archibald I/O load — the job script never executed:

JobResult jobresult.v1: success=false exit_code=1, summary="ssh connect failed: SSH operation failed: connect 127.0.0.1:42993: Connection reset by peer (os error 104)".
After the 19:07 failure the WI sat untouched for >5 hours with no re-dispatch, despite the corrective loop logging solstice.ci_red_redispatched. Manual intervention (clear the solstice_dispatch row + reset failed_dispatch_count) was required to get another CI run.

Even a healthy fleet will hit transient VM-bring-up flakes; the loop must absorb them, not stall.

Root cause (code)

crates/anima-server/src/solstice.rs (JobResult handling, ~L140–260):

On any success=false, calls WorkItemRepo::increment_failed_dispatch_count(pool, wi, cap) (cap = ANIMA_SCHED_FAILED_DISPATCH_CAP, default 3) and re-dispatches the implementer with the log tail. There is no branch on why the job failed.
reconcile_missing_ci (crates/anima-server/src/scheduler.rs) only re-dispatches WIs whose head has no solstice_dispatch row — so once a (failed) dispatch row exists for the head, nothing re-triggers CI for that commit.

The JobResult already carries enough to classify: an infra failure has a harness summary (ssh connect failed…, VM never booted) and the steps never ran, vs. a job failure where the steps ran and returned non-zero.

Proposed fix

Classify JobResults into infra_failure (harness/VM/SSH/runner error — build did not run), toolchain_failure (steps ran but died in environment setup, before the project's own code was compiled), and job_failure (the project's build/test returned non-zero). Prefer an explicit signal from the runner/orchestrator over string-matching summary.
On infra_failure or toolchain_failure: do NOT increment failed_dispatch_count, and do NOT re-prompt the implementer. Auto re-dispatch the same commit's CI with exponential backoff, up to a separate cap. Only job_failure goes through the implementer-corrective loop + park cap.
Bound retries: after N retries, surface the WI as needs-human with a clear reason (CI infrastructure failing / CI toolchain broken) — never park it as if its code failed.
Ensure a re-CI path exists for a head that already has a (superseded) solstice_dispatch row.
Scope every CI verdict to the commit it belongs to. The forcing prompt must only carry a CI failure whose commit_sha equals the work item's current head. If there is no CI result for the current head, say exactly that rather than reaching backwards.
Carry the commit_sha and the run's timestamp alongside any CI text put in front of an agent.
Alert when a lane's failure rate saturates. 23/24 identical failures over 24h is a signal no dashboard raised. A lane failing >80% with a single recurring signature is a broken lane, not N broken work items, and must be surfaced as one incident rather than N parks.

Acceptance criteria

A JobResult whose build never ran does not increment the park counter and does not re-prompt the implementer.
A toolchain failure (e.g. protoc too old, missing rustfmt) is classified separately from a code failure and does not park the WI.
Such a WI auto-retries its CI with backoff and lands green once the underlying problem is fixed, with no manual DB surgery.
After the retry cap, the WI is flagged needs-human with a reason distinct from a code-failure park.
A genuine job failure still flows through the existing loop and park cap unchanged.
A work item whose head has no CI result is never given another commit's verdict. Regression test: seed an old failed run for commit A, advance the head to commit B with no run, assemble the prompt, assert it contains no CI failure text and names commit B.
A lane whose recent failures share one signature raises a single lane-level alert.
Regression test: simulate each of the three JobResult classes and assert the distinct handling.

Files

crates/anima-server/src/solstice.rs (result handling / classification / re-dispatch)
crates/anima-server/src/dispatch.rs (re-dispatch CI vs implementer; head-revision scoping)
crates/anima-server/src/scheduler.rs (reconcile_missing_ci predicate)
crates/anima-solstice/src/results.rs + jobresult.v1 shape (explicit failure-class signal)

PR #369 — the protoc fix. Merging it stops the bleeding; this work item stops it recurring.
019f9980 — pin CI verdicts to the commit under test, and surface native vs Actions lane health separately.
019f9079 micro-VM executor SPIKE (removes the I/O-saturation root of the flakiness).
019f8602 reopen WIs when orphaned sessions fail (same family) — itself one of the five parked.
019f903c-eeb1-7e71-928b-e248db024384 — fleet observability.
019f98b5-2ccd-7d80-9842-ec3e395b0d12 — a failed turn reads as an empty one.

Anima work item `019f9698-6f38-70d3-a1f8-468e966dabc2`. ## CORRECTION 2026-07-25 14:50 — read before the rest The closing note of the earlier revision claimed that, with `default_ci_steps` fixed, "a fresh 101 is no longer plausible and the one WI-328 saw could only have been historical." **That was wrong.** Fresh 101s are not merely plausible; they are essentially the entire failure population. Measured on `solstice_prod` for the last 24 hours: ``` Anima native (runs_on=rust) failures = 24 of which are the protoc/proto3-optional = 23 of which are a VM boot failure = 1 of which were caused by the code = 0 ``` Root cause of the 23: `.solstice/job.sh` installed protoc from jammy (3.12), which rejects the `optional string project_id = 1;` added to `thread.proto` by 5c1dd37 on 2026-07-11. The lane's last green run was the same day. Fixed in PR #369. The 1 remaining failure (`9beabe2f`, commit 333fcbaf) was a VM that failed to boot — `Dependency failed for /boot/efi`, GPT errors — i.e. the second infra class this work item already describes. **So the sharpest statement of the problem is: not one native CI failure in the last 24 hours was caused by the code under test, and every implementer handed "CI red, fix your build" was told something false.** That is a stronger mandate for the classification work below than the original two anecdotes. ### Confirmed cost Seven work items auto-parked in 24 hours via `auto-parked: CI-red failed dispatch cap reached`, five of them Anima: | parked | work item | |---|---| | 11:06 | WI-B: Executor token scope stripping | | 11:13 | WI-C: Runner interceptor + checkpoint | | 12:19 | WI-A: Backend escalation core | | 13:22 | Reopen work items when their orphaned sessions… | | 14:40 | WI-D: Frontend permission card | WI-A through WI-D is the entire ACP escalation slice, parked in one day for reasons unconnected to its code. Sessions were burned first — one ran 2200s chasing the phantom. Note this also revises the WI-328 story below: that `exit_code=101` was most likely a **real, current** protoc failure rather than a replayed three-week-old verdict. The stale-verdict defect (fix 5) is still worth implementing — commit-scoping is correct regardless — but it should no longer be presented as the confirmed explanation of what WI-328 saw. The wrong conclusion came from checking the Forgejo Actions lane, which was green throughout, instead of the native lane, which was 100% red. --- ## Problem When a native-dispatch Solstice CI job fails at the **harness/infrastructure** layer (the build never ran), the CI-red corrective loop treats it identically to a real code failure. It (1) counts the failure toward the park cap, and (2) re-dispatches the *implementer* with an infra error as "corrective context" — which the implementer cannot act on. If the implementer then produces no new commit, no new CI run ever fires and the WI **silently stalls in `working` indefinitely**. The protoc evidence above generalises this: a *toolchain* failure baked into the runner image is neither an infra flake nor a code failure, and it is invisible to both classifications. It must land in the same "not the implementer's problem" bucket. ## Observed (2026-07-24) WI-A `019f83ce-6c98-7113-a424-1be8d78685fb` (escalation backend). Both CI attempts died at VM SSH bring-up under archibald I/O load — the job script never executed: - JobResult `jobresult.v1`: `success=false exit_code=1`, `summary="ssh connect failed: SSH operation failed: connect 127.0.0.1:42993: Connection reset by peer (os error 104)"`. - After the 19:07 failure the WI sat untouched for **>5 hours** with no re-dispatch, despite the corrective loop logging `solstice.ci_red_redispatched`. Manual intervention (clear the `solstice_dispatch` row + reset `failed_dispatch_count`) was required to get another CI run. Even a healthy fleet will hit transient VM-bring-up flakes; the loop must absorb them, not stall. ## Root cause (code) `crates/anima-server/src/solstice.rs` (JobResult handling, ~L140–260): - On any `success=false`, calls `WorkItemRepo::increment_failed_dispatch_count(pool, wi, cap)` (cap = `ANIMA_SCHED_FAILED_DISPATCH_CAP`, default 3) and re-dispatches the implementer with the log tail. There is no branch on *why* the job failed. - `reconcile_missing_ci` (`crates/anima-server/src/scheduler.rs`) only re-dispatches WIs whose head has **no** `solstice_dispatch` row — so once a (failed) dispatch row exists for the head, nothing re-triggers CI for that commit. The `JobResult` already carries enough to classify: an infra failure has a harness `summary` (`ssh connect failed…`, VM never booted) and the steps never ran, vs. a job failure where the steps ran and returned non-zero. ## Proposed fix 1. **Classify JobResults** into `infra_failure` (harness/VM/SSH/runner error — build did not run), `toolchain_failure` (steps ran but died in environment setup, before the project's own code was compiled), and `job_failure` (the project's build/test returned non-zero). Prefer an explicit signal from the runner/orchestrator over string-matching `summary`. 2. **On `infra_failure` or `toolchain_failure`:** do NOT increment `failed_dispatch_count`, and do NOT re-prompt the implementer. Auto re-dispatch the same commit's CI with exponential backoff, up to a separate cap. Only `job_failure` goes through the implementer-corrective loop + park cap. 3. **Bound retries:** after N retries, surface the WI as **needs-human with a clear reason** (`CI infrastructure failing` / `CI toolchain broken`) — never park it as if its code failed. 4. Ensure a re-CI path exists for a head that already has a (superseded) `solstice_dispatch` row. 5. **Scope every CI verdict to the commit it belongs to.** The forcing prompt must only carry a CI failure whose `commit_sha` equals the work item's current head. If there is no CI result for the current head, say exactly that rather than reaching backwards. 6. Carry the `commit_sha` and the run's timestamp alongside any CI text put in front of an agent. 7. **Alert when a lane's failure rate saturates.** 23/24 identical failures over 24h is a signal no dashboard raised. A lane failing >80% with a single recurring signature is a broken lane, not N broken work items, and must be surfaced as one incident rather than N parks. ## Acceptance criteria - A JobResult whose build never ran does not increment the park counter and does not re-prompt the implementer. - A toolchain failure (e.g. protoc too old, missing rustfmt) is classified separately from a code failure and does not park the WI. - Such a WI auto-retries its CI with backoff and lands green once the underlying problem is fixed, with no manual DB surgery. - After the retry cap, the WI is flagged needs-human with a reason distinct from a code-failure park. - A genuine job failure still flows through the existing loop and park cap unchanged. - **A work item whose head has no CI result is never given another commit's verdict.** Regression test: seed an old failed run for commit A, advance the head to commit B with no run, assemble the prompt, assert it contains no CI failure text and names commit B. - A lane whose recent failures share one signature raises a single lane-level alert. - Regression test: simulate each of the three JobResult classes and assert the distinct handling. ## Files - `crates/anima-server/src/solstice.rs` (result handling / classification / re-dispatch) - `crates/anima-server/src/dispatch.rs` (re-dispatch CI vs implementer; head-revision scoping) - `crates/anima-server/src/scheduler.rs` (`reconcile_missing_ci` predicate) - `crates/anima-solstice/src/results.rs` + `jobresult.v1` shape (explicit failure-class signal) ## Related - PR #369 — the protoc fix. Merging it stops the bleeding; this work item stops it recurring. - `019f9980` — pin CI verdicts to the commit under test, and surface native vs Actions lane health separately. - `019f9079` micro-VM executor SPIKE (removes the I/O-saturation root of the flakiness). - `019f8602` reopen WIs when orphaned sessions fail (same family) — itself one of the five parked. - `019f903c-eeb1-7e71-928b-e248db024384` — fleet observability. - `019f98b5-2ccd-7d80-9842-ec3e395b0d12` — a failed turn reads as an empty one.

toasterson added 1 commit

2026-07-26 02:46:28 +00:00

CI-red corrective loop must distinguish infra failures from job failures (auto-retry CI on infra flake, don't park) 5b0c31cfbb

Anima implementation session.

toasterson force-pushed claude/wi-019f9698-ci-red-corrective-loop-must-distinguish from 5b0c31cfbb to 1f6f390e1d

2026-07-26 03:59:38 +00:00

Compare

toasterson force-pushed claude/wi-019f9698-ci-red-corrective-loop-must-distinguish from 1f6f390e1d to b3b765a2bf

2026-07-28 08:07:00 +00:00

Compare

toasterson force-pushed claude/wi-019f9698-ci-red-corrective-loop-must-distinguish from b3b765a2bf to 12846323af

2026-07-28 09:15:13 +00:00

Compare

toasterson force-pushed claude/wi-019f9698-ci-red-corrective-loop-must-distinguish from 12846323af to b67112c5a7

2026-07-28 12:13:31 +00:00

Compare

toasterson added 1 commit

2026-07-28 14:53:01 +00:00

CI-red corrective loop must distinguish infra failures from job failures (auto-retry CI on infra flake, don't park) ada0dfbce8

Anima implementation session.

toasterson changed title from ~~WIP: CI-red corrective loop must distinguish infra failures from job failures (auto-retry CI on infra flake, don't park)~~ to CI-red corrective loop must distinguish infra failures from job failures (auto-retry CI on infra flake, don't park)

2026-07-28 15:20:32 +00:00

toasterson merged commit 7251147d83 into main

2026-07-28 15:25:15 +00:00

toasterson referenced this pull request from a commit

2026-07-28 15:25:15 +00:00

Merge pull request 'CI-red corrective loop must distinguish infra failures from job failures (auto-retry CI on infra flake, don't park)' (#377) from claude/wi-019f9698-ci-red-corrective-loop-must-distinguish into main

toasterson referenced this pull request

2026-07-28 22:40:43 +00:00

The CI-red corrective loop hands the implementer no log at all — every retry after a red build is a blind guess #411

toasterson referenced this pull request

2026-08-04 16:01:02 +00:00

The janitor's two sweeps both exclude the shared cargo target dir: it regrows ~75G/day and took archibald to 0 bytes, killing Forgejo, the registry and CI #496