The CI-red corrective loop hands the implementer no log at all — every retry after a red build is a blind guess #411

Merged

toasterson merged 1 commit from claude/wi-019fa902-the-ci-red-corrective-loop-hands-the-imp into main

2026-07-29 17:20:32 +00:00

toasterson commented

2026-07-28 22:40:43 +00:00

Owner

Anima work item 019fa902-43b4-7283-a662-8da24798f35f.

Symptom

When CI goes red, handle_job_failure appends a "CI Failure (attempt N)" block to the work item's description so the corrective context rides the implementer's next forcing prompt. That block contains one line and no log. Every such revision written today, measured on 2026-07-28:

WI-352  len=180  exit_code=1.   ``` job script failed: exit_code=1 — …ERROR workflow failed exit_code=1 ```
WI-413  len=186  exit_code=101. ``` job script failed: exit_code=101 — …ERROR workflow failed exit_code=101 ```
WI-308  len=192  exit_code=1.   ``` ssh connect failed: SSH operation failed: connect 127.0.0.1:35491: … ```
WI-315  len=186  exit_code=101. ``` job script failed: exit_code=101 — …ERROR workflow failed exit_code=101 ```
WI-315  len=186  (attempt 2, byte-identical shape)
WI-413  len=186  (attempt 2, byte-identical shape)

Six revisions, 180–192 bytes each. The template's own prose — "Fix the failure below, then push again." — is followed by nothing to fix. The implementer is told a build failed and is given no compiler error, no failing test name, no assertion, no stack.

WI-413 has burned three CI attempts this way and WI-315 two. Each attempt is a full session: a slot, a clone, a build, a push, a CI run. The retry cannot converge because the loop never tells the agent what broke.

Root cause

The wire type carries no log. JobResult (vendor/solstice-ci/crates/common/src/messages.rs:166) has success, exit_code and summary: Option<String> — and no log field of any kind. Anima's shim makes that explicit:

// crates/anima-server/src/solstice.rs:705
impl JobResultExt for JobResult {
    fn combined_log_or_default(&self) -> String {
        self.summary.clone().unwrap_or_default()
    }
}

So tail_lines(&result.combined_log_or_default(), 60) at solstice.rs:546 asks for the last 60 lines of a string that is at most one line long, and usually is the runner's own top-level "workflow failed" line rather than anything from the build. summary is null outright on some results.

The logs do exist — solstice stores them per step; list_steps / tail_step_log serve them for source=forgejo runs. They are simply never fetched for the native (solstice_dispatch) lane, which is the lane every Anima work item uses. A native run's request_id is not resolvable through solstice-mcp either (run not found for e34b42b0-a065-40d1-aa50-44896c077aca), so the native lane's logs are currently unreachable from Anima and from the operator tooling.

Ask

Get the log to Anima. Either extend JobResult with the tail the runner already has in hand, or have Anima fetch it by request_id from solstice after the result lands. Prefer the fetch: an unbounded log on a message bus is its own problem, and the fetch also gives the operator surface back.
Make the native lane's runs addressable by request_id in solstice's job/step store, so list_steps / tail_step_log work for them the way they do for the Forgejo lane.
Fail loudly when there is no log. combined_log_or_default() silently substituting summary (and summary silently substituting "") is what made this invisible for as long as it has been. If no log can be retrieved, say so in the corrective block — "CI failed and no log could be retrieved (request_id=…)" is honest; a code fence containing one line of nothing is not.
Only then is retrying worth the slot: today failed_dispatch_count climbs toward the park cap on evidence the agent was never shown.

Acceptance criteria

A CI-red corrective revision for a failing cargo test contains the failing test's name and its assertion output.
A CI-red corrective revision for a compile error contains the error[EXXXX] block and the file:line.
tail_step_log returns step output for a native solstice_dispatch run given its request_id.
A result that genuinely carries no log produces a revision that says so explicitly, and that case is distinguishable in the logs from a successful retrieval.
Regression test: a JobResult with summary: None must not produce a corrective block that claims to contain a failure.

WI-340 / PR #377 — classifying infra vs job failures. Adjacent and complementary: #377 decides whether to charge the work item, this decides whether the retry can possibly work. Both are needed; neither substitutes for the other.
WI-364 — the AMQP result consumer's reconnect gap, same transport.
project_false_park_amplifiers family: this is a park amplifier, since three blind retries reach the cap.

Anima work item `019fa902-43b4-7283-a662-8da24798f35f`. ## Symptom When CI goes red, `handle_job_failure` appends a "CI Failure (attempt N)" block to the work item's description so the corrective context rides the implementer's next forcing prompt. That block contains **one line and no log**. Every such revision written today, measured on 2026-07-28: ``` WI-352 len=180 exit_code=1. ``` job script failed: exit_code=1 — …ERROR workflow failed exit_code=1 ``` WI-413 len=186 exit_code=101. ``` job script failed: exit_code=101 — …ERROR workflow failed exit_code=101 ``` WI-308 len=192 exit_code=1. ``` ssh connect failed: SSH operation failed: connect 127.0.0.1:35491: … ``` WI-315 len=186 exit_code=101. ``` job script failed: exit_code=101 — …ERROR workflow failed exit_code=101 ``` WI-315 len=186 (attempt 2, byte-identical shape) WI-413 len=186 (attempt 2, byte-identical shape) ``` Six revisions, 180–192 bytes each. The template's own prose — "Fix the failure below, then push again." — is followed by nothing to fix. The implementer is told a build failed and is given no compiler error, no failing test name, no assertion, no stack. WI-413 has burned **three** CI attempts this way and WI-315 two. Each attempt is a full session: a slot, a clone, a build, a push, a CI run. The retry cannot converge because the loop never tells the agent what broke. ## Root cause The wire type carries no log. `JobResult` (`vendor/solstice-ci/crates/common/src/messages.rs:166`) has `success`, `exit_code` and `summary: Option<String>` — and no log field of any kind. Anima's shim makes that explicit: ```rust // crates/anima-server/src/solstice.rs:705 impl JobResultExt for JobResult { fn combined_log_or_default(&self) -> String { self.summary.clone().unwrap_or_default() } } ``` So `tail_lines(&result.combined_log_or_default(), 60)` at `solstice.rs:546` asks for the last 60 lines of a string that is at most one line long, and usually is the runner's own top-level "workflow failed" line rather than anything from the build. `summary` is `null` outright on some results. The logs do exist — solstice stores them per step; `list_steps` / `tail_step_log` serve them for `source=forgejo` runs. They are simply never fetched for the native (`solstice_dispatch`) lane, which is the lane every Anima work item uses. A native run's `request_id` is not resolvable through `solstice-mcp` either (`run not found` for `e34b42b0-a065-40d1-aa50-44896c077aca`), so the native lane's logs are currently unreachable from Anima *and* from the operator tooling. ## Ask 1. **Get the log to Anima.** Either extend `JobResult` with the tail the runner already has in hand, or have Anima fetch it by `request_id` from solstice after the result lands. Prefer the fetch: an unbounded log on a message bus is its own problem, and the fetch also gives the operator surface back. 2. **Make the native lane's runs addressable by `request_id`** in solstice's job/step store, so `list_steps` / `tail_step_log` work for them the way they do for the Forgejo lane. 3. **Fail loudly when there is no log.** `combined_log_or_default()` silently substituting `summary` (and `summary` silently substituting `""`) is what made this invisible for as long as it has been. If no log can be retrieved, say so in the corrective block — "CI failed and no log could be retrieved (request_id=…)" is honest; a code fence containing one line of nothing is not. 4. Only then is retrying worth the slot: today `failed_dispatch_count` climbs toward the park cap on evidence the agent was never shown. ## Acceptance criteria - A CI-red corrective revision for a failing `cargo test` contains the failing test's name and its assertion output. - A CI-red corrective revision for a compile error contains the `error[EXXXX]` block and the file:line. - `tail_step_log` returns step output for a native `solstice_dispatch` run given its `request_id`. - A result that genuinely carries no log produces a revision that says so explicitly, and that case is distinguishable in the logs from a successful retrieval. - Regression test: a `JobResult` with `summary: None` must not produce a corrective block that claims to contain a failure. ## Related - WI-340 / PR #377 — classifying infra vs job failures. Adjacent and complementary: #377 decides *whether* to charge the work item, this decides *whether the retry can possibly work*. Both are needed; neither substitutes for the other. - WI-364 — the AMQP result consumer's reconnect gap, same transport. - [[project_false_park_amplifiers]] family: this is a park amplifier, since three blind retries reach the cap.

toasterson added 1 commit

2026-07-28 22:40:48 +00:00

The CI-red corrective loop hands the implementer no log at all — every retry after a red build is a blind guess c02e9efed5

Anima implementation session.

toasterson changed title from ~~WIP: The CI-red corrective loop hands the implementer no log at all — every retry after a red build is a blind guess~~ to The CI-red corrective loop hands the implementer no log at all — every retry after a red build is a blind guess

2026-07-28 23:13:43 +00:00

toasterson merged commit a55d584e85 into main

2026-07-29 17:20:32 +00:00

toasterson referenced this pull request from a commit

2026-07-29 17:20:34 +00:00

Merge pull request 'The CI-red corrective loop hands the implementer no log at all — every retry after a red build is a blind guess' (#411) from claude/wi-019fa902-the-ci-red-corrective-loop-hands-the-imp into main