Pin CI verdicts to the commit under test, and surface native vs Actions lane health separately #380

Merged

toasterson merged 1 commit from claude/wi-019f9980-pin-ci-verdicts-to-the-commit-under-test into main

2026-07-27 10:48:57 +00:00

toasterson commented

2026-07-26 04:33:35 +00:00

Owner

Anima work item 019f9980-ba66-7f20-90d9-d0abfe7fc35c.

Correction history — read this first

This work item has been wrong twice, in opposite directions. Both errors came from checking the wrong CI lane.

Filing 1 (wrong): claimed Anima's native CI targeted a non-existent runner label and had never produced a run. False — the rust label works, solstice-rust.qcow2 exists, and 199 jobs have run on it. That conclusion came from list_runs(owner, repo), which cannot see native-dispatch rows because they leave repo_owner/repo_name NULL.

Filing 2 (also wrong): claimed WI-328 had been handed a stale 2026-07-04 verdict while "CI was in fact green". The green signal was the Forgejo Actions lane. The native lane was genuinely red at the time.

What is actually true: the native rust lane last passed 2026-07-11 20:56 and has run 36 jobs with 0 successes since 2026-07-20. Root cause was protoc 3.12 from jammy rejecting the proto3 optional field added to thread.proto by commit 5c1dd37 on 2026-07-11. Fixed in PR #369. So the exit_code=101 fed to implementers was a real, current failure, not a stale row.

The lesson worth keeping: Anima has two CI lanes with independent health, and the one everybody looks at (Actions, green) is not the one the machine exit bar consumes (native, red for two weeks). A green badge on one lane actively concealed a total outage on the other.

Remaining scope — still worth doing

Two defects surfaced by the above are real regardless of the protoc fix:

1. Verdicts are not pinned to the commit under test. Nothing checks that a CI result's commit_sha matches the commit the work item is on. Combined with Solstice job rows that never reach a terminal state (96 running rows against 5 live VMs, four stuck since 2026-07-05 — filed as solstice-ci 019f998b-9cea-7bf3-87c4-c6a6ad24f0ef), a verdict for a different or long-dead commit can be presented as current.

2. "No verdict" and "failed" are indistinguishable to an implementer. That conflation is what makes a wrong verdict expensive rather than merely confusing.

Required

Match a CI verdict to the commit currently under test. A verdict whose commit_sha is not the work item's HEAD is not a result for this work item — treat it as "no verdict yet", never as a failure.
Never surface a verdict older than the work item's latest push as current.
Distinguish "no verdict yet" from "verdict: failed" in whatever the implementer sees.
Surface per-lane CI health separately. If the native lane is dark or failing while Actions is green, that must be visible rather than averaged away into a single "CI status".

Acceptance criteria

Given a work item at commit X with a failure recorded only for commit W, the exit bar reports no verdict for X rather than a failure.
Implementer context never contains a verdict whose commit_sha differs from the commit being worked.
A zombie running row upstream produces "pending", not pass or fail.
Native-lane and Actions-lane health are separately observable; a two-week native outage cannot again be invisible behind a green Actions badge.

PR #369 — the protoc fix that unstuck the native lane.
solstice-ci 019f998b-be04-7f70-8138-27774a142d85 — NULL repo attribution, which is what made the native lane invisible to repo-scoped queries and caused correction 1.
solstice-ci 019f998b-9cea-7bf3-87c4-c6a6ad24f0ef — jobs table never reconciles against live VMs.
019f9698-6f38-70d3-a1f8-468e966dabc2 — CI-red corrective loop.
019f903c-eeb1-7e71-928b-e248db024384 — fleet observability.

Anima work item `019f9980-ba66-7f20-90d9-d0abfe7fc35c`. ## Correction history — read this first This work item has been wrong twice, in opposite directions. Both errors came from checking the wrong CI lane. **Filing 1 (wrong):** claimed Anima's native CI targeted a non-existent runner label and had never produced a run. False — the `rust` label works, `solstice-rust.qcow2` exists, and 199 jobs have run on it. That conclusion came from `list_runs(owner, repo)`, which cannot see native-dispatch rows because they leave `repo_owner`/`repo_name` NULL. **Filing 2 (also wrong):** claimed WI-328 had been handed a *stale* 2026-07-04 verdict while "CI was in fact green". The green signal was the **Forgejo Actions** lane. The **native** lane was genuinely red at the time. **What is actually true:** the native `rust` lane last passed 2026-07-11 20:56 and has run 36 jobs with 0 successes since 2026-07-20. Root cause was protoc 3.12 from jammy rejecting the proto3 `optional` field added to `thread.proto` by commit 5c1dd37 on 2026-07-11. Fixed in PR #369. So the `exit_code=101` fed to implementers was a **real, current failure**, not a stale row. The lesson worth keeping: Anima has two CI lanes with independent health, and the one everybody looks at (Actions, green) is not the one the machine exit bar consumes (native, red for two weeks). A green badge on one lane actively concealed a total outage on the other. ## Remaining scope — still worth doing Two defects surfaced by the above are real regardless of the protoc fix: **1. Verdicts are not pinned to the commit under test.** Nothing checks that a CI result's `commit_sha` matches the commit the work item is on. Combined with Solstice job rows that never reach a terminal state (96 `running` rows against 5 live VMs, four stuck since 2026-07-05 — filed as solstice-ci `019f998b-9cea-7bf3-87c4-c6a6ad24f0ef`), a verdict for a different or long-dead commit can be presented as current. **2. "No verdict" and "failed" are indistinguishable to an implementer.** That conflation is what makes a wrong verdict expensive rather than merely confusing. ### Required - Match a CI verdict to the commit currently under test. A verdict whose `commit_sha` is not the work item's HEAD is not a result for this work item — treat it as "no verdict yet", never as a failure. - Never surface a verdict older than the work item's latest push as current. - Distinguish "no verdict yet" from "verdict: failed" in whatever the implementer sees. - Surface per-lane CI health separately. If the native lane is dark or failing while Actions is green, that must be visible rather than averaged away into a single "CI status". ### Acceptance criteria - Given a work item at commit X with a failure recorded only for commit W, the exit bar reports no verdict for X rather than a failure. - Implementer context never contains a verdict whose commit_sha differs from the commit being worked. - A zombie `running` row upstream produces "pending", not pass or fail. - Native-lane and Actions-lane health are separately observable; a two-week native outage cannot again be invisible behind a green Actions badge. ## Related - PR #369 — the protoc fix that unstuck the native lane. - solstice-ci `019f998b-be04-7f70-8138-27774a142d85` — NULL repo attribution, which is what made the native lane invisible to repo-scoped queries and caused correction 1. - solstice-ci `019f998b-9cea-7bf3-87c4-c6a6ad24f0ef` — jobs table never reconciles against live VMs. - `019f9698-6f38-70d3-a1f8-468e966dabc2` — CI-red corrective loop. - `019f903c-eeb1-7e71-928b-e248db024384` — fleet observability.

toasterson added 1 commit

2026-07-26 04:33:35 +00:00

fix(solstice): pin CI verdicts to commit under test, surface lane health 967682cf9e

Two defects from WI 019f9980:

1. Verdicts were not pinned to the commit under test. A JobResult whose
   commit_sha is not the WI's current HEAD was applied as a verdict
   anyway — combined with solstice zombie `running` rows
   (019f998b-9cea) this let a stale or mismatched commit be presented as
   a current failure. Now `handle_result` compares the result SHA
   against both the dispatched commit and the WI's latest forgejo_pr
   head; on mismatch it records the row (audit) but does not touch the
   exit bar, failed_dispatch_count, or the implementer's forcing prompt.

2. "No verdict" and "failed" were indistinguishable. Add a CI Pending
   block (stripped on the next push alongside CI Failure / Review Round)
   so the implementer sees pending, not a stale failure, when a verdict
   is for a different commit.

3. Per-lane CI health is now observable separately. Each solstice_result
   row carries `lane: native|actions` and `pinned_to_current` so a
   native outage can't be averaged away behind a green Actions badge.

Tests: pin_verdict_* (6), lane_of_* (3), strip_verdict_blocks_removes_ci_pending.

toasterson changed title from ~~WIP: Pin CI verdicts to the commit under test, and surface native vs Actions lane health separately~~ to Pin CI verdicts to the commit under test, and surface native vs Actions lane health separately

2026-07-27 10:48:51 +00:00

toasterson merged commit 61e2c00789 into main

2026-07-27 10:48:57 +00:00

toasterson referenced this pull request from a commit

2026-07-27 10:49:01 +00:00

Merge pull request 'Pin CI verdicts to the commit under test, and surface native vs Actions lane health separately' (#380) from claude/wi-019f9980-pin-ci-verdicts-to-the-commit-under-test into main