WIP: CI verdict pipeline is fully dead: orchestrator crash-loops on RabbitMQ 30-min delivery-ack timeout and every published result is orphaned by re-dispatch superseding its dispatch row #617

Draft

toasterson wants to merge 1 commit from claude/wi-01a00b8c-ci-verdict-pipeline-is-fully-dead-orches into main

toasterson commented

2026-08-18 19:23:00 +00:00

Owner

Anima work item 01a00b8c-ae43-7973-8ff4-cea281643368.

Observed 2026-08-16 during a lane babysit pass. Two interacting defects have taken CI-verdict delivery to 0% for at least 24h — jobs run and finish, but no verdict ever lands on a work item.

Defect A — orchestrator crash loop on delivery-ack timeout (45 restarts in 24h).
solstice-orchestrator consumes with prefetch=8 and (evidently) acks only after a job completes. Jobs run serially, one QEMU VM at a time, ~10–17 min each, so undelivered acks age past RabbitMQ's default 30-min consumer_timeout:

2026-08-16T17:01:17 ERROR io_loop: Channel closed channel=1 method=Close { reply_code: 406,
  reply_text: "PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out.
  Timeout value used: 1800000 ms" }
2026-08-16T17:01:34 INFO orchestrator starting ...

This repeats every ~32 minutes (12:10, 12:42, 13:15, 13:47, 14:19, 14:52, 15:24, 15:55, 16:28, 17:01 …). Each crash requeues the unacked deliveries, so the same jobs re-run repeatedly (e.g. akh-medu sha 37d35ad ran to success at 08-15T23:52, 08-16T03:25, and 08-16T14:08 — three full runs of an identical green).

Defect B — anima-server orphans every late result.
While a job sits in the jammed queue, the server's ai_corrective re-dispatch fires (~20 min cadence) with a NEW request_id and supersedes the WI's solstice_dispatch row. When the original job finally publishes, the row is gone:

2026-08-16T16:53:40 WARN anima_server::solstice: no solstice_dispatch row found; result orphaned request_id=9380ede4-…

Every single result received in the last 24h was orphaned (20+ consecutive, zero result_handled). The re-dispatch then adds another message to the very queue that is over capacity — a positive feedback loop. Dozens of Anima-project WIs plus akh-medu WI-241/267/292/297 are cycling; ~60 blocked claude-code sessions are pinned behind gates that can never clear.

Suggested fixes (any one breaks the loop, all three are right):

Orchestrator: ack on receipt + persist the job, or send periodic basic.ack keepalive alternative (e.g. lower prefetch to 1, or use per-job channels); at minimum raise consumer_timeout on the broker well above worst-case queue-drain latency (operator config change).
anima-server: match incoming results by (work_item, commit_sha) as a fallback when the request_id row is superseded — a green on the current head is a verdict regardless of which dispatch generation ran it. Do not orphan; record with a "late result" annotation.
Dispatch policy: don't re-dispatch on silence while the previous request is still queued/undelivered (needs a queue-depth or claim-age signal); re-dispatching into a saturated queue only deepens the jam.

Related: WI-534 (broker-eaten verdicts, single-loss class — this is the systemic version), WI-548 (archibald disk-bound: 6 concurrent VMs; note the orchestrator currently runs jobs serially, so throughput ~2–3 jobs/30min while dispatch inflow is higher).

Evidence: ssh archibald 'docker logs solstice-orchestrator --since 24h' and kubectl -n anima logs deploy/anima-server --since=24h | grep orphaned.

Anima work item `01a00b8c-ae43-7973-8ff4-cea281643368`. Observed 2026-08-16 during a lane babysit pass. Two interacting defects have taken CI-verdict delivery to 0% for at least 24h — jobs run and finish, but no verdict ever lands on a work item. **Defect A — orchestrator crash loop on delivery-ack timeout (45 restarts in 24h).** solstice-orchestrator consumes with `prefetch=8` and (evidently) acks only after a job completes. Jobs run serially, one QEMU VM at a time, ~10–17 min each, so undelivered acks age past RabbitMQ's default 30-min `consumer_timeout`: ``` 2026-08-16T17:01:17 ERROR io_loop: Channel closed channel=1 method=Close { reply_code: 406, reply_text: "PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms" } 2026-08-16T17:01:34 INFO orchestrator starting ... ``` This repeats every ~32 minutes (12:10, 12:42, 13:15, 13:47, 14:19, 14:52, 15:24, 15:55, 16:28, 17:01 …). Each crash requeues the unacked deliveries, so the same jobs re-run repeatedly (e.g. akh-medu sha 37d35ad ran to success at 08-15T23:52, 08-16T03:25, and 08-16T14:08 — three full runs of an identical green). **Defect B — anima-server orphans every late result.** While a job sits in the jammed queue, the server's ai_corrective re-dispatch fires (~20 min cadence) with a NEW request_id and supersedes the WI's solstice_dispatch row. When the original job finally publishes, the row is gone: ``` 2026-08-16T16:53:40 WARN anima_server::solstice: no solstice_dispatch row found; result orphaned request_id=9380ede4-… ``` Every single result received in the last 24h was orphaned (20+ consecutive, zero result_handled). The re-dispatch then adds another message to the very queue that is over capacity — a positive feedback loop. Dozens of Anima-project WIs plus akh-medu WI-241/267/292/297 are cycling; ~60 blocked claude-code sessions are pinned behind gates that can never clear. **Suggested fixes (any one breaks the loop, all three are right):** 1. Orchestrator: ack on receipt + persist the job, or send periodic basic.ack keepalive alternative (e.g. lower prefetch to 1, or use per-job channels); at minimum raise `consumer_timeout` on the broker well above worst-case queue-drain latency (operator config change). 2. anima-server: match incoming results by (work_item, commit_sha) as a fallback when the request_id row is superseded — a green on the current head is a verdict regardless of which dispatch generation ran it. Do not orphan; record with a "late result" annotation. 3. Dispatch policy: don't re-dispatch on silence while the previous request is still queued/undelivered (needs a queue-depth or claim-age signal); re-dispatching into a saturated queue only deepens the jam. Related: WI-534 (broker-eaten verdicts, single-loss class — this is the systemic version), WI-548 (archibald disk-bound: 6 concurrent VMs; note the orchestrator currently runs jobs serially, so throughput ~2–3 jobs/30min while dispatch inflow is higher). Evidence: `ssh archibald 'docker logs solstice-orchestrator --since 24h'` and `kubectl -n anima logs deploy/anima-server --since=24h | grep orphaned`.

toasterson added 1 commit

2026-08-18 19:23:01 +00:00

CI verdict pipeline is fully dead: orchestrator crash-loops on RabbitMQ 30-min delivery-ack timeout and every published result is orphaned by re-dispatch superseding its dispatch row 5c7276510d

Anima implementation session.

This pull request is marked as a work in progress.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin claude/wi-01a00b8c-ci-verdict-pipeline-is-fully-dead-orches:claude/wi-01a00b8c-ci-verdict-pipeline-is-fully-dead-orches

git switch claude/wi-01a00b8c-ci-verdict-pipeline-is-fully-dead-orches

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff claude/wi-01a00b8c-ci-verdict-pipeline-is-fully-dead-orches

git switch claude/wi-01a00b8c-ci-verdict-pipeline-is-fully-dead-orches

git rebase main

git switch main

git merge --ff-only claude/wi-01a00b8c-ci-verdict-pipeline-is-fully-dead-orches

git switch claude/wi-01a00b8c-ci-verdict-pipeline-is-fully-dead-orches

git rebase main

git switch main

git merge --no-ff claude/wi-01a00b8c-ci-verdict-pipeline-is-fully-dead-orches

git switch main

git merge --squash claude/wi-01a00b8c-ci-verdict-pipeline-is-fully-dead-orches