toasterson/Anima

Watch

Fork 0

WIP: Fleet observability: metrics + alerts for runner attach, delivery outcome, unschedulable queue #367

Draft

toasterson wants to merge 1 commit from claude/wi-019f903c-fleet-observability-metrics-alerts-for-r into main

toasterson commented

2026-07-25 13:08:20 +00:00

Owner

Anima work item 019f903c-eeb1-7e71-928b-e248db024384.

The 2026-07-23 outage (PR #358) was invisible for 4.5h: a runner detached, the fleet idled, and the only signal was 26,570 identical log lines nobody was watching. Non-delivery of a turn's work is a failure mode on par with an aborted turn and must be measured, not swallowed (today "no changes to push this turn" masked both idle and broken turns).

Prometheus + Grafana already in-cluster; anima-server sees every session transition and (per the micro-VM design) will receive a Forgejo webhook when a branch ref advances, so metrics are server-emitted (a /metrics endpoint), not log-derived.

Delivery authority (decided 2026-07-23): "delivered" = anima-server observed the branch ref advance via a Forgejo webhook, NOT the runner's self-reported push result. A runner that dies mid-push cannot fake success.

Alerts — deliberately MINIMAL (Till's call 2026-07-23). No AI triage exists yet and Till handles every alert by hand, so do NOT alert on many things. Exactly two alerts:

Aborts piling up — high aborted-turn rate over a window.
Queue not draining — items are queued (ready/working WIs) but the queue is not being processed (nothing moving). i.e. "it should move but doesn't." This single alert subsumes the 2026-07-23 condition (ready work + no connected runner advertising the executor) and any other stall.

Metrics to expose (for dashboards + to compute the two alerts), without alerting on each:

connected-runner count per executor; queue depth per executor.
turn outcome counter {delivered (ref advanced), no-op, aborted, push-failed}.
delivery latency (assign → ref advanced); rebind/abandon counter.

Everything beyond the two alerts is dashboard-only until an AI log-investigator exists to triage a noisier alert set.

Field evidence, 2026-07-25 — it happened again, for 38 hours

The 07-23 outage repeated at larger scale and was again invisible. Nothing merged to main between 2026-07-23 21:22 and 2026-07-25 11:23 while every surface read "working". All three lanes were failing, each differently, each silently:

claude lane: expired OAuth, ~20 consecutive 401 turns
opencode lane: every turn stalled to the 3600s cap, zero output
akh lane: 17 WIs auto-parked on retry-budget after producing work that was discarded

Why nothing could have caught it: anima-server has no metrics at all. There is no /metrics endpoint, no ServiceMonitor, no metrics crate — crates/anima-server/src/services/workitem.rs:1304 says so outright:

// No metrics crate is wired into anima-server yet;
// this structured log line is the observable proxy until a proper OTEL histogram is added.

So the platform's only telemetry channel is structured log lines mined out of Loki — and loki-0 had been in CreateContainerConfigError for nine days (dangling secret ref, fixed 2026-07-25 in homelab b55c0dd). For nine days Anima had literally zero observability of any kind, and the log-derived-metric workaround this WI was written to replace did not exist either.

That makes the /metrics endpoint the load-bearing part of this WI, not a nice-to-have: it is the only proposal here that does not depend on the log pipeline being alive.

Three signals that would each have caught 2026-07-25 on day one

Throughput. Merges to main per day. It was 0 for 38h against a normally-busy fleet. Cheapest possible signal, and it is the one that actually matters.
Turn outcome by stop_reason, per lane. The runner already classifies correctly (acp.rs → end_turn / error / cancelled); the server currently throws the value away (019f98b5-2ccd-7d80-9842-ec3e395b0d12). Counted per lane, the claude lane's 401s and the opencode lane's timeouts would each have been an unmistakable flat line of failures.
Park reason rate. 17 WIs hitting retry-budget inside a day is a scoring bug, not a work problem (019f990a-3948-7a73-9a6a-f57287250435). A counter by parked_reason makes that shape obvious.

Add last-successful-delivery age per lane to the dashboard: a lane whose last delivery is >6h old while queue depth is non-zero is exactly the "queue not draining" alert, computed per lane instead of globally — it would have fired three times over.

019f98b5-2ccd-7d80-9842-ec3e395b0d12 — the server ignores stop_reason; prerequisite for signal 2.
019f98d9-fe07-7651-aa05-9b34be1ede89 — no phase history; prerequisite for any time-in-phase or cycle-time panel.
019f990a-3948-7a73-9a6a-f57287250435 — analysis WIs park themselves; source of signal 3.
homelab 6b8db10 — the cluster-level half is already done: container-stuck-waiting and Loki-not-ingesting alerts, both on VictoriaMetrics so they survive the log pipeline being the broken thing.

Anima work item `019f903c-eeb1-7e71-928b-e248db024384`. The 2026-07-23 outage (PR #358) was invisible for 4.5h: a runner detached, the fleet idled, and the only signal was 26,570 identical log lines nobody was watching. Non-delivery of a turn's work is a failure mode on par with an aborted turn and must be measured, not swallowed (today "no changes to push this turn" masked both idle and broken turns). Prometheus + Grafana already in-cluster; anima-server sees every session transition and (per the micro-VM design) will receive a Forgejo webhook when a branch ref advances, so metrics are server-emitted (a /metrics endpoint), not log-derived. **Delivery authority (decided 2026-07-23):** "delivered" = anima-server observed the branch ref advance via a **Forgejo webhook**, NOT the runner's self-reported push result. A runner that dies mid-push cannot fake success. **Alerts — deliberately MINIMAL (Till's call 2026-07-23).** No AI triage exists yet and Till handles every alert by hand, so do NOT alert on many things. Exactly two alerts: 1. **Aborts piling up** — high aborted-turn rate over a window. 2. **Queue not draining** — items are queued (ready/working WIs) but the queue is not being processed (nothing moving). i.e. "it should move but doesn't." This single alert subsumes the 2026-07-23 condition (ready work + no connected runner advertising the executor) and any other stall. Metrics to expose (for dashboards + to compute the two alerts), without alerting on each: - connected-runner count per executor; queue depth per executor. - turn outcome counter {delivered (ref advanced), no-op, aborted, push-failed}. - delivery latency (assign → ref advanced); rebind/abandon counter. Everything beyond the two alerts is dashboard-only until an AI log-investigator exists to triage a noisier alert set. --- ## Field evidence, 2026-07-25 — it happened again, for 38 hours The 07-23 outage repeated at larger scale and was again invisible. Nothing merged to `main` between 2026-07-23 21:22 and 2026-07-25 11:23 while every surface read "working". All three lanes were failing, each differently, each silently: - claude lane: expired OAuth, ~20 consecutive 401 turns - opencode lane: every turn stalled to the 3600s cap, zero output - akh lane: 17 WIs auto-parked on `retry-budget` after producing work that was discarded **Why nothing could have caught it: anima-server has no metrics at all.** There is no `/metrics` endpoint, no ServiceMonitor, no metrics crate — `crates/anima-server/src/services/workitem.rs:1304` says so outright: > `// No metrics crate is wired into anima-server yet;` > `// this structured log line is the observable proxy until a proper OTEL histogram is added.` So the platform's only telemetry channel is structured log lines mined out of Loki — **and `loki-0` had been in CreateContainerConfigError for nine days** (dangling secret ref, fixed 2026-07-25 in homelab b55c0dd). For nine days Anima had literally zero observability of any kind, and the log-derived-metric workaround this WI was written to replace did not exist either. That makes the `/metrics` endpoint the load-bearing part of this WI, not a nice-to-have: it is the only proposal here that does not depend on the log pipeline being alive. ### Three signals that would each have caught 2026-07-25 on day one 1. **Throughput.** Merges to `main` per day. It was 0 for 38h against a normally-busy fleet. Cheapest possible signal, and it is the one that actually matters. 2. **Turn outcome by `stop_reason`, per lane.** The runner already classifies correctly (`acp.rs` → `end_turn` / `error` / cancelled); the server currently throws the value away (`019f98b5-2ccd-7d80-9842-ec3e395b0d12`). Counted per lane, the claude lane's 401s and the opencode lane's timeouts would each have been an unmistakable flat line of failures. 3. **Park reason rate.** 17 WIs hitting `retry-budget` inside a day is a scoring bug, not a work problem (`019f990a-3948-7a73-9a6a-f57287250435`). A counter by `parked_reason` makes that shape obvious. Add **last-successful-delivery age per lane** to the dashboard: a lane whose last delivery is >6h old while queue depth is non-zero is exactly the "queue not draining" alert, computed per lane instead of globally — it would have fired three times over. ## Related - `019f98b5-2ccd-7d80-9842-ec3e395b0d12` — the server ignores `stop_reason`; **prerequisite** for signal 2. - `019f98d9-fe07-7651-aa05-9b34be1ede89` — no phase history; prerequisite for any time-in-phase or cycle-time panel. - `019f990a-3948-7a73-9a6a-f57287250435` — analysis WIs park themselves; source of signal 3. - homelab `6b8db10` — the cluster-level half is already done: container-stuck-waiting and Loki-not-ingesting alerts, both on VictoriaMetrics so they survive the log pipeline being the broken thing.

toasterson added 1 commit

2026-07-25 13:08:24 +00:00

feat(observability): Prometheus metrics + alerts for fleet health 76011c8b87

Add fleet observability to detect invisible outages (2026-07-23 4.5h,
2026-07-25 38h). Exposes /metrics endpoint with runner attachment,
queue depth, turn outcomes, delivery latency, and park metrics.
Two critical alerts: AnimaAbortsHigh + AnimaQueueNotDraining.

## What's implemented

✅ Prometheus metrics registry (crates/anima-server/src/metrics.rs)
✅ /metrics HTTP endpoint on MCP port (unauthenticated for scraping)
✅ Runner attach/detach → anima_connected_runners per executor
✅ Webhook handler scaffold for Forgejo push events (ref advance = delivered)
✅ ServiceMonitor for Prometheus Operator (30s scrape interval)
✅ PrometheusRule with two alerts + recording rules
✅ Grafana dashboard JSON (6 panels: runners, queue, outcomes, latency, age, parks)
✅ Documentation: docs/observability.md

## What's TODO (follow-up WIs)

⚠️ Webhook → work item lookup (match branch → executor, emit delivery metrics)
⚠️ Park metrics emission on WorkItemService phase transition
⚠️ Queue depth periodic update from scheduler
⚠️ Turn outcome metrics on RunnerService TurnDone (map stop_reason)
⚠️ Mount POST /webhooks/forgejo/push route on MCP HTTP server

## Why delivery is webhook-driven (not runner-reported)

Per 2026-07-23 decision: a runner that dies mid-push cannot report failure.
The Forgejo webhook is the ONLY authoritative signal that the ref advanced.

## Why these two alerts (not more)

Till handles alerts manually (no AI triage). AnimaQueueNotDraining catches
the CRITICAL failure mode (both outages). AnimaAbortsHigh catches symptoms
(401s, timeouts, scoring bugs). Everything else is dashboard-only until an
AI log-investigator exists to triage a noisier alert set.

## Deployment

Requires Prometheus Operator in-cluster. Add to values.yaml:
  prometheus:
    enabled: true

Import Grafana dashboard:
  deploy/grafana-dashboard-anima-fleet.json

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

toasterson force-pushed claude/wi-019f903c-fleet-observability-metrics-alerts-for-r from 76011c8b87 to 0ef402c447

2026-07-25 13:52:03 +00:00

Compare

toasterson force-pushed claude/wi-019f903c-fleet-observability-metrics-alerts-for-r from 0ef402c447 to 9f92f78649

2026-07-25 14:39:23 +00:00

Compare

toasterson force-pushed claude/wi-019f903c-fleet-observability-metrics-alerts-for-r from 9f92f78649 to a3d1f81bf7

2026-07-25 21:10:55 +00:00

Compare

toasterson force-pushed claude/wi-019f903c-fleet-observability-metrics-alerts-for-r from a3d1f81bf7 to 4f85668ccf

2026-07-25 22:35:29 +00:00

Compare

toasterson force-pushed claude/wi-019f903c-fleet-observability-metrics-alerts-for-r from 4f85668ccf to 127334e9c0

2026-07-28 12:10:11 +00:00

Compare

This pull request has changes conflicting with the target branch.

Cargo.lock
crates/anima-server/src/mcp/server.rs
crates/anima-server/src/services/runner.rs

View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Merge commit title

Merge pull request 'WIP: Fleet observability: metrics + alerts for runner attach, delivery outcome, unschedulable queue' (#367) from claude/wi-019f903c-fleet-observability-metrics-alerts-for-r into main

Merge commit body

Reviewed-on: https://code.aopc.cloud/toasterson/Anima/pulls/367

Merge pull request 'WIP: Fleet observability: metrics + alerts for runner attach, delivery outcome, unschedulable queue' (#367) from claude/wi-019f903c-fleet-observability-metrics-alerts-for-r into main

Reviewed-on: https://code.aopc.cloud/toasterson/Anima/pulls/367

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin claude/wi-019f903c-fleet-observability-metrics-alerts-for-r:claude/wi-019f903c-fleet-observability-metrics-alerts-for-r

git switch claude/wi-019f903c-fleet-observability-metrics-alerts-for-r

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff claude/wi-019f903c-fleet-observability-metrics-alerts-for-r

git switch claude/wi-019f903c-fleet-observability-metrics-alerts-for-r

git rebase main

git switch main

git merge --ff-only claude/wi-019f903c-fleet-observability-metrics-alerts-for-r

git switch claude/wi-019f903c-fleet-observability-metrics-alerts-for-r

git rebase main

git switch main

git merge --no-ff claude/wi-019f903c-fleet-observability-metrics-alerts-for-r

git switch main

git merge --squash claude/wi-019f903c-fleet-observability-metrics-alerts-for-r