Runner survivability — reap, self-heal, and stop the runners table lying (2026-07-23 outage) #358

Closed

toasterson wants to merge 0 commits from claude/runner-survivability into main

toasterson commented

2026-07-23 17:20:42 +00:00

Owner

Incident

On 2026-07-23 the opencode runner detached from the server at 12:33Z and never reconnected. The container stayed Up, restart: unless-stopped never fired (the process never exited), and the runners row kept reporting status=connected, last_seen=<now>. For 4.5h the scheduler logged 26,570 unschedulable: no connected runner advertises this executor executor=opencode warnings while the entire fleet sat idle — every ready WI requests opencode, so runner-1 (claude-code only) had nothing eligible either. Two in-flight sessions died of rebind window lapsed.

Post-mortem found three independent failures; this PR fixes all three. Plan: Runner container survivability — reap, self-heal, tell the truth.

WI 1 — anima-runner must not be PID 1

Dockerfile.runner had ENTRYPOINT ["anima-runner"], so the runner was PID 1 with no child reaping. The container held 175 zombies / 183 processes, ~7/hour, including the two [opencode] <defunct> executors whose exit the runner never noticed. This was already solved once for the now-retired anima-exec (648868c, "P1.1 — tini PID 1"); the ACP runner that replaced it never inherited it.

tini as PID 1 in the runtime image.
init: true on every runner service as belt-and-braces for any image built without it.

WI 2 — a wedged runner restarts itself

The tonic attach channel had no keepalive. The read loop is inbound.message().await?; on a half-open connection (server pod replaced, gateway drops the flow without a FIN) that await blocks forever — no error, no ? — so connect_once never returns and the reconnect loop never runs. The heartbeat task's send-failure path was a silent break.

http2 + tcp keepalive so a dead peer surfaces as an error and the existing reconnect loop takes over. This alone fixes the incident.
Heartbeat send-failure logs ERROR instead of breaking silently.
Watchdog: the runner touches a health file on every proven-alive heartbeat and exit(1)s when it goes stale (>3× heartbeat), letting restart: unless-stopped restart it. Docker does not restart on an unhealthy healthcheck — only on process exit — so the healthcheck diagnoses and the watchdog cures.
Compose healthchecks stat the file.

WI 3 — the runners table must not lie

The registry is keyed by runner_id. A stale connection's teardown removed the registration after the runner had already reattached on a new connection — evicting the live one. The runner kept heartbeating (so the DB row, the fleet card, and docker ps all read healthy) while dispatch could no longer see it. This is the actual outage mechanism.

Each RunnerHandle carries a monotonic conn id; teardown only removes the handle if the live one still owns that connection (remove_if).
A heartbeat from a runner absent from the live registry now logs ERROR (was swallowed by if let Some).
The per-WI-per-tick unschedulable spam is folded to one line on the transition into/out of unschedulable, with the recovery logged too.

Tests

cargo test -p anima-server --lib services::runner::tests (16 pass) — new: stale_connection_teardown_does_not_evict_the_reattached_runner, heartbeat_for_an_unregistered_runner_is_reported. cargo test -p anima-runner --bin anima-runner attach (3 pass) — new: stale_threshold_never_undercuts_the_heartbeat.

Follow-ups (not in this PR)

A stalled executor with ready work + no connected runner should raise an Attention Inbox item, not just a log line.
The kg_delete WI (019f579f) has been re-dispatched hourly against the same akh-medu PR since 2026-07-12 producing near-zero diff — needs re-scoping or parking.

🤖 Generated with Claude Code

## Incident On 2026-07-23 the opencode runner detached from the server at 12:33Z and never reconnected. The container stayed `Up`, `restart: unless-stopped` never fired (the process never exited), and the `runners` row kept reporting `status=connected, last_seen=<now>`. For 4.5h the scheduler logged **26,570** `unschedulable: no connected runner advertises this executor executor=opencode` warnings while the entire fleet sat idle — every ready WI requests opencode, so runner-1 (claude-code only) had nothing eligible either. Two in-flight sessions died of `rebind window lapsed`. Post-mortem found three independent failures; this PR fixes all three. Plan: *Runner container survivability — reap, self-heal, tell the truth*. ## WI 1 — anima-runner must not be PID 1 `Dockerfile.runner` had `ENTRYPOINT ["anima-runner"]`, so the runner was PID 1 with no child reaping. The container held **175 zombies / 183 processes**, ~7/hour, including the two `[opencode] <defunct>` executors whose exit the runner never noticed. This was already solved once for the now-retired anima-exec (648868c, "P1.1 — tini PID 1"); the ACP runner that replaced it never inherited it. - tini as PID 1 in the runtime image. - `init: true` on every runner service as belt-and-braces for any image built without it. ## WI 2 — a wedged runner restarts itself The tonic attach channel had **no keepalive**. The read loop is `inbound.message().await?`; on a half-open connection (server pod replaced, gateway drops the flow without a FIN) that await blocks forever — no error, no `?` — so `connect_once` never returns and the reconnect loop never runs. The heartbeat task's send-failure path was a silent `break`. - http2 + tcp keepalive so a dead peer surfaces as an error and the existing reconnect loop takes over. **This alone fixes the incident.** - Heartbeat send-failure logs ERROR instead of breaking silently. - Watchdog: the runner touches a health file on every proven-alive heartbeat and `exit(1)`s when it goes stale (>3× heartbeat), letting `restart: unless-stopped` restart it. **Docker does not restart on an unhealthy healthcheck — only on process exit** — so the healthcheck diagnoses and the watchdog cures. - Compose healthchecks stat the file. ## WI 3 — the runners table must not lie The registry is keyed by `runner_id`. A stale connection's teardown removed the registration **after the runner had already reattached on a new connection** — evicting the live one. The runner kept heartbeating (so the DB row, the fleet card, and `docker ps` all read healthy) while dispatch could no longer see it. This is the actual outage mechanism. - Each `RunnerHandle` carries a monotonic `conn` id; teardown only removes the handle if the live one still owns that connection (`remove_if`). - A heartbeat from a runner absent from the live registry now logs ERROR (was swallowed by `if let Some`). - The per-WI-per-tick `unschedulable` spam is folded to one line on the transition into/out of unschedulable, with the recovery logged too. ## Tests `cargo test -p anima-server --lib services::runner::tests` (16 pass) — new: `stale_connection_teardown_does_not_evict_the_reattached_runner`, `heartbeat_for_an_unregistered_runner_is_reported`. `cargo test -p anima-runner --bin anima-runner attach` (3 pass) — new: `stale_threshold_never_undercuts_the_heartbeat`. ## Follow-ups (not in this PR) - A stalled executor with ready work + no connected runner should raise an Attention Inbox item, not just a log line. - The kg_delete WI (`019f579f`) has been re-dispatched hourly against the same akh-medu PR since 2026-07-12 producing near-zero diff — needs re-scoping or parking. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

toasterson added 1 commit

2026-07-23 17:20:43 +00:00

fix(runner): survive a wedged attach stream — reap, self-heal, stop lying

CI / conflict-check (push) Successful in 51s

Details

CI / docker (push) Successful in 11m59s

Details

76cbce86f8

2026-07-23 incident: the opencode runner detached from the server at 12:33Z
and never came back. The container stayed "Up", restart:unless-stopped never
fired (the process never exited), and the runners row kept reporting
status=connected/last_seen=now. For 4.5h the scheduler logged 26,570
"unschedulable: no connected runner advertises this executor" while the whole
fleet sat idle. Root causes and fixes:

WI 1 — anima-runner was PID 1 and never reaped (175 zombies / 183 procs,
~7/hour). Add tini as PID 1 in Dockerfile.runner and init:true on every runner
service. This regressed from anima-exec's P1.1 fix (648868c) when the ACP
runner replaced it.

WI 2 — the attach channel had no keepalive, so a half-open connection blocked
inbound.message() forever and the reconnect loop never ran. Add http2/tcp
keepalive so a dead peer surfaces as an error. Heartbeat send-failure now logs
ERROR instead of a silent break. A watchdog touches a health file on every
proven-alive heartbeat and exit(1)s when it goes stale (>3× heartbeat), letting
restart:unless-stopped restart the container — Docker does NOT restart on an
unhealthy healthcheck, only on process exit, so the healthcheck diagnoses and
the watchdog cures. Compose healthchecks stat the file.

WI 3 — server: the registry is keyed by runner_id, so a stale connection's
teardown evicted a runner that had already reattached on a new connection —
the actual outage mechanism. Tag each RunnerHandle with a monotonic conn id
and only remove_if the live handle still owns it. A heartbeat from an
unregistered runner now logs ERROR (was swallowed). Fold the per-WI-per-tick
unschedulable spam into one line on the transition in/out of unschedulable.

Regression tests cover stale-connection eviction, unregistered-heartbeat
reporting, and the watchdog staleness floor.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

toasterson referenced this pull request from a commit

2026-07-23 17:39:07 +00:00

Merge pull request #358 'Runner survivability — reap, self-heal, and stop the runners table lying (2026-07-23 outage)' from claude/runner-survivability into main

toasterson closed this pull request

2026-07-24 18:28:20 +00:00

toasterson referenced this pull request

2026-07-25 13:08:20 +00:00

WIP: Fleet observability: metrics + alerts for runner attach, delivery outcome, unschedulable queue #367

toasterson referenced this pull request

2026-07-29 13:21:09 +00:00

CI builds + pushes multi-arch deploy images on git tags #431

toasterson referenced this pull request from a commit