WIP: Stamp liveness witnesses from the ACP channel, carry witness age on the heartbeat, and evaluate the lease observe-only #607

Draft

toasterson wants to merge 1 commit from claude/wi-019ff2f4-stamp-liveness-witnesses-from-the-acp-ch into main

toasterson commented

2026-08-16 20:12:22 +00:00

Owner

Anima work item 019ff2f4-d212-7312-b0d4-ed897c7a6ff6.

What to build

The end-to-end observation path for ADR 0039's session lease, acting on nothing.

Runner witnesses. Every ACP session/update the runner already handles — AgentMessageChunk, AgentThoughtChunk, tool call and tool-call update, UsageUpdate, permission request (acp.rs ~754-860) — stamps a per-session liveness witness: last-witness instant, a witness counter, and whether a first witness has ever arrived. Renewal is a side effect of the stream the runner is already reading; it must not require the agent to cooperate or the executor to self-report.
The wire carries it. Heartbeat (proto/anima/v1/runner.proto:212) currently sends repeated string live_session_ids — a bare id list, which is exactly the self-report that reported running about a corpse for 2h37m. Add a structured per-session liveness message (session id, last-witness timestamp, witness count, first-witness-seen) and add-and-deprecate live_session_ids rather than replacing it, per the schema-version rules.
The server observes. Persist/expose witness age per live session, evaluate a candidate lease against a configurable threshold, and record every would-be reclamation without acting — no CloseSession, no outcome stamped, no slot released. Surface witness age where live sessions are already visible (lane_occupancy).
Report the distribution. Make the observed inter-witness gap distribution retrievable, because its tail is what sets the real threshold.

The demoable outcome: the two sessions from 2026-08-11 would have read "silent for 2h37m" instead of "running", and an operator can answer "what would this have killed?" before anything is allowed to kill.

Pushed context

ADR 0039 is the governing decision — read it first. Key constraints it imposes on this item: the runner is the authority on the witness, the server on the verdict (so the runner reports facts and never expires its own lease); the threshold is measured, not chosen; and observe-only doubles as the runtime kill switch, so the mode must be switchable without a redeploy.
ADR 0034 — the runner reports facts, the server classifies. ADR 0037 — leases and clocks sit on Anima's durable horizon. This item must not put expiry authority in the runner.
ADR 0021 — transient partials never persist. This item needs the timestamp of a witness, never its content; do not start persisting partial content to get liveness.
A liveness witness answers "is anyone home", not "is this getting anywhere" (CONTEXT.md). Do not add worktree-mtime, CPU or output-volume signals here — each false-kills a turn that is legitimately blocked on a model response, and progress is a separate, laxer mechanism (WI-452 owns it).
Start from: crates/anima-runner/src/acp.rs (session/update handler ~754), crates/anima-runner/src/attach.rs (heartbeat ~373-395, runner-lifetime state ~179-190), proto/anima/v1/runner.proto:212, and the server's heartbeat ingest + lane_occupancy.
Proto changed ⇒ regenerate. anima-pwa/src/gen/*_pb.ts is committed; run npm run generate or the build passes while the wire disagrees.
Deployed values for reference: ANIMA_RUNNER_MAX_SESSION_SECS=3600, max_idle_secs=1800, heartbeat_secs=15. Do not change or remove any of them in this item.

Acceptance criteria

Every ACP session/update kind the runner handles renews the session's liveness witness.
Heartbeat carries per-session last-witness timestamp, witness count and first-witness-seen; live_session_ids is deprecated, not removed; PWA codegen regenerated.
Witness age for a live session is visible to an operator without database access.
The server evaluates the lease and records would-be reclamations, and no session is ever closed, no outcome stamped and no slot released by this item.
Observe/enforce mode is switchable at runtime without a redeploy, defaulting to observe.
The observed inter-witness gap distribution is retrievable, so the threshold can be set from its tail.
A session hung after handshake shows a growing witness age; one that never emitted shows first-witness-seen false.

Blocked by

None — can start immediately. Independent of the test-rig item in this plan; both may run in any order. Enforcement is deliberately in a separate plan and must not be built here.

Anima work item `019ff2f4-d212-7312-b0d4-ed897c7a6ff6`. ## What to build The end-to-end observation path for ADR 0039's **session lease**, acting on nothing. 1. **Runner witnesses.** Every ACP `session/update` the runner already handles — `AgentMessageChunk`, `AgentThoughtChunk`, tool call and tool-call update, `UsageUpdate`, permission request (`acp.rs` ~754-860) — stamps a per-session **liveness witness**: last-witness instant, a witness counter, and whether a *first* witness has ever arrived. Renewal is a side effect of the stream the runner is already reading; it must not require the agent to cooperate or the executor to self-report. 2. **The wire carries it.** `Heartbeat` (`proto/anima/v1/runner.proto:212`) currently sends `repeated string live_session_ids` — a bare id list, which is exactly the self-report that reported `running` about a corpse for 2h37m. Add a structured per-session liveness message (session id, last-witness timestamp, witness count, first-witness-seen) and **add-and-deprecate** `live_session_ids` rather than replacing it, per the schema-version rules. 3. **The server observes.** Persist/expose witness age per live session, evaluate a candidate lease against a configurable threshold, and record every would-be reclamation **without acting** — no CloseSession, no outcome stamped, no slot released. Surface witness age where live sessions are already visible (`lane_occupancy`). 4. **Report the distribution.** Make the observed inter-witness gap distribution retrievable, because its tail is what sets the real threshold. The demoable outcome: the two sessions from 2026-08-11 would have read "silent for 2h37m" instead of "running", and an operator can answer "what would this have killed?" before anything is allowed to kill. ## Pushed context - **ADR 0039** is the governing decision — read it first. Key constraints it imposes on this item: the runner is the authority on the **witness**, the server on the **verdict** (so the runner reports facts and never expires its own lease); the threshold is **measured, not chosen**; and observe-only doubles as the runtime kill switch, so the mode must be switchable without a redeploy. - **ADR 0034** — the runner reports facts, the server classifies. **ADR 0037** — leases and clocks sit on Anima's durable horizon. This item must not put expiry authority in the runner. - **ADR 0021** — transient partials never persist. This item needs the *timestamp* of a witness, never its content; do not start persisting partial content to get liveness. - A **liveness witness** answers "is anyone home", not "is this getting anywhere" (CONTEXT.md). Do not add worktree-mtime, CPU or output-volume signals here — each false-kills a turn that is legitimately blocked on a model response, and progress is a separate, laxer mechanism (WI-452 owns it). - Start from: `crates/anima-runner/src/acp.rs` (`session/update` handler ~754), `crates/anima-runner/src/attach.rs` (heartbeat ~373-395, runner-lifetime state ~179-190), `proto/anima/v1/runner.proto:212`, and the server's heartbeat ingest + `lane_occupancy`. - **Proto changed ⇒ regenerate.** `anima-pwa/src/gen/*_pb.ts` is committed; run `npm run generate` or the build passes while the wire disagrees. - Deployed values for reference: `ANIMA_RUNNER_MAX_SESSION_SECS=3600`, `max_idle_secs=1800`, `heartbeat_secs=15`. Do not change or remove any of them in this item. ## Acceptance criteria - [ ] Every ACP `session/update` kind the runner handles renews the session's liveness witness. - [ ] `Heartbeat` carries per-session last-witness timestamp, witness count and first-witness-seen; `live_session_ids` is deprecated, not removed; PWA codegen regenerated. - [ ] Witness age for a live session is visible to an operator without database access. - [ ] The server evaluates the lease and records would-be reclamations, and **no session is ever closed, no outcome stamped and no slot released** by this item. - [ ] Observe/enforce mode is switchable at runtime without a redeploy, defaulting to observe. - [ ] The observed inter-witness gap distribution is retrievable, so the threshold can be set from its tail. - [ ] A session hung after handshake shows a growing witness age; one that never emitted shows first-witness-seen false. ## Blocked by None — can start immediately. Independent of the test-rig item in this plan; both may run in any order. Enforcement is deliberately in a separate plan and must not be built here.

toasterson added 1 commit

2026-08-16 20:12:24 +00:00

Stamp liveness witnesses from the ACP channel, carry witness age on the heartbeat, and evaluate the lease observe-only b88f388c46

Anima implementation session.

This pull request has changes conflicting with the target branch.

crates/anima-server/src/mcp/handlers.rs
crates/anima-server/src/scheduler.rs

View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Merge commit title

Merge pull request 'WIP: Stamp liveness witnesses from the ACP channel, carry witness age on the heartbeat, and evaluate the lease observe-only' (#607) from claude/wi-019ff2f4-stamp-liveness-witnesses-from-the-acp-ch into main

Merge commit body

Reviewed-on: https://code.aopc.cloud/toasterson/Anima/pulls/607

Merge pull request 'WIP: Stamp liveness witnesses from the ACP channel, carry witness age on the heartbeat, and evaluate the lease observe-only' (#607) from claude/wi-019ff2f4-stamp-liveness-witnesses-from-the-acp-ch into main

Reviewed-on: https://code.aopc.cloud/toasterson/Anima/pulls/607

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin claude/wi-019ff2f4-stamp-liveness-witnesses-from-the-acp-ch:claude/wi-019ff2f4-stamp-liveness-witnesses-from-the-acp-ch

git switch claude/wi-019ff2f4-stamp-liveness-witnesses-from-the-acp-ch

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff claude/wi-019ff2f4-stamp-liveness-witnesses-from-the-acp-ch

git switch claude/wi-019ff2f4-stamp-liveness-witnesses-from-the-acp-ch

git rebase main

git switch main

git merge --ff-only claude/wi-019ff2f4-stamp-liveness-witnesses-from-the-acp-ch

git switch claude/wi-019ff2f4-stamp-liveness-witnesses-from-the-acp-ch

git rebase main

git switch main

git merge --no-ff claude/wi-019ff2f4-stamp-liveness-witnesses-from-the-acp-ch

git switch main

git merge --squash claude/wi-019ff2f4-stamp-liveness-witnesses-from-the-acp-ch