toasterson/akh-medu
Watch
1
0
Fork
You've already forked akh-medu
 0
Code Issues 10 Pull requests 22 Projects Releases 2 Packages Wiki Activity Actions 7

WIP: Phase 39 — Distributed Inference Fabric (LlmBackend trait + Candle ROCm + reservation) #270

Draft
toasterson wants to merge 1 commit from claude/wi-019dc599-phase-39-distributed-inference-fabric-ll into main
pull from: claude/wi-019dc599-phase-39-distributed-inference-fabric-ll
merge into: toasterson:main
Conversation 0 Commits 1 Files changed 1 +101 -128
toasterson commented 2026-07-27 21:49:49 +00:00
Owner
Copy link

Anima work item 019dc599-ff37-7cf0-b1dc-0ea5d774c20c.

Phase 39 — Distributed Inference Fabric

Plan: docs/ai/plans/2026-04-25-phase39-distributed-inference.md
ADR: docs/ai/decisions/050-distributed-inference-fabric.md

What this phase is

Decouples where state lives (Phase 38) from where compute happens. Akh daemons stay where their state is (Talos cluster, M2 in cellar, etc.); LLM and embedding inference can route to any reachable backend with the right capabilities. Backends are pluggable via a LlmBackend trait, with implementations for local Candle, remote Candle, and OpenRouter as the primary three. Anthropic ships as an optional/conditional fallback — kept if convenient, dropped if it causes maintenance load. Scarce on-demand backends (the 7900 XTX desktop) use a reservation protocol so multiple Akhs don't trample each other.

No native C++ in the stack — Candle hosts everything, including the ROCm path. Performance maturity is traded for a unified Rust toolchain and tight integration with the Phase 26d neural→VSA bridge (which needs hidden-state extraction Candle gives us).

LlmBackend trait

trait LlmBackend {
    fn capabilities(&self) -> BackendCapabilities;     // model size, context, hidden-state, etc
    fn health(&self) -> Health;                        // reachability + reservation status
    fn reserve(&self, lease: Duration) -> Result<Lease>; // for on-demand backends
    fn release(&self, lease: Lease);
    async fn complete(&self, req: InferenceRequest) -> Result<InferenceResponse>;
    async fn stream(&self, req: InferenceRequest) -> Result<TokenStream>;
}

Primary backends

Backend Where Notes
CandleLocal In-process on the Akh's host Metal (M2) / CUDA / ROCm (7900 XTX)
CandleRemote HTTP/gRPC to a Candle server elsewhere Phase 26d hidden-state extraction over the wire
OpenRouter https://openrouter.ai Cheap access to OSS models — distillation target. Quotas + cost tracking critical.

Conditional / optional backends

Backend Status Notes
Anthropic Optional fallback Ships if it stays low-maintenance. Drop it if its API or prompt-caching surface causes integration pain — the three primary backends cover the workload, and dropping a paid backend has no functional cost.

Hailo-8 / Coral on RPi are NOT in scope here — they fit Phase 14 (NLU) / Phase 33 (T5 NLG) embedding workloads, not LLM serving.

Candle on ROCm

This is the hard deliverable. Candle's ROCm support today is experimental. Plan:

  • Build candle-core against the AMD HIP / ROCm toolchain on the 7900 XTX desktop
  • Identify which kernels are unstable / unsupported; file upstream PRs or shim with our own kernels
  • Acceptance criteria: Qwen2.5-7B Q4_K_M runs at >5 tok/s on the 7900 XTX with full hidden-state extraction available
  • Falls back to CPU if the ROCm path errors at runtime — the daemon shouldn't crash because of GPU instability

Reservation protocol

For scarce on-demand backends (7900 XTX desktop, M2 when its homing Akh is busy):

Akh A wants heavy synthesis
  └── BackendRouter.reserve(7900_XTX, lease=10min)
       ├── available: Lease returned, A streams to backend
       ├── reserved by B: queue or fall through to OpenRouter
       └── offline: fall through

After completion: BackendRouter.release(lease)
Auto-release on lease expiry to prevent zombie holds.

Per-backend policy decides:

  • Queue depth before fall-through
  • Default fall-through chain: CandleLocal → CandleRemote → OpenRouter (with Anthropic appended only when present)
  • Cost/latency weights

Latency-aware routing in Phase 28

The n akh semantic router already learns which backend handles which query type best. Phase 39 extends the bandit's context with:

  • Per-backend latency (rolling p95)
  • Per-backend availability (reachability + reservation status)
  • Per-backend cost (cents/token from OpenRouter and, when shipped, Anthropic; free for on-prem)

Net effect: the router naturally avoids reserved/expensive backends for queries that don't need them.

Discovery + health

  • Each backend publishes a small advertisement: capabilities, current load, ETA-to-availability
  • Akh daemon polls advertisements (gRPC or HTTP /healthz)
  • Failed health check → mark stale, fall through to next chain entry
  • Recovery → mark live, exclude from chain bypass

Cost tracking

Per-Akh global budget. Single number: cents/day across all paid backends. When the budget is exhausted, the router stops considering paid backends entirely until the next budget window. No per-archetype overrides — that's over-engineered for now; revisit only if a specific archetype is identified that needs paid inference and isn't covered by routing decisions.

Key deliverables

  • LlmBackend trait + three primary implementations (CandleLocal, CandleRemote, OpenRouter)
  • Optional Anthropic adapter — ships only if it stays low-maintenance; drop without ceremony if it causes integration pain
  • Candle-on-ROCm path tested on 7900 XTX
  • Reservation protocol with leases + auto-release
  • Discovery + health-check loop
  • Latency-aware routing extension to Phase 28
  • Cost tracking per backend (per-Akh global budget; no per-archetype overrides)
  • Per-Akh fall-through chain (configurable)
  • Graceful degradation on backend failure mid-stream

Dependencies

  • Phase 26 (Candle backend) — provides the in-process Candle path
  • Phase 28 (n akh semantic router) — extends with backend-aware routing
  • Phase 38 (Relocatable Akh) — no hard dependency, but the deployment topology Phase 39 assumes is what Phase 38 establishes

Why this priority

Without Phase 39, the 7900 XTX desktop sits unused and Akhs are stuck with whatever inference the host can do locally. With Phase 39, M2-class hosts can route heavy lifting to the desktop on demand, OpenRouter handles cheap distillation jobs, and the cluster Akh has a full fall-through chain.

Resolutions log

2026-04-25

  • P39Q5 (Anthropic prompt-caching exposure): deprioritize Anthropic. The three primary backends are CandleLocal, CandleRemote, OpenRouter. Anthropic ships only if it stays low-maintenance; drop if integration pain emerges.
  • P39Q6 (per-archetype budget overrides): dropped as over-engineering. Single global budget per Akh until a specific need is identified.

Open questions still standing

  • Hidden-state extraction over the wire (CandleRemote): how to keep Phase 26d neural-bridge invariants when hidden states travel via gRPC. Wire format design.
  • Reservation semantics under contention: queue vs hard fail vs cost-aware preemption.
  • Per-Akh global budget enforcement: cents/day budget threshold definition + reset cadence + alert on near-exhaustion.
2026-06-04 21:20 UTC

Summary

This phase introduces a distributed inference fabric by decoupling compute from state, allowing the agent to route LLM workloads dynamically across local Candle (ROCm/CPU), remote Candle, and OpenRouter backends. It adds stateful reservations for scarce hardware, latency/cost-aware routing, and global budget enforcement for paid inference.

Approach

Define the LlmBackend trait in crates/anima-ai exposing capabilities, health, lease reservation, and inference/streaming execution. Implement the trait for CandleLocal (leveraging Candle's ROCm features with a fallback to native CPU execution), CandleRemote (via a new gRPC service defined in proto/anima/v1/inference.proto supporting streaming and hidden-state extraction), OpenRouter (REST, with cost-tracking), and a conditional Anthropic adapter. Introduce a BackendRouter to manage the fall-through chain (CandleLocal -> CandleRemote -> OpenRouter), enforce the strict cents/day global budget, and handle lease states. Extend the existing semantic router in anima-ai to ingest rolling p95 latency, availability, and cost data to make backend routing decisions, and wire these components into the anima-agent daemon startup.

Files likely to change

  • crates/anima-ai/Cargo.toml
  • crates/anima-ai/src/backend/mod.rs [Guess: new module for LlmBackend trait]
  • crates/anima-ai/src/backend/candle_local.rs [Guess]
  • crates/anima-ai/src/backend/candle_remote.rs [Guess]
  • crates/anima-ai/src/backend/openrouter.rs [Guess]
  • crates/anima-ai/src/backend/anthropic.rs [Guess: optional]
  • crates/anima-ai/src/router/backend_router.rs [Guess: reservation lease and fall-through logic]
  • crates/anima-ai/src/router/semantic.rs [Guess: extending the existing router for latencies/costs]
  • proto/anima/v1/inference.proto [Guess: new protobuf file for remote Candle backends]
  • crates/anima-proto/src/lib.rs
  • crates/anima-agent/Cargo.toml
  • crates/anima-agent/src/main.rs [Guess: daemon wiring for discovery and configs]

Open questions

  • Hidden-state wire format: How can we serialize hidden-state extraction over gRPC for CandleRemote efficiently without breaking Phase 26d neural-bridge invariants or imposing excessive network overhead?
  • Contention semantics: When a backend relies on the reservation protocol and is busy, what is the exact queue depth before falling through to the next chain entry?
  • ROCm Kernel instabilities: What is the precise mechanism to safely catch Candle ROCm kernel panics/errors without bringing down the asynchronous executor and the main anima-agent process, gracefully falling back to CPU?
  • Budget enforcement specifics: How are the per-Akh global budget (cents/day) window resets synchronized, and does it need to persist across anima-agent daemon restarts?

Complexity

L: Implementing multiple backend adapters, handling distributed state through leases/health-checks, and integrating the highly experimental Candle ROCm toolchain (complete with CPU fallbacks and profiling) will require significant effort and cross-machine testing.

Anima work item `019dc599-ff37-7cf0-b1dc-0ea5d774c20c`. ## Phase 39 — Distributed Inference Fabric **Plan:** `docs/ai/plans/2026-04-25-phase39-distributed-inference.md` **ADR:** `docs/ai/decisions/050-distributed-inference-fabric.md` ### What this phase is Decouples *where state lives* (Phase 38) from *where compute happens*. Akh daemons stay where their state is (Talos cluster, M2 in cellar, etc.); LLM and embedding inference can route to any reachable backend with the right capabilities. Backends are pluggable via a `LlmBackend` trait, with implementations for local Candle, remote Candle, and OpenRouter as the primary three. Anthropic ships as an optional/conditional fallback — kept if convenient, dropped if it causes maintenance load. Scarce on-demand backends (the 7900 XTX desktop) use a reservation protocol so multiple Akhs don't trample each other. No native C++ in the stack — Candle hosts everything, including the ROCm path. Performance maturity is traded for a unified Rust toolchain and tight integration with the Phase 26d neural→VSA bridge (which needs hidden-state extraction Candle gives us). ### `LlmBackend` trait ```rust trait LlmBackend { fn capabilities(&self) -> BackendCapabilities; // model size, context, hidden-state, etc fn health(&self) -> Health; // reachability + reservation status fn reserve(&self, lease: Duration) -> Result<Lease>; // for on-demand backends fn release(&self, lease: Lease); async fn complete(&self, req: InferenceRequest) -> Result<InferenceResponse>; async fn stream(&self, req: InferenceRequest) -> Result<TokenStream>; } ``` ### Primary backends | Backend | Where | Notes | |---|---|---| | `CandleLocal` | In-process on the Akh's host | Metal (M2) / CUDA / **ROCm (7900 XTX)** | | `CandleRemote` | HTTP/gRPC to a Candle server elsewhere | Phase 26d hidden-state extraction over the wire | | `OpenRouter` | https://openrouter.ai | Cheap access to OSS models — distillation target. Quotas + cost tracking critical. | ### Conditional / optional backends | Backend | Status | Notes | |---|---|---| | `Anthropic` | Optional fallback | Ships if it stays low-maintenance. Drop it if its API or prompt-caching surface causes integration pain — the three primary backends cover the workload, and dropping a paid backend has no functional cost. | Hailo-8 / Coral on RPi are NOT in scope here — they fit Phase 14 (NLU) / Phase 33 (T5 NLG) embedding workloads, not LLM serving. ### Candle on ROCm This is the hard deliverable. Candle's ROCm support today is experimental. Plan: - Build `candle-core` against the AMD HIP / ROCm toolchain on the 7900 XTX desktop - Identify which kernels are unstable / unsupported; file upstream PRs or shim with our own kernels - Acceptance criteria: Qwen2.5-7B Q4_K_M runs at >5 tok/s on the 7900 XTX with full hidden-state extraction available - Falls back to CPU if the ROCm path errors at runtime — the daemon shouldn't crash because of GPU instability ### Reservation protocol For scarce on-demand backends (7900 XTX desktop, M2 when its homing Akh is busy): ``` Akh A wants heavy synthesis └── BackendRouter.reserve(7900_XTX, lease=10min) ├── available: Lease returned, A streams to backend ├── reserved by B: queue or fall through to OpenRouter └── offline: fall through After completion: BackendRouter.release(lease) Auto-release on lease expiry to prevent zombie holds. ``` Per-backend policy decides: - Queue depth before fall-through - Default fall-through chain: `CandleLocal → CandleRemote → OpenRouter` (with Anthropic appended only when present) - Cost/latency weights ### Latency-aware routing in Phase 28 The `n akh` semantic router already learns which *backend* handles which query type best. Phase 39 extends the bandit's context with: - Per-backend latency (rolling p95) - Per-backend availability (reachability + reservation status) - Per-backend cost (cents/token from OpenRouter and, when shipped, Anthropic; free for on-prem) Net effect: the router naturally avoids reserved/expensive backends for queries that don't need them. ### Discovery + health - Each backend publishes a small advertisement: capabilities, current load, ETA-to-availability - Akh daemon polls advertisements (gRPC or HTTP /healthz) - Failed health check → mark stale, fall through to next chain entry - Recovery → mark live, exclude from chain bypass ### Cost tracking Per-Akh global budget. Single number: cents/day across all paid backends. When the budget is exhausted, the router stops considering paid backends entirely until the next budget window. No per-archetype overrides — that's over-engineered for now; revisit only if a specific archetype is identified that needs paid inference and isn't covered by routing decisions. ### Key deliverables - `LlmBackend` trait + three primary implementations (`CandleLocal`, `CandleRemote`, `OpenRouter`) - Optional Anthropic adapter — ships only if it stays low-maintenance; drop without ceremony if it causes integration pain - Candle-on-ROCm path tested on 7900 XTX - Reservation protocol with leases + auto-release - Discovery + health-check loop - Latency-aware routing extension to Phase 28 - Cost tracking per backend (per-Akh global budget; no per-archetype overrides) - Per-Akh fall-through chain (configurable) - Graceful degradation on backend failure mid-stream ### Dependencies - Phase 26 (Candle backend) — provides the in-process Candle path - Phase 28 (`n akh` semantic router) — extends with backend-aware routing - Phase 38 (Relocatable Akh) — no hard dependency, but the deployment topology Phase 39 assumes is what Phase 38 establishes ### Why this priority Without Phase 39, the 7900 XTX desktop sits unused and Akhs are stuck with whatever inference the host can do locally. With Phase 39, M2-class hosts can route heavy lifting to the desktop on demand, OpenRouter handles cheap distillation jobs, and the cluster Akh has a full fall-through chain. ### Resolutions log #### 2026-04-25 - **P39Q5** (Anthropic prompt-caching exposure): deprioritize Anthropic. The three primary backends are CandleLocal, CandleRemote, OpenRouter. Anthropic ships only if it stays low-maintenance; drop if integration pain emerges. - **P39Q6** (per-archetype budget overrides): dropped as over-engineering. Single global budget per Akh until a specific need is identified. ### Open questions still standing - Hidden-state extraction over the wire (CandleRemote): how to keep Phase 26d neural-bridge invariants when hidden states travel via gRPC. Wire format design. - Reservation semantics under contention: queue vs hard fail vs cost-aware preemption. - Per-Akh global budget enforcement: cents/day budget threshold definition + reset cadence + alert on near-exhaustion. <!-- ANIMA TRIAGE PLAN --> 2026-06-04 21:20 UTC ## Summary This phase introduces a distributed inference fabric by decoupling compute from state, allowing the agent to route LLM workloads dynamically across local Candle (ROCm/CPU), remote Candle, and OpenRouter backends. It adds stateful reservations for scarce hardware, latency/cost-aware routing, and global budget enforcement for paid inference. ## Approach Define the `LlmBackend` trait in `crates/anima-ai` exposing capabilities, health, lease reservation, and inference/streaming execution. Implement the trait for `CandleLocal` (leveraging Candle's ROCm features with a fallback to native CPU execution), `CandleRemote` (via a new gRPC service defined in `proto/anima/v1/inference.proto` supporting streaming and hidden-state extraction), `OpenRouter` (REST, with cost-tracking), and a conditional `Anthropic` adapter. Introduce a `BackendRouter` to manage the fall-through chain (`CandleLocal` -> `CandleRemote` -> `OpenRouter`), enforce the strict cents/day global budget, and handle lease states. Extend the existing semantic router in `anima-ai` to ingest rolling p95 latency, availability, and cost data to make backend routing decisions, and wire these components into the `anima-agent` daemon startup. ## Files likely to change * `crates/anima-ai/Cargo.toml` * `crates/anima-ai/src/backend/mod.rs` [Guess: new module for LlmBackend trait] * `crates/anima-ai/src/backend/candle_local.rs` [Guess] * `crates/anima-ai/src/backend/candle_remote.rs` [Guess] * `crates/anima-ai/src/backend/openrouter.rs` [Guess] * `crates/anima-ai/src/backend/anthropic.rs` [Guess: optional] * `crates/anima-ai/src/router/backend_router.rs` [Guess: reservation lease and fall-through logic] * `crates/anima-ai/src/router/semantic.rs` [Guess: extending the existing router for latencies/costs] * `proto/anima/v1/inference.proto` [Guess: new protobuf file for remote Candle backends] * `crates/anima-proto/src/lib.rs` * `crates/anima-agent/Cargo.toml` * `crates/anima-agent/src/main.rs` [Guess: daemon wiring for discovery and configs] ## Open questions * **Hidden-state wire format:** How can we serialize hidden-state extraction over gRPC for `CandleRemote` efficiently without breaking Phase 26d neural-bridge invariants or imposing excessive network overhead? * **Contention semantics:** When a backend relies on the reservation protocol and is busy, what is the exact queue depth before falling through to the next chain entry? * **ROCm Kernel instabilities:** What is the precise mechanism to safely catch Candle ROCm kernel panics/errors without bringing down the asynchronous executor and the main `anima-agent` process, gracefully falling back to CPU? * **Budget enforcement specifics:** How are the per-Akh global budget (cents/day) window resets synchronized, and does it need to persist across `anima-agent` daemon restarts? ## Complexity L: Implementing multiple backend adapters, handling distributed state through leases/health-checks, and integrating the highly experimental Candle ROCm toolchain (complete with CPU fallbacks and profiling) will require significant effort and cross-machine testing.
wip: triage Phase 39 — Distributed Inference Fabric (LlmBackend trait + Candle ROCm + reservation)
Some checks failed
CI / check-seshat (pull_request) Successful in 19m22s
Details
CI / publish-chart (push) Successful in 19m49s
Details
CI / docker-seshd (pull_request) Successful in 23m17s
Details
CI / docker-seshd (push) Successful in 29m44s
Details
CI / publish-chart (pull_request) Failing after 29m58s
Details
CI / check-seshat (push) Failing after 30m1s
Details
27c10ea439
Some checks failed
CI / check-seshat (pull_request) Successful in 19m22s
Details
CI / publish-chart (push) Successful in 19m49s
Details
CI / docker-seshd (pull_request) Successful in 23m17s
Details
CI / docker-seshd (push) Successful in 29m44s
Details
CI / publish-chart (pull_request) Failing after 29m58s
Details
CI / check-seshat (push) Failing after 30m1s
Details
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin claude/wi-019dc599-phase-39-distributed-inference-fabric-ll:claude/wi-019dc599-phase-39-distributed-inference-fabric-ll
git switch claude/wi-019dc599-phase-39-distributed-inference-fabric-ll

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff claude/wi-019dc599-phase-39-distributed-inference-fabric-ll
git switch claude/wi-019dc599-phase-39-distributed-inference-fabric-ll
git rebase main
git switch main
git merge --ff-only claude/wi-019dc599-phase-39-distributed-inference-fabric-ll
git switch claude/wi-019dc599-phase-39-distributed-inference-fabric-ll
git rebase main
git switch main
git merge --no-ff claude/wi-019dc599-phase-39-distributed-inference-fabric-ll
git switch main
git merge --squash claude/wi-019dc599-phase-39-distributed-inference-fabric-ll
git switch main
git merge --ff-only claude/wi-019dc599-phase-39-distributed-inference-fabric-ll
git switch main
git merge claude/wi-019dc599-phase-39-distributed-inference-fabric-ll
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
toasterson/akh-medu!270
No description provided.