toasterson/akh-medu
Watch
1
0
Fork
You've already forked akh-medu
 0
Code Issues 10 Pull requests 22 Projects Releases 2 Packages Wiki Activity Actions 7

WIP: Phase 42c — Drawer ingest path + embed worker #267

Draft
toasterson wants to merge 2 commits from claude/wi-019dd0c6-phase-42c-drawer-ingest-path-embed-worke into main
pull from: claude/wi-019dd0c6-phase-42c-drawer-ingest-path-embed-worke
merge into: toasterson:main
Conversation 0 Commits 2 Files changed 14 +1384 -13
toasterson commented 2026-07-27 14:51:46 +00:00
Owner
Copy link

Anima work item 019dd0c6-1371-7183-bdde-75fee9f7f4c1.

Verbatim drawer write path + ONNX embedding worker.

Deliverables:

  • Ingest::write_drawer(akh_id, session_id, speaker, body, contact, topics) — content-hash idempotent (UNIQUE on (akh_id, body_hash)), paragraph-aware chunking (target 1500 chars, MiniLM 512-token-friendly), enqueues chunks for embedding.
  • embed_worker ported from crates/seshat/src/embed_worker.rs — same MiniLM-L6-v2 ONNX model, same ort crate. Polls heka_drawer_chunks anti-join heka_drawer_embeddings, batch=32, writes to heka_drawer_embeddings.
  • heka-capture binary: reads stdin (Claude Code stop-hook payload), parses session turns, calls Ingest::write_drawer per turn. Designed as the target of ~/.claude/settings.json Stop hook.

Reuses: crates/seshat/src/embed_worker.rs, crates/seshat/src/chunker.rs. (Refactor to shared crates/akh-embed-common/ is nice-to-have, defer if it adds friction.)

Acceptance:

  • 200-turn session produces 200 drawer rows
  • Embed worker drains 200 chunks within 30s on M-class hardware
  • heka eval longmemeval --mode raw returns >0% recall (drawers are now retrievable)
  • Idempotency: capturing same session twice produces identical drawer count

Estimated lines: ~600
Dependencies: 42a, 42b

2026-06-04 22:00 UTC

Summary

Create a new heka-capture CLI to ingest Claude Code sessions via stdin, chunk the text, and store it idempotently in the database. Introduce an ONNX-based background worker to continuously poll for new chunks and embed them in batches using a MiniLM model.

Approach

Add a new workspace crate heka-capture containing a CLI binary that reads stop-hook JSON payloads from stdin and extracts session turns. Implement Ingest::write_drawer in anima-db using a new Postgres migration that creates the chunks and embeddings tables, ensuring idempotency with a UNIQUE(akh_id, body_hash) constraint. Port the seshat chunker and ort embedding logic into anima-ai, targeting ~1500 character chunks appropriate for the 512-token limit. Finally, integrate the embed_worker into anima-server to run as a background task, executing an anti-join query against the database to fetch un-embedded chunks, processing them in batches of 32, and writing the vectors back to Postgres.

Files likely to change

  • Cargo.toml
  • crates/heka-capture/Cargo.toml
  • crates/heka-capture/src/main.rs
  • crates/anima-core/src/domain/drawer.rs
  • crates/anima-db/migrations/20240519000000_create_drawer_tables.sql
  • crates/anima-db/src/repo/drawer.rs
  • crates/anima-ai/Cargo.toml
  • crates/anima-ai/src/chunker.rs
  • crates/anima-ai/src/embed_worker.rs
  • crates/anima-server/src/main.rs

Open questions

  • Should the heka-capture binary connect directly to the PostgreSQL database, or should we define a new anima-proto gRPC endpoint and route ingestion through anima-server?
  • How is the MiniLM-L6-v2 ONNX model file distributed to the environment running anima-server (e.g., baked into a container image, downloaded on startup, or included via include_bytes!)?
  • Is heka eval longmemeval a separate tool requiring its own DB access configuration, or does it need to be added to the current workspace?

Complexity

M: 1-3 days. Porting the seshat chunking and ort embedding code mitigates risk, but wiring up a new CLI, establishing the anti-join polling worker in anima-server, and managing ONNX runtime dependencies natively requires moderate effort.

Anima work item `019dd0c6-1371-7183-bdde-75fee9f7f4c1`. Verbatim drawer write path + ONNX embedding worker. Deliverables: - `Ingest::write_drawer(akh_id, session_id, speaker, body, contact, topics)` — content-hash idempotent (UNIQUE on (akh_id, body_hash)), paragraph-aware chunking (target 1500 chars, MiniLM 512-token-friendly), enqueues chunks for embedding. - `embed_worker` ported from `crates/seshat/src/embed_worker.rs` — same MiniLM-L6-v2 ONNX model, same `ort` crate. Polls heka_drawer_chunks anti-join heka_drawer_embeddings, batch=32, writes to heka_drawer_embeddings. - `heka-capture` binary: reads stdin (Claude Code stop-hook payload), parses session turns, calls Ingest::write_drawer per turn. Designed as the target of `~/.claude/settings.json` Stop hook. Reuses: `crates/seshat/src/embed_worker.rs`, `crates/seshat/src/chunker.rs`. (Refactor to shared `crates/akh-embed-common/` is nice-to-have, defer if it adds friction.) Acceptance: - 200-turn session produces 200 drawer rows - Embed worker drains 200 chunks within 30s on M-class hardware - `heka eval longmemeval --mode raw` returns >0% recall (drawers are now retrievable) - Idempotency: capturing same session twice produces identical drawer count Estimated lines: ~600 Dependencies: 42a, 42b <!-- ANIMA TRIAGE PLAN --> 2026-06-04 22:00 UTC ## Summary Create a new `heka-capture` CLI to ingest Claude Code sessions via stdin, chunk the text, and store it idempotently in the database. Introduce an ONNX-based background worker to continuously poll for new chunks and embed them in batches using a MiniLM model. ## Approach Add a new workspace crate `heka-capture` containing a CLI binary that reads stop-hook JSON payloads from stdin and extracts session turns. Implement `Ingest::write_drawer` in `anima-db` using a new Postgres migration that creates the chunks and embeddings tables, ensuring idempotency with a `UNIQUE(akh_id, body_hash)` constraint. Port the `seshat` chunker and `ort` embedding logic into `anima-ai`, targeting ~1500 character chunks appropriate for the 512-token limit. Finally, integrate the `embed_worker` into `anima-server` to run as a background task, executing an anti-join query against the database to fetch un-embedded chunks, processing them in batches of 32, and writing the vectors back to Postgres. ## Files likely to change * `Cargo.toml` * `crates/heka-capture/Cargo.toml` * `crates/heka-capture/src/main.rs` * `crates/anima-core/src/domain/drawer.rs` * `crates/anima-db/migrations/20240519000000_create_drawer_tables.sql` * `crates/anima-db/src/repo/drawer.rs` * `crates/anima-ai/Cargo.toml` * `crates/anima-ai/src/chunker.rs` * `crates/anima-ai/src/embed_worker.rs` * `crates/anima-server/src/main.rs` ## Open questions * Should the `heka-capture` binary connect directly to the PostgreSQL database, or should we define a new `anima-proto` gRPC endpoint and route ingestion through `anima-server`? * How is the MiniLM-L6-v2 ONNX model file distributed to the environment running `anima-server` (e.g., baked into a container image, downloaded on startup, or included via `include_bytes!`)? * Is `heka eval longmemeval` a separate tool requiring its own DB access configuration, or does it need to be added to the current workspace? ## Complexity M: 1-3 days. Porting the `seshat` chunking and `ort` embedding code mitigates risk, but wiring up a new CLI, establishing the anti-join polling worker in `anima-server`, and managing ONNX runtime dependencies natively requires moderate effort.
Phase 42c — Drawer ingest path + embed worker
All checks were successful
CI / publish-chart (pull_request) Successful in 15m14s
Details
CI / publish-chart (push) Successful in 15m19s
Details
CI / check-seshat (pull_request) Successful in 16m16s
Details
CI / check-seshat (push) Successful in 16m17s
Details
CI / docker-seshd (pull_request) Successful in 23m46s
Details
CI / docker-seshd (push) Successful in 23m44s
Details
84d44ce74d 
Anima implementation session.
toasterson force-pushed claude/wi-019dd0c6-phase-42c-drawer-ingest-path-embed-worke from 84d44ce74d
All checks were successful
CI / publish-chart (pull_request) Successful in 15m14s
Details
CI / publish-chart (push) Successful in 15m19s
Details
CI / check-seshat (pull_request) Successful in 16m16s
Details
CI / check-seshat (push) Successful in 16m17s
Details
CI / docker-seshd (pull_request) Successful in 23m46s
Details
CI / docker-seshd (push) Successful in 23m44s
Details
to 8fcc2ee203
All checks were successful
CI / publish-chart (push) Successful in 20m55s
Details
CI / check-seshat (push) Successful in 21m53s
Details
CI / publish-chart (pull_request) Successful in 12m44s
Details
CI / check-seshat (pull_request) Successful in 22m45s
Details
CI / docker-seshd (push) Successful in 26m1s
Details
CI / docker-seshd (pull_request) Successful in 24m22s
Details
2026-07-27 16:39:38 +00:00 Compare
toasterson force-pushed claude/wi-019dd0c6-phase-42c-drawer-ingest-path-embed-worke from 8fcc2ee203
All checks were successful
CI / publish-chart (push) Successful in 20m55s
Details
CI / check-seshat (push) Successful in 21m53s
Details
CI / publish-chart (pull_request) Successful in 12m44s
Details
CI / check-seshat (pull_request) Successful in 22m45s
Details
CI / docker-seshd (push) Successful in 26m1s
Details
CI / docker-seshd (pull_request) Successful in 24m22s
Details
to 2c2e203eca
All checks were successful
CI / publish-chart (push) Successful in 7m6s
Details
CI / publish-chart (pull_request) Successful in 2m29s
Details
CI / check-seshat (push) Successful in 10m53s
Details
CI / check-seshat (pull_request) Successful in 10m59s
Details
CI / docker-seshd (push) Successful in 15m30s
Details
CI / docker-seshd (pull_request) Successful in 15m31s
Details
2026-07-27 18:49:08 +00:00 Compare
All checks were successful
CI / publish-chart (push) Successful in 7m6s
Details
CI / publish-chart (pull_request) Successful in 2m29s
Details
CI / check-seshat (push) Successful in 10m53s
Details
CI / check-seshat (pull_request) Successful in 10m59s
Details
CI / docker-seshd (push) Successful in 15m30s
Details
CI / docker-seshd (pull_request) Successful in 15m31s
Details
This pull request has changes conflicting with the target branch.
  • Cargo.lock
  • crates/heka/Cargo.toml
  • crates/heka/src/bin/hekad.rs
  • crates/heka/src/lib.rs
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin claude/wi-019dd0c6-phase-42c-drawer-ingest-path-embed-worke:claude/wi-019dd0c6-phase-42c-drawer-ingest-path-embed-worke
git switch claude/wi-019dd0c6-phase-42c-drawer-ingest-path-embed-worke

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff claude/wi-019dd0c6-phase-42c-drawer-ingest-path-embed-worke
git switch claude/wi-019dd0c6-phase-42c-drawer-ingest-path-embed-worke
git rebase main
git switch main
git merge --ff-only claude/wi-019dd0c6-phase-42c-drawer-ingest-path-embed-worke
git switch claude/wi-019dd0c6-phase-42c-drawer-ingest-path-embed-worke
git rebase main
git switch main
git merge --no-ff claude/wi-019dd0c6-phase-42c-drawer-ingest-path-embed-worke
git switch main
git merge --squash claude/wi-019dd0c6-phase-42c-drawer-ingest-path-embed-worke
git switch main
git merge --ff-only claude/wi-019dd0c6-phase-42c-drawer-ingest-path-embed-worke
git switch main
git merge claude/wi-019dd0c6-phase-42c-drawer-ingest-path-embed-worke
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
toasterson/akh-medu!267
No description provided.