toasterson/akh-medu
Watch
1
0
Fork
You've already forked akh-medu
 0
Code Issues 10 Pull requests 22 Projects Releases 2 Packages Wiki Activity Actions 7

WIP: P26.3 — PolarQuant KV-cache compression in the Candle inference loop #313

Draft
toasterson wants to merge 3 commits from claude/wi-019f90cb-p26-3-polarquant-kv-cache-compression-in into main
pull from: claude/wi-019f90cb-p26-3-polarquant-kv-cache-compression-in
merge into: toasterson:main
Conversation 0 Commits 3 Files changed 4 +465 -3
toasterson commented 2026-08-12 19:26:52 +00:00
Owner
Copy link

Anima work item 019f90cb-b03b-7a13-9fff-549b3b772875.

Third slice of the re-scoped Phase 26. Depends on P26.2 (the Candle backend it plugs into).

Deliverable: PolarQuant KV-cache compression integrated into the Candle inference loop, targeting ~5–6× cache reduction so 7B–8B models fit on the M2.

Why it is its own slice: a custom quantisation scheme is where numerical bugs hide, and it must be independently benchmarkable and revertible. Bundling it with the backend swap (as the original WI did) makes a regression impossible to attribute — you could not tell whether Candle or PolarQuant caused a quality drop.

Acceptance:

  • Measured cache-size reduction against the uncompressed path, on a fixed workload.
  • Output-quality check: generation on a fixed prompt set must not degrade beyond an agreed threshold versus uncompressed. State the threshold in the PR.
  • A 7B or 8B model demonstrably loads and generates on the M2 where it previously did not — that is the whole point.
  • Compression is toggleable so it can be disabled to isolate future regressions.

Forward constraint: the original WI flagged an open question about whether the compression format needs special handling when serialising hidden states for remote Candle instances (Phase 39). Answer it here — even if the answer is "not applicable, hidden states are extracted pre-cache" — and record it, so Phase 39 does not rediscover it.

⚠ akh-medu repo. Ignore any inherited crates/anima-ai/... path list.

Anima work item `019f90cb-b03b-7a13-9fff-549b3b772875`. Third slice of the re-scoped Phase 26. **Depends on P26.2** (the Candle backend it plugs into). **Deliverable:** PolarQuant KV-cache compression integrated into the Candle inference loop, targeting ~5–6× cache reduction so 7B–8B models fit on the M2. **Why it is its own slice:** a custom quantisation scheme is where numerical bugs hide, and it must be independently benchmarkable and revertible. Bundling it with the backend swap (as the original WI did) makes a regression impossible to attribute — you could not tell whether Candle or PolarQuant caused a quality drop. **Acceptance:** - Measured cache-size reduction against the uncompressed path, on a fixed workload. - **Output-quality check**: generation on a fixed prompt set must not degrade beyond an agreed threshold versus uncompressed. State the threshold in the PR. - A 7B or 8B model demonstrably loads and generates on the M2 where it previously did not — that is the whole point. - Compression is toggleable so it can be disabled to isolate future regressions. **Forward constraint:** the original WI flagged an open question about whether the compression format needs special handling when serialising hidden states for remote Candle instances (Phase 39). Answer it here — even if the answer is "not applicable, hidden states are extracted pre-cache" — and record it, so Phase 39 does not rediscover it. ⚠ akh-medu repo. Ignore any inherited `crates/anima-ai/...` path list.
P26.3 — PolarQuant KV-cache compression in the Candle inference loop
Some checks failed
CI / publish-chart (pull_request) Successful in 9m58s
Details
CI / check-seshat (pull_request) Successful in 12m44s
Details
CI / test-akh-medu (pull_request) Failing after 13m31s
Details
CI / check-akh-medu (pull_request) Failing after 13m33s
Details
CI / docker-seshd (pull_request) Successful in 17m10s
Details
c8cf895acb 
Anima implementation session.
P26.3 — PolarQuant KV-cache compression in the Candle inference loop
Some checks failed
CI / check-seshat (pull_request) Successful in 11m37s
Details
CI / test-akh-medu (pull_request) Failing after 13m10s
Details
CI / publish-chart (pull_request) Successful in 1m49s
Details
CI / check-akh-medu (pull_request) Failing after 14m21s
Details
CI / docker-seshd (pull_request) Successful in 16m59s
Details
17b8e5b7bd 
Anima implementation session.
P26.3 — PolarQuant KV-cache compression in the Candle inference loop
Some checks failed
CI / publish-chart (pull_request) Successful in 14m31s
Details
CI / check-seshat (pull_request) Successful in 16m13s
Details
CI / check-akh-medu (pull_request) Successful in 17m28s
Details
CI / test-akh-medu (pull_request) Failing after 19m57s
Details
CI / docker-seshd (pull_request) Successful in 21m22s
Details
99a428eda4 
Anima implementation session.
Some checks failed
CI / publish-chart (pull_request) Successful in 14m31s
Details
CI / check-seshat (pull_request) Successful in 16m13s
Details
CI / check-akh-medu (pull_request) Successful in 17m28s
Details
CI / test-akh-medu (pull_request) Failing after 19m57s
Details
CI / docker-seshd (pull_request) Successful in 21m22s
Details
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin claude/wi-019f90cb-p26-3-polarquant-kv-cache-compression-in:claude/wi-019f90cb-p26-3-polarquant-kv-cache-compression-in
git switch claude/wi-019f90cb-p26-3-polarquant-kv-cache-compression-in

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff claude/wi-019f90cb-p26-3-polarquant-kv-cache-compression-in
git switch claude/wi-019f90cb-p26-3-polarquant-kv-cache-compression-in
git rebase main
git switch main
git merge --ff-only claude/wi-019f90cb-p26-3-polarquant-kv-cache-compression-in
git switch claude/wi-019f90cb-p26-3-polarquant-kv-cache-compression-in
git rebase main
git switch main
git merge --no-ff claude/wi-019f90cb-p26-3-polarquant-kv-cache-compression-in
git switch main
git merge --squash claude/wi-019f90cb-p26-3-polarquant-kv-cache-compression-in
git switch main
git merge --ff-only claude/wi-019f90cb-p26-3-polarquant-kv-cache-compression-in
git switch main
git merge claude/wi-019f90cb-p26-3-polarquant-kv-cache-compression-in
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
toasterson/akh-medu!313
No description provided.