Phase 42b — LongMemEval + LoCoMo harness with honest reporter #311

Merged

toasterson merged 4 commits from claude/wi-019dd0c5-phase-42b-longmemeval-locomo-harness-wit into main 2026-08-16 17:15:38 +00:00

Conversation 0 Commits 4 Files changed 11 +1213 -48

toasterson commented 2026-08-10 16:42:02 +00:00

Owner

Copy link

Anima work item 019dd0c5-ec42-7e52-8b00-c98bd36b2a94.

Ship the eval harness BEFORE retrieval is built so each subsequent sub-phase has a measurement loop.

Deliverables:

• crates/heka/src/eval/{mod,longmemeval,locomo,reporter}.rs
• eval_data/README.md with dataset fetch script (do NOT commit datasets — fetch script only)
• heka eval longmemeval --mode raw|hybrid runner
• heka eval locomo --top-k auto runner. Auto-cap: min(5, conv_len/4) to refuse the MemPalace top_k=50-on-30-items exploit by construction. --top-k-explicit N override emits a loud warning in the report.
• JSON + Markdown reporter capturing hardware, model versions, dataset hashes, exact retrieval params, p50/p99 latency.
• CI job eval-smoke posts delta vs baseline on every PR (non-blocking).

Until 42c lands, recall is a stub returning empty — harness reports 0%. Each subsequent sub-phase pushes the number up.

Acceptance:

• heka eval longmemeval --mode raw runs end-to-end on 10-session subset
• Reporter output matches eval_data/README.md format
• CI job runs on every PR (non-blocking)

Estimated lines: ~600
Dependencies: 42a

2026-06-04 22:00 UTC

Summary

Create a new heka CLI crate in the workspace to serve as an evaluation harness for LongMemEval and LoCoMo datasets, temporarily stubbing retrieval to 0% recall. This establishes a baseline measurement loop including hardware details, latency percentiles, and an automated CI job to report evaluation diffs on pull requests.

Approach

Add a new heka binary crate to the Rust workspace and configure clap for the subcommands eval longmemeval (with --mode raw|hybrid) and eval locomo (with --top-k auto and --top-k-explicit). Implement the retrieval stub to return empty results in the respective dataset handlers, enforcing the 0% recall baseline. In locomo.rs, conditionally enforce the min(5, conv_len/4) cap when auto top-k is used, and use tracing/log to emit a loud warning for explicit overrides. In reporter.rs, utilize a crate like sysinfo to capture system metadata, record timing payloads to calculate p50/p99 latencies, and serialize the final report into both JSON and Markdown formats. Add a dataset fetch bash script to eval_data/README.md, and configure a non-blocking eval-smoke CI pipeline to run the 10-session raw subset and compute deltas against the main branch.

Files likely to change

• Cargo.toml
• crates/heka/Cargo.toml
• crates/heka/src/main.rs
• crates/heka/src/eval/mod.rs
• crates/heka/src/eval/longmemeval.rs
• crates/heka/src/eval/locomo.rs
• crates/heka/src/eval/reporter.rs
• eval_data/README.md
• .forgejo/workflows/eval-smoke.yml (guess)

Open questions

• Where should the CI pipeline retrieve the stable "baseline" JSON metrics from in order to compute the PR delta (e.g., job artifacts from main, a dedicated branch)?
• Does the dataset fetch script in eval_data/README.md need to support authenticated downloads (e.g., a HuggingFace token), or are the dataset URLs strictly public?
• Is standard GitHub Actions (.github/workflows) used for CI, or should it explicitly target Forgejo Actions (.forgejo/workflows) given the presence of the anima-forgejo crate?

Complexity

M: 1-3 days. Bootstrapping a new CLI crate, extracting hardware telemetry, implementing statistical latency tracking, and setting up the PR delta CI pipeline involves moderate effort across different domains without deep technical risk.

Anima work item `019dd0c5-ec42-7e52-8b00-c98bd36b2a94`. Ship the eval harness BEFORE retrieval is built so each subsequent sub-phase has a measurement loop. Deliverables: - `crates/heka/src/eval/{mod,longmemeval,locomo,reporter}.rs` - `eval_data/README.md` with dataset fetch script (do NOT commit datasets — fetch script only) - `heka eval longmemeval --mode raw|hybrid` runner - `heka eval locomo --top-k auto` runner. Auto-cap: `min(5, conv_len/4)` to refuse the MemPalace `top_k=50`-on-30-items exploit by construction. `--top-k-explicit N` override emits a loud warning in the report. - JSON + Markdown reporter capturing hardware, model versions, dataset hashes, exact retrieval params, p50/p99 latency. - CI job `eval-smoke` posts delta vs baseline on every PR (non-blocking). Until 42c lands, recall is a stub returning empty — harness reports 0%. Each subsequent sub-phase pushes the number up. Acceptance: - `heka eval longmemeval --mode raw` runs end-to-end on 10-session subset - Reporter output matches eval_data/README.md format - CI job runs on every PR (non-blocking) Estimated lines: ~600 Dependencies: 42a <!-- ANIMA TRIAGE PLAN --> 2026-06-04 22:00 UTC ## Summary Create a new `heka` CLI crate in the workspace to serve as an evaluation harness for LongMemEval and LoCoMo datasets, temporarily stubbing retrieval to 0% recall. This establishes a baseline measurement loop including hardware details, latency percentiles, and an automated CI job to report evaluation diffs on pull requests. ## Approach Add a new `heka` binary crate to the Rust workspace and configure `clap` for the subcommands `eval longmemeval` (with `--mode raw|hybrid`) and `eval locomo` (with `--top-k auto` and `--top-k-explicit`). Implement the retrieval stub to return empty results in the respective dataset handlers, enforcing the 0% recall baseline. In `locomo.rs`, conditionally enforce the `min(5, conv_len/4)` cap when auto top-k is used, and use `tracing`/`log` to emit a loud warning for explicit overrides. In `reporter.rs`, utilize a crate like `sysinfo` to capture system metadata, record timing payloads to calculate p50/p99 latencies, and serialize the final report into both JSON and Markdown formats. Add a dataset fetch bash script to `eval_data/README.md`, and configure a non-blocking `eval-smoke` CI pipeline to run the 10-session raw subset and compute deltas against the main branch. ## Files likely to change * `Cargo.toml` * `crates/heka/Cargo.toml` * `crates/heka/src/main.rs` * `crates/heka/src/eval/mod.rs` * `crates/heka/src/eval/longmemeval.rs` * `crates/heka/src/eval/locomo.rs` * `crates/heka/src/eval/reporter.rs` * `eval_data/README.md` * `.forgejo/workflows/eval-smoke.yml` (guess) ## Open questions * Where should the CI pipeline retrieve the stable "baseline" JSON metrics from in order to compute the PR delta (e.g., job artifacts from main, a dedicated branch)? * Does the dataset fetch script in `eval_data/README.md` need to support authenticated downloads (e.g., a HuggingFace token), or are the dataset URLs strictly public? * Is standard GitHub Actions (`.github/workflows`) used for CI, or should it explicitly target Forgejo Actions (`.forgejo/workflows`) given the presence of the `anima-forgejo` crate? ## Complexity M: 1-3 days. Bootstrapping a new CLI crate, extracting hardware telemetry, implementing statistical latency tracking, and setting up the PR delta CI pipeline involves moderate effort across different domains without deep technical risk.

toasterson added 2 commits 2026-08-10 16:42:03 +00:00

wip: triage Phase 42b — LongMemEval + LoCoMo harness with honest reporter 0fb455bd15

Phase 42b — LongMemEval + LoCoMo harness with honest reporter ... 96d3221f7f

Anima implementation session.

toasterson added 1 commit 2026-08-14 13:47:34 +00:00

Phase 42b — LongMemEval + LoCoMo harness with honest reporter ... cfd00c2258

Anima implementation session.

toasterson force-pushed claude/wi-019dd0c5-phase-42b-longmemeval-locomo-harness-wit from cfd00c2258 to 8e483e7230 2026-08-14 14:51:48 +00:00 Compare

toasterson added 1 commit 2026-08-14 15:21:08 +00:00

Phase 42b — LongMemEval + LoCoMo harness with honest reporter ... cd2821d616

Anima implementation session.

toasterson force-pushed claude/wi-019dd0c5-phase-42b-longmemeval-locomo-harness-wit from cd2821d616 to 77e87fb9d5 2026-08-15 12:59:14 +00:00 Compare

toasterson changed title from WIP: Phase 42b — LongMemEval + LoCoMo harness with honest reporter to Phase 42b — LongMemEval + LoCoMo harness with honest reporter 2026-08-16 17:15:27 +00:00

toasterson merged commit 54c2caddd9 into main 2026-08-16 17:15:38 +00:00

toasterson referenced this pull request from a commit 2026-08-16 17:15:40 +00:00

Phase 42b — LongMemEval + LoCoMo harness with honest reporter (#311)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/akh-medu!311

No description provided.