WIP: ADR 081 S0 — pay the R2 empirical bet: probe small-model schema.org emission, format-under-load, and tool suppression #310

Draft
toasterson wants to merge 5 commits from claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob into main
Owner

Anima work item 019fa8f0-8fce-7313-b5f0-b60d498c1785.

Origin

Gap G6 of docs/ai/research/2026-07-28-llm-wiki-adversarial-gap-analysis.md (Seshat: 2026-07-28-llm-wiki-adversarial-gap-analysis). The only P0 in that analysis with no existing work item.

The problem

ADR 081's own risk register says it plainly:

"The whole tiered-authoring premise rests on the R2 empirical bet — must be probed first."

and the plan ordered it first: "★ R2 is an empirical bet — de-risk first."

It was not done. ADR 081's status line records:

"Still open: the S0 empirical bet (live small-model schema.org emission — structural seams shipped with deterministic fakes instead)."

So nine vertical slices (P1, P2, S0-structural, slices 1–9) landed on top of an unvalidated premise. If the premise is false, decision 10 (tiered authoring: small local model writes append-only with type-guessing salience markers) needs rework, and the structured task tool's reliability argument goes with it.

Why now — the literature moved against it since 2026-06-30

None of this existed when ADR 081 was written:

  • The Format Tax (arXiv:2604.03616) — structured-output requirements substantially degrade reasoning across open-weight models, and "format-requesting instructions alone cause most of the accuracy loss, before any decoder constraint is applied."
  • The Constraint Tax (arXiv:2605.26128) — the assumption that hard constraints improve reliability without changing the answer "is unsafe for small models."
  • When Correct Isn't Usable (arXiv:2605.02363) — across three 7–9B models, joint task-correctness and schema-validity diverges sharply from task accuracy measured alone.
  • Tool Suppression (arXiv:2606.25605) — when tool-calling and JSON-Schema constraints are enabled simultaneously, multiple open-weight models stop calling tools while maintaining perfect schema compliance.
  • Counterweight, in fairness: arXiv:2510.03847 argues SLMs (1–12B) are "sufficient and often superior for agentic workloads where the objective is schema- and API-constrained accuracy" when paired with guided decoding. The bet is contested, not refuted — which is exactly why it needs measuring rather than assuming.

Note what is not under attack: ADR 081's mitigation — anchor on schema.org because "small models already emit it," converting invent into recall — is a good instinct and may well hold. The literature attacks the structured-emission channel, independent of vocabulary familiarity. Those are separable and must be measured separately, which is why this WI has three channels rather than one pass/fail.

Scope — three independently-failing channels

Run each separately; a single blended "does gemma4 emit good frontmatter?" result is not actionable.

1. Vocabulary recall. Does gemma4-12b emit correct schema.org type names for known entities? No formatting pressure — ask for a bare type name. This isolates the part ADR 081's mitigation actually defends. Expected to pass.

2. Format compliance under load. Does frontmatter emission survive when the model is simultaneously reasoning about content? Measure joint correctness (content right AND frontmatter valid), per arXiv:2605.02363's methodology — not format validity alone, which is the metric that hides the tax.

3. Tool-calling under constraint. Does the structured task tool suppress other tool calls when grammar constraints are active? Run the same task set with constraints on and off and compare tool-invocation counts. This is the channel that also threatens ADR 067's grammar-constrained Candle tool calling and the agentic loop generally — a positive result here is a cross-cutting finding, not just an ADR 081 one.

Harness

examples/agentic_flow.rs + AKH_AGENT_DEBUG=1 — the harness the ADR 081 plan already named. No new infrastructure. Where a comparison arm is needed, src/bench_phase44/ already provides the deterministic A/B scaffolding.

Acceptance

  • Three separate result tables, one per channel, with the model build and sampling params recorded.
  • Channel 2 reports joint correctness, not format validity alone.
  • Channel 3 reports tool-invocation counts with constraints on vs. off on a matched task set.
  • ADR 081's status line updated with the outcome — replacing "structural seams shipped with deterministic fakes instead" with a measured result either way.
  • If any channel fails: a short note on which ADR 081 decisions it invalidates (candidates: decision 10 tiered authoring, the structured task tool's reliability argument, and the AKH_GRAMMAR_* defaults).

Effort

S. Roughly a day. This is the cheapest available action that retires the largest single risk in ADR 081.

Anima work item `019fa8f0-8fce-7313-b5f0-b60d498c1785`. ## Origin Gap **G6** of [`docs/ai/research/2026-07-28-llm-wiki-adversarial-gap-analysis.md`](docs/ai/research/2026-07-28-llm-wiki-adversarial-gap-analysis.md) (Seshat: `2026-07-28-llm-wiki-adversarial-gap-analysis`). The only **P0** in that analysis with no existing work item. ## The problem ADR 081's own risk register says it plainly: > "The whole tiered-authoring premise rests on the **R2 empirical bet** — must be probed first." and the plan ordered it first: *"★ R2 is an empirical bet — de-risk first."* It was not done. ADR 081's status line records: > "Still open: the S0 **empirical** bet (live small-model schema.org emission — **structural seams shipped with deterministic fakes instead**)." So nine vertical slices (P1, P2, S0-structural, slices 1–9) landed on top of an unvalidated premise. If the premise is false, decision 10 (tiered authoring: small local model writes append-only with type-guessing salience markers) needs rework, and the structured task tool's reliability argument goes with it. ## Why now — the literature moved against it since 2026-06-30 None of this existed when ADR 081 was written: - **The Format Tax** (arXiv:2604.03616) — structured-output requirements substantially degrade reasoning across open-weight models, and *"format-requesting instructions alone cause most of the accuracy loss, before any decoder constraint is applied."* - **The Constraint Tax** (arXiv:2605.26128) — the assumption that hard constraints improve reliability without changing the answer *"is unsafe for small models."* - **When Correct Isn't Usable** (arXiv:2605.02363) — across three 7–9B models, joint task-correctness *and* schema-validity diverges sharply from task accuracy measured alone. - **Tool Suppression** (arXiv:2606.25605) — when tool-calling and JSON-Schema constraints are enabled **simultaneously**, multiple open-weight models **stop calling tools** while maintaining perfect schema compliance. - Counterweight, in fairness: arXiv:2510.03847 argues SLMs (1–12B) are "sufficient and often superior for agentic workloads where the objective is schema- and API-constrained accuracy" *when paired with guided decoding*. The bet is **contested, not refuted** — which is exactly why it needs measuring rather than assuming. Note what is *not* under attack: ADR 081's mitigation — anchor on schema.org because "small models already emit it," converting *invent* into *recall* — is a good instinct and may well hold. The literature attacks the **structured-emission channel**, independent of vocabulary familiarity. Those are separable and must be measured separately, which is why this WI has three channels rather than one pass/fail. ## Scope — three independently-failing channels Run each separately; a single blended "does gemma4 emit good frontmatter?" result is not actionable. **1. Vocabulary recall.** Does gemma4-12b emit *correct schema.org type names* for known entities? No formatting pressure — ask for a bare type name. This isolates the part ADR 081's mitigation actually defends. Expected to pass. **2. Format compliance under load.** Does frontmatter emission survive when the model is simultaneously reasoning about content? Measure **joint** correctness (content right AND frontmatter valid), per arXiv:2605.02363's methodology — not format validity alone, which is the metric that hides the tax. **3. Tool-calling under constraint.** Does the structured task tool suppress *other* tool calls when grammar constraints are active? Run the same task set with constraints on and off and compare tool-invocation counts. This is the channel that also threatens [ADR 067](docs/ai/decisions/067-candle-conversational-boundary.md)'s grammar-constrained Candle tool calling and the agentic loop generally — a positive result here is a cross-cutting finding, not just an ADR 081 one. ## Harness `examples/agentic_flow.rs` + `AKH_AGENT_DEBUG=1` — the harness the ADR 081 plan already named. No new infrastructure. Where a comparison arm is needed, `src/bench_phase44/` already provides the deterministic A/B scaffolding. ## Acceptance - Three separate result tables, one per channel, with the model build and sampling params recorded. - Channel 2 reports **joint** correctness, not format validity alone. - Channel 3 reports tool-invocation counts with constraints on vs. off on a matched task set. - ADR 081's status line updated with the outcome — replacing "structural seams shipped with deterministic fakes instead" with a measured result either way. - If any channel fails: a short note on which ADR 081 decisions it invalidates (candidates: decision 10 tiered authoring, the structured task tool's reliability argument, and the `AKH_GRAMMAR_*` defaults). ## Effort **S.** Roughly a day. This is the cheapest available action that retires the largest single risk in ADR 081.
ADR 081 S0 — pay the R2 empirical bet: probe small-model schema.org emission, format-under-load, and tool suppression
Some checks failed
CI / publish-chart (pull_request) Failing after 18m7s
CI / check-seshat (pull_request) Failing after 23m52s
CI / docker-seshd (pull_request) Failing after 28m15s
CI / check-akh-medu (pull_request) Failing after 28m42s
CI / test-akh-medu (pull_request) Failing after 33m22s
92eba39d73
Anima implementation session.
ADR 081 S0 — pay the R2 empirical bet: probe small-model schema.org emission, format-under-load, and tool suppression
All checks were successful
CI / publish-chart (pull_request) Successful in 39m58s
CI / check-seshat (pull_request) Successful in 41m51s
CI / check-akh-medu (pull_request) Successful in 43m43s
CI / docker-seshd (pull_request) Successful in 46m39s
CI / test-akh-medu (pull_request) Successful in 48m16s
a7e47995d0
Anima implementation session.
toasterson force-pushed claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob from a7e47995d0
All checks were successful
CI / publish-chart (pull_request) Successful in 39m58s
CI / check-seshat (pull_request) Successful in 41m51s
CI / check-akh-medu (pull_request) Successful in 43m43s
CI / docker-seshd (pull_request) Successful in 46m39s
CI / test-akh-medu (pull_request) Successful in 48m16s
to 80d89828f3
Some checks failed
CI / check-seshat (pull_request) Successful in 9m19s
CI / check-akh-medu (pull_request) Successful in 11m50s
CI / publish-chart (pull_request) Successful in 3m10s
CI / docker-seshd (pull_request) Successful in 15m36s
CI / test-akh-medu (pull_request) Failing after 18m26s
2026-08-11 22:34:30 +00:00
Compare
ADR 081 S0 — pay the R2 empirical bet: probe small-model schema.org emission, format-under-load, and tool suppression
Some checks failed
CI / check-seshat (pull_request) Successful in 17m27s
CI / check-akh-medu (pull_request) Successful in 29m49s
CI / test-akh-medu (pull_request) Failing after 34m10s
CI / publish-chart (pull_request) Successful in 34m35s
CI / docker-seshd (pull_request) Successful in 40m18s
3b7bb3664d
Anima implementation session.
toasterson force-pushed claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob from 3b7bb3664d
Some checks failed
CI / check-seshat (pull_request) Successful in 17m27s
CI / check-akh-medu (pull_request) Successful in 29m49s
CI / test-akh-medu (pull_request) Failing after 34m10s
CI / publish-chart (pull_request) Successful in 34m35s
CI / docker-seshd (pull_request) Successful in 40m18s
to e576149bc6
Some checks failed
CI / check-seshat (pull_request) Successful in 1h32m57s
CI / check-akh-medu (pull_request) Successful in 1h39m0s
CI / test-akh-medu (pull_request) Failing after 1h48m8s
CI / docker-seshd (pull_request) Failing after 2h2m19s
CI / publish-chart (pull_request) Failing after 2h5m0s
2026-08-15 12:19:23 +00:00
Compare
toasterson force-pushed claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob from e576149bc6
Some checks failed
CI / check-seshat (pull_request) Successful in 1h32m57s
CI / check-akh-medu (pull_request) Successful in 1h39m0s
CI / test-akh-medu (pull_request) Failing after 1h48m8s
CI / docker-seshd (pull_request) Failing after 2h2m19s
CI / publish-chart (pull_request) Failing after 2h5m0s
to 820f915ac0
Some checks failed
CI / check-akh-medu (pull_request) Failing after 2h3m43s
CI / publish-chart (pull_request) Failing after 2h3m28s
CI / test-akh-medu (pull_request) Failing after 2h3m42s
CI / check-seshat (pull_request) Failing after 2h3m59s
eval-smoke / eval-smoke (pull_request) Failing after 2h3m44s
CI / docker-seshd (pull_request) Failing after 2h3m59s
2026-08-18 20:47:37 +00:00
Compare
ADR 081 S0 — pay the R2 empirical bet: probe small-model schema.org emission, format-under-load, and tool suppression
Some checks failed
CI / check-seshat (pull_request) Failing after 2h2m51s
eval-smoke / eval-smoke (pull_request) Failing after 2h2m42s
CI / publish-chart (pull_request) Failing after 2h2m50s
CI / check-akh-medu (pull_request) Failing after 2h2m59s
CI / docker-seshd (pull_request) Failing after 2h3m36s
CI / test-akh-medu (pull_request) Failing after 2h3m39s
a665639ce2
Anima implementation session.
toasterson force-pushed claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob from a665639ce2
Some checks failed
CI / check-seshat (pull_request) Failing after 2h2m51s
eval-smoke / eval-smoke (pull_request) Failing after 2h2m42s
CI / publish-chart (pull_request) Failing after 2h2m50s
CI / check-akh-medu (pull_request) Failing after 2h2m59s
CI / docker-seshd (pull_request) Failing after 2h3m36s
CI / test-akh-medu (pull_request) Failing after 2h3m39s
to a855ab10e0
Some checks failed
CI / check-seshat (pull_request) Has been cancelled
CI / check-akh-medu (pull_request) Has been cancelled
CI / docker-seshd (pull_request) Has been cancelled
CI / test-akh-medu (pull_request) Has been cancelled
CI / publish-chart (pull_request) Has been cancelled
eval-smoke / eval-smoke (pull_request) Has been cancelled
2026-08-19 18:05:47 +00:00
Compare
toasterson force-pushed claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob from a855ab10e0
Some checks failed
CI / check-seshat (pull_request) Has been cancelled
CI / check-akh-medu (pull_request) Has been cancelled
CI / docker-seshd (pull_request) Has been cancelled
CI / test-akh-medu (pull_request) Has been cancelled
CI / publish-chart (pull_request) Has been cancelled
eval-smoke / eval-smoke (pull_request) Has been cancelled
to 086fe35ed8
Some checks failed
CI / publish-chart (pull_request) Has been cancelled
CI / test-akh-medu (pull_request) Has been cancelled
CI / check-seshat (pull_request) Has been cancelled
CI / docker-seshd (pull_request) Has been cancelled
eval-smoke / eval-smoke (pull_request) Has been cancelled
CI / check-akh-medu (pull_request) Has been cancelled
2026-08-19 19:47:02 +00:00
Compare
toasterson force-pushed claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob from 086fe35ed8
Some checks failed
CI / publish-chart (pull_request) Has been cancelled
CI / test-akh-medu (pull_request) Has been cancelled
CI / check-seshat (pull_request) Has been cancelled
CI / docker-seshd (pull_request) Has been cancelled
eval-smoke / eval-smoke (pull_request) Has been cancelled
CI / check-akh-medu (pull_request) Has been cancelled
to 8793772082
Some checks failed
CI / check-seshat (pull_request) Has been cancelled
CI / check-akh-medu (pull_request) Has been cancelled
CI / test-akh-medu (pull_request) Has been cancelled
CI / docker-seshd (pull_request) Has been cancelled
CI / publish-chart (pull_request) Has been cancelled
eval-smoke / eval-smoke (pull_request) Has been cancelled
2026-08-19 20:55:36 +00:00
Compare
toasterson force-pushed claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob from 8793772082
Some checks failed
CI / check-seshat (pull_request) Has been cancelled
CI / check-akh-medu (pull_request) Has been cancelled
CI / test-akh-medu (pull_request) Has been cancelled
CI / docker-seshd (pull_request) Has been cancelled
CI / publish-chart (pull_request) Has been cancelled
eval-smoke / eval-smoke (pull_request) Has been cancelled
to d5ed56fe81
Some checks failed
CI / check-seshat (pull_request) Has been cancelled
CI / check-akh-medu (pull_request) Has been cancelled
CI / test-akh-medu (pull_request) Has been cancelled
CI / docker-seshd (pull_request) Has been cancelled
CI / publish-chart (pull_request) Has been cancelled
eval-smoke / eval-smoke (pull_request) Has been cancelled
2026-08-19 21:31:02 +00:00
Compare
toasterson force-pushed claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob from d5ed56fe81
Some checks failed
CI / check-seshat (pull_request) Has been cancelled
CI / check-akh-medu (pull_request) Has been cancelled
CI / test-akh-medu (pull_request) Has been cancelled
CI / docker-seshd (pull_request) Has been cancelled
CI / publish-chart (pull_request) Has been cancelled
eval-smoke / eval-smoke (pull_request) Has been cancelled
to d3b6b0c71d
Some checks are pending
CI / check-seshat (pull_request) Waiting to run
CI / check-akh-medu (pull_request) Waiting to run
CI / test-akh-medu (pull_request) Waiting to run
CI / docker-seshd (pull_request) Waiting to run
CI / publish-chart (pull_request) Waiting to run
eval-smoke / eval-smoke (pull_request) Waiting to run
2026-08-19 22:02:12 +00:00
Compare
Some checks are pending
CI / check-seshat (pull_request) Waiting to run
CI / check-akh-medu (pull_request) Waiting to run
CI / test-akh-medu (pull_request) Waiting to run
CI / docker-seshd (pull_request) Waiting to run
CI / publish-chart (pull_request) Waiting to run
eval-smoke / eval-smoke (pull_request) Waiting to run
This pull request is marked as a work in progress.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob:claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob
git switch claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob
git switch claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob
git rebase main
git switch main
git merge --ff-only claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob
git switch claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob
git rebase main
git switch main
git merge --no-ff claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob
git switch main
git merge --squash claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob
git switch main
git merge --ff-only claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob
git switch main
git merge claude/wi-019fa8f0-adr-081-s0-pay-the-r2-empirical-bet-prob
git push origin main
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
toasterson/akh-medu!310
No description provided.