WIP: sync_backfill returns a gateway timeout even when the import succeeds #11

Draft

toasterson wants to merge 1 commit from claude/wi-019f907a-sync-backfill-returns-a-gateway-timeout into main

pull from: claude/wi-019f907a-sync-backfill-returns-a-gateway-timeout

merge into: toasterson:main

toasterson:main

toasterson:claude/wi-019e534c-multi-arch-container-image-amd64-arm64

toasterson:claude/wi-019e534c-switch-from-latest-to-sha-pinned-image-t

toasterson:claude/wi-019e534c-remove-keel-auto-update-mechanism

toasterson:claude/wi-019e535b-fix-sandbox-classifier-allow-null-contri

toasterson:claude/wi-019e535b-deduplicate-sender-detection-70-duplicat

toasterson:claude/wi-019e535b-add-person-list-person-update-route-add

toasterson:claude/wi-019e535b-three-tier-processing-gate-observe-sesha

Conversation 0 Commits 1 Files changed 18 +858 -116

toasterson commented 2026-07-25 15:10:55 +00:00

Owner

Copy link

Anima work item 019f907a-101a-71c2-a950-36fbd2e94d9e.

Symptom

mcp__sia__sync_backfill (and a broad sync_search) reliably return Streamable HTTP error: upstream request timeout from the MCP gateway, even when the underlying work completes. Observed 2026-07-23 backfilling the two illumos/ipd PR threads: both calls timed out at the HTTP layer, yet the messages were fetched, classified, and routed successfully (decisions #1375/#1376, MailingList; 2/2 ok). The caller can only confirm success by inspecting decision_recent afterward.

Cause

The backfill path does real work synchronously inside the request: fetch full bodies from the provider → run the tools-less sandbox classifier (LLM) → anima gRPC upserts. For more than a message or two this exceeds the MCP gateway's response window, so the HTTP response times out while the server keeps working. The gateway timeout is unrelated to whether the import succeeds.

Impact

• Cosmetic-but-confusing: a successful backfill looks like a failure.
• Risk of accidental double-invocation (re-running a "failed" backfill). Dedup by external_ref currently saves us, but that's luck, not design.

Options

1. Make backfill return promptly by enqueuing the import as a background job and returning a job id / accepted-count immediately; expose status via a follow-up tool or the decision log.
2. Stream incremental progress so the gateway connection stays alive (per-message events) instead of one long blocking response.
3. Cheapest interim: raise the MCP gateway's per-request timeout for the backfill/search routes so realistic batches (≤50 msgs) return cleanly.

Prefer (1) for anything beyond a handful of messages; (3) is an acceptable stopgap.

Repro

mcp__sia__sync_backfill with a query: that resolves to >2 messages, or sync_search over a broad query, against the personal Gmail mailbox. HTTP times out; decision_recent shows the work landed.

Anima work item `019f907a-101a-71c2-a950-36fbd2e94d9e`. ## Symptom `mcp__sia__sync_backfill` (and a broad `sync_search`) reliably return `Streamable HTTP error: upstream request timeout` from the MCP gateway, even when the underlying work completes. Observed 2026-07-23 backfilling the two `illumos/ipd` PR threads: both calls timed out at the HTTP layer, yet the messages were fetched, classified, and routed successfully (decisions #1375/#1376, `MailingList; 2/2 ok`). The caller can only confirm success by inspecting `decision_recent` afterward. ## Cause The backfill path does real work synchronously inside the request: fetch full bodies from the provider → run the tools-less sandbox classifier (LLM) → anima gRPC upserts. For more than a message or two this exceeds the MCP gateway's response window, so the HTTP response times out while the server keeps working. The gateway timeout is unrelated to whether the import succeeds. ## Impact - Cosmetic-but-confusing: a successful backfill looks like a failure. - Risk of accidental double-invocation (re-running a "failed" backfill). Dedup by external_ref currently saves us, but that's luck, not design. ## Options 1. Make backfill return promptly by enqueuing the import as a background job and returning a job id / accepted-count immediately; expose status via a follow-up tool or the decision log. 2. Stream incremental progress so the gateway connection stays alive (per-message events) instead of one long blocking response. 3. Cheapest interim: raise the MCP gateway's per-request timeout for the backfill/search routes so realistic batches (≤50 msgs) return cleanly. Prefer (1) for anything beyond a handful of messages; (3) is an acceptable stopgap. ## Repro `mcp__sia__sync_backfill` with a `query:` that resolves to >2 messages, or `sync_search` over a broad query, against the personal Gmail mailbox. HTTP times out; `decision_recent` shows the work landed.

toasterson added 1 commit 2026-07-25 15:10:56 +00:00

sync_backfill returns a gateway timeout even when the import succeeds ... b9c2a6536d

Anima implementation session.

toasterson force-pushed claude/wi-019f907a-sync-backfill-returns-a-gateway-timeout from b9c2a6536d to 67aeaf60ef 2026-07-25 19:31:59 +00:00 Compare

All checks were successful

Hide checks

CI / test (pull_request) Successful in 36m19s

Details

CI / docker (pull_request) Has been skipped

Details

CI / test (push) Successful in 36m23s

Details

CI / docker (push) Has been skipped

Details

This pull request is marked as a work in progress.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin claude/wi-019f907a-sync-backfill-returns-a-gateway-timeout:claude/wi-019f907a-sync-backfill-returns-a-gateway-timeout

git switch claude/wi-019f907a-sync-backfill-returns-a-gateway-timeout

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff claude/wi-019f907a-sync-backfill-returns-a-gateway-timeout

git switch claude/wi-019f907a-sync-backfill-returns-a-gateway-timeout

git rebase main

git switch main

git merge --ff-only claude/wi-019f907a-sync-backfill-returns-a-gateway-timeout

git switch claude/wi-019f907a-sync-backfill-returns-a-gateway-timeout

git rebase main

git switch main

git merge --no-ff claude/wi-019f907a-sync-backfill-returns-a-gateway-timeout

git switch main

git merge --squash claude/wi-019f907a-sync-backfill-returns-a-gateway-timeout

git switch main

git merge --ff-only claude/wi-019f907a-sync-backfill-returns-a-gateway-timeout

git switch main

git merge claude/wi-019f907a-sync-backfill-returns-a-gateway-timeout

git push origin main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/sia!11

No description provided.