WI: Build QA runner CLI for CI integration #182

New issue

Closed

opened 2026-05-24 05:11:17 +00:00 by toasterson · 0 comments

toasterson commented

2026-05-24 05:11:17 +00:00

Owner

Goal

Create a cargo run -p anima-qa -- runner subcommand that reads YAML test specs and drives the QA MCP server programmatically.

Background

The MCP server handles VM lifecycle and individual actions. The runner CLI wraps this for CI: parse a test spec file, call the MCP sequentially, collect results, emit pass/fail. This is the CI-facing interface.

🔗 Remote Sources (all on code.aopc.cloud)

Anima workspace:

https://code.aopc.cloud/toasterson/Anima/src/branch/main/Cargo.toml — workspace deps (check if clap is already included, otherwise add)
https://code.aopc.cloud/toasterson/Anima/src/branch/main/crates/anima-qa/ — the MCP server crate from WI #180/#181

QA test specs: Ask Tecton for memory/projects/anima-qa.md — full YAML schema definition

🏗️ Implementation Guide

Step 1: Add subcommand structure

Extend main.rs with a runner subcommand using clap:

--mcp-url (default: http://localhost:8090)
--test (path to .qa.yaml file)
--vm-image (default: debian-bookworm-wayland-dev:latest)
--output (default: qa-results/)

Step 2: Test spec parser

Parse .qa.yaml files into typed structs: TestSuite with name, wi_ref, project, image, timeout_seconds, setup[], steps[], assertions[]. Use serde_yaml (add to Cargo.toml if not in workspace).

Step 3: MCP client

HTTP client that POSTs JSON-RPC 2.0 requests to the MCP server. Map test step types to MCP tool calls:

shell → qa_run_cli
wait_for_text → qa_wait_for_text
assert_visible → qa_assert_visible
input/type → qa_type
input/click_at → qa_click_at
input/key_combo → qa_key_combo
screenshot → qa_screenshot

Decode base64 screenshots → save to output directory.

Step 4: Execution flow

Parse → qa_start → setup commands → test steps → assertions → qa_stop (always) → write results.json

✅ Acceptance Criteria

CLI parses test spec and executes via MCP
Screenshots saved to output dir
JSON results with per-step pass/fail
VM cleanup on failure
Exit code reflects results

#181 — MCP server (dependency)
#185 — test spec parser (may overlap)
#186 — CI integration

## Goal Create a `cargo run -p anima-qa -- runner` subcommand that reads YAML test specs and drives the QA MCP server programmatically. ## Background The MCP server handles VM lifecycle and individual actions. The runner CLI wraps this for CI: parse a test spec file, call the MCP sequentially, collect results, emit pass/fail. This is the CI-facing interface. ## 🔗 Remote Sources (all on code.aopc.cloud) **Anima workspace:** - https://code.aopc.cloud/toasterson/Anima/src/branch/main/Cargo.toml — workspace deps (check if clap is already included, otherwise add) - https://code.aopc.cloud/toasterson/Anima/src/branch/main/crates/anima-qa/ — the MCP server crate from WI #180/#181 **QA test specs:** Ask Tecton for `memory/projects/anima-qa.md` — full YAML schema definition ## 🏗️ Implementation Guide ### Step 1: Add subcommand structure Extend main.rs with a `runner` subcommand using clap: - `--mcp-url` (default: http://localhost:8090) - `--test` (path to .qa.yaml file) - `--vm-image` (default: debian-bookworm-wayland-dev:latest) - `--output` (default: qa-results/) ### Step 2: Test spec parser Parse `.qa.yaml` files into typed structs: TestSuite with name, wi_ref, project, image, timeout_seconds, setup[], steps[], assertions[]. Use serde_yaml (add to Cargo.toml if not in workspace). ### Step 3: MCP client HTTP client that POSTs JSON-RPC 2.0 requests to the MCP server. Map test step types to MCP tool calls: - shell → qa_run_cli - wait_for_text → qa_wait_for_text - assert_visible → qa_assert_visible - input/type → qa_type - input/click_at → qa_click_at - input/key_combo → qa_key_combo - screenshot → qa_screenshot Decode base64 screenshots → save to output directory. ### Step 4: Execution flow Parse → qa_start → setup commands → test steps → assertions → qa_stop (always) → write results.json ## ✅ Acceptance Criteria - [ ] CLI parses test spec and executes via MCP - [ ] Screenshots saved to output dir - [ ] JSON results with per-step pass/fail - [ ] VM cleanup on failure - [ ] Exit code reflects results ## Related WIs - #181 — MCP server (dependency) - #185 — test spec parser (may overlap) - #186 — CI integration

toasterson referenced this issue

2026-05-24 05:22:17 +00:00

WI: Implement anima-qa MCP server (HTTP + JSON-RPC) #181

toasterson referenced this issue

2026-05-24 05:22:19 +00:00

WI: Implement YAML test spec parser and executor #185

toasterson referenced this issue

2026-05-24 05:30:28 +00:00

WI: Create anima-qa crate in Anima workspace #180

toasterson closed this issue

2026-05-24 05:53:21 +00:00