WI: Implement YAML test spec parser and executor #185

New issue

Closed

opened 2026-05-24 05:13:26 +00:00 by toasterson · 0 comments

toasterson commented

2026-05-24 05:13:26 +00:00

Owner

Goal

Parse YAML test specification files and execute test suites end-to-end via the QA MCP.

Background

The .qa.yaml format defines complete test suites: VM setup, ordered steps, assertions, and cleanup. This WI implements the parsing and execution engine.

🔗 Remote Sources (all on code.aopc.cloud)

Anima workspace:

https://code.aopc.cloud/toasterson/Anima/src/branch/main/crates/anima-qa/src/mcp.rs — MCP client used by runner
test specs: tests/qa/*.qa.yaml in the Anima repo

Design doc: Ask Tecton for memory/projects/anima-qa.md — full YAML schema spec with example test files

🏗️ Implementation Guide

Step 1: YAML data structures

Define structs matching the .qa.yaml format:

TestSuite: name, wi_ref, project, image, timeout_seconds, setup[], steps[], assertions[]
SetupStep: type: "shell", command, background (bool)
TestStep: description, type (wait_for_text|assert_visible|input|screenshot), type-specific params
Assertion: name, type (screenshot_exists), params

Step 2: Parser

Use serde_yaml with serde's tagged enum for type-differentiated step variants. Handle parsing errors gracefully.

Step 3: MCP client

QaMcpClient struct: connects to MCP server URL, methods for each tool, JSON-RPC request/response handling, retry with exponential backoff

Step 4: Test executor

TestRunner::run(test_suite): parse YAML → qa_start → setup commands → steps → assertions → qa_stop → write results.json

Step 5: Wire into CLI

Update main.rs runner subcommand to load TestSuite, create TestRunner, execute, print progress, exit with appropriate code.

✅ Acceptance Criteria

YAML test specs parse correctly
Setup commands execute in order
Steps map to MCP tool calls
Screenshots collected and saved
JSON results with per-step pass/fail
VM cleanup always happens

#181 — MCP server
#182 — CLI runner (overlap — coordinate on subcommand architecture)
#184 — visual primitives

## Goal Parse YAML test specification files and execute test suites end-to-end via the QA MCP. ## Background The `.qa.yaml` format defines complete test suites: VM setup, ordered steps, assertions, and cleanup. This WI implements the parsing and execution engine. ## 🔗 Remote Sources (all on code.aopc.cloud) **Anima workspace:** - https://code.aopc.cloud/toasterson/Anima/src/branch/main/crates/anima-qa/src/mcp.rs — MCP client used by runner - test specs: `tests/qa/*.qa.yaml` in the Anima repo **Design doc:** Ask Tecton for `memory/projects/anima-qa.md` — full YAML schema spec with example test files ## 🏗️ Implementation Guide ### Step 1: YAML data structures Define structs matching the .qa.yaml format: - TestSuite: name, wi_ref, project, image, timeout_seconds, setup[], steps[], assertions[] - SetupStep: type: "shell", command, background (bool) - TestStep: description, type (wait_for_text|assert_visible|input|screenshot), type-specific params - Assertion: name, type (screenshot_exists), params ### Step 2: Parser Use serde_yaml with serde's tagged enum for type-differentiated step variants. Handle parsing errors gracefully. ### Step 3: MCP client QaMcpClient struct: connects to MCP server URL, methods for each tool, JSON-RPC request/response handling, retry with exponential backoff ### Step 4: Test executor TestRunner::run(test_suite): parse YAML → qa_start → setup commands → steps → assertions → qa_stop → write results.json ### Step 5: Wire into CLI Update main.rs runner subcommand to load TestSuite, create TestRunner, execute, print progress, exit with appropriate code. ## ✅ Acceptance Criteria - [ ] YAML test specs parse correctly - [ ] Setup commands execute in order - [ ] Steps map to MCP tool calls - [ ] Screenshots collected and saved - [ ] JSON results with per-step pass/fail - [ ] VM cleanup always happens ## Related WIs - #181 — MCP server - #182 — CLI runner (overlap — coordinate on subcommand architecture) - #184 — visual primitives

toasterson referenced this issue

2026-05-24 05:22:20 +00:00

WI: NeoCDE shell startup smoke test suite #187

toasterson referenced this issue

2026-05-24 05:22:20 +00:00

WI: WayRay compositor smoke test suite #188

toasterson referenced this issue

2026-05-24 05:22:20 +00:00

WI: Anima Desktop UI smoke test suite #189

toasterson referenced this issue

2026-05-24 05:30:28 +00:00

WI: Build QA runner CLI for CI integration #182