Add GitHub App support, AMQP integration, and webhook enhancements

- Extend GitHub webhook handler with signature validation, push, and pull request event handling.
- Add GitHub App authentication via JWT and installation token retrieval.
- Parse `.solstice/workflow.kdl` for job queuing with `runs_on`, `script`, and job grouping support.
- Integrate AMQP consumer for orchestrator results and structured job enqueueing.
- Add S3-compatible storage configuration for log uploads.
- Refactor CLI options and internal state for improved configuration management.
- Enhance dependencies for signature, JSON, and AMQP handling.
- Document GitHub integration

Signed-off-by: Till Wegmueller <toasterson@gmail.com>

crates/github-integration/Cargo.toml

@@ -8,5 +8,25 @@ common = { path = "../common" }
 clap = { version = "4", features = ["derive", "env"] }
 miette = { version = "7", features = ["fancy"] }
 tracing = "0.1"
-tokio = { version = "1", features = ["rt-multi-thread", "macros", "signal"] }
+tokio = { version = "1", features = ["rt-multi-thread", "macros", "signal", "fs", "io-util", "time"] }
+# HTTP + Webhooks
 axum = { version = "0.8", features = ["macros"] }
+reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls-native-roots"] }
+serde = { version = "1", features = ["derive"] }
+serde_json = "1"
+# Signature verification
+hmac = "0.12"
+sha2 = "0.10"
+hex = "0.4"
+# GitHub App auth
+jsonwebtoken = "9"
+time = { version = "0.3", features = ["formatting"] }
+# AMQP consumer for results
+lapin = { version = "2" }
+futures-util = "0.3"
+# S3/Garage upload
+aws-config = { version = "1", default-features = false, features = ["behavior-version-latest", "rt-tokio"] }
+aws-sdk-s3 = { version = "1", default-features = false, features = ["rt-tokio", "rustls"] }
+# Workflow parsing helpers
+base64 = "0.22"
+uuid = { version = "1", features = ["v4"] }

docs/ai/2025-10-25-github-webhooks-to-jobrequest.md

@@ -0,0 +1,170 @@
+### GitHub Webhooks → JobRequest Mapping (Integration Layer)
+
+This document explains how the GitHub Integration service maps GitHub webhooks to internal `JobRequest` messages, publishes them to RabbitMQ, and reports status back via the GitHub Checks API.
+
+---
+
+### Overview
+- Service: `crates/github-integration`
+- Endpoint: `POST /webhooks/github` (configurable via `WEBHOOK_PATH`)
+- Auth: HMAC-SHA256 validation using `GITHUB_WEBHOOK_SECRET` (or `WEBHOOK_SECRET` fallback)
+- Events handled: `push`, `pull_request` (`opened`, `synchronize`, `reopened`)
+- Output: `JobRequest` (JSON) published to exchange `solstice.jobs` with routing key `jobrequest.v1`
+- Status reporting: GitHub Checks API (via GitHub App)
+
+---
+
+### Headers and Security
+- Event type header: `X-GitHub-Event`
+- Delivery ID header (optional, for logs): `X-GitHub-Delivery`
+- Signature header: `X-Hub-Signature-256`
+  - Value is `sha256=<hex>` where `<hex>` is `HMAC_SHA256(secret, raw_request_body)`.
+  - If `GITHUB_WEBHOOK_SECRET` is set, the service requires a valid signature and returns `401` on mismatch/missing header.
+  - If unset, the service accepts requests (dev mode) and logs a warning.
+
+Signature example (shell):
+```bash
+SECRET=your_shared_secret
+BODY='{"after":"deadbeef", "repository":{}}'
+SIG=$(printf %s "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -binary | xxd -p -c 256)
+# Send:
+curl -sS -X POST http://127.0.0.1:8082/webhooks/github \
+  -H "Content-Type: application/json" \
+  -H "X-GitHub-Event: push" \
+  -H "X-Hub-Signature-256: sha256=$SIG" \
+  --data "$BODY"
+```
+
+---
+
+### Payload Mapping → JobRequest
+
+We only deserialize the minimal fields required to construct a `JobRequest`. Unused fields are ignored.
+
+- Push event (`X-GitHub-Event: push`):
+  - `repo_url` <- `repository.clone_url` (fallback `repository.ssh_url`)
+  - `commit_sha` <- `after`
+  - Ignore branch deletions where `after` is all zeros
+  - `source` = `github`
+
+Minimal push payload shape:
+```json
+{
+  "after": "0123456789abcdef0123456789abcdef01234567",
+  "repository": {
+    "clone_url": "https://github.com/org/repo.git",
+    "ssh_url":   "git@github.com:org/repo.git"
+  }
+}
+```
+
+- Pull request event (`X-GitHub-Event: pull_request`):
+  - Only actions: `opened`, `synchronize`, `reopened` (others → 204 No Content)
+  - `repo_url` <- `pull_request.head.repo.clone_url` (fallback `ssh_url`)
+  - `commit_sha` <- `pull_request.head.sha`
+  - `source` = `github`
+
+Minimal PR payload shape:
+```json
+{
+  "action": "synchronize",
+  "pull_request": {
+    "head": {
+      "sha": "89abcdef0123456789abcdef0123456789abcd",
+      "repo": {
+        "clone_url": "https://github.com/org/repo.git",
+        "ssh_url":   "git@github.com:org/repo.git"
+      }
+    }
+  }
+}
+```
+
+`JobRequest` fields set now:
+- `schema_version = "jobrequest.v1"`
+- `request_id = Uuid::v4()`
+- `source = github`
+- `repo_url` as above
+- `commit_sha` as above
+- `workflow_path = null` (may be inferred later)
+- `workflow_job_id = null`
+- `runs_on = null` (future enhancement to infer)
+- `submitted_at = now(UTC)`
+
+---
+
+### Workflow Expansion (.solstice/workflow.kdl)
+If the GitHub App credentials are configured and the repo includes `.solstice/workflow.kdl`, the integration will:
+- Fetch the KDL file at the exact commit SHA (via the Contents API).
+- Parse job blocks and enqueue one `JobRequest` per job.
+- Set `workflow_path` to `.solstice/workflow.kdl` and `workflow_job_id` to the job ID.
+- Use `runs_on` from the workflow job if present, otherwise infer from labels or defaults.
+
+If the workflow is absent or cannot be parsed, a single job is enqueued.
+
+---
+
+### Checks API Status Updates
+- On webhook enqueue, the integration creates a Check Run for each job (status `queued`).
+- The Check Run `external_id` is the `request_id`, enabling later lookup without persistent storage.
+- When a `JobResult` arrives from MQ, the integration locates the matching Check Run and marks it `completed` with `success` or `failure`.
+- If no matching Check Run is found, it creates a completed Check Run so the commit still shows the result.
+
+---
+
+### AMQP Publication
+- Exchange: `solstice.jobs` (durable, direct)
+- Routing key: `jobrequest.v1`
+- Queue: `solstice.jobs.v1` (declared by both publisher and consumer)
+- DLX/DLQ: `solstice.dlx` / `solstice.jobs.v1.dlq`
+- Publisher confirms enabled; messages are persistent (`delivery_mode = 2`).
+
+Env/CLI (defaults):
+- `AMQP_URL=amqp://127.0.0.1:5672/%2f`
+- `AMQP_EXCHANGE=solstice.jobs`
+- `AMQP_ROUTING_KEY=jobrequest.v1`
+- `AMQP_QUEUE=solstice.jobs.v1`
+- `AMQP_DLX=solstice.dlx`
+- `AMQP_DLQ=solstice.jobs.v1.dlq`
+
+---
+
+### Configuration
+- HTTP address: `HTTP_ADDR` (default `0.0.0.0:8082`)
+- Webhook path: `WEBHOOK_PATH` (default `/webhooks/github`)
+- Shared secret: `GITHUB_WEBHOOK_SECRET` (required in prod)
+- GitHub API base: `GITHUB_API_BASE` (default `https://api.github.com`)
+- GitHub App ID: `GITHUB_APP_ID`
+- GitHub App key: `GITHUB_APP_KEY_PATH` or `GITHUB_APP_KEY`
+- Check name: `GITHUB_CHECK_NAME` (default `Solstice CI`)
+- Logs base URL: `LOGS_BASE_URL` (preferred) or `ORCH_HTTP_BASE` (deprecated)
+- S3 upload: `S3_ENDPOINT`, `S3_BUCKET`
+- Runs-on overrides: `RUNS_ON_DEFAULT`, `RUNS_ON_MAP` (owner/repo=label)
+
+Example run:
+```bash
+export GITHUB_WEBHOOK_SECRET=devsecret
+export GITHUB_APP_ID=123456
+export GITHUB_APP_KEY_PATH=/path/to/app.pem
+cargo run -p github-integration -- --http-addr 0.0.0.0:8082 --webhook-path /webhooks/github
+```
+
+---
+
+### GitHub App Setup (Minimum)
+1. Create a GitHub App with webhook URL `http://<your-host>:8082/webhooks/github` and set the webhook secret.
+2. Grant permissions:
+   - Checks: Read & write
+   - Contents: Read
+   - Metadata: Read
+3. Subscribe to events:
+   - Push
+   - Pull request
+4. Install the App on the target repositories.
+
+---
+
+### Notes & Next Steps
+- Consider adding idempotency keyed by `X-GitHub-Delivery` to avoid duplicate enqueues.
+- Consider supporting `check_suite` events for GitHub-native UI flows.
+- Add optional allowlists/branch filters to reduce noise.