solstice-ci/docs/ai/2025-10-25-github-webhooks-to-jobrequest.md

### GitHub Webhooks → JobRequest Mapping (Integration Layer)

This document explains how the GitHub Integration service maps GitHub webhooks to internal `JobRequest` messages, publishes them to RabbitMQ, and reports status back via the GitHub Checks API.

---

### Overview
- Service: `crates/github-integration`
- Endpoint: `POST /webhooks/github` (configurable via `WEBHOOK_PATH`)
- Auth: HMAC-SHA256 validation using `GITHUB_WEBHOOK_SECRET` (or `WEBHOOK_SECRET` fallback)
- Events handled: `push`, `pull_request` (`opened`, `synchronize`, `reopened`)
- Output: `JobRequest` (JSON) published to exchange `solstice.jobs` with routing key `jobrequest.v1`
- Status reporting: GitHub Checks API (via GitHub App)

---

### Headers and Security
- Event type header: `X-GitHub-Event`
- Delivery ID header (optional, for logs): `X-GitHub-Delivery`
- Signature header: `X-Hub-Signature-256`
  - Value is `sha256=<hex>` where `<hex>` is `HMAC_SHA256(secret, raw_request_body)`.
  - If `GITHUB_WEBHOOK_SECRET` is set, the service requires a valid signature and returns `401` on mismatch/missing header.
  - If unset, the service accepts requests (dev mode) and logs a warning.

Signature example (shell):
```bash
SECRET=your_shared_secret
BODY='{"after":"deadbeef", "repository":{}}'
SIG=$(printf %s "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -binary | xxd -p -c 256)
# Send:
curl -sS -X POST http://127.0.0.1:8082/webhooks/github \
  -H "Content-Type: application/json" \
  -H "X-GitHub-Event: push" \
  -H "X-Hub-Signature-256: sha256=$SIG" \
  --data "$BODY"
```

---

### Payload Mapping → JobRequest

We only deserialize the minimal fields required to construct a `JobRequest`. Unused fields are ignored.

- Push event (`X-GitHub-Event: push`):
  - `repo_url` <- `repository.clone_url` (fallback `repository.ssh_url`)
  - `commit_sha` <- `after`
  - Ignore branch deletions where `after` is all zeros
  - `source` = `github`

Minimal push payload shape:
```json
{
  "after": "0123456789abcdef0123456789abcdef01234567",
  "repository": {
    "clone_url": "https://github.com/org/repo.git",
    "ssh_url":   "git@github.com:org/repo.git"
  }
}
```

- Pull request event (`X-GitHub-Event: pull_request`):
  - Only actions: `opened`, `synchronize`, `reopened` (others → 204 No Content)
  - `repo_url` <- `pull_request.head.repo.clone_url` (fallback `ssh_url`)
  - `commit_sha` <- `pull_request.head.sha`
  - `source` = `github`

Minimal PR payload shape:
```json
{
  "action": "synchronize",
  "pull_request": {
    "head": {
      "sha": "89abcdef0123456789abcdef0123456789abcd",
      "repo": {
        "clone_url": "https://github.com/org/repo.git",
        "ssh_url":   "git@github.com:org/repo.git"
      }
    }
  }
}
```

`JobRequest` fields set now:
- `schema_version = "jobrequest.v1"`
- `request_id = Uuid::v4()`
- `source = github`
- `repo_url` as above
- `commit_sha` as above
- `workflow_path = null` (may be inferred later)
- `workflow_job_id = null`
- `runs_on = null` (future enhancement to infer)
- `submitted_at = now(UTC)`

---

### Workflow Expansion (.solstice/workflow.kdl)
If the GitHub App credentials are configured and the repo includes `.solstice/workflow.kdl`, the integration will:
- Fetch the KDL file at the exact commit SHA (via the Contents API).
- Parse job blocks and enqueue one `JobRequest` per job.
- Set `workflow_path` to `.solstice/workflow.kdl` and `workflow_job_id` to the job ID.
- Use `runs_on` from the workflow job if present, otherwise infer from labels or defaults.

If the workflow is absent or cannot be parsed, a single job is enqueued.

---

### Checks API Status Updates
- On webhook enqueue, the integration creates a Check Run for each job (status `queued`).
- The Check Run `external_id` is the `request_id`, enabling later lookup without persistent storage.
- When a `JobResult` arrives from MQ, the integration locates the matching Check Run and marks it `completed` with `success` or `failure`.
- If no matching Check Run is found, it creates a completed Check Run so the commit still shows the result.

---

### AMQP Publication
- Exchange: `solstice.jobs` (durable, direct)
- Routing key: `jobrequest.v1`
- Queue: `solstice.jobs.v1` (declared by both publisher and consumer)
- DLX/DLQ: `solstice.dlx` / `solstice.jobs.v1.dlq`
- Publisher confirms enabled; messages are persistent (`delivery_mode = 2`).

Env/CLI (defaults):
- `AMQP_URL=amqp://127.0.0.1:5672/%2f`
- `AMQP_EXCHANGE=solstice.jobs`
- `AMQP_ROUTING_KEY=jobrequest.v1`
- `AMQP_QUEUE=solstice.jobs.v1`
- `AMQP_DLX=solstice.dlx`
- `AMQP_DLQ=solstice.jobs.v1.dlq`

---

### Configuration
- HTTP address: `HTTP_ADDR` (default `0.0.0.0:8082`)
- Webhook path: `WEBHOOK_PATH` (default `/webhooks/github`)
- Shared secret: `GITHUB_WEBHOOK_SECRET` (required in prod)
- GitHub API base: `GITHUB_API_BASE` (default `https://api.github.com`)
- GitHub App ID: `GITHUB_APP_ID`
- GitHub App key: `GITHUB_APP_KEY_PATH` or `GITHUB_APP_KEY`
- Check name: `GITHUB_CHECK_NAME` (default `Solstice CI`)
- Logs base URL: `LOGS_BASE_URL` (preferred) or `ORCH_HTTP_BASE` (deprecated)
- S3 upload: `S3_ENDPOINT`, `S3_BUCKET`
- Runs-on overrides: `RUNS_ON_DEFAULT`, `RUNS_ON_MAP` (owner/repo=label)

Example run:
```bash
export GITHUB_WEBHOOK_SECRET=devsecret
export GITHUB_APP_ID=123456
export GITHUB_APP_KEY_PATH=/path/to/app.pem
cargo run -p github-integration -- --http-addr 0.0.0.0:8082 --webhook-path /webhooks/github
```

---

### GitHub App Setup (Minimum)
1. Create a GitHub App with webhook URL `http://<your-host>:8082/webhooks/github` and set the webhook secret.
2. Grant permissions:
   - Checks: Read & write
   - Contents: Read
   - Metadata: Read
3. Subscribe to events:
   - Push
   - Pull request
4. Install the App on the target repositories.

---

### Notes & Next Steps
- Consider adding idempotency keyed by `X-GitHub-Delivery` to avoid duplicate enqueues.
- Consider supporting `check_suite` events for GitHub-native UI flows.
- Add optional allowlists/branch filters to reduce noise.