### GitHub Webhooks → JobRequest Mapping (Integration Layer) This document explains how the GitHub Integration service maps GitHub webhooks to internal `JobRequest` messages, publishes them to RabbitMQ, and reports status back via the GitHub Checks API. --- ### Overview - Service: `crates/github-integration` - Endpoint: `POST /webhooks/github` (configurable via `WEBHOOK_PATH`) - Auth: HMAC-SHA256 validation using `GITHUB_WEBHOOK_SECRET` (or `WEBHOOK_SECRET` fallback) - Events handled: `push`, `pull_request` (`opened`, `synchronize`, `reopened`) - Output: `JobRequest` (JSON) published to exchange `solstice.jobs` with routing key `jobrequest.v1` - Status reporting: GitHub Checks API (via GitHub App) --- ### Headers and Security - Event type header: `X-GitHub-Event` - Delivery ID header (optional, for logs): `X-GitHub-Delivery` - Signature header: `X-Hub-Signature-256` - Value is `sha256=` where `` is `HMAC_SHA256(secret, raw_request_body)`. - If `GITHUB_WEBHOOK_SECRET` is set, the service requires a valid signature and returns `401` on mismatch/missing header. - If unset, the service accepts requests (dev mode) and logs a warning. Signature example (shell): ```bash SECRET=your_shared_secret BODY='{"after":"deadbeef", "repository":{}}' SIG=$(printf %s "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -binary | xxd -p -c 256) # Send: curl -sS -X POST http://127.0.0.1:8082/webhooks/github \ -H "Content-Type: application/json" \ -H "X-GitHub-Event: push" \ -H "X-Hub-Signature-256: sha256=$SIG" \ --data "$BODY" ``` --- ### Payload Mapping → JobRequest We only deserialize the minimal fields required to construct a `JobRequest`. Unused fields are ignored. - Push event (`X-GitHub-Event: push`): - `repo_url` <- `repository.clone_url` (fallback `repository.ssh_url`) - `commit_sha` <- `after` - Ignore branch deletions where `after` is all zeros - `source` = `github` Minimal push payload shape: ```json { "after": "0123456789abcdef0123456789abcdef01234567", "repository": { "clone_url": "https://github.com/org/repo.git", "ssh_url": "git@github.com:org/repo.git" } } ``` - Pull request event (`X-GitHub-Event: pull_request`): - Only actions: `opened`, `synchronize`, `reopened` (others → 204 No Content) - `repo_url` <- `pull_request.head.repo.clone_url` (fallback `ssh_url`) - `commit_sha` <- `pull_request.head.sha` - `source` = `github` Minimal PR payload shape: ```json { "action": "synchronize", "pull_request": { "head": { "sha": "89abcdef0123456789abcdef0123456789abcd", "repo": { "clone_url": "https://github.com/org/repo.git", "ssh_url": "git@github.com:org/repo.git" } } } } ``` `JobRequest` fields set now: - `schema_version = "jobrequest.v1"` - `request_id = Uuid::v4()` - `source = github` - `repo_url` as above - `commit_sha` as above - `workflow_path = null` (may be inferred later) - `workflow_job_id = null` - `runs_on = null` (future enhancement to infer) - `submitted_at = now(UTC)` --- ### Workflow Expansion (.solstice/workflow.kdl) If the GitHub App credentials are configured and the repo includes `.solstice/workflow.kdl`, the integration will: - Fetch the KDL file at the exact commit SHA (via the Contents API). - Parse job blocks and enqueue one `JobRequest` per job. - Set `workflow_path` to `.solstice/workflow.kdl` and `workflow_job_id` to the job ID. - Use `runs_on` from the workflow job if present, otherwise infer from labels or defaults. If the workflow is absent or cannot be parsed, a single job is enqueued. --- ### Checks API Status Updates - On webhook enqueue, the integration creates a Check Run for each job (status `queued`). - The Check Run `external_id` is the `request_id`, enabling later lookup without persistent storage. - When a `JobResult` arrives from MQ, the integration locates the matching Check Run and marks it `completed` with `success` or `failure`. - If no matching Check Run is found, it creates a completed Check Run so the commit still shows the result. --- ### AMQP Publication - Exchange: `solstice.jobs` (durable, direct) - Routing key: `jobrequest.v1` - Queue: `solstice.jobs.v1` (declared by both publisher and consumer) - DLX/DLQ: `solstice.dlx` / `solstice.jobs.v1.dlq` - Publisher confirms enabled; messages are persistent (`delivery_mode = 2`). Env/CLI (defaults): - `AMQP_URL=amqp://127.0.0.1:5672/%2f` - `AMQP_EXCHANGE=solstice.jobs` - `AMQP_ROUTING_KEY=jobrequest.v1` - `AMQP_QUEUE=solstice.jobs.v1` - `AMQP_DLX=solstice.dlx` - `AMQP_DLQ=solstice.jobs.v1.dlq` --- ### Configuration - HTTP address: `HTTP_ADDR` (default `0.0.0.0:8082`) - Webhook path: `WEBHOOK_PATH` (default `/webhooks/github`) - Shared secret: `GITHUB_WEBHOOK_SECRET` (required in prod) - GitHub API base: `GITHUB_API_BASE` (default `https://api.github.com`) - GitHub App ID: `GITHUB_APP_ID` - GitHub App key: `GITHUB_APP_KEY_PATH` or `GITHUB_APP_KEY` - Check name: `GITHUB_CHECK_NAME` (default `Solstice CI`) - Logs base URL: `LOGS_BASE_URL` (preferred) or `ORCH_HTTP_BASE` (deprecated) - S3 upload: `S3_ENDPOINT`, `S3_BUCKET` - Runs-on overrides: `RUNS_ON_DEFAULT`, `RUNS_ON_MAP` (owner/repo=label) Example run: ```bash export GITHUB_WEBHOOK_SECRET=devsecret export GITHUB_APP_ID=123456 export GITHUB_APP_KEY_PATH=/path/to/app.pem cargo run -p github-integration -- --http-addr 0.0.0.0:8082 --webhook-path /webhooks/github ``` --- ### GitHub App Setup (Minimum) 1. Create a GitHub App with webhook URL `http://:8082/webhooks/github` and set the webhook secret. 2. Grant permissions: - Checks: Read & write - Contents: Read - Metadata: Read 3. Subscribe to events: - Push - Pull request 4. Install the App on the target repositories. --- ### Notes & Next Steps - Consider adding idempotency keyed by `X-GitHub-Delivery` to avoid duplicate enqueues. - Consider supporting `check_suite` events for GitHub-native UI flows. - Add optional allowlists/branch filters to reduce noise.