papercrab

A Lobster-themed Rust document management system. Pluggable read-only ingest stores, an S3/Object-Lock-backed write store, Postgres + Apache AGE for metadata and graph, Barycenter for identity and authorization, and Firecrawl (plus a local OCR fallback) for content extraction.

Status: walking skeleton. Many subsystems are wired at the trait level with stub implementations. See docs/architecture.md for the shape and docs/roadmap.md for what's deferred.

Layout

crates/
  papercrab-core       domain types
  papercrab-db         SeaORM entities + repositories (+ AGE helpers, FTS)
  papercrab-db/migration  SeaORM migrations
  papercrab-storage    StorageBackend trait + OpenDAL-backed LocalFs/WebDAV/S3
  papercrab-auth       OIDC RP (authn) + Barycenter /v1/check client (authz) + KDL policy
  papercrab-extract    Extractor trait + Firecrawl client + local fallback
  papercrab-formats    ODF / OOXML metadata read/patch (skeleton)
  papercrab-ai         rig agent + OpenRouter provider
  papercrab-search     Postgres-backed search trait (skeleton)
  papercrab-jobs       apalis job types
  papercrab-api        axum HTTP API
  papercrab-cli        clap admin CLI

deploy/
  helm/papercrab/      Helm chart

Building

cargo build --workspace
cargo run -p papercrab-cli -- --help

Running

Papercrab needs: a Postgres 16 database, a Barycenter instance (OIDC + optional /v1/check), an S3-compatible blob store for the read/write backend, and (optionally) API keys for OpenRouter and Firecrawl.

Point PAPERCRAB_CONFIG at a filled-out copy of config/papercrab.example.toml:

cp config/papercrab.example.toml config/papercrab.toml
# edit …
cargo run -p papercrab-cli -- migrate
cargo run -p papercrab-cli -- serve

Consume documents

Enable [consume] with an explicit writable target_backend_id to expose the authenticated intake app at /consume. It accepts browser drag/drop and file selection, can be installed as a PWA share target on supporting platforms, expands ZIP archives with bounded path/size checks, places every resulting document in /Consume, and queues the normal extraction pipeline. Uploads are owner-scoped and identical content converges on the existing document for that owner.

For a cross-platform folder fallback, point a scheduled scan at a bounded read-only WebDAV or local prefix:

papercrab backends scan <source-backend-uuid> \
  --prefix Documents/Consume/ \
  --consume \
  --archive-target-backend <writable-backend-uuid>

The Helm chart can schedule that command with consumeScanner. A single-user deployment may infer its sole signed-in human owner; otherwise set consumeScanner.ownerSub explicitly. The scan fails closed when ownership is ambiguous. ZIP members are copied to the writable backend under content-addressed keys, while ordinary documents remain on the read-only source backend. Replays skip unchanged archives and do not create duplicate records.

Container + Helm

The Dockerfile produces a single binary image that runs papercrab serve by default. deploy/helm/papercrab/ is a generic Helm chart: Deployment + Service + ConfigMap + PVC + a pre-install/upgrade migration Job, plus an optional Consume scanner CronJob. All sensitive values are secretKeyRefs — the chart expects five Secrets in the release namespace (papercrab-postgres-url, papercrab-oidc, papercrab-openrouter, papercrab-firecrawl, papercrab-s3).

Override the deploy-specific bits (image tag, domain, storage class, backend list, Barycenter URLs, resource requests) from your Flux / Argo / helm install values file.

CI

.forgejo/workflows/build.yml runs on Solstice CI:

Trigger	Action
pull_request → main	cargo fmt --check, cargo clippy -D warnings, cargo test --workspace
push → main	the above, then build and push the container image as <sha> and latest
push → tag v*	the above, then helm package and push the chart to the Forgejo OCI registry, appVersion synced to the tag

The runner needs a REGISTRY_TOKEN secret with write:package scope on the forge — Solstice CI auto-configures Docker and Helm against the forge's registry when that's present. Pulls land at code.aopc.cloud/toasterson/papercrab (image) and oci://code.aopc.cloud/toasterson/charts/papercrab (chart).

Extension notes

• pg_trgm ships with Postgres — always available.
• pgvector ships with recent CloudNativePG operand images and standard pgvector/pgvector images.
• Apache AGE is not in stock Postgres. Migration 0002_age_graph checks pg_available_extensions and skips cleanly when AGE is missing; graph-only features stay dormant. To turn them on, use a Postgres image with AGE compiled in and shared_preload_libraries = 'age'.

Auth

Papercrab is a Barycenter relying party. Register it once (RFC 7591 dynamic client registration):

papercrab register-oidc \
  --issuer https://<your-barycenter-host> \
  --redirect-uri https://<your-papercrab-host>/auth/callback

Paste the returned client_secret into the papercrab-oidc Secret.

Authorization defaults to an in-process MockAuthz (useful for CI, dev, and deployments where Barycenter's /v1/check service isn't yet reachable). Set authz.backend = "barycenter" and point check_endpoint / expand_endpoint at your Barycenter authz listener to flip to real enforcement.

License

MPL-2.0. See LICENSE.