wayray/docs/ai/plans/001-implementation-roadmap.md

Implementation Roadmap

Phase 0: Foundation (Weeks 1-2)

0.1 Project Structure

• Set up Cargo workspace with four crates: wayray-server, wayray-client, wayray-protocol, wayray-ctl
• Configure shared dependencies, feature flags, CI (Linux + illumos)
• Set up tracing/logging infrastructure with miette error handling
• Smithay with default-features = false + portable features only in core
• Platform-specific backends behind cfg(target_os) + feature flags

0.2 Minimal Compositor (Server)

• Implement a minimal Smithay compositor based on Smallvil patterns
• Support: wl_compositor, xdg_shell, wl_shm, wl_seat, wl_output
• Use Winit backend for development/testing (works on both Linux and illumos)
• Use shm_open with memfd_create fallback for shared memory portability
• Verify basic Wayland clients (foot terminal, weston-info) can connect and render

0.3 Framebuffer Capture

• Implement ExportMem-based framebuffer capture after each render pass
• Integrate OutputDamageTracker for efficient dirty region tracking
• Benchmark: capture latency, memory bandwidth

Phase 1: Network Protocol (Weeks 3-5)

1.1 Protocol Definition

• Define WayRay wire protocol in wayray-protocol crate
• Message types: FrameUpdate, InputEvent, SessionControl, AudioChunk, USBData
• Serialization: serde + bincode or postcard for low overhead
• Version negotiation and capability exchange

1.2 QUIC Transport Layer

• Implement QUIC server (quinn) in wayray-server
• Implement QUIC client (quinn) in wayray-client
• Stream mapping:
  • Stream 0: Control channel (session mgmt, capabilities)
  • Stream 1: Display channel (frame updates, damage regions)
  • Stream 2: Input channel (keyboard, mouse, touch)
  • Stream 3: Audio channel (Opus frames)
  • Stream 4+: USB device channels (one per device)
• Connection handling: TLS certificates, authentication

1.3 Frame Encoding Pipeline

• Implement frame differencing (XOR diff against previous frame)
• Implement region-based compression (zstd for lossless regions)
• Implement content-adaptive encoding:
  • Static regions: lossless zstd diff
  • Video regions: H.264 via ffmpeg-next or VAAPI
  • Text regions: lossless PNG-style encoding
• Damage rectangle merging and optimization

Phase 2: Client Viewer (Weeks 5-7)

2.1 Display Client

• Implement wayray-client as a standalone application
• Use winit + wgpu for cross-platform display
• Frame decoding pipeline: receive -> decompress -> decode -> upload to GPU -> display
• Double-buffered rendering with VSync

2.2 Input Capture & Forwarding

• Capture keyboard events (with proper keymap forwarding via xkb)
• Capture mouse events (motion, buttons, scroll)
• Capture touch events
• Serialize and send over QUIC input stream
• Handle keyboard grab/release for compositor key passthrough

2.3 Cursor Handling

• Server-side cursor rendering (simplest)
• Client-side cursor rendering with cursor image forwarding (lower latency)
• Cursor shape protocol support

Phase 2.5: Pluggable Window Management (Weeks 7-8)

2.5.1 WM Protocol Definition

• Define wayray_wm_manager_v1 Wayland protocol XML
• Define wayray_wm_window_v1, wayray_wm_seat_v1, wayray_wm_workspace_v1
• Implement two-phase transaction model (manage + render sequences)
• Generate Rust bindings via wayland-scanner

2.5.2 WM Protocol Server (in compositor)

• Implement WM global in wayray-server
• Window lifecycle events (new, closed, properties)
• Manage phase: receive policy decisions, send configures
• Render phase: apply positions/z-order atomically
• Keybinding registration and dispatch via seat interface

2.5.3 Built-in Floating WM

• Default WM active when no external WM is connected
• Basic floating behavior: centered new windows, focus-follows-click
• Keyboard shortcuts: Alt+F4 close, Alt+Tab cycle, Super+Arrow snap
• Yields to external WM on connect

2.5.4 Example Tiling WM

• Ship a reference tiling WM as a separate binary (wayray-wm-tiling)
• Demonstrates the protocol for third-party WM developers
• Basic BSP tiling with keyboard-driven focus

Phase 3: Session Management (Weeks 8-11)

3.1 Session Persistence

• Session state machine: Created -> Active -> Suspended -> Resumed -> Destroyed
• Session storage: in-memory with optional persistence (SeaORM + SQLite)
• Session timeout and cleanup policies

3.2 Authentication

• Token-based session identification (smart card ID, badge, or software token)
• PAM integration for user authentication
• Session-token binding in session store

3.3 Hot-Desking (Session Mobility)

• Token insertion triggers session lookup across server pool
• Session reconnection: rebind existing session to new client endpoint
• Session disconnect: unbind from client, keep session running
• Sub-second reconnection target (< 500ms)

3.4 Multi-Server Support

• Server discovery protocol (mDNS or custom)
• Session registry: which sessions live on which servers
• Cross-server session redirect
• Load balancing for new session placement

Phase 4: Audio & Peripherals (Weeks 10-13)

4.1 Audio Forwarding

• Trait-based audio backend: PipeWire (Linux), PulseAudio (illumos/Linux fallback)
• PipeWire integration on server side for audio capture (Linux)
• Opus encoding for low-latency audio streaming
• Audio stream over dedicated QUIC stream
• Playback synchronization with display frames
• Microphone input forwarding (bidirectional audio)

4.2 USB Device Forwarding

• Userspace USB forwarding protocol over QUIC (not kernel USB/IP, for illumos portability)
• Consider usbredir as wire format or design custom
• Device hotplug detection on client (udev on Linux, sysevent on illumos)
• Device attach/detach over QUIC channels
• Security: device class filtering (allow/deny policies)

4.3 Clipboard Synchronization

• Intercept wl_data_device on server
• Forward clipboard content types and data over control channel
• Handle large clipboard entries (images) efficiently
• Security: optional clipboard direction restrictions

Phase 5: Production Hardening (Weeks 14-17)

5.1 Platform-Specific Backends

• Linux: DRM/KMS backend for running wayray-server on hardware (optional, feature-gated)
• Linux: Multi-GPU support via MultiRenderer
• Linux: Session management via logind/libseat
• illumos: Custom input backend for /dev/kbd + /dev/mouse (local console use)
• illumos: Zones integration for session isolation
• CI: Test matrix for both Linux and illumos

5.2 XWayland Support

• Integrate Smithay's XWayland module for X11 application compatibility
• Handle X11 clipboard integration

5.3 Performance Optimization

• Adaptive bitrate based on network conditions
• Hardware encoding path (VAAPI, NVENC)
• Zero-copy frame capture via DMA-BUF export to encoder
• Client-side frame interpolation for network jitter compensation

5.4 Security

• TLS 1.3 for all QUIC connections (mandatory)
• Certificate-based mutual authentication
• Session encryption at rest
• Audit logging for session lifecycle events
• AppArmor/seccomp profiles for server process

Phase 6: Management & Operations (Weeks 16-20)

6.1 Administration

• CLI tool: wayray-ctl for server/session management
• REST API for external integration
• Session monitoring: active sessions, resource usage, network stats

6.2 Multi-Tenancy

• User session isolation (namespaces, cgroups)
• Resource quotas (CPU, memory, GPU per session)
• Fair scheduling across sessions

6.3 High Availability

• Server failover groups
• Session state replication for seamless failover
• Health checking and automatic server removal

Milestones

Milestone	Phase	Description
M0	0	Wayland clients render in local compositor
M1	1	Remote viewer sees compositor output over network
M2	2	Interactive remote session (display + input)
M2.5	2.5	External WM can control window layout via protocol
M3	3	Session persists across client disconnects, hot-desking works
M4	4	Audio and USB forwarding functional
M5	5	Production-ready with platform backends, XWayland, illumos CI
M6	6	Multi-server deployment with HA and management tools