CloudNebulaProject/wayray
1
0
Fork 0
mirror of https://github.com/CloudNebulaProject/wayray.git synced 2026-04-10 21:20:40 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
wayray/docs/ai/adr/006-audio-forwarding.md
Till Wegmueller 167c6c17c6
Add project documentation, architecture decisions, and usage book 
Comprehensive documentation for WayRay, a SunRay-like thin client
Wayland compositor targeting illumos and Linux:

- CLAUDE.md: project context and conventions
- docs/ai/plans: 6-phase implementation roadmap
- docs/ai/adr: 9 architecture decision records (Smithay, QUIC,
  frame encoding, session management, rendering, audio, project
  structure, illumos support, pluggable window management)
- docs/architecture: system architecture overview with diagrams
- docs/protocols: WayRay wire protocol specification
- book/: mdbook user guide (introduction, concepts, server/client
  guides, admin, development)
- RESEARCH.md: deep research on remote display protocols
2026-03-28 20:47:16 +01:00

1.6 KiB
Raw Blame History

ADR-006: Audio Forwarding

Status

Accepted

Context

WayRay must forward audio bidirectionally between server and client. Server-side applications produce audio output; client-side microphones capture audio input.

Decision

PipeWire for server-side audio capture/playback, Opus codec for transport, QUIC stream for delivery.

Design

Server Side

  • Create a virtual PipeWire sink per session (applications route audio here)
  • Create a virtual PipeWire source per session (for microphone input from client)
  • Capture audio frames from the virtual sink
  • Encode to Opus (48kHz, stereo, 20ms frames for balance of latency and efficiency)
  • Transmit over dedicated QUIC audio stream

Client Side

  • Receive Opus frames from QUIC audio stream
  • Decode with opus crate
  • Output to local audio device (via cpal or PipeWire client)
  • Capture local microphone input
  • Encode to Opus, send back over QUIC audio stream

Synchronization

  • Audio frames carry timestamps aligned with display frame sequence numbers
  • Client performs adaptive jitter buffering (target: 20-60ms buffer)
  • Lip-sync: audio presented when corresponding display frame is shown

Rationale

  • Opus is the gold standard for low-latency audio: 2.5ms minimum frame size, built-in FEC
  • PipeWire is the modern Linux audio stack, replacing PulseAudio
  • Dedicated QUIC stream prevents audio stuttering from display traffic
  • Per-session virtual devices provide proper isolation

Consequences

  • PipeWire dependency on server (standard on modern Linux)
  • Opus adds ~5ms encoding latency + network RTT + ~5ms decoding
  • Must handle PipeWire session lifecycle (create/destroy with WayRay sessions)