# RoboPark Production Memory Last verified: 2026-07-17 This records production facts future operators and coding agents must preserve. Never add enrollment tokens, mesh tokens, API keys, or device tokens. ## Verified Architecture - `livekit-1` is the Windows hub. - Mesh/dashboard: TCP `47913`. - RoboPark scheduler: TCP `8080`, proxied under `/robopark`. - LiveKit: TCP `7880`. - Production voice worker container: `caal-agent`. - BMW maps to device `dev_0422e545` and character `vixen` / `VIXEN (BMW)`. - Robots publish camera and microphone tracks to LiveKit. Park camera preview uses the RoboVision relay rather than LiveKit video. ## Production Robot Network Boundary - All onsite production robots, including RoboPanda and BMW/VIXEN, are LAN satellites of `livekit-1`. They are not expected to be directly reachable from the development laptop, even when the laptop can reach `livekit-1` through Tailscale. - RoboPanda is reachable only through the `livekit-1` LAN mesh. A missing Panda node in the development laptop's direct peer list does not by itself prove Panda is offline; inspect it from `livekit-1` and through the scheduler. - Route robot commands, spawned agents, supervisor shell requests, camera relays, media tests, and production diagnostics through `livekit-1`. Do not substitute direct SSH or a development-laptop LAN route. - The control path is: operator -> Tailscale/mesh -> `livekit-1:47913` -> scheduler proxy `/robopark` -> robot heartbeat/supervisor on the onsite LAN. The camera path is: browser -> authenticated `livekit-1` RoboVision mesh relay -> robot-local RoboVision service. - `livekit-1` currently advertises LAN IPv4 `192.168.0.9`; verify it on the hub before generating commands because DHCP may change it. Never use the development laptop's LAN address as a robot hub address. - When a robot reports a stale heartbeat, diagnose from `livekit-1` first: scheduler enrollment row, hub-side mesh membership, robot heartbeat, then robot-local runtime logs. Camera troubleshooting starts only after heartbeat and mesh connectivity are healthy. ## BMW Launch Device Lock - Camera: `/dev/video0` (`H65 USB CAMERA`). - Microphone: scheduler ID `2`, `Usb Audio Device: USB Audio (hw:3,0)`. - Speaker: scheduler ID `1`, `USB Audio Device: - (hw:2,0)`. - Linux live playback must use ALSA `plughw:2,0` through `aplay`. - Production mic gain defaults to `ROBOPARK_MIC_GAIN=4.0` and is bounded to `1.0..12.0` before PCM frames enter LiveKit. - RoboVision/sounddevice numeric IDs are not PyAudio numeric IDs. - Do not use HDMI or metadata-only `/dev/video*` devices as launch defaults. ## Verified Production Flow BMW passed these measured stages in one production session: 1. `scheduler_session` 2. `livekit_join` 3. `tts_subscribed` 4. `playback_started` 5. `camera_published` 6. `microphone_published` 7. `robot_online` 8. `camera_ready` 9. `microphone_ready` 10. `speaker_ready` `playback_started` means the first remote TTS frame reached the robot. Never report end-to-end audio delivery without this event. ## Root Causes Fixed ### Silent Greeting The old ROBOVOICE greeting published a second dedicated LiveKit audio track. The Pi subscribed to both tracks and attempted to open the same speaker twice. ROBOVOICE must call `session.say(...)` so greeting and conversation share one track. `robopark agent-up` now detects the old source block, patches the ROBOVOICE checkout, and rebuilds Compose once. ### Linux Speaker Playback The preview agent previously used PyAudio for the selected speaker. BMW's proven output is ALSA `aplay -D plughw:2,0`. Infinicode `2.8.86` uses that same endpoint for live TTS whenever a Linux output selection contains `hw:`. Motion cues must use that same ALSA endpoint and finish before the greeting starts. Running a PyAudio cue concurrently with ALSA TTS creates an exclusive device race on `hw:2,0`. `playback_started` is valid only after PCM has been written to the physical speaker, not merely received from LiveKit. ### Silent Conversation After Greeting Publishing a LiveKit microphone track does not prove that PCM capture started. On Linux the production capture path is `arecord -D plughw:3,0 -f S16_LE -r 48000 -c 1`; it must work even when PyAudio is unavailable. The Pi now waits for actual PCM frames before reporting `microphone_published`, reports a blocked stage when capture does not start within five seconds, and applies bounded mic gain before publication. ### Camera Relay With Multiple Viewers RoboVision must have exactly one OpenCV camera reader. Preview, dashboard, and motion detection consume a shared latest-frame stream. Starting one `generate_frames()` camera loop per MJPEG client causes concurrent `VideoCapture.read()` calls, which can block the second client and make the hub relay time out with zero bytes. Long-running camera operation also requires recovery at every layer: the Pi releases and reopens a V4L2 handle after an eight-second blocked read, the mesh relay returns an error after ten seconds without an upstream frame, and both drawer and fullscreen camera views reconnect every three seconds. A camera request must never remain in an unbounded `connecting` state. ### Duplicate Speaker Tests Only one robot process may claim a scheduler `speaker_test`. The scheduler uses an atomic lease, and supervisor-status queries exclude speaker-test rows. ### BMW Alias Configuration Scheduler routes canonicalize alias `bmw` to `dev_0422e545` before reading character, voice-stack, and greeting configuration. ## Operational Rules - Run one unified robot runtime per robot. Do not manually start preview, vision, or mesh beside the systemd runtime. - Do not run audio diagnostics during a live conversation. - A direct ALSA test does not validate production if production uses another API, index space, sample format, or opens the device twice. - After a hub reboot, verify `47913`, `8080`, and `7880` before robot media. - `tts_subscribed` alone is insufficient; require `playback_started`. - Greeting `playback_started` alone does not prove conversation. A complete turn also requires a user/STT event, an LLM response, and response playback. - Container-local changes are temporary. Persist voice changes in the ROBOVOICE checkout/image or use `robopark agent-up --path `. ## Minimal Verification Run from the hub: ```powershell try { Invoke-RestMethod -Method Post http://127.0.0.1:8080/api/robots/bmw/pipeline-test/stop | Out-Null } catch {} Start-Sleep 3 Invoke-RestMethod -Method Post http://127.0.0.1:8080/api/robots/bmw/pipeline-test/start | Out-Null Start-Sleep 30 $s = Invoke-RestMethod http://127.0.0.1:8080/api/robots/bmw/pipeline-status $id = $s.latest_session.id $s.events | Where-Object session_id -eq $id | Select-Object timestamp,stage,status,message ``` Required result: `playback_started ok`. ## Fleet Media Reliability Contract (2.8.93+) - Linux setup installs persistent udev permissions for `video4linux` and `sound`, plus `/dev/robopark-camera` for the primary USB capture interface. - The systemd runtime receives available media groups and uses control-group shutdown so descendants cannot survive an update or restart. - `/run/lock/robopark-speaker.lock` and `/run/lock/robopark-microphone.lock` serialize production media and tests. - The vision launcher owns both camera and audio children. If either exits, the runtime restarts one coherent pair instead of leaving an orphan owner. - RoboVision opens its shared camera worker at boot. The runtime polls camera status and recycles media after three consecutive unhealthy checks. - Park media operations display camera/audio permission, camera worker state, and latest frame age from `device_inventory.media_health`. - Camera visibility is independent of conversation sessions. Fleet cards and the robot drawer use the RoboVision mesh MJPEG relay continuously and keep stream elements alive across telemetry rerenders. - The dashboard unattended readiness test requires camera health, an acoustic speaker-to-microphone round-trip, scheduler allocation, LiveKit join, camera and microphone publication, TTS subscription, and first robot playback. - An unattended readiness pass does not claim an STT/LLM turn occurred. That semantic turn still requires injected speech or a person near the robot. ## Camera Ownership Contract (2.8.95+) - RoboVision is the only process permitted to open the physical V4L2 camera. - Dashboard, motion sampling, and conversation video consume RoboVision's shared MJPEG stream; they never open `/dev/video0` independently. - Local camera motion remains enabled by default. Production mode controls whether motion creates a conversation, not whether the camera worker runs. - `auto`, `default`, and `first` resolve to `/dev/robopark-camera` when present, otherwise `/dev/video0`; those literal strings are never passed to OpenCV. ## Audio Ownership Contract (2.8.96+) - Fleet microphone selection is resolved by stable USB product label on every inventory refresh. The current stack maps microphone to `hw:3,0` and speaker to `hw:2,0`. - A LiveKit microphone track is healthy only after the robot captures and publishes real PCM. Publishing an empty track is not readiness. - Linux capture probes `plughw` at 48 kHz, falls back to 44.1 kHz with resampling, and releases the microphone lock after every failed attempt. - Pipeline failures include the selected device and exact ALSA startup error. A flat browser meter means missing PCM, not a quiet room. - Remote `speaker-test` requests support `mode=mic_groundtruth`, which captures through the same `arecord -D plughw:X,Y -f S16_LE -r 48000 -c 1` command proven onsite and returns bytes, RMS, peak, and nonzero sample counts. - Python 3.13 removed `audioop`. Version 2.8.97 replaces microphone gain, peak measurement, and fallback resampling with internal PCM16 helpers; production capture must not import `audioop`. ## Motor and Relay Contract (2.8.99+) - Each robot persists its own motor registry, timed sequences, local motor API URL, and optional greeting sequence in the scheduler database. - The Park robot drawer is the control surface for registry edits, BCM relay tests, sequence authoring, manual runs, and greeting assignment. - The full GPIO diagnostic requires operator confirmation and pulses every registered relay sequentially. It never scans or energizes unregistered pins. - Greeting motion queues only after the robot reports `playback_started`, so movement aligns with audible TTS instead of session allocation. - Voice agents may invoke only saved sequences for their active LiveKit room through the authenticated room-scoped scheduler endpoint. They cannot submit raw GPIO numbers or arbitrary shell commands. - The robot supervisor accepts only `127.0.0.1` or `localhost` motor APIs, synchronizes the saved registry, executes nodes serially, and verifies that every pulse returns to idle before advancing. - The motor API uses `lgpio`, accepts only registered BCM pins 2 through 27, rejects duplicate assignments, limits pulses to 10 seconds, serializes all actuation, and de-energizes every registered relay on errors and shutdown. - A missing or busy GPIO driver is a failed test, never a simulated success. ## Robot Speaker Echo Contract (2.8.100+) - The launch USB speaker/microphone stack has no hardware acoustic echo cancellation, so production uses robot-side half-duplex echo suppression. - While outbound TTS contains audible PCM, the Pi publishes timing-correct silence instead of amplified microphone PCM. The microphone track remains connected and resumes after a 700 ms acoustic decay tail. - Silent frames on the persistent TTS track do not hold the gate open. Output peak detection releases the visitor microphone between responses. - `ROBOPARK_HALF_DUPLEX`, `ROBOPARK_ECHO_TAIL_MS`, and `ROBOPARK_ECHO_GATE_PEAK` are explicit runtime controls. Disable half-duplex only after deploying real acoustic echo cancellation. ## Instant Greeting Contract (2.8.101+) - Motion reporting is asynchronous and never delays session allocation. - RoboVision's shared camera stream is not stopped or reopened on a trigger. - The old fixed 500 ms trigger delay is removed. - The local motion beep is disabled by default because it occupies the same exclusive ALSA speaker as TTS. `ROBOPARK_MOTION_CUE=true` is diagnostics only. - The voice worker uses the persistent AgentSession audio track and starts the greeting after robot join; it does not wait for camera subscription. - Network, scheduler, LiveKit, and TTS synthesis still impose real latency, but RoboPark adds no intentional serial delay before greeting generation. ## Adaptive Interruption Contract (2.8.102+) - The preset greeting remains non-interruptible so ambient startup noise cannot cancel it before the first audible frame. - Normal responses retain LiveKit AgentSession interruption handling. - Robot half-duplex suppression learns the microphone's active speaker-echo floor. Three consecutive near-field peaks above both the absolute threshold and the adaptive echo ratio reopen microphone PCM for the VAD. - The barge-in window remains open for 1.4 seconds, allowing LiveKit to cancel TTS; ordinary speaker echo remains suppressed with the existing decay tail. - Vision responses inherit the selected voice-stack output language. With an automatic profile, the original user transcript language is authoritative, including direct terminal vision-tool responses. ## Direct Conversation Motion Contract (2.8.124+) - `robopark conversation up --auto-start` owns the complete local trigger path: its webhook, camera-only RoboVision process, motion arming, and direct ElevenLabs session lifecycle run under one systemd service. - The same command disables and removes the matching legacy unified robot runtime before starting, so stale services cannot retain camera or ALSA ownership after migration. - The managed RoboVision child never starts the legacy audio server. The direct conversation runtime remains the only owner of the USB microphone and speaker, preventing `Device or resource busy` regressions. - `auto` camera selection prefers `/dev/robopark-camera` and falls back to `/dev/video0`. Use `--camera-device` only for an intentional override. - The runtime reapplies both the motion webhook and `active=true` every five seconds. If the camera process exits, it is restarted and rerouted without an onsite operator. - `GET http://127.0.0.1:5057/` reports `vision_routed`, `vision_process_running`, `vision_restarts`, session state, and trigger counters. A production-ready robot has `state=armed` and `vision_routed=true`. - `--no-manage-vision` is reserved for installations with an externally supervised RoboVision process. - During agent provisioning, a blank or whitespace-only ElevenLabs first message is replaced with a deterministic greeting derived from the agent name. A motion session must produce speech without waiting for visitor audio. - Direct ElevenLabs sessions pause microphone forwarding while speaker PCM is queued or playing and for a 700 ms acoustic decay tail. This prevents the robot from transcribing its own responses on hardware without acoustic echo cancellation; session disposal logs the number of suppressed mic chunks. - Direct production defaults allow 120 seconds of conversational silence, cap each cloud session at nine minutes, and use a two-second recycle cooldown. Motion received while active or cooling down is retained for 15 seconds and promoted into a continuation session after clean disposal, keeping the park responsive for hours without leaking one unbounded cloud connection. ## Deployable Character Binding Contract (2.8.140+) - `packages/robopark/scheduler/production_config.json` is the sanitized source of truth for canonical character IDs and provider bindings. - Scheduler startup reconciles names, default engines, ElevenLabs agent and branch IDs, RoboVoice profile IDs, prompt revisions, and map metadata. A pulled deployment no longer remains unbound because an old local row exists. - API keys, enrollment tokens, robot hardware assignments, audio identities, system prompts, and motor registries are not stored in the manifest. - Operator-managed system prompts and motor registries survive reconciliation. Provider prompt content is pulled explicitly through the ElevenLabs sync UI. - Legacy `panda` references migrate deterministically to canonical `ronin`. Runtime character resolution remains ID-based; no fuzzy reassignment occurs. - `GET /api/deployment-config` reports the loaded revision and whether every database binding matches the packaged manifest without exposing credentials.