# JUJU Multimodal — version 0.8.2

JUJU keeps the 74-tool full catalog introduced in the 0.7 series and in 0.8.2
adds one cross-platform owner setup path for the native dependencies behind media,
browser, speech and desktop capabilities. Install `npm install -g juju-mcp@0.8.2`,
run `juju setup`, then restart the intended server/client connection. Building a
checkout or preparing a different machine does not update an already-running or
separate global installation. The client/model interprets returned evidence; JUJU
is not a new vision model, autonomous player, or permanently listening assistant.

## Capabilities

| Capability | Tools | Implemented behavior |
|---|---|---|
| Screen context and image inspection | `screen_observe`, `image_view` | Short capture bursts from a permitted window, crop/zoom, timestamps, coordinate mapping, and pixel-change comparison |
| Video and audio evidence | `media_open`, `media_extract`, `media_read`, `media_job`, `media_close` | Immutable local-file snapshot, actual stream metadata, selected JPEG frames and WAV audio, owned processing jobs and cancellation |
| Speech and clip search | `media_transcribe`, `media_subtitles`, `media_search`, `speech_synthesize` | Local Whisper transcription, SRT/VTT import, literal transcript search with timestamps, and local speech synthesis returned as audio |
| Browser/app operation | `browser_session`, `browser_observe`, `browser_action` | Isolated Chromium, actual screenshots, bounded DOM controls, form actions, console/network evidence and HTML media play/pause/seek/mute |
| Step-based games | `game_session`, `game_step` | Observe, perform one bounded input, then observe again; browser key combinations with guaranteed key release; native discrete input through existing desktop controls |
| Demonstration workflows | `workflow_save`, `workflow_search`, `workflow_run` | Store/search client-authored steps, bind an exact revision, check before/after text and execute one step at a time |
| Readiness | `multimodal_status` | Report installed components and current capabilities without capturing a screen or opening a browser |

The original tools remain available in their original order. In particular,
`read_image`, `desktop_capture`, `desktop_accessibility` and `desktop_action` are
not replaced. The coding-assistance tools described in [ASSISTANCE.md](ASSISTANCE.md)
remain unchanged except for additive catalog discovery.

## Existing permissions remain authoritative

No new tool enables desktop access, changes OAuth/ACLs/trust, turns off a sandbox,
approves itself, or edits owner security configuration. Native captures and input
still use the existing desktop permission and snapshot machinery. Processing jobs
use the existing job manager, filtered environment, exec approval and configured
OS sandbox. Media workers request no network; they do not downgrade the sandbox if
an operation fails. Browser launch applies the existing sandbox configuration too.

`multimodal_status` and `workflow_search` are read-scoped; `workflow_save` is
write-scoped. Other new tools require `juju:exec`, including media handles and
isolated browser reads, because their evidence may originate from execution or an
application outside the workspace. The old `read_image` remains usable with read
scope for ordinary workspace image files. Inspect/edit trust modes still require
the applicable local approval before execution.

Every tool takes the current `workspaceId` and `workspaceEpoch` returned by
`project_overview`. Examples below omit only these two required context fields.
Handles bind to their workspace, runtime epoch and originating client/principal;
knowing an ID does not grant another client access. Live grant/ACL revocation is
checked again when returning evidence. Browser requests also recheck the session's
live authorization; expired/revoked browser sessions are closed by housekeeping.

## Installation and readiness

Images use pinned `sharp`; isolated browser operation uses the Playwright version
shipped by JUJU and its matching Chromium revision. Video/audio operations use
FFmpeg/ffprobe. Transcription uses whisper.cpp plus the pinned multilingual tiny
GGML model. Speech synthesis is local and returns an audio asset without playing it:
macOS uses an available `say`/eSpeak path, Windows prefers SAPI with eSpeak fallback,
and Linux uses eSpeak when prepared.

Use the same owner setup command on Windows, macOS and Linux:

```bash
npm install -g juju-mcp@0.8.2
juju setup
juju setup --check
# or only the media/browser/speech pieces:
juju setup --components ffmpeg,whisper,model,chromium,speech
```

Setup installs only missing components using reviewed recipes/pins and then runs
bounded capability probes: media decode/extract, Whisper inference, Chromium
launch/render/click/screenshot with its browser sandbox, and real local speech
rendering. The GGML model is stored under JUJU private state and verified by exact
size/SHA-256 before use. Managed tool paths are process-local to JUJU rather than
written into the machine PATH. `juju-media-setup` remains as a legacy 0.7 helper,
but `juju setup` is the 0.8.2 cross-platform entry point.

Call `multimodal_status` before a task. A missing/failed binary, model or backend is
an explicit capability error, never a fabricated transcript or success response.
Preparing software does not grant Desktop app access or enable web fetch.

## 1. Screen context and image inspection

Use the existing `desktop_windows` to choose a permitted window, then request
`screen_observe` for its `windowId`. The service can return a short sequence of
captures, not an always-on feed. The current schema bounds burst count and interval.
Each frame has an observation ID, timestamp, original desktop snapshot ID, image
asset, image geometry and coordinate mapping. Provide the previous observation ID
when comparing compatible captures. A comparison reports pixel change, not a
model-generated list of objects or an assertion that a task succeeded.

`image_view` can crop/resize a workspace image or an owned image asset. It returns
real image content, not just a filename. Coordinates in a cropped/resized view must
be mapped back using its returned mapping before using the original desktop
snapshot. Game/native operations use the full observation's coordinate contract.
Changing the window, geometry, local grant, actor or epoch invalidates relevant
observations. Screenshots expire quickly; capture again rather than recycling an
old observation for another input action.

Captured images are kept in a bounded RAM cache, not written to disk or the audit
log. Sharp's decoded-pixel bound and output byte bounds still apply. OCR is not
required for these tools; the model can inspect the actual images, while existing
Accessibility tools provide application-exposed text when available.

## 2. Video/movie evidence and audio

Start with a local file under the workspace:

```json
{ "path": "clips/tutorial.mp4", "idempotencyKey": "open-tutorial-001", "waitMs": 1000 }
```

`media_open` checks the normal secret/path/symlink/hardlink rules, makes a private
streamed copy, hashes it and starts a probe job. Save the returned `mediaId` and
job ID. The source byte budget is separate from ordinary code-file reads: local
media copies are bounded to 512 MiB per file and 1 GiB aggregate. Larger movies
must first be supplied in smaller clips; this version does not promise arbitrary
multi-gigabyte ingestion. Source bytes are not rewritten.

Use `media_extract` to request selected frame times or an audio interval. The
returned job can be read through `media_job`; completed results contain asset IDs.
`media_read` returns the actual selected JPEG/WAV content to an MCP client that
supports those content types. Long-running work is represented by real job IDs,
not hidden retry loops. Existing `job_status`, `job_output`, `job_wait` and
`job_cancel` remain available for these owned jobs.

Frame timestamps describe requested seek locations; selected frames are not proof
that every frame in the movie has been inspected. Audio and transcription report
their inspected interval. To understand a long clip, the client should request a
coarse set of frames, read/transcribe relevant audio, then inspect denser frames
around the question. Never describe a subtitle-only pass as having watched the
whole movie. Scene interpretation, summarization and choosing additional evidence
are the connected model's job.

Native processors receive fixed argv, private files and bounded output, not
caller-supplied shell code. Media protocols/demuxers are restricted; this is not a
general playlist downloader, URL fetcher, HLS loader or DRM removal tool. Decoder
errors, missing codecs, invalid ranges and cancellation are surfaced without
invented frame/transcript assets. Files/native decoders remain untrusted input and
local software; this does not remove documented same-OS-user/OS-sandbox limitations.

Close media when finished using `media_close`. It removes owned source copies,
invalidates derived access and cancels associated jobs when needed. Completed
worker output and temporary speech input are removed after successful import into
the bounded asset cache. Handles and temporary resources also expire and are
cleaned by periodic housekeeping, which performs no screen/microphone capture.

## 3. Transcripts, subtitles and speech

`media_transcribe` accepts an opened media ID, a bounded start/duration interval,
language, local model path and idempotency key. Whisper receives mono audio extracted
locally; transcription is not sent to an external speech API. Returned segments
have timeline offsets and ASR provenance. The multilingual tiny model is small,
not a promise of transcription accuracy for every speaker, language or soundtrack.

`media_subtitles` imports a workspace SRT/VTT file for a media handle. Imported
subtitles are explicitly marked as supplied subtitle evidence, not audio that the
server listened to. `media_search` performs bounded literal text search over a
transcript, returning matched segments and times; it is not semantic visual search
across unseen scenes. Use `media_read` to page through transcript segments.

`speech_synthesize` takes bounded text and optionally an installed voice. It uses
the prepared local backend for the current OS and creates real WAV output as an
asset without starting speaker playback. Text is supplied through a private input
file rather than native command-line arguments.
Reading the resulting asset returns MCP audio content. Whether a host renders or
passes that audio into a model depends on the host; the server cannot add native
speech support to an incompatible chat client.

This version handles files/selected clip audio. It does not enable a microphone,
record system audio continuously or provide a live two-way voice conversation.
No new OS permission is silently requested.

## 4. Browser and application operation

Create an isolated browser session with `browser_session`:

```json
{
  "mode": "open", "source": "workspace", "path": "demo/index.html",
  "idempotencyKey": "open-demo-001"
}
```

Workspace mode serves allowed local assets through a private synthetic HTTPS
origin. It blocks outgoing network, mutation requests and secret/protected files;
it is not unrestricted access to arbitrary localhost services. Static video
responses support bounded single HTTP byte ranges so playback can seek correctly.

Public mode requires the owner's already-enabled `allowWebFetch` setting and a
bounded list of exact public origins. DNS addresses are checked and pinned for
requests. Private/loopback/special addresses and the configured JUJU public origin
are refused; redirects/subresources must pass the same origin/address checks.
No new tool enables public access when that existing setting is off.

Each browser has a fresh isolated context, not cookies/history/passwords from your
personal browser. Popups, automatic downloads, dialogs, service workers and direct
socket-style channels are restricted. A bounded number of sessions can exist;
closing a session closes only its owned browser. Existing sandbox configuration
is honored for launch, with no automatic fallback that disables required protection.

`browser_observe` returns a screenshot, bounded visible control descriptors with
selectors, text, console/network summaries and media-element playback state.
Input/password values are not read by the DOM snapshot; screenshots and page text
may still contain private information visible in that isolated application.
Evidence is untrusted data, never instructions or authorization.

`browser_action` requires a fresh observation ID and one fixed action such as
click/fill/press/select/scroll/navigate or HTML media play/pause/seek/mute. Selectors
must be unique and come from the observation. Ordinary actions reject intervening
DOM changes. Media controls check URL/control identity while allowing the playback
clock to advance. Media operations call an actual fixed browser function and wait
for relevant media events; playback time/readiness/error state are returned for
verification. No arbitrary evaluation string or remote-debugging endpoint is exposed.

An input acknowledgement is not proof that an entire user task succeeded. Inspect
the after-image and stated postconditions. Effects use idempotency receipts;
uncertain outcomes are not automatically replayed under a new key.

Native apps continue to use the existing permitted-window desktop controls. This
version does not provide unrestricted app APIs, personal-profile browser control,
or universal compatibility with protected streaming players.

## 5. Step-based games

Open `game_session` against an existing isolated browser session or an approved
native window. Observe the state, then call `game_step` with that observation ID,
one action and an idempotency key. Every successful step returns new visual evidence.
A session does not think or play by itself between calls.

Browser games support bounded key combinations/holds and clicks. All pressed keys
are released in `finally`, including failure paths. Native games currently use the
existing discrete key/click API: multi-key timed native holds are refused explicitly,
not simulated or claimed supported. Use pauseable, turn-based or permitted offline
games and fixtures to evaluate the model's decisions before more demanding games.
There is no claim of reflex-level performance, cheating/anti-cheat bypass, universal
victory, or game-specific learned policy. Game correctness/strategy is not proven
merely by delivering an input event.

## 6. Workflows from examples

The client/model can inspect a demonstration or explain a procedure, then store it
through `workflow_save`. The backend saves explicit bounded steps and evidence
notes; it does not retrain model weights or independently infer a procedure from a
video. Records are private to the originating workspace/principal and carry a
revision hash plus the marker `client_authored_untrusted_steps_not_permissions`.

Use `workflow_search` to find a name/goal/tag or retrieve a specific workflow.
`workflow_run(mode="start")` binds the exact revision to existing browser/native
handles and observes without executing. `mode="next"` requires the expected step
index, current observation and an idempotency key. It performs at most one action,
using the same browser or native desktop machinery and existing approval policy.
Before/after text expectations are checked; mismatch pauses for review rather than
blindly continuing. Native preconditions are reread through Accessibility before
input, not accepted solely from cached text. Manual steps are labeled as client
confirmation, not machine-verified completion. `inspect` refreshes evidence and
`stop` prevents subsequent steps. Records cannot grant permission or invoke
arbitrary tools, shell code or owner controls.

## Lifecycle, privacy and limits

Sources, observations, browser/game/run handles and buffers are bounded and expire.
Source copies and processing outputs use private temporary directories; screenshots
and returned audio/image buffers use bounded caches. Durable effect receipts may
contain bounded UI evidence, and explicitly saved workflows persist in private
JUJU state. Do not put secrets in demonstration notes. Closing/switching a workspace
shuts down its owned media/browser resources and invalidates runtime handles.

Preparing data locally does not mean data never leaves the machine: when tool
images/audio/transcripts are returned to a cloud model, those selected contents
are sent to that client. This system neither records everything automatically nor
makes a chat application run an unattended agent after a conversation ends.

Native desktop/speech behavior is platform-specific: macOS, Windows x64 and Linux
X11/XWayland paths have 0.8.2 acceptance evidence. Linux pure Wayland portal-only,
Windows ARM64 and macOS Intel are not inferred from those runs. Host support for MCP
image/audio blocks and suitable model capabilities is still required. A missing
capability is reported, not silently replaced by guessed analysis.

## Tests and activation

Regression files include `tests/unit/multimodal.test.ts`,
`tests/unit/mediaByteRange.test.ts`, `tests/integration/multimodal.test.ts`,
`tests/security/multimodal.test.ts` and the 0.8.2 setup/platform suites. They cover
real media decoding, ASR/TTS, browser forms/video/game/workflow behavior and
permission/failure boundaries. Separate native acceptance additionally exercised
Windows Desktop/SRT and Linux X11/AT-SPI/bubblewrap paths.

```bash
npm run build
npm run test:all
juju setup --check
```

See [TEST_REPORT.md](TEST_REPORT.md) and [RELEASE_0.8.2.md](RELEASE_0.8.2.md) for
exact evidence and untested boundaries. To activate an upgrade, finish existing
jobs, stop the intended old process, install `juju-mcp@0.8.2`, run `juju setup`,
start JUJU again and refresh the client's tool catalog. Package publication alone
does not replace a separate global installation or restart a running process.
OAuth grants and existing security settings are retained.