# @octomil/browser
Run ML models in the browser. WebGPU-accelerated, WASM fallback, zero server required.
[](https://github.com/octomil/octomil-browser/actions/workflows/ci.yml)
[](https://www.npmjs.com/package/@octomil/browser)
[](https://github.com/octomil/octomil-browser/blob/main/LICENSE)
## What is this?
A TypeScript SDK for running ONNX models directly in the browser with WebGPU or WASM. The model downloads once, caches locally, and runs on the user's device. It is a good fit for classification, embeddings, image recognition, and other browser-safe ONNX workloads where low latency and offline support matter.
## Install
```bash
pnpm add @octomil/browser
# or
npm install @octomil/browser
```
## Authentication
`OctomilClient` requires an `auth` field. Two auth modes are supported:
### Organization API key
Use this when your app has a backend that can provision org-scoped credentials.
```typescript
auth: {
type: 'org_api_key',
apiKey: 'your-api-key',
orgId: 'your-org-id',
serverUrl: 'https://api.octomil.com', // optional, defaults to production
}
```
### Device token
Use this for device-style bootstrap and registration flows.
```typescript
auth: {
type: 'device_token',
deviceId: 'device-uuid',
bootstrapToken: 'bootstrap-token',
serverUrl: 'https://api.octomil.com', // optional
}
```
## Quick Start (Hosted/Cloud)
The unified facade sends requests to the Octomil cloud API. Use a publishable key (client-safe, `oct_pub_live_` prefix) -- never embed server API keys in browser code.
```typescript
import { Octomil } from "@octomil/browser";
// Use a publishable key -- safe for client-side code
const client = new Octomil({ publishableKey: "oct_pub_live_..." });
await client.initialize();
const response = await client.responses.create({
model: "phi-4-mini",
input: "Hello",
});
console.log(response.outputText);
```
### Embeddings (Hosted/Cloud)
```typescript
const result = await client.embeddings.create({
model: "nomic-embed-text-v1.5",
input: "On-device AI inference at scale",
});
console.log(result.embeddings[0].slice(0, 5));
```
Array input is also supported:
```typescript
const result = await client.embeddings.create({
model: "nomic-embed-text-v1.5",
input: ["first document", "second document"],
});
```
### Migrating from OctomilClient
`OctomilClient` and the low-level `ResponsesClient` APIs still work exactly as before. The `Octomil` facade is a convenience wrapper for the cloud-backed Responses path — it delegates to `ResponsesClient` internally. For local ONNX inference, continue using `OctomilClient` directly.
## Advanced Usage (OctomilClient)
```typescript
import { OctomilClient } from '@octomil/browser';
const ml = new OctomilClient({
model: 'https://models.octomil.com/sentiment-v1.onnx',
auth: {
type: 'org_api_key',
apiKey: 'your-api-key',
orgId: 'your-org-id',
serverUrl: 'https://api.octomil.com',
},
});
await ml.load();
const result = await ml.predict({ text: 'This product is incredible' });
console.log(result.label, result.score); // "1" 0.97
ml.close();
```
The SDK downloads the model, caches it via the Cache API, and picks the fastest backend available (`WebGPU` first, `WASM` fallback).
## Features
### Multiple input types
```typescript
// Text
await ml.predict({ text: 'classify this' });
// Image (canvas, img element, or raw ImageData)
await ml.predict({ image: document.querySelector('canvas') });
// Raw tensors
await ml.predict({ raw: new Float32Array(784), dims: [1, 1, 28, 28] });
```
### Automatic model caching
Models cache locally after the first download. Later loads can skip the network entirely.
```typescript
const ml = new OctomilClient({
model: 'https://models.octomil.com/sentiment-v1.onnx',
auth: {
type: 'org_api_key',
apiKey: 'your-api-key',
orgId: 'your-org-id',
},
cacheStrategy: 'cache-api', // default; also 'indexeddb' or 'none'
});
await ml.load(); // downloads once
await ml.isCached(); // true on next visit
```
### Routing policies
The SDK exports the same routing policy names as the Python and Node SDKs for API parity. Use `validateRoutingPolicy()` to check policy names and `assertBrowserCompatiblePolicy()` to reject policies that require local execution.
```typescript
import {
validateRoutingPolicy,
assertBrowserCompatiblePolicy,
VALID_ROUTING_POLICIES,
} from '@octomil/browser';
// Validate a policy name (throws on unknown names like "quality_first")
const policy = validateRoutingPolicy("cloud_first"); // ok
// Assert browser compatibility (throws on "private" and "local_only")
assertBrowserCompatiblePolicy("cloud_only"); // ok
assertBrowserCompatiblePolicy("private"); // throws POLICY_DENIED
```
Supported policies: `cloud_only`, `cloud_first`, `local_first`, `performance_first`. The `private` and `local_only` policies require local on-device execution and are rejected with a clear error in the browser SDK.
### Browser SDK limitations
The browser SDK is **hosted/cloud only** for the runtime planner. It differs from the Python and native SDKs in these ways:
- No local model artifact download or caching via the planner (ONNX-web models use a separate `OctomilClient` path)
- No local inference engine detection or benchmarking
- No offline plan resolution
- `private` and `local_only` routing policies are rejected -- these require capabilities the browser cannot provide
- The planner types (`RouteMetadata`, `RuntimeSelection`, etc.) are exported for type parity but the browser SDK does not implement a planner client that calls POST /api/v2/runtime/plan
### Smart routing (device vs. cloud) (Hosted/Cloud)
The SDK can choose between local and cloud inference based on model size and device capabilities, then fall back cleanly when conditions change.
```typescript
const ml = new OctomilClient({
model: 'phi-4-mini',
auth: {
type: 'org_api_key',
apiKey: 'oct_pub_live_...', // publishable key, safe for browser
orgId: 'your-org-id',
serverUrl: 'https://api.octomil.com',
},
routing: { prefer: 'fastest' }, // 'device' | 'cloud' | 'cheapest' | 'fastest'
});
```
### Streaming and embeddings (Hosted/Cloud)
```typescript
// Stream tokens via SSE
for await (const token of ml.predictStream('phi-4-mini', 'Explain quantum computing')) {
process.stdout.write(token.token);
}
// Generate embeddings
const { embeddings } = await ml.embed('nomic-embed-text', ['query', 'document']);
```
### Batch inference
```typescript
const results = await ml.predictBatch([
{ text: 'great product' },
{ text: 'terrible experience' },
{ text: 'it was okay' },
]);
```
### Federated learning with differential privacy
On-device training with built-in gradient clipping, noise injection, and secure aggregation. Raw user data never leaves the browser.
```typescript
import { clipGradients, addGaussianNoise } from '@octomil/browser';
const noised = addGaussianNoise(clipGradients(delta, 1.0), 0.01);
```
## Browser Support
| Browser | Backend | Notes |
|---------|---------|-------|
| Chrome 113+ | WebGPU | Full GPU acceleration |
| Edge 113+ | WebGPU | Full GPU acceleration |
| Firefox | WASM | WebGPU behind flag |
| Safari 18+ | WASM | WebGPU partial support |
| All modern browsers | WASM | Universal fallback via WASM SIMD |
The SDK auto-detects the best backend. Pass `backend: 'webgpu'` or `backend: 'wasm'` to override.
**Limitations**: Works well for models up to ~500MB. Large LLMs will hit browser memory limits. WebGPU performance varies by GPU. WASM is slower but universal.
### Control plane (optional)
The Browser SDK has optional control plane integration for device registration, desired-state sync, and heartbeats. This is separate from the main inference path.
```typescript
import { configure } from '@octomil/browser';
// Silent device registration (background, exponential backoff)
configure({
auth: { type: 'publishable_key', key: 'oct_pub_live_...' },
monitoring: { enabled: true },
});
```
Or use the control namespace directly:
```typescript
await ml.control.register();
ml.control.startHeartbeat(300_000); // 5 min interval
const state = await ml.control.fetchDesiredState();
await ml.control.sync({ modelInventory: [...] });
```
`SyncManager` is a standalone opt-in export for periodic desired-state reconciliation (not automatic):
```typescript
import { SyncManager } from '@octomil/browser';
const sync = new SyncManager({ intervalMs: 300_000, onEvent: (e) => console.log(e) });
```
**Key difference from iOS/Android:** The Browser SDK has no `AppManifest`. It is model-URL-driven — you pass a direct `.onnx` URL to the constructor. `predict()` is local ONNX inference only; `predictStream()` and `embed()` hit cloud endpoints.
## Architecture
```
OctomilClient → ModelManager (download/cache) → InferenceEngine (ONNX Runtime Web)
→ RoutingClient (device vs. cloud) → StreamingEngine (SSE tokens)
→ ControlClient (register, heartbeat, sync) — optional
→ TelemetryReporter (opt-in metrics)
configure() → SilentAuth (background registration) + HeartbeatManager
SyncManager → Periodic desired-state reconciliation (standalone, opt-in)
```
## Script Tag (no bundler)
```html
```
## API Reference
[docs.octomil.com](https://docs.octomil.com)
## Contributing
```bash
git clone https://github.com/octomil/octomil-browser.git && cd octomil-browser
pnpm install && pnpm test && pnpm run build
```
## Releasing
Releases publish to npm from GitHub Releases via trusted publishing with npm provenance. Configure npm trusted publishing for `@octomil/browser` with repository `octomil/octomil-browser` and workflow `.github/workflows/publish.yml`.
```bash
pnpm install --frozen-lockfile
pnpm run typecheck
pnpm test
pnpm run build
pnpm run exports:check
pnpm run pack:check
```
Then create a GitHub Release for the package version in `package.json`. The publish workflow runs the same gates and publishes `@octomil/browser` with public scoped-package access and provenance.
## License
MIT