---
name: Using Local Models
description: Run agents on models that live on your machine. Install the local provider, browse and download models, and use them with the agent, with agency run, and from the llm function.
---

# Using Local Models

Agency can run LLM calls on models stored on your own machine. Nothing leaves your computer, you need no API key, and the calls cost nothing. Local models are slower than hosted ones and less capable at the same size, so they shine for development, testing, offline work, and private data.

Local inference uses [llama.cpp](https://github.com/ggml-org/llama.cpp) under the hood. Models are single `.gguf` files, mostly downloaded from Hugging Face.

## Setup

Install the local provider once:

```bash
npm i -g smoltalk-llama-cpp
```

Agency finds the package automatically, whether you installed it globally or in your project. You can browse the catalog without it; any command that needs it will tell you to run this install.

## Browse the catalog

```bash
agency local list
```

The first line shows the directory models are downloaded to. Below it, you get the full catalog of curated models, with a checkmark next to the ones you have already downloaded:

```
Models directory: /Users/you/.agency-agent/models

   NAME          PARAMS  SIZE      CONTEXT  LICENSE
✓  smollm2-135m  135M    0.11 GB   8K       apache-2.0
   qwen3.5-2b    2B      1.28 GB   128K     apache-2.0
   gpt-oss-20b   20B     12.00 GB  128K     apache-2.0
   ...
```

For longer descriptions of each model, run `agency local alias list`.

**Picking a model:** the SIZE column is roughly what the model takes on disk and in memory, so it is the main thing to match against your hardware. Start small. `smollm2-135m` downloads in seconds and is good for checking that everything works. `qwen3.5-2b` is a reasonable first model for real tasks on a laptop.

## Download a model

```bash
agency local download
```

With no argument, this opens a picker so you can choose from the catalog. You can also name a model directly:

```bash
agency local download qwen3.5-2b
```

The value can be a curated name, one of your aliases, a Hugging Face URI, or a path to a `.gguf` file you already have:

```bash
agency local download hf:Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M
```

Downloads of curated models are verified against pinned SHA-256 hashes. A file that fails verification is set aside and never loaded.

You do not have to download ahead of time. Everything that runs a local model downloads it first if it is missing. Pre-downloading just moves the wait to a moment you choose.

## See where models live

The first line of `agency local list` names the models directory. By default it is `~/.agency-agent/models`. To change it, set the `AGENCY_MODELS_DIR` environment variable, or set `client.modelsDir` in `agency.json`:

```jsonc
{
  "client": {
    "modelsDir": "/data/agency-models"
  }
}
```

To free up disk space, delete a model's files with `-f`. Without `-f` the
command only removes the alias and tells you where the files are.

```bash
agency local remove qwen3.5-2b -f
```

## Refresh the catalog

The catalog of curated models updates over time. Pull the latest list without upgrading agency:

```bash
agency local refresh
```

New and updated entries land in your `agency.json` as aliases. Any alias you added yourself is never overwritten. The [local command reference](/cli/local) covers the details, including pointing `refresh` at your own catalog.

## Run the agent on a local model

```bash
agency agent --local qwen3.5-2b
```

This downloads the model if needed and points every LLM call at it, including the deep subagents. A fully local session needs no hosted API key at all. Run `agency agent --local` with no value to pick from the catalog interactively.

## Run a program on a local model

The `--local` flag on `agency run` pins the whole run to a local model:

```bash
agency run --local qwen3.5-2b hello.agency
```

The value accepts the same forms as `agency local download`: a curated name, an alias, an `hf:` URI, or a `.gguf` path. The download and verification happen before your program starts, so you see the progress in your terminal. `--local` and `--model` are mutually exclusive.

This composes well with [run budgets](/cli/run). A run that must not spend money can say so explicitly:

```bash
agency run --max-cost 0 --local qwen3.5-2b hello.agency
```

## Use local models in code

For finer control, resolve a model in code and pass it to `llm` yourself. The `registerLocalModel` function downloads the model if needed and returns its local path:

```ts
import { registerLocalModel } from "std::agency/local"

node main() {
  const model = registerLocalModel("qwen3.5-2b")
  const answer = llm("What is the capital of France?", {
    model: model,
    provider: "llama-cpp",
  })
  print(answer)
}
```

Structured output and tool calls work the same way they do on hosted models:

```ts
type Capital = {
  city: string
  population: number
}

node main() {
  const model = registerLocalModel("qwen3.5-2b")
  const answer: Capital = llm("What is the capital of France?", {
    model: model,
    provider: "llama-cpp",
  })
  print(answer.city)
}
```

This is useful when one program mixes models, for example routing cheap classification to a local model and hard reasoning to a hosted one. `std::agency/local` also exports the rest of the CLI's capabilities as functions: `downloadModel`, `listModelNames`, `listDownloadedModels`, `aliasModel`, and more. See the [std::agency/local reference](/stdlib/agency/local).

To make local the default for a whole project instead, set it in `agency.json`:

```jsonc
{
  "client": {
    "defaultProvider": "llama-cpp",
    "defaultModel": "/Users/you/.agency-agent/models/my-model.gguf"
  }
}
```

## Name your own models with aliases

An alias gives a short name to any model URI, so your team can share one `agency.json` and write `llm(..., { model: registerLocalModel("my7b"), ... })` everywhere:

```bash
agency local alias add my7b hf:Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M
agency local alias list
agency local alias remove my7b
```

Aliases work everywhere a model value is accepted: `agency local download`, `agency run --local`, `agency agent --local`, and `registerLocalModel`.

## Run MLX models on a Mac

MLX is Apple's array framework for Apple Silicon. Agency can run models through it as well as through llama.cpp. MLX models are often faster than GGUF models on a Mac, and the community publishes them at sizes llama.cpp builds rarely reach.

An MLX model is not one file. It is a directory of `.safetensors` weights next to a `config.json`, the layout Hugging Face repos use. Agency does not run it in its own process. A Python program called `mlx_lm.server` loads the model, and Agency sends it requests. Agency ships a small script around it that adds structured output, using the `llguidance` library, so a typed `llm()` call gets a reply that fits its type.

Set up Python once:

```bash
python3.12 -m venv ~/.agency-agent/mlx-env
~/.agency-agent/mlx-env/bin/pip install mlx-lm==0.31.3 llguidance==1.8.0
```

Agency looks for that environment by default. Agency never installs Python for you. To use a different Python, pass `--python`, or set `client.mlx.python` in `agency.json`.

### Download a model

```bash
agency local download mlx:mlx-community/Qwen3.8-27B-4bit
```

An `mlx:` URI names a Hugging Face repo. The download fetches the repo in parallel and verifies each file against the hash Hugging Face publishes. Interrupt it and run the same command again, and it picks up where it stopped.

Browse what is available with `agency local list`. MLX entries show `mlx` in the BACKEND column, and their names end in `-mlx` when a GGUF entry of the same model exists.

### Start the server

```bash
agency local serve
```

With no model named, this shows you the MLX and diffusers models you have downloaded, each with its kind (chat, embedding, speech, or image), and you pick the ones to serve. Name them yourself to skip the picker:

```bash
agency local serve mlx:mlx-community/Qwen3.8-27B-4bit
```

The command runs in the foreground and prints the address it is listening on. Leave it running in its own terminal. Ctrl-C stops it and every model it loaded.

You can serve several models at once. Agency starts one `mlx_lm.server` for each of them and puts a single port in front. Each model stays in memory for as long as the command runs, so watch the total against the memory your Mac has. Agency warns you when the models add up to more than that, and starts them anyway.

A server uses memory beyond its weights while it works, and it cuts short a reply that goes in circles. Both are covered under [What is different about a local model](#what-is-different-about-a-local-model) below, along with the flags that tune them.

### Run against the server

```bash
agency run --local mlx:mlx-community/Qwen3.8-27B-4bit hello.agency
agency agent --local mlx:mlx-community/Qwen3.8-27B-4bit
```

These work the same way they do for a GGUF model, with one difference. Agency does not download or load anything here. The server has to be running already, and it has to be serving the model you name.

A run sends the model name to the server, so the two must agree. `agency local resolve <name>` prints the name Agency will send.

### Use a server on another machine

`agency local serve` only answers requests made on the machine it runs on. To use it from another computer, put something in front of it that the other computer can reach. With [Tailscale](https://tailscale.com), run this on the machine with the server:

```bash
tailscale serve --bg --https=8090 http://127.0.0.1:8080
```

The other machines on your tailnet can now reach the server at an address such as `https://my-mac.tailnet.ts.net:8090/v1`.

On the other computer, you can point Agency at that address in two ways. The first is the `MLX_BASE_URL` environment variable, which every command reads:

```bash
MLX_BASE_URL=https://my-mac.tailnet.ts.net:8090/v1 agency agent --local mlx:mlx-community/Qwen3-Coder-Next-4bit
```

The second is a preset for `agency agent`, saved in `~/.agency-agent/settings.json`:

```json
{
  "presets": {
    "studio": {
      "local": "mlx:mlx-community/Qwen3-Coder-Next-4bit",
      "llmOptions": {
        "baseUrl": { "mlx": "https://my-mac.tailnet.ts.net:8090/v1" }
      }
    }
  }
}
```

`agency agent --preset studio` then sends every chat request to that machine. You can also ask the agent to save the preset for you. It asks you to approve the address first, because the server at that address sees the whole conversation and writes the model's replies. A preset can only set the address of an MLX server. Other providers send an API key with each request, and the key would go to that address.

The preset only changes where the agent's chat requests go. These still use `MLX_BASE_URL`, or the default address on this machine:

- The speech, image, and vision functions in the standard library, and `agency-lang/local`. When one of them cannot reach a server, its error names the address it tried.
- Agency code that the agent runs in a subprocess.

A program you compile yourself can set the address in `agency.local.json`, under `client.baseUrl.mlx`.

### Use a model you already have

```bash
agency local serve /Volumes/models/hf/hub/models--mlx-community--Qwen3.8-27B-4bit
```

Any directory holding `config.json` and `.safetensors` files works, so a model another tool downloaded needs no copying. A Hugging Face cache folder works too. Agency reads its `refs/main` to find the revision you last pulled.

```bash
agency local --model-dir /Volumes/models/hf/hub list
agency local --model-dir /Volumes/models/hf/hub serve
```

`--model-dir` points the whole `local` command at another directory for one run. The picker then offers everything in that directory. Agency reads a Hugging Face cache but never writes to one, so `agency local remove -f` refuses to delete a model in one.

### Watch what the server is doing

The server prints one line per request:

```
POST /v1/chat/completions  mlx-community/Qwen3.8-27B-4bit  200  2.6s  16→129 tok
```

That line gives you the endpoint, the model, the status, the time the request took, and the tokens in and out. Add `--verbose` to see the whole request and the whole reply as well:

```bash
agency local serve mlx:mlx-community/Qwen3.8-27B-4bit --verbose
```

### Shorter names

```bash
agency local alias add coder mlx:mlx-community/Qwen3-Coder-Next-4bit
agency local serve coder
agency run --local coder hello.agency
```

An alias works everywhere a model value is accepted, the same as it does for a GGUF model.

If you already have the model, you can also drop the prefix and use the repo id on its own:

```bash
agency local serve mlx-community/Qwen3-Coder-Next-4bit
```

The `mlx:` prefix is the spelling that always works. You need it for a model you have not downloaded yet, because that is what tells Agency where to fetch it from.

### Differences from GGUF models

The table at the end of [What is different about a local model](#llamacpp-and-the-mlx-server-side-by-side) sets the two backends side by side. One difference is not in it: the agent's memory feature needs an embedding model, and a chat server does not serve one unless you name one. Name it when you start the server, and name it for the agent too. `serve` knows an embedding model when it sees one, so no flag is needed:

```bash
agency local serve qwen3.5-27b-mlx qwen3-embedding-4b-mlx
agency agent --local qwen3.5-27b-mlx --model embedding=mlx/qwen3-embedding-4b-mlx
```

In `agency.json`, the same thing is `embeddings: { model, provider: "mlx" }` under the memory settings.

## Generate images on a Mac

Four open image models run on this machine:

| Model | Good for | 1024×1024 on an M5 Ultra | Download |
|---|---|---|---|
| `z-image-turbo` | Fast, photorealistic images | about 8 s | 32.8 GB |
| `chroma1-hd` | Detailed, cinematic pictures | about 90 s | 27.5 GB |
| `qwen-image-2512` | Legible text inside the image: signs, posters, diagrams | not measured yet | 57.7 GB |
| `flux2-klein-4b` | Macs with less memory; 4 steps per image | not measured yet | 16.0 GB |

`z-image-turbo` and `chroma1-hd` have no content filter in their weights; `flux2-klein-4b` is safety fine-tuned. They run on Hugging Face's diffusers library rather than MLX, so they need torch and diffusers in the same Python:

```bash
~/.agency-agent/mlx-env/bin/pip install torch==2.14.0 torchvision==0.29.2 diffusers==0.40.0 transformers==5.17.0 accelerate==1.15.0 sentencepiece==0.2.2 protobuf==7.36.2
```

You can serve images without installing any MLX packages; `serve` checks only for what the models you name need.

Download a model and serve it. `serve` reads what kind of model it is from the download, so it starts the image server without being told:

```bash
agency local download z-image-turbo
agency local serve z-image-turbo
```

Then call it from Agency code:

```ts
import { generateImageLocal } from "std::image"

node main() {
  const r = generateImageLocal("a lighthouse in a storm", "z-image-turbo", seed: 7)
  if (isFailure(r)) { print("failed: ${r.error}"); return }
  writeBinary("lighthouse.png", r.value.base64)
  print("made with seed ${r.value.seed}")
}
```

The same prompt and seed make the same image. Leave `seed` out and the server picks one, and the result says which. `steps` and `guidance` default to each model's own settings. `z-image-turbo` and `flux2-klein-4b` take no guidance and no negative prompt, and say so if you pass one.

Image models use more memory while they generate than their size on disk: `chroma1-hd` is 27.5 GB on disk and peaks at 36 GB.

### Your own style with a LoRA adapter

A LoRA adapter is a small file that teaches an image model a style or a character from a few dozen example pictures. SDXL models take them, and the illustration finetunes NoobAI-XL and Illustrious are the ones most adapters are trained for. Serve one by its repo, and name the folder your adapters are in:

```bash
agency local download diffusers:Laxhar/noobai-XL-1.1
agency local serve diffusers:Laxhar/noobai-XL-1.1
```

```json
{ "client": { "adaptersDir": "./adapters" } }
```

Then name the adapter in the call, by its file name without `.safetensors`. A request that leaves `lora` out gets the plain model:

```ts
const r = generateImageLocal("sketch, a cat on a chair", "diffusers:Laxhar/noobai-XL-1.1", lora: "sketch", loraScale: 0.9)
```

`loraScale` is how strongly the adapter is applied: 1 is as trained, less is subtler, and up to 2 is allowed. The server loads an adapter the first time it is asked for, so a file you drop into the folder after training works at once, and when you train it again under the same name, the next request uses the new weights. Adapters are `.safetensors` files only, and a request can only pick a file from the folder you configured, never name a path itself.

### Pose a character with a ControlNet

A ControlNet holds the image to a drawing you give it, so a stick figure becomes the pose. Name a folder for them, download one, and pass a drawing with the call:

```json
{ "client": { "controlnetsDir": "./controlnets" } }
```

```bash
agency local download controlnet-scribble-sdxl
```

```ts
const r = generateImageLocal("pen and ink, zxq_girl, surprised", "diffusers:Laxhar/noobai-XL-1.1",
  lora: "zxq", controlnet: "controlnet-scribble-sdxl", controlImage: "./poses/jump.png")
```

The drawing is read on this machine under `std::readImage`, the one thing a local generation asks approval for. `controlScale` is how strongly the drawing constrains the image, 1 by default. `controlnet-openpose-sdxl` takes a rendered pose skeleton instead of a scribble.

The scribble ControlNet reads white lines on a black background. A drawing made with dark lines on white paper needs `invertControlImage: true`:

```ts
const r = generateImageLocal("a dancer mid-leap", "diffusers:Laxhar/noobai-XL-1.1",
  controlnet: "controlnet-scribble-sdxl", controlImage: "./poses/pen-sketch.png", invertControlImage: true)
```

The drawing is scaled to fit `size` with its shape kept, and centered on black. A 4:3 drawing in a square image gets black bands above and below. It is never stretched.

## Look at images on a Mac

Two vision models turn a picture into words and boxes, on this machine, with nothing uploaded. `wd14-tagger` describes an image as booru tags, the vocabulary illustration models such as NoobAI-XL take in a prompt. `florence-2` finds the objects you name and returns a box for each, tags what it sees, and writes a caption. Download and serve them like any other model:

```bash
agency local download wd14-tagger florence-2
agency local serve wd14-tagger florence-2
```

Then ask. Each function does one thing and returns a `Result`, so a loop over a folder is your own code, and `cropImage` from `std::image` turns a box into a new file:

```ts
import { detectObjects, tagImage } from "std::vision"
import { cropImage } from "std::image"
import { glob } from "std::shell"

node main() {
  const pages = glob("comics/*.png") catch []
  for (page in pages) {
    const found = detectObjects(page, ["person", "desk", "chair"], "florence-2") catch []
    for (hit in found) {
      const out = "dataset/${hit.label}_${hit.id}.png"
      cropImage(page, hit.box, out, pad: 0.05)
      const tags = tagImage(out, "wd14-tagger") catch []
      const names = map(tags) as t {
        return t.tag
      }
      write("${out}.txt", names.join(", "))
    }
  }
}
```

Run it with `--approve std::glob --approve std::vision --approve std::cropImage --approve std::write` and it labels a few hundred crops unattended. Every call raises an effect naming the file it reads or writes, so a policy can allow tagging under `./dataset` and nothing else. Partial application narrows a tool before an agent gets it: `detectObjects.partial(labels: ["person"], model: "florence-2")` is a person-finder, and `cropImage` only ever writes a file that does not exist yet.

### Find your own things

A detector finds things by the name of their kind. Ask for "mug" and it finds every mug, with no way to tell yours from the one next to it. Ask for "cat" on a pen-and-ink page and it may find nothing, because it learned mostly from photos. Two more models close both gaps without training anything:

- `owlv2-base` finds the things you name, like `florence-2`, and `findRegions` asks it to box every thing in a picture with no names at all.
- `dinov2-base` turns a picture, or boxes in it, into embeddings: lists of numbers that say how each looks. Things that look alike get embeddings that are close together.

```bash
agency local download owlv2-base
agency local download dinov2-base
agency local serve owlv2-base dinov2-base
```

To find your cats on a page: box everything, embed each box, and keep the boxes that are closer to crops of your cats than to crops of other things you draw.

```ts
import { findRegions, embedImage } from "std::vision"
import { cosineSimilarity } from "std::embedding"

// How alike `vector` is to the closest of `references`, where 1 is identical.
def closest(vector: number[], references: number[][]): number {
  let best = 0
  for (reference in references) {
    const similarity = cosineSimilarity(vector, reference)
    if (isSuccess(similarity) && similarity.value > best) {
      best = similarity.value
    }
  }
  return best
}

// One embedding per crop file.
def embedCrops(paths: string[]): number[][] {
  let vectors: number[][] = []
  for (path in paths) {
    const embedded = embedImage(path, "dinov2-base")
    if (isSuccess(embedded)) {
      vectors.push(embedded.value[0])
    }
  }
  return vectors
}

node main(page: string, catCrops: string[], otherCrops: string[]) {
  const cats = embedCrops(catCrops)
  const others = embedCrops(otherCrops)
  const regions = findRegions(page, "owlv2-base")
  if (isFailure(regions)) {
    return regions.error
  }
  const boxes = map(regions.value) as region {
    return region.box
  }
  const vectors = embedImage(page, "dinov2-base", boxes: boxes)
  if (isFailure(vectors)) {
    return vectors.error
  }
  for (vector, i in vectors.value) {
    if (closest(vector, cats) > closest(vector, others)) {
      print("Cat in region ${i}")
    }
  }
}
```

Run it with `--approve std::vision`. Make the crops with `findRegions` and `cropImage` on a few pages, then sort the files into two folders by hand. Compare like with like: a tight crop and a box around the same thing compare well, but a whole photo of a mug on a desk mostly describes the desk. For your mug, use `detectObjects(photo, ["mug"], "owlv2-base")` for the boxes, since the detector does know what a mug is, and crops of other people's mugs as the other side.

How well this works has been measured on one photo so far. Two cats scored 0.55 against each other and at most 0.26 against a TV remote, so telling kinds of thing apart works. Telling your own mug from a lookalike in another photo has not been measured yet, and neither have drawings.

## Chat with pictures on a Mac

```bash
~/.agency-agent/mlx-env/bin/python -m pip install mlx-vlm==0.7.0 mlx-audio==0.5.4
agency local serve --vlm qwen3.5-9b-mlx
```

Use `--vlm` to ask a chat model questions about pictures. The initial
supported architecture is `Qwen3_5ForConditionalGeneration`, tested with
`mlx-community/Qwen3.5-9B-4bit`. Download the model first with
`agency local download qwen3.5-9b-mlx`.

```agency
import { image } from "std::thread"

node main() {
  const answer = llm(["What is in this picture?", image("photo.png")], {
    provider: "mlx",
    model: "mlx-community/Qwen3.5-9B-4bit"
  })
  return answer
}
```

The image goes through the ordinary image-read approval. Set
`MLX_BASE_URL` to the server's `/v1` URL if it uses a port other than
8080. The model stays loaded between calls.

A typed call can ask for a specific value:

```agency
import { image } from "std::thread"

node main() {
  const sides: number = llm(["How many sides does this shape have?", image("square.png")], {
    provider: "mlx",
    model: "mlx-community/Qwen3.5-9B-4bit"
  })
  return sides
}
```

This is a chat model you ask questions in words. `std::vision` runs
fixed tasks such as tagging, captioning, and detection.

The flag is required because these same weights also work as text models.
Without it, Agency keeps using its existing text server. A model served
with `--vlm` has no hedge or repeat limits, `limitAnswers`, or `--draft`.
Thinking is off unless the call turns it on. Temperature and `maxTokens`
still apply. Requests for unsupported reply limits fail with an error.

Both runtimes can share one serve command. Pinning `mlx-audio` in the
install command preserves the version Agency's speech server requires.
Use `--python /path/to/env/bin/python` if you keep a separate environment.

An alias keeps its resolved model identity: an `mlx:` URI uses the repo
ID, and a directory uses its absolute path. `GET /v1/models` lists these
names once the server is ready. Serving uses installed files with Hub
networking and telemetry disabled. Missing files fail instead of downloading.

Chat bodies are limited to about 107 MB, including previous images resent
in the conversation. Each image sent by the client is capped at 20 MiB.
The local server accepts image bytes in data URIs and refuses remote URLs,
file paths, audio parts, and alternate image part formats.


### A reply you gave up on keeps running

With mlx-vlm 0.7.0, closing a streamed reply cancelled generation in the
real-model check. Closing a non-streaming reply did not. The abandoned
request finished before the next request generated its answer. A client
timeout therefore does not guarantee that GPU work stops.

Agency starts mlx-vlm with its upstream log level set to `CRITICAL`.
Upstream error messages can contain an invalid image's entire data URI.
Agency's front door still logs request status and timings, and redacts
image data from verbose request and response logs.

## Use local models from TypeScript

A TypeScript program can call the same server with no Agency code. Import from `agency-lang/local`:

```ts
import { listModels, generateImage, tagImage } from "agency-lang/local";
import { writeFileSync } from "node:fs";

const generated = await generateImage({
  model: "z-image-turbo",
  prompt: "a lighthouse in a storm",
  size: "1024x1024",
});
if (!generated.success) {
  throw new Error(generated.error);
}
writeFileSync("lighthouse.png", generated.value.bytes);

const tags = await tagImage({ model: "wd14-tagger", image: generated.value.bytes });
if (tags.success) {
  console.log(tags.value.map((tag) => tag.tag).join(", "));
}
```

Start the server first, with both models: `agency local serve z-image-turbo wd14-tagger`.

`generateImage` takes the settings `generateImageLocal` takes, and `detectObjects`, `tagImage`, `captionImage`, `embedImage`, and `findRegions` take the settings of their `std::vision` namesakes. Each sends the request the stdlib function sends. Four things differ:

- **No approvals.** The stdlib functions ask before they read a file, because a model may have chosen the file. Your program chose it, so these functions read it directly.
- **An image is a path or bytes.** Pass `image: "page.png"` or `image: someUint8Array`. An image already in memory does not need to be written to a file first.
- **The server's address is an option.** Pass `baseUrl: "http://127.0.0.1:8081/v1"` to call a server on another port. Without it, the functions use `client.baseUrl.mlx` in `agency.json`, then `MLX_BASE_URL`, then port 8080.
- **A call can be cancelled.** Pass `signal` from an `AbortController`. Aborting it ends the call with the failure `Cancelled`.

```ts
const cancelButton = new AbortController();
const pending = generateImage({ model, prompt, signal: cancelButton.signal });
cancelButton.abort();
const result = await pending; // { success: false, error: "generateImage failed: Cancelled" }
```

`listModels()` returns the models downloaded to this machine. Each entry has the model's `name`, `kind`, `family`, `directory`, size, and the aliases that point at it:

```ts
const imageModels = listModels().filter((model) => model.kind === "image" && model.complete);
```

### Start the server from your program

`serve` starts the same server `agency local serve` does, and resolves once every model has loaded:

```ts
import { serve, generateImage } from "agency-lang/local";

const server = await serve(["z-image-turbo", "wd14-tagger"], {
  log: (line) => console.log(line),
});
const generated = await generateImage({
  baseUrl: server.url,
  model: "z-image-turbo",
  prompt: "a lighthouse in a storm",
});
await server.close();
```

With no `port`, the server takes any free port, and `server.url` is its address. Pass it as `baseUrl` to each call. `log` receives what the command would print and what the model processes write. Without it, that output is discarded.

A model can be an object when it needs a setting the command gives with a flag:

```ts
await serve([
  { model: "qwen3.5-9b-mlx", vlm: true },                   // --vlm
  { model: "/models/my-embedder", kind: "embedding" },      // --embedding
  { model: "qwen3.5-9b-mlx", draft: "qwen3.5-0.8b-mlx" },   // --draft
]);
```

The server it returns can stop and start one model while the others keep running:

```ts
await server.unload("z-image-turbo");   // frees its memory
server.status();                         // [{ model: "z-image-turbo", state: "stopped", ... }, ...]
await server.load("z-image-turbo");     // resolves when it is ready again
```

A request for an unloaded model fails with a message that says to load it. `server.failure` is a promise that resolves with a message if a model process dies.

Any client can read the same status at `GET /v1/agency/status` on the server's address.

### Serve more models than fit in memory

Mark a model `lazy` to load it on its first request, and to let the server stop it when another lazy model needs the memory:

```ts
const server = await serve([
  "florence-2",                              // loads now, stays loaded
  { model: "z-image-turbo", lazy: true },    // loads on its first request
  { model: "flux2-klein-9b", lazy: true },
]);
```

On the command line this is `agency local serve florence-2 --lazy z-image-turbo --lazy flux2-klein-9b`.

The first request for a lazy model waits for the load, and that wait counts against the caller's timeout. An image request with eight steps is given about three minutes, and a cold load of a large image model can take a good part of that. When a request has a deadline, warm the model first with `server.load(model)`, which resolves when it is ready.

When a lazy model does not fit, the server stops the lazy model that has been idle longest and has no request running, and tries again. If every loaded model is busy or is not lazy, the request fails with a 503 that says so:

```
Not enough memory to load flux2-klein-9b (needs about 22 GB, 9 GB available).
Loaded now: z-image-turbo (busy), florence-2 (not lazy).
```

A caller can wait and retry, or `unload` something itself.

### Cancel a call, or everything on a model

To cancel your own call, pass a `signal` and abort it. The call fails with `Cancelled`, and the server closes its connection to the model process. For an image model that stops the work after the step in progress.

To stop whatever is running on a model, whoever started it:

```ts
await server.cancel("qwen3.5-9b");
```

```
POST /v1/agency/cancel
{ "model": "qwen3.5-9b" }
```

Every request in progress on that model fails with a 499. For most models the process is kept, because it stops on its own when the connection closes. A chat model served with `vlm: true` that was asked for a reply without streaming keeps generating after the connection closes, so for that one case the process is stopped and, unless the model is lazy, started again. That takes as long as loading the model.

### Stop the server

```ts
await server.close();
```

```
POST /v1/agency/shutdown
```

`close` refuses new requests, tells every model process to stop, waits up to five seconds for them to exit, forces any that remain, and then closes the port. The shutdown route does the same; under `agency local serve` the command then exits, and under `serve()` your program keeps running. If the server's own process is killed outright, each model process notices and exits on its own.

`agency-lang/local` is the supported way in. Files under `agency-lang/stdlib-lib/` are internal to Agency, and their names change between releases.

There is no chat function here. A chat model served by `agency local serve` answers the OpenAI chat API at the same address, so any OpenAI client works.

## What is different about a local model

A hosted provider makes a dozen small choices for you, and you never see them. A local model makes you see every one. This section lists the choices that catch people, what each looks like when it goes wrong, and what to do about it. Most apply to both backends. Where one applies to the MLX server alone, or to llama.cpp alone, the text says so.

### The same prompt gives the same reply

Both local backends pick the single likeliest token at every step when nothing tells them otherwise. This is called greedy decoding. A greedy model writes the same reply for the same prompt every time, to the token, and takes the same time doing it. Every hosted provider samples instead, so its replies vary from call to call.

Agency makes a local model sample too when a call names no temperature, with the settings its model card asks for. Every catalog entry carries its card's numbers: Qwen3.5 asks for a temperature of 1.0 with a top-p of 0.95 and a top-k of 20, Gemma 4 for 1.0 with a top-k of 64, Mistral Small for 0.15, and so on. A model the catalog does not know gets 0.7 with a top-p of 0.95. The MLX server is sent all of it in the request. llama.cpp is sent the temperature and the cut-offs under node-llama-cpp's own names, and applies its own top-p of 0.95 and top-k of 40 to a model with no card of its own. Hosted providers sample at 1.0, but on top of samplers of their own; on the MLX server, 1.0 with nothing else is sampling from the whole distribution, which a small 4-bit model turns into noise. Ask for the repeatable behaviour when you want it:

```ts
import { setLlmOptions } from "std::llm"

setLlmOptions({ temperature: 0 })
```

Two things follow from greedy decoding. Running a greedy model three times measures nothing the first run did not, so a benchmark at temperature 0 needs one trial. And a greedy model that starts going in circles cannot get out of them, because whatever led it to write "But wait" once leads it to write the same thing again.

### Thinking spends your output budget

A thinking model writes a block of reasoning before its answer. Your program never sees that block, but it counts against `maxTokens` along with the answer. A small model can spend thousands of tokens thinking about a one-line question. When the budget runs out inside the thinking block, the call returns an empty reply with a stop reason of `length`, and a typed call fails with "reply did not fit the type".

Turn thinking off when the task does not need it:

```ts
import { setLlmOptions } from "std::llm"

setLlmOptions({ thinking: { enabled: false } })
const label: string = llm("Is this review positive or negative? ...")
```

This asks the model's chat template to open no thinking block. On a classification or extraction task it saves most of the time a thinking model takes. Where thinking helps, give it a budget instead:

```ts
const answer: NumberAnswer = llm(problem, { thinking: { enabled: true, budgetTokens: 2048 } })
```

After the budget, the thinking block is closed for the model and it has to answer. `reasoningEffort` works too, and means the same amount of thinking on a local model as it does on Gemini:

| effort | tokens of thinking |
| ------ | ------------------ |
| low    | 2048               |
| medium | 8192               |
| high   | 16384              |

Both backends honour all of this. On the MLX server the switch goes to the chat template and the budget to the server's watcher. On llama.cpp the switch goes to the chat wrapper, for the models whose wrapper has one, and the budget to llama.cpp itself. node-llama-cpp picks that wrapper by reading the model's template, and for a model it gets wrong, such as a fine-tune with a changed template, the switch reaches nothing. Name the wrapper yourself in `agency.json` with `client.llamaCpp.chatWrapper`, using node-llama-cpp's name for it (`qwen`, `gemma4`, `harmony`, `chatML`). Which models the switch reaches depends on their template: Qwen and Gemma 4 have one, gpt-oss takes a `reasoning_effort` instead (which `reasoningEffort` sends), and DeepSeek has none, so on DeepSeek off means a budget of zero and the block closes as soon as it opens. The budget is always held under `maxTokens` by enough to answer in.

### Replies that go in circles

A model that is unsure can write "But wait, is that right? Let me reconsider." for thousands of tokens, or repeat one sentence until its budget runs out. With a hosted model this costs you money. With a local one it costs you the machine. A large model can take ten minutes to write a reply nobody wants, and every other call waits behind it.

The MLX server watches every reply for three signs of this:

1. Thinking past its budget. The budget is half of `maxTokens` unless the call sets `budgetTokens`.
2. Twelve second thoughts in the last two thousand tokens. "But wait," "Wait," "Hmm," "Hold on," "Let me reconsider," and phrases like them. A loop says these every few lines; an honest long reply says them a dozen times over thousands of tokens, which is why the count runs over a window rather than the whole reply.
3. The same sentence of six or more words, written three times in that window.

The last two watch the thinking only, unless you ask. An answer repeats itself for honest reasons: a refrain, a table with a repeated row, three similar functions in a file. Thinking rarely does. `--limit-answers` on `agency local serve`, or `limit_answers: true` on a request, watches answers too.

When the server sees a sign, it cuts the reply short in the way that leaves the most usable result. A thinking block is closed, so the answer can follow. A reply that must fit a type has the text field it is writing closed, so the rest of the type can still be filled in. A plain reply ends where it is. An answer cut short is reported to the client with a stop reason of `length`, the same as one that hit `maxTokens`; closing the thinking is not a cut, because the answer still comes. An Agency program does not see the stop reason, so a cut typed reply that still fits its type succeeds like any other. The server's log is where to look: it prints a line saying which limit tripped and what it did.

You can change the limits when you start the server. `0` turns one off:

```bash
agency local serve mlx:mlx-community/Qwen3.5-2B-4bit --hedge-limit 20 --repeat-limit 0
```

A program can change them for its own calls, which matters when an answer repeats itself on purpose. `replyLimits` takes the same three settings; a field left out keeps the server's:

```ts
import { setLlmOptions } from "std::llm"

// A song's chorus repeats, and that is not a loop.
setLlmOptions({ replyLimits: { repeatLimit: 0 } })
const lyrics: string = llm("Write a song with a chorus that repeats after every verse.")

// A single careful call may hedge more than twelve times honestly.
const proof: string = llm(problem, { replyLimits: { hedgeLimit: 30 } })
```

llama.cpp has no watcher. There, a reply that goes in circles runs to `maxTokens` or to the call's timeout, whichever comes first.

### Every call has a cap and a clock

Two limits bound every call, and a local model hits both more often than a hosted one. The cap is `maxTokens`. llama.cpp caps a call at 16,384 tokens when you set none. The MLX server caps it at its `--max-tokens`, also 16,384 by default, and a call that asks for more gets that much. The clock is the runtime's per-call timeout of ten minutes. A call that hits it fails with "Request was aborted".

Set both lower for a local model, because a reply that goes in circles costs the whole cap and the whole clock:

```ts
setLlmOptions({ maxTokens: 8192, timeout: 300000 })
```

`timeout` is in milliseconds. 8,192 tokens is room enough for any honest reply, a small thinking model's working included; a reply that needs more has almost always looped.

### A reply you gave up on keeps running, unless something stops it

When a call times out, or you press Ctrl-C, your program moves on. The model does not know that. llama.cpp runs inside your process, so Agency stops it directly. The MLX server is another process behind a socket, and `mlx_lm.server` on its own only looks at that socket once the reply is finished. Agency's chat server looks every half second instead, and drops the reply at its next token. A prompt still being read is read to the end first, and on the path a draft model uses that means the whole prompt. The server's log shows it:

```
The client went away; stopping its reply.
```

An abandoned reply that nothing stops runs to `maxTokens`, and every other request shares the GPU with it and runs two to three times slower. If a local model gets slower as a run goes on, check for this first.

### Memory is the weights, plus everything the server remembers

A model's size on disk is the floor of what it needs, not the total. Each reply being generated holds attention state that grows with the prompt and the reply. On a 235B model that is about 190KB per token, so a 22,000-token prompt adds 4GB. The MLX server also keeps that state for the last ten prompts it saw, so a follow-up on the same conversation skips re-reading it. And the GPU keeps memory it has freed around for reuse.

None of this shrinks on its own. A long run on a large model grows until the GPU runs out of memory. The server then stops answering, and every later call waits out its timeout. The system log records the moment:

```
Execution of the command buffer was aborted due to an error during execution. Insufficient Memory
```

Agency limits the server to a sixteenth of the machine's memory for attention state, and the server drops its oldest prompts to stay under. Watch the `Prompt Cache` lines the server prints to see how much it holds.

The GPU does not get the whole machine. macOS gives it a working set below the total, and keeps the rest for everything else: on a 256GB Mac the GPU's share is about 223GB, so a 132GB model leaves about 90GB of that share for attention state and the prompt cache. The share can be raised, which takes memory from the rest of the machine until the next reboot:

```bash
sudo sysctl iogpu.wired_limit_mb=240000
```

Do this only when a model almost fits and nothing else on the machine matters while it runs.

### Which model fits the machine

A model's name says how its weights are stored. `4bit` on an MLX model and `Q4_K_M` on a GGUF one both mean about four bits per weight instead of sixteen, which is a quarter of the size for a small loss in quality; `8bit` and `Q8_0` are half the size for almost none. The catalog picks 4-bit builds, and at 4 bits a model takes about 0.6GB per billion parameters: 27B is 16GB, 31B is 18GB, 235B is 132GB. The weights should take at most about half the machine's memory, so the context, the prompt cache, and the rest of the machine have the other half. A mixture-of-experts model (the `A3B` in `qwen3.5-35b-a3b`) needs memory for all of its parameters and runs at the speed of its active ones.

| Mac    | Fits comfortably                                       |
| ------ | ------------------------------------------------------ |
| 24GB   | `qwen3.5-9b` (6GB); `gpt-oss-20b` (12GB) with little room |
| 36GB   | `qwen3.5-27b` (16GB), `gemma-4-26b-a4b` (15GB)          |
| 64GB   | `gemma-4-31b` (18GB), `qwen3.5-35b-a3b` (20GB)          |
| 256GB  | `qwen3-235b-a22b-2507` (132GB)                          |

`agency local alias list` shows every catalog model's size, `agency local list` the size of each one you have downloaded, and `agency local serve` warns when the models it is starting come near the machine's memory.

### Prompts cost time up front, replies cost time per token

A reply takes two kinds of time. Reading the prompt is one batch of work that grows with the prompt's length. A 22,000-token prompt took 28 seconds on a 235B model before the first token came out. Writing the reply then costs a fixed time per token. That time is set by how fast the machine can read the model's weights, not by how much arithmetic it does. That is why a 235B model with 22B active parameters and a dense 31B model both write at about 40 tokens per second on the same Mac.

Two things follow. A long prompt is expensive even when the reply is one word, so keep the long prompts to the calls that need them. And the biggest speed-up for a thinking model is not a faster machine, it is `thinking: { enabled: false }` on the calls that do not need it.

The MLX server also keeps the attention state of its recent prompts and matches a new prompt against them by their longest shared start. A call whose prompt begins the same way as a recent one skips reading that beginning. So put the part that stays the same first, the system prompt and the document, and the part that changes last, the question. Ten questions about one 22,000-token document then pay the 28 seconds once. A conversation gets this for free, because each turn starts with all the turns before it.

### Speculative decoding

Since writing a reply is bound by reading the weights, a smaller model can help a larger one. The small model, the draft, guesses the next few tokens cheaply. The large model then checks the whole guess in one pass, which costs it about the same as writing one token, and keeps every token it agrees with. The reply is exactly what the large model would have written alone, only sooner. On prose the gain is usually 1.5x to 2x. On a typed reply it is less, because the grammar makes the draft's guesses wrong more often.

```bash
agency local serve mlx:mlx-community/Qwen3-235B-A22B-Instruct-2507-4bit --draft mlx:mlx-community/Qwen3-0.6B-4bit
agency run --local qwen3.5-4b --draft qwen3.5-2b hello.agency
```

The first line drafts for an MLX model, the second for a GGUF one. On the server, `--draft` is written after the model it drafts for, and with several models each can have its own. The draft has to share the main model's tokenizer, which in practice means the smallest member of the same family, and both backends check the pair when the draft loads and refuse one that does not match. `--draft-tokens` on the server sets how many tokens the draft guesses at a time, four by default. A draft turns off batching on the MLX server, so calls run one at a time there while it is in use. The MLX server also refuses a draft for a model whose attention cache cannot give tokens back, which is every Qwen3.5 and Qwen3-Next model; Qwen3, gpt-oss, and Gemma 4 can take one.

On llama.cpp a drafted model runs greedy unless the call names a temperature. That is the one setting every pair takes: with node-llama-cpp 3.21.1, a Qwen3.5 model that samples on a draft never returns from the call, so the plugin refuses a sampled call on a drafted Qwen3.5 model, and warns once for other families, where a Qwen3 0.6B drafting for the Qwen3 8B returned as usual. `llamaCppDraftOptions.allowSampling` in the call's metadata turns that check off. The same Qwen3 pair reported no predictions used at any temperature, and ran slower than the 8B alone, so on llama.cpp a draft is not a speed-up today. The draft on the MLX server is the one that can pay.

Whether a draft pays off depends on the pair and the machine, so measure it: run the throughput case of the benchmark with and without the draft and compare the output speed. A draft that is too large gains little, because checking its guesses costs almost what it saves. On the 235B with a 0.6B draft, prose came out slower with the draft than without.

### Long prompts and the machine's memory

The MLX server reads a long prompt in chunks. A bigger chunk keeps the GPU busier, so the prompt is read sooner, but the attention scores of one chunk against the whole prompt have to fit in memory at once. Agency sizes the chunk from the machine's memory: 2048 tokens under 64GB, 4096 up to 128GB, and 8192 above. `--prefill-step` on `agency local serve` overrides that. Raise it if you have memory to spare and long prompts to read; lower it if the server runs out of memory on a long prompt.

### Several calls at once

llama.cpp runs one reply at a time inside your process, and other calls wait their turn. The MLX server batches replies: between one token and the next it looks for new requests, and a request that arrived joins the batch at the next token, so a call never waits for another to finish. Calls that overlap share the GPU and finish sooner together than one after another. Each call still takes about as long as it would alone, so this helps a program with independent calls, not a single slow one. A server with a draft model is the exception; it takes one request at a time. Make the calls concurrent the way you would any Agency work, with [`fork` or `parallel`](/guide/concurrency).

Time to first token is not measured separately by Agency yet. A hosted model's latency includes the network round trip and the provider's queue, and a local model's does not, so a comparison of latencies alone flatters the local model on short replies.

### A type shapes the reply, but cannot make it right

A typed `llm()` call on a local model gets a reply that fits the type, because a grammar forbids every token that would break it. On a thinking model the grammar waits: the model thinks freely inside its thinking block, and only what comes after it has to fit the type. Two things are worth knowing. A tool call is not JSON, so the grammar cannot be on while the model may still call a tool. On the MLX server, Agency makes a typed call that passes tools as two requests: the model first calls its tools and answers, then a second request asks for that answer in the type, with the grammar on. llama.cpp does the same from `smoltalk-llama-cpp` 0.8.0, and an older plugin leaves such a call unconstrained. A streamed call is never split, so it is not constrained when it passes tools. And the grammar shapes the reply without judging it. Asked for a date as a string, a model can write `March 14, 2026` where you wanted `2026-03-14`, and the grammar is satisfied. Say the format you want in the prompt or in the field's description.

### The context window is smaller than you think

The prompt, the thinking, and the answer all share one context window. llama.cpp gives a model 32,768 tokens of it in Agency. A prompt near that size leaves no room for a reply, and the call fails with a context error. The MLX server uses the model's own limit, which is usually larger.

### llama.cpp and the MLX server, side by side

|                        | llama.cpp                    | MLX server                             |
| ---------------------- | ---------------------------- | -------------------------------------- |
| Runs                   | inside your process          | as a process you start and stop        |
| Loads the model        | on the first call, every run | once, when you start the server        |
| Calls at the same time | one at a time, the rest wait | several, batched together              |
| Abandoned reply        | stopped at once              | stopped at its next token              |
| Thinking on, off, budget | yes, where the model's wrapper has a switch | yes                     |
| Watches for loops      | no                           | yes                                    |
| Speculative decoding   | `agency run --draft`, greedy only | `agency local serve --draft`      |
| Output cap             | 16,384 unless the call says  | the server's `--max-tokens`            |
| Context window         | 32,768 tokens                | the model's own                        |
| Where it runs          | any machine                  | Apple Silicon only                     |

## See also

- [`agency local` command reference](/cli/local) covers every subcommand and config option.
- [LLM calls](/guide/llm) covers the `llm` function itself.
- [Custom providers](/guide/custom-providers) covers plugging in providers beyond the built-ins.