# Provider Routing & Custom Models — Implementation Plan

## 1. Problem

The system has 3 model roles (`main`, `compact`, `title`), each with one `{base_url, api_key, model}` tuple. This means:
- All models must go through the same API endpoint (e.g. OpenRouter)
- No way to route specific models to their native APIs (Anthropic direct, Google AI, etc.)
- No fallback if a provider goes down
- No custom model metadata without editing the builtin `config/models.json`

## 2. Target Settings Structure

```json
{
  "providers": {
    "openrouter": {
      "type": "openai",
      "base_url": "https://openrouter.ai/api/v1",
      "api_key": "sk-or-..."
    },
    "anthropic": {
      "type": "openai",
      "base_url": "https://api.anthropic.com/v1",
      "api_key": "sk-ant-..."
    }
  },
  "routing": {
    "default": "openrouter",
    "fallback": ["anthropic"],
    "per_model": {
      "anthropic/claude-sonnet-4.6": {
        "default": "anthropic",
        "fallback": ["openrouter"]
      }
    },
    "per_agent": {
      "planner": {
        "default": "anthropic",
        "fallback": ["openrouter"]
      }
    }
  },
  "models": { ... }
}
```

**Why `routing` wrapper instead of flat keys:** Avoids `"default"` as a top-level settings key (JS reserved word, collides semantically with other settings), and groups routing concerns under one namespace.

**Provider resolution order**: `per_agent` > `per_model` > `routing.default` > legacy fallback

**`type` field**: Only `"openai"` supported now. Extension point for future engines (native Anthropic, Vertex, Ollama, etc.).

## 3. Custom Models File Hierarchy

```
config/models.json                 (builtin, read-only, from OpenRouter refresh)
~/.veil/custom_models.json       (global user overrides/additions)
.veil/custom_models.json         (project-level overrides/additions)
```

Merge order: builtin -> global -> project (project wins on conflict).

File format:
```json
{
  "models": {
    "my-company/internal-llm": {
      "name": "Internal LLM",
      "context_length": 128000,
      "max_completion_tokens": 4096,
      "pricing": { "prompt": 0.001, "completion": 0.002, "cache_read": 0.0001 }
    }
  }
}
```

---

## 4. Implementation Phases

### PHASE 1: Custom Models (no breaking changes)

#### 1.1 — Path helpers

**File:** `utils/paths.js`

Add two new functions + exports:
```js
function getGlobalCustomModelsPath() {
  return path.join(getGlobalConfigDir(), 'custom_models.json');
}
function getProjectCustomModelsPath(cwd) {
  return path.join(getProjectConfigDir(cwd), 'custom_models.json');
}
```

#### 1.2 — Separate builtin and custom model loading

**File:** `utils/models.js`

**Problem identified by review:** The current `_cache` is a single global. Making `loadModels(cwd)` cwd-dependent would break the cache (different projects = different merged results). Additionally, `getModel()`, `getContextLimit()`, `calculateCost()` are called from 9+ locations across the codebase — threading `cwd` to all of them is invasive and fragile.

**Solution:** Two-tier loading. Keep the existing `loadModels()` as-is for builtin models (cached, no cwd). Add a separate `loadCustomModels(cwd)` and a `getModelMerged(modelId, cwd)` that merges on demand:

```js
let _builtinCache = null;  // rename from _cache
let _customCache = { cwd: null, data: null };  // cwd-keyed

function loadModels() {
  // Unchanged — loads only config/models.json, globally cached
  if (_builtinCache) return _builtinCache;
  // ... existing logic ...
}

function loadCustomModels(cwd) {
  if (_customCache.cwd === cwd && _customCache.data) return _customCache.data;
  const result = {};
  const globalCustom = readJsonSafe(paths.getGlobalCustomModelsPath());
  if (globalCustom?.models) Object.assign(result, globalCustom.models);
  if (cwd) {
    const projectCustom = readJsonSafe(paths.getProjectCustomModelsPath(cwd));
    if (projectCustom?.models) Object.assign(result, projectCustom.models);
  }
  _customCache = { cwd, data: result };
  return result;
}

// New: merged lookup (use when cwd is available)
function getModelMerged(modelId, cwd) {
  const builtin = getModel(modelId);  // existing function, no cwd needed
  if (builtin) return builtin;
  if (!cwd) return null;
  const custom = loadCustomModels(cwd);
  return custom[modelId] || null;
}

// New: context limit with custom model support
function getContextLimitMerged(modelId, cwd) {
  const m = getModelMerged(modelId, cwd);
  return m ? (m.context_length || null) : null;
}

// New: cost calculation with custom model support
function calculateCostMerged(modelId, cwd, tokens) {
  const m = getModelMerged(modelId, cwd);
  if (!m) return 0;
  // ... same pricing logic as calculateCost ...
}

// New: full merged list for API responses
function listModelsMerged(cwd) {
  const { updated_at, models } = loadModels();
  const custom = cwd ? loadCustomModels(cwd) : {};
  const merged = { ...models, ...custom };
  return {
    updated_at,
    models: Object.entries(merged).map(([id, data]) => ({
      id, ...data, source: custom[id] ? 'custom' : 'builtin',
    })),
  };
}
```

**Existing functions stay unchanged** (`loadModels`, `getModel`, `getContextLimit`, `calculateCost`, `listModels`) — zero impact on existing callers.

**Callers that should use the merged versions (opt-in):**
- `infrastructure/database.js:162` — `getContextLimit(model)` → switch to `getContextLimitMerged(model, instanceFolder)` (instanceFolder is already available as a param)
- `api/routes/models.js` — `listModels()` → switch to `listModelsMerged(req.app.locals.cwd)`
- `core/loop.js:367` — `calculateCost(modelKey, ...)` → switch to `calculateCostMerged(modelKey, cwd, ...)`
- `api/routes/completions.js:88,109` — `calculateCost(resolvedModel, ...)` → switch to `calculateCostMerged(resolvedModel, req.app.locals.cwd, ...)`

**Invalidation:** `invalidateCache()` resets both `_builtinCache` and `_customCache`.

#### 1.3 — Custom models API endpoints

**File:** `api/routes/models.js`

Add:
- `GET /models/custom?level=global|project` — read raw custom_models.json for a level
- `PUT /models/custom?level=global|project` — validate + write, invalidate cache

Existing `GET /models` switches to `listModelsMerged(cwd)` so custom models appear with `source: 'custom'` flag.

#### 1.4 — Custom models schema

**File:** `schemas/custom_models.json` (new)

```json
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "VeilCLI Custom Models",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "models": {
      "type": "object",
      "additionalProperties": {
        "type": "object",
        "properties": {
          "name": { "type": "string" },
          "context_length": { "type": "integer", "minimum": 1 },
          "max_completion_tokens": { "type": "integer", "minimum": 1 },
          "pricing": {
            "type": "object",
            "properties": {
              "prompt": { "type": "number" },
              "completion": { "type": "number" },
              "cache_read": { "type": "number" },
              "cache_write": { "type": "number" }
            }
          }
        }
      }
    }
  }
}
```

---

### PHASE 2: Provider Registry

#### 2.1 — Field constants

**File:** `settings/fields.js`

Add:
```js
// Provider routing
PROVIDERS: 'providers',
ROUTING: 'routing',
ROUTING_DEFAULT: 'default',
ROUTING_FALLBACK: 'fallback',
ROUTING_PER_MODEL: 'per_model',
ROUTING_PER_AGENT: 'per_agent',
PROVIDER_TYPE: 'type',
PROVIDER_BASE_URL: 'base_url',
PROVIDER_API_KEY: 'api_key',
```

#### 2.2 — Settings schema update

**File:** `schemas/settings.json`

Add to the `"properties"` object (these are explicitly listed, keeping `"additionalProperties": false`):

```json
"providers": {
  "type": "object",
  "additionalProperties": { "$ref": "#/definitions/providerConfig" }
},
"routing": {
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "default": { "type": ["string", "null"] },
    "fallback": { "type": "array", "items": { "type": "string" } },
    "per_model": {
      "type": "object",
      "additionalProperties": { "$ref": "#/definitions/routingRule" }
    },
    "per_agent": {
      "type": "object",
      "additionalProperties": { "$ref": "#/definitions/routingRule" }
    }
  }
}
```

New definitions:
```json
"providerConfig": {
  "type": "object",
  "required": ["type", "base_url"],
  "additionalProperties": false,
  "properties": {
    "type": { "type": "string", "enum": ["openai"] },
    "base_url": { "type": "string" },
    "api_key": { "type": "string" }
  }
},
"routingRule": {
  "type": "object",
  "required": ["default"],
  "additionalProperties": false,
  "properties": {
    "default": { "type": "string" },
    "fallback": { "type": "array", "items": { "type": "string" } }
  }
}
```

#### 2.3 — Settings defaults + auth merging

**File:** `utils/settings.js`

In `getDefaults()`, add:
```js
providers: {},
routing: {
  default: null,
  fallback: [],
  per_model: {},
  per_agent: {},
},
```

In `loadSettings()`, extend auth.json merging (both global AND project auth):
```js
// After existing: if (globalAuth.models) deepMerge(result.models, globalAuth.models);
if (globalAuth.providers) {
  if (!result.providers) result.providers = {};
  deepMerge(result.providers, globalAuth.providers);
}

// ... (same block added after project auth.json loading) ...
if (projectAuth.providers) {
  if (!result.providers) result.providers = {};
  deepMerge(result.providers, projectAuth.providers);
}
```

**Merge order for providers:** Auth layers always win for `api_key` values because they are loaded AFTER the corresponding settings layer and deepMerge gives source-wins behavior. Recommended pattern: put `type` + `base_url` in `settings.json`, put `api_key` in `auth.json` (gitignored).

---

### PHASE 3: Provider Resolution Engine + Error Infrastructure

#### 3.0 — Add `.status` to LLM errors (prerequisite)

**File:** `llm/client.js`

The `isRetriableError()` function needs to check HTTP status codes. Currently `callLLM` throws plain `Error` with the status embedded in the message string. Fix this:

**Lines 66-69 (in `callLLM`)** and **lines 88-91 (in `callLLMStreaming`)**:

Before:
```js
if (!response.ok) {
  let errText = '';
  try { errText = await response.text(); } catch {}
  throw new Error(`LLM API error ${response.status}: ${errText.slice(0, 500)}`);
}
```

After:
```js
if (!response.ok) {
  let errText = '';
  try { errText = await response.text(); } catch {}
  const err = new Error(`LLM API error ${response.status}: ${errText.slice(0, 500)}`);
  err.status = response.status;
  throw err;
}
```

Same change in both `callLLM` (line ~69) and `callLLMStreaming` (line ~91).

#### 3.1 — Create `llm/provider.js` (new file)

Three exported functions:

**`resolveProviderChain(settings, modelId, agentName, role)`**

Returns `{ providers: [{ name, type, baseUrl, apiKey }], resolved: string }`

`role` parameter (optional, default `'main'`) handles the legacy hybrid case — when no providers are configured, the function uses `getModelConfig(settings, role)` to get the right base_url/api_key for the specific model role (main/compact/title).

Resolution logic:
```
1. hasProviders = settings.providers && Object.keys(settings.providers).length > 0

2. if (!hasProviders) → LEGACY MODE:
   - Use getModelConfig(settings, role) to get {base_url, api_key}
   - Return [{ name: 'legacy', type: 'openai', baseUrl, apiKey }]
   - resolved: 'legacy'

3. if (agentName && settings.routing.per_agent[agentName])
   → Build chain from that routing rule. resolved: 'per_agent'

4. else if (modelId && settings.routing.per_model[modelId])
   → Build chain from that routing rule. resolved: 'per_model'

5. else if (settings.routing.default)
   → Build chain: { default: settings.routing.default, fallback: settings.routing.fallback || [] }
   resolved: 'default'

6. else → LEGACY FALLBACK (providers exist but no routing configured)
   - Same as step 2. resolved: 'legacy'
```

For each provider name in the chain, look up `settings.providers[name]`. Validate it exists — throw clear error if not:
```
`Provider "${name}" referenced in ${resolved} routing for "${modelId || agentName}" but not defined in settings.providers`
```

**`isRetriableError(error)`**

```js
function isRetriableError(err) {
  // HTTP status-based (via err.status attached in callLLM)
  if (err.status) {
    return [429, 502, 503, 504].includes(err.status);
  }
  // Network errors (no HTTP response received)
  const msg = (err.message || '').toLowerCase();
  return ['econnrefused', 'etimedout', 'fetch failed', 'socket hang up',
          'enotfound', 'network', 'econnreset'].some(s => msg.includes(s));
}
```

**`callWithProviderFallback({ settings, modelId, agentName, role, ...llmParams })`**

```js
async function callWithProviderFallback({ settings, modelId, agentName, role = 'main', ...llmParams }) {
  const { providers, resolved } = resolveProviderChain(settings, modelId, agentName, role);
  const isStreaming = !!(llmParams.onChunk || llmParams.onToolStart);

  let lastError = null;
  for (let i = 0; i < providers.length; i++) {
    const provider = providers[i];
    if (provider.type !== 'openai') {
      throw new Error(`Provider "${provider.name}": type "${provider.type}" not supported. Only "openai" is currently supported.`);
    }

    // STREAMING SAFETY: Only attempt fallback if we haven't started streaming yet.
    // If streaming has begun (first provider partially succeeded), we can't safely
    // switch providers because partial data was already sent to the client.
    if (i > 0 && isStreaming) {
      throw lastError;  // Don't attempt fallback mid-stream
    }

    try {
      const response = await callLLM({
        baseUrl: provider.baseUrl,
        apiKey: provider.apiKey,
        model: modelId,
        ...llmParams,
      });
      return { response, providerUsed: provider.name, resolved };
    } catch (err) {
      lastError = err;
      if (!isRetriableError(err)) throw err;  // Non-retriable → fail immediately
      console.warn(`[provider] ${provider.name} failed (${err.status || 'network'}), trying next...`);
    }
  }
  throw lastError;
}
```

**Key design: streaming + fallback safety.** When `onChunk` is provided, fallback is only attempted if the **first** provider fails before streaming starts (e.g., connection refused, 503 before any data). Once streaming begins and partial content has been sent to the client, we cannot switch to another provider — the client would see garbled/duplicated output. In practice, most retriable errors (429, 502, 503) happen before any data is streamed, so fallback still works for the common case. The `callLLM` function in `client.js` throws before starting to read the stream body if `!response.ok`, so HTTP errors are caught pre-stream.

---

### PHASE 4: Migrate All 5 LLM Call Sites

Each call site changes from directly calling `callLLM()` with hardcoded `modelConfig[F.MODEL_BASE_URL]` / `modelConfig[F.MODEL_API_KEY]` to calling `callWithProviderFallback()` with `settings` + `modelId` + `agentName` + `role`.

#### 4.1 — `core/loop.js` (main agent loop)

**Lines 179, 331-343**

```js
// Import at top
const { callWithProviderFallback } = require('../llm/provider');

// Line 179: keep getModelConfig for model name resolution
const modelConfig = getModelConfig(settings, F.MODEL_MAIN);
const modelId = agent.model || modelConfig[F.MODEL_NAME];

// Lines 331-343: replace callLLM
const { response, providerUsed } = await callWithProviderFallback({
  settings,
  modelId,
  agentName: agent.name,
  role: F.MODEL_MAIN,
  messages, tools: llmTools,
  temperature: agent.temperature,
  reasoning: agent.reasoning,
  maxTokens: agent.maxTokens,
  thinking: thinking || agent.thinking || null,
  onChunk: onStreamChunk || null,
  onToolStart: onInferenceToolStart || null,
});
```

Existing retry logic (3 consecutive errors, lines 344-358) stays unchanged. Fallback happens **inside** each attempt.

Add `providerUsed` to emitted events for observability.

#### 4.2 — `core/compaction.js` (memory extraction, line 85)

```js
const { callWithProviderFallback } = require('../llm/provider');

// In extractMemoriesBeforeCompaction:
const modelConfig = getModelConfig(settings, F.MODEL_MAIN);
const modelId = modelConfig[F.MODEL_NAME];
if (!modelConfig[F.MODEL_API_KEY] || !modelId) return;  // guard stays

const { response } = await callWithProviderFallback({
  settings, modelId, agentName,
  role: F.MODEL_MAIN,
  messages: extractionMessages, tools: [],
});
```

`agentName` is already a parameter of `extractMemoriesBeforeCompaction()`.

#### 4.3 — `core/compaction.js` (context compaction, line 123)

```js
// In compactMessages — add agentName parameter:
async function compactMessages({ messages, settings, taskBrief, agentName }) {
  const compactConfig = getModelConfig(settings, F.MODEL_COMPACT);
  const modelId = compactConfig[F.MODEL_NAME];
  if (!compactConfig[F.MODEL_API_KEY] || !modelId) {
    // ... existing trim fallback ...
  }

  const { response } = await callWithProviderFallback({
    settings, modelId, agentName: agentName || null,
    role: F.MODEL_COMPACT,
    messages: compactionMessages, tools: [],
  });
  // ... rest unchanged ...
}
```

**Threading `agentName`:** `manageContext()` already receives `agentName` → pass it to `compactMessages()`. Only change is adding `agentName` to `compactMessages`'s destructured params and to the call in `manageContext()` at line 172:
```js
const compacted = await compactMessages({ messages, settings, taskBrief, agentName });
```

#### 4.4 — `core/default-compaction.js` (session compaction, line 166)

```js
const { callWithProviderFallback } = require('../llm/provider');

// Lines 150-177:
const mainConfig = getModelConfig(settings, F.MODEL_MAIN);
const modelId = (session.compact_model && session.compact_model !== 'default')
  ? session.compact_model
  : (session.model || mainConfig[F.MODEL_NAME]);

const { response } = await callWithProviderFallback({
  settings, modelId, agentName: session.agent_name,
  role: F.MODEL_MAIN,
  messages: compactMessages, tools: [], reasoning: 'low',
});
```

No synthetic `modelConfig` object needed anymore — provider routing handles base_url/api_key resolution.

#### 4.5 — `api/routes/completions.js` (lines 54-66, 81, 105)

```js
const { callWithProviderFallback } = require('../../llm/provider');

// Remove baseUrl/apiKey from llmParams construction:
const modelConfig = getModelConfig(settings, F.MODEL_MAIN);
const resolvedModel = model || modelConfig[F.MODEL_NAME];

const llmParams = {
  // NO baseUrl, NO apiKey here anymore
  messages,
  tools: Array.isArray(tools) ? tools : [],
  temperature: temperature !== undefined ? temperature : undefined,
  maxTokens: max_tokens !== undefined ? max_tokens : undefined,
  reasoning: reasoning !== undefined ? reasoning : undefined,
  thinking: thinking !== undefined ? thinking : undefined,
  modalities: modalities !== undefined ? modalities : undefined,
  audio: audio !== undefined ? audio : undefined,
};

// SSE mode:
const { response } = await callWithProviderFallback({
  settings, modelId: resolvedModel, agentName: null,
  role: F.MODEL_MAIN,
  ...llmParams,
  onChunk: (text) => sendEvent('chunk', { content: text }),
});

// JSON mode:
const { response } = await callWithProviderFallback({
  settings, modelId: resolvedModel, agentName: null,
  role: F.MODEL_MAIN,
  ...llmParams,
});
```

---

### PHASE 5: Observability & API

#### 5.1 — Debug logging in provider.js

Add `console.warn` / `console.log` calls with `[provider]` prefix:
- On resolution: `[provider] Resolved ${resolved} for model=${modelId} agent=${agentName} → ${providerName}`
- On fallback: `[provider] ${providerName} failed (${status}), trying next...`
- On final failure: `[provider] All providers exhausted for model=${modelId}`

#### 5.2 — Provider info in loop events

**File:** `core/loop.js`

Include `providerUsed` in the `message` event yield and bus events.

#### 5.3 — Providers API endpoint

**File:** `api/routes/providers.js` (new)

- `GET /providers` — list configured providers with redacted API keys, plus routing rules
- Returns: `{ providers: {...}, routing: {...} }`

Register in `api/index.js`.

---

## 5. Files Changed Summary

### New files:
| File | Purpose |
|------|---------|
| `llm/provider.js` | Provider resolution, fallback wrapper, error classification |
| `schemas/custom_models.json` | Validation schema for custom_models.json |
| `api/routes/providers.js` | GET /providers API endpoint |

### Modified files:
| File | Change |
|------|--------|
| `settings/fields.js` | Add provider/routing constants |
| `schemas/settings.json` | Add providers + routing properties and definitions |
| `utils/paths.js` | Add `getGlobalCustomModelsPath()`, `getProjectCustomModelsPath(cwd)` |
| `utils/settings.js` | Add routing to defaults, merge providers from auth.json (global + project) |
| `utils/models.js` | Add `loadCustomModels(cwd)`, `getModelMerged()`, `getContextLimitMerged()`, `calculateCostMerged()`, `listModelsMerged()` — existing functions untouched |
| `llm/client.js` | Add `err.status = response.status` to thrown errors (2 locations) |
| `core/loop.js` | Use `callWithProviderFallback()`, use `calculateCostMerged()`, emit `providerUsed` |
| `core/compaction.js` | Use `callWithProviderFallback()` at 2 call sites, thread `agentName` to `compactMessages()` |
| `core/default-compaction.js` | Use `callWithProviderFallback()`, remove synthetic modelConfig |
| `api/routes/completions.js` | Use `callWithProviderFallback()`, remove baseUrl/apiKey from llmParams, use `calculateCostMerged()` |
| `api/routes/models.js` | Add custom model endpoints, use `listModelsMerged(cwd)` |
| `api/index.js` | Register providers route |
| `infrastructure/database.js` | Use `getContextLimitMerged(model, instanceFolder)` at line 162 |

### NOT modified:
| File | Reason |
|------|--------|
| `schemas/agent.json` | Agent model field stays a string |
| Database schema | Sessions store model string; provider resolved at call time |

---

## 6. Backward Compatibility

- When `settings.providers` is empty/absent, `resolveProviderChain()` returns legacy mode using `getModelConfig(settings, role)` — behavior identical to today
- Legacy mode respects model roles: compact calls use `models.compact.base_url/api_key`, not `models.main`
- Existing `models.main/compact/title` configs continue to work
- `getModelConfig()` remains exported and functional
- Old functions (`getModel`, `getContextLimit`, `calculateCost`, `listModels`) stay unchanged — zero impact on any code that doesn't opt into custom models
- No database migrations needed
- No agent schema changes needed
- Auth.json format backward compatible (new `providers` key is additive)

---

## 7. Verification Plan

### Unit tests:
1. `resolveProviderChain()` — test all resolution paths: legacy, per_agent, per_model, default, missing provider errors, legacy with different roles (main vs compact)
2. `isRetriableError()` — test `err.status` for 429/502/503/504 (retriable), 400/401/403/404 (not retriable), network error messages
3. `callWithProviderFallback()` — mock `callLLM`: single provider success, fallback on 503, non-retriable 400 fails immediately, all providers fail, streaming prevents fallback after first provider
4. `loadCustomModels(cwd)` + `getModelMerged()` — test merge order, cache invalidation, missing files

### Integration tests:
1. Configure providers in settings, run chat, verify `providerUsed` in response events
2. Configure primary provider with invalid key, verify fallback provider used
3. Run without providers, verify identical behavior to current system
4. Run with providers + `models.compact` set (hybrid), verify compact calls use legacy path correctly
5. Add custom model via `PUT /models/custom`, verify `GET /models` includes it with `source: 'custom'`
6. Create session with custom model, verify `context_size_limit` is set correctly

### Manual smoke test:
1. Start server with no providers configured → everything works as before
2. Add `providers` + `routing.default` to settings → verify chat works through provider
3. Set up per_model routing → verify the right provider handles specific models
4. Kill the default provider (invalid URL) → verify fallback kicks in
5. Test streaming: kill default provider → verify fallback works (error happens pre-stream)
6. Add custom model to `~/.veil/custom_models.json` → verify it appears in model list with correct context_length