# 🧪 Testing Checklist - pi-model-router Fork

## Pre-requisites

- [ ] **Pi updated** to 0.67+ (for `after_provider_response` event support)
- [ ] **Node.js** 18+ available
- [ ] **Ollama plus pi-model-discovery are required only for local fallback/delegation checks**

## Installation Steps

```bash
# Install from github
pi install git:github.com/kylebrodeur/pi-model-router@main

# Start Pi session
pi

# Scaffold config
/router init
/router reload
```

## Test 1: Basic Router Functionality (Upstream Baseline)

**Goal:** Verify upstream routing still works with our changes.

```bash
# Start Pi with router extension
pi
```

- [ ] Pi starts without errors
- [ ] Console shows `[router] Feature sync complete`
- [ ] Status bar shows router info

```
# In Pi TUI
/router status
```

- [ ] Shows current profile, tier, model
- [ ] Shows `Router enabled: true`
- [ ] `Features:` line includes `rateLimit=` and does not include the removed Ollama sync state

### Test Routing Decision

```
# Switch to router profile
/router profile balanced
```

- [ ] Model switches to `router/balanced`
- [ ] Status bar updates

```
# Ask a simple question (should route to low tier)
What is 2+2?
```

- [ ] Request uses low-tier model

```
# Ask a complex design question (should route to high tier)
Design a full-stack app architecture with React, Node, and PostgreSQL
```

- [ ] Request uses high-tier model

## Test 2: Model Discovery Delegation

**Goal:** Verify `/router ollama-sync` delegates Ollama discovery ownership to pi-model-discovery.

### Check 1: Without pi-model-discovery installed

```
# In Pi TUI
/router ollama-sync
```

**Expected:**

- [ ] Notification: `Install pi-model-discovery for Ollama model discovery: pi install npm:@kylebrodeur/pi-model-discovery. Then run /providers sync.`
- [ ] Router does not access `models.json`
- [ ] Router does not execute `ollama`

### Check 2: With pi-model-discovery installed

```bash
pi install npm:@kylebrodeur/pi-model-discovery
```

```
# In Pi TUI
/router ollama-sync
```

**Expected:**

- [ ] First notification: `Delegating Ollama model discovery to /providers sync.`
- [ ] Provider extension handles `/providers sync`

## Test 3: Rate Limit Fallback

**Goal:** Verify rate limit detection and manual fallback.

### Step 1: Verify Rate Limit Monitoring

**Note:** Real rate limits are hard to trigger intentionally. This requires actual API usage.

```
# In Pi TUI, ensure tracking is active
/router status
```

**Expected:**

- [ ] Status output includes rate limit or fallback info if relevant.

### Step 2: Manual Fallback

```
# Switch to a cloud model first
/model anthropic/claude-sonnet-4

# Then trigger manual fallback
/router fallback
```

**Expected:**

- [ ] Switches to fallback model (best available matching sequence)
- [ ] Status bar shows `🏠 fallback`
- [ ] Console log shows fallback model name

### Step 3: Restore

```
/router restore
```

**Expected:**

- [ ] Restores to original cloud model
- [ ] Status bar clears `🏠 fallback`

### Step 4: Feature Disabled

```bash
echo '{ "features": { "rateLimitFallback": false } }' > ~/.pi/agent/model-router.json
```

```
/reload
```

**Expected:**

- [ ] Console shows `[router] rate-limit-fallback: disabled`
- [ ] Status bar still shows router info but no fallback indicator

## Test 4: Feature Toggles (Config Merging)

### Step 1: User-Level Config

```bash
cat > ~/.pi/agent/model-router.json << 'EOF'
{
  "features": {
    "rateLimitFallback": true,
    "perTurnRouting": true
  }
}
EOF
```

### Step 2: Project-Level Override

```bash
mkdir -p .pi
cat > .pi/model-router.json << 'EOF'
{
  "features": {
    "rateLimitFallback": false
  }
}
EOF
```

### Step 3: Verify

```
# In Pi TUI from project directory
/router
```

**Expected:**

- [ ] Console shows `[router] rate-limit-fallback: disabled`
- [ ] Per-turn routing still works (project didn't override it)

## Test 5: Edge Cases

### Missing Model Discovery Extension

```
/router ollama-sync
```

**Expected:**

- [ ] Notification: `Install pi-model-discovery for Ollama model discovery: pi install npm:@kylebrodeur/pi-model-discovery. Then run /providers sync.`
- [ ] No crash, graceful guidance

### No Fallback Models Configured

```
/router fallback
```

**Expected:**

- [ ] `No fallback models available` or equivalent error notification
- [ ] Router remains on the current model

## Test 6: Combined Features (Full Workflow)

**Goal:** Verify router features work together.

```
# Pi session
/new
```

**Expected:**

1. [ ] Session starts
2. [ ] `/router status` shows no removed Ollama sync feature state
3. [ ] `/router config` includes `rateLimitFallback` and does not list the removed Ollama sync feature
4. [ ] `/router config ollama-sync` reports `Unknown feature: ollama-sync. Try: rate-limit, classifier, budget, respect-pi-scope`
5. [ ] `/router profile balanced` switches to router
6. [ ] Simple query routes to low tier
7. [ ] Complex query routes to high tier (cloud)

## Regression Tests (Upstream Feature Checklist)

- [ ] `/router` shows status
- [ ] `/router profile <name>` switches profiles
- [ ] `/router pin high` pins tier
- [ ] `/router pin auto` releases pin
- [ ] `/router disable` disables router
- [ ] `/router debug toggle` enables debug
- [ ] `/router widget toggle` toggles widget
- [ ] `/router reload` reloads config
- [ ] Routing decisions shown in UI (if debug enabled)
- [ ] Cost tracking works (if budgeting enabled)
- [ ] Cost budget forces downgrade when exceeded
- [ ] Phase memory works across turns
- [ ] Custom rules match and override tiers
- [ ] Context trigger forces high tier on large contexts

## Troubleshooting

### Extension Not Loading

```bash
# Check file paths
ls -la ~/.pi/agent/extensions/pi-model-router/
# Should see: index.ts, config.ts, types.ts, routing.ts, provider.ts, commands.ts, state.ts, ui.ts, constants.ts, model-discovery.ts, rate-limit.ts
```

### TypeScript Errors After Update

```bash
cd ~/.pi/agent/extensions/pi-model-router
npm install  # ensure dependencies match
./node_modules/.bin/tsc --noEmit
```

### Config Not Applying

```bash
# Verify files exist
cat ~/.pi/agent/model-router.json
cat .pi/model-router.json 2>/dev/null || echo "No project config"
```

### Rate Limit Not Detected

```
# Requires Pi 0.67+ — check version
pi --version

# If outdated:
npm install -g @mariozechner/pi-coding-agent
```

### Fallback Model Not In Registry

```
/providers status
/providers sync
/model  # Check if local models appear
```

If Ollama models don't appear after `/providers sync`:

1. Verify pi-model-discovery is installed: `/providers status`
2. Check Ollama is running: `ollama list` in terminal
3. Pull at least one local model for fallback use