# SpecMem Search Tools - HOW TO USE

This document provides a comprehensive guide to using SpecMem's semantic search tools: `find_code_pointers` and `find_memory`. These tools use embeddings to search by meaning rather than keywords.

---

## Table of Contents

1. [Introduction](#introduction)
2. [Query Formulation (CRITICAL)](#query-formulation-critical)
3. [find_code_pointers - Semantic Code Search](#find_code_pointers---semantic-code-search)
4. [find_memory - Semantic Memory Search](#find_memory---semantic-memory-search)
5. [Timeout Behavior](#timeout-behavior)
6. [Common Mistakes](#common-mistakes)
7. [Best Practices](#best-practices)
8. [Troubleshooting](#troubleshooting)

---

## Introduction

SpecMem provides two powerful semantic search tools:

### find_code_pointers
- **Purpose**: Search your codebase by meaning, not keywords
- **What it finds**: Functions, classes, methods, interfaces, and other code definitions
- **Unique feature**: Shows tracebacks (who imports/calls the code you find)
- **Best for**: Understanding code architecture, finding implementations, tracing dependencies

### find_memory
- **Purpose**: Search stored memories (past conversations, decisions, context)
- **What it finds**: Episodic events, semantic facts, procedural knowledge, working memories
- **Unique feature**: Supports time queries like "yesterday", "last week", specific dates
- **Best for**: Recall past discussions, find decisions, locate context from previous sessions

---

## Query Formulation (CRITICAL)

The most important factor for getting good results is how you write your query. The two tools have different requirements:

### find_code_pointers: Use CODE TERMS

**DO write queries using code-related terms:**
- `admin login auth`
- `database connection pool`
- `api handler middleware`
- `user authentication flow`
- `json response serializer`

**DO NOT write natural language questions:**
- `"how does admin login work"` - BAD
- `"where is the database connection defined"` - BAD
- `"what middleware handles authentication"` - BAD

**Why?** find_code_pointers searches for code definitions using embeddings trained on code. Natural language questions don't match the embedding space well.

**Query examples by use case:**

| Use Case | Good Query | Bad Query |
|----------|-----------|-----------|
| Find auth logic | `authentication token validation` | `how does token auth work` |
| Find DB code | `postgres connection pool` | `where do we connect to database` |
| Find API code | `rest endpoint handler` | `how are API routes defined` |
| Find utilities | `string formatting helper` | `what utility functions exist` |

### find_memory: Natural Language Works

find_memory is more flexible - natural language queries work well:
- `"discussion about API design"`
- `"database migration decisions"`
- `"user feedback on login flow"`

Be specific about what you're looking for. Vague queries return vague results.

---

## find_code_pointers - Semantic Code Search

### Basic Usage

```javascript
find_code_pointers({ query: "admin login authentication" })
find_code_pointers({ query: "database connection pool" })
```

### Parameter Reference

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `query` | string | required | What to search for - use CODE TERMS |
| `limit` | number | 10 | Maximum results to return |
| `threshold` | number | 0.25 | Minimum similarity score (0-1). Higher = more strict |
| `language` | string | - | Filter by language: typescript, javascript, python, go, rust, etc. |
| `filePattern` | string | - | Filter by file path pattern: `routes/*.ts`, `src/api/**` |
| `definitionTypes` | array | - | Filter by type: function, method, class, interface, type, enum, variable, constant, constructor |
| `includeTracebacks` | boolean | true | Show who imports/calls the found code |
| `galleryMode` | boolean | false | Enable Mini COT analysis for deeper understanding |
| `zoom` | number | 50 | Detail level: 0=signature only, 50=balanced, 100=full context |
| `cameraRollMode` | boolean | true | Returns drilldownIDs for drill_down exploration |
| `zoomLevel` | string | - | Preset zoom: ultra-wide, wide, normal, close, macro |
| `includeMemoryLinks` | boolean | true | Link code to related memories |
| `includeRecent` | number | 0 | Force-include N most recently modified files |
| `keywordFallback` | boolean | true | Fall back to keyword search if semantic returns nothing |
| `allProjects` | boolean | false | Search ALL projects instead of current |

### Zoom Level Presets

| Preset | Results | Threshold | Best For |
|--------|---------|-----------|----------|
| `ultra-wide` | 50 | 15% | Exploration, unknown codebase |
| `wide` | 25 | 25% | Broad overview |
| `normal` | 15 | 40% | Balanced (default feel) |
| `close` | 10 | 60% | Specific known functionality |
| `macro` | 5 | 80% | Exact matches only |

### Example Queries

```javascript
// Find authentication-related code
find_code_pointers({ query: "jwt token validation", limit: 10 })

// Find database code, TypeScript only
find_code_pointers({ query: "postgres query builder", language: "typescript" })

// Find with tracebacks (who calls this)
find_code_pointers({ query: "api middleware handler", includeTracebacks: true })

// Gallery mode for deeper analysis
find_code_pointers({ query: "auth flow", galleryMode: true })

// Use zoom level preset
find_code_pointers({ query: "connection pool", zoomLevel: "close" })
```

---

## find_memory - Semantic Memory Search

### Basic Usage

```javascript
find_memory({ query: "API design decisions" })
find_memory({ query: "database migration" })
```

### Parameter Reference

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `query` | string | required | What to search for - natural language works |
| `limit` | number | 10 | Maximum results to return |
| `threshold` | number | 0.25 | Minimum similarity score (0-1). Higher = more strict |
| `memoryTypes` | array | - | Filter: episodic, semantic, procedural, working, consolidated |
| `tags` | array | - | Filter by tags (OR logic) |
| `importance` | array | - | Filter: critical, high, medium, low, trivial |
| `dateRange` | object | - | Filter by date: `{ start: "2024-01-01", end: "2024-12-31" }` |
| `includeRecent` | number | 0 | Force-include N most recent memories regardless of relevance |
| `recencyBoost` | boolean | true | Boost recent memories (last hour +20%, last day +10%) |
| `keywordFallback` | boolean | true | Fall back to keyword search if semantic returns nothing |
| `role` | string | - | Filter by message role: "user" or "assistant" |
| `summarize` | boolean | true | Return summarized content (first 500 chars) |
| `maxContentLength` | number | 500 | Truncate content to this many characters |
| `galleryMode` | boolean | false | Enable Mini COT analysis |
| `cameraRollMode` | boolean | true | Returns drilldownIDs for drill_down exploration |
| `zoomLevel` | string | - | Preset zoom: ultra-wide, wide, normal, close, macro |
| `projectPath` | string | - | Search specific project path |
| `allProjects` | boolean | false | Search ALL projects |

### Time Queries

find_memory supports natural language time expressions:
- `"yesterday"`
- `"last week"`
- `"last month"`
- `"2024-01-15"`
- `"past hour"`

Use with `dateRange` parameter for precise filtering.

### Memory Types

| Type | Description |
|------|-------------|
| `episodic` | Events, things that happened |
| `semantic` | Facts, knowledge, learned information |
| `procedural` | How-to, step-by-step processes |
| `working` | Temporary/active memory |
| `consolidated` | Merged/condensed memories |

### Example Queries

```javascript
// Basic search
find_memory({ query: "API design decisions" })

// Filter by memory type
find_memory({ query: "database", memoryTypes: ["episodic", "semantic"] })

// Time-based search
find_memory({ query: "auth", dateRange: { start: "2024-01-01", end: "2024-12-31" } })

// Get recent memories regardless of relevance
find_memory({ query: "anything", includeRecent: 5 })

// Full content, no summarization
find_memory({ query: "important decision", summarize: false })

// Cross-project search
find_memory({ query: "shared utility", allProjects: true })
```

---

## Timeout Behavior

### Understanding "processing" vs "working"

**IMPORTANT**: The embedding server may send `{"status":"processing"}` responses. This status indicates the server is **alive**, NOT that it is actively processing your query.

- **"processing" status**: Server is alive and idle - does NOT mean your query is being worked on
- **Actual working**: The server is generating embeddings or searching

### Timeout Configuration

Timeouts are controlled by environment variable:
```bash
export SPECMEM_EMBEDDING_TIMEOUT=60    # seconds, default
```

For code search specifically:
```bash
export SPECMEM_CODE_SEARCH_TIMEOUT=60  # seconds, default
```

### Timeout Values by Operation

| Operation | Default Timeout | Config Variable |
|-----------|-----------------|-----------------|
| Embedding generation | 60s | SPECMEM_EMBEDDING_TIMEOUT |
| DB search | 6x embedding timeout | SPECMEM_EMBEDDING_TIMEOUT |
| Keyword fallback | 10s | - |
| Recent files lookup | 5s | - |
| Tracebacks | 30s | - |
| Gallery mode | 2x normal | - |

### Retry Logic

Both tools have built-in retry logic for transient failures:
- **find_memory**: 2 retries by default (configurable via SPECMEM_FIND_MEMORY_RETRIES)
- **find_code_pointers**: 2 retries by default

Retries handle:
- Connection timeouts
- Socket reset errors
- Resource busy errors
- Temporary service unavailability

### Invalid Query Behavior

Queries with wrong input format should timeout quickly with a hint about correct format. If you see timeout errors:

1. Check that all required parameters are provided
2. Verify parameter types are correct
3. Ensure query string is not empty

---

## Common Mistakes

### Mistake 1: Writing Natural Language Questions for Code Search

```javascript
// WRONG - natural language
find_code_pointers({ query: "how does the admin login work" })

// RIGHT - code terms
find_code_pointers({ query: "admin login authentication flow" })
```

### Mistake 2: Using Too High Threshold

```javascript
// WRONG - too strict, returns nothing
find_code_pointers({ query: "auth", threshold: 0.8 })

// RIGHT - balanced threshold
find_code_pointers({ query: "auth", threshold: 0.25 })
```

### Mistake 3: Forgetting cameraRollMode Returns IDs

When `cameraRollMode: true` (default), results include `drilldownID` values. Use these with `drill_down()` to explore:
```javascript
// First search
const results = find_code_pointers({ query: "auth", cameraRollMode: true })

// Then drill down into a specific result
drill_down({ drilldownID: results[0].drilldownID })
```

### Mistake 4: Not Using zoomLevel Presets

Instead of guessing zoom values, use presets:
```javascript
// WRONG - guessing values
find_code_pointers({ query: "auth", zoom: 75 })

// RIGHT - use preset
find_code_pointers({ query: "auth", zoomLevel: "close" })
```

### Mistake 5: Not Checking includeRecent

When searching unfamiliar codebases, include recent files:
```javascript
find_code_pointers({ query: "auth", includeRecent: 10 })
```

### Mistake 6: Not Using keywordFallback

Keyword fallback finds code that semantic search might miss:
```javascript
// Should usually keep true (default)
find_code_pointers({ query: "auth", keywordFallback: true })
```

---

## Best Practices

### When to Use find_code_pointers

1. **Understanding code architecture** - Find how components connect
2. **Finding implementations** - Locate specific function/class definitions
3. **Tracing dependencies** - See who imports/calls code (tracebacks)
4. **Exploring unfamiliar code** - Use with `includeRecent`

### When to Use find_memory

1. **Recalling past discussions** - Find previous conversations
2. **Locating decisions** - Search for decisions made earlier
3. **Cross-session context** - Carry context across sessions
4. **Time-based queries** - Use "yesterday", "last week"

### Parameter Selection Guide

| Goal | Recommended Parameters |
|------|----------------------|
| Quick exploration | `zoomLevel: "ultra-wide"`, `limit: 50` |
| Normal search | defaults (threshold: 0.25, limit: 10) |
| Exact match | `zoomLevel: "macro"`, `threshold: 0.8` |
| Understand code | `includeTracebacks: true`, `galleryMode: true` |
| Cross-project | `allProjects: true` |
| Recent context | `includeRecent: 10`, `recencyBoost: true` |

### Combining with Drill-Down

Always use `drill_down()` to explore results:
```javascript
// 1. Search
const results = find_code_pointers({ query: "auth" })

// 2. Drill into best match
drill_down({ drilldownID: results[0].drilldownID })

// Or for memory
const memResults = find_memory({ query: "API decisions" })
drill_down({ drilldownID: memResults[0].drilldownID })
```

---

## Troubleshooting

### No Results Returned

1. **Lower the threshold**:
   ```javascript
   find_code_pointers({ query: "auth", threshold: 0.15 })
   ```

2. **Enable keyword fallback**:
   ```javascript
   find_code_pointers({ query: "auth", keywordFallback: true })
   ```

3. **Check query formulation** - use code terms, not questions

### Timeout Errors

1. **Increase timeout**:
   ```bash
   export SPECMEM_EMBEDDING_TIMEOUT=120
   ```

2. **Check embedding server status**:
   ```javascript
   embedding_status({})
   ```

3. **Restart embedding server if needed**:
   ```javascript
   embedding_stop({})
   embedding_start({})
   ```

### Low-Quality Results

1. **Use gallery mode for analysis**:
   ```javascript
   find_code_pointers({ query: "auth", galleryMode: true })
   ```

2. **Add tracebacks to understand context**:
   ```javascript
   find_code_pointers({ query: "auth", includeTracebacks: true })
   ```

3. **Include recent files for fresh context**:
   ```javascript
   find_code_pointers({ query: "auth", includeRecent: 10 })
   ```

### Memory Search Tips

1. **Use specific queries** - more context = better results
2. **Filter by memory type** - narrow down what you're looking for
3. **Use importance filter** for critical information
4. **Combine with dateRange** for historical context

---

## Summary

| Tool | Query Style | Best For |
|------|-------------|----------|
| find_code_pointers | CODE TERMS | Code search, architecture understanding |
| find_memory | Natural language | Past discussions, decisions, context |

**Key Takeaways**:
1. Use CODE TERMS for code search, natural language for memory search
2. Keep threshold at 0.25 unless you need exact matches
3. Use zoomLevel presets instead of guessing values
4. Always use drill_down() to explore results
5. Enable galleryMode for deeper analysis
6. Understand that "processing" status doesn't mean your query is being worked on
7. Use includeRecent when exploring unfamiliar codebases