# Research-First Architecture

**Technical Design Document**
**Version:** 1.0
**Author:** Program 4 (Research Swarm Executor)
**Date:** 2026-01-23

---

## Executive Summary

This document describes the **Research-First Architecture** for The Grid - a mandatory research swarm that runs BEFORE any planning phase. The architecture introduces two new agent types (Researcher and Scout) that gather context from external sources and existing codebases, enabling better-informed planning and execution.

### Key Benefits
- **Informed Planning:** Planner receives structured context about best practices, patterns, and constraints
- **Reduced Rework:** Discovering constraints BEFORE building prevents costly pivots
- **Current Knowledge:** Web search brings 2024-2025 best practices beyond training cutoff
- **Faster Execution:** Cached research eliminates repeated searches
- **Parallel Intelligence:** Multiple researchers can gather context simultaneously

---

## Architecture Overview

### Current Flow (Before)
```
User Request → Planner → Executor → Recognizer
```

### New Flow (Research-First)
```
User Request → Scout → Researcher(s) → Planner → Executor → Recognizer
                ↓           ↓
           Codebase     External
            Context      Context
```

### Flow Decision Tree
```
                    User Request
                         │
                         ▼
              ┌─────────────────────┐
              │  Quick Mode Check   │
              └─────────────────────┘
                    │         │
            ≤5 files│         │ >5 files
                    ▼         ▼
              ┌─────────┐  ┌─────────────────┐
              │ /quick  │  │ Research Phase  │
              └─────────┘  └─────────────────┘
                                  │
                    ┌─────────────┴─────────────┐
                    ▼                           ▼
              ┌─────────┐                 ┌───────────┐
              │  Scout  │                 │ Researcher│
              │(codebase)│                │ (external)│
              └─────────┘                 └───────────┘
                    │                           │
                    └─────────────┬─────────────┘
                                  ▼
                         ┌───────────────┐
                         │ Context Cache │
                         └───────────────┘
                                  │
                                  ▼
                         ┌───────────────┐
                         │    Planner    │
                         │(with context) │
                         └───────────────┘
                                  │
                                  ▼
                         ┌───────────────┐
                         │   Executor    │
                         └───────────────┘
```

---

## Component Specifications

### 1. Scout Agent (`grid-scout.md`)

**Purpose:** Rapid reconnaissance of existing codebase
**Time Budget:** < 2 minutes
**Output:** `.grid/scout/RECON_{timestamp}.md`

#### Capabilities
| Capability | Method | Time |
|------------|--------|------|
| Structure Scan | `find`, `glob` | 30s |
| Technology Detection | Package file analysis | 30s |
| Pattern Detection | Grep + file existence | 30s |
| Constraint Discovery | Schema + API analysis | 30s |

#### Key Outputs
```yaml
scout_output:
  technology_stack:
    languages: [TypeScript, Python]
    frameworks: [Next.js 14, FastAPI]
    databases: [PostgreSQL via Prisma]

  detected_patterns:
    - name: "App Router"
      confidence: HIGH
    - name: "Server Components"
      confidence: MEDIUM

  constraints:
    locked_dependencies: [react@18.2.0]
    existing_schemas: [prisma/schema.prisma]
    api_contracts: [src/app/api/*/route.ts]

  conventions:
    naming: "camelCase for files, PascalCase for components"
    imports: "absolute imports from @/"
```

#### Speed Optimizations
- Parallel file operations
- Early exit on pattern detection
- Depth limits on tree traversal
- Sample-based analysis (first N, not all)

---

### 2. Researcher Agent (`grid-researcher.md`)

**Purpose:** Gather external intelligence from web sources
**Time Budget:** 5-10 minutes (parallelizable)
**Output:** `.grid/research_cache/{topic}.md`

#### Mission Types
| Mission | Triggers | Output |
|---------|----------|--------|
| `technology` | New framework/library | Best practices, gotchas |
| `patterns` | Architecture decisions | Implementation guides |
| `similar_projects` | Novel project type | Reference architectures |
| `api_docs` | External integrations | API context, examples |

#### Search Tool Selection
| Need | Tool | Rationale |
|------|------|-----------|
| Best practices | `WebSearch` | Broad coverage |
| Code examples | `mcp__exa__get_code_context_exa` | Code-optimized |
| Company info | `mcp__exa__company_research_exa` | Structured data |
| Documentation | `WebFetch` | Direct access |
| GitHub repos | `gh` CLI | Precise queries |

#### Query Generation
```python
def generate_queries(request: dict) -> list[str]:
    queries = []

    # Technology queries (3 per tech)
    for tech in request.get('technologies', []):
        queries.extend([
            f"{tech} best practices 2024 2025",
            f"{tech} production patterns",
            f"{tech} common mistakes"
        ])

    # Pattern queries (3 per pattern)
    for pattern in request.get('patterns', []):
        queries.extend([
            f"{pattern} implementation guide",
            f"{pattern} real world examples",
            f"when not to use {pattern}"
        ])

    return queries
```

#### Confidence Assessment
| Level | Criteria |
|-------|----------|
| HIGH | 3+ corroborating sources, official docs |
| MEDIUM | 1-2 sources, reputable blogs |
| LOW | Single source, old content |

---

### 3. Research Cache

**Location:** `.grid/research_cache/`

#### Structure
```
.grid/research_cache/
├── index.json                    # Cache index
├── nextjs-14-app-router.md       # Cached research
├── prisma-best-practices.md
├── stripe-connect-api.md
└── archive/                      # Expired entries
```

#### Index Schema
```json
{
  "version": "1.0",
  "entries": [
    {
      "key": "nextjs-14-app-router",
      "queries": ["Next.js 14 best practices", "App Router patterns"],
      "created": "2026-01-23T14:00:00Z",
      "expires": "2026-01-24T14:00:00Z",
      "file": "nextjs-14-app-router.md",
      "hit_count": 3
    }
  ]
}
```

#### Cache Policy
| Policy | Value | Rationale |
|--------|-------|-----------|
| Default TTL | 24 hours | Balance freshness/speed |
| Max entries | 50 | Prevent bloat |
| LRU eviction | On limit | Remove least used |
| Force refresh | User request | Override cache |

#### Cache Hit Algorithm
```python
def check_cache(queries: list[str]) -> Optional[str]:
    index = load_index()

    for entry in index['entries']:
        # Check query similarity (Jaccard > 0.7)
        if jaccard_similarity(entry['queries'], queries) > 0.7:
            if not is_expired(entry):
                entry['hit_count'] += 1
                save_index(index)
                return read(entry['file'])

    return None  # Cache miss
```

---

## Integration with Master Control

### MC Research Phase Logic

```python
def research_phase(user_request: str, project_root: str) -> dict:
    """Execute research phase before planning."""

    # Check if research should be skipped
    if should_skip_research(user_request):
        return {"skipped": True, "reason": "Quick mode eligible"}

    context = {}

    # Phase 1: Scout existing codebase (always, if exists)
    if has_existing_code(project_root):
        scout_result = Task(
            prompt="""
First, read ~/.claude/agents/grid-scout.md for your role.

Scan the codebase at {project_root}.
Return structured scout report.
""",
            subagent_type="general-purpose",
            description="Scout codebase"
        )
        context['codebase'] = scout_result

    # Phase 2: Extract research needs
    research_needs = extract_research_needs(user_request, context.get('codebase'))

    # Phase 3: Check cache for existing research
    cached, needed = check_research_cache(research_needs)
    context['cached_research'] = cached

    # Phase 4: Spawn researchers for uncached needs (parallel)
    if needed:
        research_tasks = []
        for need in needed:
            task = Task(
                prompt=f"""
First, read ~/.claude/agents/grid-researcher.md for your role.

<research_request>
{need}
</research_request>

Research and return structured context.
""",
                subagent_type="general-purpose",
                description=f"Research {need['topic']}"
            )
            research_tasks.append(task)

        # All tasks execute in parallel
        context['fresh_research'] = await_all(research_tasks)

    return context
```

### Skip Conditions

Research is skipped when:
```python
def should_skip_research(request: str) -> bool:
    """Determine if research phase should be skipped."""

    # Quick mode eligible = skip research
    if quick_mode_eligible(request):
        return True

    # User explicitly skipped
    if "skip research" in request.lower():
        return True

    # Known technology fully cached
    if all_tech_cached(extract_technologies(request)):
        return True

    # Debug/fix mode
    if is_debug_request(request):
        return True

    return False
```

---

## Planner Integration

### Planner Prompt Enhancement

```python
def spawn_planner_with_research(user_request: str, research_context: dict):
    """Spawn Planner with research context."""

    prompt = f"""
First, read ~/.claude/agents/grid-planner.md for your role.

<codebase_context>
{research_context.get('codebase', 'Greenfield project - no existing code')}
</codebase_context>

<research_context>
{format_research(research_context.get('cached_research', []))}
{format_research(research_context.get('fresh_research', []))}
</research_context>

<user_request>
{user_request}
</user_request>

Create execution plan informed by:
1. Codebase constraints (if existing code)
2. Best practices from research
3. Anti-patterns to avoid
4. Recommended technologies

Your plan should respect existing patterns and leverage researched best practices.
"""

    return Task(
        prompt=prompt,
        subagent_type="general-purpose",
        description="Plan with research context"
    )
```

### Research-Informed Planning Rules

Planner applies research context as:
```yaml
planning_rules:
  # Constraints from Scout
  must_preserve:
    - existing_api_contracts
    - database_schema_structure
    - naming_conventions

  # Patterns from Researcher
  should_use:
    - recommended_patterns
    - best_practices_2024_2025

  # Anti-patterns from Researcher
  must_avoid:
    - documented_anti_patterns
    - deprecated_approaches

  # Technology from Researcher
  prefer:
    - libraries_with_good_research
    - patterns_with_code_examples
```

---

## Performance Considerations

### Time Budgets
| Phase | Target | Max |
|-------|--------|-----|
| Scout | 1 min | 2 min |
| Cache check | 1 sec | 5 sec |
| Single researcher | 3 min | 5 min |
| Parallel researchers | 3 min | 5 min |
| Total research phase | 4 min | 7 min |

### Parallelization Strategy
```
Sequential:
  Scout (1 min) → Cache Check (1 sec) → [If cache miss] Research Phase

Research Phase (Parallel):
  ├── Researcher 1: Framework docs (3 min)
  ├── Researcher 2: Pattern research (3 min)
  └── Researcher 3: API docs (3 min)
       └── Total: 3 min (not 9 min)
```

### Context Budget
| Agent | Context Budget |
|-------|---------------|
| Scout | 10% max |
| Researcher | 20% per instance |
| Research output to Planner | 15% max |

Output compression ensures Planner receives concise, actionable context.

---

## Configuration

### `.grid/config.json` Extensions
```json
{
  "research": {
    "enabled": true,
    "cache_ttl_hours": 24,
    "max_researchers": 3,
    "scout_timeout_seconds": 120,
    "researcher_timeout_seconds": 300,
    "skip_for_quick_mode": true
  }
}
```

### User Controls
| Command | Effect |
|---------|--------|
| "skip research" | Bypass research phase |
| "fresh research" | Ignore cache, research anew |
| "research only" | Run research, don't plan |
| "deep research" | Extended time budget |

---

## Error Handling

### Scout Failures
```yaml
failure_modes:
  timeout:
    action: "Proceed with partial results"
    fallback: "Manual constraint discovery"

  permission_denied:
    action: "Report inaccessible areas"
    fallback: "User provides constraints"

  empty_directory:
    action: "Report greenfield"
    fallback: "Standard patterns"
```

### Researcher Failures
```yaml
failure_modes:
  api_timeout:
    action: "Retry once, then skip topic"
    fallback: "Use training knowledge"

  no_results:
    action: "Report low confidence"
    fallback: "Mark for manual review"

  rate_limited:
    action: "Queue for later"
    fallback: "Proceed without"
```

---

## Observability

### Research Metrics
```yaml
metrics:
  scout:
    - scan_duration_ms
    - files_analyzed
    - patterns_detected
    - constraints_found

  researcher:
    - queries_executed
    - sources_found
    - cache_hit_rate
    - avg_query_time_ms

  overall:
    - research_phase_duration_ms
    - cache_hit_rate
    - context_compression_ratio
```

### Logging
```
.grid/logs/research/
├── scout_2026-01-23T14:00:00.log
├── researcher_nextjs_2026-01-23T14:01:00.log
└── researcher_prisma_2026-01-23T14:01:00.log
```

---

## Migration Path

### Phase 1: Scout Only (Week 1)
- Deploy Scout agent
- Add Scout to MC flow before Planner
- Measure planning improvement

### Phase 2: Researcher Integration (Week 2)
- Deploy Researcher agent
- Add cache infrastructure
- Integrate with Planner prompts

### Phase 3: Full Integration (Week 3)
- Parallel researcher spawning
- Cache optimization
- Metrics and observability

### Backward Compatibility
- Research phase can be disabled via config
- Quick mode bypasses research by default
- Existing projects continue to work

---

## Security Considerations

### Data Handling
- Research cache in `.grid/` (gitignored)
- No secrets in research output
- External queries sanitized

### Rate Limiting
- Max 10 queries per researcher
- 2-second delay between web searches
- Respect API rate limits

---

## Future Enhancements

### Planned
1. **Semantic caching:** Match by meaning, not exact query
2. **Learning loop:** Feed successful patterns back to research
3. **Team knowledge:** Shared research cache across projects
4. **Custom sources:** User-defined authoritative sources

### Considered
- RAG over codebase for deep context
- ML-based pattern detection
- Automated research scheduling

---

## Summary

The Research-First Architecture introduces mandatory intelligence gathering before planning:

1. **Scout** rapidly analyzes existing codebases (< 2 min)
2. **Researcher** gathers external best practices (3-5 min, parallelizable)
3. **Cache** prevents repeated searches (24h TTL)
4. **Planner** receives structured context for informed planning

This architecture ensures The Grid builds on current knowledge and respects existing constraints, reducing rework and improving plan quality.

**End of Line.**