# Hyly E2E Plugin - Architecture

This document describes the architecture, components, and design decisions of the Hyly E2E Plugin.

## Overview

The Hyly E2E Plugin is a comprehensive test automation solution built on Claude Code's plugin system. It provides an end-to-end pipeline that transforms requirements into executable tests through AI-powered intelligence.

## Architecture Diagram

```
┌─────────────────────────────────────────────────────────────────┐
│                    Hyly E2E Plugin                               │
├─────────────────────────────────────────────────────────────────┤
│                                                                   │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │   Skills     │  │   Agents     │  │    Hints     │          │
│  │              │  │              │  │              │          │
│  │ • selector-  │  │ • pipeline-  │  │ • 14 domain  │          │
│  │   finder     │  │   orchestrator│  │   specific   │          │
│  │ • test-      │  │ • 6 stage    │  │   hint files │          │
│  │   debugger   │  │   agents     │  │              │          │
│  └──────────────┘  └──────────────┘  └──────────────┘          │
│                                                                   │
└─────────────────────────────────────────────────────────────────┘
                              │
                    ┌─────────┴─────────┐
                    │                   │
          ┌─────────▼────────┐  ┌──────▼──────────┐
          │  Claude Code CLI │  │  MCP Servers    │
          │                  │  │                 │
          │  • Execution     │  │  • Chrome       │
          │  • Orchestration │  │    DevTools     │
          │  • User I/O      │  │  • Notion API   │
          └──────────────────┘  └─────────────────┘
```

## Component Architecture

### 1. Plugin Manifest (.claude-plugin/plugin.json)

**Purpose**: Defines plugin metadata and capabilities

**Structure**:
```json
{
  "name": "hyly-e2e-plugin",
  "version": "1.0.0",
  "capabilities": {
    "skills": [...],
    "agents": [...],
    "hints": [...],
    "scripts": [...]
  }
}
```

**Responsibilities**:
- Register plugin with Claude Code
- Declare available skills and agents
- Specify requirements and dependencies
- Configure environment variables

### 2. Skills (Agent Skills)

**Purpose**: Model-invoked capabilities that Claude automatically activates

#### selector-finder

**Architecture**:
```
User Request
    │
    ▼
selector-finder Skill
    │
    ├─► Chrome DevTools MCP ──► Live Application
    │                                │
    │                                ▼
    │                           DOM Inspection
    │                                │
    └─► Reference Patterns ──────────┤
                                     │
                                     ▼
                              Selector Recommendation
                                     │
                                     ▼
                              Usage Examples
```

**Key Features**:
- Live DOM inspection via Chrome DevTools MCP
- Semantic selector prioritization (getByRole, getByText)
- Pattern matching against proven selectors
- Complete usage examples with alternatives

**Design Decisions**:
- Uses MCP for real-time inspection
- Prioritizes accessibility over fragility
- Provides fallback options
- Teaches best practices

#### test-debugger

**Architecture**:
```
Test Failure
    │
    ▼
test-debugger Skill
    │
    ├─► Read test-results/*.json
    ├─► Analyze error traces
    ├─► Compare with reference tests
    └─► Check page objects
         │
         ▼
    Root Cause Analysis
         │
         ├─► Selector issues
         ├─► Timing problems
         ├─► Assertion failures
         └─► Authentication errors
              │
              ▼
    Specific Fix Recommendations
         │
         └─► File:line_number with before/after code
```

**Key Features**:
- Comprehensive test result analysis
- Root cause identification
- Specific fix recommendations with line numbers
- Best practice validation

**Design Decisions**:
- Reads actual test results (no guessing)
- Provides actionable fixes (not generic advice)
- References project patterns
- Teaches debugging skills

### 3. Agents (Subagents)

**Purpose**: Specialized AI assistants for specific pipeline stages

#### Agent Hierarchy

```
                pipeline-orchestrator (Super Agent)
                          │
        ┌─────────────────┼─────────────────┐
        │                 │                 │
        ▼                 ▼                 ▼
   Stage 1-2          Stage 3-4          Stage 5-6
        │                 │                 │
┌───────┴───────┐  ┌──────┴──────┐  ┌──────┴──────┐
│               │  │             │  │             │
▼               ▼  ▼             ▼  ▼             ▼
notion-req    test-  notion-   playwright-  script-   test-exec
extractor     case   test-     test-        reviewer  reporter
              gen    registrar generator
```

#### pipeline-orchestrator

**Role**: Super agent that manages all other agents

**Responsibilities**:
- Ask user for Notion URL (ALWAYS at start)
- Invoke appropriate agents for each stage
- Validate agent outputs for quality
- Coordinate human approval gates
- Maintain state across stages
- Request improvements when needed

**State Management**:
```
Pipeline State:
├─ current_stage: 1-6
├─ notion_url: string
├─ stage_outputs: Map<stage, output_file>
├─ approvals: Map<stage, boolean>
└─ context: conversation_history
```

**Critical Rules**:
1. ALWAYS ask for fresh URL (ignore cached/provided URLs)
2. NEVER proceed without human approval
3. NEVER accept agent-generated approvals
4. MAINTAIN state throughout pipeline
5. VALIDATE outputs before accepting

#### Stage Agents

Each stage agent follows this pattern:

```
Input Validation
      │
      ▼
Processing Logic
      │
      ├─► Use scripts (if applicable)
      ├─► AI intelligence
      └─► Pattern learning
           │
           ▼
    Output Generation
           │
           ├─► Structured JSON
           ├─► Human-readable summary
           └─► Quality metadata
                │
                ▼
         Quality Checks
                │
                └─► Return to orchestrator
```

**Stage 1: notion-requirements-extractor**

Process:
1. Extract raw content using script
2. MANDATORY intelligent processing
3. Generate user stories (5+)
4. Create acceptance criteria
5. Structure test scenarios

Output: `requirements.json`, `requirements-summary.md`

**Stage 2: test-case-generator**

Process:
1. Read requirements.json
2. Generate 5-7 comprehensive test cases
3. Create 15-20 detailed steps per test
4. Use actual UI patterns (not generic)
5. Include proper credentials

Output: `test-plan.json`

**Stage 3: notion-test-registrar**

Process:
1. Ask user for test case selection
2. Register ONLY selected cases using script
3. NO fallback allowed (Notion only)
4. Validate real Notion page IDs
5. Output ONLY registered cases

Output: `registration-summary.json`

**Stage 4: playwright-test-generator**

Process:
1. Read ALL test steps from registration summary
2. Learn from reference test files
3. Generate production-ready scripts
4. Use proven selectors
5. Follow Page Object Model

Output: Multiple `.spec.ts` files, `generation-summary.json`

**Stage 5: playwright-script-reviewer**

Process:
1. ASK user for hint file
2. Review scripts using hints context
3. AUTO-INVOKE Stage 4 if score < 70
4. Re-review after improvements
5. Max 2 iterations

Output: `review-summary.json`

**Stage 7: test-execution-reporter**

Process:
1. Execute all approved scripts
2. Capture results
3. Report to Notion
4. Generate execution summary

Output: `execution-summary.json`, Notion updates

### 4. Hints System

**Purpose**: Context-aware review rules for specific domains

**Architecture**:
```
Hint File Selection
      │
      ▼
Load Hint Content
      │
      ▼
Parse Rules & Patterns
      │
      ├─► MANDATORY rules
      ├─► FORBIDDEN patterns
      ├─► REQUIRED elements
      └─► BEST practices
           │
           ▼
    Apply to Script Review
           │
           ├─► Score calculation
           ├─► Issue identification
           └─► Fix recommendations
```

**Hint Categories**:

| Category | Files | Purpose |
|----------|-------|---------|
| General | master-hints, default-hints | Comprehensive rules |
| Authentication | login-test-hints | Login patterns |
| Navigation | hayley-navigation-hints | Navigation specifics |
| Components | dashboard-widget-hints, filter-hints | UI components |
| Data | table-explorer-hints, table-explorer-csv-hints | Data tables |
| Documents | pdf-template-*.md | PDF testing |
| Features | favorites-hints, help-center-hints, export-test-hints | Feature-specific |
| Quality | regression-test-hints | Regression standards |

**Design Principles**:
1. **Specificity**: Domain-specific rules over generic
2. **Actionability**: Clear dos and don'ts
3. **Examples**: Concrete code examples
4. **Patterns**: Proven working patterns
5. **Rationale**: Why rules exist

### 5. Scripts (JavaScript Helpers)

**Purpose**: Handle complex operations that require external APIs

**stage1-notion-requirements-reader.js**
- Calls Notion API
- Extracts blocks and content
- Structures raw data
- Agent performs intelligent processing on output

**stage3-notion-test-registration.js**
- Creates Notion pages
- Sets properties
- Returns page IDs
- NO fallback to local storage

**stage6-test-execution-reporting.js**
- Executes Playwright tests
- Parses test results
- Updates Notion database
- Generates execution report

**Why Scripts?**:
- Notion API requires complex authentication
- Test execution needs shell integration
- JSON processing for large files
- File system operations

## Data Flow

### Complete Pipeline Data Flow

```
Notion URL
    │
    ▼
┌─────────────────────────────────────────────────────────────┐
│ Stage 1: Requirements Extraction                            │
├─────────────────────────────────────────────────────────────┤
│ Input: Notion URL                                           │
│ Process: Extract → Intelligent Processing → Structure       │
│ Output: requirements.json                                   │
└────────────────┬────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────┐
│ Stage 2: Test Case Generation                               │
├─────────────────────────────────────────────────────────────┤
│ Input: requirements.json                                    │
│ Process: Analyze → Generate Cases → Detailed Steps          │
│ Output: test-plan.json                                      │
└────────────────┬────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────┐
│ Stage 3: Test Case Registration                             │
├─────────────────────────────────────────────────────────────┤
│ Input: test-plan.json + User Selection                      │
│ Process: Register Selected → Get Notion IDs                 │
│ Output: registration-summary.json (ONLY selected cases)     │
└────────────────┬────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────┐
│ Stage 4: Playwright Script Generation                       │
├─────────────────────────────────────────────────────────────┤
│ Input: registration-summary.json (with ALL test steps)      │
│ Process: Learn Patterns → Generate Scripts → Apply POM      │
│ Output: Multiple .spec.ts files                             │
└────────────────┬────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────┐
│ Stage 5: Script Review & Auto-Fix                           │
├─────────────────────────────────────────────────────────────┤
│ Input: generation-summary.json + Hints File                 │
│ Process: Review → Auto-Fix Loop → Re-Review                 │
│ Output: review-summary.json + Improved scripts              │
└────────────────┬────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────┐
│ Stage 7: Test Execution & Reporting                         │
├─────────────────────────────────────────────────────────────┤
│ Input: review-summary.json                                  │
│ Process: Execute Tests → Capture Results → Report to Notion │
│ Output: execution-summary.json + Notion updates             │
└─────────────────────────────────────────────────────────────┘
```

## Design Decisions

### 1. Why Claude Code Plugin?

**Decision**: Package as Claude Code plugin vs standalone tool

**Rationale**:
- **Leverage AI**: Use Claude's intelligence for content understanding
- **Integration**: Native integration with developer workflow
- **Flexibility**: Skills auto-activate based on context
- **Extensibility**: Easy to add new agents/skills
- **Collaboration**: Natural conversation interface

### 2. Why 7-Stage Pipeline?

**Decision**: Break into 7 stages vs monolithic process

**Rationale**:
- **Human Control**: Approval gates prevent runaway automation
- **Quality Assurance**: Validate outputs at each step
- **Iterative Improvement**: Fix issues before they compound
- **Transparency**: Clear visibility into what's being generated
- **Flexibility**: Run individual stages or full pipeline

### 3. Why Super Agent Pattern?

**Decision**: Use pipeline-orchestrator as super agent

**Rationale**:
- **State Management**: Centralized state across stages
- **Quality Control**: Single point for output validation
- **Communication**: Broker between user and stage agents
- **Consistency**: Enforces approval protocol
- **Recovery**: Can restart from any stage

### 4. Why Hints System?

**Decision**: Use hint files vs hardcoded rules

**Rationale**:
- **Context-Aware**: Different rules for different domains
- **Customizable**: Users can create project-specific hints
- **Maintainable**: Update rules without changing code
- **Reusable**: Share hints across teams
- **Educational**: Hints teach best practices

### 5. Why Semantic Selectors?

**Decision**: Prioritize getByRole/getByText vs CSS selectors

**Rationale**:
- **Accessibility**: Tests ensure app is accessible
- **Stability**: Semantic selectors survive UI changes
- **Readability**: Clear what element is being tested
- **Maintainability**: Less brittle tests
- **Best Practice**: Aligns with Playwright recommendations

### 6. Why Auto-Fix Loop in Stage 5?

**Decision**: Automatically trigger Stage 4 improvements

**Rationale**:
- **Efficiency**: No manual back-and-forth
- **Quality**: Iteratively improve to acceptable level
- **Learning**: Stage 4 learns from Stage 5 feedback
- **User Experience**: Reduces manual intervention
- **Bounded**: Max 2 iterations prevents infinite loops

## Extension Points

### Adding New Skills

1. Create `skills/new-skill/SKILL.md`
2. Define YAML frontmatter with name, description, allowed-tools
3. Write skill instructions
4. Add to plugin.json capabilities
5. Test skill invocation

### Adding New Agents

1. Create `agents/XX-new-agent.md`
2. Define YAML frontmatter with name, description, model
3. Write agent instructions
4. Add to plugin.json capabilities
5. Update orchestrator if part of pipeline

### Adding New Hints

1. Create `hints/new-domain-hints.md`
2. Document MANDATORY, FORBIDDEN, REQUIRED patterns
3. Provide code examples
4. Add to plugin.json capabilities
5. Reference in Stage 5 agent

### Customizing for Your Project

1. **Update Credentials**: Change TEST_EMAIL, TEST_PASSWORD
2. **Add Selectors**: Document your app's patterns in hints
3. **Create Hints**: Project-specific hint files
4. **Update Scripts**: Modify stage scripts for your workflow
5. **Add Page Objects**: Create page objects for your pages

## Performance Considerations

### Token Usage

**Strategy**: Minimize context size while maintaining quality

**Optimizations**:
- Skills use focused instructions (not full codebase)
- Agents receive only relevant data
- Hints loaded on-demand
- Stage outputs are structured (not verbose)

### Execution Time

**Pipeline Stages**:
- Stage 1: ~30-60 seconds (Notion API + processing)
- Stage 2: ~60-90 seconds (AI test plan generation)
- Stage 3: ~60-120 seconds (AI test case generation)
- Stage 4: ~30-60 seconds (Notion registration)
- Stage 5: ~60-120 seconds per script
- Stage 6: ~30-60 seconds per script + auto-fix time
- Stage 7: Variable (depends on test count)

**Total**: ~10-20 minutes for 5-7 tests (including approvals)

### Scalability

**Current Limits**:
- 5-7 test cases per pipeline run
- 15-20 steps per test case
- 2 auto-fix iterations max

**Future Improvements**:
- Parallel script generation
- Cached Notion content
- Batch test registration
- Distributed test execution

## Security Considerations

### API Keys

- Stored in `.env` (not committed)
- Loaded via environment variables
- Required for Notion integration
- Validated before use

### Test Credentials

- Stored in `.env` (not committed)
- Used only for test accounts
- Should NOT be production credentials
- Fallback to env vars in scripts

### Code Injection

- No eval() or exec() in scripts
- Validated inputs before processing
- Sanitized selectors
- Secure Notion API calls

## Testing the Plugin

### Manual Testing

1. **Install Plugin**
   ```bash
   /plugin install ./hyly-e2e-plugin
   ```

2. **Test Skills**
   ```
   "Find selector for the login button"
   "Debug the failing dashboard test"
   ```

3. **Test Pipeline**
   ```
   "Run the test automation pipeline"
   [Provide Notion URL]
   ```

4. **Verify Outputs**
   - Check generated files
   - Review Notion updates
   - Validate test scripts

### Automated Testing

Future: Add automated tests for:
- Script parsing
- Notion API mocking
- File generation
- Agent invocation

## Future Enhancements

### Short-term (v1.1)

- [ ] Add more hint files (API testing, mobile)
- [ ] Improve Stage 4 pattern learning
- [ ] Enhanced error messages
- [ ] Progress indicators

### Medium-term (v1.5)

- [ ] Support multiple testing frameworks
- [ ] Visual regression testing
- [ ] Performance testing integration
- [ ] API testing support

### Long-term (v2.0)

- [ ] Machine learning for flaky test detection
- [ ] Automatic test maintenance
- [ ] Multi-language support
- [ ] Cloud execution platform

## Contributing

To contribute to the plugin:

1. Fork the repository
2. Create feature branch
3. Make changes
4. Test thoroughly
5. Submit pull request

See CONTRIBUTING.md for detailed guidelines.

## License

MIT License - See LICENSE file

---

**Architecture Version**: 1.0.0
**Last Updated**: 2025-01-03
**Authors**: Hyly.ai Team