# Hyly E2E Test Automation Plugin

**Transform your requirements into automated Playwright tests with AI-powered intelligence.**

The Hyly E2E Plugin is a comprehensive test automation solution that converts requirements from Notion into fully automated, production-ready Playwright tests through an intelligent 7-stage pipeline.

## Features

### 🚀 7-Stage Test Automation Pipeline
Complete end-to-end automation from requirements to execution:

1. **Requirements Extraction** - Pull requirements from Notion pages
2. **Test Plan Generation** - Create strategic test plans with scenarios and coverage
3. **Test Case Generation** - Generate detailed test cases from test plans
4. **Notion Test Registration** - Register test cases in Notion database
5. **Playwright Script Generation** - Create production-ready test scripts
6. **Script Review & Auto-Fix** - Review scripts with automatic improvements
7. **Test Execution & Reporting** - Run tests and report results to Notion

### 🎯 Intelligent Selector Discovery
- **Live Application Inspection** - Use Chrome DevTools to inspect real DOM elements
- **Semantic Selectors** - Prioritize accessibility-based selectors (getByRole, getByText)
- **Pattern Learning** - Learn from existing test patterns
- **Best Practice Recommendations** - Get selector alternatives and usage examples

### 🐛 Test Debugging Assistant
- **Failure Analysis** - Analyze test results, errors, and stack traces
- **Root Cause Identification** - Determine why tests fail
- **Specific Fixes** - Get exact code changes with file paths and line numbers
- **Selector Improvements** - Recommendations based on reference files

### 💡 Context-Aware Script Review
- **Hints System** - Use domain-specific hints for intelligent review
- **Auto-Fix Loop** - Automatically request improvements from script generator
- **Quality Scoring** - Score scripts on best practices and patterns
- **Iterative Improvement** - Re-review after fixes (max 2 iterations)

### 📝 Notion Integration
- **Requirements Extraction** - Parse Notion pages for user stories and acceptance criteria
- **Test Case Registry** - Track test cases in Notion database
- **Results Reporting** - Report test execution results back to Notion

## Installation

### Prerequisites

1. **Node.js** >= 18.0.0
2. **Playwright** >= 1.40.0
3. **Chrome DevTools MCP Server** (for selector-finder skill)
4. **Notion API Access** (optional, for Notion integration)

### Install the Plugin

#### Option 1: Install from GitHub (Recommended)
```bash
/plugin install hyly-e2e-plugin@hyly-ai
```

#### Option 2: Manual Installation
1. Clone or download this repository
2. Copy the `hyly-e2e-plugin` folder to your project
3. Install via local path:
```bash
/plugin install ./hyly-e2e-plugin
```

## Configuration

### Environment Variables

Create a `.env` file in your project root with the following variables:

```env
# Notion Integration (Required for pipeline stages 1, 3, 6)
NOTION_API_KEY=your_notion_api_key_here
NOTION_DATABASE_ID=your_notion_database_id_here

# Test Credentials (Optional - for your application)
TEST_EMAIL=your_test_email@example.com
TEST_PASSWORD=your_test_password_here
```

### Notion Setup

#### 1. Create Notion Integration
1. Go to https://www.notion.so/my-integrations
2. Click "New integration"
3. Name it (e.g., "E2E Test Automation")
4. Copy the API key

#### 2. Create Test Case Database
Create a Notion database with these properties:
- **Name** (Title) - Test case name
- **Status** (Select) - Passed/Failed/Pending
- **Test ID** (Text) - Unique test identifier
- **Priority** (Select) - High/Medium/Low
- **Tags** (Multi-select) - Test categories
- **Date** (Date) - Last execution date
- **Duration** (Text) - Execution time
- **Passed/Failed/Skipped** (Number) - Test statistics

#### 3. Share Database with Integration
1. Open your database in Notion
2. Click "Share" button
3. Search for your integration name
4. Click "Invite"

#### 4. Extract Database ID
From the database URL: `https://www.notion.so/{DATABASE_ID}?v=...`
Copy the DATABASE_ID portion.

### Chrome DevTools MCP Server Setup

The `selector-finder` skill requires the Chrome DevTools MCP server:

1. Install the MCP server (if not already installed)
2. Configure in your Claude settings
3. Ensure Chrome DevTools MCP is running

## Usage

### Running the Complete Pipeline

The pipeline orchestrator manages all 6 stages:

```bash
# Invoke the pipeline orchestrator
"Run the test automation pipeline"

# Provide Notion URL when prompted
https://www.notion.so/your-requirements-page
```

The orchestrator will:
1. Extract requirements from Notion
2. Generate test cases (with human approval)
3. Register test cases to Notion (with human approval)
4. Generate Playwright scripts (with human approval)
5. Review scripts with auto-fix (with human approval)
6. Execute tests and report results

### Using Individual Skills

#### Selector Finder Skill

Find reliable Playwright selectors:

```bash
# Invoke the selector-finder skill
"Find selector for the Add button on the dashboard page"
```

The skill will:
- Inspect the live application using Chrome DevTools
- Compare with existing selector patterns
- Recommend the best selector with alternatives
- Provide page object examples

#### Test Debugger Skill

Debug failed tests:

```bash
# Invoke the test-debugger skill
"Debug the failing dashboard export test"
```

The skill will:
- Analyze test results and error traces
- Identify the root cause
- Provide specific fixes with line numbers
- Suggest improvements based on best practices

### Running Individual Pipeline Stages

You can run specific stages of the pipeline:

#### Stage 1: Extract Requirements
```bash
"Extract requirements from this Notion page: https://notion.so/..."
```

#### Stage 2: Generate Test Cases
```bash
"Generate test cases from the requirements file"
```

#### Stage 4: Generate Playwright Scripts
```bash
"Generate Playwright scripts from the registration summary"
```

#### Stage 5: Review Scripts
```bash
"Review the generated Playwright scripts"
```

When asked for hints, provide:
- `master-hints.md` - Comprehensive rules from all patterns
- `default-hints.md` - General Playwright best practices
- `login-test-hints.md` - Authentication patterns
- `dashboard-widget-hints.md` - Dashboard/widget specific
- Or any custom hint file from your project

## Plugin Components

### Skills (Agent Skills)

#### selector-finder
Finds and validates reliable Playwright selectors through live application inspection.

**When to use:**
- Finding selectors for new elements
- Replacing fragile selectors
- Learning best practices for selector usage
- Troubleshooting selector issues

**Capabilities:**
- Live DOM inspection with Chrome DevTools
- Semantic selector prioritization
- Pattern matching against proven selectors
- Complete usage examples and alternatives

#### test-debugger
Debugs failed Playwright tests with specific fix recommendations.

**When to use:**
- Test failures need investigation
- Understanding why tests break
- Getting specific code fixes
- Learning from test failures

**Capabilities:**
- Test result analysis
- Error trace interpretation
- Root cause identification
- Fix recommendations with line numbers

### Agents (Subagents)

#### pipeline-orchestrator
Coordinates the entire 7-stage pipeline with human approval gates.

**Responsibilities:**
- Manage communication between all agents
- Track pipeline state and progress
- Handle human approval checkpoints
- Validate agent outputs for quality
- Request improvements when needed

#### notion-requirements-extractor (Stage 1)
Extracts and structures requirements from Notion pages.

**Outputs:**
- `requirements.json` - Structured requirements data
- `requirements-summary.md` - Human-readable summary

#### test-case-generator (Stage 2)
Generates comprehensive test cases from requirements.

**Outputs:**
- `test-plan.json` - Test cases with detailed steps

#### notion-test-registrar (Stage 3)
Registers test cases in Notion database.

**Outputs:**
- `registration-summary.json` - Registration status with Notion page IDs

#### playwright-test-generator (Stage 4)
Generates production-ready Playwright test scripts.

**Outputs:**
- Multiple `.spec.ts` files in tests directory
- `generation-summary.json` - Generation metadata

#### playwright-script-reviewer (Stage 5)
Reviews generated scripts with context-aware analysis and auto-fix.

**Outputs:**
- `review-summary.json` - Review results with scores
- Improved scripts (if auto-fix triggered)

#### test-execution-reporter (Stage 7)
Executes tests and reports results to Notion.

**Outputs:**
- `execution-summary.json` - Test execution results
- Notion database updates with test outcomes

### Hints System

Hint files provide context-aware review rules:

- **master-hints.md** - Comprehensive rules from all test patterns
- **default-hints.md** - General Playwright best practices
- **login-test-hints.md** - Authentication and login patterns
- **dashboard-widget-hints.md** - Dashboard and widget testing
- **export-test-hints.md** - Export functionality rules
- **filter-hints.md** - Filter component patterns
- **favorites-hints.md** - Favorites feature patterns
- **help-center-hints.md** - Help center testing
- **hayley-navigation-hints.md** - Navigation app specifics
- **pdf-template-*.md** - PDF template testing rules
- **table-explorer-*.md** - Data table testing patterns
- **regression-test-hints.md** - Regression testing standards

## Project Structure

```
hyly-e2e-plugin/
├── .claude-plugin/
│   └── plugin.json           # Plugin manifest
├── skills/
│   ├── selector-finder/
│   │   └── SKILL.md         # Selector discovery skill
│   └── test-debugger/
│       └── SKILL.md         # Test debugging skill
├── agents/
│   ├── 00-pipeline-orchestrator.md
│   ├── 01-notion-requirements-extractor.md
│   ├── 02-test-case-generator.md
│   ├── 03-notion-test-registrar.md
│   ├── 04-playwright-test-generator.md
│   ├── 05-playwright-script-reviewer.md
│   └── 06-test-execution-reporter.md
├── hints/
│   ├── master-hints.md
│   ├── default-hints.md
│   └── ... (14 hint files total)
├── scripts/
│   ├── stage1-notion-requirements-reader.js
│   ├── stage3-notion-test-registration.js
│   └── stage6-test-execution-reporting.js
└── README.md
```

## Best Practices

### Selector Priority (ALWAYS Follow This Order)

1. **Semantic Selectors** (BEST)
   ```typescript
   page.getByRole('button', { name: 'Add' })
   page.getByText('Dashboard', { exact: true })
   ```

2. **Data Test IDs** (GOOD)
   ```typescript
   page.getByTestId('add-widget-button')
   ```

3. **ID Selectors** (ACCEPTABLE)
   ```typescript
   page.locator('#email-input')
   ```

4. **Class Selectors** (USE WITH CAUTION)
   ```typescript
   page.locator('.dashboard-card')
   ```

5. **Complex Selectors** (LAST RESORT)
   ```typescript
   page.locator('div > ul > li:nth-child(3)')
   ```

### Test Structure Requirements

Always include in your tests:

```typescript
import { test, expect } from '@playwright/test';
import { LoginPage } from './pages/login.page';
import { DashboardPage } from './pages/dashboard.page';

test.describe('Feature Name', () => {
  // Set timeout for slow operations
  test.setTimeout(10 * 60 * 1000);

  test('Test description', async ({ page }) => {
    // Use page objects
    const loginPage = new LoginPage(page);
    const dashboardPage = new DashboardPage(page);

    await test.step('Step 1: Login', async () => {
      await loginPage.navigate();
      await loginPage.login();
    });

    await test.step('Step 2: Navigate', async () => {
      await dashboardPage.navigate();
    });

    // Add assertions
    await expect(page).toHaveTitle(/Expected Title/);
  });
});
```

### Common Patterns to Avoid

❌ **DON'T:**
- Use `waitForLoadState('networkidle')` - causes browser closure
- Hardcode credentials - use environment variables
- Use nth() or position-based selectors without good reason
- Create tests without page objects
- Skip waitFor() before interactions

✅ **DO:**
- Use semantic selectors (getByRole, getByText)
- Use test.step() for organization
- Add proper timeouts for slow operations
- Use page object model
- Add waitFor() before interactions
- Use environment variables for config

## Troubleshooting

### Pipeline Issues

**Issue**: Stage 1 not extracting requirements properly
**Solution**: Ensure NOTION_API_KEY is set and Notion page is shared with integration

**Issue**: Scripts fail to generate in Stage 4
**Solution**: Check that registration-summary.json has valid test case data

**Issue**: Tests fail after generation
**Solution**: Run Stage 5 review with appropriate hint file, let auto-fix improve scripts

### Skill Issues

**Issue**: Selector-finder can't inspect application
**Solution**: Ensure Chrome DevTools MCP server is running and configured

**Issue**: Test-debugger can't find test results
**Solution**: Check that test-results directory exists with recent test output

### General Issues

**Issue**: Plugin not recognized after installation
**Solution**: Restart Claude Code CLI after plugin installation

**Issue**: Environment variables not loading
**Solution**: Ensure .env file is in project root and properly formatted

## Contributing

To adapt this plugin for your project:

1. **Customize Hints** - Create project-specific hint files
2. **Update Scripts** - Modify stage scripts for your workflow
3. **Add Selectors** - Document your application's selector patterns
4. **Extend Skills** - Add new skills for project-specific needs

## Support

For issues, questions, or contributions:
- GitHub Issues: [Report an issue](https://github.com/hyly-ai/hyly-e2e-plugin/issues)
- Documentation: [View full docs](https://docs.hyly.ai/e2e-plugin)
- Website: [https://hyly.ai](https://hyly.ai)

## License

MIT License - See LICENSE file for details

## Acknowledgments

Built with:
- [Playwright](https://playwright.dev/) - Web testing framework
- [Claude Code](https://www.anthropic.com/claude-code) - AI-powered development assistant
- [Notion API](https://developers.notion.com/) - Knowledge management integration

---

**Made with ❤️ by Hyly.ai**

*Automate smarter, test better, ship faster.*