# Testing Documentation

## Overview

AgentRails has comprehensive unit tests covering all core functionality with **96.36% code coverage**.

## Test Suites

### 1. Config Tests (`tests/config.test.ts`)

Tests for configuration loading and validation.

**Coverage:**

- Loading valid configuration files
- Setting default values
- Error handling for missing files
- Validation of required fields (agent, apiKey)
- Type checking for config properties

**8 tests - All passing ✓**

### 2. Parser Tests (`tests/parser.test.ts`)

Tests for YAML test file parsing.

**Coverage:**

- Parsing valid YAML files
- Extracting test cases correctly
- Handling string and object inputs
- Validating required fields (suite, tests, name, input)
- Error messages for malformed YAML
- Test case validation

**10 tests - All passing ✓**

### 3. Evaluator Tests (`tests/evaluator.test.ts`)

Tests for LLM-based evaluation.

**Coverage:**

- Creating evaluator from config
- OpenAI client initialization
- Successful evaluation (passed/failed)
- Structured input/output handling
- Error handling for API failures
- Prompt construction with expected behavior
- Prompt construction with example responses
- Model and temperature configuration

**10 tests - All passing ✓**

### 4. Runner Tests (`tests/runner.test.ts`)

Tests for test execution engine.

**Coverage:**

- Finding test files with glob patterns
- Running individual test cases
- Calling agent with correct input
- Calling evaluator with correct parameters
- Handling test failures
- Handling agent errors
- Timeout handling
- Structured input/output
- Running complete test suites
- Counting passed/failed tests
- Duration tracking
- Running all test files

**16 tests - All passing ✓**

### 5. Reporter Tests (`tests/reporter.test.ts`)

Tests for console output formatting.

**Coverage:**

- Printing passed test results
- Printing failed test results
- Showing error messages
- Verbose output mode
- Multiple test suites
- Duration formatting
- Structured response handling

**7 tests - All passing ✓**

## Test Coverage Report

```
--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
--------------|---------|----------|---------|---------|-------------------
All files     |   96.36 |    87.17 |     100 |   96.22 |
 config.ts    |   78.57 |    85.71 |     100 |   78.57 | 20-24,30
 evaluator.ts |     100 |    82.35 |     100 |     100 | 16-17,70
 parser.ts    |     100 |      100 |     100 |     100 |
 reporter.ts  |     100 |     90.9 |     100 |     100 | 59,95
 runner.ts    |     100 |    66.66 |     100 |     100 | 29,109
--------------|---------|----------|---------|---------|-------------------
```

### Coverage Highlights

- **100% function coverage** - All functions are tested
- **96.36% statement coverage** - Nearly all code paths covered
- **87.17% branch coverage** - Most conditional branches tested

### Uncovered Lines

The uncovered lines are mostly:

- Alternative config file search paths (lines 20-24, 30 in config.ts)
- Error handling branches that are hard to trigger in tests
- Branch coverage for timeouts and edge cases

## Running Tests

### Run all tests

```bash
npm test
```

### Run tests in watch mode

```bash
npm run test:watch
```

### Run tests with coverage

```bash
npm run test:coverage
```

## Test Architecture

### Mocking Strategy

- **OpenAI Client**: Mocked to avoid real API calls
- **File System**: Real files created in `tests/fixtures/` directory
- **Glob**: Mocked to control test file discovery
- **Console**: Spied on to verify output
- **Chalk**: Mocked to simplify color output testing

### Fixtures

Test fixtures are created dynamically during test runs and cleaned up afterwards:

- Config files (`.config.js`)
- YAML test files (`.test.yaml`)
- All stored in `tests/fixtures/` (ignored by git)

### Best Practices

- Each test suite is independent
- Proper setup (`beforeEach`) and cleanup (`afterAll`)
- Descriptive test names
- Testing both success and error cases
- Testing edge cases (timeouts, malformed data, missing fields)

## Adding New Tests

When adding new features:

1. Create test file in `tests/` directory
2. Follow existing naming convention: `feature.test.ts`
3. Use Jest's `describe` and `it` blocks
4. Mock external dependencies
5. Clean up fixtures in `afterAll`
6. Run tests to ensure they pass
7. Verify coverage hasn't decreased

Example:

```typescript
import { MyFeature } from "../src/my-feature";

describe("MyFeature", () => {
  it("should do something", () => {
    const feature = new MyFeature();
    expect(feature.doSomething()).toBe(true);
  });
});
```

## Continuous Integration

Tests should be run in CI/CD pipelines:

```yaml
# Example GitHub Actions workflow
- name: Run tests
  run: npm test

- name: Check coverage
  run: npm run test:coverage
```

## Future Testing Goals

- [ ] Increase branch coverage to >90%
- [ ] Add integration tests with real LLM calls (opt-in)
- [ ] Add E2E tests using the CLI
- [ ] Performance benchmarks
- [ ] Snapshot testing for output formatting