# AI Testing Suite
### Multi-Agent LangGraph Orchestration for Automated Testing
[](https://www.npmjs.com/package/ai-testing-suite)
[](https://www.npmjs.com/package/ai-testing-suite)
[](https://nodejs.org/)
[](https://www.typescriptlang.org/)
[](https://langchain-ai.github.io/langgraphjs/)
[](https://vitest.dev/)
[](https://jestjs.io/)
[](https://playwright.dev/)
[](https://owasp.org/www-project-top-ten/)
[](https://opensource.org/licenses/MIT)
**8 specialized AI agents analyze your project, generate tests, review quality, execute tests, scan for vulnerabilities, and document everything — fully automated.**
[Installation](#-installation) · [Quick Start](#-quick-start) · [Agents](#-the-8-agents) · [CLI Modes](#-cli-modes) · [Security](#-security-scanning) · [Reports](#-reports)
---
## Overview
AI Testing Suite is a **LangGraph-based multi-agent pipeline** that automatically tests Node.js/TypeScript projects. It analyzes project structure and code, plans a test strategy, generates tailored test files, reviews their quality, executes them, scans for security vulnerabilities, and creates comprehensive reports.
```
Scanner → Analyzer → Strategist → Writer → Reviewer ──→ Runner → Security → Reporter
│ ↑
└── (Retry) ───┘
```
### Why AI Testing Suite?
- **Extremely easy to integrate** — A single command is all it takes to integrate the suite into any existing project
- **100% project-tailored** — Tests are generated based on deep code analysis, not from templates
- **8 specialized agents** — Each agent has a clearly defined role in the pipeline
- **OWASP Top 10 + Zero-Day** — Comprehensive security scanning with 100+ vulnerability patterns
- **Console-driven** — Interactive CLI menu with 8 modes
- **Production-readiness assessment** — Clear verdict on whether the project is production-ready
---
## Project Structure
```
ai-testing-suite/
├── src/
│ ├── index.ts # CLI Entry Point
│ ├── cli.ts # Interactive Console Menu
│ ├── config.ts # Configuration Management
│ ├── setup.ts # Quick Setup for Existing Projects
│ ├── types.ts # Complete TypeScript Type System
│ │
│ ├── agents/ # 8 Specialized Agents
│ │ ├── scanner.agent.ts # Agent 1: Project Structure Scanner
│ │ ├── analyzer.agent.ts # Agent 2: Deep Code Analysis
│ │ ├── strategist.agent.ts # Agent 3: Test Strategy Planner
│ │ ├── writer.agent.ts # Agent 4: Test Generator
│ │ ├── reviewer.agent.ts # Agent 5: Test Quality Reviewer
│ │ ├── runner.agent.ts # Agent 6: Test Executor
│ │ ├── security.agent.ts # Agent 7: Security Auditor
│ │ └── reporter.agent.ts # Agent 8: Report Generator
│ │
│ ├── graph/ # LangGraph Orchestration
│ │ ├── workflow.ts # Pipeline Definition & Execution
│ │ ├── nodes.ts # Graph Nodes & Conditional Edges
│ │ └── state.ts # State Channel Management
│ │
│ └── utils/ # Utility Functions
│ ├── parser.ts # Code Parser (Functions, Classes, Patterns)
│ ├── file-utils.ts # File System Operations
│ └── logger.ts # Colored Console Output
│
├── tests/ # Generated Tests (Output)
│ ├── unit/ # Unit Tests
│ ├── integration/ # Integration Tests
│ ├── e2e/ # End-to-End Tests
│ ├── security/ # Security Tests
│ └── performance/ # Performance Tests
│
├── reports/ # Generated Reports (Output)
│ ├── report-*.json # Machine-readable
│ ├── report-*.md # Markdown Report
│ ├── report-*.html # Visual HTML Report
│ └── agent-log-*.md # Agent Activity Log
│
├── package.json
├── tsconfig.json
└── .env.example # Configuration Template
```
---
## Installation
### Install via npm (recommended)
```bash
npm install --save-dev ai-testing-suite
```
Then run the setup wizard:
```bash
npx ai-test-setup
```
This will:
- Create test directories (`tests/unit`, `tests/integration`, `tests/e2e`, `tests/security`, `tests/performance`)
- Install the right dependencies (Vitest or Jest, Playwright, etc.)
- Add npm scripts to your `package.json`
- Create `.env.example` with configuration template
### Quick Start in 3 Steps
```bash
# 1. Install
npm install --save-dev ai-testing-suite
# 2. Configure
cp .env.example .env
# Enter your API key (OpenAI or Anthropic) or use Ollama locally
# 3. Run
npx ai-test --mode full
```
### Setup Flags
```bash
npx ai-test-setup # Default: Vitest + Playwright
npx ai-test-setup --jest # Use Jest instead of Vitest
npx ai-test-setup --no-playwright # Skip Playwright installation
```
### Alternative: Clone from GitHub
```bash
git clone https://github.com/zurd46/AI-Testing-Suite.git
cd AI-Testing-Suite
npm install
npm run setup -- /path/to/project
```
---
## Quick Start
### Interactive Mode (recommended)
```bash
npm start
# or
npx ts-node src/index.ts
```
Displays the interactive menu:
```
╔══════════════════════════════════════════════════════════════╗
║ AI TESTING SUITE ║
║ Multi-Agent LangGraph Orchestration ║
╠══════════════════════════════════════════════════════════════╣
║ 8 Agents | OWASP Top 10 | Zero-Day Scans | 100% Coverage ║
╚══════════════════════════════════════════════════════════════╝
Select a mode:
1) Full Pipeline - Complete analysis, tests & security
2) Analyze Only - Project structure & code analysis
3) Generate Tests - Analysis + create & review tests
4) Run Tests - Execute existing tests
5) Security Scan - Security audit only
6) Custom Pipeline - Choose your own agent combination
7) Generate Report - Create report
8) Configuration - View/modify settings
0) Exit
```
### Direct Modes
```bash
# Full pipeline
npm run full -- --path /path/to/project
# Analysis only
npm run analyze -- --path /path/to/project
# Generate tests only
npm run generate -- --path /path/to/project
# Security scan only
npm run security -- --path /path/to/project
# Report only
npm run report -- --path /path/to/project
```
### CLI Options
```bash
npx ts-node src/index.ts \
--mode full \
--path /path/to/project \
--output ./tests \
--reports ./reports \
--provider openai \
--model gpt-4o \
--depth deep \
--verbose
```
| Option | Description | Default |
|--------|-------------|---------|
| `-m, --mode` | Mode: `full\|analyze\|generate\|run\|security\|report\|interactive` | `interactive` |
| `-p, --path` | Path to target project | `./` |
| `-o, --output` | Test output directory | `./tests` |
| `-r, --reports` | Report directory | `./reports` |
| `--provider` | LLM Provider: `openai\|anthropic\|openai_compatible\|ollama` | `openai` |
| `--model` | LLM Model | `gpt-4o` |
| `--runner` | Test runner: `vitest\|jest\|node` | `vitest` (auto-detected) |
| `--e2e` | E2E runner: `playwright\|supertest\|none` | `playwright` (auto-detected) |
| `--coverage` | Coverage tool: `v8\|c8\|istanbul` | `v8` (auto-detected) |
| `--depth` | Security scan depth: `basic\|standard\|deep` | `deep` |
| `-v, --verbose` | Verbose output | `true` |
| `--no-security` | Disable security scanning | - |
| `--no-zero-day` | Disable zero-day checks | - |
---
## The 8 Agents
### Agent 1: Scanner
> Scans the complete project structure
- Detects framework (Next.js, NestJS, Express, React, Vue, Angular, ...)
- Identifies entry points, config files, package manager
- Counts files, lines, languages
- Detects features (TypeScript, Prisma, GraphQL, WebSockets, Redis, ...)
### Agent 2: Analyzer
> Deep code analysis of every source file
- Parses functions, classes, interfaces, variables
- Detects API endpoints (Express, NestJS Decorators)
- Detects database operations (Query, Insert, Update, Delete)
- Builds dependency graph with circular dependency detection
- Detects design patterns (Singleton, Factory, Observer, Middleware, ...)
- Calculates cyclomatic complexity
- Detects error handling patterns
### Agent 3: Strategist
> Plans the optimal test strategy
- Creates unit test plans per function/class
- Plans API integration tests per endpoint
- Plans database integration tests
- Plans E2E tests per resource
- Plans security tests (OWASP-based)
- Plans performance tests
- Prioritizes by criticality and complexity
Generates test cases for each category:
`happy-path` · `edge-case` · `error-handling` · `boundary` · `null-undefined` · `type-safety` · `concurrency` · `security` · `performance` · `regression`
### Agent 4: Writer
> Generates tailored test files
- Writes unit tests with correct imports and mocks
- Writes integration tests with Supertest
- Writes E2E tests with Playwright (browser) or Supertest (API)
- Writes security tests with real attack payloads
- Writes performance tests (latency, concurrency, memory)
- Generates runner-specific config (vitest.config.ts, jest.config.ts, playwright.config.ts)
- Auto-adapts syntax per runner (vi.fn() vs jest.fn(), imports vs globals)
- Creates directory structure automatically
### Agent 5: Reviewer
> Reviews generated tests for quality
- Checks test structure (describe, it, beforeEach)
- Counts and evaluates assertions
- Checks test isolation (mock reset)
- Checks edge case coverage
- Checks error handling tests
- Checks mock usage and spy assertions
- Checks async patterns
- Detects placeholder assertions
- Calculates quality score (0-100)
- **Conditional Edge:** If score < threshold, pipeline loops back to the Writer
### Agent 6: Runner
> Executes all tests and collects results
- Supports **Vitest** (default, fastest), **Jest**, and **Node.js built-in test runner**
- Runs **Playwright** for E2E browser tests
- Runs tests grouped by type (Unit, Integration, E2E, Security, Performance)
- Parses JSON output from each runner
- Collects errors with test name, message, stack trace
- Auto-detects installed test runner from target project
- Dry run when runner is not installed
- Calculates success rate
### Agent 7: Security Auditor
> Comprehensive security audit
**Vulnerability Checks (100+ Patterns):**
| Category | Checks |
|----------|--------|
| SQL Injection | Template Literals, String Concatenation, Raw Queries |
| XSS | innerHTML, dangerouslySetInnerHTML, document.write, Reflected Input |
| Command Injection | exec(), execSync(), spawn() with user input |
| Path Traversal | readFile/writeFile with user input, path.join manipulation |
| Prototype Pollution | Object.assign, Spread, __proto__, constructor |
| SSRF | fetch/axios/got with user-controlled URL |
| CSRF | State-changing endpoints without token |
| ReDoS | User input in RegExp, nested quantifiers |
| Hardcoded Secrets | Passwords, API keys, private keys, connection strings |
| Insecure Randomness | Math.random() for tokens/keys |
| Broken Auth | JWT "none" algorithm, weak hashes, TLS bypass |
| Information Disclosure | Stack traces in responses, sensitive data in logs |
| Zero-Day Patterns | process.binding, dynamic Function(), Proxy manipulation, Buffer.allocUnsafe |
**Additionally:**
- Dependency vulnerability audit (known CVEs in npm packages)
- OWASP Top 10 compliance check (A01-A10:2021)
- Code smell detection (TODO, empty catch blocks, console.log in production)
- Risk score calculation (0-100)
### Agent 8: Reporter
> Creates comprehensive documentation
- **JSON Report** — Machine-readable, for CI/CD integration
- **Markdown Report** — Human-readable, for code reviews
- **HTML Report** — Visual report with charts and colors
- **Agent Log** — Chronological log of all agent actions
- **Production-readiness assessment** — YES/NO with reasoning
- **Recommendations** — Concrete steps for improvement
---
## CLI Modes
| Mode | Description | Agents |
|------|-------------|--------|
| `full` | Complete pipeline | All 8 |
| `analyze` | Analysis only | Scanner → Analyzer |
| `generate` | Generate tests | Scanner → Analyzer → Strategist → Writer → Reviewer |
| `run` | Run tests | Scanner → Analyzer → Strategist → Writer → Runner |
| `security` | Security scan | Scanner → Analyzer → Security → Reporter |
| `report` | Generate report | Reporter |
| `interactive` | Interactive menu | Selectable |
| Custom | Custom combination | Freely selectable (e.g. 1,2,7) |
---
## Supported Test Runners
AI Testing Suite auto-detects the test runner from your project. You can also override via CLI or `.env`.
| Runner | Type | Speed | Best For |
|--------|------|-------|----------|
| **Vitest** (default) | Unit/Integration | Fastest | Modern TypeScript projects, ESM |
| **Jest** | Unit/Integration | Fast | Existing Jest projects, CJS |
| **Node.js test runner** | Unit | Fast | Zero-dependency, Node >= 18 |
| **Playwright** (default) | E2E/Browser | Medium | Frontend/Fullstack, cross-browser |
| **Supertest** | E2E/API | Fast | Backend API testing |
### Coverage Tools
| Tool | Speed | Integration |
|------|-------|-------------|
| **v8** (default) | Fastest | Native V8 engine coverage |
| **c8** | Fast | V8 coverage via CLI wrapper |
| **istanbul** | Standard | Traditional instrumentation |
### Auto-Detection
The Scanner agent automatically detects your project's existing test tooling:
```
vitest in package.json → uses Vitest
jest/ts-jest → uses Jest
@playwright/test → uses Playwright for E2E
supertest → uses Supertest for E2E
c8 → uses c8 for coverage
nyc/istanbul → uses Istanbul for coverage
```
Override with CLI: `npx ai-test --mode full --runner jest --e2e playwright --coverage v8`
---
## Security Scanning
### OWASP Top 10 (2021) Coverage
| # | Category | Status |
|---|----------|--------|
| A01 | Broken Access Control | Covered |
| A02 | Cryptographic Failures | Covered |
| A03 | Injection | Covered |
| A04 | Insecure Design | Covered |
| A05 | Security Misconfiguration | Covered |
| A06 | Vulnerable Components | Covered |
| A07 | Auth Failures | Covered |
| A08 | Software Integrity | Covered |
| A09 | Logging Failures | Covered |
| A10 | SSRF | Covered |
### Zero-Day Pattern Detection
Detection of patterns that may indicate potential zero-day exploits:
- `process.binding()` — Access to internal Node.js APIs
- `Function()` with user input — Dynamic code execution
- `Proxy` with user input — Handler manipulation
- `Reflect` API with user input — Metaprogramming attacks
- Dynamic `import()` with user input — Module injection
- `constructor` bracket notation — Sandbox escape
- `Buffer.allocUnsafe()` — Uninitialized memory (memory leak)
- `globalThis`/`global[]` — Global scope manipulation
---
## Reports
### HTML Report
The HTML report provides a visual overview with:
- Overall status (Production-ready YES/NO)
- Score cards (Tests, Security, Quality)
- Vulnerability table with severity colors
- OWASP compliance status
- Recommendations
### Markdown Report
Contains:
- Summary with metrics
- Project structure details
- Code analysis results
- Generated tests by type
- Test results with errors
- Security report with all vulnerabilities
- OWASP status
- Recommendations
### JSON Report
Complete, machine-readable report for CI/CD integration:
```json
{
"projectName": "my-app",
"timestamp": "2025-01-15T10:30:00Z",
"summary": {
"totalTests": 142,
"totalPassed": 138,
"totalFailed": 4,
"securityScore": 85,
"qualityScore": 92,
"productionReady": false,
"recommendations": [...]
},
"security": {
"vulnerabilities": [...],
"owaspChecks": [...],
"overallRisk": "medium"
}
}
```
---
## Configuration
### Environment Variables (.env)
```bash
# LLM Provider
OPENAI_API_KEY=sk-... # OpenAI API Key
ANTHROPIC_API_KEY=sk-ant-... # Or Anthropic API Key
LLM_PROVIDER=openai # openai | anthropic | openai_compatible | ollama
LLM_MODEL=gpt-4o # Model name (ollama default: llama3.2)
LLM_TEMPERATURE=0.1 # Low for consistent tests
LLM_MAX_TOKENS=4096
# Paths
TARGET_PROJECT_PATH=./ # Target project
TESTS_OUTPUT_DIR=./tests # Test output
REPORTS_OUTPUT_DIR=./reports # Report output
# Test Runner Configuration
TEST_RUNNER=vitest # vitest (default, fastest) | jest | node
E2E_RUNNER=playwright # playwright (default) | supertest | none
COVERAGE_TOOL=v8 # v8 (default, native) | c8 | istanbul
# Security
SECURITY_SCAN_DEPTH=deep # basic | standard | deep
CHECK_ZERO_DAY=true
CHECK_OWASP_TOP_10=true
CHECK_DEPENDENCIES=true
# Agent
MAX_AGENT_ITERATIONS=10
AGENT_VERBOSE=true
```
---
## Supported Frameworks
| Framework | Type | Detection |
|-----------|------|-----------|
| **Next.js** | Fullstack | Automatic |
| **NestJS** | Backend | Automatic |
| **Express** | Backend | Automatic |
| **Fastify** | Backend | Automatic |
| **Koa** | Backend | Automatic |
| **Hapi** | Backend | Automatic |
| **React** | Frontend | Automatic |
| **Vue** | Frontend | Automatic |
| **Angular** | Frontend | Automatic |
| **Svelte** | Frontend | Automatic |
| **Nuxt** | Fullstack | Automatic |
| **Remix** | Fullstack | Automatic |
| **Electron** | Fullstack | Automatic |
| **Node.js** | Library/CLI | Fallback |
### Detected Features
TypeScript · Prisma · TypeORM · Mongoose · Sequelize · GraphQL · WebSockets · Redis · Passport · JWT · bcrypt · Helmet · CORS · Joi/Zod/Yup · Winston/Pino · Bull/BullMQ · Swagger/OpenAPI · Vitest · Playwright · Stryker · c8 · ESLint Security · PactumJS
---
## Prerequisites
- **Node.js** >= 18.0.0
- **npm**, **yarn**, or **pnpm**
- **TypeScript** (recommended, JavaScript is also supported)
- **API Key** for OpenAI or Anthropic (for LLM-powered features), or **Ollama** for local inference (no API key needed)
---
## Example Output
```
======================================================================
AI TESTING SUITE - FULL MODE
======================================================================
-- Project: /home/user/my-app
-- Pipeline: scanner -> analyzer -> strategist -> writer -> reviewer -> runner -> security -> reporter
-- 8 agents will be executed
>> [SCAN] Agent started...
[SCAN] 47 files found
[SCAN] 12 directories found
[SCAN] Framework detected: Express (backend)
OK [SCAN] Agent completed (0.3s)
>> [ANALYZE] Agent started...
[ANALYZE] 23 modules analyzed
[ANALYZE] 45 functions, 8 classes
[ANALYZE] 12 API endpoints found
OK [ANALYZE] Agent completed (1.2s)
>> [STRATEGY] Agent started...
Test Type | Plans | Test Cases
Unit Tests | 18 | 72
Integration | 14 | 42
E2E Tests | 4 | 8
Security Tests | 2 | 16
Performance | 1 | 3
TOTAL | 39 | 141
OK [STRATEGY] Agent completed (0.8s)
>> [WRITE] Agent started...
[WRITE] 18 unit test files written
[WRITE] 14 integration test files written
[WRITE] 4 E2E test files written
OK [WRITE] Agent completed (2.1s)
>> [REVIEW] Agent started...
36/39 passed (Score: 87.3)
OK [REVIEW] Agent completed (0.5s)
>> [RUN] Agent started...
PASS auth.test.ts (124ms)
PASS users.test.ts (89ms)
FAIL payment.test.ts (203ms)
OK [RUN] Agent completed (12.4s)
>> [SECURITY] Agent started...
[CRITICAL] hardcoded-credentials: Hardcoded API key (config.ts)
[HIGH] sql-injection: Raw SQL with Concatenation (users.service.ts)
[MEDIUM] xss: innerHTML assignment (dashboard.tsx)
Security Score: 72/100
OK [SECURITY] Agent completed (1.8s)
>> [REPORT] Agent started...
Reports saved to: ./reports
OK [REPORT] Agent completed (0.4s)
======================================================================
TEST RESULTS
======================================================================
Project: my-app
Framework: Express (backend)
Total Tests: 141
Passed: 137
Failed: 4
Security Score: 72/100
Quality Score: 87/100
!! NOT PRODUCTION-READY - Please review recommendations:
?? -> Fix 4 failing tests
?? -> Address security vulnerabilities
?? -> Fix 1 critical vulnerability immediately!
```
---
## License
MIT
---
**Built with LangGraph, TypeScript, and the power of 8 AI agents.**