# Test Runner MCP A Model Context Protocol (MCP) server for running and parsing test results from multiple testing frameworks. This server provides a unified interface for executing tests and processing their outputs, supporting: - Bats (Bash Automated Testing System) - Pytest (Python Testing Framework) - Flutter Tests - Jest (JavaScript Testing Framework) - Go Tests - Rust Tests (Cargo test) - Generic (for arbitrary command execution) Test Runner MCP server ## Installation ```bash npm install test-runner-mcp ``` ## Prerequisites The following test frameworks need to be installed for their respective test types: - Bats: `apt-get install bats` or `brew install bats` - Pytest: `pip install pytest` - Flutter: Follow [Flutter installation guide](https://flutter.dev/docs/get-started/install) - Jest: `npm install --save-dev jest` - Go: Follow [Go installation guide](https://go.dev/doc/install) - Rust: Follow [Rust installation guide](https://www.rust-lang.org/tools/install) ## Usage ### Configuration Add the test-runner to your MCP settings (e.g., in `claude_desktop_config.json` or `cline_mcp_settings.json`): ```json { "mcpServers": { "test-runner": { "command": "node", "args": ["/path/to/test-runner-mcp/build/index.js"], "env": { "NODE_PATH": "/path/to/test-runner-mcp/node_modules", // Flutter-specific environment (required for Flutter tests) "FLUTTER_ROOT": "/opt/homebrew/Caskroom/flutter/3.27.2/flutter", "PUB_CACHE": "/Users/username/.pub-cache", "PATH": "/opt/homebrew/Caskroom/flutter/3.27.2/flutter/bin:/usr/local/bin:/usr/bin:/bin" } } } } ``` Note: For Flutter tests, ensure you replace: - `/opt/homebrew/Caskroom/flutter/3.27.2/flutter` with your actual Flutter installation path - `/Users/username/.pub-cache` with your actual pub cache path - Update PATH to include your system's actual paths You can find these values by running: ```bash # Get Flutter root flutter --version # Get pub cache path echo $PUB_CACHE # or default to $HOME/.pub-cache # Get Flutter binary path which flutter ``` ### Running Tests Use the `run_tests` tool with the following parameters: ```json { "command": "test command to execute", "workingDir": "working directory for test execution", "framework": "bats|pytest|flutter|jest|go|rust|generic", "outputDir": "directory for test results", "timeout": "test execution timeout in milliseconds (default: 300000)", "env": "optional environment variables", "securityOptions": "optional security options for command execution" } ``` Example for each framework: ```json // Bats { "command": "bats test/*.bats", "workingDir": "/path/to/project", "framework": "bats", "outputDir": "test_reports" } // Pytest { "command": "pytest test_file.py -v", "workingDir": "/path/to/project", "framework": "pytest", "outputDir": "test_reports" } // Flutter { "command": "flutter test test/widget_test.dart", "workingDir": "/path/to/project", "framework": "flutter", "outputDir": "test_reports", "FLUTTER_ROOT": "/opt/homebrew/Caskroom/flutter/3.27.2/flutter", "PUB_CACHE": "/Users/username/.pub-cache", "PATH": "/opt/homebrew/Caskroom/flutter/3.27.2/flutter/bin:/usr/local/bin:/usr/bin:/bin" } // Jest { "command": "jest test/*.test.js", "workingDir": "/path/to/project", "framework": "jest", "outputDir": "test_reports" } // Go { "command": "go test ./...", "workingDir": "/path/to/project", "framework": "go", "outputDir": "test_reports" } // Rust { "command": "cargo test", "workingDir": "/path/to/project", "framework": "rust", "outputDir": "test_reports" } // Generic (for arbitrary commands, CI/CD tools, etc.) { "command": "act -j build", "workingDir": "/path/to/project", "framework": "generic", "outputDir": "test_reports" } // Generic with security overrides { "command": "sudo docker-compose -f docker-compose.test.yml up", "workingDir": "/path/to/project", "framework": "generic", "outputDir": "test_reports", "securityOptions": { "allowSudo": true } } ``` ### Security Features The test-runner includes built-in security features to prevent execution of potentially harmful commands, particularly for the `generic` framework: 1. **Command Validation** - Blocks `sudo` and `su` by default - Prevents dangerous commands like `rm -rf /` - Blocks file system write operations outside of safe locations 2. **Environment Variable Sanitization** - Filters out potentially dangerous environment variables - Prevents overriding critical system variables - Ensures safe path handling 3. **Configurable Security** - Override security restrictions when necessary via `securityOptions` - Fine-grained control over security features - Default safe settings for standard test usage Security options you can configure: ```json { "securityOptions": { "allowSudo": false, // Allow sudo commands "allowSu": false, // Allow su commands "allowShellExpansion": true, // Allow shell expansion like $() or backticks "allowPipeToFile": false // Allow pipe to file operations (> or >>) } } ``` ### Flutter Test Support The test runner includes enhanced support for Flutter tests: 1. Environment Setup - Automatic Flutter environment configuration - PATH and PUB_CACHE setup - Flutter installation verification 2. Error Handling - Stack trace collection - Assertion error handling - Exception capture - Test failure detection 3. Output Processing - Complete test output capture - Stack trace preservation - Detailed error reporting - Raw output preservation ### Rust Test Support The test runner provides specific support for Rust's `cargo test`: 1. Environment Setup - Automatically sets RUST_BACKTRACE=1 for better error messages 2. Output Parsing - Parses individual test results - Captures detailed error messages for failed tests - Identifies ignored tests - Extracts summary information ### Generic Test Support For CI/CD pipelines, GitHub Actions via `act`, or any other command execution, the generic framework provides: 1. Automatic Output Analysis - Attempts to segment output into logical blocks - Identifies section headers - Detects pass/fail indicators - Provides reasonable output structure even for unknown formats 2. Flexible Integration - Works with arbitrary shell commands - No specific format requirements - Perfect for integration with tools like `act`, Docker, and custom scripts 3. Security Features - Command validation to prevent harmful operations - Can be configured to allow specific elevated permissions when necessary ## Output Format The test runner produces structured output while preserving complete test output: ```typescript interface TestResult { name: string; passed: boolean; output: string[]; rawOutput?: string; // Complete unprocessed output } interface TestSummary { total: number; passed: number; failed: number; duration?: number; } interface ParsedResults { framework: string; tests: TestResult[]; summary: TestSummary; rawOutput: string; // Complete command output } ``` Results are saved in the specified output directory: - `test_output.log`: Raw test output - `test_errors.log`: Error messages if any - `test_results.json`: Structured test results - `summary.txt`: Human-readable summary ## Development ### Setup 1. Clone the repository 2. Install dependencies: ```bash npm install ``` 3. Build the project: ```bash npm run build ``` ### Running Tests ```bash npm test ``` The test suite includes tests for all supported frameworks and verifies both successful and failed test scenarios. ### CI/CD The project uses GitHub Actions for continuous integration: - Automated testing on Node.js 18.x and 20.x - Test results uploaded as artifacts - Dependabot configured for automated dependency updates ## Contributing 1. Fork the repository 2. Create your feature branch 3. Commit your changes 4. Push to the branch 5. Create a Pull Request ## License This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.