# Sonar-Forge

**JMeter test orchestrator** - YAML-driven JMX modification and execution.

Inspired by [Blazemeter Taurus](https://gettaurus.org/), Sonar-Forge has been built as a simplified and streamlined alternative to work within Sonar's performance testing framework and ecosystem. It enables clean, version-controlled test and tool configuration specific to each test without needting to modify JMX files and Jmeter properties, or embedding complex properties and variables within each script.

## Overview

Sonar-Forge separates test configuration from test scripts by:
- Reading existing JMX files to generate an initial YAML configuration specifying test and thread properties/configuration
- Users are able to extend the configuration, or make several versions (e.g. smoke test, peak test, soak test, etc.) by updating the initial configuration without needing to modify the JMX itself.
- Executing tests with automatic result collection and error logging
- Refreshing the YAML configuration file when the JMX gets updated (e.g. new thread groups get added, etc.)

## Installation

```bash
npm install -g sonar-forge
```

## Quick Start

### 1. Initialize from existing JMX

```bash
sonar-forge init my-test.jmx                   # Config named after JMX
sonar-forge init my-test.jmx smoke-test        # Custom config name
```

This creates a project structure:
```
├── config/
│   └── smoke-test.yaml       # Test configuration
├── scripts/
│   └── my-test.jmx           # Original JMX file
├── data/                     # Test data files
├── logs/                     # JMeter logs
└── results/                  # Test results
```

### 2. Edit configuration

Edit `config/my-test.yaml`:

```yaml
test_script: ../scripts/my-test.jmx

output:
  results_dir: ../results
  logs_dir: ../logs

jmeter:
  memory:
    heap_min: 512m
    heap_max: 4g
    metaspace_max: 512m

  properties:
    jmeter.save.saveservice.output_format: csv
    jmeter.save.saveservice.timestamp_format: ms
    # ... additional JMeter properties

error_log:
  enabled: true
  filename: errors.xml

thread_groups:
  - name: Load Test
    enabled: true
    threads: 50
    rampup: 10
    iterations: -1
    duration: 300

    lifecycle:
      EnableLifeCycleController: true
      GenerateParentSample: true
      UseFixedPacing: true
      PacingInterval: 5000
```

### 3. Run your test

```bash
cd my-test
sonar-forge run config/my-test.yaml
```

Results are saved with timestamps:
- `results/results_2025-01-15_14-30-00.jtl` - CSV format with all metrics
- `results/errors_2025-01-15_14-30-00.xml` - XML format with full error details

## Commands

### `init <jmx-file> [config-name]`
Generate project from existing JMX file. Optionally specify a custom config filename.

```bash
sonar-forge init my-test.jmx                    # Creates config/my-test.yaml
sonar-forge init my-test.jmx smoke-test          # Creates config/smoke-test.yaml
sonar-forge init my-test.jmx peak-test.yaml      # Creates config/peak-test.yaml
```

### `validate <config-file>`
Validate configuration file.

```bash
sonar-forge validate config/my-test.yaml
```

### `build <config-file>`
Build effective JMX without running test.

```bash
sonar-forge build config/my-test.yaml
sonar-forge build config/my-test.yaml --dry-run
sonar-forge build config/my-test.yaml --run-id "test-001"  # Set unique run ID
```

### `run <config-file>`
Build and execute JMeter test.

```bash
sonar-forge run config/my-test.yaml
sonar-forge run config/my-test.yaml -v                    # Verbose output
sonar-forge run config/my-test.yaml --build-only          # Build only, don't run
sonar-forge run config/my-test.yaml --keep-jmx            # Keep effective JMX after run
sonar-forge run config/my-test.yaml --run-id "test-001"   # Set unique run ID
sonar-forge run config/my-test.yaml --lgid "lg-01"       # Load generator subdirectory
```

### `refresh [config-file]`
Update configuration from modified JMX file.

```bash
cd my-test

# Single config file (auto-detect)
sonar-forge refresh              # Merge mode (default) - preserves config
sonar-forge refresh --replace    # Replace mode - overwrites with JMX data

# Multiple config files (specify which one)
sonar-forge refresh test1.yaml
sonar-forge refresh test2.yaml --replace

# Process all config files
sonar-forge refresh --all        # Merge mode for all configs
sonar-forge refresh --all --replace   # Replace mode for all configs

# Verbose output
sonar-forge refresh -v
```

**Merge mode**: Preserves your configuration values, adds new thread groups from JMX.

**Replace mode**: Completely replaces configuration with current JMX state.

**--all flag**: Processes all YAML config files in the `config/` directory, showing a summary of successes and failures.

## Configuration

### Logging Level

Control JMeter's console and log file verbosity:

```yaml
# JMeter logging level (DEBUG, INFO, WARN, ERROR)
log_level: WARN
```

**Available levels** (from most to least verbose):
- `DEBUG` - Detailed debugging information
- `INFO` - General informational messages (default if not specified)
- `WARN` - Warning messages only
- `ERROR` - Error messages only

### Thread Groups

```yaml
thread_groups:
  - name: Thread Group Name       # Must match JMX testname
    enabled: true                 # Enable/disable thread group
    threads: 50                   # Number of threads
    rampup: 10                    # Ramp-up period in seconds
    iterations: -1                # Number of iterations (-1 = infinite)
    duration: 300                 # Thread lifetime in seconds (null to disable)
```

**Execution modes**:
- **Duration only**: Set duration, leave iterations at -1 (runs until time limit)
- **Iterations only**: Set iterations, set duration to null (runs until iteration count)
- **Both**: Stops when either limit is reached
- **Infinite**: iterations: -1, duration: null (runs forever)

### Lifecycle Controller

For tests using Sonar's LifecycleController:

```yaml
thread_groups:
  - name: My Test
    # ... thread configuration
    lifecycle:
      EnableLifeCycleController: true
      GenerateParentSample: true
      TroubleshootingMode: false
      UseFixedPacing: true
      PacingInterval: 5000
      PacingRandomizationPct: 5
```

### Run ID

The `run_id` provides a unique identifier for test runs that is automatically applied to all backend listeners. This enables tracking and correlating metrics across multiple backend systems.

**Three-tier precedence** (highest to lowest):
1. **CLI flag**: `--run-id "my-run-id"`
2. **YAML config**: `run_id: "my-run-id"`
3. **Auto-generated**: UUID automatically generated if not specified

**Examples**:

```bash
# CLI flag (highest precedence)
sonar-forge run config/my-test.yaml --run-id "production-2025-01-15"

# YAML config
cat config/my-test.yaml
run_id: "test-run-001"

# Auto-generated (no configuration needed)
# Automatically generates UUID like "550e8400-e29b-41d4-a716-446655440000"
```

The run_id is automatically injected into the `runId` property of all enabled backend listeners, overriding any values configured in the YAML backend listener properties.

### Backend Listeners

Sonar-Forge automatically detects and configures backend listeners for real-time test result reporting.

#### Supported Listeners

**TimescaleDB Listener** (recommended for Sonar framework):
```yaml
backend_listeners:
  - name: Sonar - TimescaleDB Listener
    type: timescaledb
    enabled: true

    properties:
      dbHost: timescaledb.example.com
      dbPort: 6432
      dbName: tsdb
      dbUser: username
      dbPassword: password
      runId: test-run
      reportLabel: ""
      batchSize: 5000
      flushInterval: 4000
      poolSize: 20
      errorThreshold: 5
      sampleTableName: performance_metrics_full
      threadTableName: performance_threads_full
      recordSubSamples: true
      saveResponseBodyOfFailures: true
      responseBodyLength: 2000
```

**InfluxDB v2 Listener**:
```yaml
backend_listeners:
  - name: InfluxDBv2 Backend Listener
    type: influxdb2
    enabled: false

    properties:
      testName: Test
      nodeName: Test-Node
      runId: test-run
      influxDBURL: "http://localhost:8086/"
      influxDBToken: "your-token-here"
      influxDBOrganization: performance_testing
      influxDBBucket: jmeter
      influxDBFlushInterval: 4000
      influxDBMaxBatchSize: 2000
      influxDBThresholdError: 5
      samplersList: .*
      useRegexForSamplerList: true
      recordSubSamples: true
      saveResponseBodyOfFailures: false
      responseBodyLength: 2000
```

**Spectre Integration Listener**:
```yaml
backend_listeners:
  - name: Sonar - Spectre Integration
    type: spectre
    enabled: true

    properties:
      spectreUrl: "https://spectre2.example.com/api/jmeter/teardown"
      Application: MyApp
      Environment: Performance
      RunID: test-run
      Organisation: MyOrg
      RunType: Shakeout
      FriendlyName: "Smoke test run"
```

**Other Listeners**:
For other backend listener types, only name, classname, and enabled status are captured:
```yaml
backend_listeners:
  - name: Custom Backend Listener
    type: other
    classname: com.example.CustomListener
    enabled: false
```

**Warning**: If no TimescaleDB or InfluxDB2 listeners are detected, a warning comment appears in the YAML:
```yaml
# WARNING: No real-time reporting listeners (TimescaleDB/InfluxDB2) detected in this test
backend_listeners: []
```

### JVM Memory

Configure JVM heap and metaspace for JMeter. These are passed via `JVM_ARGS` to the JMeter process.

```yaml
jmeter:
  memory:
    heap_min: 512m
    heap_max: 4g
    metaspace_max: 512m
```

If not specified, JMeter uses its own defaults. Any existing `JVM_ARGS` environment variable is preserved and appended.

### JMeter Properties

Configure JTL output format and other JMeter settings:

```yaml
jmeter:
  properties:
    jmeter.save.saveservice.output_format: csv
    jmeter.save.saveservice.time: true
    jmeter.save.saveservice.latency: true
    jmeter.save.saveservice.bytes: true
    # ... additional properties
```

### Error Logging

Capture full request/response details for failed samples:

```yaml
error_log:
  enabled: true
  filename: errors.xml
```

Generates timestamped XML files with complete error diagnostics.

### Output Configuration

```yaml
output:
  results_dir: ../results        # JTL results directory
  logs_dir: ../logs              # JMeter log directory
```

## Workflow

### Making JMX Changes

After modifying your JMX file in `scripts/`, sync your configuration using the `refresh` command.

#### Single Configuration File

If you have one config file, the refresh command auto-detects it:

```bash
cd my-test
sonar-forge refresh              # Merge: preserves your config values
sonar-forge refresh --replace    # Replace: overwrites with JMX values
sonar-forge refresh -v           # Verbose output shows details
```

#### Multiple Configuration Files

If you maintain multiple configurations (e.g., smoke, load, soak tests), you have several options:

**Refresh a specific config:**
```bash
cd my-test
sonar-forge refresh smoke-test.yaml
sonar-forge refresh load-test.yaml --replace
```

**Refresh all configs at once:**
```bash
cd my-test
sonar-forge refresh --all                 # Merge mode for all
sonar-forge refresh --all --replace       # Replace mode for all
sonar-forge refresh --all -v              # Verbose for all
```

The `--all` flag processes all YAML files in `config/` and shows a summary:
```
Found 3 config file(s) to refresh
...
Summary
============================================================
Mode: merge
Total files: 3
✓ Successful: 2
✗ Failed: 1
```

#### What Refresh Detects

**Merge mode** (default):
- ✅ Adds new thread groups from JMX
- ✅ Removes thread groups deleted from JMX
- ✅ Preserves your custom configuration values (threads, duration, etc.)
- ✅ Keeps your lifecycle controller settings
- ✅ Maintains all YAML comments and formatting

**Replace mode** (`--replace`):
- Completely replaces thread group configuration with current JMX state
- Useful when you want to reset config to match JMX exactly
- ⚠️ Overwrites your custom settings

#### Common Scenarios

**Scenario 1: Added new thread group to JMX**
```bash
# Edit scripts/my-test.jmx, add new thread group "API Tests"
sonar-forge refresh              # Auto-adds "API Tests" to config
# Edit config/my-test.yaml to customize the new thread group
```

**Scenario 2: Multiple test profiles**
```bash
# You have: smoke.yaml, load.yaml, soak.yaml
# All reference scripts/my-test.jmx
# JMX was updated with new thread group

sonar-forge refresh --all        # Updates all 3 configs
# Each config preserves its custom settings (threads, duration, etc.)
```

**Scenario 3: Reset config to match JMX**
```bash
# Your config has experimental changes, want to reset
sonar-forge refresh --replace    # Overwrites config with JMX defaults
```

### Version Control

**Commit these files**:
- `config/*.yaml` - Test configuration (source of truth)
- `scripts/*.jmx` - Test scripts

**Add to .gitignore**:
- `results/` - Test results
- `logs/` - JMeter logs
- `data/` - Test data files (unless needed)
- `*_effective.jmx` - Generated effective JMX files

## Upcoming Features
- Disable section that allows specific elements at the Test Plan level to be explicitly enabled/disabled
- Improved test execution UI (similar to Taurus, but customised for Sonar framework)