# Performance Benchmarking

This directory outlines the performance benchmarking strategy for SOPHIAClaw and provides documentation on establishing baselines, monitoring for regressions, and optimizing critical paths.

## Running Benchmarks

We use [Vitest](https://vitest.dev/guide/features/benchmarks.html) with `@vitest/bench` for our benchmarking suite. The benchmarks are located in the `test/benchmarks/` directory.

To run the full suite of benchmarks, execute the following command:

```bash
pnpm test:bench
```

Under the hood, this runs:

```bash
vitest bench test/benchmarks/
```

### Comparing against baselines

You can compare performance regressions against a previous baseline using Vitest's comparison feature. Ensure you run a benchmark on your main branch and output a JSON file:

```bash
vitest bench --outputJson bench.json test/benchmarks/
```

Then on your feature branch, you can run:

```bash
vitest bench --compare bench.json test/benchmarks/
```

## Performance Baselines

The following are the established initial baselines for critical paths. Note that these metrics will vary depending on your hardware, so establishing local or CI-specific baselines is recommended for accurate comparisons.

### ResourceMonitor

The `ResourceMonitor` handles monitoring system CPU, memory, open files, and disk usage to enforce system limits.

| Benchmark Case              | Operations/sec (Hz) | Mean Time (ms) | P99 (ms) |
| --------------------------- | ------------------- | -------------- | -------- |
| Instantiate ResourceMonitor | ~14,700,000         | 0.0001         | 0.0002   |
| Get Status                  | ~6,500,000          | 0.0002         | 0.0003   |
| Collect and Check           | ~110,000            | 0.0089         | 0.0157   |

_(Baselines recorded on a MacBook Pro environment. See raw run output below)_

```
  ✓ test/benchmarks/resource-monitor.bench.ts > ResourceMonitor Benchmarks
      name                           hz     min     max    mean     p75     p99    p995    p999     rme  samples
    · Instantiate ResourceMonitor  14,795,882.31  0.0000  2.4388  0.0001  0.0001  0.0001  0.0002  0.0005  ±0.98%  7397942
    · Get Status                    6,509,234.74  0.0000  0.1814  0.0002  0.0002  0.0002  0.0003  0.0005  ±0.24%  3254618
    · Collect and Check               111,907.73  0.0070  0.2672  0.0089  0.0090  0.0134  0.0157  0.0309  ±0.37%    55954
```

## Performance Regression Monitoring

Regression monitoring can be integrated into the CI/CD pipeline. Currently, developers should verify their changes manually:

1. Identify the critical paths altered in a PR.
2. Run `pnpm test:bench` before the changes to get a local baseline.
3. Run `pnpm test:bench` after the changes.
4. Ensure no significant degradation in operations per second (Hz) or average execution time (mean).

## Performance Characteristics and Optimizations

Benchmarks guide our performance optimizations by targeting the slowest paths or paths run frequently.

- The `ResourceMonitor` `collectAndCheck` method has a mean time around 0.0089ms which is very fast and well within budget. Its overhead for the collection interval (e.g., 1-5 seconds) is negligible.
- By tracking memory allocations, CPU calculations, and Promise resolutions, we can locate performance bottlenecks in real-time. We optimize string concatenations, disk reads/writes, and tight loops iteratively, guided by metrics gathered by this suite.