# ⚡️ Lightning-Fast Benchmarking for JavaScript and TypeScript

[![npm version](https://badge.fury.io/js/%40pawel-up%2Fbenchmark.svg)](https://badge.fury.io/js/%40pawel-up%2Fbenchmark)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**Tired of slow, inaccurate, or overly complex benchmarking tools?**  `@pawel-up/benchmark` is a modern, lightweight, and highly accurate benchmarking library designed for JavaScript and TypeScript. It provides everything you need to measure the performance of your code with confidence.

**Why Choose `@pawel-up/benchmark`?**

* **Accuracy:**  `@pawel-up/benchmark` uses advanced techniques like warm-up iterations, adaptive inner iterations, and outlier removal to ensure highly accurate and reliable results.
* **Ease of Use:**  Get started with benchmarking in minutes with our intuitive and straightforward API.
* **Flexibility:** Benchmark synchronous and asynchronous functions with ease.
* **Detailed Reporting:**  Generate comprehensive reports with key statistical metrics, including operations per second, relative margin of error, standard deviation, and more.
* **Extensibility:**  Easily extend the library with custom reporters to fit your specific needs. Note that the core library provides the foundation for creating reporters, but only includes basic file and terminal output reporters. You are free to create your own reporters for external operations.**
* **TypeScript Support:**  Built with TypeScript, providing excellent type safety and developer experience.
* **Debugging:** The library provides a debug mode and detailed logs to help you understand the benchmark process.
* **Node.js and Browser support** The library works in both Node.js and browser environments.

**This library is designed to be a lean and powerful core for benchmarking. Integrations for CLI, file output, and other features are intended to be built on top of this core.**

## Key Features

* **Warm-up Iterations:**  Automatic warm-up to allow the JavaScript engine to optimize the code before measurements.
* **Adaptive Inner Iterations:**  Dynamically adjusts the number of inner iterations to ensure accurate measurements, even for extremely fast functions.
* **Outlier Removal:**  Automatically removes outliers using the IQR method to improve the reliability of results.
* **Synchronous and Asynchronous Support:**  Benchmark both synchronous and asynchronous functions seamlessly.
* **Foundation for Custom Reporters:**  Create custom reporters to output results in various formats (e.g., JSON, CSV, HTML) or to different destinations (e.g., files, databases). **The core library provides the necessary tools for creating reporters, but does not include any built-in reporters.**
* **Suite Support:**  Organize multiple benchmarks into suites for better management and reporting. **The core library does not include any built-in reporters.**
* **Setup function** Add a setup function to the suite to prepare the environment before running the benchmarks.
* **Detailed Statistics:**  Reports include:
  * `ops` - Operations per second.
  * `rme` - Relative Margin of Error (RME).
  * `me` - Margin of error.
  * `stddev` - Sample standard deviation.
  * `mean` - Sample arithmetic mean.
  * `sample` - The sample of execution of times.
  * `sem`- The standard error of the mean.
  * `variance`- The sample variance.
  * `size` - Sample size.
  * `cohensd` - Cohen's d effect size.
  * `sed` - The standard error of the difference in means.
  * `dmedian` - The difference between the sample medians of the two benchmark runs.
  * `pmedian` - The percentage difference between the sample medians of the two benchmark runs.
* **Debug mode** The library provides a debug mode and detailed logs to help you understand the benchmark process.

## Quick Start

1. **Installation:**

    ```bash
    npm install @pawel-up/benchmark
    # or
    yarn add @pawel-up/benchmark
    ```

2. **Basic Usage (Single Benchmark):**

    ```typescript
    import { Benchmarker } from '@pawel-up/benchmark';

    // Your function to benchmark
    async function myAsyncFunction() {
      // ... your code ...
      await new Promise(resolve => setTimeout(resolve, 10));
    }

    async function main() {
      const benchmarker = new Benchmarker('My Async Benchmark', myAsyncFunction, {
        maxIterations: 100,
        maxExecutionTime: 5000,
      });
      await benchmarker.run();
      const report = benchmarker.getReport();
      console.log(report);
    }

    main();
    // Note: This example uses `console.log` for demonstration purposes. The core library does not include any built-in reporters.
    ```

3. **Using Benchmark Suites:**

    ```typescript
    import { Suite } from '@pawel-up/benchmark';

    // Your functions to benchmark
    function myFunction1() {
      // ... your code ...
    }

    function myFunction2() {
      // ... your code ...
    }

    async function main() {
      const suite = new Suite('My Benchmark Suite', { maxExecutionTime: 10000 });
      suite.setSetup(async () => {
        console.log('Running setup function...');
        // Do some setup work here...
        await new Promise(resolve => setTimeout(resolve, 1000)); // Example async setup
        console.log('Setup function completed.');
      });
      suite.setup();
      suite.add('Function 1', myFunction1);
      suite.setup();
      suite.add('Function 2', myFunction2);

      await suite.run();
      const report = suite.getReport();
      console.log(report);
    }

    main();
    // Note: This example uses `console.log` for demonstration purposes. The core library does not include any built-in reporters.
    ```

4. **Using compareFunction:**

    ```typescript
    import { compareFunction, SuiteReport } from '@pawel-up/benchmark';
    import * as fs from 'fs/promises';

    async function main() {
      // Load suite reports from files (example)
      const suiteReport1 = JSON.parse(await fs.readFile('suite_report_1.json', 'utf-8')) as SuiteReport;
      const suiteReport2 = JSON.parse(await fs.readFile('suite_report_2.json', 'utf-8')) as SuiteReport;
      const suiteReport3 = JSON.parse(await fs.readFile('suite_report_3.json', 'utf-8')) as SuiteReport;
      const suiteReport4 = JSON.parse(await fs.readFile('suite_report_4.json', 'utf-8')) as SuiteReport;

      const suiteReports = [suiteReport1, suiteReport2, suiteReport3, suiteReport4];

      // Example 1: Compare with JSON output
      compareFunction('myFunction', suiteReports, { format: 'json' });

      // Example 2: Compare with CSV output
      compareFunction('myFunction', suiteReports, { format: 'csv' });

      // Example 3: Compare with default table output
      compareFunction('myFunction', suiteReports);
    }

    main().catch(console.error);
    // Note: This example uses `console.log` for demonstration purposes. The core library does not include any built-in reporters.
    ```

## The Power of Statistical Benchmarking

`@pawel-up/benchmark` goes beyond simple timing measurements. It leverages statistical methods to provide a more accurate and meaningful assessment of function performance. Here's why this approach is crucial:

* **Addressing Variability:**  JavaScript execution environments (like Node.js and web browsers) are complex and can introduce variability in execution times. Simple timing measurements can be misleading due to this inherent variability.
* **Statistical Significance:**  The library uses statistical tests (like the t-test) to determine if observed performance differences are likely due to real changes in the code or just random fluctuations.
* **Effect Size:**  Metrics like Cohen's d help you understand the *magnitude* of performance differences, allowing you to distinguish between statistically significant but practically insignificant changes and changes that have a real-world impact.
* **Reliable Results:** Techniques like warm-up iterations, adaptive inner iterations, and outlier removal are used to minimize the impact of external factors and produce more reliable results.
* **Confidence:** The library provides confidence intervals, which help you understand the range of plausible values for the true performance difference.

By using a statistical approach, `@pawel-up/benchmark` helps you make data-driven decisions about your code's performance, leading to more effective optimizations and a better understanding of your library's behavior.

## API Overview

### `Benchmarker` Class

* **`new Benchmarker(name: string, fn: () => unknown | Promise<unknown>, options?: BenchmarkOptions)`**
  * Creates a new `Benchmarker` instance.
  * `name`: The name of the benchmark.
  * `fn`: The function to benchmark (can be synchronous or asynchronous).
  * `options`: An optional `BenchmarkOptions` object to configure the benchmark.
* **`async run(): Promise<void>`**
  * Runs the benchmark.
* **`getReport(): BenchmarkReport`**
  * Generates a `BenchmarkReport` object with the benchmark results.

### `Suite` Class

* **`new Suite(name: string, options?: BenchmarkOptions)`**
  * Creates a new `Suite` instance.
  * `name`: The name of the suite.
  * `options`: An optional `BenchmarkOptions` object to configure the suite.
* **`add(name: string, fn: () => unknown | Promise<unknown>): this`**
  * Adds a benchmark to the suite.
  * `name`: The name of the benchmark.
  * `fn`: The function to benchmark.
* **`addReporter(reporter: Reporter, timing: ReporterExecutionTiming): this`**
  * Adds a reporter to the suite.
  * `reporter`: The reporter instance.
  * `timing`: When the reporter should be executed (`'after-each'` or `'after-all'`).
* **`setSetup(fn: () => unknown | Promise<unknown>): this`**
  * Sets the setup function for the suite.
  * `fn`: The setup function.
* **`setup(): this`**
  * Adds the setup function to the execution queue.
* **`async run(): Promise<SuiteReport>`**
  * Runs all benchmarks in the suite.
* **`getReport(): SuiteReport`**
  * Generates a `SuiteReport` object with the suite results.

### `Reporter` Class

* **`async run(report: BenchmarkReport | SuiteReport): Promise<void>`**
  * Abstract method that reporters must implement to process and output the benchmark report.

### `BenchmarkOptions` Interface

* `maxExecutionTime?: number`
* `warmupIterations?: number`
* `innerIterations?: number`
* `maxInnerIterations?: number`
* `timeThreshold?: number`
* `minsize?: number`
* `maxIterations?: number`
* `debug?: boolean`
* `logLevel?: number`

### `BenchmarkReport` Interface

* `kind: 'benchmark'`
* `name: string`
* `ops: number` - Operations per Second
* `rme: number` - Relative Margin of Error (RME)
* `stddev: number` - Sample Standard Deviation
* `mean: number` - Sample Arithmetic Mean
* `me: number` - Margin of error
* `sample: number[]` - The sample of execution of times
* `sem: number`- The standard error of the mean.
* `variance: number` - The sample variance
* `size: number` - Sample size
* `date: string`

### `SuiteReport` Interface

* `kind: 'suite'`
* `name: string`
* `date: string`
* `results: BenchmarkReport[]`

## Browser Support

`@pawel-up/benchmark` ships a dedicated browser entry point that excludes all Node.js-only APIs (`fs`, `path`, `FileReporter`, `FileStore`).

### Importing in a browser project

```typescript
// Explicit browser import — always resolves to the browser bundle
import { Suite, Benchmarker, SuiteConfig, BrowserReporter } from '@pawel-up/benchmark/browser';
```

If you use a bundler (Vite, Rollup, webpack) that respects the `exports` `"browser"` condition, the default import path also works:

```typescript
// Automatically resolved to the browser bundle by your bundler
import { Suite, Benchmarker, BrowserReporter } from '@pawel-up/benchmark';
```

### `BrowserReporter`

`BrowserReporter` is the browser counterpart to `CliReporter`. It uses CSS-styled `console.log`, `console.group`, and `console.table` — no terminal dependencies.

```typescript
import { Suite, BrowserReporter } from '@pawel-up/benchmark/browser';

const suite = new Suite('My Suite');
suite.add('Array.from', () => Array.from({ length: 1000 }));
suite.add('spread', () => [...Array(1000)]);
suite.addReporter(new BrowserReporter({ format: 'short' }), 'after-each');

await suite.run();
```

Output formats:

* **`short`** (default) — one styled line per benchmark with color-coded RME (green / yellow / red).
* **`long`** — `console.group` header, `console.table` with numeric metrics, and separate color-coded lines for RME, standard deviation, and margin of error.

### What is available in the browser bundle

| Export | Available |
| --- | --- |
| `Benchmarker` | ✅ |
| `Suite` | ✅ |
| `SuiteConfig` | ✅ |
| `BrowserReporter` | ✅ |
| `Reporter` (base class) | ✅ |
| `compare` / `compareFunction` / `compareSuites` | ✅ |
| All TypeScript types | ✅ |
| `CliReporter` | ❌ Node only |
| `FileReporter` | ❌ Node only |
| `FileStore` | ❌ Node only |

### Node.js explicit import

When writing Node.js scripts that should never accidentally pick up the browser bundle:

```typescript
import { Suite, CliReporter, FileReporter, FileStore } from '@pawel-up/benchmark/node';
```

## Lupa Integration

`@pawel-up/benchmark` ships first-class integration with the [Lupa](https://github.com/pawel-up/lupa) browser test framework. Benchmarks run in the browser alongside your tests, and results are forwarded over Lupa's IPC channel to Node where they are printed to the terminal via `CliReporter`.

### How it works

The integration is two Lupa plugins that work together:

* **`@pawel-up/benchmark/lupa/browser`** — a Lupa `testPlugin` that runs in the browser. It wires the Lupa emitter into the benchmark module so that every `createSuite()` result automatically forwards benchmark data to Node over the Vite WebSocket channel.
* **`@pawel-up/benchmark/lupa/node`** — a Lupa `runnerPlugin` that runs in the Node orchestrator. Its `plan` hook auto-configures benchmark suites with a lower execution priority and disables them in watch mode. Its `execute` hook receives the forwarded results and prints them with `CliReporter`.

### Installation

```bash
npm install --save-dev @pawel-up/benchmark
```

### Recommended: suites-based config

The recommended pattern is to declare benchmarks as a separate named suite. This keeps benchmarks opt-in — they only run when you include the `benchmarks` suite, making them easy to gate behind CI or run on demand without changing any source files.

```typescript
// lupa.config.ts
import { defineConfig } from '@pawel-up/lupa'
import { benchmarkPlugin } from '@pawel-up/benchmark/lupa/node'

export default defineConfig({
  suites: [
    { name: 'unit',       files: ['tests/**/*.spec.ts'] },
    { name: 'functional', files: ['tests/**/*.functional.ts'] },
    { name: 'benchmarks', files: ['tests/**/*.benchmark.ts'] },
    // priority: 50 and disableInWatchMode: true are set automatically
    // by benchmarkPlugin for any suite named 'benchmark'/'benchmarks'
    // or whose file pattern contains '.benchmark.'
  ],
  runnerPlugins: [benchmarkPlugin()],
  testPlugins: ['@pawel-up/benchmark/lupa/browser'],
})
```

Run only the benchmark suite on demand:

```bash
lupa test --suite benchmarks
```

### Writing benchmark files

Use `createSuite` from `@pawel-up/benchmark/lupa` instead of `new Suite()`. It pre-wires the `LupaReporter` so results are forwarded automatically:

```typescript
// tests/array.benchmark.ts
import { createSuite } from '@pawel-up/benchmark/lupa'

const suite = createSuite('Array creation')
suite.add('Array.from', () => Array.from({ length: 1_000 }))
suite.add('spread',     () => [...Array(1_000)])
await suite.run()
```

### Execution order

Lupa executes suites in descending priority order (default: `100`). The plugin sets benchmark suites to `priority: 50`, so they always run **after** all regular test suites finish. Terminal output flows naturally: test results appear first, then benchmark results.

### Watch mode

Benchmark suites are automatically skipped during `lupa test --watch`. The plugin sets `disableInWatchMode: true` on all detected benchmark suites, keeping the watch feedback loop fast.

This augments Lupa's `RunnerEvents` with `benchmark:result` and `benchmark:suite:end`, enabling type-safe access if you subscribe to those events in your own plugins.

## Learn More

* Interpreting Benchmark Results
* Increasing Sample Size
* Why RME Might Be Above 5%
* Interpreting Cohen's d
* API Reference

## Contributing

Contributions are welcome! Please see the contributing guidelines for more information.

## License

This project is licensed under the MIT License.