# bencho

> A command-line benchmarking tool.

> :book: This is an early version of the package.

Bencho is inspired by [hyperfine](https://github.com/sharkdp/hyperfine), but it measures explicit markers emitted by the benchmarked process instead of only measuring the process runtime. This is useful when you need to benchmark a specific operation inside a larger program or compare application-level metrics.

Markers are plain text lines written to stdout or stderr. They can represent time, memory, or any numeric value. You can emit any number of markers from any language, framework, logger, or tracing tool.

See [`./examples/min.js`](./examples/min.js) for a complete example.

```console
$ bencho 'node --expose-gc ./examples/min.js {mode}' -n 'min.js {mode}' -r 1 -l mode=math,math-spread,loop,sort,reduce --reporter benchmark --allow-multiple
```

```php
Marker        Type    Command             Mean       StdDev       StdDev, %  Ratio   Lowest
------------  ------  ------------------  ---------  -----------  ---------  ------  ------
time          time    min.js math           0.148ms      0.017ms    11.744%    1.00     yes
time          time    min.js math-spread    0.152ms      0.027ms    17.487%    1.03
time          time    min.js loop           0.241ms      0.341ms   141.105%    1.63
time          time    min.js sort          18.774ms      0.528ms     2.815%  126.58
time          time    min.js reduce         1.246ms      0.066ms     5.270%    8.40

memory        memory  min.js math         4.180 MiB  130.722 kiB     3.054%    1.00     yes
memory        memory  min.js math-spread  4.251 MiB  104.033 kiB     2.390%    1.02
memory        memory  min.js loop         4.358 MiB  415.165 kiB     9.304%    1.04
memory        memory  min.js sort         6.026 MiB  163.173 kiB     2.644%    1.44
memory        memory  min.js reduce       4.411 MiB  132.157 kiB     2.926%    1.06

max           value   min.js math             99999            0     0.000%    1.00     yes
max           value   min.js math-spread      99999            0     0.000%    1.00     yes
max           value   min.js loop             99999            0     0.000%    1.00     yes
max           value   min.js sort             99999            0     0.000%    1.00     yes
max           value   min.js reduce           99999            0     0.000%    1.00     yes

process.time  time    min.js math            4.018s          0ms     0.000%    1.00
process.time  time    min.js math-spread     4.011s          0ms     0.000%    1.00     yes
process.time  time    min.js loop            4.028s          0ms     0.000%    1.00
process.time  time    min.js sort            5.873s          0ms     0.000%    1.46
process.time  time    min.js reduce          4.124s          0ms     0.000%    1.03
```

## How does it work?

Add markers to the code you want to benchmark. A marker is a line with this format:

```php
bencho_marker:<marker_label>;<marker_type>;<marker_value>
```

Examples:

```php
bencho_marker:time;time;12.5
bencho_marker:memory;memory;1048576
bencho_marker:counter;value;42
```

Bencho starts your command, reads stdout and stderr, extracts marker lines, groups them by label and type, and reports statistics.

> :book: Bencho does not transform or instrument your code.
>
> You can print marker lines manually with `console.log` or `console.error`, or use the helper functions exported by `bencho`. If markers should not be present in production builds, gate or remove them in your own build process.

## Installation

> :book: Right now we only support installation from the `npm` registry.

```console
npm install --save-dev bencho
```

## Usage

Use `bencho --help` for the full CLI reference. More examples are available in the [`./examples`](./examples) directory.

The examples below use `bencho`. For a local project install, run it as `npx bencho` or from an npm script.

Before adding markers:

```js
// ./code.mjs
async function findAllEmptyResolutions() { /* code */ }

await findAllEmptyResolutions();
```

After adding markers:

```js
// ./code.mjs
import { performance } from 'node:perf_hooks';
import * as bencho from 'bencho';

async function findAllEmptyResolutions() { /* code */ }

const startTime = performance.now();

await findAllEmptyResolutions();

const time = performance.now() - startTime;
const memory = process.memoryUsage().heapUsed;

bencho.time('time', time);
bencho.memory('memory', memory);
```

Run a fixed number of measured runs:

```shell
$ bencho 'node ./code.mjs' --warmup 10 --runs 50
```

`--warmup` runs are executed before measurement and are not included in the result. `--runs` selects exact mode and performs exactly that many measured runs.

When `--runs` is omitted, Bencho uses adaptive mode. In this mode, the first measured run determines how many measured runs will be collected:

```js
max(floor(3000 / first_process_time_ms), min_runs)
```

The calculated count includes the first measured run. By default, adaptive mode uses at least 10 measured runs and an internal 3 second target. `--min-runs` changes the minimum, and `--max-runs` caps the measured run count.

```shell
$ bencho 'node ./code.mjs' --warmup 10 --min-runs 10
```

With this command:

- if the first measured run takes about 400ms, Bencho runs 10 warmup runs and 10 measured runs;
- if it takes about 100ms, Bencho runs 10 warmup runs and 30 measured runs.

The internal 3 second target is only used to estimate the run count. It is not a timeout and does not limit total benchmark duration. Use `--max-runs` when fast commands would otherwise run too many times.

`--runs` cannot be combined with `--min-runs` or `--max-runs`.

Results:

```php
Benchmark #1
Command:    node ./code.mjs
            node ./code.mjs

Marker        Type    Mean       StdDev   StdDev, %  Min        Max
------------  ------  ---------  -------  ---------  ---------  ---------
time          time      0.011ms  0.002ms    19.064%    0.008ms    0.017ms
process.time  time     36.135ms  2.163ms     5.985%   33.077ms   43.087ms
memory        memory  5.127 MiB  0.000 B     0.000%  5.127 MiB  5.127 MiB
```