criterion performance measurements
overview
want to understand this report?
encode/encodePacker
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.475718184529927e-7 | 5.630737048423427e-7 | 6.040089647478438e-7 |
| Standard deviation | 2.214220597354752e-8 | 8.030647204350723e-8 | 1.6511351186628752e-7 |
Outlying measurements have severe (0.9477921226781341%) effect on estimated standard deviation.
encode/encodeSimplePokeMonad
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.3305099726139537e-7 | 4.347009773597342e-7 | 4.3634824044244875e-7 |
| Standard deviation | 4.867015605591515e-9 | 5.4928491922097154e-9 | 6.257644692324075e-9 |
Outlying measurements have moderate (0.11633384132469499%) effect on estimated standard deviation.
encode/encodeStore
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.338118522053943e-7 | 4.3510081696570493e-7 | 4.373095241353874e-7 |
| Standard deviation | 3.3854072585705934e-9 | 5.691399002031444e-9 | 1.0608157297476373e-8 |
Outlying measurements have moderate (0.12464469369636823%) effect on estimated standard deviation.
encode/encodeCereal
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.7264419766685444e-5 | 1.7541586649162307e-5 | 1.787655594787388e-5 |
| Standard deviation | 9.147132864483453e-7 | 9.863277386263642e-7 | 1.0246597328621164e-6 |
Outlying measurements have severe (0.6438218072696295%) effect on estimated standard deviation.
encode/cbor
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.352772696000648e-6 | 5.411668575036211e-6 | 5.494111866866681e-6 |
| Standard deviation | 1.7994057204088378e-7 | 2.3483522700802193e-7 | 2.7854693611383373e-7 |
Outlying measurements have severe (0.5500081503875457%) effect on estimated standard deviation.
decode/decodePacker
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3180302773361786e-6 | 2.344191039701862e-6 | 2.3682589316904026e-6 |
| Standard deviation | 6.10073821387604e-8 | 8.526172955312548e-8 | 1.0812460783728875e-7 |
Outlying measurements have moderate (0.484779250740743%) effect on estimated standard deviation.
decode/decodeSimplePeek
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.182929162795577e-7 | 9.307187604093999e-7 | 9.46487804298666e-7 |
| Standard deviation | 3.9651855346054795e-8 | 4.480066648674503e-8 | 4.8798599226371805e-8 |
Outlying measurements have severe (0.6465054408776154%) effect on estimated standard deviation.
decode/decodeStore
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.326492834753665e-7 | 9.460970147109965e-7 | 9.616465195054183e-7 |
| Standard deviation | 4.6114913205472467e-8 | 5.112935517742105e-8 | 5.3971932109644835e-8 |
Outlying measurements have severe (0.7008241823489774%) effect on estimated standard deviation.
decode/decodeCereal
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.667284540243614e-6 | 4.729597047284985e-6 | 4.764701791274515e-6 |
| Standard deviation | 8.253412888519859e-8 | 1.5290953458387076e-7 | 2.1785903885213245e-7 |
Outlying measurements have moderate (0.4074172074555851%) effect on estimated standard deviation.
decode/cbor
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.593086486070497e-6 | 6.68059531797972e-6 | 6.813501348497385e-6 |
| Standard deviation | 2.772203459135689e-7 | 3.4321247788179703e-7 | 3.9506631469366606e-7 |
Outlying measurements have severe (0.6310142617530713%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.