criterion performance measurements

overview

want to understand this report?

fromList/HashMap.Strict

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.430087062139572e-2 5.5306861611212614e-2 5.6675910888740404e-2
Standard deviation 1.3997181820719936e-3 2.0765077546525262e-3 3.1342865183418074e-3

Outlying measurements have slight (7.691876843351375e-2%) effect on estimated standard deviation.

fromList/LinkedHashMap.Seq

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.1118768702656383 0.11365900137786449 0.11530133712650943
Standard deviation 1.8375882037615183e-3 2.518686102060762e-3 3.348851243754148e-3

Outlying measurements have moderate (0.109375%) effect on estimated standard deviation.

fromList/LinkedHashMap.IntMap

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.13616156056465584 0.13991178703941676 0.1465616479489535
Standard deviation 2.495435373816608e-3 7.137081410157575e-3 1.0954786477752082e-2

Outlying measurements have moderate (0.11327217785466746%) effect on estimated standard deviation.

fromList/LinkedHashSet

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.10832844180195576 0.11112111195283281 0.11456346264201635
Standard deviation 3.495516077831306e-3 4.783726891218055e-3 6.395679181516847e-3

Outlying measurements have slight (9.95459176947333e-2%) effect on estimated standard deviation.

insert/HashMap.Strict

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 9.36859746050086e-2 9.564038285186564e-2 9.765368661116038e-2
Standard deviation 2.1439719721177337e-3 3.059855153748262e-3 4.096123128351332e-3

Outlying measurements have slight (9.876543209876541e-2%) effect on estimated standard deviation.

insert/LinkedHashMap.Seq

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.12925660774424838 0.13214900029892288 0.13636362410684086
Standard deviation 3.2869328172669327e-3 5.008508351789427e-3 7.444706798703425e-3

Outlying measurements have moderate (0.109375%) effect on estimated standard deviation.

insert/LinkedHashMap.IntMap

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.1381321991664535 0.1408451551438188 0.14382993468392277
Standard deviation 2.593116369287535e-3 3.994899078013638e-3 5.690067129933355e-3

Outlying measurements have moderate (0.12244897959183673%) effect on estimated standard deviation.

insert/LinkedHashSet

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.12084342120217309 0.12276720309503271 0.12496266379612098
Standard deviation 2.1659102822166884e-3 2.8456342197920293e-3 3.6654346083441975e-3

Outlying measurements have moderate (0.109375%) effect on estimated standard deviation.

toList/HashMap.Strict

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.1250000938920172e-2 1.1564078052990583e-2 1.1916531254640409e-2
Standard deviation 6.836980631636226e-4 8.690256302020189e-4 1.1079661243709234e-3

Outlying measurements have moderate (0.37760649966406246%) effect on estimated standard deviation.

toList/LinkedHashMap.Seq

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 9.668126983910865e-3 1.0264724493343624e-2 1.134531974756007e-2
Standard deviation 1.1502333521670755e-3 2.1723309778124496e-3 3.7370394414292024e-3

Outlying measurements have severe (0.8579320155725961%) effect on estimated standard deviation.

toList/LinkedHashMap.IntMap

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.915954672256332e-3 6.455850815980168e-3 7.105598447030257e-3
Standard deviation 1.3042923623282844e-3 1.6841884988224372e-3 2.573598913431011e-3

Outlying measurements have severe (0.9136793173275716%) effect on estimated standard deviation.

toList/LinkedHashSet

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.059316847661113e-3 6.573212020409705e-3 7.337737697536092e-3
Standard deviation 1.376369784330068e-3 1.8097239184442714e-3 2.5927745775057246e-3

Outlying measurements have severe (0.9142145289337344%) effect on estimated standard deviation.

lookup/HashMap.Strict

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.473729355735698e-2 2.504179551543866e-2 2.5553656780240602e-2
Standard deviation 5.91114966714086e-4 8.6654628690708e-4 1.2780424853415851e-3

Outlying measurements have slight (9.765347406472392e-2%) effect on estimated standard deviation.

lookup/LinkedHashMap.Seq

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.08267747073094e-2 3.093286253272737e-2 3.12540073378293e-2
Standard deviation 1.2408752013628844e-4 3.421632926602405e-4 6.268687888922882e-4

Outlying measurements have slight (5.536332179930796e-2%) effect on estimated standard deviation.

lookup/LinkedHashMap.IntMap

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.426628612118356e-2 3.488420362072054e-2 3.570621402615471e-2
Standard deviation 1.0725611598340382e-3 1.464618428605021e-3 1.843688983459404e-3

Outlying measurements have moderate (0.11634377161476366%) effect on estimated standard deviation.

lookup/LinkedHashSet

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8353393095660316e-2 1.8592965281378555e-2 1.8854385608461005e-2
Standard deviation 4.905290524957243e-4 6.002411348437584e-4 7.889004841042409e-4

Outlying measurements have slight (8.527715398147008e-2%) effect on estimated standard deviation.

delete/HashMap.Strict

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.657259152672139e-2 6.880135554728915e-2 7.184987411789664e-2
Standard deviation 2.910503358264081e-3 4.35907675998751e-3 6.886786435481943e-3

Outlying measurements have moderate (0.17012030837842088%) effect on estimated standard deviation.

delete/LinkedHashMap.Seq

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.3088634549358604 0.312881759024926 0.31803884141788674
Standard deviation 8.651382806075028e-4 5.108554903587085e-3 7.063215807089006e-3

Outlying measurements have moderate (0.15999999999999998%) effect on estimated standard deviation.

delete/LinkedHashMap.IntMap

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.29083887617878446 0.3197353696833014 0.3428401871998568
Standard deviation 1.731677156351609e-2 3.0840905315095434e-2 4.202518238869635e-2

Outlying measurements have moderate (0.1865650164292625%) effect on estimated standard deviation.

delete/LinkedHashSet

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.31055797004782754 0.3163024372206242 0.32054798234246284
Standard deviation 3.120241838309029e-3 6.005503526427491e-3 7.690315050020456e-3

Outlying measurements have moderate (0.16%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.