Diagnostic Plots for BH-friendly, lfsr-hostile Correlated N(0, 1) Scores

Last updated: 2017-04-23

Code version: 3c00d15

Introduction

We take a look at those data sets where BH produces few (false) discoveries, and at the same time, ASH’s lfsr produces many (false) discoveries. For each data set, \(z\) scores, \(p\) values, and diagnostic \(\hat F_j\) are plotted.

library(ashr)
z = read.table("../output/z_null_liver_777.txt")
p = read.table("../output/p_null_liver_777.txt")
pihat0 = as.numeric(unlist(read.table("../output/pihat0_z_null_liver_777.txt")))
fd.bh = as.numeric(unlist(read.table("../output/fd.bh.0.05_null_liver_777.txt")))
fd.lfsr = as.numeric(unlist(read.table("../output/fd.lfsr.0.05.z.normal_null_liver_777.txt")))

The 118 data sets where `BH` didn’t produce false discoveries but `lfsr` does.

In this simulation, BH used the FDR level \(\alpha = 0.05\), whereas ASH declared a discovery when lfsr \(\leq 0.05\).

N0. 1 : Data Set 693 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1890 ; pihat0 = 0.01296388

N0. 2 : Data Set 33 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1111 ; pihat0 = 0.01048871

N0. 3 : Data Set 537 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1060 ; pihat0 = 0.01174394

N0. 4 : Data Set 885 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1012 ; pihat0 = 0.01285109

N0. 5 : Data Set 338 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 991 ; pihat0 = 0.01832042

N0. 6 : Data Set 942 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 844 ; pihat0 = 0.02694257

N0. 7 : Data Set 627 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 771 ; pihat0 = 0.02034137

N0. 8 : Data Set 176 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 763 ; pihat0 = 0.03005686

N0. 9 : Data Set 41 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 671 ; pihat0 = 0.03013827

N0. 10 : Data Set 407 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 648 ; pihat0 = 0.02274312

N0. 11 : Data Set 467 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 648 ; pihat0 = 0.0513426

N0. 12 : Data Set 404 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 642 ; pihat0 = 0.01757131

N0. 13 : Data Set 649 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 630 ; pihat0 = 0.02077128

N0. 14 : Data Set 411 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 484 ; pihat0 = 0.05617472

N0. 15 : Data Set 341 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 463 ; pihat0 = 0.04295436

N0. 16 : Data Set 538 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 449 ; pihat0 = 0.04976013

N0. 17 : Data Set 562 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 416 ; pihat0 = 0.05014307

N0. 18 : Data Set 705 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 365 ; pihat0 = 0.03966321

N0. 19 : Data Set 937 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 346 ; pihat0 = 0.04736585

N0. 20 : Data Set 893 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 309 ; pihat0 = 0.0422105

N0. 21 : Data Set 324 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 252 ; pihat0 = 0.04925775

N0. 22 : Data Set 915 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 217 ; pihat0 = 0.03371413

N0. 23 : Data Set 511 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 204 ; pihat0 = 0.04832825

N0. 24 : Data Set 3 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 193 ; pihat0 = 0.04912382

N0. 25 : Data Set 476 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 193 ; pihat0 = 0.06738787

N0. 26 : Data Set 985 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 182 ; pihat0 = 0.04389071

N0. 27 : Data Set 133 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 168 ; pihat0 = 0.04729238

N0. 28 : Data Set 495 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 167 ; pihat0 = 0.04841265

N0. 29 : Data Set 976 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 161 ; pihat0 = 0.06447058

N0. 30 : Data Set 809 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 156 ; pihat0 = 0.03198968

N0. 31 : Data Set 499 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 155 ; pihat0 = 0.07477045

N0. 32 : Data Set 999 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 154 ; pihat0 = 0.04802764

N0. 33 : Data Set 83 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 151 ; pihat0 = 0.06797304

N0. 34 : Data Set 780 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 145 ; pihat0 = 0.05197813

N0. 35 : Data Set 182 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 141 ; pihat0 = 0.04835373

N0. 36 : Data Set 982 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 129 ; pihat0 = 0.04228667

N0. 37 : Data Set 800 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 124 ; pihat0 = 0.0352261

N0. 38 : Data Set 972 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 118 ; pihat0 = 0.06009152

N0. 39 : Data Set 40 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 117 ; pihat0 = 0.04225311

N0. 40 : Data Set 895 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 113 ; pihat0 = 0.05313983

N0. 41 : Data Set 968 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 109 ; pihat0 = 0.05946434

N0. 42 : Data Set 644 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 108 ; pihat0 = 0.06256917

N0. 43 : Data Set 232 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 96 ; pihat0 = 0.07981507

N0. 44 : Data Set 15 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 94 ; pihat0 = 0.047921

N0. 45 : Data Set 251 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 89 ; pihat0 = 0.1130911

N0. 46 : Data Set 685 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 89 ; pihat0 = 0.04238147

N0. 47 : Data Set 659 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 85 ; pihat0 = 0.04122881

N0. 48 : Data Set 734 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 82 ; pihat0 = 0.05525044

N0. 49 : Data Set 484 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 73 ; pihat0 = 0.03296313

N0. 50 : Data Set 638 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 71 ; pihat0 = 0.05038079

N0. 51 : Data Set 268 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 70 ; pihat0 = 0.07180346

N0. 52 : Data Set 425 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 67 ; pihat0 = 0.1239249

N0. 53 : Data Set 781 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 65 ; pihat0 = 0.04563832

N0. 54 : Data Set 379 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 64 ; pihat0 = 0.03824364

N0. 55 : Data Set 303 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 59 ; pihat0 = 0.06374509

N0. 56 : Data Set 871 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 59 ; pihat0 = 0.07502045

N0. 57 : Data Set 606 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 56 ; pihat0 = 0.08713022

N0. 58 : Data Set 259 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 48 ; pihat0 = 0.06952575

N0. 59 : Data Set 526 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 45 ; pihat0 = 0.06426254

N0. 60 : Data Set 831 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 44 ; pihat0 = 0.08150764

N0. 61 : Data Set 853 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 37 ; pihat0 = 0.07850494

N0. 62 : Data Set 402 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 34 ; pihat0 = 0.07096583

N0. 63 : Data Set 190 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 33 ; pihat0 = 0.08695344

N0. 64 : Data Set 977 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 29 ; pihat0 = 0.06351771

N0. 65 : Data Set 452 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 24 ; pihat0 = 0.04829788

N0. 66 : Data Set 477 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 23 ; pihat0 = 0.05702149

N0. 67 : Data Set 633 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 22 ; pihat0 = 0.1001185

N0. 68 : Data Set 505 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 21 ; pihat0 = 0.09021379

N0. 69 : Data Set 953 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 21 ; pihat0 = 0.06148864

N0. 70 : Data Set 653 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 19 ; pihat0 = 0.09313401

N0. 71 : Data Set 907 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 18 ; pihat0 = 0.03578098

N0. 72 : Data Set 296 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 17 ; pihat0 = 0.08233874

N0. 73 : Data Set 845 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 16 ; pihat0 = 0.09543107

N0. 74 : Data Set 857 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 16 ; pihat0 = 0.07138568

N0. 75 : Data Set 423 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 15 ; pihat0 = 0.08643177

N0. 76 : Data Set 434 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 14 ; pihat0 = 0.0835438

N0. 77 : Data Set 349 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 13 ; pihat0 = 0.08685835

N0. 78 : Data Set 588 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 13 ; pihat0 = 0.2121708

N0. 79 : Data Set 622 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 12 ; pihat0 = 0.0782977

N0. 80 : Data Set 986 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 12 ; pihat0 = 0.2194227

N0. 81 : Data Set 866 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 11 ; pihat0 = 0.1081335

N0. 82 : Data Set 231 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 9 ; pihat0 = 0.1099259

N0. 83 : Data Set 532 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 9 ; pihat0 = 0.1006315

N0. 84 : Data Set 241 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 8 ; pihat0 = 0.2951308

N0. 85 : Data Set 433 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 8 ; pihat0 = 0.06930144

N0. 86 : Data Set 949 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 8 ; pihat0 = 0.09631564

N0. 87 : Data Set 618 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 6 ; pihat0 = 0.07763429

N0. 88 : Data Set 711 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 6 ; pihat0 = 0.09199754

N0. 89 : Data Set 44 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 5 ; pihat0 = 0.07141674

N0. 90 : Data Set 153 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 5 ; pihat0 = 0.05761454

N0. 91 : Data Set 671 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 5 ; pihat0 = 0.1067464

N0. 92 : Data Set 849 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 4 ; pihat0 = 0.1557612

N0. 93 : Data Set 32 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.1531939

N0. 94 : Data Set 71 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.1989475

N0. 95 : Data Set 266 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.2153199

N0. 96 : Data Set 353 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.1153508

N0. 97 : Data Set 487 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.3311973

N0. 98 : Data Set 799 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.1880441

N0. 99 : Data Set 868 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.1153411

N0. 100 : Data Set 877 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 3 ; pihat0 = 0.1894112

N0. 101 : Data Set 39 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 2 ; pihat0 = 0.1052546

N0. 102 : Data Set 132 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 2 ; pihat0 = 0.109358

N0. 103 : Data Set 424 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 2 ; pihat0 = 0.1877424

N0. 104 : Data Set 502 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 2 ; pihat0 = 0.1171748

N0. 105 : Data Set 552 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 2 ; pihat0 = 0.1703334

N0. 106 : Data Set 27 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.2143327

N0. 107 : Data Set 70 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.30275

N0. 108 : Data Set 97 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.184987

N0. 109 : Data Set 180 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.1318437

N0. 110 : Data Set 188 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.2319604

N0. 111 : Data Set 199 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.1978562

N0. 112 : Data Set 280 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.1960971

N0. 113 : Data Set 291 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.1825971

N0. 114 : Data Set 393 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.1925067

N0. 115 : Data Set 475 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.06974273

N0. 116 : Data Set 584 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.0723416

N0. 117 : Data Set 721 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.1049651

N0. 118 : Data Set 892 ; Number of False Discoveries by BH: 0 ; Number of False Discoveries by ASH's lfsr: 1 ; pihat0 = 0.2868517

The 7 data sets where `lfsr` is more conservative than `BH`.

N0. 1 : Data Set 901 ; Number of False Discoveries by BH: 2 ; Number of False Discoveries by ASH's lfsr: 0 ; pihat0 = 0.9997685

N0. 2 : Data Set 187 ; Number of False Discoveries by BH: 1 ; Number of False Discoveries by ASH's lfsr: 0 ; pihat0 = 0.6662317

N0. 3 : Data Set 285 ; Number of False Discoveries by BH: 1 ; Number of False Discoveries by ASH's lfsr: 0 ; pihat0 = 0.5178013

N0. 4 : Data Set 483 ; Number of False Discoveries by BH: 1 ; Number of False Discoveries by ASH's lfsr: 0 ; pihat0 = 0.9998828

N0. 5 : Data Set 817 ; Number of False Discoveries by BH: 1 ; Number of False Discoveries by ASH's lfsr: 0 ; pihat0 = 1

N0. 6 : Data Set 837 ; Number of False Discoveries by BH: 1 ; Number of False Discoveries by ASH's lfsr: 0 ; pihat0 = 0.828704

N0. 7 : Data Set 955 ; Number of False Discoveries by BH: 1 ; Number of False Discoveries by ASH's lfsr: 0 ; pihat0 = 0.9998902

Comparison of false discoveries by `BH` and `lfsr`

To get a more comprehensive picture, we plot as follows the numbers of false discoveries by BH and lfsr for each data set. These data sets were selected out of \(1000\) of them, because they produced at least one false discoveries by either method. Note that for every data set there are \(10K\) correlated null statistics.

Session information

sessionInfo()

R version 3.3.3 (2017-03-06)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: macOS Sierra 10.12.4

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] backports_1.0.5 magrittr_1.5    rprojroot_1.2   tools_3.3.3    
 [5] htmltools_0.3.5 yaml_2.1.14     Rcpp_0.12.10    stringi_1.1.2  
 [9] rmarkdown_1.3   knitr_1.15.1    git2r_0.18.0    stringr_1.2.0  
[13] digest_0.6.11   evaluate_0.10

This R Markdown site was created with workflowr

Diagnostic Plots for `BH`-friendly, `lfsr`-hostile Correlated \(N(0, 1)\) Scores

Lei Sun

2017-04-23

Introduction

The 118 data sets where `BH` didn’t produce false discoveries but `lfsr` does.

The 7 data sets where `lfsr` is more conservative than `BH`.

Comparison of false discoveries by `BH` and `lfsr`

Session information

Diagnostic Plots for BH-friendly, lfsr-hostile Correlated \(N(0, 1)\) Scores

Lei Sun

2017-04-23

Introduction

The 118 data sets where BH didn’t produce false discoveries but lfsr does.

The 7 data sets where lfsr is more conservative than BH.

Comparison of false discoveries by BH and lfsr

Session information

Diagnostic Plots for `BH`-friendly, `lfsr`-hostile Correlated \(N(0, 1)\) Scores

The 118 data sets where `BH` didn’t produce false discoveries but `lfsr` does.

The 7 data sets where `lfsr` is more conservative than `BH`.

Comparison of false discoveries by `BH` and `lfsr`