A mixture model based approach for estimating the FDR in replicated microarray data

Abstract

One of the mostly used methods for estimating the false discovery rate (FDR) is the permutation based method. The permutation based method has the well-known granularity problem due to the discrete nature of the permuted null scores. The granularity problem may produce very unstable FDR estimates. Such instability may cause scientists to over- or under-estimate the number of false positives among the genes declared as significant, and hence result in inaccurate interpretation of biological data. In this paper, we propose a new model based method as an improvement of the permutation based FDR estimation method of SAM [1] The new method uses the t-mixture model which can model the microarray data better than the currently used normal mixture model. We will show that our proposed method provides more accurate FDR estimates than the permutation based method and is free of the problems of the permutation based FDR estimators. Finally, the proposed method is evaluated using extensive simulation and real microarray data.