Improved variance estimation of classification performance via reduction of bias caused by small sample size.

@article{citeulike:554287, abstract = {ABSTRACT: BACKGROUND: Supervised learning for classification of cancer uses a set of design examples to learn how to discriminate between tumors. In practice it is crucial to confirm that the classifier is robust with good generalization performance to new examples, or at least that it has a performance that is better than random guessing. A suggested alternative is to obtain a confidence interval of the error rate using repeated design and test sets selected from available examples. However, it is known that even in the ideal situation of repeated designs and tests with completely novel samples in each cycle, a small test set size leads to a large bias in the estimate of the true variance between design sets. Therefore different methods for small sample performance estimation such as a recently proposed procedure called Repeated Random Sampling (RSS) is also expected to result in heavily biased estimates, which are translated into biased confidence intervals. We explored such biases and developed a refined algorithm called Repeated Independent Design and Test (RIDT). RESULTS: Our simulations show that repeated designs and tests based on resampling in a fixed bag of samples yield a biased variance estimate. We also demonstrate that it is possible to obtain an improved variance estimate by means of a procedure that explicitly models how this bias depends on the number of samples used for testing. For the special case of repeated designs and tests using new samples for each design and test, we present an exact analytical expression for how the expected value of the bias decreases with the size of the test set. CONCLUSIONS: We show that via modeling and subsequent reduction of the small sample bias, it is possible to obtain an improved estimate of the variance of classifier performance between design sets. However, even though the method provides unbiased performance estimates, the uncertainty of the variance estimate is large in the simulations performed, which indicates that the method in its present form cannot be directly applied to small data sets.}, author = {Wickenberg-Bolin, Ulrika and Goransson, Hanna and Fryknas, Marten and Gustafsson, Mats G. and Isaksson, Anders }, citeulike-article-id = {554287}, doi = {http://dx.doi.org/10.1186/1471-2105-7-127}, issn = {1471-2105}, journal = {BMC Bioinformatics}, keywords = {classification, statistical}, month = {March}, number = {1}, posted-at = {2006-03-16 14:08:05}, priority = {1}, title = {Improved variance estimation of classification performance via reduction of bias caused by small sample size.}, url = {http://dx.doi.org/10.1186/1471-2105-7-127}, volume = {7}, year = {2006} }

TY - JOUR ID - citeulike:554287 L3 - citeulike-article-id:554287 TI - Improved variance estimation of classification performance via reduction of bias caused by small sample size. JF - BMC Bioinformatics VL - 7 IS - 1 SN - 1471-2105 N2 - ABSTRACT: BACKGROUND: Supervised learning for classification of cancer uses a set of design examples to learn how to discriminate between tumors. In practice it is crucial to confirm that the classifier is robust with good generalization performance to new examples, or at least that it has a performance that is better than random guessing. A suggested alternative is to obtain a confidence interval of the error rate using repeated design and test sets selected from available examples. However, it is known that even in the ideal situation of repeated designs and tests with completely novel samples in each cycle, a small test set size leads to a large bias in the estimate of the true variance between design sets. Therefore different methods for small sample performance estimation such as a recently proposed procedure called Repeated Random Sampling (RSS) is also expected to result in heavily biased estimates, which are translated into biased confidence intervals. We explored such biases and developed a refined algorithm called Repeated Independent Design and Test (RIDT). RESULTS: Our simulations show that repeated designs and tests based on resampling in a fixed bag of samples yield a biased variance estimate. We also demonstrate that it is possible to obtain an improved variance estimate by means of a procedure that explicitly models how this bias depends on the number of samples used for testing. For the special case of repeated designs and tests using new samples for each design and test, we present an exact analytical expression for how the expected value of the bias decreases with the size of the test set. CONCLUSIONS: We show that via modeling and subsequent reduction of the small sample bias, it is possible to obtain an improved estimate of the variance of classifier performance between design sets. However, even though the method provides unbiased performance estimates, the uncertainty of the variance estimate is large in the simulations performed, which indicates that the method in its present form cannot be directly applied to small data sets. KW - classification KW - statistical AU - Wickenberg-Bolin, Ulrika AU - Goransson, Hanna AU - Fryknas, Marten AU - Gustafsson, Mats AU - Isaksson, Anders PY - 2006/03/13/ UR - http://dx.doi.org/10.1186/1471-2105-7-127 ER -