Support Vector Machinery for Infinite Ensemble Learning

@article{citeulike:2412040, abstract = {Ensemble learning algorithms such as boosting can achieve better performance by averaging over the predictions of some base hypotheses. Nevertheless, most existing algorithms are limited to combining only a finite number of hypotheses, and the generated ensemble is usually sparse. Thus, it is not clear whether we should construct an ensemble classifier with a larger or even an infinite number of hypotheses. In addition, constructing an infinite ensemble itself is a challenging task. In this paper, we formulate an infinite ensemble learning framework based on the support vector machine (SVM). The framework can output an infinite and nonsparse ensemble through embedding infinitely many hypotheses into an SVM kernel. We use the framework to derive two novel kernels, the stump kernel and the perceptron kernel. The stump kernel embodies infinitely many decision stumps, and the perceptron kernel embodies infinitely many perceptrons. We also show that the Laplacian radial basis function kernel embodies infinitely many decision trees, and can thus be explained through infinite ensemble learning. Experimental results show that SVM with these kernels is superior to boosting with the same base hypothesis set. In addition, SVM with the stump kernel or the perceptron kernel performs similarly to SVM with the Gaussian radial basis function kernel, but enjoys the benefit of faster parameter selection. These properties make the novel kernels favorable choices in practice.}, author = {Lin, Hsuan-Tien and Li, Ling }, citeulike-article-id = {2412040}, journal = {Journal of Machine Learning Research}, keywords = {ensemble-diversity, ensembles, kernel-machines}, month = {February}, pages = {285--312}, posted-at = {2008-03-02 16:55:46}, priority = {2}, title = {Support Vector Machinery for Infinite Ensemble Learning}, url = {http://www.jmlr.org/papers/volume9/lin08a/lin08a.pdf}, volume = {9}, year = {2007} }

TY - JOUR ID - citeulike:2412040 L3 - citeulike-article-id:2412040 TI - Support Vector Machinery for Infinite Ensemble Learning JF - Journal of Machine Learning Research VL - 9 SP - 285 EP - 312 N2 - Ensemble learning algorithms such as boosting can achieve better performance by averaging over the predictions of some base hypotheses. Nevertheless, most existing algorithms are limited to combining only a finite number of hypotheses, and the generated ensemble is usually sparse. Thus, it is not clear whether we should construct an ensemble classifier with a larger or even an infinite number of hypotheses. In addition, constructing an infinite ensemble itself is a challenging task. In this paper, we formulate an infinite ensemble learning framework based on the support vector machine (SVM). The framework can output an infinite and nonsparse ensemble through embedding infinitely many hypotheses into an SVM kernel. We use the framework to derive two novel kernels, the stump kernel and the perceptron kernel. The stump kernel embodies infinitely many decision stumps, and the perceptron kernel embodies infinitely many perceptrons. We also show that the Laplacian radial basis function kernel embodies infinitely many decision trees, and can thus be explained through infinite ensemble learning. Experimental results show that SVM with these kernels is superior to boosting with the same base hypothesis set. In addition, SVM with the stump kernel or the perceptron kernel performs similarly to SVM with the Gaussian radial basis function kernel, but enjoys the benefit of faster parameter selection. These properties make the novel kernels favorable choices in practice. KW - ensemble-diversity KW - ensembles KW - kernel-machines AU - Lin, Hsuan-Tien AU - Li, Ling PY - 2007/02// UR - http://www.jmlr.org/papers/volume9/lin08a/lin08a.pdf ER -