Statistics and Its Interface

Volume 8 (2015)

Number 2

Special Issue on Modern Bayesian Statistics (Part II)

Guest Editor: Ming-Hui Chen (University of Connecticut)

Difficulty of selecting among multilevel models using predictive accuracy

Pages: 153 – 160

DOI: https://dx.doi.org/10.4310/SII.2015.v8.n2.a3

Authors

Wei Wang (Department of Statistics, Columbia University, New York, N.Y., U.S.A.)

Andrew Gelman (Department of Statistics and Political Science, Columbia University, New York, N.Y., U.S.A.)

Abstract

As a simple and compelling approach for estimating out-of-sample prediction error, cross-validation naturally lends itself to the task of model comparison. However, even with moderate sample size, it can be surprisingly difficult to compare multilevel models based on predictive accuracy. Using a hierarchical model fit to large survey data with a battery of questions, we demonstrate that even though cross-validation might give good estimates of pointwise out-of-sample prediction error, it is not always a sensitive instrument for model comparison.

Keywords

multilevel models, predictive accuracy, model selection, sample survey, cross-validation

2010 Mathematics Subject Classification

Primary 62F15. Secondary 62D05.

Published 6 March 2015