Model selection
This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these template messages)

Model selection is the task of selecting a statistical model from a set of candidate models, given data. In the simplest cases, a preexisting set of data is considered. However, the task can also involve the design of experiments such that the data collected is wellsuited to the problem of model selection. Given candidate models of similar predictive or explanatory power, the simplest model is most likely to be the best choice (Occam's razor).
Konishi & Kitagawa (2008, p. 75) state, "The majority of the problems in statistical inference can be considered to be problems related to statistical modeling". Relatedly, Cox (2006, p. 197) has said, "How [the] translation from subjectmatter problem to statistical model is done is often the most critical part of an analysis".
Model selection may also refer to the problem of selecting a few representative models from a large set of computational models for the purpose of decision making or optimization under uncertainty.^{[1]}
Introduction[edit]
In its most basic forms, model selection is one of the fundamental tasks of scientific inquiry. Determining the principle that explains a series of observations is often linked directly to a mathematical model predicting those observations. For example, when Galileo performed his inclined plane experiments, he demonstrated that the motion of the balls fitted the parabola predicted by his model^{[citation needed]}.
Of the countless number of possible mechanisms and processes that could have produced the data, how can one even begin to choose the best model? The mathematical approach commonly taken decides among a set of candidate models; this set must be chosen by the researcher. Often simple models such as polynomials are used, at least initially^{[citation needed]}. Burnham & Anderson (2002) emphasize throughout their book the importance of choosing models based on sound scientific principles, such as understanding of the phenomenological processes or mechanisms (e.g., chemical reactions) underlying the data.
Once the set of candidate models has been chosen, the statistical analysis allows us to select the best of these models. What is meant by best is controversial. A good model selection technique will balance goodness of fit with simplicity^{[citation needed]}. More complex models will be better able to adapt their shape to fit the data (for example, a fifthorder polynomial can exactly fit six points), but the additional parameters may not represent anything useful. (Perhaps those six points are really just randomly distributed about a straight line.) Goodness of fit is generally determined using a likelihood ratio approach, or an approximation of this, leading to a chisquared test. The complexity is generally measured by counting the number of parameters in the model.
Model selection techniques can be considered as estimators of some physical quantity, such as the probability of the model producing the given data. The bias and variance are both important measures of the quality of this estimator; efficiency is also often considered.
A standard example of model selection is that of curve fitting, where, given a set of points and other background knowledge (e.g. points are a result of i.i.d. samples), we must select a curve that describes the function that generated the points.
Two directions of model selection[edit]
There are two main objectives in inference and learning from data. One is for scientific discovery, understanding of the underlying datagenerating mechanism, and interpretation of the nature of the data. Another objective of learning from data is for predicting future or unseen observations. In the second objective, the data scientist does not necessarily concern an accurate probabilistic description of the data. Of course, one may also be interested in both directions.
In line with the two different objectives, model selection can also have two directions: model selection for inference and model selection for prediction.^{[2]} The first direction is to identify the best model for the data, which will preferably provide a reliable characterization of the sources of uncertainty for scientific interpretation. For this goal, it is significantly important that the selected model is not too sensitive to the sample size. Accordingly, an appropriate notion for evaluating model selection is the selection consistency, meaning that the most robust candidate will be consistently selected given sufficiently many data samples.
The second direction is to choose a model as machinery to offer excellent predictive performance. For the latter, however, the selected model may simply be the lucky winner among a few close competitors, yet the predictive performance can still be the best possible. If so, the model selection is fine for the second goal (prediction), but the use of the selected model for insight and interpretation may be severely unreliable and misleading.^{[2]} Moreover, for very complex models selected this way, even predictions may be unreasonable for data only slightly different from those on which the selection was made.^{[3]}
Methods to assist in choosing the set of candidate models[edit]
Criteria[edit]
Below is a list of criteria for model selection. The most commonly used criteria are (i) the Akaike information criterion and (ii) the Bayes factor and/or the Bayesian information criterion (which to some extent approximates the Bayes factor), see Stoica & Selen (2004) for a review.
 Akaike information criterion (AIC), a measure of the goodness fit of an estimated statistical model
 Bayes factor
 Bayesian information criterion (BIC), also known as the Schwarz information criterion, a statistical criterion for model selection
 Bridge criterion (BC), a statistical criterion that can attain the better performance of AIC and BIC despite the appropriateness of model specification.^{[4]}
 Crossvalidation
 Deviance information criterion (DIC), another Bayesian oriented model selection criterion
 False discovery rate
 Focused information criterion (FIC), a selection criterion sorting statistical models by their effectiveness for a given focus parameter
 Hannan–Quinn information criterion, an alternative to the Akaike and Bayesian criteria
 Kashyap information criterion (KIC) is a powerful alternative to AIC and BIC, because KIC uses Fisher information matrix
 Likelihoodratio test
 Mallows's C_{p}
 Minimum description length
 Minimum message length (MML)
 PRESS statistic, also known as the PRESS criterion
 Structural risk minimization
 Stepwise regression
 Watanabe–Akaike information criterion (WAIC), also called the widely applicable information criterion
 Extended Bayesian Information Criterion (EBIC) is an extension of ordinary Bayesian information criterion (BIC) for models with high parameter spaces.
 Extended Fisher Information Criterion (EFIC) is a model selection criterion for linear regression models.
Among these criteria, crossvalidation is typically the most accurate, and computationally the most expensive, for supervised learning problems.^{[citation needed]}
Burnham & Anderson (2002, §6.3) say the following:
There is a variety of model selection methods. However, from the point of view of statistical performance of a method, and intended context of its use, there are only two distinct classes of methods: These have been labeled efficient and consistent. (...) Under the frequentist paradigm for model selection one generally has three main approaches: (I) optimization of some selection criteria, (II) tests of hypotheses, and (III) ad hoc methods.
See also[edit]
 All models are wrong
 Analysis of competing hypotheses
 Automated machine learning (AutoML)
 Biasvariance dilemma
 Feature selection
 Freedman's paradox
 Grid search
 Identifiability Analysis
 Loglinear analysis
 Model identification
 Occam's razor
 Optimal design
 Parameter identification problem
 Scientific modelling
 Statistical model validation
 Stein's paradox
Notes[edit]
 ^ Shirangi, Mehrdad G.; Durlofsky, Louis J. (2016). "A general method to select representative models for decision making and optimization under uncertainty". Computers & Geosciences. 96: 109–123. Bibcode:2016CG.....96..109S. doi:10.1016/j.cageo.2016.08.002.
 ^ ^{a} ^{b} Ding, Jie; Tarokh, Vahid; Yang, Yuhong (2018). "Model Selection Techniques: An Overview". IEEE Signal Processing Magazine. 35 (6): 16–34. arXiv:1810.09583. doi:10.1109/MSP.2018.2867638. ISSN 10535888. S2CID 53035396.
 ^ Su, J.; Vargas, D.V.; Sakurai, K. (2019). "One Pixel Attack for Fooling Deep Neural Networks". IEEE Transactions on Evolutionary Computation. 23 (5): 828–841. arXiv:1710.08864. doi:10.1109/TEVC.2019.2890858.
 ^ Ding, J.; Tarokh, V.; Yang, Y. (June 2018). "Bridging AIC and BIC: A New Criterion for Autoregression". IEEE Transactions on Information Theory. 64 (6): 4024–4043. arXiv:1508.02473. doi:10.1109/TIT.2017.2717599. ISSN 15579654. S2CID 5189440.
References[edit]
 Aho, K.; Derryberry, D.; Peterson, T. (2014), "Model selection for ecologists: the worldviews of AIC and BIC", Ecology, 95 (3): 631–636, doi:10.1890/131452.1, PMID 24804445
 Akaike, H. (1994), "Implications of informational point of view on the development of statistical science", in Bozdogan, H. (ed.), Proceedings of the First US/JAPAN Conference on The Frontiers of Statistical Modeling: An Informational Approach—Volume 3, Kluwer Academic Publishers, pp. 27–38
 Anderson, D.R. (2008), Model Based Inference in the Life Sciences, Springer, ISBN 9780387740751
 Ando, T. (2010), Bayesian Model Selection and Statistical Modeling, CRC Press, ISBN 9781439836156
 Breiman, L. (2001), "Statistical modeling: the two cultures", Statistical Science, 16: 199–231, doi:10.1214/ss/1009213726
 Burnham, K.P.; Anderson, D.R. (2002), Model Selection and Multimodel Inference: A Practical InformationTheoretic Approach (2nd ed.), SpringerVerlag, ISBN 0387953647 [this has over 38000 citations on Google Scholar]
 Chamberlin, T.C. (1890), "The method of multiple working hypotheses", Science, 15 (366): 92–6, Bibcode:1890Sci....15R..92., doi:10.1126/science.ns15.366.92, PMID 17782687 (reprinted 1965, Science 148: 754–759 [1] doi:10.1126/science.148.3671.754)
 Claeskens, G. (2016), "Statistical model choice" (PDF), Annual Review of Statistics and Its Application, 3 (1): 233–256, Bibcode:2016AnRSA...3..233C, doi:10.1146/annurevstatistics041715033413^{[permanent dead link]}
 Claeskens, G.; Hjort, N.L. (2008), Model Selection and Model Averaging, Cambridge University Press, ISBN 9781139471800
 Cox, D.R. (2006), Principles of Statistical Inference, Cambridge University Press
 Kashyap, R.L. (1982), "Optimal choice of AR and MA parts in autoregressive moving average models", IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE, PAMI4 (2): 99–104, doi:10.1109/TPAMI.1982.4767213, PMID 21869012, S2CID 18484243
 Konishi, S.; Kitagawa, G. (2008), Information Criteria and Statistical Modeling, Springer, Bibcode:2007icsm.book.....K, ISBN 9780387718866
 Lahiri, P. (2001), Model Selection, Institute of Mathematical Statistics
 Leeb, H.; Pötscher, B. M. (2009), "Model selection", in Anderson, T. G. (ed.), Handbook of Financial Time Series, Springer, pp. 889–925, doi:10.1007/9783540712978_39, ISBN 9783540712961
 Lukacs, P. M.; Thompson, W. L.; Kendall, W. L.; Gould, W. R.; Doherty, P. F. Jr.; Burnham, K. P.; Anderson, D. R. (2007), "Concerns regarding a call for pluralism of information theory and hypothesis testing", Journal of Applied Ecology, 44 (2): 456–460, doi:10.1111/j.13652664.2006.01267.x
 McQuarrie, Allan D. R.; Tsai, ChihLing (1998), Regression and Time Series Model Selection, Singapore: World Scientific, ISBN 981023242X
 Massart, P. (2007), Concentration Inequalities and Model Selection, Springer
 Massart, P. (2014), "A nonasymptotic walk in probability and statistics", in Lin, Xihong (ed.), Past, Present, and Future of Statistical Science, Chapman & Hall, pp. 309–321, ISBN 9781482204988
 Navarro, D. J. (2019), "Between the Devil and the Deep Blue Sea: Tensions between scientific judgement and statistical model selection", Computational Brain & Behavior, 2: 28–34, doi:10.1007/s421130180019z
 Resende, Paulo Angelo Alves; Dorea, Chang Chung Yu (2016), "Model identification using the Efficient Determination Criterion", Journal of Multivariate Analysis, 150: 229–244, arXiv:1409.7441, doi:10.1016/j.jmva.2016.06.002, S2CID 5469654
 Shmueli, G. (2010), "To explain or to predict?", Statistical Science, 25 (3): 289–310, arXiv:1101.0891, doi:10.1214/10STS330, MR 2791669, S2CID 15900983
 Stoica, P.; Selen, Y. (2004), "Modelorder selection: a review of information criterion rules" (PDF), IEEE Signal Processing Magazine, 21 (4): 36–47
 Wit, E.; van den Heuvel, E.; Romeijn, J.W. (2012), "'All models are wrong...': an introduction to model uncertainty" (PDF), Statistica Neerlandica, 66 (3): 217–236, doi:10.1111/j.14679574.2012.00530.x
 Wit, E.; McCullagh, P. (2001), Viana, M. A. G.; Richards, D. St. P. (eds.), "The extendibility of statistical models", Algebraic Methods in Statistics and Probability, pp. 327–340
 Wójtowicz, Anna; Bigaj, Tomasz (2016), "Justification, confirmation, and the problem of mutually exclusive hypotheses", in Kuźniar, Adrian; OdrowążSypniewska, Joanna (eds.), Uncovering Facts and Values, Brill Publishers, pp. 122–143, doi:10.1163/9789004312654_009, ISBN 9789004312654
 Owrang, Arash; Jansson, Magnus (2018), "A Model Selection Criterion for HighDimensional Linear Regression", IEEE Transactions on Signal Processing , 66 (13): 3436–3446, Bibcode:2018ITSP...66.3436O, doi:10.1109/TSP.2018.2821628, ISSN 19410476, S2CID 46931136