The concept of model validation
Model validation is the process of demonstration that the model can reproduce its performance with reasonable accuracy to a different population or setting that was not used to develop the model. The purpose of model validation is to demonstrate that the model is accurate for the intended population (dataset) for whom the model was developed and performs well in other populations (datasets) which were not used to develop the model.
Preferably, a model should be evaluated on samples that were not used to develop the model so that a model’s effectiveness can be assessed unbiasedly. However often models are developed in one part of the sample and evaluated in the other part of the sample or the same sample is used through resampling to develop and evaluate the model. Although this kind of model evaluation belongs to model validation formally known as internal validation, this does not guarantee that the model will perform well in a different dataset from a different population. Evaluation of a model’s performance in an entirely different population is formally known as external validation and is always advised to establish the generalizability of the model. Within model validation, there are different types each with its advantages and disadvantages. Once a model is validated to a different sample or population, its performance needs to be assessed. There are also different ways to assess the performance of a model. We discuss the types of model validation and how to assess the performance of a validated model within the model validation steps.