The concept of model validation
Model validation is the process of demonstration that the model can
reproduce its performance with reasonable accuracy to a different
population or setting that was not used to develop the model. The
purpose of model validation is to demonstrate that the model is accurate
for the intended population (dataset) for whom the model was developed
and performs well in other populations (datasets) which were not used to
develop the model.
Preferably, a model should be evaluated on samples that were not used to
develop the model so that a model’s effectiveness can be assessed
unbiasedly. However often models are developed in one part of the sample
and evaluated in the other part of the sample or the same sample is used
through resampling to develop and evaluate the model. Although this kind
of model evaluation belongs to model validation formally known as
internal validation, this does not guarantee that the model will perform
well in a different dataset from a different population. Evaluation of a
model’s performance in an entirely different population is formally
known as external validation and is always advised to establish the
generalizability of the model. Within model validation, there are
different types each with its advantages and disadvantages. Once a model
is validated to a different sample or population, its performance needs
to be assessed. There are also different ways to assess the performance
of a model. We discuss the types of model validation and how to assess
the performance of a validated model within the model validation steps.