Model validation and predictions
Using in-sample data for model training and out-of-sample data for model
testing, we used the following four validations: 1) randomly sampled
80% of weeks without replacement; 2) leave out 52 non-contiguous
randomly sampled weeks; 3) leave out 20% of randomly sampled schools,
and 4) leave one influenza season out (i.e., model training used all but
one season and the out-of-sample season was used for model testing) to
account for influenzas’ seasonal variation. Estimated
R2 used linear regressions of out-of-sample observed
influenza cases (outcome) and predicted cases (independent variable).
Prediction metrics used mean absolute error (MAE) and relative mean
absolute error (relMAE). Mean absolute error was defined as the mean of
the absolute value of model prediction errors(18). Relative MAE is the
ratio comparing a model’s MAE to a reference MAE (i.e., from a model
including calendar week, and average weekly temperature, and relative
humidity), where relMAE of 1.0 indicated the same prediction error for
two models. We visually compared observed and predicted cumulative
distributions and time-series of influenza cases.