Model Validation

This is a repo for the Model Validation module of the AI for Good COVID-19 Simulator project.

Please take a look at model_metrics_test.py for a sample usage.

Forecast errors

A forecast “error” is the difference between an observed value and its forecast. Here “error” does not mean a mistake, it means the unpredictable part of an observation.

Note that forecast errors are different from residuals in two ways. First, residuals are calculated on the training set while forecast errors are calculated on the test set. Second, residuals are based on one-step forecasts while forecast errors can involve multi-step forecasts.

We can measure forecast accuracy by summarising the forecast errors in different ways.

Scale-dependent errors

The forecast errors are on the same scale as the data. Accuracy measures that are based only on etet are therefore scale-dependent and cannot be used to make comparisons between series that involve different units.

The two most commonly used scale-dependent measures are based on the absolute errors or squared errors:

When comparing forecast methods applied to a single time series, or to several time series with the same units, the MAE is popular as it is easy to both understand and compute. A forecast method that minimises the MAE will lead to forecasts of the median, while minimising the RMSE will lead to forecasts of the mean. Consequently, the RMSE is also widely used, despite being more difficult to interpret.

Percentage errors

The percentage error is given by:

Percentage errors have the advantage of being unit-free, and so are frequently used to compare forecast performances between data sets. The most commonly used measure is:Mean absolute percentage error: MAPE=mean(|pt|).

How to Use Model Validation:

There are two Model Validation Modules developed:

1. Model Metrics (model_validation/model_metrics.py
2. Model Validation Plots (model_validation/model_validation_plot.py)

The sample calls for these two modules could be found in this test program:

https://github.com/AIforGoodSimulator/model_validation/blob/master/model_validation_test.ipynb

We also have developed the following use cases for Model validation:

(1) Compare age group cases against the overall population

Here is an sample for CM model : model_validation/validate_cm_output.py

https://s3.us-west-2.amazonaws.com/secure.notion-static.com/4055612f-d210-41b4-ad7c-8ef08ede897d/Untitled.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAT73L2G45O3KS52Y5%2F20201021%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20201021T170417Z&X-Amz-Expires=86400&X-Amz-Signature=a44b0f5589813fa9d54b2935aab8e61dd78d6f474f47064022dddb9480714b7f&X-Amz-SignedHeaders=host&response-content-disposition=filename%20%3D%22Untitled.png%22

The full CSV sample output could be found in this file in GitHub https://github.com/AIforGoodSimulator/model_validation/blob/master/cm_model_validation_metrics.csv

(2) Compare two different model outputs

Here is an sample for CM model : model_validation/compare_cm_output.py

The sample model validation output for comparing two different runs:

https://s3.us-west-2.amazonaws.com/secure.notion-static.com/156595cf-206f-4562-84c3-54c119f13825/Untitled.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAT73L2G45O3KS52Y5%2F20201021%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20201021T170137Z&X-Amz-Expires=86400&X-Amz-Signature=b7e2fbbc38553ae0f83e1a72e89a36a7ebb59b0b32cc2de55e3a1d15b3bde65f&X-Amz-SignedHeaders=host&response-content-disposition=filename%20%3D%22Untitled.png%22

References for Model Validation

1. IHME COVID-19 Model Comparison Team. Predictive performance of international COVID-19 mortality forecasting models. MedRxiv. 14 July 2020. https://www.medrxiv.org/content/10.1101/2020.07.13.20151233v4

2. Forecasting: Principles and Practice, Rob J Hyndman and George Athanasopoulos.