From 931450a32d7a0aa1ffc8ea54999bfeeda6c3fd82 Mon Sep 17 00:00:00 2001 From: Sam Date: Wed, 19 Apr 2023 11:19:07 +0100 Subject: [PATCH 1/2] edit pass on transform_forecasts and NEWS --- NEWS.md | 194 ++++++++++++++++++------------------- R/convenience-functions.R | 62 +++++++----- man/transform_forecasts.Rd | 57 ++++++----- 3 files changed, 170 insertions(+), 143 deletions(-) diff --git a/NEWS.md b/NEWS.md index cc5cba946..5d95e05cf 100644 --- a/NEWS.md +++ b/NEWS.md @@ -2,13 +2,12 @@ ## Feature updates -- added a new function, `transform_forecasts()` to make it easy to transform forecasts before scoring them, as suggested in Bosse et al. (2023), https://www.medrxiv.org/content/10.1101/2023.01.23.23284722v1. -- added another function, `log_shift()` that implements the default transformation function. The function allows add an offset before applying the logarithm. - +- Added a new function, `transform_forecasts()` to make it easy to transform forecasts before scoring them, as suggested in Bosse et al. (2023), https://www.medrxiv.org/content/10.1101/2023.01.23.23284722v1. +- Added a function, `log_shift()` that implements the default transformation function. The function allows add an offset before applying the logarithm. # scoringutils 1.1.1 -- added a small change to `interval_score()` which explicitly converts the logical vector to a numeric one. This should happen implicitly anyway, but is now done explicitly in order to avoid issues that may come up if the input vector has a type that doesn't allow the implict conversion. +- Added a small change to `interval_score()` which explicitly converts the logical vector to a numeric one. This should happen implicitly anyway, but is now done explicitly in order to avoid issues that may come up if the input vector has a type that doesn't allow the implicit conversion. # scoringutils 1.1.0 @@ -21,8 +20,8 @@ A minor update to the package with some bug fixes and minor changes. - Removed the on attach message which warned of breaking changes in `1.0.0`. - Renamed the `metric` argument of `summarise_scores()` to `relative_skill_metric`. This argument is now deprecated and will be removed in a future version of the package. Please use the new argument instead. - Updated the documentation for `score()` and related functions to make the soft requirement for a `model` column in the input data more explicit. -- Updated the documentation for `score()`, `pairwise_comparison()` and `summaris_scores()` to make it clearer what the unit of a single forecast is that is required for computations -- Simplified the function `plot_pairwise_comparison()` which now only supports plotting mean score ratios or p-values and removed the hybrid option to print both at the same time. +- Updated the documentation for `score()`, `pairwise_comparison()` and `summarise_scores()` to make it clearer what the unit of a single forecast is that is required for computations +- Simplified the function `plot_pairwise_comparison()` which now only supports plotting mean score ratios or p-values and removed the hybrid option to print both at the same time. ## Bug fixes @@ -35,170 +34,167 @@ A minor update to the package with some bug fixes and minor changes. Major update to the package and most package functions with lots of breaking changes. ## Feature updates -- new and updated Readme and vignette -- the proposed scoring workflow was reworked. Functions were changed so they + +- New and updated Readme and vignette. +- The proposed scoring workflow was reworked. Functions were changed so they can easily be piped and have simplified arguments and outputs. -### new functions and function changes -- the function `eval_forecasts()` was replaced by a function `score()` with a -much reduced set of function arguments. +### New functions and function changes + +- The function `eval_forecasts()` was replaced by a function `score()` with a much reduced set of function arguments. - Functionality to summarise scores and to add relative skill scores was moved to a function `summarise_scores()` -- new function `check_forecasts()` to analyse input data before scoring -- new function `correlation()` to compute correlations between different metrics -- new function `add_coverage()` to add coverage for specific central prediction -intervals -- new function `avail_forecasts()` allows to visualise the number of available -forecasts -- new function `find_duplicates()` to find duplicate forecasts which cause an -error -- all plotting functions were renamed to begin with `plot_`. Arguments were -simplified -- the function `pit()` now works based on data.frames. The old `pit` function -was renamed to `pit_sample()`. PIT p-values were removed entirely. -- the function `plot_pit()` now works directly with input as produced by `pit()` -- many data-handling functions were removed and input types for `score()` were -restricted to sample-based, quantile-based or binary forecasts. -- the function `brier_score()` now returns all brier scores, rather than taking -the mean before returning an output. -- `crps`, `dss` and `logs` were renamed to `crps_sample()`, `dss_sample()`, and +- New function `check_forecasts()` to analyse input data before scoring +- New function `correlation()` to compute correlations between different metrics +- New function `add_coverage()` to add coverage for specific central prediction intervals. +- New function `avail_forecasts()` allows to visualise the number of available forecasts. +- New function `find_duplicates()` to find duplicate forecasts which cause an error. +- All plotting functions were renamed to begin with `plot_`. Arguments were +simplified. +- The function `pit()` now works based on data.frames. The old `pit` function +was renamed to `pit_sample()`. PIT p-values were removed entirely. +- The function `plot_pit()` now works directly with input as produced by `pit()` +- Many data-handling functions were removed and input types for `score()` were +restricted to sample-based, quantile-based or binary forecasts. +- The function `brier_score()` now returns all brier scores, rather than taking +the mean before returning an output. +- `crps()`, `dss()` and `logs()` were renamed to `crps_sample()`, `dss_sample()`, and `logs_sample()` ### Bug fixes + - Testing was expanded -- minor bugs were fixed, for example a bug in the sample_to_quantile function +- Minor bugs were fixed, for example a bug in the `sample_to_quantile()` function (https://github.com/epiforecasts/scoringutils/pull/223) -### package data updated -- package data is now based on forecasts submitted to the European Forecast Hub -(https://covid19forecasthub.eu/). -- all example data files were renamed to begin with `example_` -- a new data set, `summary_metrics` was included that contains a summary of the -metrics implemented in `scoringutils` +### Package data updated + +- Package data is now based on forecasts submitted to the European Forecast Hub +(https://covid19forecasthub.eu/). +- All example data files were renamed to begin with `example_`. +- A new data set, `summary_metrics` was included that contains a summary of the metrics implemented in `scoringutils`. ## Other breaking changes -- The 'sharpness' component of the weighted interval score was renamed to -dispersion. This was done to make it more clear what the component represents -and to maintain consistency with what is used in other places. + +- The 'sharpness' component of the weighted interval score was renamed to dispersion. This was done to make it more clear what the component represents and to maintain consistency with what is used in other places. # scoringutils 0.1.8 ## Feature updates -- now added a function `check_forecasts()` that runs some basic checks on the -input data and provides feedback + +- Added a function `check_forecasts()` that runs some basic checks on the +input data and provides feedback. # scoringutils 0.1.7.2 ## Package updates -- minor bug fixes (previously, 'interval_score' needed to be among the -selected metrics) -- all data.tables are now returned as `table[]` rather than as `table`, -such that they don't have to be called twice to display the contents. + +- Minor bug fixes (previously, 'interval_score' needed to be among the selected metrics). +- All data.tables are now returned as `table[]` rather than as `table`, such that they don't have to be called twice to display the contents. # scoringutils 0.1.7 ## Feature updates -- added a function, `pairwise_comparison()` that runs pairwise comparisons -between models on the output of `eval_forecasts()` -- added functionality to compute relative skill within `eval_forecasts()` -- added a function to visualise pairwise comparisons + +- Added a function, `pairwise_comparison()` that runs pairwise comparisons between models on the output of `eval_forecasts()` +- Added functionality to compute relative skill within `eval_forecasts()`. +- Added a function to visualise pairwise comparisons. ## Package updates -- The WIS definition change introduced in version 0.1.5 was partly corrected -such that the difference in weighting is only introduced when summarising -over scores from different interval ranges + +- The WIS definition change introduced in version 0.1.5 was partly corrected such that the difference in weighting is only introduced when summarising over scores from different interval ranges. - "sharpness" was renamed to 'mad' in the output of [score()] for sample-based -forecasts. +forecasts. # scoringutils 0.1. ## Feature updates + - `eval_forecasts()` can now handle a separate forecast and truth data set as -as input +as input. - `eval_forecasts()` now supports scoring point forecasts along side quantiles -in a quantile-based format. Currently the only metric used is the absolute error +in a quantile-based format. Currently the only metric used is the absolute error. ## Package updates + - Many functions, especially `eval_forecasts()` got a major rewrite. While functionality should be unchanged, the code should now be easier to maintain - Some of the data-handling functions got renamed, but old names are supported -as well for now. - +as well for now. # scoringutils 0.1.5 ## Package updates -- changed the default definition of the weighted interval score. Previously, + +- Changed the default definition of the weighted interval score. Previously, the median prediction was counted twice, but is no only counted once. If you want to go back to the old behaviour, you can call the interval_score function -with the argument `count_median_twice = FALSE`. +with the argument `count_median_twice = FALSE`. # scoringutils 0.1.4 ## Feature updates -- we added basic plotting functionality to visualise scores. You can now + +- Added basic plotting functionality to visualise scores. You can now easily obtain diagnostic plots based on scores as produced by `score`. -- `correlation_plot` shows correlation between metrics -- `plot_ranges` shows contribution of different prediction intervals to some -chosen metric -- `plot_heatmap` visualises scores as heatmap -- `plot_score_table` shows a coloured summary table of scores +- `correlation_plot()` shows correlation between metrics. +- `plot_ranges()` shows contribution of different prediction intervals to some chosen metric. +- `plot_heatmap()` visualises scores as heatmap. +- `plot_score_table()` shows a coloured summary table of scores. ## package updates -- renamed "calibration" to "coverage" -- renamed "true_values" to "true_value" in data.frames -- renamed "predictions" to "prediction" in data.frames -- renamed "is_overprediction" to "overprediction" -- renamed "is_underprediction" to "underprediction" + +- Renamed "calibration" to "coverage". +- Renamed "true_values" to "true_value" in data.frames. +- Renamed "predictions" to "prediction" in data.frames. +- Renamed "is_overprediction" to "overprediction". +- Renamed "is_underprediction" to "underprediction". # scoringutils 0.1.3 ## (Potentially) Breaking changes -- the by argument in `score` now has a slightly changed meaning. It -now denotes the lowest possible grouping unit, i.e. the unit of one observation -and needs to be specified explicitly. The default is now `NULL`. The reason for -this change is that most metrics need scoring on the observation level and this -the most consistent implementation of this principle. The pit function receives + +- The by argument in `score` now has a slightly changed meaning. It now denotes the lowest possible grouping unit, i.e. the unit of one observation and needs to be specified explicitly. The default is now `NULL`. The reason for +this change is that most metrics need scoring on the observation level and this the most consistent implementation of this principle. The pit function receives its grouping now from `summarise_by`. In a similar spirit, `summarise_by` has to be specified explicitly and e.g. doesn't assume anymore that you want 'range' -to be included. -- for the interval score, `weigh = TRUE` is now the default option. -- (potentially planned) rename true_values to true_value and predictions to prediction. +to be included. +- For the interval score, `weigh = TRUE` is now the default option. +- Renamed true_values to true_value and predictions to prediction. ## Feature updates -- updated quantile evaluation metrics in `score`. Bias as well as -calibration now take all quantiles into account -- Included option to summarise scores according to a `summarise_by` argument in -`score` The summary can return the mean, the standard deviation as well -as an arbitrary set of quantiles. -- `score` can now return pit histograms. -- switched to ggplot2 for plotting + +- Updated quantile evaluation metrics in `score`. Bias as well as calibration now take all quantiles into account. +- Included option to summarise scores according to a `summarise_by` argument in `score()` The summary can return the mean, the standard deviation as well +as an arbitrary set of quantiles. +- `score()` can now return pit histograms. +- Switched to `ggplot2` for plotting. # scoringutils 0.1.2 ## (Potentially) Breaking changes -- all scores in score were consistently renamed to lower case. -Interval_score is now interval_score, CRPS is now crps etc. + +- All scores in score were consistently renamed to lower case. `Interval_score` is now `interval_score`, `CRPS` is now `crps` etc. ## Feature updates -- included support for grouping scores according to a vector of column names -in `score` -- included support for passing down arguments to lower-level functions in -`score` -- included support for three new metrics to score quantiles with -`score`: bias, sharpness and calibration + +- Included support for grouping scores according to a vector of column names +in `score()`. +- Included support for passing down arguments to lower-level functions in `score()` +- Included support for three new metrics to score quantiles with `score()`: bias, sharpness and calibration ## Package updates -- example data now has a horizon column to illustrate the use of grouping -- documentation updated to explain the above listed changes + +- Example data now has a horizon column to illustrate the use of grouping. +- Documentation updated to explain the above listed changes. # scoringutils 0.1.1 ## Feature updates -- included support for a long as well as wide input formats for -quantile forecasts that are scored with `score` + +- Included support for a long as well as wide input formats for quantile forecasts that are scored with `score()`. ## Package updates -- updated documentation for the `score` -- added badges to the Readme +- Updated documentation for the `score()`. +- Added badges to the `README`. diff --git a/R/convenience-functions.R b/R/convenience-functions.R index 2b173e909..7c22cf393 100644 --- a/R/convenience-functions.R +++ b/R/convenience-functions.R @@ -19,25 +19,35 @@ #' rule. #' #' @inheritParams score +#' #' @param fun A function used to transform both true values and predictions. #' The default function is [log_shift()], a custom function that is essentially #' the same as [log()], but has an additional arguments (`offset`) -#' that allows you add an offset before applying the logarithm. -#' @param append whether or not to append a transformed version of the data to -#' the currently existing data (default is TRUE). If selected, the data gets -#' transformed and appended to the existing data frame, making it possible to -#' use the outcome directly in [score()]. An additional column, 'scale', gets -#' created that denotes which rows or untransformed ('scale' has the value -#' "natural") and which have been transformed ('scale' has the value passed to -#' the argument `label`). +#' that allows you add an offset before applying the logarithm. This is often +#' helpful as the natural log transformation is not define at zero. A common, +#' and pragmatic solution, is to add a small offset to the data before applying +#' the log transformation. In our work we have often used an offset of 1 but +#' the precise value will depend on your application. +#' +#' @param append Logical, defaults to `TRUE`. Whether or not to append a +#' transformed version of the data to the currently existing data (`TRUE`). If +#' selected, the data gets transformed and appended to the existing data frame, +#' making it possible to use the outcome directly in [score()]. An additional +#' column, 'scale', gets created that denotes which rows or untransformed +#' ('scale' has the value "natural") and which have been transformed ('scale' +#' has the value passed to the argument `label`). +#' #' @param label A string for the newly created 'scale' column to denote the #' newly transformed values. Only relevant if `append = TRUE`. -#' @param ... Additional parameters to pass to the function you supplied. -#' @return A data.table with either a transformed version of the data, or one -#' with both the untransformed and the transformed data. includes the original data as well as a -#' transformation of the original data. There will be one additional column, -#' 'scale', present which will be set to "natural" for the untransformed -#' forecasts. +#' +#' @param ... Additional parameters to pass to the function you supplied. For +#' the default option of [log_shift()] this could be the `offset` argument. +#' +#' @return A `data.table` with either a transformed version of the data, or one +#' with both the untransformed and the transformed data. includes the original +#' data as well as a transformation of the original data. There will be one +#' additional column, `scale', present which will be set to "natural" for the +#' untransformed forecasts. #' #' @importFrom data.table ':=' is.data.table copy #' @author Nikos Bosse \email{nikosbosse@@gmail.com} @@ -48,7 +58,7 @@ #' Sebastian Funk #' medRxiv 2023.01.23.23284722 #' \doi{https://doi.org/10.1101/2023.01.23.23284722} -#' # nolint +#' #' @keywords check-forecasts #' @examples #' @@ -58,20 +68,27 @@ #' # negative values need to be handled (here by replacing them with 0) #' example_quantile %>% #' .[, true_value := ifelse(true_value < 0, 0, true_value)] %>% -#' transform_forecasts(append = FALSE) +#' # Here we use the default function log_shift() which is essentially the same +#' # as log(), but has an additional arguments (offset) that allows you add an +#' # offset before applying the logarithm. +#' transform_forecasts(append = FALSE) %>% +#' head() #' #' # alternatively, integrating the truncation in the transformation function: -#' transform_forecasts(example_quantile, -#' fun = function(x) {log_shift(pmax(0, x))}, -#' append = FALSE) +#' example_quantile %>% +#' transform_forecasts( +#' fun = function(x) {log_shift(pmax(0, x))}, append = FALSE +#' ) %>% +#' head() #' #' # specifying an offset for the log transformation removes the #' # warning caused by zeros in the data #' example_quantile %>% #' .[, true_value := ifelse(true_value < 0, 0, true_value)] %>% -#' transform_forecasts(offset = 1, append = FALSE) +#' transform_forecasts(offset = 1, append = FALSE) %>% +#' head() #' -#' # adding square root transformed forecasts to the orginal ones +#' # adding square root transformed forecasts to the original ones #' example_quantile %>% #' .[, true_value := ifelse(true_value < 0, 0, true_value)] %>% #' transform_forecasts(fun = sqrt, label = "sqrt") %>% @@ -82,7 +99,8 @@ #' example_quantile %>% #' .[, true_value := ifelse(true_value < 0, 0, true_value)] %>% #' transform_forecasts(fun = log_shift, offset = 1) %>% -#' transform_forecasts(fun = sqrt, label = "sqrt") +#' transform_forecasts(fun = sqrt, label = "sqrt") %>% +#' head() transform_forecasts <- function(data, fun = log_shift, diff --git a/man/transform_forecasts.Rd b/man/transform_forecasts.Rd index 34d4873a3..f3a54b655 100644 --- a/man/transform_forecasts.Rd +++ b/man/transform_forecasts.Rd @@ -37,27 +37,32 @@ are examples for each format (\link{example_quantile}, \link{example_continuous} \item{fun}{A function used to transform both true values and predictions. The default function is \code{\link[=log_shift]{log_shift()}}, a custom function that is essentially the same as \code{\link[=log]{log()}}, but has an additional arguments (\code{offset}) -that allows you add an offset before applying the logarithm.} +that allows you add an offset before applying the logarithm. This is often +helpful as the natural log transformation is not define at zero. A common, +and pragmatic solution, is to add a small offset to the data before applying +the log transformation. In our work we have often used an offset of 1 but +the precise value will depend on your application.} -\item{append}{whether or not to append a transformed version of the data to -the currently existing data (default is TRUE). If selected, the data gets -transformed and appended to the existing data frame, making it possible to -use the outcome directly in \code{\link[=score]{score()}}. An additional column, 'scale', gets -created that denotes which rows or untransformed ('scale' has the value -"natural") and which have been transformed ('scale' has the value passed to -the argument \code{label}).} +\item{append}{Logical, defaults to \code{TRUE}. Whether or not to append a +transformed version of the data to the currently existing data (\code{TRUE}). If +selected, the data gets transformed and appended to the existing data frame, +making it possible to use the outcome directly in \code{\link[=score]{score()}}. An additional +column, 'scale', gets created that denotes which rows or untransformed +('scale' has the value "natural") and which have been transformed ('scale' +has the value passed to the argument \code{label}).} \item{label}{A string for the newly created 'scale' column to denote the newly transformed values. Only relevant if \code{append = TRUE}.} -\item{...}{Additional parameters to pass to the function you supplied.} +\item{...}{Additional parameters to pass to the function you supplied. For +the default option of \code{\link[=log_shift]{log_shift()}} this could be the \code{offset} argument.} } \value{ -A data.table with either a transformed version of the data, or one -with both the untransformed and the transformed data. includes the original data as well as a -transformation of the original data. There will be one additional column, -'scale', present which will be set to "natural" for the untransformed -forecasts. +A \code{data.table} with either a transformed version of the data, or one +with both the untransformed and the transformed data. includes the original +data as well as a transformation of the original data. There will be one +additional column, `scale', present which will be set to "natural" for the +untransformed forecasts. } \description{ Function to transform forecasts and true values before scoring. @@ -87,20 +92,27 @@ library(magrittr) # pipe operator # negative values need to be handled (here by replacing them with 0) example_quantile \%>\% .[, true_value := ifelse(true_value < 0, 0, true_value)] \%>\% - transform_forecasts(append = FALSE) +# Here we use the default function log_shift() which is essentially the same +# as log(), but has an additional arguments (offset) that allows you add an +# offset before applying the logarithm. + transform_forecasts(append = FALSE) \%>\% + head() # alternatively, integrating the truncation in the transformation function: -transform_forecasts(example_quantile, - fun = function(x) {log_shift(pmax(0, x))}, - append = FALSE) +example_quantile \%>\% + transform_forecasts( + fun = function(x) {log_shift(pmax(0, x))}, append = FALSE + ) \%>\% + head() # specifying an offset for the log transformation removes the # warning caused by zeros in the data example_quantile \%>\% .[, true_value := ifelse(true_value < 0, 0, true_value)] \%>\% - transform_forecasts(offset = 1, append = FALSE) + transform_forecasts(offset = 1, append = FALSE) \%>\% + head() -# adding square root transformed forecasts to the orginal ones +# adding square root transformed forecasts to the original ones example_quantile \%>\% .[, true_value := ifelse(true_value < 0, 0, true_value)] \%>\% transform_forecasts(fun = sqrt, label = "sqrt") \%>\% @@ -111,7 +123,8 @@ example_quantile \%>\% example_quantile \%>\% .[, true_value := ifelse(true_value < 0, 0, true_value)] \%>\% transform_forecasts(fun = log_shift, offset = 1) \%>\% - transform_forecasts(fun = sqrt, label = "sqrt") + transform_forecasts(fun = sqrt, label = "sqrt") \%>\% + head() } \references{ Transformation of forecasts for evaluating predictive @@ -120,7 +133,7 @@ Nikos I. Bosse, Sam Abbott, Anne Cori, Edwin van Leeuwen, Johannes Bracher, Sebastian Funk medRxiv 2023.01.23.23284722 \doi{https://doi.org/10.1101/2023.01.23.23284722} -\url{https://www.medrxiv.org/content/10.1101/2023.01.23.23284722v1} # nolint +\url{https://www.medrxiv.org/content/10.1101/2023.01.23.23284722v1} } \author{ Nikos Bosse \email{nikosbosse@gmail.com} From 0733d9ff16e7eca55054527fbe661b29e8ec1085 Mon Sep 17 00:00:00 2001 From: Sam Abbott Date: Wed, 19 Apr 2023 12:16:58 +0100 Subject: [PATCH 2/2] Update R/convenience-functions.R Co-authored-by: Nikos Bosse <37978797+nikosbosse@users.noreply.github.com> --- R/convenience-functions.R | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/R/convenience-functions.R b/R/convenience-functions.R index 7c22cf393..49ce96e55 100644 --- a/R/convenience-functions.R +++ b/R/convenience-functions.R @@ -24,7 +24,7 @@ #' The default function is [log_shift()], a custom function that is essentially #' the same as [log()], but has an additional arguments (`offset`) #' that allows you add an offset before applying the logarithm. This is often -#' helpful as the natural log transformation is not define at zero. A common, +#' helpful as the natural log transformation is not defined at zero. A common, #' and pragmatic solution, is to add a small offset to the data before applying #' the log transformation. In our work we have often used an offset of 1 but #' the precise value will depend on your application.