Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
194 changes: 95 additions & 99 deletions NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,12 @@

## Feature updates

- added a new function, `transform_forecasts()` to make it easy to transform forecasts before scoring them, as suggested in Bosse et al. (2023), https://www.medrxiv.org/content/10.1101/2023.01.23.23284722v1.
- added another function, `log_shift()` that implements the default transformation function. The function allows add an offset before applying the logarithm.

- Added a new function, `transform_forecasts()` to make it easy to transform forecasts before scoring them, as suggested in Bosse et al. (2023), https://www.medrxiv.org/content/10.1101/2023.01.23.23284722v1.
- Added a function, `log_shift()` that implements the default transformation function. The function allows add an offset before applying the logarithm.

# scoringutils 1.1.1

- added a small change to `interval_score()` which explicitly converts the logical vector to a numeric one. This should happen implicitly anyway, but is now done explicitly in order to avoid issues that may come up if the input vector has a type that doesn't allow the implict conversion.
- Added a small change to `interval_score()` which explicitly converts the logical vector to a numeric one. This should happen implicitly anyway, but is now done explicitly in order to avoid issues that may come up if the input vector has a type that doesn't allow the implicit conversion.

# scoringutils 1.1.0

Expand All @@ -21,8 +20,8 @@ A minor update to the package with some bug fixes and minor changes.
- Removed the on attach message which warned of breaking changes in `1.0.0`.
- Renamed the `metric` argument of `summarise_scores()` to `relative_skill_metric`. This argument is now deprecated and will be removed in a future version of the package. Please use the new argument instead.
- Updated the documentation for `score()` and related functions to make the soft requirement for a `model` column in the input data more explicit.
- Updated the documentation for `score()`, `pairwise_comparison()` and `summaris_scores()` to make it clearer what the unit of a single forecast is that is required for computations
- Simplified the function `plot_pairwise_comparison()` which now only supports plotting mean score ratios or p-values and removed the hybrid option to print both at the same time.
- Updated the documentation for `score()`, `pairwise_comparison()` and `summarise_scores()` to make it clearer what the unit of a single forecast is that is required for computations
- Simplified the function `plot_pairwise_comparison()` which now only supports plotting mean score ratios or p-values and removed the hybrid option to print both at the same time.

## Bug fixes

Expand All @@ -35,170 +34,167 @@ A minor update to the package with some bug fixes and minor changes.
Major update to the package and most package functions with lots of breaking changes.

## Feature updates
- new and updated Readme and vignette
- the proposed scoring workflow was reworked. Functions were changed so they

- New and updated Readme and vignette.
- The proposed scoring workflow was reworked. Functions were changed so they
can easily be piped and have simplified arguments and outputs.

### new functions and function changes
- the function `eval_forecasts()` was replaced by a function `score()` with a
much reduced set of function arguments.
### New functions and function changes

- The function `eval_forecasts()` was replaced by a function `score()` with a much reduced set of function arguments.
- Functionality to summarise scores and to add relative skill scores was moved
to a function `summarise_scores()`
- new function `check_forecasts()` to analyse input data before scoring
- new function `correlation()` to compute correlations between different metrics
- new function `add_coverage()` to add coverage for specific central prediction
intervals
- new function `avail_forecasts()` allows to visualise the number of available
forecasts
- new function `find_duplicates()` to find duplicate forecasts which cause an
error
- all plotting functions were renamed to begin with `plot_`. Arguments were
simplified
- the function `pit()` now works based on data.frames. The old `pit` function
was renamed to `pit_sample()`. PIT p-values were removed entirely.
- the function `plot_pit()` now works directly with input as produced by `pit()`
- many data-handling functions were removed and input types for `score()` were
restricted to sample-based, quantile-based or binary forecasts.
- the function `brier_score()` now returns all brier scores, rather than taking
the mean before returning an output.
- `crps`, `dss` and `logs` were renamed to `crps_sample()`, `dss_sample()`, and
- New function `check_forecasts()` to analyse input data before scoring
- New function `correlation()` to compute correlations between different metrics
- New function `add_coverage()` to add coverage for specific central prediction intervals.
- New function `avail_forecasts()` allows to visualise the number of available forecasts.
- New function `find_duplicates()` to find duplicate forecasts which cause an error.
- All plotting functions were renamed to begin with `plot_`. Arguments were
simplified.
- The function `pit()` now works based on data.frames. The old `pit` function
was renamed to `pit_sample()`. PIT p-values were removed entirely.
- The function `plot_pit()` now works directly with input as produced by `pit()`
- Many data-handling functions were removed and input types for `score()` were
restricted to sample-based, quantile-based or binary forecasts.
- The function `brier_score()` now returns all brier scores, rather than taking
the mean before returning an output.
- `crps()`, `dss()` and `logs()` were renamed to `crps_sample()`, `dss_sample()`, and
`logs_sample()`

### Bug fixes

- Testing was expanded
- minor bugs were fixed, for example a bug in the sample_to_quantile function
- Minor bugs were fixed, for example a bug in the `sample_to_quantile()` function
(https://github.com/epiforecasts/scoringutils/pull/223)

### package data updated
- package data is now based on forecasts submitted to the European Forecast Hub
(https://covid19forecasthub.eu/).
- all example data files were renamed to begin with `example_`
- a new data set, `summary_metrics` was included that contains a summary of the
metrics implemented in `scoringutils`
### Package data updated

- Package data is now based on forecasts submitted to the European Forecast Hub
(https://covid19forecasthub.eu/).
- All example data files were renamed to begin with `example_`.
- A new data set, `summary_metrics` was included that contains a summary of the metrics implemented in `scoringutils`.

## Other breaking changes
- The 'sharpness' component of the weighted interval score was renamed to
dispersion. This was done to make it more clear what the component represents
and to maintain consistency with what is used in other places.

- The 'sharpness' component of the weighted interval score was renamed to dispersion. This was done to make it more clear what the component represents and to maintain consistency with what is used in other places.

# scoringutils 0.1.8

## Feature updates
- now added a function `check_forecasts()` that runs some basic checks on the
input data and provides feedback

- Added a function `check_forecasts()` that runs some basic checks on the
input data and provides feedback.

# scoringutils 0.1.7.2

## Package updates
- minor bug fixes (previously, 'interval_score' needed to be among the
selected metrics)
- all data.tables are now returned as `table[]` rather than as `table`,
such that they don't have to be called twice to display the contents.

- Minor bug fixes (previously, 'interval_score' needed to be among the selected metrics).
- All data.tables are now returned as `table[]` rather than as `table`, such that they don't have to be called twice to display the contents.

# scoringutils 0.1.7

## Feature updates
- added a function, `pairwise_comparison()` that runs pairwise comparisons
between models on the output of `eval_forecasts()`
- added functionality to compute relative skill within `eval_forecasts()`
- added a function to visualise pairwise comparisons

- Added a function, `pairwise_comparison()` that runs pairwise comparisons between models on the output of `eval_forecasts()`
- Added functionality to compute relative skill within `eval_forecasts()`.
- Added a function to visualise pairwise comparisons.

## Package updates
- The WIS definition change introduced in version 0.1.5 was partly corrected
such that the difference in weighting is only introduced when summarising
over scores from different interval ranges

- The WIS definition change introduced in version 0.1.5 was partly corrected such that the difference in weighting is only introduced when summarising over scores from different interval ranges.
- "sharpness" was renamed to 'mad' in the output of [score()] for sample-based
forecasts.
forecasts.

# scoringutils 0.1.

## Feature updates

- `eval_forecasts()` can now handle a separate forecast and truth data set as
as input
as input.
- `eval_forecasts()` now supports scoring point forecasts along side quantiles
in a quantile-based format. Currently the only metric used is the absolute error
in a quantile-based format. Currently the only metric used is the absolute error.

## Package updates

- Many functions, especially `eval_forecasts()` got a major rewrite. While
functionality should be unchanged, the code should now be easier to maintain
- Some of the data-handling functions got renamed, but old names are supported
as well for now.

as well for now.

# scoringutils 0.1.5

## Package updates
- changed the default definition of the weighted interval score. Previously,

- Changed the default definition of the weighted interval score. Previously,
the median prediction was counted twice, but is no only counted once. If you
want to go back to the old behaviour, you can call the interval_score function
with the argument `count_median_twice = FALSE`.
with the argument `count_median_twice = FALSE`.

# scoringutils 0.1.4

## Feature updates
- we added basic plotting functionality to visualise scores. You can now

- Added basic plotting functionality to visualise scores. You can now
easily obtain diagnostic plots based on scores as produced by `score`.
- `correlation_plot` shows correlation between metrics
- `plot_ranges` shows contribution of different prediction intervals to some
chosen metric
- `plot_heatmap` visualises scores as heatmap
- `plot_score_table` shows a coloured summary table of scores
- `correlation_plot()` shows correlation between metrics.
- `plot_ranges()` shows contribution of different prediction intervals to some chosen metric.
- `plot_heatmap()` visualises scores as heatmap.
- `plot_score_table()` shows a coloured summary table of scores.

## package updates
- renamed "calibration" to "coverage"
- renamed "true_values" to "true_value" in data.frames
- renamed "predictions" to "prediction" in data.frames
- renamed "is_overprediction" to "overprediction"
- renamed "is_underprediction" to "underprediction"

- Renamed "calibration" to "coverage".
- Renamed "true_values" to "true_value" in data.frames.
- Renamed "predictions" to "prediction" in data.frames.
- Renamed "is_overprediction" to "overprediction".
- Renamed "is_underprediction" to "underprediction".

# scoringutils 0.1.3

## (Potentially) Breaking changes
- the by argument in `score` now has a slightly changed meaning. It
now denotes the lowest possible grouping unit, i.e. the unit of one observation
and needs to be specified explicitly. The default is now `NULL`. The reason for
this change is that most metrics need scoring on the observation level and this
the most consistent implementation of this principle. The pit function receives

- The by argument in `score` now has a slightly changed meaning. It now denotes the lowest possible grouping unit, i.e. the unit of one observation and needs to be specified explicitly. The default is now `NULL`. The reason for
this change is that most metrics need scoring on the observation level and this the most consistent implementation of this principle. The pit function receives
its grouping now from `summarise_by`. In a similar spirit, `summarise_by` has to
be specified explicitly and e.g. doesn't assume anymore that you want 'range'
to be included.
- for the interval score, `weigh = TRUE` is now the default option.
- (potentially planned) rename true_values to true_value and predictions to prediction.
to be included.
- For the interval score, `weigh = TRUE` is now the default option.
- Renamed true_values to true_value and predictions to prediction.

## Feature updates
- updated quantile evaluation metrics in `score`. Bias as well as
calibration now take all quantiles into account
- Included option to summarise scores according to a `summarise_by` argument in
`score` The summary can return the mean, the standard deviation as well
as an arbitrary set of quantiles.
- `score` can now return pit histograms.
- switched to ggplot2 for plotting

- Updated quantile evaluation metrics in `score`. Bias as well as calibration now take all quantiles into account.
- Included option to summarise scores according to a `summarise_by` argument in `score()` The summary can return the mean, the standard deviation as well
as an arbitrary set of quantiles.
- `score()` can now return pit histograms.
- Switched to `ggplot2` for plotting.

# scoringutils 0.1.2

## (Potentially) Breaking changes
- all scores in score were consistently renamed to lower case.
Interval_score is now interval_score, CRPS is now crps etc.

- All scores in score were consistently renamed to lower case. `Interval_score` is now `interval_score`, `CRPS` is now `crps` etc.

## Feature updates
- included support for grouping scores according to a vector of column names
in `score`
- included support for passing down arguments to lower-level functions in
`score`
- included support for three new metrics to score quantiles with
`score`: bias, sharpness and calibration

- Included support for grouping scores according to a vector of column names
in `score()`.
- Included support for passing down arguments to lower-level functions in `score()`
- Included support for three new metrics to score quantiles with `score()`: bias, sharpness and calibration

## Package updates
- example data now has a horizon column to illustrate the use of grouping
- documentation updated to explain the above listed changes

- Example data now has a horizon column to illustrate the use of grouping.
- Documentation updated to explain the above listed changes.

# scoringutils 0.1.1

## Feature updates
- included support for a long as well as wide input formats for
quantile forecasts that are scored with `score`

- Included support for a long as well as wide input formats for quantile forecasts that are scored with `score()`.

## Package updates
- updated documentation for the `score`
- added badges to the Readme

- Updated documentation for the `score()`.
- Added badges to the `README`.
Loading