Skip to content

Avoid computation of p-values in add_relative_skill() #978

@elray1

Description

@elray1

The main source of this request is the following example where we use add_relative_skill to compute a relative absolute error, which generates a bunch of warnings from p-value computations which are not used or saved in the output:

library(scoringutils)
#> scoringutils 2.0.0 introduces major changes. We'd love your feedback!
#> <https://github.com/epiforecasts/scoringutils/issues>. To use the old version,
#> run: `remotes::install_github('epiforecasts/scoringutils@v1.2.2')`
#> This message is displayed once per session.
forecast_quantile <- example_quantile |>
  as_forecast_quantile(
    forecast_unit = c(
      "location", "forecast_date", "target_end_date", "target_type", "model", "horizon"
    )
  )
#> ℹ Some rows containing NA values may be removed. This is fine if not
#>   unexpected.

scores <- forecast_quantile |> 
  score(metrics = get_metrics(forecast_quantile, "ae_median"))

scores_w_rel_skill <- scores |>
  add_relative_skill(compare = "model", by = "location", metric = "ae_median")
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with zeroes
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with zeroes
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with zeroes
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with zeroes

colnames(scores_w_rel_skill)
#> [1] "model"                    "location"                
#> [3] "forecast_date"            "target_end_date"         
#> [5] "target_type"              "horizon"                 
#> [7] "ae_median"                "ae_median_relative_skill"

Created on 2025-01-03 with reprex v2.1.1

It would be nice to skip the computation of p-values in a call to add_relative_skill. I think a solution could involve these changes:

  • Add support to compare_forecasts for test_type = NULL, in which case no test is performed. To avoid changing inputs/outputs, it could set pval = NA_real_ in that case.
  • In add_relative_skill, when calling get_pairwise_comparisons, pass test_type = NULL.

This is related to discussion in #750, but I'm suggesting smaller changes here.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions