The main source of this request is the following example where we use add_relative_skill to compute a relative absolute error, which generates a bunch of warnings from p-value computations which are not used or saved in the output:
library(scoringutils)
#> scoringutils 2.0.0 introduces major changes. We'd love your feedback!
#> <https://github.com/epiforecasts/scoringutils/issues>. To use the old version,
#> run: `remotes::install_github('epiforecasts/scoringutils@v1.2.2')`
#> This message is displayed once per session.
forecast_quantile <- example_quantile |>
as_forecast_quantile(
forecast_unit = c(
"location", "forecast_date", "target_end_date", "target_type", "model", "horizon"
)
)
#> ℹ Some rows containing NA values may be removed. This is fine if not
#> unexpected.
scores <- forecast_quantile |>
score(metrics = get_metrics(forecast_quantile, "ae_median"))
scores_w_rel_skill <- scores |>
add_relative_skill(compare = "model", by = "location", metric = "ae_median")
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with zeroes
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with zeroes
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with zeroes
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with zeroes
colnames(scores_w_rel_skill)
#> [1] "model" "location"
#> [3] "forecast_date" "target_end_date"
#> [5] "target_type" "horizon"
#> [7] "ae_median" "ae_median_relative_skill"
Created on 2025-01-03 with reprex v2.1.1
It would be nice to skip the computation of p-values in a call to add_relative_skill. I think a solution could involve these changes:
- Add support to
compare_forecasts for test_type = NULL, in which case no test is performed. To avoid changing inputs/outputs, it could set pval = NA_real_ in that case.
- In
add_relative_skill, when calling get_pairwise_comparisons, pass test_type = NULL.
This is related to discussion in #750, but I'm suggesting smaller changes here.
The main source of this request is the following example where we use
add_relative_skillto compute a relative absolute error, which generates a bunch of warnings from p-value computations which are not used or saved in the output:Created on 2025-01-03 with reprex v2.1.1
It would be nice to skip the computation of p-values in a call to
add_relative_skill. I think a solution could involve these changes:compare_forecastsfortest_type = NULL, in which case no test is performed. To avoid changing inputs/outputs, it could setpval = NA_real_in that case.add_relative_skill, when callingget_pairwise_comparisons, passtest_type = NULL.This is related to discussion in #750, but I'm suggesting smaller changes here.