Rework score.scoringutils quantile()2#421
Rework score.scoringutils quantile()2#421nikosbosse merged 13 commits intoadd-coverage-deviationfrom
Conversation
…ion of coverage_deviation per quantile as a metric
…or now that cannot be run anymore.
…differing quantiles
There was a problem hiding this comment.
fixes a few issues wit
Like what? This is the kind of thing the PR info is for or even better yet issues.
comments out tests that are currently failing since a lot of functions depended on the previous way that scores were returned (i.e. one score per quantile)
Make an issue for this.
It seems weird that the default metrics are being stored as data when they are just function? If you made it a function you could also make it easier for people to compose lists of related metrics (i.e to get multiple coverage metrics or all the wis types without hating their life).
|
|
||
| available_metrics <- function() { | ||
| return(unique(scoringutils::metrics$Name)) | ||
| return(unique(c(scoringutils::metrics$Name, |
There was a problem hiding this comment.
I think the right place to break this line is after return( and then unique( and so on vs randomly at the end.
| "coverage_50" = \(...) {run_safely(..., range = 50, fun = interval_coverage_quantile)}, | ||
| "coverage_90" = \(...) {run_safely(..., range = 90, fun = interval_coverage_quantile)}, | ||
| "coverage_deviation" = interval_coverage_deviation_quantile | ||
| "coverage_deviation" = interval_coverage_deviation_quantile, |
There was a problem hiding this comment.
why are we storing these as data when they are really a list of functions? Why isn't it just a function?
There was a problem hiding this comment.
Not sure I completely understand what you're suggesting. Made an issue: #449
There was a problem hiding this comment.
instead of storing as a data object have this be the return value of some function.
2 participants
This PR
ae_median_...()that were forgotten in the last PRpit()wherepit()relied on the existence of quantile coverage for quantile-based forecastsscore.scoringutils_quantile()score.scoringutils_quantile(), where the way that the data is split internally is improved to make sure that all forecasts in a single group have the same quantile-levels (vs. previously just the same number of unique quantile levels)interval_scoretowiseverywhere