Rework quantile scores#388
Merged
nikosbosse merged 94 commits intoscoringutils-reviewfrom Nov 16, 2023
Merged
Conversation
…(no separate results yet)
…p. Function is horribly named and we need to decide what to do with it
… quantile must be equal to the length of predictions if there is only one observation
…hly hoping this is a bit easier / cleaner...
… well by introducing a new function bias_quantile_single
…nstances where the lower bound of an interval is higher than an upper bound
Collaborator
Author
|
@jhellewell14 maybe this is not the most terrible PR to have a look at if you feel inclined |
Contributor
|
Do you want this reviewed @nikosbosse? |
Collaborator
Author
|
Yes please! |
…e-based forecasts
Update branch to include coverage functions
…cause we can't pass the `forecast_unit` arg to apply_metrics
…wasn't working and everything else feels more complicated.
seabbs
reviewed
Nov 15, 2023
Contributor
seabbs
left a comment
There was a problem hiding this comment.
If you could clean this up into the actual proposal that would be good as trying to review across two PRs without sufficient information is very hard.
Base automatically changed from
rework-quantile-to-interval-format
to
scoringutils-review
November 15, 2023 14:31
First step towards reworking `score.scoringutils_quantile()` - adding coverage as a metric
Update `bias_quantile()` to work with vectors / matrices instead of data.table
…, `underprediction()` and `dispersion()`
Update functions to compute the absolute median
…uantile()2 Rework score.scoringutils quantile()2
Move tests around
Expand tests2
Rework add coverage to work with raw forecasts
Simplify `score()`
Add separate functions for wis components
Fix set forecast unit
Add coverage deviation as a metric
seabbs
approved these changes
Nov 16, 2023
Contributor
seabbs
left a comment
There was a problem hiding this comment.
I think this is effectively unreviewable in its current form and given where we are with the rest of the development we should merge and aim to improve the use of planning issues and modular PRs going forward.
Rework add coverage()
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update: I reworked the part about
bias_quantile()again in #396. Maybe it would be easiest to just ignorequantile_bias()for now? I can also merge #396 into this one (although there are some updates in between) or I could replicate the changes in #396 here to make reviewing it a bit easier.This PR
score()at the moment where you get one score per quantile. After talking to Seb it's a bit unclear whether we want / need that.bias_quantile()to work with several observations instead of only one (accepting the same input formats aswis(). This makes use of a functionbias_quantile_single()which ultimately could be replaced by justdata.tablecode in a single function. In the spirit of not cramming to many things in here (which I appreciate I completely failed to do yet again) I created an issue for that Mergebias_quantile()andbias_quantile_single()#387Next step in a future PR is then to update
score.quantile()to work with these functions.