Implement Consistent naming of scoring rules/metrics

[nikosbosse]

Replaces #476 and #401. Everyone ready for another round of naming discussions? 🙈

Here is a list of places that use the words scores, metrics, rules, score_names or something similar.

Functions

• available_metrics() --> should be removed entirely
• get_score_names() "Get Names Of The Scoring Rules That Were Used For Scoring"
• validate_metrics() --> should probably be validate_scoring_rules()
• apply_rules()
• default scoring rules, e.g. rules_point() etc., + select_rules()

Function arguments

• correlation(): "Correlation Between Metrics" --> Should be "Correlation between scores"

correlation <- function(scores,
                        metrics = NULL,
                        digits = NULL)

• pairwise_comparison()

pairwise_comparison <- function(scores,
                                by = "model",
                                metric = "auto",
                                baseline = NULL,
                                ...)

• add_pairwise_comparison()

add_pairwise_comparison <- function(scores,
                                    by = NULL,
                                    relative_skill_metric = "auto",
                                    baseline = NULL)

• score() methods, e.g.

score.forecast_binary <- function(data, metrics = rules_binary(), ...)

Other

• score_names is the attribute that holds the names of the scores/scoring rules used for scoring in score(). (that's what's returned by get_score_names()

Proposal

I suggest the following:

• we call the scoring functions "scoring rules". (see Create documentation/vignette for the naming scheme of our functions #520)
• we call the output of a scoring rule "score"
• we call the name of the score "score name". In particular, "score name" would refer to the name of the column that holds the score. (I agree that "score name" is usually identical to the name of the scoring rule...)

This would result in the following function names/argument names:

• get_score_names() - Returns the names of the scoring rules that were used for scoring - which correspond to the names of columns that hold the scores

• validate_rules() validate_scoring_rules() - validates the scoring rules passed to score()

• apply_rules() - applies the scoring rules inside score(). Could also rename to apply_scoring_rules()

• default scoring rules like rules_point() etc., + select_rules() to select them

• correlation(scores, score_names, digits) computes correlations between scores (the output of the scoring rules). score_names denotes the column names of those scores for which a correlation should be computed.

• pairwise_comparison(scores, by, score_name, baseline, ...). Computes pairwise comparisons between two models based on the scores they achieved.

• add_pairwise_comparison(scores, by, score_name, baseline, ...)

• score.forecast_binary(data, rules =, ....) etc. (or alternatively, score.forecast_binary(data, scoring_rules =, ....)

Since we call the output of pairwise_comparison() something like wis_relative_skill, maybe the argument could also just be relative_skill

[nikosbosse]

@sbfnk in #600 you mentioned you didn't think the distinction between "score" (output of a scoring rule) and "scoring rule" (function to compute the score) made much sense. What's your current thinking on this? I think, for example, it doesn't make much sense to talk of correlations between different scoring rules. If we want to omit that distinction, then I guess it would make more sense to call everything a "score".

Pinging @nickreich, @elray1 for a pair of fresh eyes on the whole subject. We've been going in circles a bit in the past...

[nickreich]

I spent a few minutes trying to catch up on these discussions. I think @nikosbosse's suggestions above make pretty good sense. My one quibble is that the distinction between "scoring name" and "scoring rule" feels fuzzy. Like, couldn't you just enforce that the name of each scoring_rule function be clear enough that it could stand in for the score_name?

[nikosbosse]

@nickreich so a user would be able to pass in their own names. score() currently expects a named list of functions. Those names will then be used as the column names in the output. That's why the output has an attribute that tracks the names of the scoring rules used.
I think in practice the names of the scores are identical to the names of the scoring rules, it's more like you wouldn't compute correlations between scoring rules, but between the scores they produced. Does that address what you thought about?

[seabbs]

Everyone ready for another round of naming discussions?

Round and round we go.

something_scoring_rules vs something_rules

As we are in a package called scoringutils do we need to name everything scoring this and that? There is some clarity benefit but it also makes all of the variable and argument names longer (in what is already a very verbose package). I vote for using rules as a shorthand for scoring rules in places where that is obvious and clearly documenting this choice.

score.forecast_binary(data, scoring_rules =, ....)

This for example makes me a tired man.

Something we have mentioned elsewhere is what happens when a scoring rule has multiple outputs. For example WIS decomposition. I think there is an argument its not a score and so should have a totally different name and there is an argument that we just want to make a distinction between the thing we use to get scores (the rule) and the scores themselves (as this can then include WIS decomposition etc.).

[nikosbosse]

I'm fine with using rules consistently. (EDIT: instead of scoring_rules)
Then I think the remaining question is whether we should replace score_names by rules as well. This boils down to a choice between

Option A	Option B
get_rules()	get_score_names()
correlation(scores, rules, digits)	correlation(scores, score_names, digits)
pairwise_comparison(scores, by, rule, ...)	pairwise_comparison(scores, by, score_name, ...)
Attribute rules (names of scoring rules used)	Attribute score_names (column names of scores produced)

In a way, it makes more sense to compute a correlation between scores than between rules. But I also tend to agree with @nickreich (and I think @sbfnk) that adding another term to distinguish between the name of a score and the name of a scoring rule is maybe not all that helpful.

An additional option for the pairwise comparisons would be pairwise_comparison(scores, by, relative_skill, ...).

[seabbs]

I'm fine with using rules consistently.

This wasn't my suggestion (I was arguing we keep it where we have it to refer to scoring rules and continue to use score/score name for the output) but am happy if that is the way we go.

adding another term to distinguish between the name of a score and the name of a scoring rule is maybe not all that helpful.

😢 this was your idea in the first place...

[nikosbosse]

This wasn't my suggestion

I meant using rules instead of scoring_rules

😢 this was your idea in the first place...

I know 😢. And to me it makes sense - but apparently, it's confusing to others...

[sbfnk]

you mentioned you didn't think the distinction between "score" (output of a scoring rule) and "scoring rule" (function to compute the score) made much sense. What's your current thinking on this?

I didn't mean to say this, I meant to say the distinction between score_name and scoring_rule was potentially not very helpful.

[sbfnk]

In a way, it makes more sense to compute a correlation between scores than between rules. But I also tend to agree with @nickreich (and I think @sbfnk) that adding another term to distinguish between the name of a score and the name of a scoring rule is maybe not all that helpful.

But you are passing scores to correlation (the table holding the outcomes of applying scoring rules). And you're specifying the scores corresponding to which rules you're comparing. In this sense correlation(scores, rules, digits) makes sense I think.

I think there is an argument its not a score and so should have a totally different name and there is an argument that we just want to make a distinction between the thing we use to get scores (the rule)

I guess this is where metric as a more general concept than a scoring rule came from? But at the same time scoring rules are so central to the concept of forecast evaluation that we want to name them as such when they are scoring rules? Maybe metric wasn't so bad after all...