In Summaries scores, relative skill scores change when there is another unrelated column added

[nikosbosse]

There could e.g. be a warning if there is a column included that should not be there or something like that? Unclear.

[seabbs]

This needs more detail to be addressable

[nikosbosse]

Reprex:

example_quantile |>
  score() |>
  summarise_scores(by = c("model", "target_type"), relative_skill = TRUE) |>
  pull(relative_skill)


example_quantile |>
  dplyr::mutate(
    arbitrary_colum = rep(
      c("Coruscant", "Endor", "Dagobah", "Tatooine", "Naboo"), 
      20545/5)) |>
  score() |>
  summarise_scores(by = c("model", "target_type"), relative_skill = TRUE) |>
  pull(relative_skill)

So adding a nonsense column causes scores to change, even though I specified a "summarise_by". I think this is expected behaviour, but it really confused me when that occurred to me and I definitely wasn't expecting it. The other columns do define the unit of a single forecast (that's why there was a two separate by arguments in the previous version of that function. But I think that should be documented more clearly (the summarise_by documentation doesn't give much info).

I briefly thought about a message "it was summarised by x, using xyz as the unit of a single forecast". Maybe that's overkill, since there is already a check_forecast function

[seabbs]

This really confuses me as well honestly... why does it cause scores to change and why is this expected behaviour? Is it because in your example they change the definition of a single forecast? Or is it something else?

[seabbs]

Can this not be explained by a changing forecast unit and hence is expected behaviour (but does require the user to pay attention to what they are doing).

Full reprex:

library(scoringutils)
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union

check_forecasts(example_quantile)
#> The following messages were produced when checking inputs:
#> 1.  144 values for `prediction` are NA in the data provided and the corresponding rows were removed. This may indicate a problem if unexpected.
#> Your forecasts seem to be for a target of the following type:
#> $target_type
#> [1] "integer"
#> 
#> and in the following format:
#> $prediction_type
#> [1] "quantile"
#> 
#> The unit of a single forecast is defined by:
#> $forecast_unit
#> [1] "location"        "target_end_date" "target_type"     "location_name"  
#> [5] "forecast_date"   "model"           "horizon"        
#> 
#> Cleaned data, rows with NA values in prediction or true_value removed:
#> $cleaned_data
#> Key: <location, target_end_date, target_type>
#>        location target_end_date target_type true_value location_name
#>          <char>          <Date>      <char>      <num>        <char>
#>     1:       DE      2021-05-08       Cases     106987       Germany
#>     2:       DE      2021-05-08       Cases     106987       Germany
#>     3:       DE      2021-05-08       Cases     106987       Germany
#>     4:       DE      2021-05-08       Cases     106987       Germany
#>     5:       DE      2021-05-08       Cases     106987       Germany
#>    ---                                                              
#> 20397:       IT      2021-07-24      Deaths         78         Italy
#> 20398:       IT      2021-07-24      Deaths         78         Italy
#> 20399:       IT      2021-07-24      Deaths         78         Italy
#> 20400:       IT      2021-07-24      Deaths         78         Italy
#> 20401:       IT      2021-07-24      Deaths         78         Italy
#>        forecast_date quantile prediction                 model horizon
#>               <Date>    <num>      <int>                <char>   <num>
#>     1:    2021-05-03    0.010      82466 EuroCOVIDhub-ensemble       1
#>     2:    2021-05-03    0.025      86669 EuroCOVIDhub-ensemble       1
#>     3:    2021-05-03    0.050      90285 EuroCOVIDhub-ensemble       1
#>     4:    2021-05-03    0.100      95341 EuroCOVIDhub-ensemble       1
#>     5:    2021-05-03    0.150      99171 EuroCOVIDhub-ensemble       1
#>    ---                                                                
#> 20397:    2021-07-12    0.850        352  epiforecasts-EpiNow2       2
#> 20398:    2021-07-12    0.900        397  epiforecasts-EpiNow2       2
#> 20399:    2021-07-12    0.950        499  epiforecasts-EpiNow2       2
#> 20400:    2021-07-12    0.975        611  epiforecasts-EpiNow2       2
#> 20401:    2021-07-12    0.990        719  epiforecasts-EpiNow2       2
#> 
#> Number of unique values per column per model:
#> $unique_values
#>                    model location target_end_date target_type true_value
#>                   <char>    <int>           <int>       <int>      <int>
#> 1: EuroCOVIDhub-ensemble        4              12           2         96
#> 2: EuroCOVIDhub-baseline        4              12           2         96
#> 3:  epiforecasts-EpiNow2        4              12           2         95
#> 4:       UMass-MechBayes        4              12           1         48
#>    location_name forecast_date quantile prediction horizon
#>            <int>         <int>    <int>      <int>   <int>
#> 1:             4            11       23       3969       3
#> 2:             4            11       23       3733       3
#> 3:             4            11       23       3903       3
#> 4:             4            11       23       1058       3
#> 
#> $messages
#> [1] "144 values for `prediction` are NA in the data provided and the corresponding rows were removed. This may indicate a problem if unexpected."
example_quantile |>
  score() |>
  summarise_scores(by = c("model", "target_type"), relative_skill = TRUE) |>
  pull(relative_skill)
#> The following messages were produced when checking inputs:
#> 1.  144 values for `prediction` are NA in the data provided and the corresponding rows were removed. This may indicate a problem if unexpected.
#> [1] 1.2947445 2.2958723 0.8156514 0.5966310 0.7475873 0.9469157 0.9765276


ex <- example_quantile |>
  mutate(
    arbitrary_colum = rep(
      c("Coruscant", "Endor", "Dagobah", "Tatooine", "Naboo"), 
      20545/5))
check_forecasts(ex)
#> The following messages were produced when checking inputs:
#> 1.  144 values for `prediction` are NA in the data provided and the corresponding rows were removed. This may indicate a problem if unexpected.
#> Warning in check_forecasts(ex): The following warnings were produced when checking inputs:
#> 1.  Some forecasts have different numbers of rows (e.g. quantiles or samples). scoringutils found: 5, 4. This is not necessarily a problem, but make sure this is intended.
#> Your forecasts seem to be for a target of the following type:
#> $target_type
#> [1] "integer"
#> 
#> and in the following format:
#> $prediction_type
#> [1] "quantile"
#> 
#> The unit of a single forecast is defined by:
#> $forecast_unit
#> [1] "location"        "target_end_date" "target_type"     "location_name"  
#> [5] "forecast_date"   "model"           "horizon"         "arbitrary_colum"
#> 
#> Cleaned data, rows with NA values in prediction or true_value removed:
#> $cleaned_data
#> Key: <location, target_end_date, target_type>
#>        location target_end_date target_type true_value location_name
#>          <char>          <Date>      <char>      <num>        <char>
#>     1:       DE      2021-05-08       Cases     106987       Germany
#>     2:       DE      2021-05-08       Cases     106987       Germany
#>     3:       DE      2021-05-08       Cases     106987       Germany
#>     4:       DE      2021-05-08       Cases     106987       Germany
#>     5:       DE      2021-05-08       Cases     106987       Germany
#>    ---                                                              
#> 20397:       IT      2021-07-24      Deaths         78         Italy
#> 20398:       IT      2021-07-24      Deaths         78         Italy
#> 20399:       IT      2021-07-24      Deaths         78         Italy
#> 20400:       IT      2021-07-24      Deaths         78         Italy
#> 20401:       IT      2021-07-24      Deaths         78         Italy
#>        forecast_date quantile prediction                 model horizon
#>               <Date>    <num>      <int>                <char>   <num>
#>     1:    2021-05-03    0.010      82466 EuroCOVIDhub-ensemble       1
#>     2:    2021-05-03    0.025      86669 EuroCOVIDhub-ensemble       1
#>     3:    2021-05-03    0.050      90285 EuroCOVIDhub-ensemble       1
#>     4:    2021-05-03    0.100      95341 EuroCOVIDhub-ensemble       1
#>     5:    2021-05-03    0.150      99171 EuroCOVIDhub-ensemble       1
#>    ---                                                                
#> 20397:    2021-07-12    0.850        352  epiforecasts-EpiNow2       2
#> 20398:    2021-07-12    0.900        397  epiforecasts-EpiNow2       2
#> 20399:    2021-07-12    0.950        499  epiforecasts-EpiNow2       2
#> 20400:    2021-07-12    0.975        611  epiforecasts-EpiNow2       2
#> 20401:    2021-07-12    0.990        719  epiforecasts-EpiNow2       2
#>        arbitrary_colum
#>                 <char>
#>     1:           Endor
#>     2:         Dagobah
#>     3:        Tatooine
#>     4:           Naboo
#>     5:       Coruscant
#>    ---                
#> 20397:       Coruscant
#> 20398:           Endor
#> 20399:         Dagobah
#> 20400:        Tatooine
#> 20401:           Naboo
#> 
#> Number of unique values per column per model:
#> $unique_values
#>                    model location target_end_date target_type true_value
#>                   <char>    <int>           <int>       <int>      <int>
#> 1: EuroCOVIDhub-ensemble        4              12           2         96
#> 2: EuroCOVIDhub-baseline        4              12           2         96
#> 3:  epiforecasts-EpiNow2        4              12           2         95
#> 4:       UMass-MechBayes        4              12           1         48
#>    location_name forecast_date quantile prediction horizon arbitrary_colum
#>            <int>         <int>    <int>      <int>   <int>           <int>
#> 1:             4            11       23       3969       3               5
#> 2:             4            11       23       3733       3               5
#> 3:             4            11       23       3903       3               5
#> 4:             4            11       23       1058       3               5
#> 
#> $messages
#> [1] "144 values for `prediction` are NA in the data provided and the corresponding rows were removed. This may indicate a problem if unexpected."
#> 
#> $warnings
#> [1] "Some forecasts have different numbers of rows (e.g. quantiles or samples). scoringutils found: 5, 4. This is not necessarily a problem, but make sure this is intended."

ex |>
  score() |>
  summarise_scores(by = c("model", "target_type"), relative_skill = TRUE) |>
  pull(relative_skill)
#> The following messages were produced when checking inputs:
#> 1.  144 values for `prediction` are NA in the data provided and the corresponding rows were removed. This may indicate a problem if unexpected.
#> Warning in check_forecasts(data): The following warnings were produced when checking inputs:
#> 1.  Some forecasts have different numbers of rows (e.g. quantiles or samples). scoringutils found: 5, 4. This is not necessarily a problem, but make sure this is intended.
#> Warning in pairwise_comparison(scores = scores, metric =
#> relative_skill_metric, : Some values for the metric 'interval_score' are NA.
#> These have been removed. Maybe choose a different metric?
#> [1] 1.0000000 1.8335767 1.0000000 1.0000000 1.0000000 1.0000000 0.5453821

^{Created on 2023-01-13 with reprex v2.0.2}

[nikosbosse]

Yeah, I checked this and it's expected behaviour.

Maybe my use case was a little stupid, what happened was:

• I did the checks on the data
• computed the scores
• appended a column to the scores after they were computed
• summarised those modified scores, which resulted in unnoticeable changes.

Maybe we can just assume people aren't that stupid? Alternative would be to either

• have pairwise comparison always output a message that gives you the current forecast unit
• warn about doing that explicitly somwhere in the docs

What do you think?

[seabbs]

Yeah, I checked this and it's expected behaviour.

Panic over!!

warn about doing that explicitly somewhere in the docs

I think at a minimum this one as you ran into but having pairwise_comparison output a message might be nice (this might also confuse people when used in score() as I am not sure it would be clear where it was coming from.

[nikosbosse]

in summarise_score() you mean? yeah that's what I did. But summarise_score() calls pairwise_comparison(), so I think that should be fine

[nikosbosse]

One other thing I was thinking about: Maybe it would be a good idea to add a forecast_unit argument back in, at least for the lower level functions. That could make the code more robust when used in production somewhere. All higher level functions use ... anyway, so we wouldn't have to make the documentation more complicated.