Move computation of p-values out of get_pairwise_comparisons()?

[nikosbosse]

Currently, three functions exist that do something related to pairwise comparisons:

• get_pairwise_comparisons() outputs a data.table with three different things:
  • pairwise mean score ratios between models
  • relative skill scores (geometric mean of the mean score ratios)
  • pairwise p-values
• add_relative_skill() adds relative skill scores to an existing scores object (calling get_pairwise_comparisons()`)
• plot_pairwise_comparisons() visualises either mean score ratios or the p-values.

Should the calculation of p-values and mean score ratios/relative skill scores be done by the same function?

Pro:

• both mean score ratios and computation of p-values require a similar mechanic in which two models are compared against each other

• the function is currently called get_pairwise_comparisons() so it makes sense to have those two things together

Contra:

• it might be cleaner to separate the code a bit. Both get_pairwise_comparisons() and plot_pairwise_comparisons() currently have code for both things.
• the workflows feel a bit different. The mean score ratios feel a bit more like "visualisation of performance" and the p-values feel more like "rigorous statistical testing".

In terms of currently suggested workflows we have the following:

For getting relative skill scores, you call as_forecast(data) |> score() |> add_relative_skill().

For visualising mean score ratios, you call

pairwise <- example_quantile |>
  as_forecast() |>
  score() |>
  get_pairwise_comparisons() 

plot_pairwise_comparisons(pairwise)

For visualising p-values, you call

plot_pairwise_comparisons(pairwise, type = "pval")

We previously even had a nice plot that showed both p-values and mean score ratios in a single plot (using the upper and lower triangle), but that broke and we ditched a while ago.

---

Options:

1. leave everything as is for now.
2. remove computation of p-values for now. Maybe rename get_pairwise_comparisons() to get_score_ratios(). Re-introduce functionality later.
3. do a rewrite with two different workflows before the next CRAN release of version 2.0.0
4. other

[nikosbosse]

@sbfnk @seabbs
@nickreich @elray1 maybe you have thoughts or preferences as well?

[sbfnk]

4. Could get_pairwise_comparisons() have a metric (or the like) option which could be mean_score_ratio (default) or p_value? The plot function could then plot whichever is there.

[nikosbosse]

4. Could get_pairwise_comparisons() have a metric (or the like) option which could be mean_score_ratio (default) or p_value? The plot function could then plot whichever is there.

It would get two metric arguments then :). My intuition is to prefer the status quo over that proposal for the following reasons:

• we'd have to introduce and name an additional argument (or rather, the argument that decides what to do would be moved from plot_pairwise_comparisons() to get_pairwise_comparisons(). To me, the status quo feels a bit simpler.
• we would still have the same code complexity issue with two functions covering two slightly different use cases.

[seabbs]

I think my preference is 1 (i.e do nothing)

[nikosbosse]

ok. Moving this to a later release then.