Testing: Make sure pit() and plot_pit() are correct

[nikosbosse]

Check the functions thoroughly and write more tests for them to make sure they are correct.

• in the past, we sometimes had instances where the bins did not sum to 1
• think about how pit_sample() fits in with the other metrics - it's return type is different, because it is a many-to-many relationship.

Maybe related: tidyverse/ggplot2#2323

[nikosbosse]

@sbfnk could we interest you in having a look at this one? :)

[sbfnk]

I'd be thrilled.

[nikosbosse]

In addition, found this in my notes:

• get_pit should call pit_sample() in the sample case and quantile_coverage in the quantile case and we should be able to say that in the docs. Also probably get_pit() should not necessarily have the n_replicates arg.
• plot_wis has this:

  #' @param relative_contributions Logical. Show relative contributions instead 
  #' of absolute contributions? Default is `FALSE` and this functionality is not 
  #' available yet.
  

But maybe that's a different issue...

[nikosbosse]

@sbfnk I imagine the thrill may have died down since April where you so kindly and voluntarily volunteered to have a look at this :) Is this still something you'd be willing to have a look at?

[nikosbosse]

I found an issue with the pit plot breaks:

 observed <- rnorm(30, mean = 1:30)
  predicted <- replicate(200, rnorm(n = 30, mean = 1:30))
  p2 <- plot_pit(pit, num_bins = 5)

produces this plot with its lovely 4 bins...

[sbfnk]

get_pit should call pit_sample() in the sample case and quantile_coverage in the quantile case and we should be able to say that in the docs. Also probably get_pit() should not necessarily have the n_replicates arg.

Wouldn't the easiest way to implement different behaviours by forecast type to make get_pit() an S3 method?

[nikosbosse]

Wouldn't the easiest way to implement different behaviours by forecast type to make get_pit() an S3 method?

ahh. yeah... now that you say it that seems very obviously a good idea

[seabbs]

when in doubt make it an s3 methods the new scoringutils catch phrase.