Additional proper scoring rules

[seabbs]

@sbfnk suggests adding another proper scoring rule into the mix such as the absolute or relative error (to allow comparison of point estimate vs distribution performance). Another alternative would be to use the Weighted interval score approximation to the CRPRS and then show the decomposition into over/under prediction + sharpness.

On the face of it this all makes sense. We are a bit space constrained, already show relative performance, and have just quite a lot going on. That being said I think it would make sense to try adding and if we think its overwhelming put into the SI and just discuss and its useful information.,

[parksw3]

I think this is a good idea, especially since other people will be more familiar with things like absolute errors.

We are a bit space constrained...

I also agree with this. And also agree with putting into the SI.

I had some summary statistics that I wanted to see as well and so will take a crack at it.

[parksw3]

Here are three plots showing bias (the more classical one), RMSE, and coverage. I think these three plots would be pretty intuitive for most people? I feel like fancier summary statistics might be a bit more difficult for broader audience. So I think it's a good idea to show the traditional ones like these.

[seabbs]

Just realised I didn't respond to this with actual words. These are nice and we should put them in. Not sure how we should compare for space etc but lets give it a whirl