Check 1 wk ahead forecast is somewhere near data

[kathsherratt]

Check that the 1 wk ahead incident point forecast is within +/- 20% of either

• last data point (last week incidence) or
• three week mean incidence

Not very sophisticated but looks like this should help catch some of the odder forecasts.

It would be good to add something to the evaluation plots so we can tell immediately which ones get dropped in this check. Will look at adding but would be glad for review of this step first. Happy to take suggestions on some alternative metric for checking for outlandish forecasts.

[seabbs]

Thanks for this Kath - a good idea.

So three points.

1. I think 20% might be too little as a reasonable observed doubling time could be something like 7 days.
2. I assume that most of the crazy is coming from the Rt forecast (or maybe not?) and so it might be better to do model by model checks and then reset crazy forecasts to something less mad or drop them from the ensemble (but that gets quite complicated when weighting as if a model is not present the weighting will no longer add up to 1).
3. My preference is that we start submitting forecasts for all of the targets always. Given that it would be good if we swapped crazy forecasts with some kind of null model (no change with overdispersion from recent obs - I think @nikosbosse has code for this).

For now happy to go ahead with this approach but perhaps bump the check up to 200% of observed data (I realise this will likely lead to some crazy getting through but any lower feels like it might exclude forecasts that aren't outliers (but are likely bad...).

So for 3. the pseudo code would be if not present add in null model forecast and otherwise don't?

[kathsherratt]

Thanks, Sam - changed to exclude forecasts which are less than half or more than double recent data.

The other point about checking each model is prob best dealt with separately (#86).

I guess we can merge this PR and use until we have that in place, or close the PR and focus on checking individual models - don't mind either way.

[seabbs]

Is it okay if we hold off on this for a few days. Can't quite work out what is/isn't a good idea.....

[kathsherratt]

great - no rush from me. will just leave open while we work out what to do next.