score_forecasts() not working right for huge numbers of forecasts

[nickreich]

I can't tell for sure, but I think this might have to do with the way scoringutils is parsing huge numbers of rows - there is some aggregating going on somewhere in the following code. I haven't been able to pinpoint yet if it is in covidHubUtils or in scoringutils. @nikosbosse any ideas?

This issue is based on some code that @eycramer was running and I modified into a "simple" example below.

library(covidHubUtils)
library(tidyverse)

the_locations <- covidHubUtils::hub_locations %>% filter(geo_type == "state") %>% pull(fips)

mondays <- seq(as.Date("2020-11-09"), as.Date("2020-12-07"), by = "week")

## load truth data
truth_dat_inc <- load_truth(
  truth_source = "JHU",
  target_variable = "inc death",
  locations = the_locations)

## load and score a lot of forecasts (~1m rows)

### load forecasts
forecasts_inc <- map_dfr(
  mondays, 
  function(the_weeks) {
    load_latest_forecasts(
      last_forecast_date = the_weeks,
      forecast_date_window_size = 7,
      locations = the_locations,
      types = "quantile",
      targets = paste(1:4, "wk ahead inc death"),
      source = "zoltar")
  }
)

### score forecasts
score_inc <- score_forecasts(forecasts = forecasts_inc,
  truth = truth_dat_inc)
## message is printed here that says:
## "Aggregate function missing, defaulting to 'length'"

### view a selected few, all WIS scores are identical
score_inc %>% 
  filter(location=="US", horizon==1, forecast_date == "2020-11-08") %>%
  select(model, location, horizon, forecast_date, wis, abs_error)

## load and score just a small set of forecasts
forecasts_inc_small <- load_latest_forecasts(
      last_forecast_date = "2020-11-08",
      forecast_date_window_size = 7,
      locations = "US",
      types = "quantile",
      targets = paste(1:4, "wk ahead inc death"),
      source = "local_hub_repo",
      hub_repo_path = "../covid19-forecast-hub")

### score the small set, no message returned
score_inc_small <- score_forecasts(forecasts = forecasts_inc_small,
  truth = truth_dat_inc)

### view a subset, here the WIS are unique
score_inc_small %>% 
  filter(location=="US", horizon==1, forecast_date == "2020-11-08") %>%
  select(model, location, horizon, forecast_date, wis, abs_error)

[nickreich]

duplicate of #91