score_forecasts() not working when there are nearly 1m rows.

[nickreich]

I can't tell for sure, but I think this might have to do with the way scoringutils is parsing huge numbers of rows - there is some aggregating going on somewhere in the following code. I haven't been able to pinpoint yet if it is in covidHubUtils or in scoringutils. @nikosbosse any ideas?

This issue is based on some code that @eycramer was running and I modified into a "simple" example below.

library(covidHubUtils)
library(tidyverse)

the_locations <- covidHubUtils::hub_locations %>% filter(geo_type == "state") %>% pull(fips)

mondays <- seq(as.Date("2020-11-09"), as.Date("2020-12-07"), by = "week")

## load truth data
truth_dat_inc <- load_truth(
  truth_source = "JHU",
  target_variable = "inc death",
  locations = the_locations)

## load and score a lot of forecasts (~1m rows)

### load forecasts
forecasts_inc <- map_dfr(
  mondays, 
  function(the_weeks) {
    load_latest_forecasts(
      last_forecast_date = the_weeks,
      forecast_date_window_size = 7,
      locations = the_locations,
      types = "quantile",
      targets = paste(1:4, "wk ahead inc death"),
      source = "zoltar")
  }
)

### score forecasts
score_inc <- score_forecasts(forecasts = forecasts_inc,
  truth = truth_dat_inc)
## message is printed here that says:
## "Aggregate function missing, defaulting to 'length'"

### view a selected few, all WIS scores are identical
score_inc %>% 
  filter(location=="US", horizon==1, forecast_date == "2020-11-08") %>%
  select(model, location, horizon, forecast_date, wis, abs_error)

## load and score just a small set of forecasts
forecasts_inc_small <- load_latest_forecasts(
      last_forecast_date = "2020-11-08",
      forecast_date_window_size = 7,
      locations = "US",
      types = "quantile",
      targets = paste(1:4, "wk ahead inc death"),
      source = "local_hub_repo",
      hub_repo_path = "../covid19-forecast-hub")

### score the small set, no message returned
score_inc_small <- score_forecasts(forecasts = forecasts_inc_small,
  truth = truth_dat_inc)

### view a subset, here the WIS are unique
score_inc_small %>% 
  filter(location=="US", horizon==1, forecast_date == "2020-11-08") %>%
  select(model, location, horizon, forecast_date, wis, abs_error)

[elray1]

I'm getting a similar error with a similar number of forecasts to score, but another error:

Aggregate function missing, defaulting to 'length'
Error in `[.data.table`(quantile_data, , `:=`(aem, abs_err(true_value,  : 
  Supplied 4 items to be assigned to group 1 of size 92 in column 'aem'. The RHS length must either be 1 (single values are ok) or match the LHS length exactly. If you wish to 'recycle' the RHS please use rep() explicitly to make this intent clear to readers of your code.

Unfortunately my code is not highly reproducible. Maybe best to ignore my error and solve Nick's better example for now, unless you have specific ideas about what might be happening.

[nikosbosse]

They both look like scoringutils errors (usually something like that happens when grouping is not unique, i.e. there are two forecasts where there should be one. But not sure that's the case here). I'm going to investigate it.

[nikosbosse]

I think the issue is that there are duplicates in the data. Once I remove the duplicates, the function works as expected as far as I can tell

[nikosbosse]

some more diagnostics:

[nickreich]

Super helpful, thanks for the diagnostics, Nikos! I am going to close this issue, although:

• I will open one on scoringutils to check for duplicated rows - seems sensible to at least throw a more clear warning/error.
• I will track down the source of this problem on our end as well!