Cross Year Aggregations

[rmartz]

Overview

Introduces the ability to perform time aggregations across year boundaries, allowing winter measurements to be handled as a single segment using the timespan value offset_yearly.

Prerequisite for #169

Demo

"2097-2098": {
  "max": 30.30512817382882,
  "avg": 18.430609043666067,
  "min": -0.6856005859384425
},
"2051-2052": {
  "max": 22.797236328124978,
  "avg": 12.2029003906249,
  "min": -2.618150634765442
},
"2029-2030": {
  "max": 21.10369384765582,
  "avg": 11.077372000558032,
  "min": -15.550047607421233
},
"2022-2023": {
  "max": 17.06918457031156,
  "avg": 7.002739083426256,
  "min": -3.1465655517586018
},
"2099-2100": {
  "max": 27.82869995117114,
  "avg": 22.212894461495313,
  "min": 10.794453124999979

Notes

• When performing a cross-year aggregation, partial year data at the earliest and latest data points are discarded as there is not another year's data to pair them with.
• The query for cross-year aggregation is currently very slow, and requests may take 3-10 seconds to complete
• This completely removes the daily indicator aggregation that was kinda sorta removed in Remove Daily Raw Data Indicators #172 because it's not trivial to construct the ISO-8601 format in Django querysets

Testing Instructions

• Ensure tests still pass
• Run two requests for min_low_temperature side by side, one for time_aggregation=offset_yearly and the other for time_aggregation=yearly
• Pick years from the yearly instance and compare their min value to the min in offset_yearly for winters ending or starting with that year
• • In many instances, the min should be in the starting year, others in the ending year
• • In some instances, the min for the winter span may be a completely different value higher or lower, based on the fact that the coldest point in two sequential years of yearly aggregation can skip an entire winter by occurring in late winter of one and early winter of the other.

Checklist

• Do tests pass?
• Does this PR require an update to the API Documentation?

Connects #198

[KlaasH]

Small issue: I'm getting 2006-2007 and 2099-2100 values from http://localhost:8080/api/climate-data/31/RCP85/indicator/frost_days?time_aggregation=offset_yearly (i.e. it's not excluding spans that are missing one side of the data)

Other than that, just a few small comments below.

[KlaasH]

Stale comment. This one doesn't use get_intervals.

[KlaasH]

Is it worth adding an __init__ with an assert that the intervals add up to a year? It would just be protecting against errors on our part, but it seems like the sort of error that could crop up and it would be nice for it to present in an obvious way if it does.

[rmartz]

That's a good thought, I think I'll try to do that in a unit test so we don't have to wait to hit a runtime error to detect a class definition problem.

[KlaasH]

Maybe add a docstring note here that "Custom" means "custom within a year".

[rmartz]

2006 and 2100 should be valid dates, they're included in the GDDP data file list for RCP45 and RCP85. Is the data for those intervals coming back weird? If it's getting cut off an indicator like heating_degree_days will be very low because it only has data for half the winter.

[rmartz]

That's a good thought, I think I'll try to do that in a unit test so we don't have to wait to hit a runtime error to detect a class definition problem.

[rmartz]

The edge year problem should be fixed, turns out some scenarios have data to 2100 and others only to 2099, so we had to add some logic to detect what the year boundaries are for a given data source and filter for the days we can use within that. A bit complex, but thankfully not as bad as feared.

[KlaasH]

Well, I've somehow destroyed or lost track of my database container, so re-testing that last little change would be a major production. But it looks good. 👍