Extract additional NWP fields

[JackKelly]

Prep work:

• Search through SNS messages to find exact names used for the NWP fields we want.
• Refactor ec2.py. Maybe need new way to represent which fields we want; possibly in a YAML file? e.g. [{'name': 'humidity', 'height_meters': 'ALL', 'ensemble_members': ['UKV', 'CONTROL'], 'compression': 'RLE', 'mask': 'land'}. members defaults to all members; and set sane defaults for compression and mask.

Additional NWP fields to add:

• irradiance at the surface
• apply land mask to irradiance
• temperature at the surface (for PV)
• wind speed, direction & gust at the surface (for PV)
• cloud fraction & cloud type

To reduce space requirements, save these for just UKV and the control ensemble member:

• At multiple vertical levels, up to ~ 10 km (to help nowcast clouds):
  • temperature
  • wind speed, direction & gust
  • humidity
• At surface:
  • rainfall (probably affects PV, right?)
  • snow
  • rainfall & snow will be mostly zeros; so will benefit from compression which squishes zeros.

[JackKelly]

All MOGREPs-UK params (and NetCDF file sizes in MB):

air_pressure                                                                             225.058732
air_pressure_at_sea_level                                                                  3.286515
air_temperature                                                                           22.361765
atmosphere_boundary_layer_thickness                                                        3.487302
cloud_area_fraction                                                                        2.294713
cloud_area_fraction_assuming_only_consider_surface_to_1000_feet_asl                        1.132317
cloud_base_altitude_assuming_only_consider_cloud_area_fraction_greater_than_2p5_oktas      1.230863
cloud_base_altitude_assuming_only_consider_cloud_area_fraction_greater_than_4p5_oktas      1.169574
cloud_volume_fraction_in_atmosphere_layer                                                 32.069940
dew_point_temperature                                                                      4.121010
fog_area_fraction                                                                          0.290074
freezing_level_altitude                                                                    4.141373
high_type_cloud_area_fraction                                                              1.398838
land_binary_mask                                                                           0.209811
low_type_cloud_area_fraction                                                               2.290723
lwe_graupel_and_hail_fall_rate                                                             0.147302
lwe_snowfall_rate                                                                          0.141513
lwe_thickness_of_graupel_and_hail_fall_amount                                              0.146463
lwe_thickness_of_snowfall_amount                                                           0.142890
lwe_thickness_of_surface_snow_amount                                                       0.370675
medium_type_cloud_area_fraction                                                            1.513929
number_of_lightning_flashes_per_unit_area                                                  0.139374
rainfall_rate                                                                              0.543209
relative_humidity                                                                         71.864523
soil_temperature                                                                           5.233638
surface_air_pressure                                                                       6.969847
surface_altitude                                                                           2.405621
surface_diffusive_downwelling_shortwave_flux_in_air                                        4.824533
surface_direct_downwelling_shortwave_flux_in_air                                           4.574237
surface_downwelling_longwave_flux_in_air                                                   6.172594
surface_downwelling_shortwave_flux_in_air                                                  5.174557
surface_downwelling_ultraviolet_flux_in_air                                                1.040426
surface_temperature                                                                        2.239603
surface_upward_sensible_heat_flux                                                          4.852735
surface_upwelling_ultraviolet_flux_in_air                                                  0.384810
thickness_of_rainfall_amount                                                               0.628758
total_radar_reflectivity                                                                  24.493266
total_radar_reflectivity_max_in_column                                                     3.957536
upward_air_velocity                                                                      114.111978
visibility_in_air                                                                          7.149662
wet_bulb_freezing_level_altitude                                                           4.416840
wet_bulb_potential_temperature                                                             4.893202
wind_from_direction                                                                       80.949014
wind_speed                                                                                35.525234
wind_speed_of_gust                                                                         3.247313

[JackKelly]

For now, we want:

• air_temperature (1.5m; not at 5, 10, ..., 6,000m)
• surface_diffusive_downwelling_shortwave_flux_in_air (no height)
• surface_direct_downwelling_shortwave_flux_in_air (no height)
• surface_downwelling_shortwave_flux_in_air (or is this just direct + diffuse?) (no height)
• surface_temperature (no height)

Long term, we want:

• air_temperature (available at 1.5m; and 5, 10, ..., 6,000m)
• surface_diffusive_downwelling_shortwave_flux_in_air (no height)
• surface_direct_downwelling_shortwave_flux_in_air (no height)
• surface_downwelling_shortwave_flux_in_air (or is this just direct + diffuse?) (no height)
• surface_temperature (no height)
• upward_air_velocity (available with no height, and at 5, 10, ..., 6,000m)

At cloud heights:

• cloud_volume_fraction_in_atmosphere_layer (available at 5, 10, ..., 6,000m)
• relative_humidity (available at 5, 10, ..., 6,000m)

[JackKelly]

TODO (short term):

• Modify code so it accepts messages with the heights we want (at present, MetOfficeMessage.is_wanted() will return False if the message isn't multi-level). See ideas about refactoring at top of this issue.
• Change filter on SQS to accept new params.