Define and calculate composites with temporal dependencies #1748
Comments
gerritholl
added
component:compositors
component:multiscene
future ideas
|
I'm very torn about how this could and how this should be implemented. From an xarray point of view the user should probably be in complete control over resampling, especially in time. It seems wrong to put that requirement/specification in something like a composite definition. However, I also see how a user might not know what's best and there should be a logical default and that that default could depend on the algorithm that will be applied. The other part of this that is difficult is that this request involves a lot of various steps (grouping files, loading datasets, loading composites/composite definitions, temporal resampling/aggregation, time-based requests in composite-like definitions, etc) and it is hard to think of what parts should be something the user directly has control over and what parts are just kind of magically configured in this temporal-composite definition. So I don't have any real solutions, just wanted to voice my concerns. |
|
The user would remain in control. What I'm after is creating the infrastructure for the user to integrate time-dependent products into Satpy. Today, a user-developer can create a custom Compositor and use that to implement any algorithm that is based on a single Scene. For example, fogpy creates the "composites" |
|
To clarify, what I meant by user control, instead of something like mscn = MultiScene(...)
time_dataset = mscn.blend(time_series)
t_resampled_dataset = time_dataset.satpy.resample(method='temporal_average', threshold="T-15", align="left")
result = t_resampled_dataset.satpy.run_algorithm('some_name')
scn = Scene()
scn['my_result'] = result
scn.save_datasets()This is an extremely bad example for what is probably better and more common practice with xarray objects, but it demonstrates the idea of "hide everything behind a single call" versus "break each step into a separate interface that the user has to control". I realize that this makes trollflow2 stuff harder as well. Maybe these interfaces exist, but are used by temporal composites underneath. |
|
I suppose I should have opened two issues, as there are really two related but distinct cases here:
|
|
This could also be used for "dynamic RGBs", such as R=10.8, G=10.8[t-30 minutes], B=10.8, which might help forecasters as an additional tool (in addition to loops) to highlight areas with (rapid) changes. |
|
Taking up the idea of @gerritholl 's 'dynamic RGB' that gives the forecaster the opportunity to judge/measure the evolution of a cloud system, here is a sample image of what it could look like if you assign cloud imagery at t0-2, t0-1,t0 to the R,G,B planes, respectively. The sample (HRV SEVIRI band) shows the evolution of a fog patch (and other low cloud patches) over the Caspian Sea - blue/red is most/least recent position/occurrence of the feature. If the time steps are equally spaced you glean acceleration(deceleration of the system in one go (in the sample the fog appears to move with constant speed while loosing its tail on the way).
|
|
I added this issue to the PCW 2023 project board. I think this could be further split to the following sub-tasks:
|
|
(Continuing the discussion from #2488 here)
Unless we add a way for loading time-dependent composites directly from the MultiScene. But as @djhoese pointed out, a MultiScene does not have to consist of multiple Scenes with the same area but different timesteps. It can also be multiple Scenes with the same time, but different areas. I keep forgetting that use case because I've never used it. |
|
Oh, I didn't know that different area possibility! I'll dig in to how dependencies are loaded/handled currently in |
|
@pnuu So you're saying you want a criteria in the dependency loading to pull from the existing DataArrays in the |
I guess was just thinking doing the usual The blended |
Actually I "knew" that existed because I had tested the temporal RGB stuff also with polar orbiter data, just didn't realise the inherently were from different areas 🙈 |
|
I like the idea to first blend the MultiScene into a Scene as a timeseries, and then load the temperal composites. This requires to load from a blended scene — see #1749 (an issue I opened around 2 hours after this one, so probably I had this thought before). #1749 was closed as completed, but actually loading from a blended scene is still not possible (rather the use case in #1749 was covered in a different way): import hdf5plugin
from glob import glob
from satpy.multiscene import MultiScene, timeseries
from satpy.utils import debug_on; debug_on()
ms = MultiScene.from_files(
glob("/media/nas/x21308/MTG_test_data/2022_05_MTG_Testdata/RC010[012]/*BODY*.nc"),
reader="fci_l1c_nc",
group_keys=["repeat_cycle_in_day"])
channels = ["ir_105", "ir_133", "wv_63", "wv_73"]
sc = ms.blend(blend_function=timeseries)
sc.load(channels)this fails with |
If I understand correctly, that would be something like: then after blending the temporal composite would be generated? |
|
Yes, that's what I was thinking. |
|
We already have an satpy/satpy/composites/__init__.py Lines 49 to 50 in 72893bc |
|
My understanding based on the discussion we had earlier today: the initial A problem: on the new scene, |
|
Possible solution: rather than starting with an empty scene, That might have consequences I do not currently oversee. It also doesn't address the possibility of different wishlists between scenes. Would we ever expect different wishlists between scenes in a MultiScene? |
More importantly the timeseries blend function gets
I'm scared of this too. I think if we're really worried about making |
|
How would the MultiScene or the timeseries blend function recognise a temporal composite?
|
|
They could all derive from a common base class, which would check for a time dimension in with the Satpy requirement management figuring out the prerequisites. Rather, we would have with the temporal requirement being enforced in the code and not encoded in the YAML. The downside of the second approach is that it makes it harder to experiment with different ways of combining time-steps in different RGBs or other composites, as this would be hardcoded in the compositor rather than configurable in YAML. Unless a base TemporalCompositor class would somehow do a second pass of requirement processing... |
Another option is to do it like I did in #2488 and assume a fixed number of inputs (currently in a specific order). This wouldn't be using the compositor configuration interface at all for the temporal part, though. The datasets/composites (in your example |
|
I am interested in a framework that allows the user to use the compositor configuration interface for the temporal part. |
|
A base class sounds fine but as discussed in the meeting today (my morning, your afternoon), the MultiScene shouldn't need to know the type of the composite. It should "just try" by generating possible composites and if the composite can't be made it still raises the incompatible times (or whatever) or incompatible areas (if not resampled or can't handle time-based DataArrays) and those composites remain as "missing". I think a base class would be nice for time-based composites. Possibly a questionable idea, but you could overwrite |
|
By the time the compositor is called the second time, it is passed a 3-dimensional data-array coming from |
|
I don't think you can have a dimension coordinate that is not monotonic (always decreasing or always increasing) in an xarray DataArray. |
|
Right, so there's nothing to do at the compositor stage, is there? If the input data have messed up times that should fail by the latest when calling |
|
I think that's correct, yes. Well...I guess if there was no |
|
#2495 depends on a time coordinate being available. |
Feature Request
Is your feature request related to a problem? Please describe.
Satpy can calculate composites from datasets within a single
Scene. For some use cases, it would be desirable to calculate composites from datasets from multiple scenes, such as multiple time steps in aMultiScene. In nowcasting, the temporal component contains important information for forecasting the near future: rising air parcels may indicate convective initiation, a vertically growing cloud may indicate a rapidly developing thunderstorm, a cloud growing rapidly horizontally may tell us something about the expected lifetime of a convective system, and a rapid increase in lightning density likewise. Currently, it is difficult to build such time-dependent "composites" into Satpy.Describe the solution you'd like
I would like to be able to define composites that need multiple time steps of a certain variable. Such composites could only be loaded from a
MultiScene, or maybe from a blendedScenewhere the datasets have a time component.In
composites.yaml, we would define what time range or time steps are needed. Maybe a product that needs 6.2 µm and 7.3 µm for the present and the previous scene could be defined as:The compositor would be passed four inputs: two from the present scene, and two from the previous scene.
Or a composite that needs the last five minutes of flash extent density:
The compositor would be passed a
DataArraywith a time dimension, containing the last five minutes offlash_extent_density.Inevitably, there will be time steps, probably the first time steps, where those conditions cannot be met. For those time steps, the composite would be filled with fill values.
Compositors could also reduce the data. If we have a
MultiScenewith sixtyScenes containingflash_extent_density, oneScenefor every minute, the second example above could be used to calculate a running average starting at t=5 minutes. But what if, instead of a running average, we want to get aMultiScenewith only 12 scenes, each containing the summedflash_extent_densityfor a block of 5 minutes? It would be interesting if we could "load" this in Satpy as well, but this has more potential caveats than non-reducing operations. Maybe rather than a.load(...)operation that reduces the number of scenes in theMultiScene, this should have an API that returns a new, reducedMultiSceneleaving the original intact.Describe any changes to existing user workflow
In principle, there should be no backward compatibility concerns, as this concerns a new feature. However, it's possible that it would require such an overhaul of
MultiSceneand perhapsScene, that user workflow or existing unit tests would need to be changed. I don't foresee right now what those changes would be.Additional context
I have code that takes a
MultiSceneand calculates a value based on the Δ between two subsequent scenes (akin to the first example above), and then manually assigns this to eachScene. That works, but it does not very well fit in theSatpyframework. TheSceneclass is not really aware of the resulting dataset (see also, for example, #1747), and I cannot define this procedure in a composites file. An even dirtier workaround that would be possible in present satpy is to create a reader that reads multiple time steps and defines datasets such as "BT73_1_minute_ago". That would be a bad solution for various reasons.See also #1747 and #1469.
The text was updated successfully, but these errors were encountered: