Dataset metadata spec

[jpoehnelt]

Just getting it down somewhere...

[cholmes]

I think it could be good here to not reuse datetime and geometry, as those are item specific, and instead really talk about the extent.

Ideal to me would be to align with WFS3, so we can add the dataset fields directly to their 'collection' metadata. They have it like:

"extent": {
"spatial": [ 7.01, 50.63, 7.22, 50.78 ],
"temporal": [ "2010-02-15T12:34:56Z", "2018-03-18T12:11:00Z" ]
},

See 'example 4' on https://rawgit.com/opengeospatial/WFS_FES/master/docs/17-069.html

I could be up for a different form, like flatten to spatial_extent and temporal_extent - but I don't think we should reuse the item fields, and I don't believe a dataset should need to conform to the Item schema. If we go with something different I'd want to advocate to WFS3 group to align with us. And I'd see a WFS 3 compliant STAC implementation using these dataset fields in their /collections/{collectionId}/ endpoint.

[m-mohr]

I'm fine with renaming it to spatial extent and temporal extent, we just used those names for consistency within STAC. (Edit: Changed.)
I don't like the the way temporal extents are specified in WFS though as it doesn't allow to leave the end time out (i.e. open date ranges).

[cholmes]

What is this? And what's the definition of it? May make sense as an extension?

[m-mohr]

Process graph was added for provenance, but will definitely be an extension. The same applies for dimensions. Changed that, too.

[matthewhanson]

Is region primarily an AWS thing or is it general to all cloud providers?

[m-mohr]

The schema was written before we knew about the idea of storage profiles in #148. I would really like to have profiles instead of having the storage details directly baked into the dataset spec.

region would probably be an AWS specific thing, which should be in a separate profile as proposed in #148. If others have regions, too, then they should have it separately in their profiles aswell.

[simonff]

GCS has regions too

[matthewhanson]

Ok, that makes sense though - each cloud provider should define it's own storage profile (extension?)

[m-mohr]

I'd go that route, yes. Is that dataset specific or could that be also catalog or item specific?

[matthewhanson]

If some sort of unit field is provided it should probably be per-band. But I'm not convinced it's needed, what are the possible values here?

[m-mohr]

Yes, makes sense to have it per band (and per dimension). Moved it to the bands. An example would be °C for temperatures, but could be any unit of measurement. Preferably SI, but could also link to UDUNITS or the dictionary of UoM.

[matthewhanson]

I do think nodata, scale, and offset are useful to have per band, but not sure about data_type. The other fields may not be provided in the actual datafile (although providers should be encouraged to set them), but data_type will always be available.

[m-mohr]

data_type was a field I ported from openEO and got spec'ed by my predecessor. I agree, I think it doesn't really make sense. Removed it for now.

[matthewhanson]

In the EO spec we have gsd (Ground Sample Distance) at the top level, and it's also provided per band because resolution may vary by band. At the top level it represents the best resolution to enable searching. I think it makes sense at the Dataset level rather than the Item level.

[m-mohr]

@matthewhanson What do you think? Would it make sense to extend and share the EO extension across datasets and items or to have it separated? Or should there be one extension, which has sections on items and datasets? I think I'd prefer to share the same extension... some definitions probably make sense for items and datasets equally.

[matthewhanson]

I'm not sure I entirely get what you are saying.
I think datasets are a concept which will be used in general with STAC - although I'm still not sure if the intention is that datasets are always present and if they are themselves part of core. I personally think they should part of core, the include core fields: temporal and spatial extent are the unions of core fields, license, provider...

So the EO extension, or any extension, should define additions to both the Dataset and the Item.

[m-mohr]

I hope datasets are part of the core and always present, but not sure what others think.

I agree with what you are saying about extensions and that answers basically my question. So I would expect that the additional EO fields we are proposing for datasets will be incorporated into the EO extension. If you are okay with that, I'd already move them to the eo extension in the branch we are currently working on. They are currently a bit badly located in the dataset-spec.

[matthewhanson]

Yes, I think those additional fields belong in the EO extension...although I think that I'd still like to see some of the non-varying asset info in Datasets, but I'll make the case and provide some examples in the EO extension.

[m-mohr]

Not sure whether speaking about the same. I am speaking about an EO extension that - whenever meaningful - is shared between Dataset and Item and can be used in both locations! Are you just talking about an EO extension limited to items? Otherwise I don't get the point you make with "I'd still like to see some of the non-varying asset info in Datasets".

[matthewhanson]

I think we are talking about the same thing, EO extension is shared between Dataset and Item.

What I'm saying is that some of the Asset information, such as the list of possible assets and what their types are, can be added at the Dataset level. I think I explained it a bit better elsewhere. Maybe need to make a new issue for it, these PR is getting a bit hard to follow.

[m-mohr]

Great, then I'd like to have your support here (regarding global extensions): #186 ;-)

Sure, I think more issues would make sense! I think we are also discussing that in #174. I'd like a proposal on this. I don't think it's simply copy and paste (minus url) from the assets spec in the items?

[matthewhanson]

Been thinking about this more and scale and offset definitely need to be specified per asset, not per band. Individual items can have different scale and offsets. While Landsat collection 1 uses a consistent scale/offset for all scenes, this was not always the case and a provider may certainly provide them per Item, it is not really a band characteristic, it is an asset characteristic.
I had thought of it because there are other reasons too - for instance I might take the RGB bands and scale them to a Byte and provide that as a visible True Color image, or other bands to make a False Color image, but still want to reference the bands that make up that 3-band image.

[matthewhanson]

In the same vain, units really is per asset as well. I might provide a TOA datafile as well as a Surface reflectance file.

And in those cases I might want to use different nodata values as well, so they really should all be per-asset. Units and nodata can be specified per asset in the Dataset, but scale/offset needs to be specified for each asset in each Item.

[fredliporace]

@m-mohr @matthewhanson gain (scale) and offsets may have distinct values depending on the date, see the case for LS4 and LS5. How should that cases be represented? I also prefer gain instead of scale, but this may be a Landsat thing.

[simonff]

@matthewhanson : I don't quite follow. What is an asset in this case? My understanding that by assets STAC people usually mean files on disk. Do you mean a specific file for a specific item? But a gdal file can have two bands that have different gain/offset.

For reference, datasets in EE almost always have the same gain/offset for each band across all items. The two exceptions are Landsat and SeaWiFS OceanColor dataset that has different processing for different items in the same dataset. We did not dare to change Landsat, but for SeaWiFS we applied gain/offset before ingestion to make the data easier to use in ARD-like manner.

So in general, yes, each item can have its own gain/offset. I've never seen nodata values differ across items, though. If they do, it might make more sense to split the dataset in two or more.

[m-mohr]

@matthewhanson In the normal "STAC way" with Items and Assets that might be feasible, but for openEO (and GEE?), we still need those information - whenever meaningful - per dataset or band, as we don't have any items/assets where this information is available! So we probably need some of these properties in all locations.

@fredliporace

• Honestly, it's just a name, but I am more used to scale. Don't really care whether its gain or scale. It has the same meaning, correct?
• Not sure whether we can cover all of that, currently it can't represented. Maybe just leave it out if it depends on another property? I don't have a proper solution for it yet - but open for proposals.

[matthewhanson]

@fredliporace Right, that's why I'm saying scale and offset need to be provided per asset rather than at some higher level like the dataset.

I actually prefer gain as well, but scale and offset are the terms used by GDAL which make them a bit more universal I think.

[matthewhanson]

@simonff What I'm saying is that the scales and offsets need to be provided for each specific asset, and they need to be arrays because as you say the asset could have multiple bands.

I agree nodata values probably won't be different across items, it would probably be fine to specify them per asset at the Dataset level.

[m-mohr]

Well, than it's easy to solve, right? If scale/offset are per asset, define it in the asset. If it's per band, define it in the band (any maybe also in the asset, not sure).

And again, openEO can't use it per item, so I'd need both ways anyway.

[fredliporace]

@m-mohr Is this to be interpreted as 'possible extent' or as 'current extent'? My concern here is that some missions have the capacity to image most of the Earth but do not make a systematic acquisitions - CBERS for instance.
The footprint for CBERS-4 MUX scenes may be obtained here, just select CBERS on the upper right panel. You'll see that most of northern Canada is not covered yet by the dataset, but a new scene may be acquired anytime in the future. In that case should the spatial_extent be changed? If is is changed could the dataset version be the same?

[simonff]

I would prefer possible extent to keep search results (over datasets, not over items in a dataset) more consistent.

[m-mohr]

For openEO we had it defined as current extent and so I had that in mind, but I am open to both. Whoever has the best arguments wins. ;-)

[cholmes]

I think I prefer possible extent. That's a lot easier to implement for static providers. And someone who wants it to be 'current extent' can do that if they want - the current extent is certainly within the possible extent.

I don't want catalogs to feel like they have to always be updating this field.

[cholmes]

I lean towards 'possible extent'. An implementor could choose to make theirs 'current extent', since the current should be a subset of the possible. But I think it's better to not require static catalogs to keep updating their extent every single time there's new data outside their current.

[m-mohr]

"possible" vs "current" also applies for the temporal_extent as open date ranges would always be "possible". I would like to have them and for consistency and the reasons mentioned above, I now slightly prefer "possible", too. But with open date ranges we are not compatible with WFS, see also opengeospatial/ogcapi-features#155.

[m-mohr]

Added "potential" to both extents.

[fredliporace]

@m-mohr @matthewhanson gain (scale) and offsets may have distinct values depending on the date, see the case for LS4 and LS5. How should that cases be represented? I also prefer gain instead of scale, but this may be a Landsat thing.

[simonff]

I suggest a single provider here. I added support for multiple providers in the EE catalog, and never found any use for it other than expressing the processing chain, which we want to do more systematically elsewhere.

[m-mohr]

I am hesitant to only rely on the process chain extension (or however we call it). It is still a long way to have standard to properly define these processing information apart from maybe a provider and a dataset url. Still, we should have something small in the core, I think. It should be easy to users to get at least some information about the history. I think the processing chain information would be much much harder to express and then it will just be left out. We also discussed the field derived_from. Can we combine that somehow?

And for me it's also to give proper credit and having them all makes clear what to put here. A single provider again leaves it open whether it's the RAW data provider or the last one processing it.

Maybe we could also just have something like "history" in the core, which has a list of provider name + provider homepage + dataset url (derived_from). As we don't need the dataset url for the last provider (it's that catalog) the last provider would be the provider in the dataset. A process chain extension just extends the History Object and the Dataset Object, so that a process_chain can simply be added to each history element, too.

Example follows...

[m-mohr]

{
  "id":"sentinel2-processed",
  "description":"Sentinel 2 NDVI max composite processed by Final Processor and Another Processing Comp, data originates from ESA.",
  "spatial_extent":{

  },
  "temporal_extent":"2015/2018",
  "license":"Apache-2.0",
  "provider":{
    "name":"Final processor, Inc.",
    "url":"http://www.final-corporation.com"
  },
  "pc:process_chain":{
    "process":"max_time_composite"
  },
  "history":[
    {
      "provider":{
        "organization":"Another Processing Comp, Inc.",
        "url":"http://processing.inc"
      },
      "dataset_url":"http://processing.inc/datasets/sentinel2-processed/catalog.json",
      "pc:process_chain":{
        "process":"ndvi"
      }
    },
    {
      "provider":{
        "organization":"ESA",
        "url":"http://esa.eu"
      },
      "dataset_url":"http://esa.eu/data/sentinel-2"
    }
  ]
}

We could also put the last provider directly into the history, dataset_url would be to self or omitted. Then we would have no direct provider in the top-level, but that would be okay for me.

{
  "id":"sentinel2-processed",
  "description":"Sentinel 2 NDVI max composite processed by Final Processor and Another Processing Comp, data originates from ESA.",
  "spatial_extent":{

  },
  "temporal_extent":"2015/2018",
  "license":"Apache-2.0",
  "history":[
    {
      "provider":{
        "name":"Final processor, Inc.",
        "url":"http://www.final-corporation.com"
      },
      "pc:process_chain":{
        "process":"max_time_composite"
      }
    },
    {
      "provider":{
        "organization":"Another Processing Comp, Inc.",
        "url":"http://processing.inc"
      },
      "dataset_url":"http://processing.inc/datasets/sentinel2-processed/catalog.json",
      "pc:process_chain":{
        "process":"ndvi"
      }
    },
    {
      "provider":{
        "organization":"ESA",
        "url":"http://esa.eu"
      },
      "dataset_url":"http://esa.eu/data/sentinel-2"
    }
  ]
}

[simonff]

So far I only used this as en enum describing the way averaging is down during pyramiding: MEAN for normal continuous bands like EO observations or temperature/DEM/etc, MODE for categorical values (eg, landcover classes), SAMPLE for the rest (typically this means bitmask bands). SAMPLE means just take one possibly random value, as nothing else makes sense. (In reality in this case we always take UL value for each 2x2 grid).

[m-mohr]

@simonff, could you come up with a proposal for this field? And maybe also asset_schema? These initially were your ideas and I think you have a much better idea to define them.

[m-mohr]

Added a first draft for the dimension extension, is open for review.
Was partly proposed by a partner of openEO on basis of the OGC Coverage Implementation Schema.

[m-mohr]

Another issue we should have in mind: #150

[cholmes]

It could be really good to try to get compatibility with WFS on the extent fields. They use:

      "extent": {
        "spatial": [ 7.01, 50.63, 7.22, 50.78 ],
        "temporal": [ "2010-02-15T12:34:56Z", "2018-03-18T12:11:00Z" ]
      },

If we want we can try to influence them to adopt our convention, but we should have good reasoning.

[m-mohr]

• I'm perfectly fine with the additional level of hierarchy (i.e. putting them both in an object with the key extent).
• Spatial is fine, too.
• Temporal on the other hand is a problem. Open date ranges are not supported. I already opened an issue on that: Issues with temporal data, including (open) ranges / periods opengeospatial/ogcapi-features#155
• What is missing from your example is trs and crs. Do we need to support them?

[cholmes]

I just used 'example 4' from the WFS spec. It notes: 'Coordinate reference system information is not provided as the service provides geometries only in the default system (WGS84 longitude/latitude)'. So seems like we could just say for dataset we require default.

[m-mohr]

Okay, than that's fine. Curious what the WFS crew is coming up with for the other issues mentioned. Will change that in the dataset spec. Temporal extent is still a string for now and need to find out how they define 3D bboxes (= incl. the z-axis).

[m-mohr]

Should be mostly WFS compatible now.

[fredliporace]

@m-mohr possible accidental text 'cut'

[m-mohr]

Fixed, thank you for the hint.

[m-mohr]

I fleshed out the spec a bit more.
Additionally, I cleaned up this PR to just contain the dimension extension and the dataset spec for now.
I think it is time for a review to get this in so we can develop and discuss better based on this first proposal.

[ghost]

This looks pretty good to me. I think reading through this got me a little closer helped me understand how the concepts of "catalog", "collection" and "dataset" are being handled.

I'm not sure it's clear where the extensions fit in? Are they just added as additional properties in the root dataset object? Is dimensions a dataset specific extension or does it also apply to Items?

It sounds like any extension (eo in particular was discussed) can be applied at the dataset level? If so, do we need to indicate that item metadata takes precedence over dataset metadata?

[cholmes]

Looks great. Added a few typo corrections, but I don't see any blocking - would be good to get this in.

[cholmes]

Is it that STAC Items 'use' the spec? I guess 'use' is less clear to me, it seems to imply that they should maybe implement the fields? Maybe describe what 'use' means? Like 'STAC Items are recommended to provide a link to a dataset definition'?

[m-mohr]

Improved that as suggested.

[cholmes]

I -> It

Also not sure if 'is a superset' is the best way to explain it. I'd say more like 'extends the catalog spec'. Probably would add a bit of color, like 'extends the catalog spec with additional fields to describe the set of items in the catalog'.

[m-mohr]

Both fixed/added.

[cholmes]

I think you said somewhere that multiple providers are allowed? If so I'd make it a bit more explicit here. If not then we should provide guidance on which one of the 'any...that created or processed'.

[m-mohr]

The Provider object provides information about a single provider, but the Dataset object can hold multiple providers in an array. I made that more clear in the provider field.

[cholmes]

Makes sent, sounds great.

[cholmes]

typo: id -> is

[m-mohr]

Fixed.

[cholmes]

Is the S2 license 'proprietary'? If it is in our current definition maybe we should expand the definition a bit. Or ideally find the ideal spdx license. For landsat I think we just used https://spdx.org/licenses/PDDL-1.0.html as getting across the intent of US public domain stuff...

[m-mohr]

Honestly, I don't really know and don't feel in the position to decide that. That example is taken from GEE and I handled it the way GEE does.

[cholmes]

Ok, cool. Super minor issue, so let's just leave it, and following GEE sounds good.

[m-mohr]

Mostly fixed the issues mentioned by Chris. One more review required.

@hgs-truthe01 We still need to elaborate more on the extensions and "inheritance".

[cholmes]

Looks good to me, and looks like my green review checkmark is still there. Just need one more review - @matthewhanson or @hgs-truthe01 ?

[m-mohr]

I just made another compatibility change, which improves the WFS 3 compatibility as we are now using their specification for temporal extents with one exception. We were using ISO8601 date ranges before and WFS3 uses a two element array. We changed to two element arrays, but here it is allowed to set one of the elements to null to support open date ranges. Proposed that also in the WFS issue tracker, maybe we get that into WFS (but doesn't seem anybody is currently very interested in temporal extents ;-) ).

[ghost]

I guess I will save any extension/inheritance discussion for another thread. At the very least this cleans up the root catalog stuff nicely.

[ghost]

One last comment.

We have so far been standardizing on GeoJSON and RCF3339 for space and time. The extent stuff is introducing a new way to specify similar information.

It is a slightly different case, so I could be convinced that the extent definitions should be different, but it is less developer friendly to use two competing formats for representing information. One of the issues I have seen with geospatial api's is around extents and projections. I would prefer to only represent geospatial coordinates in one projection to keep clients from needing a full projection engine.

I only bring this up because Space and Time are the core parts of the spec. If we are not consistent, it feels weird to me.

[m-mohr]

@hgs-truthe01 We are using RFC3339 now. That's what I wanted to say with my latest comment. ;)
GeoJSON is not applicable for temporal extents. For spatial extents we are following WFS3, which is also WGS84, but doesn't allow GeoJSON. So I think we got it combined mostly and can't do much more.

[simonff]

I think a single provider is enough for the vast majority of the cases.

[cholmes]

Cool. Are you saying we should just mention that this will usually just be one provider? Or that we should shift from array to a single object?

[simonff]

I suggest shifting to single object to reduce compelxity. (In the internal EE catalog, I started with providers a list and found that this is not necessary in 99.9% of cases, and the remaining 0.1% is better handled by provenance.) I'm open to revisiting this, but I'd like to see more examples first.

[m-mohr]

I think this need more discussion, but I think an issue would be the better place as this PR is already very messy.

[simonff]

In addition to the 'host' field, I suggest the 'source' field of the same type Host that points at the canonical source of the data. This is needed for 99% of datasets in the EE catalog, as we are mostly a mirror.

[m-mohr]

Open to add that, but I think we can add that with another PR shortly after this PR.

[simonff]

we don't have any examples of data with extra dimensions yet, so I suggest dropping dimensions from this version (and just mentioning it as 'planned" here, like we do for the provenance extension)

[m-mohr]

We have at least a draft for this extension in contrast to provenance. And I have data for the dimension extension. I'll ask my colleague to publish that. But we can move that to a separate PR, if that's preferred by you or others.

[simonff]

Right, I'd rather reduce this PR and then calmly discuss the change separately.

[m-mohr]

Okay, will separate when I'm back in the office in 9 hours ;)

[m-mohr]

Dimensions extension has been moved to PR #227.

[simonff]

Should we distinguish between links to official license pages vs local copies? Eg, rel="license_copy"

[m-mohr]

Why should we distinguish them? Don't see a good reason yet and we don't do it for items, too.

[simonff]

Having to create a local license file means that the data provider has not organized their licenses well and may change them later, making local copies obsolete. But this is speculative, so I'm okay with not changing this now.

[m-mohr]

Okay, great. Please, make sure to open an issue on this one so we don't forget.

[m-mohr]

Moved Dimensions extension as requested. Can be merged now, I think.