Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update SETSM product filename metadata parsing #332

Closed
ehusby opened this issue Nov 9, 2022 · 1 comment · Fixed by #334
Closed

Update SETSM product filename metadata parsing #332

ehusby opened this issue Nov 9, 2022 · 1 comment · Fixed by #334
Labels
bug Something isn't working

Comments

@ehusby
Copy link

ehusby commented Nov 9, 2022

In PGC's most recent October release of ArcticDEM and REMA datasets (in which all of the DEM Strips were reprocessed with a new "s2s041" version) we changed the filename convention of our DEM Strip data to the following:

Key: ALGORTHM_VERSION_SENSOR_DATE_IMAGE1_IMAGE2_SEGMENT_RESOLUTION_FILETYPE
Example: SETSM_s2s041_WV02_20150615_10300100443C2D00_1030010043373000_seg1_2m_dem

(see our new DEM Guide, "Metadata -> Strip DEM Products")

I believe the geoutils/satimg.parse_metadata_from_fn function needs to be adjusted here so it parses the DATE part of the filename appropriately:

attrs = ("WorldView", spl[1], "ArcticDEM/REMA", spl[7], None, dt.datetime.strptime(spl[2], "%Y%m%d"))

Whether you include backwards compatibility for old DEM Strip data filenames is something else to consider.

We're looking to start using xdem more internally, and recommending it to our users for DEM coregistration, merging, and differencing applications. Personally, I've always wanted to have python libraries like those in geoutils and xdem for our DEM work! PGC developers would be happy to collaborate more on these two great code projects wherever you need help.

@ehusby ehusby added the bug Something isn't working label Nov 9, 2022
@rhugonnet
Copy link
Member

Hi @ehusby,

Ah perfect! I didn't know about the syntax change, thanks a lot for pointing that out 😄. I'll make the fix sometime next week (was going to start using your new datasets very soon too, so the timing is pretty amazing!).

Great to hear about your interest in xDEM and GeoUtils at PGC! It's been a lot of effort building these two packages during the past years. The last 6 months we focused a lot on improving CI tests to ensure our features are consistent and robust. There's still a bit of work to reach a first stable 0.1 version for both modules, but I'd say we're almost there. Maybe sometime early-to-mid 2023?

We are currently working on streamlining the coregistration module (GlacioHack/xdem#329), and we expect to finalize a new documentation in the next months (GlacioHack/xdem#303) that we'll also write for GeoUtils. Only at this point, and after a couple other bug fixes and feature improvements, would we be comfortable releasing a 0.1 version for both packages and recommend them to broader range of users.

In any case, we'd be delighted to collaborate with you at PGC if you decide to start working with xDEM and GeoUtils! There's a lot of points that we've had in mind where your combining your expertise to these efforts would make great sense, such as optimizing routines further for very high resolution data (e.g., random subsampling within chunks and lazy processing with Dask). It is becoming increasingly common to have such big data nowadays, and would benefit a broad range of users! Additionally, we (main developpers) all have a current situation for which we know we'll remain active, and so we could easily help you get your hands on the packages if issues arise 👍.

In short, really psyched to join efforts! And, in terms of timing and stability, it likely makes sense to wait just a bit until we manage to push that first stable version out with completed documentation 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants