Merge branch 'wv' into '0.7.0'

Merge wv into 0.7.0

See merge request extracteo/eoreader!14

CHANGES.md

@@ -2,20 +2,40 @@
 
 ## X.Y.Z (YYYY-MM-DD)
 
-## 0.7.0 (2021-MM-DD)
-
-- ENH: Implementing RADARSAT-Constellation products (as `RCM`)
-- ENH: Adding `RH`, `RV`, `RH_DSPK` and `RV_DSPK` SAR bands
-- ENH: Loading by size -> round resolution to the closest meter (or decimeter for resolution < 1.0m)
+## 0.7.0 (2021-09-DD)
+
+- **ENH: Implementing RADARSAT-Constellation products (as `RCM`)**
+- **ENH: Implementing Maxar products (such as `GE01, WV02, WV03, WV04`, but others should be supported too)**
+- **ENH: Implementing TanDEM-X products (as `TDX`)**
+- **ENH: Adding `RH`, `RV`, `RH_DSPK` and `RV_DSPK` SAR bands**
+- **ENH: Adding the `YELLOW` optical band (for `WorldView-2`, `WorldView-3` and `Sentinel-3 OLCI`)**
+- **ENH: Adding [WorldView index](https://resources.maxar.com/optical-imagery/multispectral-reference-guide) (without the ones using SWIR)**
+- **ENH: Loading by size -> round resolution to the closest meter (or decimeter for resolution < 1.0m)**
+- **ENH: Super class for VHR data**
+- FIX: Fixing reading PlanetScope archived products (error in read band)
 - FIX: Fix band name with complex resolutions
 - FIX: Fixing minor bug in RADARSAT-2 data when looking for product type
 - FIX: Fixing SAR band search in BEAM-DIMAP files
 - FIX: Fixing python version in environment.yml
+- FIX: Discard unused MIR and FNIR bands
+- FIX: Check for existence of given path when reading any product
+- FIX: Workaround for a bug involving some downloaded but badly formatted archives for Sentinel-2
+- FIX: Allow NARROW_NIR for and DIMAP data (== NIR)
+- FIX: Better management of writeable band folder
+- DOC: Fix documentation of the NDWI index
+- DOC: Update graph for optical band mapping
 - CI: Adding a test loading invalid band name
+- CI: Setting CI log level to DEBUG
+- CI: Accelerating the CI processes
+
+## 0.6.4 (2021-09-15)
+
+- FIX: Sentinel-3 band mapping (`Coastal Aerosol` <-> `03`, `BLUE` <-> `04`)
+- DOC: Adding an interactive graph for optical band mapping
 
 ## 0.6.3 (2021-09-10)
 
-- ENH: Load works with string bands (`prod.load('BLUE')`)
+- **ENH: Load works with string bands (`prod.load('BLUE')`)**
 - FIX: Fixing missing `_remove_tmp_process` for products needing extraction
 - FIX: Remove multi converting for Sentinel-3
 
@@ -30,7 +50,7 @@
 
 ## 0.6.0 (2021-09-02)
 
-- ENH: Ensuring EOReader supports Dask
+- **ENH: Ensuring EOReader supports Dask**
 - FIX: Fixing and adding BAIS2 index in alias
 - FIX: Fixing GLI index
 - FIX: Fixing a bug when writing reprojected DIMAP band
@@ -40,13 +60,13 @@
 
 ## 0.5.0 (2021-08-24)
 
-- ENH: Adding the [BAIS2](https://www.researchgate.net/publication/323964124_BAIS2_Burned_Area_Index_for_Sentinel-2)
-  index
-- ENH: Read metadata/namespaces only once and store it as a private member. Keep accessing it through the `read_mtd`
-  function (#9)
+- **ENH: Adding the [BAIS2](https://www.researchgate.net/publication/323964124_BAIS2_Burned_Area_Index_for_Sentinel-2)
+  index**
+- **ENH: Read metadata/namespaces only once and store it as a private member. Keep accessing it through the `read_mtd`
+  function (#9)**
   **WARNING**: Breaking change for Landsat: `read_mtd()` loses the argument `force_pd=True` as it always returns an
   Etree
-- ENH: Reads Sentinel-3 global attributes as metadata:
+- **ENH: Reads Sentinel-3 global attributes as metadata:**
     - `absolute_orbit_number`
     - `comment`
     - `contact`
@@ -65,8 +85,8 @@
     - `ac_subsampling_factor` (`OLCI` only)
     - `al_subsampling_factor` (`OLCI` only)
     - `track_offset` (`SLSTR` only)
-- ENH: Refining Despeckle Graph (#6) to use a more usual filter (`Refined Lee`)
-- ENH: Allowing the user to open the datatake metadata for Sentinel-2 products
+- **ENH: Refining Despeckle Graph (#6) to use a more usual filter (`Refined Lee`)**
+- **ENH: Allowing the user to open the datatake metadata for Sentinel-2 products**
 - FIX: Decoupling classic metadata reading from the name as EOReader accepts now modified product names (#9)
 - FIX: Better handling of cloud-stored DEM (raising an exception for non-ortho DIMAP data as GDAL and rasterio does not
   handle that case)
@@ -82,7 +102,7 @@
 
 ## 0.4.8 (2021-07-23)
 
-- ENH: Allowing `stack` to take single band in input instead of a list
+- **ENH: Allowing `stack` to take single band in input instead of a list**
 - FIX: Fixing a regression loading optical bands which have been previously cleaned (Landsat, Theia, possibly
   PlanetScope)
 - FIX: `load` and `stack` always returns `float32` arrays
@@ -94,10 +114,10 @@
 
 ## 0.4.7 (2021-07-23)
 
-- ENH: Adding a `default_transform` function returning data from default band (without warping it) -> *
-  mapping `calculate_default_transform` from `rasterio`*
-- ENH: Adding a `clean_tmp` function allowing the user to clean the product's temporary output by hand
-- ENH: Simplifying DEM warping code
+- **ENH: Adding a `default_transform` function returning data from default band (without warping it)**
+  -> *mapping `calculate_default_transform` from `rasterio`*
+- **ENH: Adding a `clean_tmp` function allowing the user to clean the product's temporary output by hand**
+- **ENH: Simplifying DEM warping code**
 - FIX: `DIMAP` products return always projected (in UTM) default bands (`get_default_band_path`
   uses `_get_default_utm_band`)
 - FIX: Theia Footprint returns a `GeoDataFrame` instead of a `GeoSeries`
@@ -140,44 +160,44 @@
 
 ## 0.4.3 (2021-07-05)
 
-- [DIMAP Products] Optimizing loading cloud bands
+- **ENH: Optimizing loading cloud bands for DIMAP Products**
 - `stack` accepts `**kwargs` in order to pass options to `rioxarray.to_raster()`
-- Fixing not found masks with S3+zip Sentinel-2 products
-- [CI] BUG: Fixing network directories with pathlib
-- Fixing some type hints
+- FIX: Fixing not found masks with S3+zip Sentinel-2 products
+- FIX: Fixing some type hints
+- CI: Fixing network directories with pathlib
 
 ## 0.4.2 (2021-07-01)
 
-- Feature: Enabling the use of products stored in the cloud
-  (S3, S3 compatible storage, Google, Azure...) through [`cloudpathlib`](https://cloudpathlib.drivendata.org/)
-- Enhancement: Using correct band names in long_name
+- **ENH: Enabling the use of products stored in the cloud
+  (S3, S3 compatible storage, Google, Azure...) through [`cloudpathlib`](https://cloudpathlib.drivendata.org/)**
+- **ENH: Using correct band names in long_name**
 - CI: Use pre-computed cleaned band if existing
-- Doc: Adding examples for using S3 data, especially for S3 compatible storage
+- DOC: Adding examples for using S3 data, especially for S3 compatible storage
 
 ## 0.4.1.post0 (2021-06-21)
 
-- Bug fix: cloud mask values were inverted in Sentinel-2 cloud masks
-- Bug fix: Landsat collection 2 cloud masks are now OK
+- FIX: cloud mask values were inverted in Sentinel-2 cloud masks
+- FIX: Landsat collection 2 cloud masks are now OK
 
 ## 0.4.1 (2021-06-21)
 
-- Improving stacks saved as uint16:
+- FIX: Improving stacks saved as uint16:
     - Only satellite bands and index are scaled (*10.000)
     - DEM bands are just rounded
     - Cloud bands (booleans) are saved as is
-- Fixing a rasterization bug affecting S2 and DIMAP masks, happening when the vectors have another size than the image
-- Adding a warning on bad georeferencing when using GS and GT Landsat products
-- Minor updates in documentation and code
+- FIX: Fixing a rasterization bug affecting S2 and DIMAP masks, happening when the vectors have another size than the image
+- FIX: Adding a warning on bad georeferencing when using GS and GT Landsat products
+- CI: Minor updates in documentation and code
 
 ## 0.4.0 (2021-06-10)
 
 ### Features
 
-- Adding **THR** data support:
+- **ENH: Adding THR data support:**
     - **PlanetScope**
     - **Pleiades**
     - **SPOT 6-7**
-- [SAR] Better handling of SNAP DEMs (using External DEM and other available SNAP DEMs)
+- **ENH: Better handling of SNAP DEMs (using External DEM and other available SNAP DEMs)**
 
 ### Fix
 

CI/SCRIPTS/test_index.py

@@ -38,7 +38,8 @@ def test_index():
         logging.getLogger("botocore").setLevel(
             logging.WARNING
         )  # BOTO has way too much verbosity
-        idx = prod.load(get_all_index(), resolution=RES)
+        idx_list = [idx for idx in get_all_index() if prod.has_band(idx)]
+        idx = prod.load(idx_list, resolution=RES)
 
         for idx_fct, idx_arr in idx.items():
             idx_name = idx_fct.__name__

CI/SCRIPTS/test_satellites.py

@@ -8,7 +8,7 @@
 from cloudpathlib import AnyPath
 from geopandas import gpd
 from lxml import etree
-from sertit import ci, files, logs, rasters
+from sertit import ci, files, rasters
 
 from eoreader.bands.alias import *
 from eoreader.env_vars import (
@@ -20,7 +20,7 @@
     TEST_USING_S3_DB,
 )
 from eoreader.products.product import Product, SensorType
-from eoreader.reader import CheckMethod, Platform
+from eoreader.reader import CheckMethod
 from eoreader.utils import EOREADER_NAME
 
 from .scripts_utils import (
@@ -131,7 +131,6 @@ def _test_core(
     with xr.set_options(warn_for_unclosed_files=debug):
 
         # Init logger
-        logs.init_logger(LOGGER)
         logging.getLogger("boto3").setLevel(
             logging.WARNING
         )  # BOTO has way too much verbosity
@@ -173,16 +172,19 @@ def _test_core(
                     prod.output = tmp_dir
 
                     # Env var
-                    if (
-                        prod.platform == Platform.S3
-                        or prod.sensor_type == SensorType.SAR
-                    ):
-                        os.environ[CI_EOREADER_BAND_FOLDER] = str(
-                            get_ci_data_dir().joinpath(prod.condensed_name)
-                        )
-                    else:
-                        if CI_EOREADER_BAND_FOLDER in os.environ:
-                            os.environ.pop(CI_EOREADER_BAND_FOLDER)
+                    # if (
+                    #     prod.platform == Platform.S3
+                    #     or prod.sensor_type == SensorType.SAR
+                    # ):
+                    #     os.environ[CI_EOREADER_BAND_FOLDER] = str(
+                    #         get_ci_data_dir().joinpath(prod.condensed_name)
+                    #     )
+                    # else:
+                    #     if CI_EOREADER_BAND_FOLDER in os.environ:
+                    #         os.environ.pop(CI_EOREADER_BAND_FOLDER)
+                    os.environ[CI_EOREADER_BAND_FOLDER] = str(
+                        get_ci_data_dir().joinpath(prod.condensed_name)
+                    )
 
                     # Manage S3 resolution to speed up processes
                     if prod.sensor_type == SensorType.SAR:
@@ -304,8 +306,8 @@ def _test_core(
                 # Clean temp
                 if not debug:
                     LOGGER.info("Cleaning tmp")
-                    # prod.clean_tmp()
-                    # assert len(list(prod._tmp_process.glob("*"))) == 0
+                    prod.clean_tmp()
+                    assert len(list(prod._tmp_process.glob("*"))) == 0
 
 
 @s3_env
@@ -434,6 +436,24 @@ def test_spot7():
     _test_core_optical("*IMG_SPOT7*", dem_path=dem_path)
 
 
+@s3_env
+@dask_env
+def test_wv02_wv03():
+    """Function testing the correct functioning of the optical satellites"""
+    # This test orthorectifies DIMAP data, so we need a DEM stored on disk
+    dem_path = os.path.join(
+        ci.get_db2_path(), "BASES_DE_DONNEES", *MERIT_DEM_SUB_DIR_PATH
+    )
+    _test_core_optical("*P001_MUL*", dem_path=dem_path)
+
+
+@s3_env
+@dask_env
+def test_ge01_wv04():
+    """Function testing the correct functioning of the optical satellites"""
+    _test_core_optical("*P001_PSH*")
+
+
 @s3_env
 @dask_env
 def test_s1():

README.md

@@ -8,13 +8,13 @@
 
 # ![eoreader_logo](https://eoreader.readthedocs.io/en/latest/_static/favicon.png) EOReader
 
-**EOReader** is a **multi-satellite reader** allowing you to open
-[optical](https://eoreader.readthedocs.io/en/latest/optical.html)
-and [SAR](https://eoreader.readthedocs.io/en/latest/sar.html) data.
+**EOReader** is a remote-sensing opensource python library reading [optical](https://eoreader.readthedocs.io/en/latest/optical.html)
+and [SAR](https://eoreader.readthedocs.io/en/latest/sar.html) sensors, loading and stacking bands,
+clouds, DEM and index in a sensor-agnostic way.
 
 |**Optical sensors** | **SAR sensors**|
 | --- | ---|
-|Sentinel-2 and Sentinel-2 Theia<br>Sentinel-3 OLCI and Sentinel-3 SLSTR<br>Landsat 1 to 8 (MSS, TM, ETM and OLCI)<br>PlanetScope<br>Pleiades<br>SPOT 6-7| Sentinel-1<br>COSMO-Skymed<br>TerraSAR-X<br>RADARSAT-2<br>RADARSAT-Constellation|
+|Sentinel-2 and Sentinel-2 Theia<br>Sentinel-3 OLCI and Sentinel-3 SLSTR<br>Landsat 1 to 8 (MSS, TM, ETM and OLCI)<br>PlanetScope<br>Pleiades<br>SPOT 6-7<br>WorldView-2 to 4, GeoEye-1 and other Maxar satellites| Sentinel-1<br>COSMO-Skymed<br>TerraSAR-X & TanDEM-X<br>RADARSAT-2<br>RADARSAT-Constellation|
 
 It also implements additional **sensor-agnostic** features:
 
@@ -45,10 +45,6 @@ For optical data:
 >>> eoreader = Reader()
 >>> l5_prod = eoreader.open(l5_path)  # The Reader will recognize the satellite type from its structure
 
->>> # Get the footprint of the product (usable data) and its extent (envelope of the tile)
->>> footprint = l5_prod.footprint()
->>> extent = l5_prod.extent()
-
 >>> # Specify a DEM to load DEM bands
 >>> import os
 >>> from eoreader.env_vars import DEM_PATH
@@ -77,10 +73,6 @@ For SAR data:
 >>> eoreader = Reader()
 >>> s1_prod = eoreader.open(s1_path)  # The Reader will recognize the satellite type from its name
 
->>> # Get the footprint of the product (usable data) and its extent (envelope of the tile)
->>> footprint = s1_prod.footprint()
->>> extent = s1_prod.extent()
-
 >>>  # Specify a DEM to load DEM bands
 >>> import os
 >>> from eoreader.env_vars import DEM_PATH