upstream updates (#1)

* Fix sertit#140: replace QB satellite ID with QB02 satellite ID * Fix sertit#142: catch StopIteration and return InvalidProductError * Fix sertit#142: catch StopIteration in several products * Fix sertit#142: raise InvalidProductError instead of log * Fix sertit#145: remove nested=-1 for SPOT constellations, use the default nested=0 instead * Fix sertit#145: update CHANGES.md * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix sertit#142: catch StopIteration and return InvalidProductError * Avoid KeyError if a product metadata doesn't have a namespace * sertit#142: catch StopIteration and return InvalidProductError * sertit#156: catch InvalidProductError when we try to get cloud_coverage property * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * sertit#156: catch all InvalidProductError due to properties not found in metadata * sertit#161: return an error if SPECTRAL_PROCESSING not found in metadata * [pre-commit.ci] pre-commit autoupdate updates: - [github.com/PyCQA/flake8: 7.0.0 → 7.1.0](PyCQA/flake8@7.0.0...7.1.0) * FIX: Handle RCM and RS2 products that doesn't bundle their extent in a KML file sertit#155 * OPTIM: Search correctly nested metadata in the Reader (without accidentally using a recursive glob) * sertit#152: add SGF in RS2 product type * FIX: Fix error in looking for bands in Sentinel-2 L1C archived products sertit#168 * FIX: Handle wrongly recognized Planet products because of the recursive nested mtd in the Reader sertit#169 * FIX: Force the loading of `DimapV1` bands in `float32` * FIX: Handle the case where fiona isn't installed anymore (with `geopandas 1.0`) * FIX: Don't make `pystac` a mandatory requirement * Use sertit.types.is_iterable * Use sertit.types.make_iterable * Correct typo * FIX: Fix an unknown Planet bug that just appeared (`'...Path' has no len()`) * CI: Remove for now end-to-end tests with Python 3.11 and 3.10. * v0.21.2 * INSTALL: Remove `pystac[validation]` (as it is an optional dependency) from setup.py, and create a `stac` extra feature. * Updates in CHANGES.md * v0.21.2 * [pre-commit.ci] pre-commit autoupdate updates: - [github.com/PyCQA/flake8: 7.1.0 → 7.1.1](PyCQA/flake8@7.1.0...7.1.1) - [github.com/psf/black: 24.4.2 → 24.8.0](psf/black@24.4.2...24.8.0) * FIX: Get better window name (if available) when writing bands on disk (in tmp folder) + update sertit's version to benefit to the vector's name --------- Co-authored-by: jteulade <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Rémi Braun <[email protected]> Co-authored-by: BRAUN REMI <[email protected]>
pixalytics-ltd · Aug 12, 2024 · f63d6d6 · f63d6d6
1 parent bacd004
commit f63d6d6
Show file tree

Hide file tree

Showing 46 changed files with 433 additions and 302 deletions.
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
@@ -117,61 +117,61 @@ pytest_end_to_end:
     - if: $CI_COMMIT_TAG
       when: always
 
-pytest_end_to_end310:
-  image: $EO_CONTAINER:geo_310
-  stage: test
-  variables:
-    # Use Dask for python 3.10
-    EOREADER_USE_DASK: "1"
-  before_script:
-    - python -m pip install --upgrade pip
-    - pip install --ignore-installed PyYAML
-    - pip install pytest coverage pytest-cov pylint
-    - pip install -e .
-  script:
-    - python -m pytest -v --durations=0 --cov-report term --cov-report html:${CI_PROJECT_DIR}/cov_e2e.html --cov=eoreader  --cov-config=.coveragerc CI/SCRIPTS --log-cli-level DEBUG --capture=tee-sys
-  artifacts:
-    paths:
-      - ${CI_PROJECT_DIR}/cov_e2e.html
-  coverage: '/TOTAL\s+\d+\s+\d+\s+(\d+%)/'
-  tags:
-    - sertit
-    - linux
-    - high_memory
-  rules:
-    - if: '$CI_PIPELINE_SOURCE == "schedule" && $SCHEDULE_NAME == "END_TO_END"'
-      when: always
-    - if: $CI_COMMIT_TAG
-      when: always
-  needs: [ "pytest_end_to_end" ]
-
-pytest_end_to_end311:
-  image: $EO_CONTAINER:geo_311
-  stage: test
-  variables:
-    # Use Dask for python 3.11
-    EOREADER_USE_DASK: "1"
-  before_script:
-    - python -m pip install --upgrade pip
-    - pip install --ignore-installed PyYAML
-    - pip install pytest coverage pytest-cov pylint
-    - pip install -e .
-  script:
-    - python -m pytest -v --durations=0 --cov-report term --cov-report html:${CI_PROJECT_DIR}/cov.html --cov=eoreader  --cov-config=.coveragerc CI/SCRIPTS --log-cli-level DEBUG --capture=tee-sys
-  artifacts:
-    paths:
-      - ${CI_PROJECT_DIR}/cov_e2e.html
-  coverage: '/TOTAL\s+\d+\s+\d+\s+(\d+%)/'
-  tags:
-    - sertit
-    - linux
-    - high_memory
-  rules:
-    - if: '$CI_PIPELINE_SOURCE == "schedule" && $SCHEDULE_NAME == "END_TO_END"'
-      when: always
-    - if: $CI_COMMIT_TAG
-      when: always
-  needs: [ "pytest_end_to_end310" ]
+#pytest_end_to_end310:
+#  image: $EO_CONTAINER:geo_310
+#  stage: test
+#  variables:
+#    # Use Dask for python 3.10
+#    EOREADER_USE_DASK: "1"
+#  before_script:
+#    - python -m pip install --upgrade pip
+#    - pip install --ignore-installed PyYAML
+#    - pip install pytest coverage pytest-cov pylint
+#    - pip install -e .
+#  script:
+#    - python -m pytest -v --durations=0 --cov-report term --cov-report html:${CI_PROJECT_DIR}/cov_e2e.html --cov=eoreader  --cov-config=.coveragerc CI/SCRIPTS --log-cli-level DEBUG --capture=tee-sys
+#  artifacts:
+#    paths:
+#      - ${CI_PROJECT_DIR}/cov_e2e.html
+#  coverage: '/TOTAL\s+\d+\s+\d+\s+(\d+%)/'
+#  tags:
+#    - sertit
+#    - linux
+#    - high_memory
+#  rules:
+#    - if: '$CI_PIPELINE_SOURCE == "schedule" && $SCHEDULE_NAME == "END_TO_END"'
+#      when: always
+#    - if: $CI_COMMIT_TAG
+#      when: always
+#  needs: [ "pytest_end_to_end" ]
+#
+#pytest_end_to_end311:
+#  image: $EO_CONTAINER:geo_311
+#  stage: test
+#  variables:
+#    # Use Dask for python 3.11
+#    EOREADER_USE_DASK: "1"
+#  before_script:
+#    - python -m pip install --upgrade pip
+#    - pip install --ignore-installed PyYAML
+#    - pip install pytest coverage pytest-cov pylint
+#    - pip install -e .
+#  script:
+#    - python -m pytest -v --durations=0 --cov-report term --cov-report html:${CI_PROJECT_DIR}/cov.html --cov=eoreader  --cov-config=.coveragerc CI/SCRIPTS --log-cli-level DEBUG --capture=tee-sys
+#  artifacts:
+#    paths:
+#      - ${CI_PROJECT_DIR}/cov_e2e.html
+#  coverage: '/TOTAL\s+\d+\s+\d+\s+(\d+%)/'
+#  tags:
+#    - sertit
+#    - linux
+#    - high_memory
+#  rules:
+#    - if: '$CI_PIPELINE_SOURCE == "schedule" && $SCHEDULE_NAME == "END_TO_END"'
+#      when: always
+#    - if: $CI_COMMIT_TAG
+#      when: always
+#  needs: [ "pytest_end_to_end310" ]
 
 #pytest_end_to_end312:
 #  image: $EO_CONTAINER:geo_312

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -20,12 +20,12 @@ repos:
       - id: check-merge-conflict
 
   - repo: 'https://github.com/PyCQA/flake8'
-    rev: 7.0.0
+    rev: 7.1.1
     hooks:
       - id: flake8
 
   - repo: 'https://github.com/psf/black'
-    rev: 24.4.2
+    rev: 24.8.0
     hooks:
       - id: black
 

diff --git a/CHANGES.md b/CHANGES.md
@@ -1,12 +1,28 @@
 # Release History
 
-## 0.21.2 (2024-mm-dd)
+## 0.21.3 (2024-mm-dd)
+
+- FIX: Get better window name (if available) when writing bands on disk (in tmp folder) 
+
+## 0.21.2 (2024-07-30)
 
 - ENH: `to_str` and `to_band`: add a `as_list` argument defaulting to `True`. When set as False, return a str from `to_str` and a band from `to_band` ([#138](https://github.com/sertit/eoreader/issues/138)). Thanks @jsetty!
+- FIX: `Sentinel-2` product with `StopIteration` error ([#142](https://github.com/sertit/eoreader/issues/142))
+- FIX: Fix error in looking for bands in `Sentinel-2 L1C` archived products ([#168](https://github.com/sertit/eoreader/issues/168))
 - FIX: Fix issue with geocoding with unzipped `Sentinel-3 OLCI` product ([#137](https://github.com/sertit/eoreader/issues/137))
-- FIX: Fix iceye product when extent file (*.kml) not found ([#135](https://github.com/sertit/eoreader/pull/135))
-- FIX: Add missing `pystac[validation]` in setup.py
-- CI: Fix S3 endpoint management with `sertit==1.37.0`
+- FIX: In `SPOT` products, METADATA.DIM and IMAGERY.TIF must be at the root of the product ([#145](https://github.com/sertit/eoreader/issues/145))
+- FIX: Fix `Maxar` product with `QB02` satellite ID ([#140](https://github.com/sertit/eoreader/issues/140))
+- FIX: Fix `ICEYE` product when extent file (*.kml) not found ([#135](https://github.com/sertit/eoreader/pull/135))
+- FIX: Handle `RCM` and `RS2` products that doesn't bundle their extent in a KML file ([#155](https://github.com/sertit/eoreader/issues/155))
+- FIX: Handle wrongly recognized `Planet` products because of the recursive nested mtd in the Reader ([#169](https://github.com/sertit/eoreader/issues/169))
+- FIX: Fix an unknown `Planet` bug that just appeared (`'...Path' has no len()`)
+- FIX: Force the loading of `DimapV1` bands in `float32`
+- FIX: Handle the case where `fiona` isn't installed anymore (with `geopandas 1.0`)
+- FIX: Don't make `pystac` a mandatory requirement
+- OPTIM: Search correctly nested metadata in the Reader (without accidentally using a recursive glob)
+- CI: Fix S3 endpoint management with `sertit>=1.37`
+- CI: Remove for now end-to-end tests with Python 3.11 and 3.10. 
+- INSTALL: Remove `pystac[validation]` (as it is an optional dependency) from setup.py, and create a `stac` extra feature.
 
 ## 0.21.1 (2024-04-03)
 

diff --git a/CI/SCRIPTS_WEEKLY/test_all_sat_end_to_end_on_disk.py b/CI/SCRIPTS_WEEKLY/test_all_sat_end_to_end_on_disk.py
@@ -7,7 +7,7 @@
 
 import xarray as xr
 from lxml import etree
-from sertit import AnyPath, ci, path
+from sertit import AnyPath, ci, path, types
 
 from CI.scripts_utils import (
     CI_EOREADER_S3,
@@ -166,8 +166,7 @@ def _test_core(
 
     with xr.set_options(warn_for_unclosed_files=debug):
         # DATA paths
-        if not isinstance(prod_dirs, list):
-            prod_dirs = [prod_dirs]
+        prod_dirs = types.make_iterable(prod_dirs)
 
         pattern_paths = []
         for prod_dir in prod_dirs:

diff --git a/docs/sar.md b/docs/sar.md
@@ -43,8 +43,8 @@ mostly because SNAP doesn't handle them.
 | `RADARSAT Constellation Mission`    | GRC, GCC, GCD                                                                                                                                                 | ⚠       |
 | `RADARSAT Constellation Mission`    | GRD                                                                                                                                                           | ✅       | 
 | `RADARSAT-2`                        | SLC                                                                                                                                                           | ✅       | 
-| `RADARSAT-2`                        | SGX, SCN, SCW,<br>SCF, SCS, SSG, SPG                                                                                                                          | ⚠       |
-| `RADARSAT-2`                        | SGF                                                                                                                                                           | ✅       |
+| `RADARSAT-2`                        | SGF, SGX, SSG                                                                                                                                                 | ✅       |
+| `RADARSAT-2`                        | SCN, SCF, SCW, SCS, SPG                                                                                                                                       | ⚠       |
 | `Sentinel-1`                        | SLC                                                                                                                                                           | ✅       | 
 | `Sentinel-1`                        | GRD                                                                                                                                                           | ✅       |
 | `Sentinel-1`                        | RTC ([ASF](https://hyp3-docs.asf.alaska.edu/guides/rtc_product_guide/#readme-file) and [MPC](https://planetarycomputer.microsoft.com/dataset/sentinel-1-rtc)) | ✅       |

diff --git a/environment-doc.yml b/environment-doc.yml
@@ -5,5 +5,4 @@ dependencies:         # everything under this, installed by conda
   - python=3.9
   - pip
   - pip:                # everything under this, installed by pip
-      - --extra-index-url ${WINDOWS_WHEELS}  # Use windows wheels on Windows
       - -r ./requirements-doc.txt
diff --git a/eoreader/__meta__.py b/eoreader/__meta__.py
@@ -17,7 +17,7 @@
 """
 **EOReader** library
 """
-__version__ = "0.21.1"
+__version__ = "0.21.2"
 __title__ = "eoreader"
 __description__ = (
     "Remote-sensing opensource python library reading optical and SAR constellations, "

diff --git a/eoreader/bands/__init__.py b/eoreader/bands/__init__.py
@@ -416,6 +416,7 @@ def to_band(
         list: converted values
 
     """
+    from sertit import types
 
     def convert_to_band(tc) -> BandNames:
         band_or_idx = None
@@ -460,14 +461,14 @@ def convert_to_band(tc) -> BandNames:
 
     if as_list:
         band_list = []
-        if not isinstance(to_convert, list):
-            to_convert = [to_convert]
+        to_convert = types.make_iterable(to_convert)
+
         for tc in to_convert:
             tc_band = convert_to_band(tc=tc)
             band_list.append(tc_band)
         return band_list
     else:
-        if isinstance(to_convert, list):
+        if types.is_iterable(to_convert):
             raise _ite(f"Set as_list=True(default) for list arguments")
         return convert_to_band(to_convert)
 
@@ -491,9 +492,10 @@ def to_str(
     Returns:
         list: str bands
     """
+    from sertit import types
+
     if as_list:
-        if not isinstance(to_convert, list):
-            to_convert = [to_convert]
+        to_convert = types.make_iterable(to_convert)
 
         bands_str = []
         for tc in to_convert:
@@ -508,7 +510,7 @@ def to_str(
             bands_str.append(band_str)
         return bands_str
     else:
-        if isinstance(to_convert, list):
+        if types.is_iterable(to_convert):
             raise _ite(f"Set as_list=True(default) for list arguments")
         try:
             band_str = tc.name

diff --git a/eoreader/bands/band_names.py b/eoreader/bands/band_names.py
@@ -1,12 +1,12 @@
 from typing import Union
 
-from sertit.misc import ListEnum
+from sertit import misc, types
 
 from eoreader.exceptions import InvalidTypeError
 from eoreader.stac import StacCommonNames
 
 
-class BandNames(ListEnum):
+class BandNames(misc.ListEnum):
     """Super class for band names, **do not use it**."""
 
     @classmethod
@@ -25,8 +25,8 @@ def from_list(cls, name_list: Union[list, str]) -> list:
         Returns:
             list: List of enums
         """
-        if not isinstance(name_list, list):
-            name_list = [name_list]
+        name_list = types.make_iterable(name_list)
+
         try:
             band_names = [cls(name) for name in name_list]
         except ValueError as ex:

diff --git a/eoreader/products/custom_product.py b/eoreader/products/custom_product.py
@@ -28,7 +28,7 @@
 from lxml.builder import E
 from rasterio import crs
 from rasterio.enums import Resampling
-from sertit import logs, misc, path, rasters
+from sertit import logs, misc, path, rasters, types
 from sertit.misc import ListEnum
 from sertit.types import AnyPathStrType, AnyPathType
 
@@ -423,8 +423,7 @@ def _load_bands(
             return {}
 
         # Get band paths
-        if not isinstance(bands, list):
-            bands = [bands]
+        bands = types.make_iterable(bands)
 
         if pixel_size is None and size is not None:
             pixel_size = self._pixel_size_from_img_size(size)

diff --git a/eoreader/products/optical/gs2_product.py b/eoreader/products/optical/gs2_product.py
@@ -405,14 +405,13 @@ def get_curr_band_mtd(root_mtd):
 
             # Compute band in radiance
             band_arr = bias + band_arr * gain
+            band_arr = self._toa_rad_to_toa_refl(band_arr, band, e_sun, sun_earth_dist)
 
-            # To float32
-            if band_arr.dtype != np.float32:
-                band_arr = band_arr.astype(np.float32)
+        # To float32
+        if band_arr.dtype != np.float32:
+            band_arr = band_arr.astype(np.float32)
 
-            return self._toa_rad_to_toa_refl(band_arr, band, e_sun, sun_earth_dist)
-        else:
-            return band_arr
+        return band_arr
 
     @cache
     def _read_mtd(self) -> (etree._Element, dict):

diff --git a/eoreader/products/optical/hls_product.py b/eoreader/products/optical/hls_product.py
@@ -31,7 +31,7 @@
 import xarray as xr
 from lxml import etree
 from rasterio.enums import Resampling
-from sertit import path, rasters, rasters_rio, xml
+from sertit import path, rasters, rasters_rio, types, xml
 from sertit.misc import ListEnum
 from sertit.types import AnyPathType
 
@@ -873,8 +873,7 @@ def _load_bands(
             return {}
 
         # Get band paths
-        if not isinstance(bands, list):
-            bands = [bands]
+        bands = types.make_iterable(bands)
 
         if pixel_size is None and size is not None:
             pixel_size = self._pixel_size_from_img_size(size)