Checking pixellimit for r.import commands (#491)

* started implementation for querying pixellimit for r.import commands * continued implementation of pixel limit check for rimport commands * considered resampling/reprojection for raster size + cleanup of created vrt * linting * markdown formating * add tests for pixellimit check * improved check in test + added fix for determining the estimated resolution of raster * Update tests/test_raster_import_pixellimit.py Co-authored-by: Markus Neteler <[email protected]> * Update tests/test_raster_import_pixellimit.py Co-authored-by: Markus Neteler <[email protected]> * Update tests/test_raster_import_pixellimit.py Co-authored-by: Carmen Tawalika <[email protected]> * Update tests/test_raster_import_pixellimit.py Co-authored-by: Carmen Tawalika <[email protected]> * Update src/actinia_core/processing/actinia_processing/ephemeral_processing.py Co-authored-by: Carmen Tawalika <[email protected]> * add test for importer --------- Co-authored-by: Markus Neteler <[email protected]> Co-authored-by: Carmen Tawalika <[email protected]>
actinia-org · Nov 21, 2023 · a01244a · a01244a
1 parent 6fa50df
commit a01244a
Show file tree

Hide file tree

Showing 6 changed files with 382 additions and 23 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -23,32 +23,35 @@ https://github.com/actinia-org/actinia-core/compare/4.3.1...main
 released from main
 ...
 
-## [4.11.0] - 2023-11-02
+## [4.11.0](https://github.com/actinia-org/actinia-core/releases/tag/4.11.0) - 2023-11-02
+
 released from main
+
 ### Added
-* enhance docker installation docs by @mmacata in https://github.com/actinia-org/actinia-core/pull/470
-* mdformat by @mmacata in https://github.com/actinia-org/actinia-core/pull/472
-* Add pre-commit to renovate config by @mmacata in https://github.com/actinia-org/actinia-core/pull/473
-* docs: add more details for user management by @neteler in https://github.com/actinia-org/actinia-core/pull/490
-* Enable import via vsicurl by @mmacata in https://github.com/actinia-org/actinia-core/pull/482
-* Allow separate config for worker Part 1 by @mmacata in https://github.com/actinia-org/actinia-core/pull/376
+
+- enhance docker installation docs by @mmacata in https://github.com/actinia-org/actinia-core/pull/470
+- mdformat by @mmacata in https://github.com/actinia-org/actinia-core/pull/472
+- Add pre-commit to renovate config by @mmacata in https://github.com/actinia-org/actinia-core/pull/473
+- docs: add more details for user management by @neteler in https://github.com/actinia-org/actinia-core/pull/490
+- Enable import via vsicurl by @mmacata in https://github.com/actinia-org/actinia-core/pull/482
+- Allow separate config for worker Part 1 by @mmacata in https://github.com/actinia-org/actinia-core/pull/376
 
 ### Fixed
-* mundialis->actinia-org by @mmacata in https://github.com/actinia-org/actinia-core/pull/471
-* Update actions/checkout action to v4 by @renovate in https://github.com/actinia-org/actinia-core/pull/474
-* Update docker/login-action action to v3 by @renovate in https://github.com/actinia-org/actinia-core/pull/477
-* Update docker/build-push-action action to v5 by @renovate in https://github.com/actinia-org/actinia-core/pull/476
-* Update docker/setup-buildx-action action to v3 by @renovate in https://github.com/actinia-org/actinia-core/pull/479
-* Update docker/setup-qemu-action action to v3 by @renovate in https://github.com/actinia-org/actinia-core/pull/480
-* Update docker/metadata-action action to v5 by @renovate in https://github.com/actinia-org/actinia-core/pull/478
-* Update update-version.yml by @mmacata in https://github.com/actinia-org/actinia-core/pull/475
-* Update pyproj in requirements for ubuntu by @mmacata in https://github.com/actinia-org/actinia-core/pull/484
-* chore(deps): update dependency werkzeug and Flask to v3 [security] by @renovate in https://github.com/actinia-org/actinia-core/pull/489
 
-**Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.10.0...4.11.0
+- mundialis->actinia-org by @mmacata in https://github.com/actinia-org/actinia-core/pull/471
+- Update actions/checkout action to v4 by @renovate in https://github.com/actinia-org/actinia-core/pull/474
+- Update docker/login-action action to v3 by @renovate in https://github.com/actinia-org/actinia-core/pull/477
+- Update docker/build-push-action action to v5 by @renovate in https://github.com/actinia-org/actinia-core/pull/476
+- Update docker/setup-buildx-action action to v3 by @renovate in https://github.com/actinia-org/actinia-core/pull/479
+- Update docker/setup-qemu-action action to v3 by @renovate in https://github.com/actinia-org/actinia-core/pull/480
+- Update docker/metadata-action action to v5 by @renovate in https://github.com/actinia-org/actinia-core/pull/478
+- Update update-version.yml by @mmacata in https://github.com/actinia-org/actinia-core/pull/475
+- Update pyproj in requirements for ubuntu by @mmacata in https://github.com/actinia-org/actinia-core/pull/484
+- chore(deps): update dependency werkzeug and Flask to v3 \[security\] by @renovate in https://github.com/actinia-org/actinia-core/pull/489
 
-generated with `gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name=4.11.0 -f target_commitish=main -q .body` 
+**Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.10.0...4.11.0
 
+generated with `gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name=4.11.0 -f target_commitish=main -q .body`
 
 ## [4.10.0](https://github.com/actinia-org/actinia-core/releases/tag/4.10.0) - 2023-08-31
 

diff --git a/docs/docs/index.md b/docs/docs/index.md
@@ -1,6 +1,7 @@
 # Actinia - The GRASS GIS REST API
 
 <!-- **** Begin Fork-Me-On-Gitlab-Ribbon-HTML. See MIT License at https://gitlab.com/seanwasere/fork-me-on-gitlab **** -->
+
 <a href="https://github.com/actinia-org/actinia-core/tree/main/docs/docs">
     <span id="fork-me" style="font-family: tahoma; font-size: 18px; position:fixed; top:50px; right:-45px; display:block; -webkit-transform: rotate(45deg); -moz-transform: rotate(45deg); color:white; padding: 4px 30px 4px 30px; z-index:99; opacity:0.6">Fork Me On GitHub</span>
 </a>

diff --git a/src/actinia_core/processing/actinia_processing/ephemeral_processing.py b/src/actinia_core/processing/actinia_processing/ephemeral_processing.py
@@ -192,6 +192,7 @@ def __init__(self, rdc):
         self.temp_mapset_path = None
 
         self.ginit = None
+        self.ginit_tmpfiles = list()
 
         # Successfully finished message
         self.finish_message = "Processing successfully finished"
@@ -1203,6 +1204,154 @@ def _cleanup(self):
             and os.path.isdir(self.temp_grass_data_base)
         ):
             shutil.rmtree(self.temp_grass_data_base, ignore_errors=True)
+        if self.ginit_tmpfiles:
+            for tmpfile in self.ginit_tmpfiles:
+                try:
+                    os.remove(tmpfile)
+                except Exception as e:
+                    self.message_logger.debug(
+                        f"Temporary file {tmpfile} can't be removed: {e}"
+                    )
+
+    def _check_pixellimit_rimport(self, process_executable_params):
+        """Check the current r.import command against the user cell limit.
+
+        Raises:
+            This method will raise an AsyncProcessError exception
+
+        """
+        rimport_inp = [x for x in process_executable_params if "input=" in x][
+            0
+        ].split("=")[1]
+        rimport_out = [x for x in process_executable_params if "output=" in x][
+            0
+        ].split("=")[1]
+        vrt_out = f"{rimport_out}_{os.getpid()}_tmp.vrt"
+        self.ginit_tmpfiles.append(vrt_out)
+
+        # define extent_region if set (otherwise empty list)
+        extent_region = [
+            x for x in process_executable_params if "extent=" in x
+        ]
+
+        # build VRT of rimport input
+        gdabuildvrt_params = list()
+        # if extent=region set, vrt only for region, not complete input
+        if extent_region:
+            # first query region extents
+            errorid, stdout_gregion, stderr_gregion = self.ginit.run_module(
+                "g.region", ["-ug"]
+            )
+            if errorid != 0:
+                raise AsyncProcessError(
+                    "Unable to check the computational region size"
+                )
+            # parse region extents for creation of vrt (-te flag from gdalbuildvrt)
+            list_out_gregion = stdout_gregion.split("\n")
+            gdabuildvrt_params.append("-te")
+            gdabuildvrt_params.append(list_out_gregion[4])  # xmin/w
+            gdabuildvrt_params.append(list_out_gregion[3])  # ymin/s
+            gdabuildvrt_params.append(list_out_gregion[5])  # xmax/e
+            gdabuildvrt_params.append(list_out_gregion[2])  # ymax/n
+        # out and input for gdalbuildvrt
+        gdabuildvrt_params.append(vrt_out)
+        gdabuildvrt_params.append(rimport_inp)
+        # build vrt with previous defined parameters
+        (
+            errorid,
+            stdout_gdalbuildvrt,
+            stderr_gdalbuildvrt,
+        ) = self.ginit.run_module("/usr/bin/gdalbuildvrt", gdabuildvrt_params)
+
+        # gdalinfo for created vrt
+        gdalinfo_params = [vrt_out]
+        errorid, stdout_gdalinfo, stderr_gdalinfo = self.ginit.run_module(
+            "/usr/bin/gdalinfo", gdalinfo_params
+        )
+        # parse "Size" output of gdalinfo
+        rastersize_list = (
+            stdout_gdalinfo.split("Size is")[1].split("\n")[0].split(",")
+        )
+        # size = x-dim*y-dim
+        rastersize_x = int(rastersize_list[0])
+        rastersize_y = int(rastersize_list[1])
+        rastersize = rastersize_x * rastersize_y
+
+        # if different import/reprojection resolution set:
+        rimport_res = [
+            x for x in process_executable_params if "resolution=" in x
+        ]
+        res_val = None
+        # If raster exceeds cell limit already in original resolution, next part can be skipped
+        if rimport_res and (rastersize < self.cell_limit):
+            # determine estimated resolution
+            errorid, stdout_estres, stderr_estres = self.ginit.run_module(
+                "r.import", [vrt_out, "-e"]
+            )
+            if "Estimated" in stderr_estres:
+                # if data in different projection get rest_est with output of r.import -e
+                res_est = float(stderr_estres.split("\n")[-2].split(":")[1])
+            else:
+                # if data in same projection can use gdalinfo output
+                res_xy = (
+                    stdout_gdalinfo.split("Pixel Size = (")[1]
+                    .split(")\n")[0]
+                    .split(",")
+                )
+                # get estimated resolution
+                # (analoug as done within r.import -e: estres = math.sqrt((n - s) * (e - w) / cells))
+                res_est = math.sqrt(abs(float(res_xy[0]) * float(res_xy[1])))
+            # determine set resolution value
+            resolution = rimport_res[0].split("=")[1]
+            if resolution == "value":
+                res_val = [
+                    float(
+                        [
+                            x
+                            for x in process_executable_params
+                            if "resolution_value=" in x
+                        ][0].split("=")[1]
+                    )
+                ] * 2
+            elif resolution == "region":
+                # if already queried above reuse, otherwise execute g.region command
+                try:
+                    stdout_gregion
+                except Exception:
+                    (
+                        errorid,
+                        stdout_gregion,
+                        stderr_gregion,
+                    ) = self.ginit.run_module("g.region", ["-ug"])
+                res_val_ns = float(
+                    [x for x in stdout_gregion.split("\n") if "nsres=" in x][
+                        0
+                    ].split("=")[1]
+                )
+                res_val_ew = float(
+                    [x for x in stdout_gregion.split("\n") if "ewres=" in x][
+                        0
+                    ].split("=")[1]
+                )
+                res_val = [res_val_ns, res_val_ew]
+        if res_val:
+            if (res_val[0] < res_est) | (res_val[1] < res_est):
+                # only check if smaller resolution set
+                res_change_x = res_est / res_val[1]
+                res_change_y = res_est / res_val[0]
+                # approximate raster size after resampling
+                # by using factor of changed resolution
+                rastersize = (
+                    rastersize_x * res_change_x * rastersize_y * res_change_y
+                )
+
+        # compare estimated raster output size with pixel limit
+        # and raise exception if exceeded
+        if rastersize > self.cell_limit:
+            raise AsyncProcessError(
+                "Processing pixel limit exceeded for raster import. "
+                "Please set e.g. region smaller."
+            )
 
     def _check_reset_region(self):
         """Check the current region settings against the user cell limit.
@@ -1479,10 +1628,14 @@ def _run_module(self, process, poll_time=0.05):
             )
             self._send_resource_update(message)
 
+        # Check pixel limit for r.import operations
+        if process.executable == "r.import":
+            self._check_pixellimit_rimport(process.executable_params)
+
         # Check reset region if a g.region call was present in the process
         # chain. By default the initial value of last_module is "g.region" to
-        # assure for first run of a process from the process chain, the
-        # region settings are evaluated
+        # assure for first run of a process from the process chain, the region
+        # settings are evaluated
         if (
             self.last_module == "g.region"
             and process.skip_permission_check is False

diff --git a/tests/test_download_cache.py b/tests/test_download_cache.py
@@ -136,7 +136,7 @@ def test_download_cache(self):
     #         "HTML status code is wrong %i" % rv.status_code,
     #     )
     #     self.assertEqual(
-    #         rv.mimetype, "application/json", "Wrong mimetype %s" % rv.mimetype
+    #        rv.mimetype, "application/json", "Wrong mimetype %s" % rv.mimetype
     #     )
 
     # TODO: configure wrong download cache path differently (via config file)
@@ -158,7 +158,7 @@ def test_download_cache(self):
     #         "HTML status code is wrong %i" % rv.status_code,
     #     )
     #     self.assertEqual(
-    #         rv.mimetype, "application/json", "Wrong mimetype %s" % rv.mimetype
+    #        rv.mimetype, "application/json", "Wrong mimetype %s" % rv.mimetype
     #     )