geopandas 1.0.0 or pyogrio 0.9.0 doesn't support OGR_GMT output? #3301

seisman · 2024-06-25T05:56:42Z

geopandas v1.0.0 was released a few hours ago and now our Docs workflow fails to build (https://github.com/GenericMappingTools/pygmt/actions/runs/9559533732/job/26634813440) which is not caught in our "GMT Dev Tests" workflow (why?):

Extension error:
Here is a summary of the problems encountered when running the examples:

Unexpected failing examples:

    ../examples/get_started/04_table_inputs.py failed leaving traceback:

    Traceback (most recent call last):
      File "/home/runner/work/pygmt/pygmt/examples/get_started/04_table_inputs.py", line 111, in <module>
        fig.plot(data=gdf, style="c0.2c", fill="purple")
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/helpers/decorators.py", line 609, in new_module
        return module_func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/helpers/decorators.py", line 773, in new_module
        return module_func(*bound.args, **bound.kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/src/plot.py", line 258, in plot
        with lib.virtualfile_in(
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/contextlib.py", line 137, in __enter__
        return next(self.gen)
               ^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/helpers/tempfile.py", line 164, in tempfile_from_geojson
        geojson.to_file(**ogrgmt_kwargs)
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/geodataframe.py", line 1536, in to_file
        _to_file(self, filename, driver, schema, index, **kwargs)
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 686, in _to_file
        _to_file_pyogrio(df, filename, driver, schema, crs, mode, metadata, **kwargs)
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 748, in _to_file_pyogrio
        pyogrio.write_dataframe(df, filename, driver=driver, metadata=metadata, **kwargs)
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/geopandas.py", line 456, in write_dataframe
        path, driver = _get_write_path_driver(path, driver, append=append)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/raw.py", line 564, in _get_write_path_driver
        if not ogr_driver_supports_write(driver):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "pyogrio/_ogr.pyx", line 105, in pyogrio._ogr.ogr_driver_supports_write
      File "pyogrio/_ogr.pyx", line 287, in pyogrio._ogr._get_driver_metadata_item
    pyogrio.errors.DataSourceError: Could not obtain driver: OGR_GMT (check that it was installed correctly into GDAL)

    ../examples/gallery/lines/roads.py failed leaving traceback:

    Traceback (most recent call last):
      File "/home/runner/work/pygmt/pygmt/examples/gallery/lines/roads.py", line 19, in <module>
        gdf = gpd.read_file(
              ^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 294, in _read_file
        return _read_file_pyogrio(
               ^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 547, in _read_file_pyogrio
        return pyogrio.read_dataframe(path_or_bytes, bbox=bbox, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/geopandas.py", line 261, in read_dataframe
        result = read_func(
                 ^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/raw.py", line 196, in read
        return ogr_read(
               ^^^^^^^^^
      File "pyogrio/_io.pyx", line 1239, in pyogrio._io.ogr_read
      File "pyogrio/_io.pyx", line 219, in pyogrio._io.ogr_open
    pyogrio.errors.DataSourceError: No driver registered.

The text was updated successfully, but these errors were encountered:

seisman · 2024-06-26T09:28:53Z

That's weird. I tried to create the same environment locally.

mamba create --name pygmt-docs python=3.12 gmt=6.5.0 ghostscript=10.03.1 numpy pandas xarray netCDF4 packaging contextily geopandas ipython rioxarray build make pip myst-parser panel sphinx sphinx-autodoc-typehints sphinx-copybutton sphinx-design sphinx-gallery sphinx_rtd_theme

Now I have geopandas 1.0.0, pyogrio 0.9.0 and gdal 3.9.0 installed, but the script works well for me.

In [1]: import pygmt

In [2]: import geopandas as gpd

In [3]: # Example GeoDataFrame
   ...: gdf = gpd.GeoDataFrame(
   ...:     {
   ...:         "geometry": gpd.points_from_xy([2, 5, 9], [2, 3, 4]),
   ...:         "value": [10, 20, 30],
   ...:     }
   ...: )
   ...:
   ...: # Use the GeoDataFrame to specify the 'data' parameter
   ...: fig = pygmt.Figure()
   ...: fig.basemap(region=[0, 10, 0, 5], projection="X10c/5c", frame=True)
   ...: fig.plot(data=gdf, style="c0.2c", fill="purple")
   ...: fig.show()

seisman · 2024-06-26T10:52:31Z

There is one more failure:

../examples/gallery/lines/roads.py unexpectedly failed to execute correctly:

    Traceback (most recent call last):
      File "/home/runner/work/pygmt/pygmt/examples/gallery/lines/roads.py", line 19, in <module>
        gdf = gpd.read_file(
              ^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 294, in _read_file
        return _read_file_pyogrio(
               ^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 547, in _read_file_pyogrio
        return pyogrio.read_dataframe(path_or_bytes, bbox=bbox, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/geopandas.py", line 261, in read_dataframe
        result = read_func(
                 ^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/raw.py", line 196, in read
        return ogr_read(
               ^^^^^^^^^
      File "pyogrio/_io.pyx", line 1239, in pyogrio._io.ogr_read
      File "pyogrio/_io.pyx", line 219, in pyogrio._io.ogr_open
    pyogrio.errors.DataSourceError: No driver registered.

seisman · 2024-06-26T16:19:36Z

pyogrio.errors.DataSourceError: No driver registered.

This error comes from GDAL https://github.com/OSGeo/gdal/blob/a66c71d0e1e19a5ef9d94b33bef90074173be9c2/gcore/gdaldataset.cpp#L3965C7-L3965C7

weiji14 · 2024-07-09T04:17:19Z

geopandas v1.0.0 was released a few hours ago and now our Docs workflow fails to build (https://github.com/GenericMappingTools/pygmt/actions/runs/9559533732/job/26634813440) which is not caught in our "GMT Dev Tests" workflow (why?)

The "GMT Dev Tests" installs geopandas from PyPI, whereas the Docs workflow installs geopandas from conda-forge. Are there differences in GDAL version used? I'm reading https://github.com/geopandas/pyogrio/blob/v0.9.0/docs/source/install.md#troubleshooting-installation-errors, and it says errors like pyogrio.errors.DataSourceError: Could not obtain driver ... may be due to incompatible versions of GDAL in fiona and pyogrio, but we don't have fiona installed anymore 😕

Your example at #3301 (comment) works for me too if I run it individually. But when I do cd docs; make clean; make all to run the sphinx build, it fails the first time, but works if I run it twice. Looks like there's some flaky behaviour going on.

seisman · 2024-07-09T04:29:15Z

The "GMT Dev Tests" installs geopandas from PyPI, whereas the Docs workflow installs geopandas from conda-forge. Are there differences in GDAL version used?

The "Tests" workflow also has geopandas from conda-forge, but the geopandas tests all pass.

weiji14 · 2024-07-09T05:36:37Z

Found a related issue upstream at geopandas/pyogrio#144. There is supposed to be some logic to register GDAL drivers when pyogrio is imported (geopandas/pyogrio#145), but it is not working properly in Sphinx build for some reason, and causing that pyogrio.errors.DataSourceError: Could not obtain driver: OGR_GMT (check that it was installed correctly into GDAL) error. Note that the OGR_GMT driver does exist in our GDAL installation, just that the driver isn't loaded properly.

Workaround is to manually register the drivers like so:

import pyogrio

pyogrio.core._register_drivers()

See also https://github.com/GenericMappingTools/pygmt/pull/3305/files#r1669745089 where this patch is applied.

weiji14 · 2024-07-10T00:44:19Z

Have reported this upstream, and got a super helpful response at geopandas/pyogrio#448 (comment). It seems like there may be some conflict with the order of how rioxarray/rasterio and pyogrio is loading the GDAL drivers.

weiji14 · 2024-12-02T22:12:39Z

Hmm, hitting into this issue again, but this time on rasterio. Traceback from https://github.com/GenericMappingTools/pygmt/actions/runs/12128509215/job/33815018072#step:7:956 at #3667:

Unexpected failing examples (2):

    ../examples/gallery/maps/tilemaps.py failed leaving traceback:

    Traceback (most recent call last):
      File "/home/runner/work/pygmt/pygmt/examples/gallery/maps/tilemaps.py", line 37, in <module>
        fig.tilemap(
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/helpers/decorators.py", line 610, in new_module
        return module_func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/helpers/decorators.py", line 773, in new_module
        return module_func(*bound.args, **bound.kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/src/tilemap.py", line 145, in tilemap
        raster = raster.rio.reproject(dst_crs="OGC:CRS84")
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/rioxarray/raster_array.py", line 444, in reproject
        dst_affine, dst_width, dst_height = _make_dst_affine(
                                            ^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/rioxarray/raster_array.py", line 146, in _make_dst_affine
        dst_affine, dst_width, dst_height = rasterio.warp.calculate_default_transform(
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/rasterio/env.py", line 413, in wrapper
        return f(*args, **kwds)
               ^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/rasterio/warp.py", line 556, in calculate_default_transform
        dst_affine, dst_width, dst_height = _calculate_default_transform(
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "rasterio/_warp.pyx", line 744, in rasterio._warp._calculate_default_transform
      File "rasterio/_base.pyx", line 221, in rasterio._base.open_dataset
      File "rasterio/_err.pyx", line 360, in rasterio._err.exc_wrap_pointer
    SystemError: Unknown GDAL Error. To debug: https://rasterio.readthedocs.io/en/latest/topics/errors.html#debugging-internal-gdal-functions

    ../examples/gallery/images/rgb_image.py failed leaving traceback:

    Traceback (most recent call last):
      File "/home/runner/work/pygmt/pygmt/examples/gallery/images/rgb_image.py", line 24, in <module>
        with rioxarray.open_rasterio(
             ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/rioxarray/_io.py", line 1128, in open_rasterio
        riods = manager.acquire()
                ^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 193, in acquire
        file, _ = self._acquire_with_cache_info(needs_lock)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
        file = self._opener(*self._args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/rasterio/env.py", line 463, in wrapper
        return f(*args, **kwds)
               ^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/rasterio/__init__.py", line 356, in open
        dataset = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "rasterio/_base.pyx", line 312, in rasterio._base.DatasetBase.__init__
    rasterio.errors.RasterioIOError: No driver registered.

Thinking if this has something to do with https://github.com/rasterio/rasterio/releases/tag/1.4.3, possibly due to changes in rasterio/rasterio#3260?

seisman added the bug Something isn't working label Jun 25, 2024

This was referenced Jun 27, 2024

CI: Temporarily pin geopandas<1.0 in the Docs workflow #3304

Merged

CI: Unpin geopandas and rasterio after the fix in #3672 #3305

Merged

weiji14 mentioned this issue Jul 9, 2024

pyogrio.errors.DataSourceError: No driver registered geopandas/pyogrio#448

Closed

weiji14 added the upstream Bug or missing feature of upstream core GMT label Jul 10, 2024

weiji14 mentioned this issue Dec 3, 2024

List function aliases in a more compact layout #3666

Merged

This was referenced Dec 3, 2024

CI: Pin rasterio < 1.4.3 temporarily #3668

Merged

clib.session: Add the GMT_SESSION_NOGDALCLOSE flag to keep GDAL open #3672

Merged

seisman added this to the 0.14.0 milestone Dec 4, 2024

seisman removed the upstream Bug or missing feature of upstream core GMT label Dec 4, 2024

seisman closed this as completed in #3672 Dec 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

geopandas 1.0.0 or pyogrio 0.9.0 doesn't support OGR_GMT output? #3301

geopandas 1.0.0 or pyogrio 0.9.0 doesn't support OGR_GMT output? #3301

seisman commented Jun 25, 2024 •

edited

Loading

seisman commented Jun 26, 2024 •

edited

Loading

seisman commented Jun 26, 2024

seisman commented Jun 26, 2024

weiji14 commented Jul 9, 2024

seisman commented Jul 9, 2024

weiji14 commented Jul 9, 2024 •

edited

Loading

weiji14 commented Jul 10, 2024 •

edited

Loading

weiji14 commented Dec 2, 2024 •

edited

Loading

geopandas 1.0.0 or pyogrio 0.9.0 doesn't support OGR_GMT output? #3301

geopandas 1.0.0 or pyogrio 0.9.0 doesn't support OGR_GMT output? #3301

Comments

seisman commented Jun 25, 2024 • edited Loading

seisman commented Jun 26, 2024 • edited Loading

seisman commented Jun 26, 2024

seisman commented Jun 26, 2024

weiji14 commented Jul 9, 2024

seisman commented Jul 9, 2024

weiji14 commented Jul 9, 2024 • edited Loading

weiji14 commented Jul 10, 2024 • edited Loading

weiji14 commented Dec 2, 2024 • edited Loading

seisman commented Jun 25, 2024 •

edited

Loading

seisman commented Jun 26, 2024 •

edited

Loading

weiji14 commented Jul 9, 2024 •

edited

Loading

weiji14 commented Jul 10, 2024 •

edited

Loading

weiji14 commented Dec 2, 2024 •

edited

Loading