Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix fixture_xr_image to open earth_day_01d_p directly with rioxarray #2963

Merged
merged 8 commits into from
Jan 11, 2024
Merged

Fix fixture_xr_image to open earth_day_01d_p directly with rioxarray #2963

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
eeb6c63
Fix fixture_xr_image to open earth_day_01d_p directly with rioxarray
weiji14 Jan 7, 2024
f4e263a
Merge branch 'main' into fix/xr_image_fixture
weiji14 Jan 8, 2024
4f02cb3
Update baseline images for test_tilemap_* with GMT 6.5.0 and gs 10.02.1
weiji14 Jan 8, 2024
98f30ac
Trigger re-caching of earth_day_01d_p file
weiji14 Jan 8, 2024
64cd663
Merge branch 'main' into fix/xr_image_fixture
weiji14 Jan 8, 2024
b80652f
Merge branch 'main' into fix/xr_image_fixture
weiji14 Jan 8, 2024
b8c9c3c
Remove elif block for manually creating RGB image
weiji14 Jan 8, 2024
2a41ddb
Merge branch 'main' into fix/xr_image_fixture
weiji14 Jan 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions pygmt/helpers/caching.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
"""
Functions for download remote data files as cache.
Functions for downloading remote data files to cache.
"""
from pygmt.src import which


def cache_data():
"""
Download GMT remote data files used in PyGMT tests and docs as cache files.
Download GMT remote data files used in PyGMT tests and docs to cache folder.
"""
# List of datasets to download
datasets = [
Expand Down Expand Up @@ -57,7 +57,7 @@ def cache_data():
# Earth WDMAM grids
"@earth_wdmam_01d_g",
"@S90E000.earth_wdmam_03m_g.nc", # Specific grid for 03m test
# Earth day/night grids
# Earth day/night images
"@earth_day_01d_p",
# Other cache files
"@capitals.gmt",
Expand Down
5 changes: 3 additions & 2 deletions pygmt/tests/baseline/test_tilemap_no_clip_False.png.dvc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
outs:
- md5: e2a9a57f61bdca0c21ddec3e1ae5d819
size: 16595
- md5: e55d9848135fc14c9cfc3d2c85bd82bb
size: 16472
path: test_tilemap_no_clip_False.png
hash: md5
5 changes: 3 additions & 2 deletions pygmt/tests/baseline/test_tilemap_no_clip_True.png.dvc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
outs:
- md5: 86ce085faad17433dfabba675d0379ee
size: 38836
- md5: 53949854f32b450e55f1be86102387e9
size: 38816
path: test_tilemap_no_clip_True.png
hash: md5
5 changes: 3 additions & 2 deletions pygmt/tests/baseline/test_tilemap_ogc_wgs84.png.dvc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
outs:
- md5: 6da0601f305a6c3391e4a1aa0aee6173
size: 37139
- md5: 5f225e4dd26f44e07bcbd8e713c67dbe
size: 37343
path: test_tilemap_ogc_wgs84.png
hash: md5
5 changes: 3 additions & 2 deletions pygmt/tests/baseline/test_tilemap_web_mercator.png.dvc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
outs:
- md5: 2965f6711c5878014491c7e6f791048e
size: 62077
- md5: 14f7afd5d48ed6aed88d495783f0ba42
size: 62398
path: test_tilemap_web_mercator.png
hash: md5
26 changes: 2 additions & 24 deletions pygmt/tests/test_grdimage_image.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,9 @@
"""
Test Figure.grdimage on 3-band RGB images.
"""
import numpy as np
import pandas as pd
import pytest
import xarray as xr
from pygmt import Figure, which

rasterio = pytest.importorskip("rasterio")
rioxarray = pytest.importorskip("rioxarray")


Expand All @@ -19,26 +15,8 @@ def fixture_xr_image():
"""
geotiff = which(fname="@earth_day_01d_p", download="c")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The earth_day_01d_p.tif grid downloaded with GMT 6.5.0 shows this:

$ gmt which @earth_day_01d_p
$ gdalinfo ~/.gmt/cache/earth_day_01d_p.tif
Driver: GTiff/GeoTIFF
Files: ~/.gmt/cache/earth_day_01d_p.tif
Size is 360, 180
Coordinate System is:
GEOGCRS["unknown",
    DATUM["unknown",
        ELLIPSOID["unknown",6378137,0,
            LENGTHUNIT["metre",1,
                ID["EPSG",9001]]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433,
            ID["EPSG",9122]]],
    CS[ellipsoidal,2],
        AXIS["latitude",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]],
        AXIS["longitude",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]]]
Data axis to CRS axis mapping: 2,1
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (-180.0000000,  90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)
Lower Left  (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)
Upper Right ( 180.0000000,  90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)
Center      (   0.0000000,   0.0000000) (  0d 0' 0.01"E,  0d 0' 0.01"N)
Band 1 Block=360x7 Type=Byte, ColorInterp=Undefined
  Min=10.000 Max=255.000 
  Minimum=10.000, Maximum=255.000, Mean=49.508, StdDev=65.299
  NoData Value=0
  Metadata:
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=49.507654320988
    STATISTICS_MINIMUM=10
    STATISTICS_STDDEV=65.299010755209
    STATISTICS_VALID_PERCENT=100
Band 2 Block=360x7 Type=Byte, ColorInterp=Undefined
  Min=10.000 Max=255.000 
  Minimum=10.000, Maximum=255.000, Mean=49.911, StdDev=62.129
  NoData Value=0
  Metadata:
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=49.910941358025
    STATISTICS_MINIMUM=10
    STATISTICS_STDDEV=62.129078727248
    STATISTICS_VALID_PERCENT=100
Band 3 Block=360x7 Type=Byte, ColorInterp=Undefined
  Min=10.000 Max=255.000 
  Minimum=10.000, Maximum=255.000, Mean=65.605, StdDev=44.820
  NoData Value=0
  Metadata:
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=65.605154320988
    STATISTICS_MINIMUM=10
    STATISTICS_STDDEV=44.819618148349
    STATISTICS_VALID_PERCENT=100

If I run gmt which @earth_day_01d_p with GMT 6.4.0, it errors with gmtwhich [ERROR]: File earth_day_01d_p.tif not found!. Did something change on the GMT data server?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the old @earth_day_01d_p (unzip earth_day_01d_p.tif.zip) from #2937 cached at https://github.com/GenericMappingTools/pygmt/actions/runs/7376169058 that was used in GMT 6.4.0:

$ gdalinfo earth_day_01d_p.tif
Driver: GTiff/GeoTIFF
Files: earth_day_01d_p.tif
Size is 360, 180
Coordinate System is:
GEOGCRS["unknown",
    DATUM["unknown",
        ELLIPSOID["unknown",6378137,0,
            LENGTHUNIT["metre",1,
                ID["EPSG",9001]]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433,
            ID["EPSG",9122]]],
    CS[ellipsoidal,2],
        AXIS["latitude",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]],
        AXIS["longitude",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]]]
Data axis to CRS axis mapping: 2,1
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=BAND
  PREDICTOR=2
Corner Coordinates:
Upper Left  (-180.0000000,  90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)
Lower Left  (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)
Upper Right ( 180.0000000,  90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)
Center      (   0.0000000,   0.0000000) (  0d 0' 0.01"E,  0d 0' 0.01"N)
Band 1 Block=360x22 Type=Byte, ColorInterp=Palette
  Color Table (RGB with 256 entries)
    0: 248,224,168,255
    1: 240,224,152,255
    2: 104,104,96,255
    3: 16,24,56,255
    4: 24,28,24,255
    5: 104,80,48,255
    6: 220,224,228,255
    7: 48,56,32,255
    8: 184,192,176,255
    9: 104,104,40,255
   10: 16,48,8,255
   11: 96,96,56,255
   12: 188,184,184,255
   13: 128,112,60,255
   14: 48,72,24,255
   15: 8,16,48,255
   16: 216,200,192,255
   17: 80,80,28,255
   18: 56,48,44,255
   19: 72,88,24,255
   20: 204,168,140,255
   21: 128,136,136,255
   22: 72,68,40,255
   23: 144,120,68,255
   24: 144,144,136,255
   25: 88,80,48,255
   26: 168,160,152,255
   27: 88,104,40,255
   28: 8,16,32,255
   29: 168,116,84,255
   30: 120,100,48,255
   31: 200,192,184,255
   32: 176,168,160,255
   33: 136,112,80,255
   34: 204,204,200,255
   35: 184,152,124,255
   36: 96,76,48,255
   37: 88,72,48,255
   38: 184,196,192,255
   39: 28,56,48,255
   40: 92,100,124,255
   41: 184,176,176,255
   42: 160,160,152,255
   43: 132,144,144,255
   44: 208,192,184,255
   45: 128,100,64,255
   46: 72,100,24,255
   47: 36,48,16,255
   48: 140,140,80,255
   49: 152,144,136,255
   50: 56,64,16,255
   51: 168,168,168,255
   52: 224,200,144,255
   53: 176,176,168,255
   54: 80,88,28,255
   55: 152,144,144,255
   56: 156,148,84,255
   57: 160,160,160,255
   58: 16,16,48,255
   59: 168,176,168,255
   60: 216,200,176,255
   61: 16,8,48,255
   62: 72,80,24,255
   63: 132,140,124,255
   64: 88,80,40,255
   65: 152,152,176,255
   66: 16,16,56,255
   67: 20,48,24,255
   68: 200,200,192,255
   69: 56,48,72,255
   70: 40,56,24,255
   71: 120,136,56,255
   72: 152,160,152,255
   73: 8,8,24,255
   74: 192,184,176,255
   75: 248,244,172,255
   76: 116,124,136,255
   77: 88,68,40,255
   78: 64,72,32,255
   79: 172,180,176,255
   80: 176,152,132,255
   81: 24,40,24,255
   82: 60,100,24,255
   83: 168,148,128,255
   84: 176,132,88,255
   85: 200,192,192,255
   86: 76,76,48,255
   87: 152,116,80,255
   88: 8,16,60,255
   89: 152,152,144,255
   90: 240,216,152,255
   91: 52,80,32,255
   92: 20,24,40,255
   93: 132,144,156,255
   94: 88,88,48,255
   95: 20,32,8,255
   96: 84,68,28,255
   97: 172,164,180,255
   98: 152,152,152,255
   99: 120,112,56,255
  100: 88,88,40,255
  101: 56,80,8,255
  102: 80,80,40,255
  103: 176,176,160,255
  104: 48,56,8,255
  105: 104,84,68,255
  106: 48,120,76,255
  107: 184,176,168,255
  108: 108,96,56,255
  109: 232,224,212,255
  110: 16,16,40,255
  111: 96,88,48,255
  112: 144,152,144,255
  113: 104,112,68,255
  114: 220,180,148,255
  115: 28,32,60,255
  116: 56,72,8,255
  117: 104,96,40,255
  118: 56,72,24,255
  119: 24,64,24,255
  120: 148,164,160,255
  121: 180,176,140,255
  122: 216,208,196,255
  123: 156,128,92,255
  124: 72,84,40,255
  125: 56,80,16,255
  126: 72,72,24,255
  127: 48,44,12,255
  128: 124,108,96,255
  129: 64,80,24,255
  130: 132,136,148,255
  131: 200,200,184,255
  132: 120,100,68,255
  133: 16,28,24,255
  134: 144,144,144,255
  135: 116,120,124,255
  136: 248,224,184,255
  137: 184,184,168,255
  138: 72,80,32,255
  139: 152,148,160,255
  140: 56,76,44,255
  141: 232,208,144,255
  142: 176,168,168,255
  143: 104,96,88,255
  144: 60,56,32,255
  145: 168,152,92,255
  146: 184,184,176,255
  147: 152,144,124,255
  148: 232,180,120,255
  149: 208,204,192,255
  150: 152,152,128,255
  151: 188,180,108,255
  152: 48,56,16,255
  153: 28,60,40,255
  154: 56,64,24,255
  155: 160,152,144,255
  156: 48,72,8,255
  157: 168,160,160,255
  158: 228,228,184,255
  159: 48,64,8,255
  160: 160,152,152,255
  161: 148,112,64,255
  162: 32,48,8,255
  163: 80,68,40,255
  164: 132,140,100,255
  165: 84,104,28,255
  166: 36,48,28,255
  167: 188,192,184,255
  168: 168,168,160,255
  169: 240,248,248,255
  170: 60,52,24,255
  171: 248,248,220,255
  172: 36,36,8,255
  173: 144,152,152,255
  174: 32,64,8,255
  175: 96,80,80,255
  176: 72,140,148,255
  177: 48,72,16,255
  178: 200,184,176,255
  179: 148,144,152,255
  180: 136,140,136,255
  181: 244,200,144,255
  182: 180,200,212,255
  183: 160,168,160,255
  184: 40,64,8,255
  185: 56,72,16,255
  186: 48,56,24,255
  187: 64,72,24,255
  188: 176,184,168,255
  189: 48,64,16,255
  190: 104,116,56,255
  191: 28,24,56,255
  192: 132,124,136,255
  193: 72,72,32,255
  194: 80,88,40,255
  195: 56,88,12,255
  196: 236,204,184,255
  197: 72,64,28,255
  198: 64,84,32,255
  199: 24,48,8,255
  200: 88,80,84,255
  201: 240,232,236,255
  202: 148,100,60,255
  203: 148,116,100,255
  204: 160,144,140,255
  205: 208,132,88,255
  206: 36,32,24,255
  207: 148,132,132,255
  208: 52,84,24,255
  209: 196,180,168,255
  210: 204,188,140,255
  211: 24,16,52,255
  212: 104,120,40,255
  213: 240,184,144,255
  214: 72,88,32,255
  215: 104,84,40,255
  216: 60,52,16,255
  217: 208,200,184,255
  218: 32,56,8,255
  219: 24,56,8,255
  220: 160,168,152,255
  221: 116,136,164,255
  222: 204,196,172,255
  223: 48,84,12,255
  224: 88,128,80,255
  225: 64,64,24,255
  226: 248,244,192,255
  227: 172,180,192,255
  228: 144,144,60,255
  229: 40,56,8,255
  230: 76,72,88,255
  231: 40,72,24,255
  232: 196,180,160,255
  233: 152,168,132,255
  234: 64,88,24,255
  235: 64,64,16,255
  236: 40,48,8,255
  237: 220,152,124,255
  238: 92,84,28,255
  239: 204,208,220,255
  240: 104,128,96,255
  241: 132,168,164,255
  242: 40,76,8,255
  243: 188,184,196,255
  244: 56,76,80,255
  245: 156,148,104,255
  246: 244,240,156,255
  247: 68,80,16,255
  248: 28,64,80,255
  249: 152,176,188,255
  250: 132,84,56,255
  251: 8,8,48,255
  252: 8,8,56,255
  253: 8,40,24,255
  254: 8,72,96,255
  255: 248,248,248,255

See how the GMT 6.4.0 version has 1 band with a Color Table, compared to 3 bands with GMT 6.5.0 release as in https://github.com/GenericMappingTools/pygmt/pull/2963/files#r1444072938.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, asked about the dataset changes upstream at GenericMappingTools/gmtserver-admin#257.

with rioxarray.open_rasterio(filename=geotiff) as rda:
if len(rda.band) == 1:
with rasterio.open(fp=geotiff) as src:
df_colormap = pd.DataFrame.from_dict(
data=src.colormap(1), orient="index"
)
array = src.read()

red = np.vectorize(df_colormap[0].get)(array)
green = np.vectorize(df_colormap[1].get)(array)
blue = np.vectorize(df_colormap[2].get)(array)
# alpha = np.vectorize(df_colormap[3].get)(array)

rda.data = red
da_red = rda.astype(dtype=np.uint8).copy()
rda.data = green
da_green = rda.astype(dtype=np.uint8).copy()
rda.data = blue
da_blue = rda.astype(dtype=np.uint8).copy()

xr_image = xr.concat(objs=[da_red, da_green, da_blue], dim="band")
if len(rda.band) == 3:
xr_image = rda.load()
Comment on lines 17 to +19
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait for GenericMappingTools/gmtserver-admin#257, to see if the 3-band GeoTIFF file stays, or we revert back to the quantized/indexed GeoTIFF file from before Sep 2023.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gonna assume that the 3-band GeoTIFF will stay for now, so that we can move on with the GMT 6.5.0 update. Marked this PR as ready for review.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about splitting this PR into two separate RPs so if the upstream decide to to back to the indexed geotiff file, we can easily revert the changes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kinda want to take the code from this fixture and just make a load_blue_marble dataset function after this 🙂 If we do go back to the indexed GeoTIFF files, then the modifications would go in that load_blue_marble function.

assert xr_image.sizes == {"band": 3, "y": 180, "x": 360}
return xr_image

Expand Down
Loading