Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix fixture_xr_image to open earth_day_01d_p directly with rioxarray #2963

Merged
merged 8 commits into from
Jan 11, 2024
Merged

Fix fixture_xr_image to open earth_day_01d_p directly with rioxarray #2963

merged 8 commits into from
Jan 11, 2024

Conversation

weiji14
Copy link
Member

@weiji14 weiji14 commented Jan 7, 2024

Description of proposed changes

Rioxarray can now read the 3-bands from @earth_day_01d_p directly without needing to parse the colorinterp information from the GeoTIFF file.

Addresses some of the failing tests in #2961 (comment), resolves #2590 (comment)

Reminders

  • Run make format and make check to make sure the code follows the style guide.
  • Add tests for new features or tests that would have caught the bug that you're fixing.
  • Add new public functions/methods/classes to doc/api/index.rst.
  • Write detailed docstrings for all functions/methods.
  • If wrapping a new module, open a 'Wrap new GMT module' issue and submit reasonably-sized PRs.
  • If adding new functionality, add an example to docstrings or tutorials.
  • Use underscores (not hyphens) in names of Python files and directories.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform
specific operations. Supported slash command is:

  • /format: automatically format and lint the code

@weiji14
Fix fixture_xr_image to open earth_day_01d_p directly with rioxarray
eeb6c63 
Rioxarray can now read the 3-bands from `@earth_day_01d_p` directly without needing to parse the colorinterp information from the GeoTIFF file.
@weiji14 weiji14 added the maintenance Boring but important stuff for the core devs label Jan 7, 2024
@weiji14 weiji14 added this to the 0.11.0 milestone Jan 7, 2024
@weiji14 weiji14 self-assigned this Jan 7, 2024
weiji14
weiji14 commented Jan 7, 2024
View reviewed changes
pygmt/tests/test_grdimage_image.py
@@ -19,7 +19,9 @@ def fixture_xr_image():
"""
geotiff = which(fname="@earth_day_01d_p", download="c")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The earth_day_01d_p.tif grid downloaded with GMT 6.5.0 shows this:

$ gmt which @earth_day_01d_p
$ gdalinfo ~/.gmt/cache/earth_day_01d_p.tif
Driver: GTiff/GeoTIFF
Files: ~/.gmt/cache/earth_day_01d_p.tif
Size is 360, 180
Coordinate System is:
GEOGCRS["unknown",
    DATUM["unknown",
        ELLIPSOID["unknown",6378137,0,
            LENGTHUNIT["metre",1,
                ID["EPSG",9001]]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433,
            ID["EPSG",9122]]],
    CS[ellipsoidal,2],
        AXIS["latitude",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]],
        AXIS["longitude",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]]]
Data axis to CRS axis mapping: 2,1
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (-180.0000000,  90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)
Lower Left  (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)
Upper Right ( 180.0000000,  90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)
Center      (   0.0000000,   0.0000000) (  0d 0' 0.01"E,  0d 0' 0.01"N)
Band 1 Block=360x7 Type=Byte, ColorInterp=Undefined
  Min=10.000 Max=255.000 
  Minimum=10.000, Maximum=255.000, Mean=49.508, StdDev=65.299
  NoData Value=0
  Metadata:
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=49.507654320988
    STATISTICS_MINIMUM=10
    STATISTICS_STDDEV=65.299010755209
    STATISTICS_VALID_PERCENT=100
Band 2 Block=360x7 Type=Byte, ColorInterp=Undefined
  Min=10.000 Max=255.000 
  Minimum=10.000, Maximum=255.000, Mean=49.911, StdDev=62.129
  NoData Value=0
  Metadata:
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=49.910941358025
    STATISTICS_MINIMUM=10
    STATISTICS_STDDEV=62.129078727248
    STATISTICS_VALID_PERCENT=100
Band 3 Block=360x7 Type=Byte, ColorInterp=Undefined
  Min=10.000 Max=255.000 
  Minimum=10.000, Maximum=255.000, Mean=65.605, StdDev=44.820
  NoData Value=0
  Metadata:
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=65.605154320988
    STATISTICS_MINIMUM=10
    STATISTICS_STDDEV=44.819618148349
    STATISTICS_VALID_PERCENT=100

If I run gmt which @earth_day_01d_p with GMT 6.4.0, it errors with gmtwhich [ERROR]: File earth_day_01d_p.tif not found!. Did something change on the GMT data server?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the old @earth_day_01d_p (unzip earth_day_01d_p.tif.zip) from #2937 cached at https://github.com/GenericMappingTools/pygmt/actions/runs/7376169058 that was used in GMT 6.4.0:

$ gdalinfo earth_day_01d_p.tif
Driver: GTiff/GeoTIFF
Files: earth_day_01d_p.tif
Size is 360, 180
Coordinate System is:
GEOGCRS["unknown",
    DATUM["unknown",
        ELLIPSOID["unknown",6378137,0,
            LENGTHUNIT["metre",1,
                ID["EPSG",9001]]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433,
            ID["EPSG",9122]]],
    CS[ellipsoidal,2],
        AXIS["latitude",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]],
        AXIS["longitude",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]]]
Data axis to CRS axis mapping: 2,1
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=BAND
  PREDICTOR=2
Corner Coordinates:
Upper Left  (-180.0000000,  90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)
Lower Left  (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)
Upper Right ( 180.0000000,  90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)
Center      (   0.0000000,   0.0000000) (  0d 0' 0.01"E,  0d 0' 0.01"N)
Band 1 Block=360x22 Type=Byte, ColorInterp=Palette
  Color Table (RGB with 256 entries)
    0: 248,224,168,255
    1: 240,224,152,255
    2: 104,104,96,255
    3: 16,24,56,255
    4: 24,28,24,255
    5: 104,80,48,255
    6: 220,224,228,255
    7: 48,56,32,255
    8: 184,192,176,255
    9: 104,104,40,255
   10: 16,48,8,255
   11: 96,96,56,255
   12: 188,184,184,255
   13: 128,112,60,255
   14: 48,72,24,255
   15: 8,16,48,255
   16: 216,200,192,255
   17: 80,80,28,255
   18: 56,48,44,255
   19: 72,88,24,255
   20: 204,168,140,255
   21: 128,136,136,255
   22: 72,68,40,255
   23: 144,120,68,255
   24: 144,144,136,255
   25: 88,80,48,255
   26: 168,160,152,255
   27: 88,104,40,255
   28: 8,16,32,255
   29: 168,116,84,255
   30: 120,100,48,255
   31: 200,192,184,255
   32: 176,168,160,255
   33: 136,112,80,255
   34: 204,204,200,255
   35: 184,152,124,255
   36: 96,76,48,255
   37: 88,72,48,255
   38: 184,196,192,255
   39: 28,56,48,255
   40: 92,100,124,255
   41: 184,176,176,255
   42: 160,160,152,255
   43: 132,144,144,255
   44: 208,192,184,255
   45: 128,100,64,255
   46: 72,100,24,255
   47: 36,48,16,255
   48: 140,140,80,255
   49: 152,144,136,255
   50: 56,64,16,255
   51: 168,168,168,255
   52: 224,200,144,255
   53: 176,176,168,255
   54: 80,88,28,255
   55: 152,144,144,255
   56: 156,148,84,255
   57: 160,160,160,255
   58: 16,16,48,255
   59: 168,176,168,255
   60: 216,200,176,255
   61: 16,8,48,255
   62: 72,80,24,255
   63: 132,140,124,255
   64: 88,80,40,255
   65: 152,152,176,255
   66: 16,16,56,255
   67: 20,48,24,255
   68: 200,200,192,255
   69: 56,48,72,255
   70: 40,56,24,255
   71: 120,136,56,255
   72: 152,160,152,255
   73: 8,8,24,255
   74: 192,184,176,255
   75: 248,244,172,255
   76: 116,124,136,255
   77: 88,68,40,255
   78: 64,72,32,255
   79: 172,180,176,255
   80: 176,152,132,255
   81: 24,40,24,255
   82: 60,100,24,255
   83: 168,148,128,255
   84: 176,132,88,255
   85: 200,192,192,255
   86: 76,76,48,255
   87: 152,116,80,255
   88: 8,16,60,255
   89: 152,152,144,255
   90: 240,216,152,255
   91: 52,80,32,255
   92: 20,24,40,255
   93: 132,144,156,255
   94: 88,88,48,255
   95: 20,32,8,255
   96: 84,68,28,255
   97: 172,164,180,255
   98: 152,152,152,255
   99: 120,112,56,255
  100: 88,88,40,255
  101: 56,80,8,255
  102: 80,80,40,255
  103: 176,176,160,255
  104: 48,56,8,255
  105: 104,84,68,255
  106: 48,120,76,255
  107: 184,176,168,255
  108: 108,96,56,255
  109: 232,224,212,255
  110: 16,16,40,255
  111: 96,88,48,255
  112: 144,152,144,255
  113: 104,112,68,255
  114: 220,180,148,255
  115: 28,32,60,255
  116: 56,72,8,255
  117: 104,96,40,255
  118: 56,72,24,255
  119: 24,64,24,255
  120: 148,164,160,255
  121: 180,176,140,255
  122: 216,208,196,255
  123: 156,128,92,255
  124: 72,84,40,255
  125: 56,80,16,255
  126: 72,72,24,255
  127: 48,44,12,255
  128: 124,108,96,255
  129: 64,80,24,255
  130: 132,136,148,255
  131: 200,200,184,255
  132: 120,100,68,255
  133: 16,28,24,255
  134: 144,144,144,255
  135: 116,120,124,255
  136: 248,224,184,255
  137: 184,184,168,255
  138: 72,80,32,255
  139: 152,148,160,255
  140: 56,76,44,255
  141: 232,208,144,255
  142: 176,168,168,255
  143: 104,96,88,255
  144: 60,56,32,255
  145: 168,152,92,255
  146: 184,184,176,255
  147: 152,144,124,255
  148: 232,180,120,255
  149: 208,204,192,255
  150: 152,152,128,255
  151: 188,180,108,255
  152: 48,56,16,255
  153: 28,60,40,255
  154: 56,64,24,255
  155: 160,152,144,255
  156: 48,72,8,255
  157: 168,160,160,255
  158: 228,228,184,255
  159: 48,64,8,255
  160: 160,152,152,255
  161: 148,112,64,255
  162: 32,48,8,255
  163: 80,68,40,255
  164: 132,140,100,255
  165: 84,104,28,255
  166: 36,48,28,255
  167: 188,192,184,255
  168: 168,168,160,255
  169: 240,248,248,255
  170: 60,52,24,255
  171: 248,248,220,255
  172: 36,36,8,255
  173: 144,152,152,255
  174: 32,64,8,255
  175: 96,80,80,255
  176: 72,140,148,255
  177: 48,72,16,255
  178: 200,184,176,255
  179: 148,144,152,255
  180: 136,140,136,255
  181: 244,200,144,255
  182: 180,200,212,255
  183: 160,168,160,255
  184: 40,64,8,255
  185: 56,72,16,255
  186: 48,56,24,255
  187: 64,72,24,255
  188: 176,184,168,255
  189: 48,64,16,255
  190: 104,116,56,255
  191: 28,24,56,255
  192: 132,124,136,255
  193: 72,72,32,255
  194: 80,88,40,255
  195: 56,88,12,255
  196: 236,204,184,255
  197: 72,64,28,255
  198: 64,84,32,255
  199: 24,48,8,255
  200: 88,80,84,255
  201: 240,232,236,255
  202: 148,100,60,255
  203: 148,116,100,255
  204: 160,144,140,255
  205: 208,132,88,255
  206: 36,32,24,255
  207: 148,132,132,255
  208: 52,84,24,255
  209: 196,180,168,255
  210: 204,188,140,255
  211: 24,16,52,255
  212: 104,120,40,255
  213: 240,184,144,255
  214: 72,88,32,255
  215: 104,84,40,255
  216: 60,52,16,255
  217: 208,200,184,255
  218: 32,56,8,255
  219: 24,56,8,255
  220: 160,168,152,255
  221: 116,136,164,255
  222: 204,196,172,255
  223: 48,84,12,255
  224: 88,128,80,255
  225: 64,64,24,255
  226: 248,244,192,255
  227: 172,180,192,255
  228: 144,144,60,255
  229: 40,56,8,255
  230: 76,72,88,255
  231: 40,72,24,255
  232: 196,180,160,255
  233: 152,168,132,255
  234: 64,88,24,255
  235: 64,64,16,255
  236: 40,48,8,255
  237: 220,152,124,255
  238: 92,84,28,255
  239: 204,208,220,255
  240: 104,128,96,255
  241: 132,168,164,255
  242: 40,76,8,255
  243: 188,184,196,255
  244: 56,76,80,255
  245: 156,148,104,255
  246: 244,240,156,255
  247: 68,80,16,255
  248: 28,64,80,255
  249: 152,176,188,255
  250: 132,84,56,255
  251: 8,8,48,255
  252: 8,8,56,255
  253: 8,40,24,255
  254: 8,72,96,255
  255: 248,248,248,255

See how the GMT 6.4.0 version has 1 band with a Color Table, compared to 3 bands with GMT 6.5.0 release as in https://github.com/GenericMappingTools/pygmt/pull/2963/files#r1444072938.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, asked about the dataset changes upstream at GenericMappingTools/gmtserver-admin#257.

This was referenced Jan 7, 2024
Open
Closed
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 8, 2024 •
edited
Loading

Summary of changed images

This is an auto-generated report of images that have changed on the DVC remote

Status Path
modified pygmt/tests/baseline/test_tilemap_no_clip_False.png
modified pygmt/tests/baseline/test_tilemap_no_clip_True.png
modified pygmt/tests/baseline/test_tilemap_ogc_wgs84.png
modified pygmt/tests/baseline/test_tilemap_web_mercator.png

Image diff(s)

Added images

Modified images

Path Old New
test_tilemap_no_clip_False.png
test_tilemap_no_clip_True.png
test_tilemap_ogc_wgs84.png
test_tilemap_web_mercator.png

Report last updated at commit 2a41ddb

weiji14 added 2 commits January 8, 2024 13:25
@weiji14
Trigger re-caching of earth_day_01d_p file
98f30ac 
Modify pygmt/helpers/caching.py file slightly to create new cache.
@weiji14
Merge branch 'main' into fix/xr_image_fixture
64cd663
@seisman
Copy link
Member

seisman commented Jan 8, 2024

Rioxarray can now read the 3-bands from @earth_day_01d_p directly without needing to parse the colorinterp information from the GeoTIFF file.

What's the required rioxarray version?

weiji14 added 2 commits January 9, 2024 09:18
@weiji14
Merge branch 'main' into fix/xr_image_fixture
b80652f
@weiji14
Remove elif block for manually creating RGB image
b8c9c3c 
Downloading @earth_day_01d_p using GMT 6.4.0 actually works. The new earth_day_01d_p.tif file processed on 2023-09-29 doesn't have the quantized colormap, so no need for the extra handling. See GenericMappingTools/gmtserver-admin#257 for more info.
@weiji14
Copy link
Member Author

weiji14 commented Jan 8, 2024 •
edited
Loading

Rioxarray can now read the 3-bands from @earth_day_01d_p directly without needing to parse the colorinterp information from the GeoTIFF file.

What's the required rioxarray version?

Sorry, should have worded that differently. This should work with any rioxarray version. The change to the @earth_day_01d_p GeoTIFF file from the quantized/indexed version to a non-quantized/3-band version as mentioned at GenericMappingTools/gmtserver-admin#257 means that we don't need the complicated parsing logic (see also #2590 (comment)). This also means we can easily create a load_blue_marble or load_black_marble (#1442) later.

weiji14
weiji14 commented Jan 9, 2024
View reviewed changes
pygmt/tests/test_grdimage_image.py
Comment on lines 17 to +19
with rioxarray.open_rasterio(filename=geotiff) as rda:
if len(rda.band) == 1:
with rasterio.open(fp=geotiff) as src:
df_colormap = pd.DataFrame.from_dict(
data=src.colormap(1), orient="index"
)
array = src.read()

red = np.vectorize(df_colormap[0].get)(array)
green = np.vectorize(df_colormap[1].get)(array)
blue = np.vectorize(df_colormap[2].get)(array)
# alpha = np.vectorize(df_colormap[3].get)(array)

rda.data = red
da_red = rda.astype(dtype=np.uint8).copy()
rda.data = green
da_green = rda.astype(dtype=np.uint8).copy()
rda.data = blue
da_blue = rda.astype(dtype=np.uint8).copy()

xr_image = xr.concat(objs=[da_red, da_green, da_blue], dim="band")
if len(rda.band) == 3:
xr_image = rda.load()
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait for GenericMappingTools/gmtserver-admin#257, to see if the 3-band GeoTIFF file stays, or we revert back to the quantized/indexed GeoTIFF file from before Sep 2023.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gonna assume that the 3-band GeoTIFF will stay for now, so that we can move on with the GMT 6.5.0 update. Marked this PR as ready for review.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about splitting this PR into two separate RPs so if the upstream decide to to back to the indexed geotiff file, we can easily revert the changes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kinda want to take the code from this fixture and just make a load_blue_marble dataset function after this 🙂 If we do go back to the indexed GeoTIFF files, then the modifications would go in that load_blue_marble function.

@weiji14 weiji14 marked this pull request as ready for review January 11, 2024 02:58
@weiji14 weiji14 added the needs review This PR has higher priority and needs review. label Jan 11, 2024
@weiji14 weiji14 merged commit 5bfcbeb into main Jan 11, 2024
14 of 24 checks passed
@weiji14 weiji14 deleted the fix/xr_image_fixture branch January 11, 2024 04:52
@seisman seisman removed the needs review This PR has higher priority and needs review. label Jan 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@seisman seisman seisman approved these changes

Assignees

@weiji14 weiji14

Labels
maintenance Boring but important stuff for the core devs
Projects
None yet
Milestone
0.11.0
Development

Successfully merging this pull request may close these issues.
2 participants
@weiji14 @seisman