GTiff: segmentation fault building overviews for Float32

[metzm]

Expected behavior and actual behavior.

Expected:
Creating a COG of datatype Float32 works, and building overviews for a GTiff of datatype Float32 works.

Problem:
Segmentation fault at https://github.com/OSGeo/gdal/blob/master/gdal/frmts/gtiff/libtiff/tif_dirwrite.c#L2064 because
tif->tif_dir.td_colormap[0] is NULL.

This segmentation fault occurs e.g. with COG when building overviews calling GTIFFBuildOverviewsEx().

Creating COG files of datatype Byte works just fine, there must be a problem with datatype Float32.

I am not familiar with the GDAL-internal libtiff. Any hints on the problem tif->tif_dir.td_colormap[0] for Float32 that could help me to produce a reasonable PR?

Steps to reproduce the problem.

1. create a GTiff with datatype Float32
2. use gdaladdo to add overviews

Alternative:
use gdal_translate -of COG with any raster dataset with datatype Float32

Original problem:
In GRASS GIS, using the North Carolina sample dataset:

g.region -p n=225500 s=218000 w=633000 e=642000 res=10
r.out.gdal -fmt input=elevation output=temp_elevation.tif format=COG

Operating system

Fedora 32, Ubuntu ?

GDAL version and provenance

GDAL 3.1.3 compiled from source, also other GDAL 3.1.? versions on other operating systems

[rouault]

Can you attach a sample input file demonstrating the issue ? I can't replicate that with regular Float32 TIFF files. I believe the root cause might be GRASS attaches a ColorMap to a Float32 TIFF which is quite unusual.

[metzm]

GRASS tries to attach a color map independent of the datatype unless the user suppresses writing color rules with r.out.gdal -c. So far GRASS relied on GDAL if color rules can be written to the requested output format and datatype. GDAL does not write color rules if not supported, right? r.out.gdal -c avoids the segfault, but the expected behaviour is that GDAL ignores color maps if not supported. I will try to construct a test file to replicate the issue.

[rouault]

GDAL does not write color rules if not supported, right?

it shouldn't. As far as I can see the GDAL GTiff driver only allows to create a color map for Byte and UInt16 data type, which makes sense (actually reading the TIFF6 spec, colormaps should only be used for 4-bit or 8-bit data...). But perhaps I'm missing a code path that would still allow that.
But doing something like "gdal_translate ../autotest/gcore/data/test_average_palette.tif test.tif -ot float32" raises a "Warning 1: Unable to export color table to GeoTIFF file. Color tables can only be written to 1 band or 2 bands Byte or UInt16 GeoTIFF files."

[metzm]

In contrast to the GTiff driver, the COG driver only supports GDALDriver::CreateCopy(), not GDALDriver::Create(). Therefore we use the MEM driver to create the dataset, then GDALCreateCopy() to copy from MEM to COG driver. Apparently at this step the color tables are also written to COG, there is no warning "Warning 1: Unable to export color table to GeoTIFF file. Color tables can only be written to 1 band or 2 bands Byte or UInt16 GeoTIFF files."
I do get this warning If I do the same with the GTiff driver (create a MEM dataset, then GDALCreateCopy() to get the final GTiff dataset.
Creating a GTiff dataset directly with Float32 raises as expected the (harmless) error "SetColorTable() only supported for Byte or UInt16 bands in TIFF format."
Using gdal_translate to create a COG file with e.g. gdal_translate ../autotest/gcore/data/test_average_palette.tif test.tif -ot float32 -of COG" also works, with the above warning.

[rouault]

Minimum reproducer found with your above indications

from osgeo import gdal
src_ds = gdal.GetDriverByName('MEM').Create('', 1000,1000,1,gdal.GDT_Float32) 
ct = gdal.ColorTable()
src_ds.GetRasterBand(1).SetColorTable(ct)
gdal.GetDriverByName('COG').CreateCopy('/vsimem/my.cog.tif', src_ds) # segfault