[CVE-2018-17439, HDFFV-10589] Stack buffer overflow in H5O_dtype_decode_helper()

[e4t]

Description
A malformed hdf5 file may cause a stack overflow in H5IMget_image_info() if the rank of the data space exceeds the array size allocated on the stack:

herr_t H5IMget_image_info( ...)
{ 
   ...
   hsize_t     dims[IMAGE24_RANK];
   ...
   if ( H5Sget_simple_extent_dims( sid, dims, NULL) < 0)
        goto out;
...
}

With:

H5Sget_simple_extent_dims(hid_t space_id, hsize_t dims[] /*out*/, hsize_t maxdims[] /*out*/) {
   int ret_value = -1;
   H5S_t *ds;
   ...
   ret_value = H5S_get_simple_extent_dims(ds, dims, maxdims);
   ...
}

int
H5S_get_simple_extent_dims(const H5S_t *ds, hsize_t dims[], hsize_t max_dims[])
{
   int ret_value = -1;
   if ((ret_value = H5S_extent_get_dims(&ds->extent, dims, max_dims)) < 0) {
       error
   }
...
}

int
H5S_extent_get_dims(const H5S_extent_t *ext, hsize_t dims[], hsize_t max_dims[])
{
 ...
    int ret_value = (int)ext->rank;
    for (i = 0; i < ret_value; i++) {
                if (dims)
                    dims[i] = ext->size[i];
      ...
    }
 ...
}

This is somewhat critical on systems with stacks growing downwards as the return address will be located above the base address of the array and thus is reachable by an out-of-bounds array index. This way, the return address can be manipulated which may open the door to arbitrary code execution.
Luckily, the fix is simple as a call to H5Sget_simple_extent_dims() with NULL arguments will simply return the rank which can then be checked against the array size.
On later versions of HDF5 (including 1.10.8) the reproducer of CVE-2018-17439 no longer triggers: it contains a dataspace rank that doesn't match its buffer size. Due to the fix for CVE-2018-14460 H5IMget_image_info() bails in H5Dopen2() already. AFAICT these issues are independent of each other: an image dataspace with rank > IMAGE24_RANK but the correct buffer size would not trigger the detection added for CVE-2018-14460 and be vulnerable to this exploit.

Platform

• 1.10.7 and earlier
• openSUSE Leap 15.4/ SLE 15 SP4 / openSUSE Tumbleweed
• gcc7
• Autotools
• config --enable-fortran --enable-unsupported --enable-hl --enable-shared --enable-threadsafe --enable-build-mode=production --enable-cxx --with-pthread

[derobins]

Definitely fixed in 1.14.3, 1.12.3, and 1.10.11.