Merge remote-tracking branch 'origin/develop' into pause_errors_3

README.md

@@ -90,10 +90,12 @@ are tentative.
 
 | Release | New Features |
 | ------- | ------------ |
-| 1.14.4 | native float16 support, cloud-optimized HDF5 improvements |
-| 1.14.5 | Complex number support |
-| TBD | VFD SWMR |
-| 2.0.0 | TBD |
+| 1.14.5 | oss-fuzz fixes, ros3 VFD improvements |
+| 1.14.6 | Last maintenance release of 1.14 |
+| 1.16.0 | Complex number support, updated library defaults (cache sizes, etc.) |
+| 2.0.0 | Multi-threaded HDF5, crashproofing / metadata journaling, Full (VFD) SWMR, encryption, digital signatures, semantic versioning |
+
+Some HDF5 2.0.0 features listed here may be released in a 1.16.x release.
 
 This list of feature release versions is also tentative, and the specific release
 in which a feature is introduced may change.

doc/img/release-schedule.plantuml

@@ -23,17 +23,26 @@ Project starts 2023-01-01
 [1.12.3] happens 2023-11-30
 [1.12] is colored in #88CCEE
 
-[1.14] starts at 2023-01-01 and lasts 104 weeks
+[1.14] starts at 2023-01-01 and lasts 122 weeks
 [1.14.1] happens at 2023-04-30
 [1.14.2] happens at 2023-08-31
 [1.14.3] happens at 2023-10-31
-[1.14.4] happens at 2024-03-31
+[1.14.4.2] happens at 2024-04-15
+[1.14.4.3] happens at 2024-05-22
 [1.14.5] happens at 2024-09-30
+[1.14.6] happens at 2025-04-30
 [1.14.1] displays on same row as [1.14.1]
 [1.14.2] displays on same row as [1.14.1]
 [1.14.3] displays on same row as [1.14.1]
-[1.14.4] displays on same row as [1.14.1]
+[1.14.4.2] displays on same row as [1.14.1]
 [1.14.5] displays on same row as [1.14.1]
+[1.14.6] displays on same row as [1.14.1]
 [1.14] is colored in #B187CF
 
+[1.16] starts at 2025-04-30 and lasts 35 weeks
+[1.16.0] happens at 2025-04-30
+[1.16.1] happens at 2025-09-30
+[1.16.1] displays on same row as [1.16.0]
+[1.16] is colored in #02BFA0
+
 @endgantt

doxygen/dox/VOLConnGuide.dox

@@ -664,7 +664,11 @@ structure. It is the opposite of the <em>to_str</em> callback.
 
 \subsection subsecVOLRefWrap Object Wrap Callbacks
 The object wrap callbacks are used by passthrough connectors to wrap/unwrap objects and contexts when
-passing them up and down the VOL chain.
+passing them up and down the VOL chain. Each passthrough VOL must define an object wrapping structure and a wrap context.
+The object wrapping structure should contain the information necessary to recursively unwrap the object - at a minimum, this is the object provided from and the ID of the next connector in the stack.
+The wrap context should contain the information necessary to recursively wrap the object - at a minimum, this is the ID and the wrap context of the next VOL connector in the stack.
+
+Each callback should use <em>H5VL&lt;callback&gt;</em> to recursively invoke the same callback in all lower passthrough VOl connectors.
 
 <em>Wrap class for object wrapping routines, H5VLconnector.h</em>
 \code
@@ -678,7 +682,13 @@ typedef struct H5VL_wrap_class_t {
 \endcode
 
 \subsubsection subsubsecVOLRefWrapobj wrap: get_object
-Retrieves an underlying object.
+Retrieves the underlying object from a wrapped object. Should return a pointer to the underlying object belonging to the terminal VOL connector.
+
+This will generally be done by unwrapping this VOL's object wrapping structure, before recursively calling <em>H5VLget_object</em> to invoke the <em>get_object</em>
+callback for all lower VOL connectors which define it. <em>H5VLget_object</em> requires the object returned by the next VOL and the next VOL's ID. Both of 
+these fields should be stored by the VOL somehow, generally on the wrapped object structure.
+
+This callback should not cleanup or modify the provided object wrapping structure.
 <table>
 <tr>
 <th>Signature:</th>
@@ -696,14 +706,17 @@ Retrieves an underlying object.
 <tr>
 <td>
 \code
-  obj  (IN): Object being unwrapped.
+  obj  (IN): The object to be unwrapped.
 \endcode
 </td>
 </tr>
 </table>
 
 \subsubsection subsubsecVOLRefWrapctx wrap: get_wrap_ctx
 Get a VOL connector's object wrapping context.
+
+The context should be returned in dynamically allocated memory under <em>*wrap_ctx</em>.
+Any resources this callback allocates should be freed within <em>free_wrap_ctx</em>.
 <table>
 <tr>
 <th>Signature:</th>
@@ -721,15 +734,19 @@ Get a VOL connector's object wrapping context.
 <tr>
 <td>
 \code
-  obj      (IN):  Object for which we need a context.
+  obj      (IN):  Object wrapped by this VOL connector, for which we need a context.
   wrap_ctx (OUT): Context.
 \endcode
 </td>
 </tr>
 </table>
 
 \subsubsection subsubsecVOLRefWrapwrap wrap: wrap_object
-Asks a connector to wrap an underlying object.
+Asks a connector to wrap an underlying object. This callback should use <em>H5VLwrap_object</em> 
+to recursively have the object wrapped by all lower VOL connectors before performing its own wrapping.
+
+The wrapped object should provide the information necessary for <em>unwrap_object</em> to recursively 
+unwrap it - at a minimum, the object provided from and the ID of the next VOL connector.
 <table>
 <tr>
 <th>Signature:</th>
@@ -747,7 +764,7 @@ Asks a connector to wrap an underlying object.
 <tr>
 <td>
 \code
-  obj       (IN): Object being wrapped.
+  obj       (IN): The object to be wrapped.
   obj_type  (IN): Object type (see H5Ipublic.h).
   wrap_ctx  (IN): Context.
 \endcode
@@ -756,7 +773,10 @@ Asks a connector to wrap an underlying object.
 </table>
 
 \subsubsection subsubsecVOLRefWrapunwrap wrap: unwrap_object
-Unwrap an object from connector.
+Unwrap an object from connector. Any resources allocated during <em>wrap_object</em> should be released and cleaned up here.
+
+This callback should clean up this VOL's object wrapping structure before recursively invoking <em>H5VLunwrap_object</em>.
+
 <table>
 <tr>
 <th>Signature:</th>
@@ -774,14 +794,15 @@ Unwrap an object from connector.
 <tr>
 <td>
 \code
-  obj  (IN): Object being unwrapped.
+  obj  (IN): Object to be unwrapped.
 \endcode
 </td>
 </tr>
 </table>
 
 \subsubsection subsubsecVOLRefWrapfree wrap: free_wrap_ctx
-Release a VOL connector's object wrapping context.
+Release a VOL connector's object wrapping context. This should free any resources allocated during <em>get_wrap_ctx</em>, and recursively invoke <em>H5VLfree_wrap_ctx</em>
+to execute the free callback for the lower VOL connectors in the stack.
 <table>
 <tr>
 <th>Signature:</th>

fortran/src/H5_f.c

@@ -539,6 +539,7 @@ h5init_flags_c(int_f *h5d_flags, size_t_f *h5d_size_flags, int_f *h5e_flags, hid
     h5f_flags[27] = (int_f)H5F_LIBVER_V110;
     h5f_flags[28] = (int_f)H5F_LIBVER_V112;
     h5f_flags[29] = (int_f)H5F_LIBVER_V114;
+    h5f_flags[30] = (int_f)H5F_LIBVER_V116;
 
     /*
      *  H5FD flags

fortran/src/H5_ff.F90

@@ -56,7 +56,7 @@ MODULE H5LIB
   !
   ! H5F flags declaration
   !
-  INTEGER, PARAMETER :: H5F_FLAGS_LEN = 30
+  INTEGER, PARAMETER :: H5F_FLAGS_LEN = 31
   INTEGER, DIMENSION(1:H5F_FLAGS_LEN) :: H5F_flags
   !
   ! H5generic flags declaration
@@ -379,6 +379,7 @@ END FUNCTION h5init1_flags_c
     H5F_LIBVER_V110_F              = H5F_flags(28)
     H5F_LIBVER_V112_F              = H5F_flags(29)
     H5F_LIBVER_V114_F              = H5F_flags(30)
+    H5F_LIBVER_V116_F              = H5F_flags(31)
     !
     ! H5generic flags
     !

fortran/src/H5f90global.F90

@@ -236,6 +236,7 @@ MODULE H5GLOBAL
   !DEC$ATTRIBUTES DLLEXPORT :: H5F_LIBVER_V110_F
   !DEC$ATTRIBUTES DLLEXPORT :: H5F_LIBVER_V112_F
   !DEC$ATTRIBUTES DLLEXPORT :: H5F_LIBVER_V114_F
+  !DEC$ATTRIBUTES DLLEXPORT :: H5F_LIBVER_V116_F
   !DEC$ATTRIBUTES DLLEXPORT :: H5F_FSPACE_STRATEGY_FSM_AGGR_F
   !DEC$ATTRIBUTES DLLEXPORT :: H5F_FSPACE_STRATEGY_PAGE_F
   !DEC$ATTRIBUTES DLLEXPORT :: H5F_FSPACE_STRATEGY_AGGR_F
@@ -269,6 +270,7 @@ MODULE H5GLOBAL
   INTEGER :: H5F_LIBVER_V110_F     !< H5F_LIBVER_V110
   INTEGER :: H5F_LIBVER_V112_F     !< H5F_LIBVER_V112
   INTEGER :: H5F_LIBVER_V114_F     !< H5F_LIBVER_V114
+  INTEGER :: H5F_LIBVER_V116_F     !< H5F_LIBVER_V116
   INTEGER :: H5F_FSPACE_STRATEGY_FSM_AGGR_F !< H5F_FSPACE_STRATEGY_FSM_AGGR
   INTEGER :: H5F_FSPACE_STRATEGY_PAGE_F     !< H5F_FSPACE_STRATEGY_PAGE
   INTEGER :: H5F_FSPACE_STRATEGY_AGGR_F     !< H5F_FSPACE_STRATEGY_AGGR

fortran/test/tH5P_F03.F90

@@ -228,6 +228,13 @@ SUBROUTINE test_create(total_error)
   CALL VERIFY("***ERROR: Returned wrong low libver_bounds", low, H5F_LIBVER_V114_F, total_error)
   CALL VERIFY("***ERROR: Returned wrong high libver_bounds", high, H5F_LIBVER_V114_F, total_error)
 
+  CALL h5pset_libver_bounds_f(fapl, H5F_LIBVER_V116_F, H5F_LIBVER_V116_F, error)
+  CALL check("h5pset_libver_bounds_f",error, total_error)
+  CALL h5pget_libver_bounds_f(fapl, low, high, error)
+  CALL check("h5pget_libver_bounds_f",error, total_error)
+  CALL VERIFY("***ERROR: Returned wrong low libver_bounds", low, H5F_LIBVER_V116_F, total_error)
+  CALL VERIFY("***ERROR: Returned wrong high libver_bounds", high, H5F_LIBVER_V116_F, total_error)
+
   CALL H5Pset_libver_bounds_f(fapl, H5F_LIBVER_LATEST_F, H5F_LIBVER_LATEST_F, error)
   CALL check("H5Pset_libver_bounds_f",error, total_error)
   CALL h5pget_libver_bounds_f(fapl, low, high, error)

src/H5Aint.c

@@ -95,6 +95,7 @@ const unsigned H5O_attr_ver_bounds[] = {
     H5O_ATTR_VERSION_3,     /* H5F_LIBVER_V110 */
     H5O_ATTR_VERSION_3,     /* H5F_LIBVER_V112 */
     H5O_ATTR_VERSION_3,     /* H5F_LIBVER_V114 */
+    H5O_ATTR_VERSION_3,     /* H5F_LIBVER_V116 */
     H5O_ATTR_VERSION_LATEST /* H5F_LIBVER_LATEST */
 };
 

src/H5Dlayout.c

@@ -48,6 +48,7 @@ const unsigned H5O_layout_ver_bounds[] = {
     H5O_LAYOUT_VERSION_4,     /* H5F_LIBVER_V110 */
     H5O_LAYOUT_VERSION_4,     /* H5F_LIBVER_V112 */
     H5O_LAYOUT_VERSION_4,     /* H5F_LIBVER_V114 */
+    H5O_LAYOUT_VERSION_4,     /* H5F_LIBVER_V116 */
     H5O_LAYOUT_VERSION_LATEST /* H5F_LIBVER_LATEST */
 };
 

src/H5Dpkg.h

@@ -633,7 +633,7 @@ H5_DLLVAR const H5B2_class_t H5D_BT2[1];
 H5_DLLVAR const H5B2_class_t H5D_BT2_FILT[1];
 
 /*  Array of versions for Layout */
-H5_DLLVAR const unsigned H5O_layout_ver_bounds[H5F_LIBVER_NBOUNDS];
+H5_DLLVAR const unsigned H5O_layout_ver_bounds[H5F_LIBVER_NBOUNDS + 1];
 
 /******************************/
 /* Package Private Prototypes */

src/H5Fsuper.c

@@ -71,6 +71,7 @@ static const unsigned HDF5_superblock_ver_bounds[] = {
     HDF5_SUPERBLOCK_VERSION_3,     /* H5F_LIBVER_V110 */
     HDF5_SUPERBLOCK_VERSION_3,     /* H5F_LIBVER_V112 */
     HDF5_SUPERBLOCK_VERSION_3,     /* H5F_LIBVER_V114 */
+    HDF5_SUPERBLOCK_VERSION_3,     /* H5F_LIBVER_V116 */
     HDF5_SUPERBLOCK_VERSION_LATEST /* H5F_LIBVER_LATEST */
 };
 

src/H5Ofill.c

@@ -153,6 +153,7 @@ const unsigned H5O_fill_ver_bounds[] = {
     H5O_FILL_VERSION_3,     /* H5F_LIBVER_V110 */
     H5O_FILL_VERSION_3,     /* H5F_LIBVER_V112 */
     H5O_FILL_VERSION_3,     /* H5F_LIBVER_V114 */
+    H5O_FILL_VERSION_3,     /* H5F_LIBVER_V116 */
     H5O_FILL_VERSION_LATEST /* H5F_LIBVER_LATEST */
 };
 

src/H5Ofsinfo.c

@@ -69,6 +69,7 @@ static const unsigned H5O_fsinfo_ver_bounds[] = {
     H5O_FSINFO_VERSION_1,     /* H5F_LIBVER_V110 */
     H5O_FSINFO_VERSION_1,     /* H5F_LIBVER_V112 */
     H5O_FSINFO_VERSION_1,     /* H5F_LIBVER_V114 */
+    H5O_FSINFO_VERSION_1,     /* H5F_LIBVER_V116 */
     H5O_FSINFO_VERSION_LATEST /* H5F_LIBVER_LATEST */
 };
 #define N_FSINFO_VERSION_BOUNDS H5F_LIBVER_NBOUNDS

src/H5Oint.c

@@ -127,6 +127,7 @@ const unsigned H5O_obj_ver_bounds[] = {
     H5O_VERSION_2,     /* H5F_LIBVER_V110 */
     H5O_VERSION_2,     /* H5F_LIBVER_V112 */
     H5O_VERSION_2,     /* H5F_LIBVER_V114 */
+    H5O_VERSION_2,     /* H5F_LIBVER_V116 */
     H5O_VERSION_LATEST /* H5F_LIBVER_LATEST */
 };
 

src/H5Opkg.h

@@ -626,10 +626,10 @@ H5_DLL herr_t H5O__attr_count_real(H5F_t *f, H5O_t *oh, hsize_t *nattrs);
  * Object header, Attribute/Fill value/Filter pipeline messages
  */
 /* Layout/Datatype/Dataspace arrays of versions are in H5Dpkg.h, H5Tpkg.h and H5Spkg.h */
-H5_DLLVAR const unsigned H5O_obj_ver_bounds[H5F_LIBVER_NBOUNDS];
-H5_DLLVAR const unsigned H5O_attr_ver_bounds[H5F_LIBVER_NBOUNDS];
-H5_DLLVAR const unsigned H5O_fill_ver_bounds[H5F_LIBVER_NBOUNDS];
-H5_DLLVAR const unsigned H5O_pline_ver_bounds[H5F_LIBVER_NBOUNDS];
+H5_DLLVAR const unsigned H5O_obj_ver_bounds[H5F_LIBVER_NBOUNDS + 1];
+H5_DLLVAR const unsigned H5O_attr_ver_bounds[H5F_LIBVER_NBOUNDS + 1];
+H5_DLLVAR const unsigned H5O_fill_ver_bounds[H5F_LIBVER_NBOUNDS + 1];
+H5_DLLVAR const unsigned H5O_pline_ver_bounds[H5F_LIBVER_NBOUNDS + 1];
 
 /* Testing functions */
 #ifdef H5O_TESTING

src/H5Opline.c

@@ -88,6 +88,7 @@ const unsigned H5O_pline_ver_bounds[] = {
     H5O_PLINE_VERSION_2,     /* H5F_LIBVER_V110 */
     H5O_PLINE_VERSION_2,     /* H5F_LIBVER_V112 */
     H5O_PLINE_VERSION_2,     /* H5F_LIBVER_V114 */
+    H5O_PLINE_VERSION_2,     /* H5F_LIBVER_V116 */
     H5O_PLINE_VERSION_LATEST /* H5F_LIBVER_LATEST */
 };
 

src/H5S.c

@@ -61,6 +61,7 @@ const unsigned H5O_sdspace_ver_bounds[] = {
     H5O_SDSPACE_VERSION_2,     /* H5F_LIBVER_V110 */
     H5O_SDSPACE_VERSION_2,     /* H5F_LIBVER_V112 */
     H5O_SDSPACE_VERSION_2,     /* H5F_LIBVER_V114 */
+    H5O_SDSPACE_VERSION_2,     /* H5F_LIBVER_V116 */
     H5O_SDSPACE_VERSION_LATEST /* H5F_LIBVER_LATEST */
 };
 

src/H5Shyper.c

@@ -235,12 +235,13 @@ const H5S_select_class_t H5S_sel_hyper[1] = {{
 }};
 
 /* Format version bounds for dataspace hyperslab selection */
-const unsigned H5O_sds_hyper_ver_bounds[] = {
+static const unsigned H5O_sds_hyper_ver_bounds[] = {
     H5S_HYPER_VERSION_1, /* H5F_LIBVER_EARLIEST */
     H5S_HYPER_VERSION_1, /* H5F_LIBVER_V18 */
     H5S_HYPER_VERSION_2, /* H5F_LIBVER_V110 */
     H5S_HYPER_VERSION_3, /* H5F_LIBVER_V112 */
     H5S_HYPER_VERSION_3, /* H5F_LIBVER_V114 */
+    H5S_HYPER_VERSION_3, /* H5F_LIBVER_V116 */
     H5S_HYPER_VERSION_3  /* H5F_LIBVER_LATEST */
 };
 

src/H5Spkg.h

@@ -382,7 +382,7 @@ H5_DLLVAR const H5S_select_class_t H5S_sel_none[1];
 H5_DLLVAR const H5S_select_class_t H5S_sel_point[1];
 
 /* Array of versions for Dataspace */
-H5_DLLVAR const unsigned H5O_sdspace_ver_bounds[H5F_LIBVER_NBOUNDS];
+H5_DLLVAR const unsigned H5O_sdspace_ver_bounds[H5F_LIBVER_NBOUNDS + 1];
 
 /* Extent functions */
 H5_DLL herr_t H5S__extent_release(H5S_extent_t *extent);

src/H5Spoint.c

@@ -120,12 +120,13 @@ const H5S_select_class_t H5S_sel_point[1] = {{
 }};
 
 /* Format version bounds for dataspace point selection */
-const unsigned H5O_sds_point_ver_bounds[] = {
+static const unsigned H5O_sds_point_ver_bounds[] = {
     H5S_POINT_VERSION_1, /* H5F_LIBVER_EARLIEST */
     H5S_POINT_VERSION_1, /* H5F_LIBVER_V18 */
     H5S_POINT_VERSION_1, /* H5F_LIBVER_V110 */
     H5S_POINT_VERSION_2, /* H5F_LIBVER_V112 */
     H5S_POINT_VERSION_2, /* H5F_LIBVER_V114 */
+    H5S_POINT_VERSION_2, /* H5F_LIBVER_V116 */
     H5S_POINT_VERSION_2  /* H5F_LIBVER_LATEST */
 };
 

src/H5T.c

@@ -609,6 +609,7 @@ const unsigned H5O_dtype_ver_bounds[] = {
     H5O_DTYPE_VERSION_3,     /* H5F_LIBVER_V110 */
     H5O_DTYPE_VERSION_4,     /* H5F_LIBVER_V112 */
     H5O_DTYPE_VERSION_4,     /* H5F_LIBVER_V114 */
+    H5O_DTYPE_VERSION_4,     /* H5F_LIBVER_V116 */
     H5O_DTYPE_VERSION_LATEST /* H5F_LIBVER_LATEST */
 };
 

src/H5Tpkg.h

@@ -344,7 +344,7 @@ typedef struct {
 typedef herr_t (*H5T_operator_t)(H5T_t *dt, void *op_data /*in,out*/);
 
 /*  Array of versions for Datatype */
-H5_DLLVAR const unsigned H5O_dtype_ver_bounds[H5F_LIBVER_NBOUNDS];
+H5_DLLVAR const unsigned H5O_dtype_ver_bounds[H5F_LIBVER_NBOUNDS + 1];
 
 /*
  * Alignment constraints for HDF5 types.  Accessing objects of these

src/H5VLcallback.c

@@ -739,7 +739,10 @@ H5VL_get_wrap_ctx(const H5VL_class_t *connector, void *obj, void **wrap_ctx)
 /*---------------------------------------------------------------------------
  * Function:    H5VLget_wrap_ctx
  *
- * Purpose:     Get a VOL connector's object wrapping context
+ * Purpose:     Get a VOL connector's object wrapping context. The output
+ *              wrap context is stored in memory allocated by the VOL callback
+ *              under *wrap_ctx and must be freed by the caller through
+ *              H5VLfree_wrap_ctx().
  *
  * Return:      Success:    Non-negative
  *              Failure:    Negative

src/H5VLconnector.h

@@ -845,7 +845,8 @@ typedef struct H5VL_info_class_t {
 } H5VL_info_class_t;
 
 /* VOL object wrap / retrieval callbacks */
-/* (These only need to be implemented by "pass through" VOL connectors) */
+/* (These must be implemented by "pass through" VOL connectors, and should not be implemented by terminal VOL
+ * connectors) */
 typedef struct H5VL_wrap_class_t {
     void *(*get_object)(const void *obj); /* Callback to retrieve underlying object       */
     herr_t (*get_wrap_ctx)(

src/H5VLint.c

@@ -55,7 +55,16 @@
 /* Local Typedefs */
 /******************/
 
-/* Object wrapping context info */
+/* Object wrapping context info for passthrough VOL connectors.
+ * Passthrough VOL connectors must wrap objects returned from lower level(s) of the VOL connector stack
+ * so that they may be passed back up the stack to the library, and must unwrap objects
+ * passed down the stack before providing them to the next lower VOL connector.
+ * This is generally done individually within each VOL object callback. However, the library sometimes
+ * needs to wrap objects from places that don't pass through the VOL layer.
+ * In this case, the wrap callbacks defined in H5VL_wrap_class_t are used, and the VOL-defined wrap context
+ * (obj_wrap_ctx) provides necessary information - at a minimum, the object originally returned by the lower
+ * VOL connector, and the ID of the next VOL connector.
+ */
 typedef struct H5VL_wrap_ctx_t {
     unsigned rc;           /* Ref. count for the # of times the context was set / reset */
     H5VL_t  *connector;    /* VOL connector for "outermost" class to start wrap */