Hdf5 merge issue 488 v110 (#535)

* close #195. (#196) * Update HDF5PluginMacros.cmake * Update HDF5PluginMacros.cmake * Modify temporary rpath for testing in java example scripts. (#230) * Fix undefined left shifting of negative numbers (#338) Undefined Bahavior Sanitizer errored here about left shifting negative numbers. * Update license url (#332) * Modify temporary rpath for testing in java example scripts. * Update URL in source file Copyright headers for web copy of COPYING file - src and test directories. * Cleans up a couple of MSVC warnings in testhdf5 (#475) * Fixes a few testhdf5 warnings raised in Visual Studio Visual Studio is grumpier about treating pointers like integers than gcc. * Committing clang-format changes Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Fix typos and grammar errors. (#476) * Suppresses the tcheck_version test's abort dialog on Windows (#477) * Suppresses the tcheck_version test's abort dialog on Windows Windows raises a modal abort/retry/ignore dialog box when CRT calls abort(). This change installs a report hook that suppresses the dialog so that the CMake tests don't time out waiting for a nonexistent user to click a dialog box. * Committing clang-format changes * Removes __cdecl from callback Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Purge the buffer used in type conversion. (#263) Some of the uniniitialized bits in the buffer may get carried through all the way to disk, creating a risk for leaks. We observed an msan error during the floating point output conversion. Due to the encoding certain bits could remain untouched during the conversion. In this draft we zero initialize the dbuf used by every convertor. * Fixed HDFFV-10480 (CVE-2018-11206) and HDFFV-11159 (CVE-2018-14033) (#405) * Fixed HDFFV-10480 (CVE-2018-11206) and HDFFV-11159 (CVE-2018-14033) Description Checked against buffer size to prevent segfault, in case of data corruption. + HDFFV-11159 CVE-2018-14033 Buffer over-read in H5O_layout_decode + HDFFV-10480 CVE-2018-11206 Buffer over-read in H5O_fill_new[/old]_decode Platforms tested: Linux/64 (jelly) * Accidentally left in another occurrence of the previous patch from user after a more correct fix was applied, that is the check now accounted for the previous advance of the buffer pointer. Removed it. * Typo * Fixed format issues. * Added test. * Changed arguments to ADD_H5_TEST * Fixing arguments to ADD_H5_TEST again. * Fixing arguments again. * Took out the CMake changes until Allen can help. * Added files: tCVE_2018_11206_fill_old.h5 tCVE_2018_11206_fill_new.h5 * Revert "Took out the CMake changes until Allen can help." This reverts commit c21324d. * Revert "Fixing arguments again." This reverts commit 5832a70. * Revert "Fixing arguments to ADD_H5_TEST again." This reverts commit b45de82. * Revert "Changed arguments to ADD_H5_TEST" This reverts commit 1671982. * Added first argument to ADD_H5_TEST for HDFFV-10480 fix. * Changed argument 0 to 1 * Revert "Changed argument 0 to 1" This reverts commit b343d66. * Revert "Added first argument to ADD_H5_TEST for HDFFV-10480 fix." This reverts commit b8a0f9a. * Added first argument and corrected the second. * Updated fixes for HDFFV-10480 and HDFFV-11159/HDFFV-11049 * Improved error messages. * Added description of the current HDF5 branches; added a draft of contribution policy. (#445) * Added description of the current HDF5 branches. * Removed capitalization in from Develop, Release, Feature to reflect the real naming schema * Added a draft of contributions guidance document. * Fixed typos. * Fixed section title. * Fixed typo. * Fixed typos and formatting. * Fixed many typos and simplified the text (e.g., removed testing instructions and left pointers to where to find them, etc.) * Fixed a typo. Please enter the commit message for your changes. Lines starting * Added contributing.md file and rearranged doc entries in alphabetical order. * ddressed Gerd's review comments; found and fixed more typos. * Addressed comments from Larry and Scot. * Fix CMake error message location. (#478) Print error message if Perl is not found. * Committing clang-format changes * Update MANIFEST. Co-authored-by: H. Joe Lee <[email protected]> Co-authored-by: Sean McBride <[email protected]> Co-authored-by: Dana Robinson <[email protected]> Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Yu Feng <[email protected]> Co-authored-by: bmribler <[email protected]> Co-authored-by: epourmal <[email protected]>
HDFGroup · Apr 2, 2021 · 5f72e0b · 5f72e0b
1 parent 87d1813
commit 5f72e0b
Show file tree

Hide file tree

Showing 11 changed files with 211 additions and 17 deletions.
diff --git a/MANIFEST b/MANIFEST
@@ -203,6 +203,9 @@
 
 ./config/site-specific/BlankForm
 
+./doc/branches-explained.md
+./doc/contributing.md
+
 ./doxygen/aliases
 ./doxygen/Doxyfile.in
 ./doxygen/dox/api-compat-macros.dox
@@ -2052,6 +2055,8 @@
 ./tools/testfiles/twithddl.exp
 ./tools/testfiles/twithddlfile.ddl
 ./tools/testfiles/twithddlfile.exp
+./tools/testfiles/tCVE_2018_11206_fill_old.h5
+./tools/testfiles/tCVE_2018_11206_fill_new.h5
 
 # h5dump test error files
 ./tools/test/h5dump/errfiles/filter_fail.err

diff --git a/doc/branches-explained.md b/doc/branches-explained.md
@@ -0,0 +1,41 @@
+# HDF5 Git Branching Model Explained
+
+This document describes current HDF5 branches. 
+
+Branches are tested nightly and testing results are available at https://cdash-internal.hdfgroup.org/ and https://cdash.hdfgroup.org/. 
+Commits that break daily testing should be fixed by 3:00 pm Central time or reverted.  
+We encourage code contributors to check the status of their commits. If you have any questions, please contact [email protected].
+
+## `develop`
+Develop is the main branch whose source code always reflects a state with the latest delivered development changes for the next major release of HDF5. 
+This is also considered the integration branch, as **all** new features are integrated into this branch from respective feature branches. 
+
+## `Maintenance branches`
+
+Each currently supported release-line of HDF5 (e.g. 1.8.x, 1.10.x, 1.12.x) has a support branch with the name 1_8, 1_10, 1_12. 
+Maintenance branches are similar to the develop branch, except the source code in a maintenance branch always reflects a state 
+with the latest delivered development changes for the next **maintenance** release of that particular supported release-line of HDF5. 
+**Some** new features will be integrated into a release maintenance branch, depending on whether or not those features can be 
+introduced in minor releases.  Maintenance branches are removed when a release-line is retired from support.
+
+## `feature/*`
+Feature branches are temporary branches used to develop new features in HDF5.
+Feature branches branch off of develop and exist as long as the feature is under development. 
+When the feature is complete, the branch is merged back into develop, as well as into any support branches in which the change will be included, and then the feature branch is removed.
+
+## `release/*`
+Release branches are used to prepare a new production release. They are primarily used to allow for last minute dotting of i's and crossing of t's 
+(things like setting the release version, finalizing release notes, et cetera) and do not include new development. 
+They are created from the maintenance branch at the time of the maintenance release and have 
+names 1_8_N, 1_10_N, 1_12_N, where N is the minor release number. Once the release is  done it is tagged. 
+Patches can be applied to the release branch for patch releases that are treated as "scaled down" maintenance releases as defined by Release coordinator.
+
+## `1.X/master/*` where X is 8, 10 or 12
+These branches are used to tag 1.X.* maintenance releases.
+
+## `inactive/<name>/*`
+These branches are for experimental features that were developed in the past  and have not been merged to develop, and are not under active development. The features 
+can be out of sync with the develop branch.
+
+This document was last updated on March 16, 2021
+
diff --git a/doc/contributing.md b/doc/contributing.md
@@ -0,0 +1,87 @@
+# How to contribute to HDF5 (Draft)
+
+The HDF Group encourages community members to contribute to the HDF5 project. We accept and are very grateful for any type of contributions 
+from small typos and bug fixes to new features. The HDF Group is committed to work with the code contributors and make contribution process simple and enjoyable.
+
+This document describes guiding principles for the HDF5 code contributors and does not pretend to address any possible 
+contribution. If in doubt, please do not hesitate to ask us for guidance. 
+***Note that no contribution may be accepted unless the donor agrees with the HDF Group software license terms
+found in the COPYING file in the top source directory of every branch.***
+
+
+> We will assume that you are familiar with `git` and `GitHub`.  If not, you may go through the GitHub tutorial found at [https://guides.github.com/activities/hello-world/](https://guides.github.com/activities/hello-world/).  This tutorial should only take around 10 minutes.
+
+## Table of Contents
+
+* [Workflow](#workflow)
+* [Acceptance criteria for pull request](#criteria)
+* [Check List](#checklist)
+
+# Workflow <A NAME="workflow"></A>
+
+The process for contributing code to HDF5 is as follows:
+
+* Open an issue on [HDF5 GitHub](https://github.com/HDFGroup/hdf5/issues).
+
+> This step is ***required*** unless the change is minor (e.g., typo fix). 
+
+* Fork the [HDF5](https://github.com/HDFGroup/hdf5) repository.
+* Make the desired changes to the HDF5 software.
+	* New features should always go to develop branch first and later should be merged to the appropriate maintenance branches.
+	* Bug fixes should go to all appropriate branches (develop and maintenance). 
+* Build and test your changes. Detailed instructions on how to build and test HDF5 can be found in the `INSTALL*` files in the `release_docs` directory.
+* Push your changes to GitHub.
+* Issue a pull request and address any code formatting and testing issues reported.
+
+Once a pull request is correctly formatted and passes **ALL** CI tests, it will be reviewed and evaluated by The HDF Group developers and HDF5 community members who can approve pull requests..
+The HDF Group developers will work with you to assure that the pull request satisfies acceptance criteria described in the next section. 
+
+# Acceptance criteria for pull request <A NAME="criteria"></A>
+
+We appreciate every contribution we receive, but we may not accept them all.  Those that we *do* accept satisfy the following criteria:
+
+* **The pull request has a clear purpose** - What does the pull request address? How does it benefit the HDF5 community? 
+If the pull request does not have a clear purpose and benefits it will not be accepted. 
+
+* **The pull request is documented** - The HDF5 developers must understand not only *what* a change is doing, but *how* it is doing it.
+  Documenting the code makes it easier for us to understand your patch and will help to maintaine the code in the future. 
+
+* **The pull request passes HDF5 regression testing** - Any issue fixed or functionality added should be accompanied by the corresponding 
+tests and pass HDF5 regression testing run by The HDF Group. We do not expect you to perform comprehensive testing across multiple platforms 
+before we accept the pull request. If the pull request does not pass regression testing after the merge, The HDF Group developers will i
+work with you on the fixes. 
+
+* **The pull request does not compromise the principles behind HDF5** - HDF5 has a 100% commitment to backward compatibility.  
+	* Any file ever created with HDF5 must be readable by any future version of HDF5.
+   If the purpose of your patch  is to modify HDF5 data model or file format,
+ **please** discuss this with us first. File format changes and features required by those changes can be introduced only in a new major release. 
+	* HDF5 has a commitment to remaining *machine-independent*; data created on one platform/environment/architecture **must** remain readable by HDF5 on any other. 
+	* For binary compatibility no changes are allowed to public APIs and data structures in the maintenance releases; new APIs can be added.
+
+* **New features are documented** - Any new features should have proper documentation; talk to us if you have any questions.
+
+
+# Checklist <A NAME="checklist"></A>
+
+Please make sure that you check the items applicable to your pull request:
+
+* Code 
+  * [ ] Does the pull request have a corresponding GitHub issue and clear purpose?
+  * [ ] Does the pull request follow HDF5 best practices (naming conventions, code portability, code structure, etc.)? <<TODO: link to the document>>
+  * [ ] If changes were done to autotools build were they added to CMake and vice versa?
+  * [ ] Is the pull request applicable to any other branches? If yes, which ones? Please document it in the GitHub issue.
+  * [ ] Is the new code sufficiently documented for future maintenance?
+  * [ ] Does the new feature require a change to an existing API? See "API Compatibility Macros" document (https://portal.hdfgroup.org/display/HDF5/API+Compatibility+Macros)
+* Documentation
+  * [ ] Was the change described in the release_docs/RELEASE.txt file?
+  * [ ] Was MANIFEST updated if new files had been added to the source?
+  * [ ] Was the new function documented in the corresponding public header file using Doxygen? <<TODO: link to Doxygen instructions>>
+  * [ ] Was new functionality documented for the HDF5 community (the level of documentation depends on the feature; ask us what would be appropriate)
+* Testing
+  * [ ] Does the pull request have tests?
+  * [ ] Does the pull request affect HDF5 library perfromance?
+
+We want as many contributions as we can get, and we are here to help. Feel free to reach out to us if you have any questions
+
+Thank you for your contribution!
+
diff --git a/release_docs/RELEASE.txt b/release_docs/RELEASE.txt
@@ -274,6 +274,26 @@ Bug Fixes since HDF5-1.10.7 release
 ===================================
     Library
     -------
+    - Fixed CVE-2018-17435
+
+        The tool h5dump produced a segfault when the size of a fill value
+        message was corrupted and caused a buffer overflow.
+
+        The problem was fixed by verifying the fill value's size
+        against the buffer size before attempting to access the buffer.
+
+        (BMR - 2021/03/15, HDFFV-10480)
+
+    - Fixed CVE-2018-14033 (same issue as CVE-2020-10811)
+
+        The tool h5dump produced a segfault when the storage size message
+        was corrupted and caused a buffer overflow.
+
+        The problem was fixed by verifying the storage size against the
+        buffer size before attempting to access the buffer.
+
+        (BMR - 2021/03/15, HDFFV-11159/HDFFV-11049)
+
     - Remove underscores on header file guards
 
         Header file guards used a variety of underscores at the beginning of the define.

diff --git a/src/H5Olayout.c b/src/H5Olayout.c
@@ -17,7 +17,7 @@
  * Purpose:     Messages related to data layout.
  */
 
-#define H5D_FRIEND     /*suppress error about including H5Dpkg	  */
+#define H5D_FRIEND     /*suppress error about including H5Dpkg       */
 #include "H5Omodule.h" /* This source code file is part of the H5O module */
 
 #include "H5private.h"   /* Generic Functions                        */
@@ -896,13 +896,13 @@ H5O__layout_reset(void *_mesg)
 } /* end H5O__layout_reset() */
 
 /*-------------------------------------------------------------------------
- * Function:	H5O__layout_free
+ * Function:    H5O__layout_free
  *
- * Purpose:	Free's the message
+ * Purpose:     Free's the message
  *
- * Return:	Non-negative on success/Negative on failure
+ * Return:      Non-negative on success/Negative on failure
  *
- * Programmer:	Quincey Koziol
+ * Programmer:  Quincey Koziol
  *              Saturday, March 11, 2000
  *
  *-------------------------------------------------------------------------

diff --git a/src/H5Tconv.c b/src/H5Tconv.c
@@ -1664,7 +1664,7 @@ H5T__conv_b_b(hid_t src_id, hid_t dst_id, H5T_cdata_t *cdata, size_t nelmts, siz
     size_t         olap;                    /*num overlapping elements    */
     size_t         half_size;               /*1/2 of total size for swapping*/
     uint8_t *      s, *sp, *d, *dp;         /*source and dest traversal ptrs*/
-    uint8_t        dbuf[256];               /*temp destination buffer    */
+    uint8_t        dbuf[256] = {0};         /*temp destination buffer    */
     size_t         msb_pad_offset;          /*offset for dest MSB padding    */
     size_t         i;
     uint8_t *      src_rev   = NULL;         /*order-reversed source buffer  */
@@ -3612,8 +3612,8 @@ H5T__conv_i_i(hid_t src_id, hid_t dst_id, H5T_cdata_t *cdata, size_t nelmts, siz
     size_t         half_size;            /*half the type size        */
     size_t         olap;                 /*num overlapping elements    */
     uint8_t *      s, *sp, *d, *dp;      /*source and dest traversal ptrs*/
-    uint8_t *      src_rev = NULL;       /*order-reversed source buffer  */
-    uint8_t        dbuf[64];             /*temp destination buffer    */
+    uint8_t *      src_rev  = NULL;      /*order-reversed source buffer  */
+    uint8_t        dbuf[64] = {0};       /*temp destination buffer    */
     size_t         first;
     ssize_t        sfirst;                   /*a signed version of `first'    */
     size_t         i;                        /*Local index variables         */
@@ -4055,8 +4055,8 @@ H5T__conv_f_f(hid_t src_id, hid_t dst_id, H5T_cdata_t *cdata, size_t nelmts, siz
     size_t       olap;                 /*num overlapping elements    */
     ssize_t      bitno = 0;            /*bit number            */
     uint8_t *    s, *sp, *d, *dp;      /*source and dest traversal ptrs*/
-    uint8_t *    src_rev = NULL;       /*order-reversed source buffer  */
-    uint8_t      dbuf[64];             /*temp destination buffer    */
+    uint8_t *    src_rev  = NULL;      /*order-reversed source buffer  */
+    uint8_t      dbuf[64] = {0};       /*temp destination buffer    */
     uint8_t      tmp1, tmp2;           /*temp variables for swapping bytes*/
 
     /* Conversion-related variables */
@@ -4714,7 +4714,7 @@ H5T__conv_s_s(hid_t src_id, hid_t dst_id, H5T_cdata_t *cdata, size_t nelmts, siz
             dst_delta = (ssize_t)direction * (ssize_t)(buf_stride ? buf_stride : dst->shared->size);
 
             /* Allocate the overlap buffer */
-            if (NULL == (dbuf = (uint8_t *)H5MM_malloc(dst->shared->size)))
+            if (NULL == (dbuf = (uint8_t *)H5MM_calloc(dst->shared->size)))
                 HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, FAIL, "memory allocation failed for string conversion")
 
             /* The conversion loop. */
@@ -8176,8 +8176,8 @@ H5T__conv_f_i(hid_t src_id, hid_t dst_id, H5T_cdata_t *cdata, size_t nelmts, siz
     size_t       tsize;           /*type size for swapping bytes  */
     size_t       olap;            /*num overlapping elements    */
     uint8_t *    s, *sp, *d, *dp; /*source and dest traversal ptrs*/
-    uint8_t *    src_rev = NULL;  /*order-reversed source buffer  */
-    uint8_t      dbuf[64];        /*temp destination buffer    */
+    uint8_t *    src_rev  = NULL; /*order-reversed source buffer  */
+    uint8_t      dbuf[64] = {0};  /*temp destination buffer    */
     uint8_t      tmp1, tmp2;      /*temp variables for swapping bytes*/
 
     /* Conversion-related variables */
@@ -8799,8 +8799,8 @@ H5T__conv_i_f(hid_t src_id, hid_t dst_id, H5T_cdata_t *cdata, size_t nelmts, siz
     size_t       tsize;           /*type size for swapping bytes  */
     size_t       olap;            /*num overlapping elements    */
     uint8_t *    s, *sp, *d, *dp; /*source and dest traversal ptrs*/
-    uint8_t *    src_rev = NULL;  /*order-reversed source buffer  */
-    uint8_t      dbuf[64];        /*temp destination buffer    */
+    uint8_t *    src_rev  = NULL; /*order-reversed source buffer  */
+    uint8_t      dbuf[64] = {0};  /*temp destination buffer    */
     uint8_t      tmp1, tmp2;      /*temp variables for swapping bytes*/
 
     /* Conversion-related variables */

diff --git a/test/CMakeTests.cmake b/test/CMakeTests.cmake
@@ -965,9 +965,9 @@ if (ENABLE_EXTENDED_TESTS)
         ENVIRONMENT "srcdir=${HDF5_TEST_BINARY_DIR}/H5TEST/flushrefresh_test"
         WORKING_DIRECTORY ${HDF5_TEST_BINARY_DIR}/H5TEST/flushrefresh_test
     )
+  else ()
+    message (STATUS "Cannot execute TEST flushrefresh - perl not found")
   endif ()
-else ()
-  message (STATUS "Cannot execute TEST flushrefresh - perl not found")
 endif ()
 
 ##############################################################################

diff --git a/tools/test/h5dump/CMakeTests.cmake b/tools/test/h5dump/CMakeTests.cmake
@@ -319,6 +319,8 @@
       ${HDF5_TOOLS_DIR}/testfiles/tvlstr.h5
       ${HDF5_TOOLS_DIR}/testfiles/tvms.h5
       ${HDF5_TOOLS_DIR}/testfiles/t128bit_float.h5
+      ${HDF5_TOOLS_DIR}/testfiles/tCVE_2018_11206_fill_old.h5
+      ${HDF5_TOOLS_DIR}/testfiles/tCVE_2018_11206_fill_new.h5
       ${HDF5_TOOLS_DIR}/testfiles/zerodim.h5
   )
   set (HDF5_ERROR_REFERENCE_TEST_FILES
@@ -1131,6 +1133,10 @@
   # test to verify HDFFV-9407: long double full precision
 #  ADD_H5_GREP_TEST (t128bit_float 1 "1.123456789012345" -m %.35Lg t128bit_float.h5)
 
+  # test to verify HDFFV-10480: out of bounds read in H5O_fill_new[old]_decode
+  ADD_H5_TEST (tCVE_2018_11206_fill_old 1 tCVE_2018_11206_fill_old.h5)
+  ADD_H5_TEST (tCVE_2018_11206_fill_new 1 tCVE_2018_11206_fill_new.h5)
+
 ##############################################################################
 ###    P L U G I N  T E S T S
 ##############################################################################

diff --git a/tools/test/h5dump/testh5dump.sh.in b/tools/test/h5dump/testh5dump.sh.in
@@ -177,6 +177,8 @@ $SRC_H5DUMP_TESTFILES/tvldtypes5.h5
 $SRC_H5DUMP_TESTFILES/tvlenstr_array.h5
 $SRC_H5DUMP_TESTFILES/tvlstr.h5
 $SRC_H5DUMP_TESTFILES/tvms.h5
+$SRC_H5DUMP_TESTFILES/tCVE_2018_11206_fill_old.h5
+$SRC_H5DUMP_TESTFILES/tCVE_2018_11206_fill_new.h5
 "
 
 LIST_OTHER_TEST_FILES="
@@ -861,6 +863,35 @@ TOOLTEST5() {
     fi
 }
 
+# same as TOOLTEST1 but expects h5dump to fail
+#
+TOOLTEST_FAIL() {
+
+    infile=$1
+    expect="$TESTDIR/`basename $1 exp`.ddl"
+    actual="$TESTDIR/`basename $1 .exp`.out"
+
+    # Run test.
+    TESTING $DUMPER $@
+    (
+        cd $TESTDIR
+        $RUNSERIAL $DUMPER_BIN "$@" $infile
+    ) >&$actual
+    RET=$?
+    # Segfault occurred
+    if [ $RET == 139 ] ; then
+        nerrors="`expr $nerrors + 1`"
+        echo "*FAILED - test on $infile failed with segmentation fault"
+    # Should fail but didn't
+    elif [ $RET == 0 ] ; then
+        nerrors="`expr $nerrors + 1`"
+        echo "*FAILED - test on $infile did not fail as expected"
+    else
+        echo " PASSED"
+    fi
+
+}
+
 # ADD_HELP_TEST
 TOOLTEST_HELP() {
 
@@ -1435,6 +1466,10 @@ TOOLTEST3 non_existing.ddl --enable-error-stack tgroup.h5 non_existing.h5
 # test to verify HDFFV-9407: long double full precision
 #GREPTEST OUTTXT "1.123456789012345" t128bit_float.ddl -m %.35Lf t128bit_float.h5
 
+# test to verify HDFFV-10480: out of bounds read in H5O_fill_new[old]_decode
+TOOLTEST_FAIL tCVE_2018_11206_fill_old.h5
+TOOLTEST_FAIL tCVE_2018_11206_fill_new.h5
+
 # Clean up temporary files/directories
 CLEAN_TESTFILES_AND_TESTDIR
 

diff --git a/tools/testfiles/tCVE_2018_11206_fill_new.h5 b/tools/testfiles/tCVE_2018_11206_fill_new.h5
diff --git a/tools/testfiles/tCVE_2018_11206_fill_old.h5 b/tools/testfiles/tCVE_2018_11206_fill_old.h5