Split doxygen pages from spec and TN into files (#5165)

doxygen/dox/AppDbgging.dox

@@ -0,0 +1,158 @@
+
+/** \page APPDBG Debugging HDF5 Applications
+
+\section sec_adddbg_intro Introduction
+The HDF5 library contains a number of debugging features to make programmers' lives
+easier including the ability to print detailed error messages, check invariant
+conditions, display timings and other statistics.
+
+\subsection subsec_adddbg_intro_err Error Messages
+Error messages are normally displayed automatically on the standard error stream and
+include a stack trace of the library including file names, line numbers, and function
+names. The application has complete control over how error messages are displayed and
+can disable the display on a permanent or temporary basis. Refer to the documentation
+ for the H5E error handling package.
+
+\subsection subsec_adddbg_intro_invar Invariant Conditions
+Unless NDEBUG is defined during compiling, the library will include code to verify that
+invariant conditions have the expected values. When a problem is detected the library will
+display the file and line number within the library and the invariant condition that
+failed. A core dump may be generated for post mortem debugging. The code to perform these
+checks can be included on a per-package bases.
+
+\subsection subsec_adddbg_intro_stats Timings and Statistics
+The library can be configured to accumulate certain statistics about things like cache
+performance, datatype conversion, data space conversion, and data filters. The code is
+included on a per-package basis and enabled at runtime by an environment variable.
+
+\subsection subsec_adddbg_intro_trace API Tracing
+Tracing API calls has been removed from the library.
+
+The statistics can be displayed on any output stream (including streams opened by
+the shell) with output from different packages even going to different streams.
+
+\section sec_adddbg_msg Error Messages
+By default any API function that fails will print an error stack to the standard error stream.
+\code
+HDF5-DIAG: Error detected in thread 0.  Back trace follows.
+  #000: H5F.c line 1245 in H5Fopen(): unable to open file
+    major(04): File interface
+    minor(10): Unable to open file
+  #001: H5F.c line 846 in H5F_open(): file does not exist
+    major(04): File interface
+    minor(10): Unable to open file
+\endcode
+The error handling package (H5E) is described elsewhere.
+
+\section sec_adddbg_invars Invariant Conditions
+To include checks for invariant conditions the library should be configured
+with --disable-production, the default for versions before 1.2. The library
+designers have made every attempt to handle error conditions gracefully but
+an invariant condition assertion may fail in certain cases. The output from
+a failure usually looks something like this:
+\code
+Assertion failed: H5.c:123: i<NELMTS(H5_debug_g)
+IOT Trap, core dumped.
+\endcode
+
+\section sec_adddbg_stats Timings and Statistics
+Code to accumulate statistics is included at compile time by using the --enable-debug
+configure switch. The switch can be followed by an equal sign and a comma-separated
+list of package names or else a default list is used.
+<table>
+<tr>
+<th><strong>Name</strong>
+</th>
+<th><strong>Default</strong>
+</th>
+<th><strong>Description</strong>
+</th>
+</tr>
+<tr>
+<td>a</td><td>No</td><td>Attributes</td>
+</tr>
+<tr>
+<td>ac</td><td>Yes</td><td>Meta data cache</td>
+</tr>
+<tr>
+<td>b</td><td>Yes</td><td>B-Trees</td>
+</tr>
+<tr>
+<td>d</td><td>Yes</td><td>Datasets</td>
+</tr>
+<tr>
+<td>e</td><td>Yes</td><td>Error handling</td>
+</tr>
+<tr>
+<td>f</td><td>Yes</td><td>Files</td>
+</tr>
+<tr>
+<td>g</td><td>Yes</td><td>Groups</td>
+</tr>
+<tr>
+<td>hg</td><td>Yes</td><td>Global heap</td>
+</tr>
+<tr>
+<td>hl</td><td>No</td><td>Local heaps</td>
+</tr>
+<tr>
+<td>i</td><td>Yes</td><td>Interface abstraction</td>
+</tr>
+<tr>
+<td>mf</td><td>No</td><td>File memory management</td>
+</tr>
+<tr>
+<td>mm</td><td>Yes</td><td>Library memory management</td>
+</tr>
+<tr>
+<td>o</td><td>No</td><td>Object headers and messages</td>
+</tr>
+<tr>
+<td>p</td><td>Yes</td><td>Property lists</td>
+</tr>
+<tr>
+<td>s</td><td>Yes</td><td>Data spaces</td>
+</tr>
+<tr>
+<td>t</td><td>Yes</td><td>Datatypes</td>
+</tr>
+<tr>
+<td>v</td><td>Yes</td><td>Vectors</td>
+</tr>
+<tr>
+<td>z</td><td>Yes</td><td>Raw data filters</td>
+</tr>
+</table>
+
+In addition to including the code at compile time the application must enable each package at
+runtime. This is done by listing the package names in the HDF5_DEBUG environment variable. That
+variable may also contain file descriptor numbers (the default is '2') which control the output
+for all following packages up to the next file number. The word 'all' refers to all packages. Any
+word my be preceded by a minus sign to turn debugging off for the package.
+
+\subsection subsec_adddbg_stats_sample Sample debug specifications
+<table>
+<tr>
+<td>all
+</td>
+<td>This causes debugging output from all packages to be sent to the standard error stream.
+</td>
+</tr>
+<tr>
+<td>all -t -s
+</td>
+<td>Debugging output for all packages except datatypes and data spaces will appear on the standard error stream.
+</td>
+</tr>
+<tr>
+<td>-all ac 255 t,s
+</td>
+<td>This disables all debugging even if the default was to debug something, then output
+from the meta data cache is send to the standard error stream and output from data types
+and spaces is sent to file descriptor 255 which should be redirected by the shell.
+</td>
+</tr>
+</table>
+The components of the HDF5_DEBUG value may be separated by any non-lowercase letter.
+
+*/

doxygen/dox/GettingStarted.dox

@@ -50,15 +50,15 @@ The high-level HDF5 library includes several sets of convenience and standard-us
 </tr>
 <tr>
 <td style="background-color:#F5F5F5">
-\ref IntroParHDF5
+@ref IntroParHDF5
 </td>
 <td>
 A brief introduction to Parallel HDF5. If you are new to HDF5 please see the @ref LearnBasics topic first.
 </td>
 </tr>
 <tr>
 <td style="background-color:#F5F5F5">
-\ref ViewTools
+@ref ViewTools
 </td>
 <td>
 \li @ref LearnHDFView
@@ -71,16 +71,16 @@ A brief introduction to Parallel HDF5. If you are new to HDF5 please see the @re
 New Features since HDF5-1.10
 </td>
 <td>
-\li \ref VDS
-\li \ref SWMR
+\li @ref VDSTN
+\li @ref SWMRTN
 </td>
 </tr>
 <tr>
 <td style="background-color:#F5F5F5">
 Example Programs
 </td>
 <td>
-\ref HDF5Examples
+@ref HDF5Examples
 </td>
 </tr>
 <tr>

doxygen/dox/SWMRTechNote.dox

@@ -0,0 +1,146 @@
+
+/** \page SWMRTN Introduction to Single-Writer/Multiple-Reader (SWMR)
+
+\section sec_swmr_intro Introduction to SWMR
+The Single-Writer / Multiple-Reader (SWMR) feature enables multiple processes to read an HDF5 file
+while it is being written to (by a single process) without using locks or requiring communication between processes.
+<img src=tutr-swmr1.png alt="tutr-swmr1.png" width=500>
+
+All communication between processes must be performed via the HDF5 file. The HDF5 file under SWMR access must
+reside on a system that complies with POSIX write() semantics.
+
+The basic engineering challenge for this to work was to ensure that the readers of an HDF5 file always
+see a coherent (though possibly not up to date) HDF5 file.
+
+The issue is that when writing data there is information in the metadata cache in addition to the physical file on disk:
+<img src=tutr-swmr2.png alt="tutr-swmr2.png" width=500>
+
+However, the readers can only see the state contained in the physical file:
+<img src=tutr-swmr3.png alt="tutr-swmr3.png" width=500>
+
+The SWMR solution implements dependencies on when the metadata can be flushed to the file. This ensures that metadata cache
+flush operations occur in the proper order, so that there will never be internal file pointers in the physical file
+that point to invalid (unflushed) file addresses.
+
+A beneficial side effect of using SWMR access is better fault tolerance. It is more difficult to corrupt a file when using SWMR.
+
+\subsection subsec_swmr_doc Documentation
+\subsubsection subsubsec_swmr_doc_guide User Guide
+<a href="https://\DOCURL/features/SWMR/HDF5_SWMR_Users_Guide.pdf">SWMR User Guide</a>
+
+\subsubsection subsubsec_swmr_doc_apis HDF5 Library APIs
+<ul>
+<li>#H5Fstart_swmr_write — Enables SWMR writing mode for a file</li>
+<li>#H5DOappend — Appends data to a dataset along a specified dimension</li>
+<li>#H5Pset_object_flush_cb — Sets a callback function to invoke when an object flush occurs in the file</li>
+<li>#H5Pget_object_flush_cb — Retrieves the object flush property values from the file access property list</li>
+<li>#H5Odisable_mdc_flushes — Prevents metadata entries for an HDF5 object from being flushed from the metadata cache to storage</li>
+<li>#H5Oenable_mdc_flushes — Enables flushing of dirty metadata entries from a file's metadata cache</li>
+<li>#H5Oare_mdc_flushes_disabled — Determines if an HDF5 object has had flushes of metadata entries disabled</li>
+</ul>
+
+\subsubsection subsubsec_swmr_doc_tools Tools
+\li h5watch — Outputs new records appended to a dataset as the dataset grows
+\li h5format_convert — Converts the layout format version and chunked indexing types of datasets created with
+HDF5-1.10 so that applications built with HDF5-1.8 can access them
+\li h5clear — Clears superblock status_flags field, removes metadata cache image, prints EOA and EOF, or sets EOA of a file
+
+\subsubsection subsubsec_swmr_doc_design Design Documents
+
+\subsection subsec_swmr_model Programming Model
+Please be aware that the SWMR feature requires that an HDF5 file be created with the latest file format. See
+#H5Pset_libver_bounds for more information.
+
+To use SWMR follow the the general programming model for creating and accessing HDF5 files and objects along with the steps described below.
+
+\subsubsection subsubsec_swmr_model_writer SWMR Writer
+The SWMR writer either opens an existing file and objects or creates them as follows.
+
+Open an existing file:
+Call #H5Fopen using the #H5F_ACC_SWMR_WRITE flag.
+Begin writing datasets.
+Periodically flush data.
+
+Create a new file:
+Call #H5Fcreate using the latest file format.
+Create groups, datasets and attributes, and then close the attributes.
+Call #H5Fstart_swmr_write to start SWMR access to the file.
+Periodically flush data.
+
+<h4 id="example-code">Example Code:</h4>
+Create the file using the latest file format property:
+\code
+   fapl = H5Pcreate (H5P_FILE_ACCESS);
+   status = H5Pset_libver_bounds (fapl, H5F_LIBVER_LATEST, H5F_LIBVER_LATEST);
+   fid = H5Fcreate (filename, H5F_ACC_TRUNC, H5P_DEFAULT, fapl);
+   // Create objects (files, datasets, ...).
+   // Close any attributes and named datatype objects.
+   // Groups and datasets may remain open before starting SWMR access to them.
+
+   // Start SWMR access to the file:
+   status = H5Fstart_swmr_write (fid);
+
+   // Reopen the datasets and then start writing, periodically flushing data:
+   status = H5Dwrite (dset_id, ...);
+   status = H5Dflush (dset_id);
+\endcode   
+
+\subsubsection subsubsec_swmr_model_reader SWMR Reader
+The SWMR reader must continually poll for new data:
+
+Call #H5Fopen using the #H5F_ACC_SWMR_READ flag.
+Poll, checking the size of the dataset to see if there is new data available for reading.
+Read new data, if any.
+
+<h4 id="example-code-1">Example Code:</h4>
+\code
+   // Open the file using the SWMR read flag:
+   fid = H5Fopen (filename, H5F_ACC_RDONLY | H5F_ACC_SWMR_READ, H5P_DEFAULT);
+   // Open the dataset and then repeatedly poll the dataset, by getting the dimensions, reading new data, and refreshing:
+   dset_id = H5Dopen (...);
+   space_id = H5Dget_space (...);
+   while (...) {
+      status = H5Dread (dset_id, ...);
+      status = H5Drefresh (dset_id);
+      space_id = H5Dget_space (...);
+   }
+\endcode
+
+\subsection subsec_swmr_scope Limitations and Scope
+An HDF5 file under SWMR access must reside on a system that complies with POSIX write()
+semantics. It is also limited in scope as follows.
+
+The writer process is only allowed to modify raw data of existing datasets by;
+Appending data along any unlimited dimension.
+Modifying existing data
+The following operations are not allowed (and the corresponding HDF5 files will fail)
+\li The writer cannot add new objects to the file.
+\li The writer cannot delete objects in the file.
+\li The writer cannot modify or append data with variable length, string or region reference datatypes.
+\li File space recycling is not allowed. As a result the size of a file modified by a SWMR writer may be larger than a file modified by a non-SWMR writer.</p>
+
+\subsection subsec_swmr_tools Tools for Working with SWMR
+Two new tools, h5watch and h5clear, are available for use with SWMR. The other HDF5 utilities have also been modified to recognize SWMR
+\li The h5watch tool allows a user to monitor the growth of a dataset.
+\li The h5clear tool clears the status flags in the superblock of an HDF5 file.
+\li The rest of the HDF5 tools will exit gracefully but not work with SWMR otherwise.
+
+\subsection subsec_swmr_example Programming Example
+A good example of using SWMR is included with the HDF5 tests in the source code. You can run it while reading
+the file it creates. If you then interrupt the application and reader and look at the resulting file, you will
+see that the file is still valid. Follow these steps:
+\li Download the HDF5 source code to a local directory on a filesystem (that complies with POSIX write() semantics).
+Build the software. No special configuration options are needed to use SWMR.
+\li Invoke two command terminal windows. In one window go into the bin directory of the built binaries.
+In the other window go into the test directory of the HDF5-1.10 source code that was just built.
+\li In the window in the test directory compile and run use_append_chunk.c. The example writes a three
+dimensional dataset by planes (with chunks of size 1 x 256 x 256).
+\li In the other window (in the bin directory) run h5watch on the file created by
+use_append_chunk.c (use_append_chunk.h5). It should be run while use_append_chunk is executing and you
+will see valid data displayed with h5watch.
+\li Interrupt use_append_chunk while it is running, and stop h5watch.
+\li Use h5clear to clear the status flags in the superblock of the HDF5 file (use_append_chunk.h5).
+\li View the file with h5dump. You will see that it is a valid file even though the application did not
+close properly. It will contain data up to the point that it was interrupted.
+
+*/