Updated HPSS paths; modified other user/project-centric paths for con…

…sistency (Fixes olcf#208) * Updated HPSS paths; modified other user/project-centric paths for consistency * Fixed code snip * Fixed typo. Co-authored-by: Jack Morrison <[email protected]>
gmarkomanolis · Feb 24, 2020 · d94f718 · d94f718
1 parent 9dfcfab
commit d94f718
Show file tree

Hide file tree

Showing 6 changed files with 333 additions and 182 deletions.
diff --git a/accounts/olcf_policy_guide.rst b/accounts/olcf_policy_guide.rst
diff --git a/data/archiving.rst b/data/archiving.rst
@@ -101,17 +101,11 @@ HPSS, you might use:
 and ``hsi mput`` can be used to retrieve multiple files. To send a file to HPSS,
 you might use:
 
-     ``hsi put a.out``
-
-To put a file in a pre-existing directory on hpss:
-
-
-     ``hsi “cd MyHpssDir; put a.out”``
+     ``hsi put a.out : /hpss/prod/[projid]/users/[userid]/a.out``
 
 To retrieve one, you might use:
 
-
-     ``hsi get /proj/projectid/a.out``
+     ``hsi get /hpss/prod/[projid]/proj-shared/a.out``
 
 Here is a list of commonly used hsi commands.
 
@@ -159,32 +153,32 @@ As with the standard Unix ``tar`` utility the ``-c``, ``-x``, and ``-t``
 options, respectively, function to create, extract, and list tar archive files.
 The ``-K`` option verifies an existing tarfile in HPSS and the ``-X`` option can
 be used to re-create the index file for an existing archive. For example, to
-store all files in the directory ``dir1`` to a file named ``allfiles.tar`` on
-HPSS, use the command:
+store all files in the directory ``dir1`` to a file named
+``/hpss/prod/[projid]/users/[userid]/allfiles.tar`` on HPSS, use the command:
 
-     ``htar -cvf allfiles.tar dir1/*``
+     ``htar -cvf /hpss/prod/[projid]/users/[userid]/allfiles.tar dir1/*``
 
 To retrieve these files:
 
-     ``htar -xvf allfiles.tar``
+     ``htar -xvf  /hpss/prod/[projid]/users/[userid]/allfiles.tar``
 
 ``htar`` will overwrite files of the same name in the target directory.  **When
 possible, extract only the files you need from large archives.** To display the
 names of the files in the ``project1.tar`` archive file within the HPSS home
 directory:
 
-     ``htar -vtf project1.tar``
+     ``htar -vtf  /hpss/prod/[projid]/users/[userid]/project1.tar``
 
 To extract only one file, ``executable.out``, from the ``project1`` directory in
-the Archive file called ``project1.tar``:
+the Archive file called `` /hpss/prod/[projid]/users/[userid]/project1.tar``:
 
      ``htar -xm -f project1.tar project1/ executable.out``
 
 To extract all files from the ``project1/src`` directory in the archive file
 called ``project1.tar``, and use the time of extraction as the modification
 time, use the following command:
 
-     ``htar -xm -f project1.tar project1/src``
+     ``htar -xm -f  /hpss/prod/[projid]/users/[userid]/project1.tar project1/src``
 
 HTAR Limitations
 -----------------
@@ -219,7 +213,7 @@ that 64GB, the following error message will appear:
 
 .. code::
 
-   $ htar -cvf hpss_test.tar hpss_test/
+   $ htar -cvf  /hpss/prod/[projid]/users/[userid]/hpss_test.tar hpss_test/
 
    INFO: File too large for htar to handle: hpss_test/75GB.dat (75161927680 bytes)
    ERROR: 1 oversize member files found - please correct and retry

diff --git a/data/policies.rst b/data/policies.rst
@@ -9,32 +9,62 @@ Policy
 A brief description of each area and basic guidelines to follow are provided in
 the table below:
 
-+-------------------+---------------------------------------------+---------------------------+-------------+---------+---------+-------+--------------------------+
-| *Name*            |   Path                                      |     Type                  | Permissions | Backups |  Purged | Quota | Mounted on Compute nodes |
-+===================+=============================================+===========================+=============+=========+=========+=======+==========================+
-| *User Home*       |   ``$HOME``                                 |     NFS                   |   User Set  |    yes  |    no   | 50GB  |   Read-only              |
-+-------------------+---------------------------------------------+---------------------------+-------------+---------+---------+-------+--------------------------+
-| *Project Home*    | ``/ccs/proj/[projid]``                      |     NFS                   |      770    |    yes  |    no   |  50GB |  Read-only               |
-+-------------------+---------------------------------------------+---------------------------+-------------+---------+---------+-------+--------------------------+
-| *User Archive*    | ``/home/$USER``                             |     HPSS                  |   User Set  |    no   |    no   |  2TB  |    No                    |
-+-------------------+---------------------------------------------+---------------------------+-------------+---------+---------+-------+--------------------------+
-| *Project Archive* | ``/proj/[projid]``                          |     HPSS                  |      770    |    no   |    no   | 100TB |     No                   |
-+-------------------+---------------------------------------------+---------------------------+-------------+---------+---------+-------+--------------------------+
-| *Member Work*     | ``/gpfs/alpine/scratch/[userid]/[projid]/`` | Spectrum Scale (ex. GPFS) |      700    |    no   | 90 days |  50TB |  Yes                     |
-+-------------------+---------------------------------------------+---------------------------+-------------+---------+---------+-------+--------------------------+
-| *Project Work*    | ``/gpfs/alpine/proj-shared/[projid]``       | Spectrum Scale (ex. GPFS) |      770    |    no   | 90 days |  50TB |  Yes                     |
-+-------------------+---------------------------------------------+---------------------------+-------------+---------+---------+-------+--------------------------+
-| *World Work*      | ``/gpfs/alpine/world-shared/[projid]``      | Spectrum Scale (ex. GPFS) |      775    |    no   | 90 days |  50TB |  Yes                     |
-+-------------------+---------------------------------------------+---------------------------+-------------+---------+---------+-------+--------------------------+
-
-
-On Summit paths to the various project-centric work storage areas are simplified
-by the use of environment variables that point to the proper directory on a
-per-user basis:
-
-- Member Work Directory:  ``$MEMBERWORK/[projid]``
-- Project Work Directory: ``$PROJWORK/[projid]``
-- World Work Directory: ``$WORLDWORK/[projid]``
++---------------------+---------------------------------------------+----------------+-------------+--------+---------+---------+------------+------------------+
+| Area                | Path                                        | Type           | Permissions |  Quota | Backups | Purged  | Retention  | On Compute Nodes |
++=====================+=============================================+================+=============+========+=========+=========+============+==================+
+| User Home           | ``/ccs/home/[userid]``                      | NFS            | User set    |  50 GB | Yes     | No      | 90 days    | Read-only        |
++---------------------+---------------------------------------------+----------------+-------------+--------+---------+---------+------------+------------------+
+| User Archive [#f1]_ | ``/home/[userid]``                          | HPSS           | User set    |  2TB   | No      | No      | 90 days    | No               |
++---------------------+---------------------------------------------+----------------+-------------+--------+---------+---------+------------+------------------+
+| User Archive [#f2]_ | ``/home/[userid]``                          | HPSS           | 700         |  N/A   | N/A     | N/A     | N/A        | No               |
++---------------------+---------------------------------------------+----------------+-------------+--------+---------+---------+------------+------------------+
+| Project Home        | ``/ccs/proj/[projid]``                      | NFS            | 770         |  50 GB | Yes     | No      | 90 days    | Read-only        |
++---------------------+---------------------------------------------+----------------+-------------+--------+---------+---------+------------+------------------+
+| Member Work         | ``/gpfs/alpine/[projid]/scratch/[userid]``  | Spectrum Scale | 700 [#f3]_  |  50 TB | No      | 90 days | N/A [#f4]_ | Yes              |
++---------------------+---------------------------------------------+----------------+-------------+--------+---------+---------+------------+------------------+
+| Project Work        | ``/gpfs/alpine/[projid]/proj-shared``       | Spectrum Scale | 770         |  50 TB | No      | 90 days | N/A [#f4]_ | Yes              |
++---------------------+---------------------------------------------+----------------+-------------+--------+---------+---------+------------+------------------+
+| World Work          | ``/gpfs/alpine/[projid]/world-shared``      | Spectrum Scale | 775         |  50 TB | No      | 90 days | N/A [#f4]_ | Yes              |
++---------------------+---------------------------------------------+----------------+-------------+--------+---------+---------+------------+------------------+
+| Member Archive      | ``/hpss/prod/[projid]/users/$USER``         | HPSS           | 700         | 100 TB | No      | No      | 90 days    | No               |
++---------------------+---------------------------------------------+----------------+-------------+--------+---------+---------+------------+------------------+
+| Project Archive     | ``/hpss/prod/[projid]/proj-shared``         | HPSS           | 770         | 100 TB | No      | No      | 90 days    | No               |
++---------------------+---------------------------------------------+----------------+-------------+--------+---------+---------+------------+------------------+
+| World Archive       | ``/hpss/prod/[projid]/world-shared``        | HPSS           | 775         | 100 TB | No      | No      | 90 days    | No               |
++---------------------+---------------------------------------------+----------------+-------------+--------+---------+---------+------------+------------------+
+
+| *Area -* The general name of storage area.
+| *Path -* The path (symlink) to the storage area's directory.
+| *Type -* The underlying software technology supporting the storage area.
+| *Permissions -* UNIX Permissions enforced on the storage area's top-level directory.
+| *Quota -* The limits placed on total number of bytes and/or files in the storage area.
+| *Backups -* States if the data is automatically duplicated for disaster recovery purposes.
+| *Purged -* Period of time, post-file-access, after which a file will be marked as eligible for permanent deletion.
+| *Retention -* Period of time, post-account-deactivation or post-project-end, after which data will be marked as eligible for permanent deletion.
+| *On Compute Nodes -* Is this filesystem available on compute nodes (yes, no, or available but read-only)
+
+    **Important!** Files within "Work" directories (i.e., Member Work,
+    Project Work, World Work) are *not* backed up and are *purged* on a
+    regular basis according to the timeframes listed above.
+
+.. rubric:: Footnotes
+
+.. [#f1] This entry is for legacy User Archive directories which contained user data on January 14, 2020.
+
+.. [#f2] User Archive directories that were created (or had no user data) after January 14, 2020. Settings other than permissions are not applicable because directories are root-owned and contain no user files.
+
+.. [#f3] Permissions on Member Work directories can be controlled to an extent by project members. By default, only the project member has any accesses, but accesses can be granted to other project members by setting group permissions accordingly on the Member Work directory. The parent directory of the Member Work directory prevents accesses by "UNIX-others" and cannot be changed (security measures).
+
+.. [#f4] Retention is not applicable as files will follow purge cycle.
+
+
+
+On Summit, Rhea and the DTNs, additional paths to the various project-centric work areas are available
+via the following symbolic links and/or environment variables:
+
+- Member Work Directory:  ``/gpfs/alpine/scratch/[userid]/[projid]`` or ``$MEMBERWORK/[projid]``
+- Project Work Directory: ``/gpfs/alpine/proj-shared/[projid]`` or ``$PROJWORK/[projid]``
+- World Work Directory: ``/gpfs/alpine/world-shared/[projid]`` or ``$WORLDWORK/[projid]``
 
 Information
 ============