zstash_problem_set.md

zstash problem set

This problem set will help you learn more about zstash. Try to solve the problem by looking through the documentation and/or discussion pages.

Use cases

Problem 1

Suppose you just ran a simulation on a non-NERSC Machine. Is it possible to still archive it on NERSC HPSS? If so, how could we do that?

Hints

If the machine has a Globus endpoint, can zstash help us transfer to NERSC HPSS?

The zstash usage docs can provide some direction.

Solution

You just need to tell zstash to use Globus, by setting

--hpss=globus://<Globus endpoint UUID/<path>

For NERSC HPSS specifically, that'd be:

--hpss=globus://9cd89cfd-6d04-11e5-ba46-22000b92c6ec/<path>

Luckily, zstash has a built-in default for this endpoint:

--hpss=globus://NERSC/<path>

Problem 2

Suppose we want to archive a simulation to HPSS, but we also want the generated tars locally. How could we do that?

Hints

Is there a command line option that could help us?

Solution

We juse need to add the --keep flag.

Problem 3

Suppose we want to archive a simulation locally, but not to HPSS, perhaps because the machine we're on doesn't have HPSS. How could we do that?

Hints

Is there a command line option that could help us?

Solution

We juse need to set --hpss=none. Then, the cache effectively replaces the HPSS archive.

Debugging

Problem 1

zstash create --hpss=tutorial_archive_20240507 --include=archive/atm/hist/extendedOutput.v3.LR.historical_0101.eam.h0.2000-1*.nc /global/cfs/cdirs/e3sm/www/Tutorials/2024/simulations/extendedOutput.v3.LR.historical_0101/

Why does the above command give a permissions error?

Hints

zstash temporarily stores files before transferring them to HPSS. Can we change where we temporarily store files -- to somewhere where we do have permissions?

Solution

We need to add the cache parameter:

--cache=/pscratch/sd/f/forsyth/e3sm_tutorial/workdir/zstash_v3.LR.historical_0101