Does zstash create copy all data?

[forsyth2]

Does zstash create copy all data? That is, if I have x TB of data, will I have another x TB of data after running zstash create, resulting in 2x TB of data total?

[forsyth2]

Case 1: zstash create --hpss=<archive-name>

Data will be archived into a tar, which is then transferred and deleted locally. Repeat until all data is transferred.

This means that there is no point when more than the --maxsize (default --maxsize=256 GB) is stored twice on the same system. At the end you will have x TB of data locally and x TB of data on HPSS.

Example: https://e3sm-project.github.io/zstash/_build/html/master/tutorial.html#simple-case

Case 2: zstash create --hpss=none

Data will be archived into a tar, which is then kept (not transferred, not deleted). Repeat until all data is in tar files.

At the end, you will have 2x TB of data locally.

Example: https://e3sm-project.github.io/zstash/_build/html/master/tutorial.html#no-hpss

Case 3: zstash create --hpss=<archive> --keep

Data will be archived into a tar, which is then transferred but NOT deleted. Repeat until all data is transferred.

At the end you will have 2x TB of data locally and x TB of data on HPSS.

Example: https://e3sm-project.github.io/zstash/_build/html/master/tutorial.html#keep-files

Case 4: zstash create --hpss=globus://<globus-path>

Data will be archived into a tar, which is then transferred via Globus and afterwards deleted locally. Repeat until all data is transferred.

This means that there is no point when more than the --maxsize (default --maxsize=256 GB) is stored twice on the same system. At the end you will have x TB of data locally and x TB of data on HPSS on another system.

Example: https://e3sm-project.github.io/zstash/_build/html/master/usage.html#create

Case 5: zstash create --hpss=globus://<globus-path> --non-blocking

Data will be archived into a tar, which is submitted to the Globus transfer queue. More data is collected into a tar to be submitted next. Once a Globus transfer completes successfully, the corresponding tar is deleted locally. Repeat until all data is transferred.

This means that there is no point when more than the n * --maxsize (default --maxsize=256 GB, where n is the number of tars waiting to be transferred) is stored twice on the same system. At the end you will have x TB of data locally and x TB of data on HPSS on another system.

Note that this functionality will be updated with #214 -- a Globus transfer will then be able to have multiple tar files in it.

[forsyth2]

@golaz @lukaszlacinski Please let me know if I got any of the above wrong. Thanks.

[forsyth2]

Actually, looking at ls /global/cfs/cdirs/e3sm/forsyth/E3SMv2/v2.LR.historical_0201/zstash/ following my test of #214, I see:

000000.tar  000005.tar  00000a.tar  00000f.tar  000014.tar  000019.tar  00001e.tar  000023.tar  000028.tar  00002d.tar
000001.tar  000006.tar  00000b.tar  000010.tar  000015.tar  00001a.tar  00001f.tar  000024.tar  000029.tar  00002e.tar
000002.tar  000007.tar  00000c.tar  000011.tar  000016.tar  00001b.tar  000020.tar  000025.tar  00002a.tar  00002f.tar
000003.tar  000008.tar  00000d.tar  000012.tar  000017.tar  00001c.tar  000021.tar  000026.tar  00002b.tar  index.db
000004.tar  000009.tar  00000e.tar  000013.tar  000018.tar  00001d.tar  000022.tar  000027.tar  00002c.tar

So, that seems to imply Case 5 (presumably Case 4 too) actually do result in 2x TB of data.