Recover from md5 checksum error

[nanr]

I am archive a large number of netcdf files on HPSS using zstash create, which results in multiple tar files. When I check the zstash archive, I get md5 checksum errors on the FIRST tar file (00000.tar). the other tar files seem to be fine.

How can I recreate a single tar file with that replaces only the files that are on the failed archive tar file? Is there a way to tell zstash to recreate the 00000.tar file and skip the others? or do I need to create a whole new zstash archive?

thanks in advance.

Error:
INFO: Opening tar archive zstash/000000.tar
INFO: Checking log.o
INFO: Checking log.o60192129
INFO: Checking log.o60256424
INFO: Checking log.o60605844
INFO: Checking zstash_create_0171_20220725.log
ERROR: md5 mismatch for: zstash_create_0171_20220725.log
ERROR: md5 of extracted file: 0983653672142c0a1a2c160b101cddf9
ERROR: md5 of original file: 8df08517a68c930eabf219e74eea860f
INFO: Checking archive/atm/hist/v2.LR.SSP370_0171.eam.h0.2015-01.nc
Traceback (most recent call last):
File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.7.0_nompi/lib/python3.9/site-packages/zstash/extract.py", line 472, in extractFiles
tarinfo: tarfile.TarInfo = tar.tarinfo.fromtarfile(tar)
File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.7.0_nompi/lib/python3.9/tarfile.py", line 1122, in fromtarfile
obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors)
File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.7.0_nompi/lib/python3.9/tarfile.py", line 1066, in frombuf
raise InvalidHeaderError("bad checksum")
tarfile.InvalidHeaderError: bad checksum

[forsyth2]

I moved this from "issue" to "Q&A", since it's asking a question, not requesting a specific feature.

The extract command does allow for specifying which tar to extract (with --tars). However, I don't think we have anything in place for the create command to re-generate a particular tar. @golaz @tangq Have either of you encountered this before / been able to re-create a particular tar?

[golaz]

@nanr, @forsyth2,

This problem arises from the location of the zstash log file (zstash_create_0171_20220725.log). It is in the top level directory, so it gets included in the zstash archive at the same time as it is written to. That causes conflicts. It is best to put the zstash log files under the zstash/ sub-directory so that they don't get archived. Alternatively, they could explicitly be excluded using the --exclude command line option.

[nanr]

This makes complete sense. Is there any way to remedy the situation (i.e. by fixing the first tar file) without recreating the entire archive?

[golaz]

@nanr: there are two options to recover from this: (1) simpler but more time consuming, (2) more complicated, but faster.

(1) If all the files are still on disk, delete the entire zstash archive and start over, placing the zstash log files under the zstash sub-directory. (You may need to create it first). This may take a while, however.

(2) Remove the bad tar file (000000.tar). Remove all the corresponding entries from the database. Run zstash update

In more details:

• Delete (or move) the bad 000000.tar file, both in the cache and on HPSS.
• Delete the zstash log files (to avoid future confusion).
• Edit the database (index.db)

cd zstash
sqlite3 index.db
delete from tars where name="000000.tar";
delete from files where tar="000000.tar";
.exit

Run zstash update <options>. The files that were originally in 000000.tar should be picked up by the update.

[nanr]

Thanks Chris @golaz ~

I'm testing the second option now. I'm cautiously optimistic since it is picking up the same files that were listed in the first tar file. Running the check now. Thanks a million for your help!