Thanks, @pwinckles .

Your approach to '1.' sounds good: Make a new "FITS zip tool".

Regarding '2.', would you mind asking in Slack (c4l#fits-community) and on the list (https://groups.google.com/g/fits-users)? I will check locally as well.

Hi @pwinckles,
You're right that we don't want originalSize and compressionMethod to disappear. They end up in the preservation record via containerMD. The "FITS zip tool" idea sounds good to me.

About the Droid recursion issue, we have no current use for file metadata below the first level. I'm concerned that the increased processing time for recursion would be an issue for some of our partners. I'm also wary of having a forked/hacked version of Droid in there. If there is a way to cleanly limit recursion to 1 level, or to make the depth configurable, I would vote for that. If not, it would be helpful to understand the performance implications of recursion on our typical use cases before deciding next steps.

@awoods @cvicary:

Thanks for the input. I still need to investigate options for limiting Droid's recursion.

Note that with these changes, FITS would now be recursing into the following archive formats, which are all of the formats that Droid supports. I know that you'd expressed interest in having more formats supported in the past. Let me know if you are good with this list, or if you'd like any of them disabled.

ZIP
TAR
GZ
ARC
WARC
BZ
7Z
ISO
RAR
FAT

Additionally, which of those formats would you like supported with the new tool that will be computing originalSize and compressionMethod? This would require unique code per format, and, obviously, it doesn't really make sense for some formats, like tar.

We need to support: zip, gz, and 7z

@awoods @cvicary

Okay, there's some updated WIP code in the branch that limits the recursion to a depth of 1, and also adds back originalSize and compressionMethod. I thought of a fairly easy way of doing this. All but one of the tests is passing now unmodified.

New questions:

1. Can you provide me with example gz and 7z files to add to the test collection?
2. I assume that compressionMethod, which can currently only have the values deflate or stored, only makes sense for zip files? Do your metadata folks have guidance for the other formats, or should I just emit for zip, which would not be a deviation from prior behavior because FITS has historically only supported zip here.

@awoods @cvicary

The one test that isn't passing as is is for https://github.com/harvard-lts/fits/blob/main/testfiles/input/aliceDynamic_images_metadata_tableOfContents.epub

Droid doesn't consider that file to be an epub because it's missing a root mimetype file (spec), so it just treats it as a zip. I'm planning on accepting this because the file is invalid.

Which raises the question of whether or not there is interest in integrating epubcheck into FITS? If so, we could create a separate ticket for that.

I created a draft PR that has what I consider to be final a version of the code, just pending your input on #387 (comment)

@awoods @cvicary Droid 6.7.0 has been released now, and I updated the PR. The PR is still pending answers to these questions:

1. Can you provide me with example gz and 7z files to add to the test collection?
2. I assume that compressionMethod, which can currently only have the values deflate or stored, only makes sense for zip files? Do your metadata folks have guidance for the other formats, or should I just emit for zip, which would not be a deviation from prior behavior because FITS has historically only supported zip here.

I can add made up gz and 7z test files if you'd prefer, but I thought it might be nice if they were representative of what you were interested in processing.

@awoods @cvicary: Pinging this again in the new year. I understand if this is not a priority, just checking in. I have a draft PR up with the droid upgrade (#388). It would be great to include sample gz and 7z test files that demonstrate it working on your archives. However, I could also create some example archives, if that is not feasible.

@pwinckles : Thanks for resurfacing this issue and associated PR. I will work with @cvicary to dig up example/test files for gz and 7z.

For question 2, let's stick with only emitting compressionMethod for zip.

Thanks again.

@pwinckles : Please use the 7z and gz test files found here:

• https://github.com/harvard-lts/drs-translation-service/blob/main/tests/data/epadd-export-test-7z/test.7z
• https://github.com/harvard-lts/drs-translation-service/blob/main/tests/data/epadd-export-test-gz/test.gz

@awoods Thanks!

See the current output for those files here: https://gist.github.com/pwinckles/6b76bee3d37b9142081781ffcfa35c3e

Points of interest:

1. For the 7z file the droid output is not included because its file id does not include a mime type. I can open an issue with them to correct this. (Opened: 7z file format is missing mime type digital-preservation/droid#1065)
2. There is not an agreement on the 7z format name. We see various tools report it as "7-zip archive data", "7-zip archive", and "7Zip format". Do you all have a preference for how this is normalized?
3. For the gz file, the mime is being reported as "application/gzip" and as "application/x-gzip". Would you like this normalized? If so, which do you prefer?