Default to an archive with same name as the target directory

[todd-a-jacobs]

Expected Behavior

I intuitively expect that when passed a directory named foo that zpaqfranz would create a foo.zpaq archive using the name of the directory as a default. For example:

zpaqfranz a foo

Actual Behavior

This is likely because it's not obvious what a default output archive should be named, but zpaqfranz a foo just prints the usage information. Instead, I have to specify both the directories and files after naming an archive, like so:

zpaqfranz a foo.zpaq foo

This is in contrast to tools like lrztar which will do the "sensible thing" and simply create a sensibly-named archive unless one already exists in which case I need to specify the -f flag to force an overwrite, or the -o flag to specify the output file.

Side Note

It would be nice if zpaqfranz played nicely with lrztar, which many other formats do (including zpaq, although zpipe doesn't currently work correctly on the M2 processor making lrztar with standard zpaq currently unusable).

[fcorbelli]

In case your command does not match the expected syntax, you will get the usage as output.
For example

zpaqfranz l public.zpaq -toz z:\pippo

I chose not to display the help for the individual command because, often, it is very long

zpaqfranz h a

I don't normally use lrztar, I'll take a closer look tomorrow

[fcorbelli]

It would be nice if zpaqfranz played nicely with lrztar, which many other formats do (including zpaq, although zpipe doesn't currently work correctly on the M2 processor making lrztar with standard zpaq currently unusable).

zpaqfranz does support stdin and even ordered archiving (for -stdout)

(whatever) | zpaqfranz a thefile.zpaq thefilename.something -stdin -stdout

[fcorbelli]

I am not really sure
What exactly should do

zpaqfranz a pippero?

if ./pippero folder does exists => if ./pippero.zpaq does NOT exists => archive ./pippero folder inside ./pippero.zpaq
?

[todd-a-jacobs]

Here's what's in my head. I wrote it in Ruby since that's what I know best; treat it as pseudo-code if you prefer.

file_or_dir_arg = "pippero"

if File.exist? "#{file_or_dir_arg}.zpaq"
  warn("file or folder already exists: use '--force' to overwrite it")
  exit 2
else
  output_archive = "#{file_or_dir_arg}.zpaq"
end

if File.directory? file_or_dir_arg
  recursion = true
end

You can always do this in a way that lets people override the default behaviors with flags, like:

File.exist?("#{file_or_dir_arg}.zpaq") && ARGV.include?("--force")

or some such so that explicit CLI flags always take precedence over the default convention-over-configuration. If you have a lot of these, I'd probably do some of this during argument parsing or build a case statement, but that's an implementation detail. Overall, though, I think it would help with making the application easier to use, but (to be entirely fair) there are plenty of applications that force users to specify both an input and an output file.

I also realize (after more time and experimentation with your application) that zpaqfranz is intended more for backups than for a ZPAQ drop-in replacement, so if this doesn't fit your design goals I'll understand. I just tend to think about usability issues, which may not be a priority here.

If you can point me to the right sections, I'd also be willing to help fix the typos and update the wiki. I don't program in C++, so I can't help directly with the code, but I'm always happy to help with documentation!

[fcorbelli]

Every single help is highly appreciated, and my documentation support is modest, to say the least

[fcorbelli]

Coming back to the question, if I understand right, when you do something like

zpaqfranz a pippero

this should happen

1. if pippero folder does exists => update/add into ./pippero.zpaq the ./pippero folder
2. if pippero folder DOES NOT exists => get an error

Do you agree?

[todd-a-jacobs]

@fcorbelli Almost right. See comments and examples below for a (hopefully) clearer set of examples.

With Current Command-Line Signature and No Output-File Flag

Assuming this only works when naming a single target file or directory (which it almost has to currently since the target .zpaq file needs to be named before the files/directories to add to the archive) and if there's no flag to define an output file, then:

1. It's possible that pippero is a file or tarball, in which case it should do the same thing but without the recursion since it's a file and not a directory.
2. There should probably be an ENOENT error whether pippero is a file or a folder since you're trying to add something that isn't there.

Deciding Whether to Delay Errors or Abort Immediately

With an Output-File Flag

However, if there were a flag for naming the zpaq archive and all non-arguments were just files or directories to compress, then you might want to think about whether it makes sense to error immediately before spending a lot of time compressing things if only some of them are missing, or whether (like tar) you want to delay any ENOENT or other errors until after the rest of the archiving is done. For example, if we imagine a -o foo or -o foo.zpaq flag:

# Add pippero1 and pippero2 to foo.zpaq; if pippero1 is missing, report that later
# and compress pippero2 anyway. The user could then call `zpaqfranz a` to add
# the missing/misspelled/incorrect file to foo.zpaq later.
$ zpaqfranz a pippero1 pippero2 -o foo.zpaq

# Add pippero1 and pippero2 to foo.zpaq, which "magically" gets the .zpaq extension
# if it's not already present in the string passed to the `-o` output flag. For example,
# in Ruby:
#
#     extension = ".zpaq"
#     output_file+=extension unless output_file.end_with? extension
#
$ zpaqfranz a pippero1 pippero2 -o foo

then you can either error right away if any files are missing and avoid spending time compressing an incomplete archive, or you could delay errors and compress what you can and add any additional files afterwards. Delaying errors prevents a race condition if files/directories are moved, deleted, or renamed after zpaqfranz starts running (which can happen during a large backup, for example, on a read-write filesystem) but that's true now anyway unless zpaqfranz already handles that use case somehow. So, if you're not delaying errors, you could error out at any time.

When Streaming to Standard Output, Redirections, or Pipes

What to do when the output is standard output or file redirection is less clear-cut. You have the same choices, but sending incomplete data down a long-running pipe or stream might not make a lot of sense, such as trying to stream zpaq-compressed data from stdout to foo.zpaq over an SSH connection, or pipe the stream to a process that doesn't really know how to read or append to .zpaq archives. In that case, what does zpaqfranz do now, and what should it do?

It's kind of a separate question that isn't directly related to the issue I originally raised, but answering your question made me wonder what the optimal behavior for streams with missing compression targets would be from a user's perspective. I'm not 100% sure what's correct, or how it works today, but the slowness of zpaq makes this a bigger issue for zpaqfranz than it does for lzo or gzip.

[fcorbelli]

With Current Command-Line Signature and No Output-File Flag

Assuming this only works when naming a single target file or directory (which it almost has to currently since the target .zpaq file needs to be named before the files/directories to add to the archive) and if there's no flag to define an output file, then:

1. It's possible that pippero is a file or tarball, in which case it should do the same thing but without the recursion since it's a file and not a directory.
2. There should probably be an ENOENT error whether pippero is a file or a folder since you're trying to add something that isn't there.

It seems a very rare situation, the purpose of which (saving the indication of the name of the folder to be added) is not essential.
However may be in the next version it will be implemented

Deciding Whether to Delay Errors or Abort Immediately

With an Output-File Flag

I'm not in favor, because the file naming style is Windows (native), not *nix. So I don't think I will implement such a radical change, the risk of breaking every single script made for zpaq is too high

However, if there were a flag for naming the zpaq archive and all non-arguments were just files or directories to compress, then you might want to think about whether it makes sense to error immediately before spending a lot of time compressing things if only some of them are missing

The (intended) behavior of zpaq and zpaqfranz is to compress everything possible, without being blocked, for example, by a badly indicated folder. This is critical for backup scripts

then you can either error right away if any files are missing and avoid spending time compressing an incomplete archive, or you could delay errors and compress what you can and add any additional files afterwards. Delaying errors prevents a race condition if files/directories are moved, deleted, or renamed after zpaqfranz starts running (which can happen during a large backup, for example, on a read-write filesystem) but that's true now anyway unless zpaqfranz already handles that use case somehow. So, if you're not delaying errors, you could error out at any time.

The above applies. The backup must be completed, in every possible case, even if the files change during the copy.
Incidentally zpaqfranz has a (albeit crude) control over file sizes.
Basically if these change in size, during the copy, a warning is issued ("houston")

38271: HOUSTON expected 9.003.293.316, done 9.003.293.318, diff 2
38271: Corrupted source files? Lost connection? Cannot access? Media full?
38271: =>The updated .zpaq archive is almost certainly incompleted

When you want to verify this type of case (double-check), there is the appropriate v (verify) command which with the -ssd switch (for solid state drives) operates at speeds comparable to the maximum possible for a certain HW

C:\dropbox\Dropbox>zpaqfranz v z:\prova.zpaq -ssd
zpaqfranz v58.4k-JIT-LBLAKE3,SFX64 v55.1,(2023-06-13)
franz:-ssd
z:/prova.zpaq:
1 versions, 22.868 files, 7.585.730.265 bytes (7.06 GB)

Verify hashes of one version vs filesystem (multithreaded)
Total files 19.616 -> in 032 threads -> 19.616 to be checked
--------------------------------------------------------------------
OK       XXHASH64 : 00019615 of 00019616 (     8.38 GB hash check against file on disk)
FAIL     XXHASH64 : 00000001 of 00019616 (   FAILED    hash check against file on disk)
--------------------------------------------------------------------

0.657 seconds (00:00:00) (with errors)

In this real-world example @ more than 10GB/s
The "real" solution, of course, is gettin zfs snapshots BEFORE the backup. This is the "normal" way to consistently update an archive

When Streaming to Standard Output, Redirections, or Pipes

It's kind of a separate question that isn't directly related to the issue I originally raised, but answering your question made me wonder what the optimal behavior for streams with missing compression targets would be from a user's perspective. I'm not 100% sure what's correct, or how it works today, but the slowness of zpaq makes this a bigger issue for zpaqfranz than it does for lzo or gzip.

With regards to slowness, it is the opposite.

When used as a backup, there is simply no comparison (in favor of zpaq) to gzip, lzo, 7z or whatever, by several orders of magnitude in the typical case, which is to update a backup (i.e. add only the changed data) every X time
It is completely normal, with zpaq/zpaqfranz, to have hundreds or thousands of versions inside the same file: unthinkable with gzip, lzo etc

Regarding stdin-stdout

zpaq just doesn't support it, period
Technically, stdout is not doable for a block deduplication system
In fact, zpaq does NOT allow you to mix versioned files with the so called streamed format in a .zpaq file
The zpaq files in streamed format are very slow for any operation (such as listing etc), which can even last tens of minutes

I had to work a lot to get this result (at the price of a significant reduction in the space saved), retaining zpaq-versioned file with monotonic-ordered-based-archived data

Usage is absolutely NOT thought just in a "normal" pipe chain, *nix style

Here the main "scenarios"

stdin side

• Is to make database dump backups (mysql/mariadb for example), where zpaqfranz is monstrously superior to any lzo gzip etc. It's like comparing a bow and arrow to a machine gun
• It also allows sector-level backups of Windows machines, essentially as a kind of "superdd".
  I use it in my work for deduplicated forensic copies (or backup really everything on Windows)
• Finally it is used to archive zfs send streams, for zfs based backups (almost every server I manage runs on zfs)

The stdout

complements for zfs send, to perform zfs receive without having to extract the whole archive, as needed with versioned zpaqs, or direct restoring of mysql's dumps
As mentioned archiving isn't as efficient as it could be, but it saves half the recovery space needed, and about 2/3 the time

[todd-a-jacobs]

@fcorbelli I appreciate your detailed answer above. It seems like zpaqfranz is designed for a different use case than what I had expected, and your answers definitely make sense within that context. I appreciate you taking the time to explain the various reasons.

While I'd still like to see automatic extensions, I agree that it's not absolutely essential. It's mostly a user convenience for those of us who might be using it for use cases other than which it was originally designed. So, I'd still love it if you'd implement that piece of it in future, but you can close the issue for now if you like since my questions were resolved in the sense that they're largely out of scope for the intended use cases.

[fcorbelli]

I am working on a brand-new mode (the fasttxt), I'll embed some changes too

zpaqfranz a foo

if file "./foo" or folder "./foo" does esists, create or update "./foo.zpaq" with the "./foo"
This should not broke scripts

[fcorbelli]

https://github.com/fcorbelli/zpaqfranz/releases/tag/58.5