Windows 32/64 binaries, 64 bit-HW accelerated

-DNOJIT is now -nojit switch

Now the definition of compilation with -DNOJIT should be obsolete. zpaqfranz automatically detects whether the system supports JIT execution, both at the CPU instruction level and the operating system level. Some systems (e.g., NetBSD) prevent code execution from allocated memory areas (PROT_EXEC). Essentially, the CPU may support JIT, but the operating system doesn't quite agree. It’s not yet fully tested, especially on virtual systems, where the JIT might be incorrectly disabled. Managing all virtualization systems with "fake" CPUs is challenging. I’ll probably add a "forcejit" switch in the next release. In summary: there’s no need to use -DNOJIT during compilation. JIT, as a reminder, accelerates data extraction, while compression speed remains the same.

-tmp

The -tmp switch causes multipart files to be named ".tmp" instead of ".zpaq," renaming them only when they are completed. This allows for running tests and updates in parallel, and also makes it compatible with systems (like Syncthing) that detect file changes and automatically trigger remote updates.

Better password handling

Improved password input handling (with the -key switch), with support (at least theoretically!) for the delete key and cursor movement. When creating an archive, it now prompts for the password twice (which must obviously be the same). Example: zpaqfranz a z:\1.zpaq *.cpp -key.

1980

During creation, the files are stamped with the date 1/1/1980. This makes it easy to identify if the file is incomplete (for example, because the compression was interrupted for any reason, such as a full disk, process kill, and so on). The date is set when the final header, jidac, is written

More placehoderls

On Windows, it's possible to use "placeholders" in file names, such as $pcname, $computername, and $username, which will be automatically replaced. For example, zpaqfranz a z:\pippo_$username c:\nz.

New ZETA hasher / -backupzeta

Now there are (actually, two) new ZETA hashes. They are used to calculate pseudo XXHASH64 hashes in conjunction with the -backupzeta switch of the backup command. OK, it's a rather complicated thing, there's an explanation in the issues. Let me summarize.
When using the backup command, a text file is created (or updated) where MD5 hashes (default behavior) or XXH3 hashes (with the -backupxxh3 switch) are written for integrity checks (they actually also contain QUICK hashes). This requires a re-reading of the written data. That is, when the new chunk is created, for example, okane_00003456.zpaq, zpaqfranz will re-read the okane_00003456.zpaq file (after creating it) to update the index TXT file. Normally, this isn’t a big issue (multipart files are usually small). However, they can become very large (e.g., virtual disks). In this case, the -backupzeta switch will generate a pseudo XXHASH64 hash while generating the archive, WITHOUT needing to re-read the okane_00003456.zpaq file after writing it (it actually also calculates the CRC-32).
As of now, it doesn’t work with encrypted multipart archives (but it will in the future). If anything is unclear, feel free to ask.

-nomore

The -nomore switch disables the internal more command, making external text processing faster. For example, zpaqfranz h h -nomore | less. On 64-bit Windows systems, it also enables support for LargePages (experimental). In reality, it doesn’t improve anything, so I’ll probably remove it in the next release.

Minor fixes for very old compilers (slackware) and others

It’s difficult to maintain compatibility with very old compilers, 32-bit systems, and so on. However, I try to do so with a minimal number of differences compared to the more modern versions. I don’t always succeed, but after all, it’s a hobby project, so please be patient. On the other hand, if anyone has an unused Apple Mx machine, they can let me use it for testing on Apple Silicon

consolidatebackup command renamed to consolidate

List of common switches

You can quickly get with zpaqfranz h common

p7m

On Windows, there is the option to check the FEQ digital signature of hash files for updates. Here's the spiegone https://github.com/fcorbelli/zpaqfranz/wiki/Windows-update

Updated compatibility with HPPA CPUs

Fixed a potential issue with CRC-32 calculation on CPUs that require very strict memory alignment (slower computation, but should be fine for very old and unusual processors, such as 32-bit HP RISC).

IPv6 support (untested)

Using the compilation define -DIVP6 should, and I emphasize "should," make the upgrade command work on IPv6 systems as well. I don't have one available, so I cannot guarantee anything; it's impossible to test and debug. Maybe in the future.

HINT

You can often find small threads about the changes made within the issues, that is, at this link. You might find some "whys" about why certain things are done in a particular way.

https://github.com/fcorbelli/zpaqfranz/issues?q=is%3Aissue

Windows 32/64 binaries, 64 bit-HW accelerated

-backupzeta

This new switch, to be used in the backup command, generates checksums on the fly, without reading them at the end of the .zpaq file creation, saving a lot of time, especially on slow disk drives like HDDs. The generated checksums are quite robust ("almost" XXHASH64 and CRC-32). In the future, it will also support encrypted volumes #139

Improved build compatibility on BSD operating systems

• OpenBSD
• NetBSD
• DragonFly

Set the creation date of .zpaq archives to 1/1/1980

so it is easy to quickly identify those that were not written completely #138

New hash algorithms ZETA and ZETAENC.

Can be selected with -zeta and -zetaenc. You can find the explanation here #139 (comment)

-destination

With the new switch, it is possible to load a series of lines (from a text file) as if they were multiple -to options. The explanation is here #136 (comment)

-nodelete

This switch does not mark files as deleted if they are not found during the path scan. It is used for bulk manipulation of the file list. The explanation is here #136 (comment)

-salt

This switch forces an empty salt (i.e., 32 bytes of zeros). It is something you should NOT normally use. It serves as a development mechanism (i.e., it is something that is useful for ME, not for YOU).

-hdd

With this switch, you use the computer's memory (including virtual memory, i.e., the swap file) to sequentially write the extracted data. It is useful if you have a lot of RAM or an SSD system drive and want to extract to an HDD. In this case, everything will first be decompressed into RAM and then written to the HDD, without any seeks (head movement) in the output. It is not suitable for gigantic files, but for medium sizes, it can halve the extraction time.
#135

-ramdisk

Internal for -hdd

fix on -input

Minor bug fixed on Windows

warnings in yellows

Because sometimes colours... counts...

Everywhere

This release includes numerous new features; hopefully, not too many bugs 😄

• The license of one component has been changed to make it compatible with Fedora policies.
• It's now possible to create a file with a list of errors to reduce confusion in the logs (-errorlog).
• Captchas can be bypassed using -nocaptcha.
• By default, it now uses the number of physical CPUs (no Hyperthread); you can use -ht to bypass this and return to the previous method.
• On Solaris, it (should) now detect the number of CPUs.
• Error messages now appear in red by default.
• UTF8 file output on Windows has been completely redone. New work command added.
• zpaqfranz tracks memory usage (more or less) in the very last output line.
• It shows more understandable error messages (!) when running out of memory due to overly small fragments.
• Output lines have been renumbered.
• With the -input switch, you can load a list of files to be added.
• The -715 switch in the l (list) command restores an output that is nearly identical, binary-wise, to zpaq 7.15.
• In the sfx command (for Windows), it's possible to extract the zpaq from the exe using the -to switch.
• In the sum command, the -home switch replaces -checksum.
• On Windows, the -fixreserved switch removes the ":" character during extraction.
• Sometimes, it heuristically takes the name of an existing backup.

• more

I am, laboriously, updating the wiki, so I do not feel like making a very detailed list of changes
If you want to elaborate, or to report bugs, feel free to open an issue
https://github.com/fcorbelli/zpaqfranz/issues

Windows 32/64 binaries, 64 bit-HW accelerated

This release contains a large number of new features and therefore, as usual, may contain a new bugs.
So be careful, test on your own that the archived files are truly archived correctly.
Some features are introduced to facilitate my work, that is it's possible to obtain the same result with a combination of various other tools (awk, grep etc), but they are complicated to use and fine-tune.
While the *nix logic is do one thing, and do it well, that of zpaqfranz is the opposite
Do everything I need. Because, often, you don't have the tools you would like (e.g. ESXi, NAS and so on)

Better -stdin management

Now files added with -stdin should be deduplicated as well as those added manually (up to the previous version the deduplicator was less efficient, for a whole series of reasons long to explain)

New comparehex command

This command performs an operation that happens very frequently, namely comparing two lines (typically HASH codes) present in two files. If the codes are the same it exits with OK. You can also impose the expected length of the hash.
Attention, all non-HEX characters are discarded

Example:

zpaqfranz comparehex z:\1.txt z:\2.txt "GLOBAL SHA256:" 64

New count command

Counts within more than one file the occurrences of a certain string, and returns OK if the number is the expected one In essence, within a log file, it counts how many positive results there are.
If you don't select a string to search for, it will use the default one for OKs with the -big switch

Example:

zpaqfranz count z:\*.txt 3 "all OK"
zpaqfranz count z:\*.txt 10

Better Control-C handling

During program termination with Control-C press, a better management of the deletion of files created with -chunk is performed.
It might also perform rollback of .zpaq files, maybe in the future.
Apparently it should work in a portable way across various platforms, or at least I hope so.
The problem of terminating a multithreaded system based on a condition is not trivial, even from a performance perspective. I used "dirty tricks", we'll see in the future.

gettempdirectory

Improvement in the creation of temporary files, for the use of multiple zpaqfranz processes simultaneously
Temporary files are created inside different subfolders (marked with mtime) in order to reduce the risk of "collision" if
multiple zpaqfranz are launched simultaneously.

new work command with many verbs

Used to automate many convenient functions for backup logs.
Maybe for most users they are useless, but they make my work a lot easier to prepare easily readable log files.

Write something big                  work big "count the ok"
Write something big                  work big "COUNT THE OK"
Write something big                  work big forza inter
Pad number 123 to 00000123           work pad 123
Pad number 123 to 0123               work pad 123 -n 4
Extract path                         work filepathnotrailing z:\doc\2\3.eml
Write datetime                       work date -terse
Write year_month_day                 work date "%year_%month_%day" -terse
Write -----                          work printbar -terse
Write !!!!!                          work printbar "!" -terse

Newer -crc32 switch for t (test)

Run a triple CRC-32 check (!) against the filesystem
Use -find/replace to fix path (if needed); -ssd for M/T

Translation

The t command (without a path) do NOT read something from filesystem
You have to use the t command WITH a path (or use the -verify switch)

The brand new -crc32 switch is makes a triple CRC-32 comparing, READING from the filesystem.
In this case, the -find and -replace switches can be useful to transform the stored paths into physical paths (if they are different)

What's so exciting about it?
The first CRC-32 is the one calculated by the filesystem during the file addition phase
The second CRC-32 is the one RE-calculated from the blocks of the compressed file.
In other words, it's the CRC-32 of the file once extracted and written to the filesystem
The third CRC-32 is the one that is re-read from the filesystem

If these three CRC-32s match, it means that the file (during processing) has not been modified, and that the compressed file once decompressed will have the same CRC-32 that the file now has in the filesystem.
In short, that by DECOMPRESSING it will be correct

Obviously it is NOT equivalent to the p command (paranoid), to t -paranoid and not even to w, where files are really extracted.
But it's enormously faster, and can process in multithread with the -ssd switch

Triple check CRC-32:                 t z:\\1.zpaq -crc32");
3 check w/path manipulation:         t z:\\1.zpaq -crc32 -find \"x:/memme/\" -replace \"c:/nz/\"");

New -terse switch

Almost everywhere it transforms the output into a version without additional information
That is, it shows the lines typically for redirection to other programs.
If you think about the difference in output between the ls command on Linux and dir on Windows, it will be immediate.

New -csv and -csvhf switches for l (list)

Using these switches you can get an output more easily processable with other programs
In other words, if you want to extract the list of files in a zpaqfranz archive, for example to load it into an Excel file or wherever you want, you can use these switches, using \t instead of TAB.
Keep in mind that -csvhf stands for "csv header footer", that is any string indicated at the beginning and end

For example pippo pluto paperino can become "pippo","pluto","paperino"
Otherwise it would remain pippo","pluto","paperino

TAB-delimited CSV output             l z:\1.zpaq -terse -csv "\t"
Enclose fields between double quote  l z:\1.zpaq -terse -csv "\",\"" -csvhf "\""

-external in add

It is now possible to specify a command that is executed BEFORE the add function, save its output to a text file, and automatically add it to the archive.
It can be used for activities such as creating .zfs snapshots, but also - and above all - to perform an independent check, for truly paranoid people

Translation

Let's say you want to have a... squared certainty that your files are truly identical to the original ones (we are in the hypothesis of using some remote backup system, so you CANNOT compare the files with the local filesystem, because they are not there)

But zpaqfranz can have bugs, right?

You can use zpaqfranz for a long time with the -hashdeep switch, which will generate a file in the
hashdeep format
But it's still using zpaqfranz: any bugs could still affect its reliability
For delicate situations we use an external program, which therefore doesn't have the possible bugs of zpaqfranz, such as hashdeep (in the Windows version hashdeep64 is faster)

Now we will store in a virtual file (of zpaqfranz) VFILE-l-external.txt the list of sha256 hashes generated by hashdeep (or any other similar program).
In this case the string %files and $files is replaced with the files specified in zpaqfranz (they are different for Windows and Linux).
In this example, the command will be executed
c:\nz\hashdeep64 -r -c sha256 c:\nz >temporaryfile.txt
The temporary file will be compressed as usual and added to the version

zpaqfranz a z:\2.zpaq c:\nz -external "c:\nz\hashdeep64 -r -c sha256 %files"

Now let's send the .zpaq file to another computer "somehow".
Let's suppose now we extract the c:\nz folder (object of the example) into the z:\memme folder

zpaqfranz x z:\2.zpaq -to z:\memme

We will now find in the z:\memme folder a folder c (containing the content of the files) AND also the VFILE-l-external.txt file, which contains the output of hashdeep
Now we can use hashdeep "by hand" to test the identity of the stored hashes with those of the folder
If we are not so paranoid there is the zpaqfranz versum command that understands the hashdeep format and allows to manipulate paths
So pay attention to -find and -replace

zpaqfranz versum z:\memme\VFILE-l-external.txt -hashdeep -find "c:\nz" -replace "z:\memme\c\nz" -ssd

-external in the x extract command

You can directly extract the virtual external file, to have an output to redirect

Show external VFILE                  x z:\2.zpaq -external -silent

-silent, in this example, disables the output of zpaqfranz. You can then, if needed, easily redirect for example >mygoodoutput.txt

Various minor fixes

Increased the maximum number of versions shown by the i command

-symlink in add (on Windows)

Ignores symlinks completely. The magical world of NTFS is so complex that every now and then
a more or less unknown functionality pops up. Anyway for one of my clients I need to ignore them, so I do it. It will probably be evolved in the future.

Always show output if there's the -big switch

By limiting the output with various switches the final OK output (gigantic) is still written
It's for the recognition (with the count command) of logs that went well
In essence, it reduces the verbosity of the output, but maintaining the final result

Execution dates are stored in backups

And the testbackup command shows the last one (if it exists)
This way you can understand if an automated backup procedure works
I had to use very dirty tricks to maintain backwards compatibility with older zpaqfranz, but it should work

newer franzomips CPU

Added some new CPUs to the franzomips list

It's possible to set file dates with -touch X

During the a add phase, you can force the timestamp of added files with the X switch
where X represents a date or a date and time
It also works for files added with -stdin

Windows + ARM NAS + Intel NAS + ESXi

Faster Windows 64-bit binary

At least on AMD runs 5% (average) to 20% faster (best case)

C:\zpaqfranz>release\60_1\zpaqfranz a "" c:\dropbox -noeta
zpaqfranz v60.1k-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2024-07-01)
(...)
Files  added +17.218
21.938 seconds (00:00:21) (all OK)

C:\zpaqfranz>zpaqfranz a "" c:\dropbox -noeta
zpaqfranz v60.5d-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2024-07-20)
(...)
Files  added +17.218
18.875 seconds (00:00:18) (all OK)

New switch -stat in add

Show files added/removed/updated

zpaqfranz a z:\test.zpaq c:\zpaqfranz\*.cpp -stat

New switch -stat in i (info)

Show uncompressed data size (slower)

zpaqfranz i z:\test.zpaq -stat

New switch -quick in test with paths (do not hash, only size/date)

zpaqfranz t z:\test.zpaq c:\zpaqfranz\*.cpp -quick

The testbackup command show global SHA256 (of hashes) in verify

Quick look comparing local and cloud backups

Local Synology drive

ash-4.4# /volume1/copie/bin/zpaqfranz_nas testbackup /volume1/copie/cloud/nas_test
zpaqfranz v60.5e-NAS-L(2024-07-20)
franz:testbackup                                _ - command
============================================================================
part0 /volume1/copie/cloud/nas_test_00000000.zpaq i_filename /volume1/copie/cloud/nas_test_????????.zpaq
Multipart backup looks good
Loading backupfile... /volume1/copie/cloud/nas_test_00000000_backup.txt
Rows in backup 00000026 from 00000001 to 00000026
Enabling XXH3 (in reading) hasher

Quick hash check

GLOBAL SHA256: EDBEE1D3D826F07D476B1332E21E0147C0CBE7AE9142E443497BF4D7A8C9DBA5
Chunks checked OK: 26 (233.636.698.925 @ 77.878.899.641.666/s)
0.003 seconds (00:00:00) (all OK)

Remote FreeBSD server.

root@franco:/tmp/zp # /usr/local/bin/zpaqfranz testbackup /home/moz1/cloud/nas_test
zpaqfranz v60.5e-JIT-L,HW SHA1/2,(2024-07-20)
franz:testbackup                                _ - command
*** WARNING: original backup path <</volume1/copie/cloud/>> != current index path <</home/moz1/cloud/>>
*** I highly recommend using -paranoid! ***
*** I highly recommend using -paranoid! ***
*** I highly recommend using -paranoid! ***
============================================================================
part0 /home/moz1/cloud/nas_test_00000000.zpaq i_filename /home/moz1/cloud/nas_test_????????.zpaq
Multipart backup looks good
Loading backupfile... /home/moz1/cloud/nas_test_00000000_backup.txt
Rows in backup 00000026 from 00000001 to 00000026
Enabling XXH3 (in reading) hasher

Quick hash check

GLOBAL SHA256: EDBEE1D3D826F07D476B1332E21E0147C0CBE7AE9142E443497BF4D7A8C9DBA5
Chunks checked OK: 26 (233.636.698.925 @ 38.939.449.820.833/s)
0.006 seconds (00:00:00) (all OK)

GLOBAL SHA256 is the very same (this is GOOD!)

For paranoid people a -verify will ask for a full-sized rehashing of everything

de-overloaded fclose()

Helpful for debugging

minor fix in chunked add

A possible double close file

Windows 32/64 binary, 64 bit-HW accelerated

-nosynology

Switch to disable hidden-system Synology's folders during a

*/@recycle/*
*/#recycle/*
*/#snapshot/*
*/@ActiveBackup/*
*/@sharebin/*
*/@appconf/*
*/@appdata/*
*/@apphome/*
*/@appshare/*
*/@apptemp/*
*/@clamav/*
*/@database/*
*/@docker/*
*/@eaDir/*
*/@maillog/*
*/@MailScanner/*
*/@postfix/*
*/@S2S/*
*/@sharesnap/*
*/@synoconfd/*
*/@SynologyDrive/*
*/@SynoFinder-etc-volume/*
*/@SynoFinder-log/*
*/@tmp/*
*/@USBCopy/*
*/@userpreference/*

Fixes for "strange" things

Should compile and run on Haiku and Macintosh and Solaris

Celeron J4125 bench

Added franzomips for Synology DS224+ NAS

isjitable

Inside the b (benchmark) command a WARNING is issued if the source code is compiled without -DNOJIT, but the CPU does non seems Intel/AMD. Will be evolved in future

Some examples

root@theserverone:~# zpaqfranz b
zpaqfranz v60.4b-JIT-L,HW SHA1/2,(2024-07-12)
This seems Intel/AMD 'normal' CPU
uname x86_64

root@franco:~ # zpaqfranz b
zpaqfranz v60.4b-JIT-L,HW SHA1/2,(2024-07-12)
I am not sure this is Intel/AMD CPU (virtual? arm?)
WARNING: non Intel/AMD CPUs should be compiled with -DNOJIT
uname amd64

With -debug you can get some more infos (if available)

root@franco:~ # zpaqfranz b -debug
38761: error on realpath of zpaqfranz
41554: Pretend to be FreeBSD
43948: This seems a LITTLE ENDIAN CPU (aka:'normal')
zpaqfranz v60.4b-JIT-L,HW SHA1/2,(2024-07-12)
46184: FULL exename <</usr/local/bin/zpaqfranz>>
42993: The chosen algo 3 SHA-1
04058: SSSE3 :NO
04064: SSE41 :NO
04070: SHA   :NO
46776: franz:-debug -verbose
98567: Multiple not-intel vendor_id: Common KVM processor
68456: I am not sure this is Intel/AMD CPU (virtual? arm?)
68467: WARNING: non Intel/AMD CPUs should be compiled with -DNOJIT

In this example the vendor_id (Common KVM processor) is "strange" => warning

There is no easy, portable way across platforms to figure out exactly what type of CPU.
As mentioned, over tim, I will improve

Binary for everywhere

Version 60.3, with a few microfixes for compilability on various systems

Sorry, no Mac today 😄

• zpaqfranz.exe Windows 64 bit
• zpaqfranz32.exe Windows 32 bit
• zpaqfranzhw.exe Windows 64 bit with SHA-1 asm (AMD)
• zpaqfranz_armv8 ARM-NAS compatible build (QNAP, Synology)
• zpaqfranz_esxi ESXi build
• zpaqfranz_freebsd FreeBSD 64 bit
• zpaqfranz_linux32 Generic Linux 32 bit
• zpaqfranz_linux64 Generic Linux 64 bit
• zpaqfranz_nas Generic Linux 64 bit Intel-powered NAS
• zpaqfranz_openbsd OpenBSD 64 bit
• zpaqfranz_haiku Haiku 64 bit

Windows 32/64 binary, 64 bit-HW accelerated

Improved dump command

Show the technical information of the archives in a more readable way (!) and, in particular, the type of archive format
Mind you, it is not meant to work on giant files; it is just a debugging tool. It can therefore crash for large archives (multigigabyte), this is normal, it get a very naive approach. Maybe in the future I will write a better implementation, for now it is enough for me.

There are 3 switches

• : -summary Brief
• : -verbose Show useful infos
• : -all A bit deeper

BTW will show archive compatibility level
60+ = zpaqfranz from v60 and more ~jul 2024
<60 = zpaqfranz up to v59.x
715 = standard zpaq 7.15 (or zpaqfranz -715)

I think it is pretty clear

C:\zpaqfranz\release\60_2>zpaqfranz dump z:\715.zpaq -summary
zpaqfranz v60.2b-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2024-07-03)
(...)
This seems a 715 archive (original zpaq format)

In this example there are older FRANZOBLOCKs with XXHASH64 (older hash, not "b")

C:\zpaqfranz\release\60_2>zpaqfranz dump z:\v59.zpaq -summary
zpaqfranz v60.2b-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2024-07-03)
(...)
FRANZOBLOCK             050          24
HASHTYPE           XXHASH64          24
This seems <60 archive (older zpaqfranz v59)

This is a newer HASH64B ("binary") => a mark 60 zpaqfranz

C:\zpaqfranz\release\60_2>zpaqfranz dump z:\v60.zpaq -summary
(...)
FRANZOBLOCK             050          24
HASHTYPE          XXHASH64B          24
This seems 60+ archive (newer zpaqfranz v60)

This is a more interesting example. It is a mix of small (050) and large (190) blocks , composed of two different hash types, binary XXHASH64 and old whirlpool
It means that by using the -frugal command to append data with a short block format (i.e., with the default hash, i.e., without any particular indication) the program will CRASH . Which is normal (in this case)

C:\zpaqfranz\release\60_2>zpaqfranz dump z:\v60_mixed.zpaq -summary
(...)
FRANZOBLOCK             050          24
FRANZOBLOCK             190          88
HASHTYPE          WHIRLPOOL          88
HASHTYPE          XXHASH64B          24
This seems 60+ archive (newer zpaqfranz v60)
*** WARNING DO NOT USE -frugal adding data to this archive!

Perhaps, in the future, I will put an automatic control. Short version: use the -frugal switch if you know what you are doing, otherwise avoid. It's for advanced users

Minor bug fixing

Could create unnecessarily large backup command files if there were no files to add

Windows 32/64 binary, 64 bit-HW accelerated, Intel/arm based

This branch introduces several new features, also new bugs, so it is up to you to use it with awareness

I understand that the "spiegone" is heavy, however, the alternative is to write nothing at all

The biggest improvement is the use of a new binary format for storing FRANZBLOCKs.

These are (or rather can) be smaller, reducing (though not by much) the overall size of the archive file.

The side effect is the possibility of "breaking" something.
To maintain some degree of safety, I kept all the old switches and added new ones that end in b (as binary)

So the new blake3 storage format is called -blake3b, while the "historical" one (up to version 59) is called -blake3 (as usual)

Warning: if you do not specify any switch the new default binary format -xxhash64b will be used. If you want to use an old format you MUST specify it explicitly

-frugal

There is one additional significant modification, which can lead to program crashes if misused.
It is the -frugal switch, which reduces memory consumption in storing internal structures (significantly)
This makes sense for storing archives with large amounts of files (10 or 100 MILLION) where the memory occupied can become very large, even exceeding the physical memory of the computer (yes, some people use zpaqfranz to store tens of millions of files)

There are no particular problems in using it IF you do not mix different types (in size of FRANZOblocks)

Translation
Suppose you store within an archive files always with the same hash (the default is xxhash64b).
It doesn't matter which one it is, small (e.g., blake3) or large (e.g., whirlpool)
The problem arises when you create an archive with a small hash (blake3 for example) then add data
with a larger hash (ex whirlpool) and then you want to add data with a smaller one (suppose sha3)
This "mix" of formats will crash zpaqfranz if you use the optional -frugal switch.
So if you always use the same hash you have no problem, you can use -frugal (should you have problems with giant numbers of files)
Or don't use it and live happily
Short version: if you use -frugal DO NOT mix different types of hashes.
If you don't have tens of millions of files to add, -frugal is not needed.

Backward compatibility

Barring bugs the new format can be listed, extracted and added by both zpaq and zpaqfranz up to version 59
Archives created with version 60 (and the new binary format) cannot be tested by earlier versions.
Basically everything works, but the t (test) command loses the ability to check CRC32 codes and, in general, all commands cannot read stored hashes (v command and so on).
Therefore if for some reason you want to maintain 100% backward compatibility with version 59 you have to specify the older versions of the switches (e.g. -xxhash64 or -blake3 or maybe -sha3). Those without the final binary "b".

There is a new -date switch that shows the date the files were created (stored in the new format)

The new format, in addition to being smaller when compressed, also stores the date the files were created. Always (on Windows, by default) and optionally for *nix systems (with the -date switch).
This difference is given by the slower processing of this information (modest on Windows, heavy on *nix)
Also - as is well known - there is no "true" method to determine the creation date on non-Windows systems (the so-called birth date) so zpaqfranz uses a heuristic for the various operating systems, with no guarantee of perfect operation (not my fault)

The new format also stores the number of files added, modified, and deleted for each version

Command l list

The l command has been heavily rewritten, result now slower (!) though more beautiful.
Introduces additional information, such as an estimate of file compression ratio, number of files added (+), modified (#) or deleted (-)
Auto-resizes column size width to maximize space for file names
Uses colors by default (disabled during redirection to files either with system variable NO_COLOR or switch -nocolor to disable)
Changes way of showing up with switches -attr, -checksum (e.g., -crc32), -date, -terse)
A lot of operations are performed that slow down, maybe in the future I will put an uglier but faster version.
More informations here

New list and ads: no way

ADS management has not yet been aligned, so, in case you want to force the recalculation, remember that you can force it to

Backup command with -index

I had a request to make the indexes created by the backup command "mobile" (with the -index switch somewhere). That is, to detach the index files from the archives containing the .zpaq data. The purpose is to make it compatible with Worm systems, writing the data inside a "protected" folder and the indexes in a "normal" one
This, however, makes the backup more fragile, as there is nothing to prevent pairing the WRONG indexes with the .zpaq data
It is up to you to be aware of this and operate accordingly. zpaqfranz cannot do miracles
More informations here

t command with parameters and replace/find/to

A user requested the possibility of using the -test switch, with substitution of strings operated by, for example, -find/-replace, during a test command with parameter (!)
It is quite a complex and specific topic, I leave a link with detailed explanations
More informations here

There is a new compilation switch, -DNAS, which is used to make a mixture of reducing

memory usage and maintaining functionality (in short, it is a relaxed ANCIENT)
It serves, typically, to create executables compatible with the popular Synology and QNAP NAS

autotest

The self-test -all -to something command has been extended considerably.
The generated dotest file operates many more tests, in more areas, some of them deliberately to expected failure,
in short to get a picture I hope fairly reliable

I do not exclude that under certain circumstances it may give false positives, that is, detect errors that are not actually there.
To date it is tested on Windows, Debian, FreeBSD, and QNAP-arm, I have not carried out tests on "weird" things like Macintosh, Solaris, and so on.

The completion time is longer (depends on the speed of the system, of course) even a few tens of minutes or hours on cheap NASes and the space needed is about 15GB (maybe it is too big for small VPS, I will see maybe later)

If you have come this far, and you are very confused, patience

If you have problems open an issue on github or send me a direct email
Remember: the continuous improvement (kaizen) process of zpaqfranz depends on YOU
The more reports and suggestions, the better the program will become

Windows 32/64 binary, 64 bit-HW accelerated

-index for backup

It is now possible to use a different folder to store backup's index for WORM storage. Beware, it is up to you to couple the right files!

zpaqfranz backup z:\pippo *.cpp -index c:\temp

-thunderbird flag (Windows)

Automatically adds the two Thunderbird folders (on Windows) inside appdata local and roaming.
Can kill the thunderbird.exe process (before making backups)

In this example get the user utente, killing (if any) thunderbird for a consistent backup

zpaqfranz a z:\email.zpaq c:\users\utente -thunderbird -kill

Bug fixing

Fixed an error for listing non-standard hashes

Support on *nix of NO_COLOR

If you want to disable coloring via a shell variable