Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how to extract a single file? #63

Closed
luckman212 opened this issue Jul 22, 2023 · 35 comments
Closed

how to extract a single file? #63

luckman212 opened this issue Jul 22, 2023 · 35 comments

Comments

@luckman212
Copy link

luckman212 commented Jul 22, 2023

I read (multiple times) all examples, docs, searched directly the code from the repo... Sorry I couldn't figure it out.

I also looked at #13 but even the example given is not working for me

I am using zpaqfranz v58.6c-NOJIT-L(2023-07-19) on macOS 13.4.1 - installed via Homebrew

Simple example, I have a folder I want to back up:

/Users/luke/important-files/
   file1.txt
   file2.txt
   folderB/
      file3.png

I issue this command:

zpaqfranz a ~/backup.zpaq /Users/luke/important-files

All fine so far... archive is created, and I can also update, seems to work fine

Now I want to restore just file3.png...

I made many failed attempts...

list (ok)

~ $ zpaqfranz l ~/backup.zpaq
zpaqfranz v58.6c-NOJIT-L(2023-07-19)
/Users/luke/backup.zpaq:
1 versions, 6 files, 3.676 bytes (3.59 KB)


- 2023-07-22 20:31:48                   0 d0755 /Users/luke/important-files/
- 2023-07-22 20:32:03               6.148  0644 /Users/luke/important-files/.DS_Store
- 2023-07-22 20:31:38                   4  0644 /Users/luke/important-files/file1.txt
- 2023-07-22 20:31:38                   4  0644 /Users/luke/important-files/file2.txt
- 2023-07-22 20:32:08                   0 d0755 /Users/luke/important-files/folderB/
- 2023-07-22 20:31:53               3.486  0644 /Users/luke/important-files/folderB/file3.png

                9.642 (9.42 KB) of 9.642 (9.42 KB) in 6 files shown
                3.676 compressed

0.001 seconds (00:00:00) (all OK)

extract - 1st try (fail)

~ $ zpaqfranz x ~/backup.zpaq file3.png
zpaqfranz v58.6c-NOJIT-L(2023-07-19)
/Users/luke/backup.zpaq:
1 versions, 6 files, 3.676 bytes (3.59 KB)
Extract 0 bytes (0.00 B) in 0 files (0 folders) / 8 T


0.002 seconds (00:00:00) (all OK)

extract - 2nd try (fail)

~ $ zpaqfranz x ~/backup.zpaq '/Users/luke/important-files/folderB/file3.png' -to /tmp/file3.png
zpaqfranz v58.6c-NOJIT-L(2023-07-19)
/Users/luke/backup.zpaq:
1 versions, 6 files, 3.676 bytes (3.59 KB)
Extract 3.486 bytes (3.40 KB) in 1 files (0 folders) / 8 T
Free space on <<-to /tmp/file3.png>>
is                          0
needed                  3.486
33646: -to is a single file, NOT a folder. Maybe you miss the final /?
Aborting extraction. Use -space to bypass and enforcing.
23013: zpaqfranz error: Not enough free space

0.003 seconds (00:00:00) (with errors)

extract - 3rd try (fail)

~ $ zpaqfranz x ~/backup.zpaq '/Users/luke/important-files/folderB/file3.png' -to /private/tmp/file3.png
zpaqfranz v58.6c-NOJIT-L(2023-07-19)
/Users/luke/backup.zpaq:
1 versions, 6 files, 3.676 bytes (3.59 KB)
Extract 3.486 bytes (3.40 KB) in 1 files (0 folders) / 8 T
Free space on <<-to /private/tmp/file3.png>>
is                          0
needed                  3.486
33646: -to is a single file, NOT a folder. Maybe you miss the final /?
Aborting extraction. Use -space to bypass and enforcing.
23013: zpaqfranz error: Not enough free space

0.003 seconds (00:00:00) (with errors)

extract - 4th try (fail)

~ $ zpaqfranz x ~/backup.zpaq '/Users/luke/important-files/folderB/file3.png' -to /private/tmp/
zpaqfranz v58.6c-NOJIT-L(2023-07-19)
31744: MAGIC: selected 1 file extracting to a folder => merge to /private/tmp/file3.png
/Users/luke/backup.zpaq:
1 versions, 6 files, 3.676 bytes (3.59 KB)
Extract 3.486 bytes (3.40 KB) in 1 files (0 folders) / 8 T
Free space on <<-to /private/tmp/file3.png>>
is                          0
needed                  3.486
33646: -to is a single file, NOT a folder. Maybe you miss the final /?
Aborting extraction. Use -space to bypass and enforcing.
23013: zpaqfranz error: Not enough free space

0.002 seconds (00:00:00) (with errors)

extract - 5th try (fail)

~ $ zpaqfranz x ~/backup.zpaq '/Users/luke/important-files/folderB/file3.png'
zpaqfranz v58.6c-NOJIT-L(2023-07-19)
/Users/luke/backup.zpaq:
1 versions, 6 files, 3.676 bytes (3.59 KB)
00000001 ?existing files skipped (-force overwrites).
Extract 0 bytes (0.00 B) in 0 files (0 folders) / 8 T


0.002 seconds (00:00:00) (all OK)

How do I do it? 🤷

@luckman212
Copy link
Author

luckman212 commented Jul 22, 2023

I figured out that free space detection seems broken on macOS, so if add -space to ignore it, I was able to extract a single file using

zpaqfranz x ~/backup.zpaq '/Users/luke/important-files/folderB/file3.png' -space -to /tmp/file3.png

But, when I try

zpaqfranz x ~/backup.zpaq '/Users/luke/important-files/folderB/file3.png' -to /tmp/ -space -all 4

...it fails. So, new question: is it possible to extract all versions of a single file at the same time as overriding destination with -to?

@fcorbelli
Copy link
Owner

Yes, you can, with ranges
Please read the examples by asking help

zpaqfranz h x

@fcorbelli
Copy link
Owner

And you can quickly select files with only

-only *.png
-only *somefile.jpg

@luckman212
Copy link
Author

Hmm, ok well sorry to have bothered you. I can't figure it out, just doesn't work on my system. But I will keep trying.

@fcorbelli
Copy link
Owner

Relax
Tomorrow I will answer better, it is my fault if something is not easy enough

@fcorbelli
Copy link
Owner

Extract into folder muz7:            x z:\1.zpaq -to z:\muz7\
0-bytes files (check restoration):   x z:\1.zpaq -to z:\muz7\ -kill
Extract into single directory:       x z:\1.zpaq -to z:\muz7\ -flat
Extract without utf,<255,.eml:       x z:\1.zpaq -to z:\muz7\ -utf -fix255 -fixeml
Extract forcing overwrite:           x z:\1.zpaq -to z:\muz7\ -force
Extract version K:                   x z:\1.zpaq -to z:\muz7\ -until K
Extract last versions of multipart:  x "z:\a_???" -to z:\ugo
Extract into muz7 every versions:    x z:\1.zpaq -to z:\muz7\ -all
Show the filelist (if any):          x z:\1.zpaq -filelist
Show the filelist (if any) of v3:    x z:\1.zpaq -filelist -until 3
Extract all *.xls into new archive:  x z:\1.zpaq *.xls -repack onlyxls.zpaq
Extract from a VSS (Windows):        x z:\1.zpaq \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1\path -to d:\output
Replace path in extract              x 1.zpaq -find /tank/ -replace z:\uno\
Change path in extract, longpath     x 1.zpaq -replace z:\uno\ -longpath
Create files' tree                   x 1.zpaq -to z:\testdir\ -debug -kill -zero -longpath
Range-extract version                x copia.zpaq -only *comp.pas -to z:\allcomp -all -range 100-1000
Restore datetime and comments        x copia.zpaq -to z:\prova\ -all -comment
Restore huge image w/smart progress  x copia.zpaq -to z:\prova\ -image
Single file to stdout                x copia.zpaq 00000015.zfs >z:\15.zfs
Extract multiple zpaqs               x vu*.zpaq -to z:\pippo\

In your case, all version of a file, something like

zpaqfranz x copia.zpaq -only *comp.pas -to z:\allcomp -all -range 100-1000

You can use whatever you want for range, better NOT using - (I'll fix the help)

Example

  • -range :10000 (from 1 to 10.000)
  • -range 1000: (from 1.000 to 99.999.999)
  • -range 3 (from 3 to 3)
  • -range 10:100 (from 10 to 100)

The free space test definitely needs to be improved; it doesn't actually work if the folder doesn't exist.
So extracting to /tmp should always go, but extracting to /tmp/nonexistent requires the switch - space
Thanks for the report, I will try to improve

@ruptotus
Copy link

Sorry for little off topic: I sometimes use bandizip (https://en.bandisoft.com/bandizip/) to extract single files. It support zpaq files. It is Windows only, but maybe it will work with wine or something.

@fcorbelli
Copy link
Owner

Well... I use my own GUI extractor :)
PAKKA
Bandizip have a poor multiversion support

@fcorbelli
Copy link
Owner

Please check the attached-prelease (not tested on Mac, only on BSD)
58_7g.zip
You must compile with something like

g++ -Dunix  -O3  zpaqfranz.cpp -o zpaqfranz -pthread

Of course with g++ (/clang) installed. On newer Mac at first invocation everything should be "automagically" download
I am not a regular Mac User, please help me and report :)

@luckman212 luckman212 changed the title how to extract a single file?? how to extract a single file? Jul 23, 2023
@luckman212
Copy link
Author

luckman212 commented Jul 23, 2023

Thank you for trying to help @fcorbelli

I did compile the custom build using the exact command you posted, and it runs successfully 🙏

However, it is outputting errors when I try to extract.

$ zpaqfranz x my-test-backup.zpaq -only "/some/dir/foo.png" -to "/tmp/" -all

zpaqfranz v58.7g-JIT-L(2023-07-23)
franz:-only                 /some/dir/foo.png
--------------------------------------------------------------------------------------------------------------------
franz:-all                                      4
my-test-backup.zpaq:
Skipping block at 378669404:  2249: allocx failed for 8.192
Skipping block at 378669404:  2249: allocx failed for 8.192
Skipping block at 378669404:  2249: allocx failed for 8.192
Skipping block at 378669404:  2249: allocx failed for 8.192

(...error repeats 120 more times...)

1 versions, 0 files, 378.669.404 bytes (361.13 MB)
Extract 0 bytes (0.00 B) in 0 files (0 folders) / 8 T


Extracted 0 files (124 errors) using 0 bytes x 8 threads

0.021 seconds (00:00:00) (with warnings)

Also, for extraction purposes, if we have a tree structure where the file basename alone is not enough to guarantee uniqueness, e.g.

/foo
  /bar
    baz123.png
  /quux
    baz123.png

I guess the correct way to extract only /foo/quux/baz123.png is to use the full path name instead of -only '*/baz123.png'? I thought about using the excellent fzf utility to filter and select files for extraction, but I can't figure out if there is a switch in zpaqfranz list mode that will output purely filenames without all of the extraneous info, headers etc.

@fcorbelli
Copy link
Owner

fcorbelli commented Jul 23, 2023

Thank you for trying to help @fcorbelli

I did compile the custom build using the exact command you posted, and it runs successfully 🙏

However, it is outputting errors when I try to extract.

$ zpaqfranz x my-test-backup.zpaq -only "/some/dir/foo.png" -to "/tmp/" -all

(...)
This seems a broken archive.
What kind of CPU do you have?
Please try a

zpaqfranz autotest -all -to /tmp/whatever

running the .sh

BTW what exactly you want to do, with -only and -all?
Extract ALL the versions of a single file?
In this case you should use range x:y

Also, for extraction purposes, if we have a tree structure where the file basename alone is not enough to guarantee uniqueness, e.g.

/foo
  /bar
    baz123.png
  /quux
    baz123.png

I guess the correct way to exxtract only /foo/quux/baz123.png is to use the full path name instead of -only '*/baz123.png'?

You are right. Of course the "right" way is the full path
You can use -only as a quick-and-dirty access

I thought about using the excellent fzf utility to filter and
select files for extraction, but I can't figure out if there is a switch in zpaqfranz list mode that will
output purely filenames without all of the extraneous info, headers etc.

I do not know fzf

You can try the attached pre-release (quick and dirty, not refined of course) with the
58_7i.zip
new... fzf command

zpaqfranz fzf thefile.zpaq |fzf

On Windows there is the text-based GUI, but it takes too much effort on *nix

@fcorbelli
Copy link
Owner

Asked for help on fzf
junegunn/fzf#3379

@luckman212
Copy link
Author

Thank you for the continued effort!

In my situation, my "goal" would be to extract the last 3 versions, for example. I don't think the range x:y can achieve this, right? It would be something like range -3:

I compiled latest test version from 58_7i.zip, tests seem to pass

$ zpaqfranz autotest -all -to /tmp/foo/
zpaqfranz v58.7i-JIT-L(2023-07-23)
franz:-all                                      4
Self-test for correct internal functioning
Creating autotest folder in <</tmp/foo/>>
Iteration 0/9 chunksize    1.000.000
Iteration 1/9 chunksize      333.333
Iteration 2/9 chunksize      111.111
Iteration 3/9 chunksize       37.037
Iteration 4/9 chunksize       12.345
Iteration 5/9 chunksize        4.115
Iteration 6/9 chunksize        1.371
Iteration 7/9 chunksize          457
Iteration 8/9 chunksize          152
Iteration 9/9 chunksize           50
This seems a LITTLE ENDIAN CPU (aka:'normal')
         BLAKE3 : OK
         CRC-32 : OK
        CRC-32C : OK
     HIGHWAY128 : OK
     HIGHWAY256 : OK
      HIGHWAY64 : OK
            MD5 : OK
    SHA-256-PUT : OK
  SHA-256-WRITE : OK
          SHA-3 : OK
       SHA1-PUT : OK
     SHA1-WRITE : OK
      WHIRLPOOL : OK
           XXH3 : OK
       XXHASH64 : OK
      XXHASH64Y : OK
Time 47.62 seconds for bytes 1.748.003.691


The test batchfile is: /tmp/foo/dotest.sh

47.628 seconds (000:00:47) (all OK)

But, with my old, existing .zpaq archive, these new builds can't operate:

$ zpaqfranz fzf my-test-backup.zpaq

Skipping block at 378669404:  2249: allocx failed for 8.192
Skipping block at 378669404:  2249: allocx failed for 8.192
Skipping block at 378669404:  2249: allocx failed for 8.192
Skipping block at 378669404:  2249: allocx failed for 8.192
(repeated 124 times)

The old v58.6c-NOJIT-L(2023-07-19) version works without errors, so yes I guess this is some kind of compilation problem on my end due to macOS, or other code change.

@fcorbelli
Copy link
Owner

Thank you for the continued effort!

In my situation, my "goal" would be to extract the last 3 versions, for example. I don't think the range x:y can achieve this, right? It would be something like range -3:

The "-" is not good (because confuse the parser)

I compiled latest test version from 58_7i.zip, tests seem to pass
(...)
$ zpaqfranz fzf my-test-backup.zpaq

Skipping block at 378669404: 2249: allocx failed for 8.192
Skipping block at 378669404: 2249: allocx failed for 8.192
Skipping block at 378669404: 2249: allocx failed for 8.192
Skipping block at 378669404: 2249: allocx failed for 8.192
(repeated 124 times)


The old `v58.6c-NOJIT-L(2023-07-19)` version works without errors, so yes I guess this is some kind of compilation problem on my end due to macOS, or other code change.

What kind of CPU do you have? Maybe M1 or M2?
If you DO NOT have an Intel, you MUST insert -DNOJIT when compiling!!

You can see in the header
zpaqfranz v58.6c-NOJIT-L(2023-07-19)
zpaqfranz v58.7i-JIT-L(2023-07-23)

Please try

g++ -Dunix  -O3 -DNOJIT zpaqfranz.cpp -o zpaqfranz -pthread  

@luckman212
Copy link
Author

Ah, yes I am working on M1 and M2 cpus, sorry.

I rebuilt using -DNOJIT and now it is working, no more archive errors when reading!

The initial test of fzf flag output is good, the only thing is: there is a single blank line output at the start. It would be good to eliminate it.

@fcorbelli
Copy link
Owner

Work in progress... :)

@luckman212
Copy link
Author

Awesome! I have not played with fzf on Windows, but there are Windows binaries in case you haven't found them, could be helpful in testing

@fcorbelli
Copy link
Owner

zpaqfranz (for Windows) already include a text-based "GUI" :)

@fcorbelli
Copy link
Owner

gui

@fcorbelli
Copy link
Owner

58_7j.zip
Please check this pre-release

Should

  1. do fzf
  2. list-extract the last N versions with -all -range ::N
    (not tested at all)

@luckman212
Copy link
Author

@fcorbelli this is great! -range ::N is working!

well, I only have 2 comments:

  1. if a zpaq file has e.g. 6 snapshots, and I ask for last 3 versions of file xxx using -range ::3, but file has not changed since v0002, then there is no extraction. I don't know if this is expected. I would expect to get the 2 versions contained from snapshot 0001 and 0002. Not sure if this is possible or not. Otherwise you will need to really know before hand what the last version is where your file was changed.

  2. I have to prefix all -only flag with a * otherwise it doesn't work. Even if I give the exact full pathname like

zpaqfranz x my-test-archive.zpaq -only "/some/dir/foo.png" -to "/tmp/" -all

It doesn't work, I have to use

zpaqfranz x my-test-archive.zpaq -only "*/some/dir/foo.png" -to "/tmp/" -all

Not sure if it's a bug...

@fcorbelli
Copy link
Owner

@fcorbelli this is great! -range ::N is working!

well, I only have 2 comments:

  1. if a zpaq file has e.g. 6 snapshots, and I ask for last 3 versions of file xxx using -range ::3, but file has not changed since v0002, then there is no extraction. I don't know if this is expected. I would expect to get the 2 versions contained from snapshot 0001 and 0002. Not sure if this is possible or not. Otherwise you will need to really know before hand what the last version is where your file was changed.

Bit too complex to do, I don't think I will be implement anytime soon.
It is NOT a "selection", but a DE-selection:rather crude, but enough for my needs

if ((version)<g_rangefrom) || (version)>g_rangeto))
   issel=false;
  1. I have to prefix all -only flag with a * otherwise it doesn't work. Even if I give the exact full

Not sure if it's a bug...

Not a bug, at best a feature :)
You can use as many -onlys as you want, not just one, and they normally specify the full filename. With wildcards you can use as a "trick" to do not write the full path
Some time ago I had thought about a regex parser, but in the end the game was not worth the candle.
With "only" you can quickly catch about everthing

@luckman212
Copy link
Author

luckman212 commented Jul 24, 2023

Not a bug, at best a feature :) You can use as many -onlys as you want, not just one, and they normally specify the full filename. With wildcards you can use as a "trick" to do not write the full path

Ok fair enough, but why does -only /some/dir/foo.png not work? Do I still misunderstand the syntax? I am trying to be as explicit as possible with the extraction, so it seems odd that I have to add the * even when I am passing an exact matching full pathname.

@fcorbelli
Copy link
Owner

Not a bug, at best a feature :) You can use as many -onlys as you want, not just one, and they normally specify the full filename. With wildcards you can use as a "trick" to do not write the full path

Ok fair enough, but why does -only /some/dir/foo.png not work? Do I still misunderstand the syntax? I am trying to be as explicit as possible with the extraction, so it seems odd that I have to add the * even when I am passing an exact matching full pathname.

You are extracting a file to a folder
A file should be extracted to a file, with even a different name

C:\zpaqfranz>zpaqfranz l versioni.zpaq
zpaqfranz v58.7k-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-07-24)
franz:-hw
versioni.zpaq:
4 versions, 20 files, 23.669 bytes (23.11 KB)


- 2022-10-09 15:11:54                   0 A     1.txt
- 2022-09-17 16:56:17                 570 A     2.txt
- 2022-09-18 15:12:54                 874 A     3.txt
- 2022-09-18 15:12:54                 426 A     4.txt
- 2022-10-07 14:10:08                   1 A     c.txt
- 2022-09-08 15:54:43              13.050 A     cpuz.txt
- 2022-09-08 15:54:43                 153 A     lavoretti.txt
- 2022-09-18 19:25:43                 190 A     news.txt
- 2022-09-08 15:54:43              21.966 A     nz.txt
- 2022-09-08 15:54:43               1.398 A     nz2.txt
- 2022-10-03 15:06:06                   0 D     prova/
- 2022-10-03 15:06:12                   0 D     prova/directory3/
- 2022-10-03 15:06:14                   4 A     prova/directory3/versione3.txt
- 2022-10-03 15:05:15                   4 A     prova/file1.txt
- 2022-10-03 15:05:22                   5 A     prova/filea.txt
- 2022-10-03 15:05:48                  14 A     prova/versione2.txt
- 2022-09-08 15:54:43              14.338 A     provapc.txt
- 2022-09-08 15:54:43                 434 A     thelog.txt

A file to a file

zpaqfranz x versioni.zpaq prova/versione2.txt -to z:\ripristinato_versione2.txt

The file(s) inside a folder to a folder

>zpaqfranz x versioni.zpaq prova/directory3 -to z:\newfolder

A file to a folder, creating subfolders as needed

zpaqfranz x versioni.zpaq -only prova/filea.txt -to z:\thetree
C:\zpaqfranz>dir z:\ /s
 
 Directory di z:\

03/10/2022  15:06    <DIR>          newfolder
03/10/2022  15:05                14 ripristinato_versione2.txt
24/07/2023  21:01    <DIR>          thetree
               1 File             14 byte

 Directory di z:\newfolder

03/10/2022  15:06    <DIR>          .
03/10/2022  15:06    <DIR>          ..
03/10/2022  15:06                 4 versione3.txt
               1 File              4 byte

 Directory di z:\thetree

24/07/2023  21:01    <DIR>          .
24/07/2023  21:01    <DIR>          ..
24/07/2023  21:01    <DIR>          prova
               0 File              0 byte

 Directory di z:\thetree\prova

24/07/2023  21:01    <DIR>          .
24/07/2023  21:01    <DIR>          ..
03/10/2022  15:05                 5 filea.txt
               1 File              5 byte

     Totale file elencati:
               3 File             23 byte

@fcorbelli
Copy link
Owner

C:\zpaqfranz>tree z:\
Elenco del percorso delle cartelle per il volume RamDisk
Numero di serie del volume: 000001A2-A619:3A2C
Z:\
├───newfolder
└───thetree
    └───prova

@luckman212
Copy link
Author

I understand the examples you give above, but I think some things are not working right on the macOS version, or at least not working as they do on Windows.

For me, with the following syntax, zpaqfranz is ignoring the -to directive and simply extracting the file(s) to the current directory.

$ zpaqfranz x my-test-backup.zpaq '/Users/luke/some/path/file.png' -to /private/tmp/ -all

Instead of creating a 0001 dir in /private/tmp and extracting there:

/private/tmp
          |
          0001
            └── Users
                └── luke
                    └── some
                        └── path
                            └── file.png

It gets created in my current directory ($HOME):

/Users/luke
         |
         0001
           └── Users
               └── luke
                   └── some
                       └── path
                           └── file.png

Also, in your examples, you have e.g. prova/directory3/versione3.txt. In my .zpaq, all paths are shown absolute, I have no relative paths. So it would be shown as /Users/luke/prova/directory3/versione3.txt

This variant with -only does not match any files unless I add a * in front of the filename, and also add the -space option:

fails:

$ zpaqfranz x my-test-backup.zpaq -only '/Users/luke/some/path/file.png' -to /private/tmp -all

succeeds:

$ zpaqfranz x my-test-backup.zpaq -only '*/Users/luke/some/path/file.png' -to /private/tmp -all -space

Lastly, in fzf discussion you show a sample output (maybe theoretical) where files are listed along with their version#s:

0000999945|0001|file.txt
0000777124|0003|file.txt
0000777125|0002|file.txt

Is this output possible? Current fzf mode outputs only filenames.

@fcorbelli
Copy link
Owner

fzf should show versions with -all

On the rest
For a single file -to does NOT extract to a folder, but to a file

Why? Because dr.Mahoney choose this way
You can find the historic example here

https://mattmahoney.net/dc/zpaqdoc.html

Almost everything is maintained for backward compatibility as a drop-in replacement (not 100% in fact, the output is way different, I do not like zpaq's output at all)

@luckman212
Copy link
Author

luckman212 commented Jul 25, 2023

Ok, I have been playing around with the syntax. I understand now that you don't want to change the syntax to stay as close to Dr. Mahoney's spec as possible. I will say that in all my years of working in the shell, this has to be one of the craziest syntax/arg parsers I've ever seen!

I still don't understand why this command does not work:

zpaqfranz x my-test-backup.zpaq '/Users/luke/some/path/foo.md' -to '/private/tmp/foo.md' -range 0009

zpaqfranz v58.7k-NOJIT-L(2023-07-24)
franz:-range                                 0009
franz:rangefrom (version)                       9
franz:rangeto   (version)                       9
/Users/luke/Backups/my-test-backup.zpaq:
9 versions, 11.007 files, 379.350.254 bytes (361.78 MB)
Extract 0 bytes (0.00 B) in 0 files (0 folders) / 8 T        <<< fail
21571: Path does not exists   /private/tmp/foo.md
21584: Getting free space for /private/tmp/


0.186 seconds (00:00:00) (all OK)

But this one (with -only and *) does:

zpaqfranz x my-test-backup.zpaq -only '*/Users/luke/some/path/foo.md' -to '/private/tmp/' -range 0009

zpaqfranz v58.7k-NOJIT-L(2023-07-24)
franz:-only                 */Users/luke/some/path/foo.md
----------------------------------------------------------------------------------------------
franz:-range                                 0009
franz:rangefrom (version)                       9
franz:rangeto   (version)                       9
/Users/luke/Backups/my-test-backup.zpaq:
9 versions, 11.007 files, 379.350.254 bytes (361.78 MB)
Extract 465 bytes (465.00 B) in 1 files (0 folders) / 8 T    <<< succeed


0.189 seconds (00:00:00) (all OK)

Also, this version (without -only but with *) completely ignores the -to directive and just extracts to the current directory, into a dir called 0009 (?)

zpaqfranz x my-test-backup.zpaq '*/Users/luke/some/path/foo.md' -to '/private/tmp/foo.md' -range 0009

I am using the exact syntax you gave as a "file to a file" in your example above.

For now, I am at least able to extract a specific file @ a specific snapshot version, but there seem to be some things broken here.

@fcorbelli
Copy link
Owner

Ok, I have been playing around with the syntax. I understand now that you don't want to change the syntax to stay as close to Dr. Mahoney's spec as possible. I will say that in all my years of working in the shell, this has to be one of the craziest syntax/arg parsers I've ever seen!

I indeed agree :)
And this is NOT the "craziest thing" within zpaq :)
I can tell you that from time to time I discover things that perplex me, even though I have been developing for more than 35 years :)

I still don't understand why this command does not work:

zpaqfranz x my-test-backup.zpaq '/Users/luke/some/path/foo.md' -to '/private/tmp/foo.md' -range 0009

This is a bug on the new "mac-friendly-get-the-free-space"?
Try -space
But do NOT use -range 9, but -until 9 (later more)

For now, I am at least able to extract a specific file @ a specific snapshot version, but there seem to be some things broken here.
Well, I am pretty confident that it is possible to restore everything inside a zpaq, even on systems without utf or longfilename support (the -flat)
Even millions of files and tens of terabytes. There is also a specific function for a third-party certification, with hashdeep :)

Can you please attach some kind of your file to make some tests?
This would make everything much easier
If there is any correction or improvement I am always ready

Unfortunately, on *nix you don't know what a file is, or even a folder, or even "where" a file is (there are mounts)

So figuring out if /tmp/foo.txt is a file, or a folder, is not trivial, not at all.
Just as it is not trivial to figure out whether it is on the computer's hard drive, and not perhaps a mount of an NFS folder or whatever

The "old" zpaq does not have any kind of verification on written data. If space runs out (which happens frequently in planned backups) the .zpaq file will be silently "truncated," but without zpaq giving an error message [data of course non exctractable]

As you can see this is a gigantic problem (for crontab executions)

So zpaqfranz tries to do three things (which succeed well on Windows, not so well on *nix, but better then nothing)
1- Figure out if the target space is non-zero (you try to write to a filled-up filesystem)
2- Figure out how big a file is expected to be written.
3- Verify that the file being written is as large as it was expected to be

On Windows it is quite simple (ignoring symlinks that actually exist, but are rare), not so simple on network shares

Short version: practically everywhere by putting a -space these tests are not done, just to allow starting even in the case of "confusion"

Another note: to restore a single version the switch is -until. Range does a whole range of other things.
-until means it restores a file (or all files) of version X (aka: all files not deleted from 1 to X)
-range extracts within a single folder many different subfolders, each with files in THAT version. I use it for source versioning

Translation: if in version 9 you have 100 files, and in version 10 you have 3 files, with -until (I am simplifying, there are 1000 different cases, nothing is easy with zpaq) you will get a folder with 103 files

With range you will get a folder with 3 files

@fcorbelli
Copy link
Owner

Let's take this one as a real-world example

zpaqfranz v58.7k-JIT-L,HW SHA1/2,(2023-07-24)
franz:-limit                                   10
replicata_ll3p.zpaq:
798 versions, 174.459 files, 56.289.378.748 bytes (52.42 GB)

 limit to first 10
- 2023-07-02 10:46:26                   0 d0777 /tank/d/
- 2023-07-24 16:41:00                   0 d0755 /tank/d/documenti/
- 2023-07-24 15:00:41                   0 d0777 /tank/d/documenti/ANNA/
- 2023-07-21 14:15:52                   0 d0777 /tank/d/documenti/ANNA/APPUNTI/
- 2023-04-13 16:06:51              96.256  0666 /tank/d/documenti/ANNA/APPUNTI/115 cpc.doc
- 2018-10-18 14:43:20              58.368  0666 /tank/d/documenti/ANNA/APPUNTI/116 cpc.doc
- 2023-06-06 15:06:53              29.184  0666 /tank/d/documenti/ANNA/APPUNTI/1226 cc VALUTAZIONE EQUITATIVA.doc
- 2023-05-17 09:14:11              25.088  0666 /tank/d/documenti/ANNA/APPUNTI/1227 E DOPPIO NESSO CAUSALE - 2016.doc
- 2023-05-17 08:57:17              24.064  0666 /tank/d/documenti/ANNA/APPUNTI/1227 caso particolare CONCORSO VITTIMA SE CI SONO DUE CONCAUSE.doc
- 2023-03-01 10:51:31              46.080  0666 /tank/d/documenti/ANNA/APPUNTI/1227 cc.doc

              279.040 (272.50 KB) of 82.753.797.536 (77.07 GB) in 10 files shown
       56.289.378.748 compressed

20.068 seconds (000:00:20) (all OK)

We want to extract the second to last (the 24.064 bytes long) single file

root@aserver:/zroot/interna # zpaqfranz x replicata_ll3p.zpaq "/tank/d/documenti/ANNA/APPUNTI/1227 caso particolare CONCORSO VITTIMA SE CI SONO DUE CONCAUSE.doc"  -to /tmp/ripristinato.doc
zpaqfranz v58.7k-JIT-L,HW SHA1/2,(2023-07-24)
replicata_ll3p.zpaq:
798 versions, 174.459 files, 56.289.378.748 bytes (52.42 GB)
Extract 24.064 bytes (23.50 KB) in 1 files (0 folders) / 16 T
21571: Path does not exists   /tmp/ripristinato.doc
21584: Getting free space for /tmp/


18.846 seconds (000:00:18) (all OK)

Here it is

root@aserver:/zroot/interna # ls -l /tmp/ripristinato.doc
-rw-rw-rw-  1 root  wheel  24064 May 17 10:57 /tmp/ripristinato.doc

Restore single file WITHOUT space-check (-space) (no 21571 and 21584 in output)

root@aserver:/zroot/interna # zpaqfranz x replicata_ll3p.zpaq "/tank/d/documenti/ANNA/APPUNTI/1227 caso particolare CONCORSO VITTIMA SE CI SONO DUE CONCAUSE.doc"  -to /tmp/ripristinato_nospace.doc -space
zpaqfranz v58.7k-JIT-L,HW SHA1/2,(2023-07-24)
franz:-space
replicata_ll3p.zpaq:
798 versions, 174.459 files, 56.289.378.748 bytes (52.42 GB)
Extract 24.064 bytes (23.50 KB) in 1 files (0 folders) / 16 T


19.448 seconds (000:00:19) (all OK)

If you want to get the X version (of this file) use -until X, NOT range X

@fcorbelli
Copy link
Owner

Now you want to know "where" to find the versions of a file
Here the -only *something dirty trick

We want to find (and get) different version of "something". Please note -only *something and -all

root@aserver:/zroot/interna # zpaqfranz l replicata_ll3p.zpaq -only "*/tank/d/documenti/ANNA/APPUNTI/115 cpc.doc" -all 
zpaqfranz v58.7k-JIT-L,HW SHA1/2,(2023-07-24)
franz:-only                 */tank/d/documenti/ANNA/APPUNTI/115 cpc.doc
--------------------------------------------------------------------------------------
franz:-all                                      4
replicata_ll3p.zpaq:
798 versions, 174.459 files, 56.289.378.748 bytes (52.42 GB)


- 2022-04-22 08:25:38              89.600  0666 0001|/tank/d/documenti/ANNA/APPUNTI/115 cpc.doc
- 2022-11-25 10:01:13              92.672  0666 0153|/tank/d/documenti/ANNA/APPUNTI/115 cpc.doc
- 2023-03-01 10:48:41              94.720  0666 0521|/tank/d/documenti/ANNA/APPUNTI/115 cpc.doc
- 2023-04-13 16:06:51              96.256  0666 0695|/tank/d/documenti/ANNA/APPUNTI/115 cpc.doc

              373.248 (364.50 KB) of 373.248 (364.50 KB) in 4 files shown
       56.289.378.748 compressed

17.315 seconds (000:00:17) (all OK)

If you want the 2nd one you will do a x (blablabla) -until 153

Of course, in "real world", I would prefer something "shorter"

root@aserver:/zroot/interna # zpaqfranz l replicata_ll3p.zpaq -only "*115 cpc.doc" -all 
zpaqfranz v58.7k-JIT-L,HW SHA1/2,(2023-07-24)
franz:-only                          *115 cpc.doc
--------------------------------------------------------------------------------------
franz:-all                                      4
replicata_ll3p.zpaq:
798 versions, 174.459 files, 56.289.378.748 bytes (52.42 GB)


- 2022-04-22 08:25:38              89.600  0666 0001|/tank/d/documenti/ANNA/APPUNTI/115 cpc.doc
- 2022-11-25 10:01:13              92.672  0666 0153|/tank/d/documenti/ANNA/APPUNTI/115 cpc.doc
- 2023-03-01 10:48:41              94.720  0666 0521|/tank/d/documenti/ANNA/APPUNTI/115 cpc.doc
- 2023-04-13 16:06:51              96.256  0666 0695|/tank/d/documenti/ANNA/APPUNTI/115 cpc.doc

              373.248 (364.50 KB) of 373.248 (364.50 KB) in 4 files shown
       56.289.378.748 compressed

Sidenote: running a range 153 extraction will take everything inside the version 153

root@aserver:/zroot/interna # zpaqfranz i replicata_ll3p.zpaq  |grep 00000153
00000153 2022-11-25 14:01:00  +00000139 -00000072 ->            9.538.876

In this example there are 139 files (/folders) inside version 153; 72 previous files/folders no more present, ~9.5MB
If you extract with -until 153 you will get (about) 50GB, every file until 153

Therefore range 153 is NOT just like -until 153, it is FROM 153 TO 153
I hope this is clear (!)

@luckman212
Copy link
Author

luckman212 commented Jul 25, 2023

Thank you @fcorbelli

Indeed, this format works (example)

zpaqfranz x my-test-backup.zpaq "/Users/luke/foo/bar.md" -to "/tmp/restored/bar.md" -space -until 9

I am happy 🙏 ... confused, but happy... which is fine

@fcorbelli
Copy link
Owner

Actually, after a couple of years of digging, it starts to make sense :)

PS should work without -space too. Maybe... who knows? :) :)

@fcorbelli
Copy link
Owner

A bit of explanation on the heuristic

/tmp/restored/bar.md

is /tmp/restored/bar.md an existing folder? => yes, get free space, no => continue, split the path in "pieces", ad rebuild backward until the n-1 (take the "father" of /tmp/restored/bar.md, that should be /tmp/restored)

If /tmp/restored is a path that does exists, then zpaqfranz will "automagically" get the free space on /tmp/restored

If /tmp/restored does NOT exists, then zpaqfranz throws in the towel and "thinks" there is no space

In fact, it is true, a non existend path does not have free space

I didn't make further attempts, otherwise sooner or later you get to "/," and there the space is (generally) always present

Short version: if you are dealing with non-existent output paths, use -space to bypass any checks.

@fcorbelli
Copy link
Owner

I think I can close, at least for now, after explaination and examples

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants