Restoring failed with "cannot receive incremental stream: invalid backup stream"

[ROCKTAKEY]

Hi, thanks for such a great backup tool. This is the most important tool for me to backup ZFS.

I try to restore a snapshot from B2 backend, but it failed with message:

2024/09/15 22:30:42 Error while trying to read from volume tank/encrypted/main|zfs-auto-snap_daily-2024-09-12-2125.zstream.gz.vol215 - io: read/write on closed pipe
2024/09/15 22:30:42 Error waiting for zfs command to finish - exit status 1: cannot receive incremental stream: invalid backup stream

Note that the vol number 215 is not reproducible. In another execution, it was 333. So each vol does not seem to be really invalid.

I would like to fix it. Would you have any idea?

Information

Procedure

send

• Command for sending

/root/go/bin/zfsbackup-go send --fullIfOlderThan 8760h --raw --replication --logLevel debug --snapshotPrefix zfs-auto-snap_daily tank/encrypted/main b2://<bucket-name>

Backup-ed ZFS informations

zfs get all tank/encrypted/main

NAME                 PROPERTY              VALUE                     SOURCE
tank/encrypted/main  type                  filesystem                -
tank/encrypted/main  creation              Thu Aug  8  2:29 2024     -
tank/encrypted/main  used                  457G                      -
tank/encrypted/main  available             4.75T                     -
tank/encrypted/main  referenced            256K                      -
tank/encrypted/main  compressratio         1.74x                     -
tank/encrypted/main  mounted               yes                       -
tank/encrypted/main  quota                 none                      default
tank/encrypted/main  reservation           none                      default
tank/encrypted/main  recordsize            128K                      default
tank/encrypted/main  mountpoint            /tank/encrypted/main      default
tank/encrypted/main  sharenfs              off                       default
tank/encrypted/main  checksum              on                        default
tank/encrypted/main  compression           on                        default
tank/encrypted/main  atime                 on                        default
tank/encrypted/main  devices               on                        default
tank/encrypted/main  exec                  on                        default
tank/encrypted/main  setuid                on                        default
tank/encrypted/main  readonly              off                       default
tank/encrypted/main  zoned                 off                       default
tank/encrypted/main  snapdir               hidden                    default
tank/encrypted/main  aclmode               discard                   default
tank/encrypted/main  aclinherit            restricted                default
tank/encrypted/main  createtxg             405                       -
tank/encrypted/main  canmount              on                        default
tank/encrypted/main  xattr                 on                        default
tank/encrypted/main  copies                1                         default
tank/encrypted/main  version               5                         -
tank/encrypted/main  utf8only              off                       -
tank/encrypted/main  normalization         none                      -
tank/encrypted/main  casesensitivity       sensitive                 -
tank/encrypted/main  vscan                 off                       default
tank/encrypted/main  nbmand                off                       default
tank/encrypted/main  sharesmb              off                       default
tank/encrypted/main  refquota              none                      default
tank/encrypted/main  refreservation        none                      default
tank/encrypted/main  guid                  14248786990059512964      -
tank/encrypted/main  primarycache          all                       default
tank/encrypted/main  secondarycache        all                       default
tank/encrypted/main  usedbysnapshots       304K                      -
tank/encrypted/main  usedbydataset         256K                      -
tank/encrypted/main  usedbychildren        457G                      -
tank/encrypted/main  usedbyrefreservation  0B                        -
tank/encrypted/main  logbias               latency                   default
tank/encrypted/main  objsetid              345                       -
tank/encrypted/main  dedup                 off                       default
tank/encrypted/main  mlslabel              none                      default
tank/encrypted/main  sync                  standard                  default
tank/encrypted/main  dnodesize             legacy                    default
tank/encrypted/main  refcompressratio      1.00x                     -
tank/encrypted/main  written               0                         -
tank/encrypted/main  logicalused           435G                      -
tank/encrypted/main  logicalreferenced     87K                       -
tank/encrypted/main  volmode               default                   default
tank/encrypted/main  filesystem_limit      none                      default
tank/encrypted/main  snapshot_limit        none                      default
tank/encrypted/main  filesystem_count      none                      default
tank/encrypted/main  snapshot_count        none                      default
tank/encrypted/main  snapdev               hidden                    default
tank/encrypted/main  acltype               off                       default
tank/encrypted/main  context               none                      default
tank/encrypted/main  fscontext             none                      default
tank/encrypted/main  defcontext            none                      default
tank/encrypted/main  rootcontext           none                      default
tank/encrypted/main  relatime              on                        default
tank/encrypted/main  redundant_metadata    all                       default
tank/encrypted/main  overlay               on                        default
tank/encrypted/main  encryption            aes-256-gcm               -
tank/encrypted/main  keylocation           none                      default
tank/encrypted/main  keyformat             raw                       -
tank/encrypted/main  pbkdf2iters           0                         default
tank/encrypted/main  encryptionroot        tank/encrypted            -
tank/encrypted/main  keystatus             available                 -
tank/encrypted/main  special_small_blocks  0                         default
tank/encrypted/main  snapshots_changed     Sun Sep 15 22:45:03 2024  -
tank/encrypted/main  prefetch              all                       default

receive

I try to restore to new zfs, named tank/test3

• Ensure tank/test3 does not exist

  root@hostname:/# zfs list tank/test3
  cannot open 'tank/test3': dataset does not exist
  

• Restore

  /root/go/bin/zfsbackup-go receive --auto tank/encrypted/main@zfs-auto-snap_daily-2024-09-12-2125 b2://<bucket-name> tank/test3 --logLevel debug 2> /root/log/zfsbackup-go.err.log

Log file

• zfsbackup-go-masked.err.log
• Note that the bucket name is replaced with <bucket-name>

Environment

zfsbackup-go

	Program Name:	zfsbackup
	Version:	v0.3
	OS Target:	linux
	Arch Target:	amd64
	Compiled With:	gc
	Go Version:	go1.22.6

zfs

zfs-2.2.4-pve1
zfs-kmod-2.2.4-pve1

zpool

  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 11:15:37 with 0 errors on Sun Sep  8 11:39:38 2024
config:

        NAME                                            STATE     READ WRITE CKSUM
        tank                                            ONLINE       0     0     0
          ata-WDC_WD80EAAZ-00BXBB0_WD-RD0MTKAE          ONLINE       0     0     0
        cache
          nvme-KIOXIA-EXCERIA_PLUS_G3_SSD_9D8KF0JSZ0EA  ONLINE       0     0     0

errors: No known data errors

uname

Linux pve 6.8.8-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.8-2 (2024-06-24T09:00Z) x86_64 GNU/Linux

[ROCKTAKEY]

Because the ZFS I used is too large to test repeatedly, I tried with smaller one.

Even use smaller one, same error occurred.

In addition, when I downloaded all file by hand and used file:// backend, any error occurred.
So it seems to be networking error on on download.