[6.5.x, 2.2.99git] PANIC: unencrypted block in encrypted object set 2051 #15275
Comments
|
I split the mirror and put the NVME into a USB carrier to access with a laptop. I cannot import it: It has hung here, apparently indefinitely. Hoping I didn't lose the pool. I'm going to update this laptop to zfs head just in case that helps. |
|
After updating ZFS to |
65a
changed the title
65a
changed the title
|
Recovered PANIC error message by grepping the source code, the initial error was in dmu_send, but now the pool cannot be imported with the stack trace above. |
|
EDIT: don't run these commands if you found this bug, see the commands further down that give a clean read only import. All of the -X and -F stuff is not required and could damage your data and leave it unrecoverable. zdb shows history, but I don't have a dateset called |
|
Yeah, encryption is kinda broken on kernel >= 6.3.1, see #15140. Ironically also for using Linux for a Windows VM instance. I've also bisected the first commit that causes my issue in the kernel itself, but as of today no one has pitched in and I'm still stuck at 6.1 LTS. I wanted to try to see if with the latest commit something changed, but I'm not gonna risk my encrypted boot pool now. I'm sorry for what's happened to it, but this issues with encryption should be mentioned in the readme or properly fixed as of now, especially since they are claiming 6.5 kernel compat, which is quite risky to claim if people just using the FS can kill their dataset on their systems. I'm evaluating switching back to a mainline supported FS at this point, at least FS bugs there are properly addressed rather than let linger on for an unknown amount of time. |
|
Those seem quite different, e.g. I am running 2.99.1-git, which is not 2.1 and is 6.5 compatible (re 15258) , and I have a very different stack trace/error and cannot repro the issue in 15140 on a machine running similar software. At this point I am mainly looking for advice on how to get my data back. I suppose I should be just trying the previous command on various txgs until I find one that will import? |
65a
changed the title
|
I managed to get the pool to import readonly using some combination of the flags above, disabling zil replay, and eventually it imported ok. I can't zfs send to my backup pool which is older, but I managed to get a backup using tar, so at least I have my data back. Still no idea what happened here, or why this system's pool was corrupted. |
65a
changed the title
|
If it no longer breaks wildly on a new pool after updating to latest git, I might wildly speculate interactions with #14989. |
|
That's possible, the machine has been running -git for some time, but updated right around then. My recovery efforts just reproduced the same error on a Broadwell-EX e5, but that might be due to pool corruption. I also thinks this still means I may have lost data, but I am experimenting on one side of the mirror only to find a fix or get a clean backup. I do have the original error message now, or something very like it (and tar getting sad right after): |
65a
changed the title
|
As for recovery, I can import the pool with zil recovery disabled in readonly mode, but when it hits a specific file in objset 2051 (naturally, my home directory), it will hang. The code in question is in dbuf.c, and I'm not sure, but it seems like it's lock sensitive code, and I'm not sure how the zfs_panic_recover interacts with that. My hypothesis is that it interacts poorly, so I removed the panic and am trying to rsync my data out again. I was able to interact with the suspected file (received IO error, as expected) without triggering a panic or hang with the panic_recover commented out, which gives me some hope my hypothesis is correct. This would also have indirectly triggered the original issue. My current theory on events is as follows:
|
|
After trying some hacky command line flags, the ZIL was still too corrupt to touch, as zil_parse was eating a bad entry that was causing page faults. I made a local patch to do two things: avoid the panic_recover deadlock (which seems like a real bug?) and tell the zil to not claim log records at all. This may be a terrible idea, but it did allow the pool to import without errors: |
|
I would suspect the unencrypted behavior is probably the same thing that caused a bug report a while ago about managing to sneak an embedded_data block onto an encrypted dataset, which you can't...do. |
|
I know exactly which file was the problem now, and it was moved into a directory shared with Windows 10 over Virtio-FS (which is terrible and often broken/weird). The VM hung, and dmesg had page fault related errors, which I'd seen before, as well as getting the PANIC: Unencrypted block error, I don't remember the precise order. Subsequent files touched after that were also corrupt, including a last minute attempt to rebuild the zfs dkms. @rincebrain since I have the pool imported now and am running a scrub (I got a good rsync of my home directory now, and this is still one half of the mirror, so risks can be taken), does it make sense to checkpoint it and rewind it to try to clean the zil? Or is it likely the bad zil_log_record will be ok after an import/export cycle and a scrub? I'm sort of tempted to see if I can actually save this pool now (though for safety it would be retired after a backup). |
65a
changed the title
|
I wouldn't expect scrub to fix anything in the ZIL - it shouldn't be the case that it won't correct things that could be correctable in ZIL replay. |
65a
changed the title
|
@rincebrain since I have read-only access working, I was able to determine this issue (or at least the version as of pool crash) was version 2.2.99+r83+g12ce45f260, which is commit 12ce45f, and unfortunately already included your SPR errata patch (though SPR9 also looks interesting?). I don't think the QAT change is actually relevant, as this CPU does not have QAT. I also realized my hacky patch is basically doing the same thing as the zfs_recover=1 tunable, so I guess that would have been easier. I don't think I can get zfs send or rw access working on this pool, the space map is hosed. |
|
You can send from a read-only dataset directly, if that would be convenient. You can't use a lot of flags, but you can do it. |
|
With no arguments, I get a vague 'invalid argument' error, so I defaulted to rsync. I did get a full backup though and things seem fine with the data, thankfully this was a workstation and nothing critical. It would be great if #3111 existed, since I'm fairly certain the pool could have survived. I briefly debated trying to see if I could hack the space map code into working for this particular corruption, but it's easier to rebuild at this point. |
|
I have now same problem after sharing one dataset to win10 machine as smb share. I was running also git version of zfs. |
|
@65a Could you describe bit more in detail how were you able to fix the problem or mount the pool in readonly mode? |
|
If you have the same problem, this is a basic summary that worked for me. If it's your boot pool you'll need to use a different machine or a live CD or something.
If 4 doesn't work, you might want to create a new issue, since it probably indicates something different is happening. |
|
Thanks @65a ! Managed to get backup with rsync from live CD by following those instructions. |
|
#15293 seems like a potential root cause here? VM or windows does something that tickles the UAF, corrupts the block pointer for a file, and then because the encrypted bit is not set, triggers the zfs_panic_recover in dbuf.c, which by defaultl (no zfs_recover=1) causes complete IO death, forcing a the user to hard reboot. I'm not sure where my zil corruption factored in, I was able to boot several times with a corrupted dataset before the pool hung on import. |
|
@heppu any stack traces? I haven't tried to repro this yet, but once I have more time I may be able to. |
|
Sorry, I already destroyed the pool and re-created it. How ever it was exactly the the same page fault with |
|
Interesting, so you also saw ZIL corruption? I wasn't sure if that was a secondary effect in my case or not. Did you also get the |
|
Yes, it was ZIL corruption caused by my smb share as far as I understood it but I didn't notice |
|
So I lost another pool for this on voidlinux with freshly released zfs 2.2. stacktrace
[ 659.271649] BUG: unable to handle page fault for address: ffffa0553197e00c
[ 659.271880] #PF: supervisor read access in kernel mode
[ 659.271999] #PF: error_code(0x0000) - not-present page
[ 659.272116] PGD 100000067 P4D 100000067 PUD 1bccd7067 PMD 2c4e46067 PTE 0
[ 659.272240] Oops: 0000 [#1] PREEMPT SMP NOPTI
[ 659.272358] CPU: 94 PID: 6198 Comm: dmu_objset_find Tainted: P OE 6.5.8_1 #1
[ 659.272480] Hardware name: ASUS System Product Name/Pro WS WRX80E-SAGE SE WIFI, BIOS 1106 02/10/2023
[ 659.272604] RIP: 0010:zil_claim_log_record+0x47/0xd0 [zfs]
[ 659.272877] Code: 83 f8 09 74 6c 48 83 f8 18 75 55 48 85 d2 74 50 48 83 7e 40 00 48 8b 6f 38 74 45 45 31 e4 31 d2 48 89 d0 48 c1 e0 07 48 01 d8 40 7c 80 75 22 48 83 78 48 00 75 07 48 83 78 50 00 74 14 48 c1
[ 659.273136] RSP: 0018:ffffa054871939d0 EFLAGS: 00010286
[ 659.273268] RAX: ffffa0553197df90 RBX: ffffa0553189bf10 RCX: 0000000000000000
[ 659.273403] RDX: 0000000000001c41 RSI: ffffffff823164a5 RDI: ffffffff822f5c44
[ 659.273540] RBP: ffff9166100e0000 R08: ffff91678d639540 R09: 9ae16a3b2f90404f
[ 659.273679] R10: ffff9165c44dbd20 R11: 00000000f50b1000 R12: 0000000000001c41
[ 659.273819] R13: ffff916789aec540 R14: ffffffffffffffff R15: 0000000000014e15
[ 659.273961] FS: 0000000000000000(0000) GS:ffff91e2fe580000(0000) knlGS:0000000000000000
[ 659.274105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 659.274248] CR2: ffffa0553197e00c CR3: 0000004178a20000 CR4: 0000000000350ee0
[ 659.274393] Call Trace:
[ 659.274537]
[ 659.274678] ? __die+0x23/0x70
[ 659.274823] ? page_fault_oops+0x159/0x460
[ 659.274970] ? srso_return_thunk+0x5/0x10
[ 659.275116] ? fixup_exception+0x26/0x310
[ 659.275262] ? srso_return_thunk+0x5/0x10
[ 659.275409] ? exc_page_fault+0xd1/0x170
[ 659.275557] ? asm_exc_page_fault+0x26/0x30
[ 659.275708] ? zil_claim_log_record+0x47/0xd0 [zfs]
[ 659.276000] zil_parse+0x54a/0x980 [zfs]
[ 659.276288] ? __pfx_zil_claim_log_record+0x10/0x10 [zfs]
[ 659.276577] ? __pfx_zil_claim_log_block+0x10/0x10 [zfs]
[ 659.276874] ? _raw_spin_lock+0x17/0x40
[ 659.277025] ? srso_return_thunk+0x5/0x10
[ 659.277176] ? zrl_exit+0x4c/0x60 [zfs]
[ 659.277464] ? srso_return_thunk+0x5/0x10
[ 659.277616] ? srso_return_thunk+0x5/0x10
[ 659.277765] ? dmu_objset_open_impl+0x5cf/0x9c0 [zfs]
[ 659.278069] ? srso_return_thunk+0x5/0x10
[ 659.278220] ? preempt_count_add+0x6e/0xa0
[ 659.278370] ? srso_return_thunk+0x5/0x10
[ 659.278520] ? _raw_spin_lock+0x17/0x40
[ 659.278669] ? srso_return_thunk+0x5/0x10
[ 659.278819] ? srso_return_thunk+0x5/0x10
[ 659.278967] ? preempt_count_add+0x6e/0xa0
[ 659.279114] ? srso_return_thunk+0x5/0x10
[ 659.279261] ? srso_return_thunk+0x5/0x10
[ 659.279405] ? srso_return_thunk+0x5/0x10
[ 659.279548] ? dmu_objset_from_ds+0x85/0x160 [zfs]
[ 659.279841] ? srso_return_thunk+0x5/0x10
[ 659.279982] ? dmu_objset_own_obj+0xa2/0xd0 [zfs]
[ 659.280275] zil_claim+0x115/0x290 [zfs]
[ 659.280558] dmu_objset_find_dp_impl+0x145/0x3e0 [zfs]
[ 659.280852] dmu_objset_find_dp_cb+0x29/0x40 [zfs]
[ 659.281140] taskq_thread+0x2c1/0x4e0 [spl]
[ 659.281287] ? __pfx_default_wake_function+0x10/0x10
[ 659.281427] ? __pfx_taskq_thread+0x10/0x10 [spl]
[ 659.281561] kthread+0xf7/0x130
[ 659.281687] ? __pfx_kthread+0x10/0x10
[ 659.281811] ret_from_fork+0x34/0x50
[ 659.281934] ? __pfx_kthread+0x10/0x10
[ 659.282055] ret_from_fork_asm+0x1b/0x30
[ 659.282181]
[ 659.282297] Modules linked in: tcp_diag inet_diag 8021q garp mrp stp llc ipmi_ssif intel_rapl_msr iwlmvm intel_rapl_common amd64_edac snd_usb_audio edac_mce_amd mac80211 snd_usbmidi_lib kvm_amd snd_rawmidi ch341 mc libarc4 usbserial joydev input_leds kvm raid1 eeepc_wmi asus_wmi battery irqbypass md_mod ledtrig_audio sparse_keymap iwlwifi platform_profile snd_hda_intel video rapl snd_intel_dspcfg wmi_bmof snd_intel_sdw_acpi snd_hda_codec acpi_cpufreq btusb pcspkr cfg80211 btrtl snd_hda_core btbcm btintel acpi_ipmi tpm_crb snd_hwdep btmtk snd_pcm tpm_tis ipmi_si i2c_designware_platform tpm_tis_core i2c_designware_core evdev mac_hid tpm i2c_piix4 k10temp ipmi_devintf ipmi_msghandler tiny_power_button sg snd_seq snd_seq_device snd_timer snd soundcore vhost_vsock vmw_vsock_virtio_transport_common vsock vhost_net vhost vhost_iotlb tap hci_vhci bluetooth ecdh_generic rfkill ecc vfio_iommu_type1 vfio iommufd uhid uinput userio ppp_generic slhc tun nvram cuse fuse ext4 crc16 mbcache jbd2 dm_snapshot dm_bufio isofs cdrom
[ 659.282400] squashfs uas usb_storage hid_multitouch hid_generic crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic gf128mul usbhid ghash_clmulni_intel sha512_ssse3 ast aesni_intel sd_mod hid i2c_algo_bit nvme_tcp crypto_simd cryptd nvme_fabrics zfs(POE) xhci_pci drm_shmem_helper ahci xhci_pci_renesas drm_kms_helper libahci xhci_hcd mxm_wmi drm ixgbe ccp mpt3sas mlx4_core libata usbcore spl(OE) rng_core xfrm_algo agpgart raid_class dca usb_common sp5100_tco scsi_transport_sas wmi button sunrpc dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod scsi_common loop btrfs blake2b_generic xor raid6_pq libcrc32c crc32c_generic crc32c_intel
[ 659.284679] CR2: ffffa0553197e00c
[ 659.284829] ---[ end trace 0000000000000000 ]---
[ 659.290489] pstore: backend (erst) writing error (-28)
[ 659.290645] RIP: 0010:zil_claim_log_record+0x47/0xd0 [zfs]
[ 659.290950] Code: 83 f8 09 74 6c 48 83 f8 18 75 55 48 85 d2 74 50 48 83 7e 40 00 48 8b 6f 38 74 45 45 31 e4 31 d2 48 89 d0 48 c1 e0 07 48 01 d8 40 7c 80 75 22 48 83 78 48 00 75 07 48 83 78 50 00 74 14 48 c1
[ 659.291289] RSP: 0018:ffffa054871939d0 EFLAGS: 00010286
[ 659.291459] RAX: ffffa0553197df90 RBX: ffffa0553189bf10 RCX: 0000000000000000
[ 659.291632] RDX: 0000000000001c41 RSI: ffffffff823164a5 RDI: ffffffff822f5c44
[ 659.291806] RBP: ffff9166100e0000 R08: ffff91678d639540 R09: 9ae16a3b2f90404f
[ 659.291985] R10: ffff9165c44dbd20 R11: 00000000f50b1000 R12: 0000000000001c41
[ 659.292154] R13: ffff916789aec540 R14: ffffffffffffffff R15: 0000000000014e15
[ 659.292324] FS: 0000000000000000(0000) GS:ffff91e2fe580000(0000) knlGS:0000000000000000
[ 659.292497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 659.292668] CR2: ffffa0553197e00c CR3: 0000004178a20000 CR4: 0000000000350ee0
[ 659.292843] note: dmu_objset_find[6198] exited with irqs disabled
|
|
Now got the same error after 1 hour creating the pool. There was no SMB shares on pool, how ever there was one NFS share. |
|
And now this happened on completely different machine which should rule out any hardware reason. |
|
Fwiw, i hit the same bug after upgrading my machine from 2.1.x to 2.2.0: Everything seemed fine after the reboot and after about 1h of uptime, i executed a 20 min later, ZFS started to complain about a corruped file in my Maildir folder (a newly delivered mail) No SMB nor NFS involved. |
|
I managed to trigger this now few times on two different machines. Both running voidlinux and zfs 2.2.0 with root on zfs using zfsbootmenu. Running generate-zbm seemed to trigger this at least 3 times. |
|
@heppu you might want to see if that linked PR fixes it for you or not, I'm not longer using ZFS encryption for now. |
|
This should be fixed in #15465, please verify. For the filesystem to remain consistent, the files copied before the patch must be deleted. |
|
I'm glad I saw these bugs before updating; I'll definitely be holding off since I've been using encryption for a long time. Given the severity of these pool corruptions, especially with how easily #15465 can be triggered, should 2.2.0 even be left up as the current release? |
|
Same issue on multiple fresh pools. zfs v2.2.2
|
|
@behlendorf I think this is fixed by #15464 and can be closed or not? |
|
Yes, this is resolved by #15464 which is back backported for a 2.2.1 release. |
|
This does not seem to have been fixed (or I'm testing this wrong). I'm on Ubuntu 23.10. So I think my zfs is recent (I compiled 2.2.2 from scratch): But error is still there (same as with the original Ubuntu Mantic zfs module (2.2.0-rc3). I have a 100% repeatable case of this bug: This could be remnant possibly from an earlier corruption, but not sure how to fix this (zpool scrub is fine). Any hints are appreciated. |
|
Delete the file, or someone earlier commented out the assertion trip to let them read it and delete the original copy. |
System information
Describe the problem you're observing
Machine panicked on accessing certain datasets. First observed reading a VIRTIOFS into Windows (blech) through KVM from a local encrypted ZFS dataset. PANIC followed by several oopses and timeouts. Stopped booting, pool cannot be imported on other machines.
Describe how to reproduce the problem
On this machine, now any high queue depth activity (e.g., compile latest ZFS from HEAD) causes the issue. Unfortunately the machine now hangs at boot, after complaining one file was corrupt. I am going to try to coax this board into allowing an alternate boot medium (unrelated problem) and I can probably recover better logs for this issue, which I will attach.
Include any warning/errors/backtraces from the system logs
PANIC: unecrypted in encrypted...sometime later...
kernel hung task warning for autotrim
(at this point, no FS operations complete, and machine cannot be shutdown normally, power was removed to shut down)
The text was updated successfully, but these errors were encountered: