Missing contrib/debian/control when building deb dkms package on Ubuntu 23.10
#15586
Comments
|
Small Update: |
|
@tonyhutter seems to have not yet read the bug reports since 2.2.0 came out that his process for generating release tarballs is producing things that will never work on FreeBSD and fail to build the debs on Linux. Cloning the git repo it should work better. |
|
I managed to compile 2.2.1 a few days ago on one system (Tarball downloaded yesterday 2023-11-26 at 06h27 GMT). Now I am running into the same issue as @FL140 with Tested on two separate Rock 5B SBC (same Software Versions though). They both fail today. Attached is the buildinfo of the successfull build of yesterday, in case it could help somebody. Arch is AARCH64/ARM64 on Rock 5B SBC. openzfs-linux_2.2.1-1_arm64.buildinfo.txt I am not sure if in the meantime some of my toolchain/system tools got updated though. If you want a diff / apt show packages etc I can provide that as well. Just let me know. |
|
That being said the tarball from yesterday seemed to uncompress to zfs-zfs-2.2.1 (weird !), while the one from today seemed to uncompress to zfs-2.2.1. Seems like the tarball on the Release page got changed ... |
|
@FL140 : I just noticed when I installed yesterday I used While today I was using Note that today still the first uncompresses to zfs-2.2.1 while the second uncompresses to zfs-zfs-2.2.1. |
|
https://github.com/openzfs/zfs/releases/download/zfs-2.2.1/zfs-2.2.1.tar.gz seems to untar to zfs-2.2.1/*, for me, while the first one produces zfs-zfs-2.2.1, I assume because it's projectname-tagname dynamically generated. |
|
And in one case you need to run ./autogen.sh while in the other you don't. But even if you run autogen.sh it seems it's not creating the necessary configuration for Debian. ./configure works, make does as well, but buildpkg fails. |
|
In case somebody wants to replicate, I'm trying with this right now. |
|
Trying to make both native deb packages at once breaks, would be my guess about what's breaking for you there. |
|
Yeah keep in mind I was playing around a lot, so I kinda wrote them inside a script. But it doesn't meant it 100% works. It's more to remember what I did
|
|
I managed to compile on one of the systems, the other one failed without very clear message. On the system where it compiled successfully one warning was generated though The exact commands used were (just to make sure I didn't update anything since last time) |
|
On another note, is it possible to omit the "openzfs" prefix when building deb packages ? So that the name would be the same as the "standard" e.g. zfs-dkms package as in APT Repository ? I had a brief look at autogen.sh, configure & Makefile files, but didn't see any obvious "prefix" directive. |
|
The entire point is to make them separate so that you can clearly say "you cannot mix and match these" and not have to worry about people's version numbers overlapping. |
|
So it's a "feature" not a "bug", alright, I guess I can live with that |
|
After a bit of sleep and reinvestigating again it turns out the source of the problem is exact the same as in #15404 But marking it as a pure duplicate is probably not the best thing as the the two scripts provided here will provide a way to compile the package for Ubuntu 22.10 (hopefully). I will merge the new information from the issue above and provide an update. |
|
@luckylinux I will try to merge our scripts to make it more generic. And I think we should switch over to the Just a small hint, you should prefere |
|
@FL140 actually I thought that apt-get like apt-cache was getting deprecated in favor of simply apt. I many script (besides this one) I actually use aptitude. I just saw today that apt is not really recommended in scripts. Duh. Also sometimes apt and aptitude yield different dependencoes resolution... |
I also got recently bite by that, that's why I wrote the comment here in the first place. I also prefer |
|
The following script builds an Ubuntu package for 2.2.2 from tonyhutter and will be updated to the official branch after the official 2.2.2 release. Note: currently the script exits prior the deb package installation because either dracut or initramfs is used, act according to your setup. |
|
Apparently 2.2.2 is released, but if I got (using my script) using git clone approach Then it compiles to 2.2.99 for some reason |
|
@luckylinux please be careful with 2.2.2 and/or the above script for now! I ran into a bad pool crash and currently can't say where it comes from! see: |
Well you can try my script from above, it is mostly comments to keep track of what has been changed and why. You only need to comment out the The script currently exits prior the actual deb package install because the packages depend on if you use dracut or initramfs. Just move the one you don't need out of the way. Apart from a few |
|
Well with my script & initramfs it just finished installing the packages - at which point I saw your comment about the pool crash -_-. |
|
My updated script for ZFS 2.2.2 and AMD64 with apt-get instead of apt (thanks again !) |
Sorry, we are writing in parallel and I am currently very tired and therefor, slow.... ;-( BUT the panic occured during running of the block clone info script. I booted again and I am running on 2.2.2 here since that (1-2h) without any complaints AFAIK. |
|
OK, so apart from a few debug error messages I can't recall from the tonyhutter branch The building of the deb packages went just fine. |
|
I'm also tempted to try with the newly built ZFS 2.2.2 - wish me luck :D |
Same here official 2.2.2 compiles and installed fine (using dracut with a custom EFI setup and NOT the official GRUB / initramfs combo). Reboot was uneventful and system is up and running so far, I just don't dare to run the block clone info script for now. ;-) |
|
Well that's promising. But I'm starting to get more and more disappointed that a file system that proud itself for preventing data corruption now (and not for the first time) is causing data corruption after an update. Bugs seems to cause more data corruption than bad ram / disks / cables it seems |
|
I'm not really sure where you're getting the idea that the update is causing more corruption, and not that the existing issue on disk is causing a panic when you try to read it later. It's hard to guess without the actual crash message, but that would be my bet. Also, zdb doesn't touch the running kernel, so if something is causing the running kernel to panic, it's not directly zdb. It might be that the script is walking the filesystem and hitting metadata that's munged, or something, but zdb itself can't mangle your bits. |
Well TBH I also had to hold my temper over the last two weeks sometimes, but at the end everyone is just trying to get over with the damage done. To be fair the underlying bug is time critical and those are very hard to find if you don't know what you are looking for. Over the last decades of unsing file systems I ran into a bunch of very unpleasant things with different file systems (ntfs, xfs, extN,...) so nothing is perfect. But I have to admit the amount of bugs that manifested over the last two weeks where a lot and I wonder if more in depth (heavy parallel load!) testing would have saved us from what we are all in now. |
Yeah, seeing the output in the other issue it's either crashing when walking the filesystem or when trying to find the path/filename of a specific object. |
I also think, the pool panic came from the data that got corrupted between 2.2.0 and 2.2.1 but I need to investigate as already written. The "panic" was related to the pool not the kernel. So while the machine crashed, I expect that happened because the root pool was put in panic mode. And not due to a kernel crash, but I also have to investigate that further. But after 16h into this I need a break. |
I am thankful I wasn't hit yet (knock on wood)... Yesterday I once again got stuck with podman / docker because of ACLs permission working weirdly with zfs. Then programs left hanging and only solution is to reboot once every 10 minutes. I don't know... It just feels like everything keep breaking. You should always update for security and vulnerabilities reasons, but if every time stuff breaks 🙄 |
Well that is my biggest taking after the last two weeks that all my remote backups are also in danger, which is the worst case scenario. /* paranoid mode on / So in the future I think I will additionally make backups with and rsync script (as prior zfs) to a remote site with having the backup on a different filesystem. Just in case everything blows up at once. / paranoid mode off */ |
No, you should wait a little while and see what breaks, unless it fixes something that you need in your environment, and either way, test it somewhere less important before updating the important things. I'm not saying people don't try to make things reliable to the best of their ability, but there are reasons I advise people to not install the latest and greatest immediately on things where any breakage could be bad. By definition, any amount of testing is going to be less than "everyone running it on all their varied systems" for anything popular enough, and no matter how well you test it, your tests are only as good as how varied the places you run them are. If it's not broke, don't fix it. (None of the above is specific to ZFS in any way - just, you know, the lower your tolerance for risk, the more cautious you need to be.) |
I agree to some extent to the "if it isn't broken, don't fix it" approach. I am usually not that scared of updating Debian for instance. Track record is good so far (knock on wood). Ever since Ubuntu managed to self-uninstall of system tools during a distro upgrade I have become vary and skeptical of it. But ZFS I used to have more confidence. Sometimes upgrade is required. But then you have to upgrade kernel. But zfs is not supported natively. Then you need to manually compile zfs. And so on and so forth. Then I have to upgrade podman, but I also need all of the other tools etc. I am usually quite happy with the distro shipped version of zfs. Except this buggy one on Ubuntu which is still a development version nonetheless 🙄. |
That is exactly the problem for users of Ubuntu. And what is really unacceptable IMHO is how all of the last two weeks is just ignored by Ubuntu. The know of it because 2.2.1 is there for nobel, but still no hotfix 2.2.1 for mantic. I am not even starting to talk about 2.2.2. So while the testing with the 2.2.2 patches suggest that the "block cloning" bug is very rare in real life for everything prior 2.2.1 and fixed in 2.2.2. I at least can confirm serious problems (pool panic) with a real world notebook, non heavy used and Ubuntu 23.10 after Upgrade to 23.10 (2.2.0 rc3? IIRC, as shipped with Mantic). It is still too early to tell what really is the exact problem, but 2.2.0 was the cause for 99% (the pool gets regular scrubs at least once a month and the pool panic is caused by accessing BRT information with resulting system freeze) and since "we just don't know yet" and need to look further into this, just ignoring those Updates, is not a smart move, to put it mildly, by Ubuntu. So since I have hard evidence now, I am going to fill a bug report with Ubuntu, but I just can hope this one gets taken more seriously finally by Ubuntu. |
It is not, since the crash you're seeing happens in a different command that dumps the list of all blocks* without any BRT-related information *rather, it dumps the whole list of all objects which includes all blocks but none of the information from the BRT |
@0x5c Sorry looks like I am not so deep into this to see that on first glance, but fact is that this simple script triggers a pool panic and a following system freeze. All on a notebook running stock Ubuntu 23.10 (fast installed 2.2.1 and now with 2.2.2) which was scrubed on a regular base without prior errors. For sure this was OK prior updrade from 23.04 to 23.10. This is good enough for me to take that all very seriously. So any help on nailing this down is highly appreciated! |
@luckylinux after having some sleep and having I deeper look into the zdb man page I should be now up dot date. So what is your problem and with what do you need help? |
|
Cleaned up script for building the deb packages: The script builds 2.2.2 with initramfs in default mode. But you can provide the version to build and the build tool (either initramfs or dracut). The script was tested sucessfully on Ubuntu 23.10 running a personalized dracut based boot process using UKI. (Which shouldn't matter for you, initramfs is just not tested.) |
I described it in the other thread actually #15404 (comment) But TLDR is I cannot get the zdb script to work and list the corrupted blocks (if any). |
Well actually, besides a few debugging differences (my script is just "open-loop" command after command, yours actually handles some of the errors which might occur) we should be getting the same result. I'm not that advanced in bash 😉. I can also test yours. Only question is does it build 2.2.2 when checked out from git with the tag? It built 2.2.99 for me, that's why I fell back on the tar tarball which is working (if you download the correct one). |
I tried your script. Attached the build log. Last ~ 10 lines of the log are most relevant. |
|
apt-get needs full paths, not relative. |
With my script it also works with relative paths: (it was already installed - re-ran the command to prove my point) |
|
The file pattern for apt-get is perfectly fine. so instead of @FL140 I'm not sure what the consequences, are regarding the package build, if you try to build it without the dh-dkms package. Also: results in a permission denied error, when trying to change the user/group. |
You can pass the version you want to build in as the first parameter, it uses 2.2.2 as default see I went the git clone path since the tar wasn't working at the time and in addition with the git clone approach it is possible to apply additional commits. |
Well the last minute changes you know ;-), I had the popd exactly before the cleanup, but I moved it with the idea to also clean up the build directory earlier and make an option to keep the deb files. But it was verly late that day... Anyways will have look again over it.
hmmm, IDK how we handle that on older releases on 23.10 it's there, this will need a bit of investigation if someone needs it. I will also test it on Debian today.
Well if the script is called as root no, otherwise yes. I had it there to prevent normal users entering the directory. But you are right, it's better to run the build process as normal user and sudo install later. |
|
The recently tagged 2.3.0-rc1 release included a couple of fixed. It'd be very helpful if anyone who had issues could try again so we can work through any potential removing issues. We'll backport these for 2.2.7. |
System information
Describe the problem you're observing
I am trying to build a deb zfs dkms package for Ubuntu 23.10 from the release 2.2.1 tar package from GitHub according to the recommendation given in #15529 (comment) if followed the instructions given in https://openzfs.github.io/openzfs-docs/Developer%20Resources/Custom%20Packages.html .
I am trying this because of the block clone bug and the following fixes introduced with 2.2.1, unfortunately there is still no package for Ubuntu available and I need to fix the systems I am responsible for ASAP.
Unfortunately it seems as the instructions and release package have issues.
The following issues were found:
contrib/debian/controlis missing in the release tar package.Get the tar package directly from GitHub.
zfs-${OPENZFS_VERSION}notzfsalienpackage (and others?) should no longer be needed since 2.2.0 (if I understand the instructions correctly). (Note: The control file not found error pops up with and without packagealieninstalled.)Only the first issue needs solving, I already solved the other issues (except that the documentation needs an update).
Describe how to reproduce the problem
I wrote an installation script:
The command
make native-deb-utilstriggers the error:Include any warning/errors/backtraces from the system logs
Script output:
The text was updated successfully, but these errors were encountered: