Strengthen Linux kernel capabilities detection #13351

szubersk · 2022-04-20T21:43:39Z

Motivation and Context

Issues found while testing #13339 amended by a discussion in #13341.

Description

Add CONFIG_BLOCK Linux config requirement to
ZFS_AC_KERNEL_CONFIG_DEFINED. OpenZFS won't compile without
that block device support due to large amount of functional
dependencies on it.
Remove dependency on groups_alloc() in
ZFS_AC_KERNEL_SRC_GROUP_INFO_GID to circumvent the missing stub
in Linux 4.X kernel headers.

How Has This Been Tested?

$ make

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Performance enhancement (non-breaking change which improves efficiency)
Code cleanup (non-breaking change which makes code smaller or more readable)
Breaking change (fix or feature that would cause existing functionality to change)
Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
Documentation (a change to man pages or other documentation)

Checklist:

My code follows the OpenZFS code style requirements.
I have updated the documentation accordingly.
I have read the contributing document.
I have added tests to cover my changes.
I have run the ZFS Test Suite with this change applied.
All commit messages are properly formatted and contain Signed-off-by.

behlendorf

Looks good. Thanks!

config/kernel-group-info.m4

- Add `CONFIG_BLOCK` Linux config requirement to `ZFS_AC_KERNEL_CONFIG_DEFINED`. OpenZFS won't compile without that block device support due to large amount of functional dependencies on it. - Remove dependency on `groups_alloc()` in `ZFS_AC_KERNEL_SRC_GROUP_INFO_GID` to circumvent the missing stub in Linux 4.X kernel headers. Signed-off-by: szubersk <[email protected]>

- Add `CONFIG_BLOCK` Linux config requirement to `ZFS_AC_KERNEL_CONFIG_DEFINED`. OpenZFS won't compile without that block device support due to large amount of functional dependencies on it. - Remove dependency on `groups_alloc()` in `ZFS_AC_KERNEL_SRC_GROUP_INFO_GID` to circumvent the missing stub in Linux 4.X kernel headers. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: szubersk <[email protected]> Closes openzfs#13351

- Add `CONFIG_BLOCK` Linux config requirement to `ZFS_AC_KERNEL_CONFIG_DEFINED`. OpenZFS won't compile without that block device support due to large amount of functional dependencies on it. - Remove dependency on `groups_alloc()` in `ZFS_AC_KERNEL_SRC_GROUP_INFO_GID` to circumvent the missing stub in Linux 4.X kernel headers. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: szubersk <[email protected]> Closes #13351

* Improve the inline descriptions of the ARC module parameters These are displayed as the descriptions of the sysctl's on FreeBSD Reviewed-by: George Melikov <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Allan Jude <[email protected]> Closes openzfs#13334 * linux: module: weld all but spl.ko into zfs.ko Originally it was thought it would be useful to split up the kmods by functionality. This would allow external consumers to only load what was needed. However, in practice we've never had a case where this functionality would be needed, and conversely managing multiple kmods can be awkward. Therefore, this change merges all but the spl.ko kmod in to a single zfs.ko kmod. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13274 * scripts: zfs.sh: remove cat Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13274 * scripts: zfs.sh: make usage make sense We don't pass the arguments as arguments Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13274 * scripts: zfs.sh: unload zfs with dependencies Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13274 * linux: module: uninstall legacy modules on (un)installation This can be reverted once we're sure nobody's using them anymore (post-3.0 release?) Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13274 * zpool_history_unpack: return correct errno on nvlist_unpack failure Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Damian Szuberski <[email protected]> Signed-off-by: WHR <[email protected]> Closes openzfs#13321 * Document zfs inherit -S's interaction with noninheritable properties Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Damian Szuberski <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#11894 Closes openzfs#13335 * rpm -> deb doesn't fail when optional packages are missing Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: szubersk <[email protected]> Closes openzfs#13331 Closes openzfs#13336 * man: ... -> … again zfs-program.8 is left, but that's literal Lua syntax Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13255 * FreeBSD: Fix translation from ABD to physical pages In hypothetical case of non-linear ABD with single segment, multiple to page size but not aligned to it, vdev_geom_fill_unmap_cb() could fill one page less into bio_ma array. I am not sure it is exploitable, but better to be safe than sorry. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Reported-by: Mark Johnston <[email protected]> Signed-off-by: Alexander Motin <[email protected]> Closes openzfs#13345 * Corrected oversight in ZERO_RANGE behavior It turns out, no, in fact, ZERO_RANGE and PUNCH_HOLE do have differing semantics in some ways - in particular, one requires KEEP_SIZE, and the other does not. Also added a zero-range test to catch this, corrected a flaw that made the punch-hole test succeed vacuously, and a typo in file_write. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Rich Ercolani <[email protected]> Closes openzfs#13329 Closes openzfs#13338 * contrib: dracut: parse-zfs: drop initqueue-finished for i/f The switch was released in dracut 009 in March 2011, we can safely get rid of the compatibility hook Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13291 * contrib: dracut: parse-zfs: stop pretending we support FILESYSTEM= It was added in the original ae26d04 ("Add dracut support") commit in 2011, and was then broken a bit later with the advent of dracut-zfs-generator, or maybe earlier as part of other churn Either way, it's broken, and has been in 2.0+ as well, and no-one complained. Stop pretending we support it at all Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13291 * contrib; dracut: centralise root= parsing, actually support root=s So far, everything parsed root= manually, which meant that while zfs-parse.sh was updated, and supposedly supported + -> ' ' conversion, it meant nothing Instead, centralise parsing, and allow: root= root=zfs root=zfs: root=zfs:AUTO root=ZFS=data/set root=zfs:data/set root=zfs:ZFS=data/set (as a side-effect; allowed but undocumented) rootfstype=zfs AND root=data/set <=> root=data/set rootfstype=zfs AND root= <=> root=zfs:AUTO So rootfstype=zfs /also/ behaves as expected, and + decoding works Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13291 * contrib; dracut: flatten zfs-load-key, simplify zfs-env-bootfs Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13291 * contrib: dracut: zfs-lib: simplify ask_for_password The only user is mount-zfs.sh (non-systemd systems), so reduce it to what it needs Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13291 * contrib: dracut: zfs-lib: remove find_bootfs Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13291 * contrib: dracut: inline single-use import_pool, move single-use ask_for_password Also don't set ROOTFS_MOUNTED; the final mention was removed in dracut 011 from July 2011 Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13291 * contrib: dracut: don't require essentials to be under the same encroot Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13291 * contrib: dracut: zfs-{rollback,snapshot}-bootfs: order after key loading This fixes at least one race I got with an encrypted root Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13291 * contrib: dracut: zfs-needshutdown: don't list Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13291 * Add dracut.zfs.7 Thorough documentation with a dracut.bootup(7)-style flowchart, dracut.cmdline(7)-style cmdline listing, and per-file docs like the old README Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13291 * contrib: dracut: remove getargbool polyfill It was originally released in dracut 008 in February 2011; we can probably drop it now Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13291 * Strengthen Linux kernel capabilities detection - Add `CONFIG_BLOCK` Linux config requirement to `ZFS_AC_KERNEL_CONFIG_DEFINED`. OpenZFS won't compile without that block device support due to large amount of functional dependencies on it. - Remove dependency on `groups_alloc()` in `ZFS_AC_KERNEL_SRC_GROUP_INFO_GID` to circumvent the missing stub in Linux 4.X kernel headers. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: szubersk <[email protected]> Closes openzfs#13351 * scripts: zfs.sh: explicitly ignore unloaded modules when unloading Reviewed-by: Brian Atkinson <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13356 * scripts: zfs.sh: explicitly unload all modules via rmmod modprobe -r only works for depmodded modules, but this also means we have to re-iterate legacy modules, and in the right order Reviewed-by: Brian Atkinson <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13356 * linux: module: zfs: sysfs: constify types and attrs Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13357 * Linux 5.18 compat: kobj_type.default_attrs replaced with default_groups Upstream-commit: cdb4f26a63c391317e335e6e683a614358e70aeb ("kobject: kobj_type: remove default_attrs") Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13357 * zvol_wait: Ignore locked zvols "When an encrypted zvol is locked the zfs-volume-wait service does not start. The /sbin/zvol_wait should not wait for links when the volume has property keystatus=unavailable." -- https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1888405 Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Damian Szuberski <[email protected]> Thanks: James Dingwall <[email protected]> Signed-off-by: Richard Laager <[email protected]> Closes openzfs#10662 * man: zfs-send.8: fix -X synopses and description Also clean up the horrendously verbose -X handling in zfs_main() Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13352 * tests: cli_user: zfs_001_neg: print the problematic lines Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13352 * Improve zpool status output, list all affected datasets Currently, determining which datasets are affected by corruption is a manual process. The primary difficulty in reporting the list of affected snapshots is that since the error was initially found, the snapshot where the error originally occurred in, may have been deleted. To solve this issue, we add the ID of the head dataset of the original snapshot which the error was detected in, to the stored error report. Then any time a filesystem is deleted, the errors associated with it are deleted as well. Any time a clone promote occurs, we modify reports associated with the original head to refer to the new head. The stored error reports are identified by this head ID, the birth time of the block which the error occurred in, as well as some information about the error itself are also stored. Once this information is stored, we can find the set of datasets affected by an error by walking back the list of snapshots in the given head until we find one with the appropriate birth txg, and then traverse through the snapshots of the clone family, terminating a branch if the block was replaced in a given snapshot. Then we report this information back to libzfs, and to the zpool status command, where it is displayed as follows: pool: test state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A scan: scrub repaired 0B in 00:00:00 with 800 errors on Fri Dec 3 08:27:57 2021 config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 sdb ONLINE 0 0 1.58K errors: Permanent errors have been detected in the following files: test@1:/test.0.0 /test/test.0.0 /test/1clone/test.0.0 A new feature flag is introduced to mark the presence of this change, as well as promotion and backwards compatibility logic. This is an updated version of openzfs#9175. Rebase required fixing the tests, updating the ABI of libzfs, updating the man pages, fixing bugs, fixing the error returns, and updating the old on-disk error logs to the new format when activating the feature. Reviewed-by: Matthew Ahrens <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Mark Maybee <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Co-authored-by: TulsiJain <[email protected]> Signed-off-by: George Amanakis <[email protected]> Closes openzfs#9175 Closes openzfs#12812 * Improve log spacemap load time Previous flushing algorithm limited only total number of log blocks to the minimum of 256K and 4x number of metaslabs in the pool. As result, system with 1500 disks with 1000 metaslabs each, touching several new metaslabs each TXG could grow spacemap log to huge size without much benefits. We've observed one of such systems importing pool for about 45 minutes. This patch improves the situation from five sides: - By limiting maximum period for each metaslab to be flushed to 1000 TXGs, that effectively limits maximum number of per-TXG spacemap logs to load to the same number. - By making flushing more smooth via accounting number of metaslabs that were touched after the last flush and actually need another flush, not just ms_unflushed_txg bump. - By applying zfs_unflushed_log_block_pct to the number of metaslabs that were touched after the last flush, not all metaslabs in the pool. - By aggressively prefetching per-TXG spacemap logs up to 16 TXGs in advance, making log spacemap load process for wide HDD pool CPU-bound, accelerating it by many times. - By reducing zfs_unflushed_log_block_max from 256K to 128K, reducing single-threaded by nature log processing time from ~10 to ~5 minutes. As further optimization we could skip bumping ms_unflushed_txg for metaslabs not touched since the last flush, but that would be an incompatible change, requiring new pool feature. Reviewed-by: Matthew Ahrens <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Alexander Motin <[email protected]> Sponsored-By: iXsystems, Inc. Closes openzfs#12789 * autoconf: Pretend `CONFIG_MODULES` is always on - Unconditionally inject `CONFIG_MODULES` make variable and `#define CONFIG_MODULES` to Kbuild in `ZFS_LINUX_COMPILE` autoconf function to emulate loadable kernel modules support. This allows OpenZFS to perform Linux checks despite `CONFIG_MODULES=n` in the actual Linux config. - Add `ZFS_AC_KERNEL_CONFIG_MODULES` check which encompasses the logic from `ZFS_AC_KERNEL_TEST_MODULE` with additional diagnostic messages to the user - Removed `ZFS_AC_KERNEL_TEST_MODULE` as it merely duplicates every check in `ZFS_AC_KERNEL_CONFIG_DEFINED` - Moved `ZFS_AC_MODULE_SYMVERS` after `ZFS_AC_KERNEL_CONFIG_DEFINED` so the user has a chance to see the proper diagnostic from the steps before. A workaround for Linux's ``` commit 3e3005df73b535cb849cf4ec8075d6aa3c460f68 Author: Masahiro Yamada <[email protected]> Date: Wed Mar 31 22:38:03 2021 +0900 kbuild: unify modules(_install) for in-tree and external modules If you attempt to build or install modules ('make modules(_install)' with CONFIG_MODULES disabled, you will get a clear error message, but nothing for external module builds. Factor out the modules and modules_install rules into the common part, so you will get the same error message when you try to build external modules with CONFIG_MODULES=n. Signed-off-by: Masahiro Yamada <[email protected]> ``` Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: szubersk <[email protected]> Closes openzfs#10832 Closes openzfs#13361 * PPC get_user workaround Linux 5.12 PPC 5.12 get_user() and __copy_from_user_inatomic() inline helpers very indirectly include a reference to the GPL'd array mmu_feature_keys[] and fails to build. Workaround this by using copy_from_user() and throwing EFAULT for any calls to __copy_from_user_inatomic(). This is a workaround until a fix for Linux commit 7613f5a66becfd0e43a0f34de8518695888f5458 "powerpc/64s/kuap: Use mmu_has_feature()" is fully addressed. Reviewed-by: Brian Behlendorf <[email protected]> Authored-by: Colin Ian King <[email protected]> Signed-off-by: szubersk <[email protected]> Closes openzfs#11958 Closes openzfs#12590 Closes openzfs#13367 * zfs: holds: general cleanup Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13373 * zfs: holds: dequadratify Before: 15 0m0.177s 30 0m0.653s 45 0m1.289s 60 0m2.129s 75 0m3.264s 90 0m4.397s 100 0m5.996s 117 0m8.552s After: 30 0m0.053s 117 0m0.125s Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes openzfs#13372 Closes openzfs#13373 * Linux 5.18 compat: replace __set_page_dirty_nobuffers Replace __set_page_dirty_nobuffers with filemap_dirty_folio. Upstream-commit: 6b1f86f8e9c7f9de7ca1cb987b2cf25e99b1ae3a ("Merge tag 'folio-5.18b' of git://git.infradead.org/users/willy/pagecache ") Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Authored-by: Satadru Pramanik <[email protected]> Signed-off-by: Satadru Pramanik <[email protected]> Closes openzfs#13325 Closes openzfs#13380 * Fix O_APPEND for Linux 3.15 and older kernels When using a Linux kernel which predates the iov_iter interface the O_APPEND flag should be applied in zpl_aio_write() via the call to generic_write_checks(). The updated pos variable was incorrectly ignored resulting in the current offset being used. This issue should only realistically impact the RHEL/CentOS 7.x kernels which are based on Linux 3.10. Reviewed-by: Tony Hutter <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes openzfs#13370 Closes openzfs#13377 Co-authored-by: Allan Jude <[email protected]> Co-authored-by: наб <[email protected]> Co-authored-by: Low-power <[email protected]> Co-authored-by: Damian Szuberski <[email protected]> Co-authored-by: Alexander Motin <[email protected]> Co-authored-by: Rich Ercolani <[email protected]> Co-authored-by: Richard Laager <[email protected]> Co-authored-by: George Amanakis <[email protected]> Co-authored-by: Satadru Pramanik <[email protected]> Co-authored-by: Brian Behlendorf <[email protected]>

- Add `CONFIG_BLOCK` Linux config requirement to `ZFS_AC_KERNEL_CONFIG_DEFINED`. OpenZFS won't compile without that block device support due to large amount of functional dependencies on it. - Remove dependency on `groups_alloc()` in `ZFS_AC_KERNEL_SRC_GROUP_INFO_GID` to circumvent the missing stub in Linux 4.X kernel headers. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: szubersk <[email protected]> Closes openzfs#13351

behlendorf approved these changes Apr 20, 2022

View reviewed changes

config/kernel-group-info.m4 Outdated Show resolved Hide resolved

behlendorf added the Status: Code Review Needed Ready for review and testing label Apr 20, 2022

behlendorf added Status: Accepted Ready to integrate (reviewed, tested) and removed Status: Code Review Needed Ready for review and testing labels Apr 21, 2022

behlendorf merged commit 2f31b58 into openzfs:master Apr 21, 2022

szubersk deleted the szubersk-checks branch May 7, 2022 01:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strengthen Linux kernel capabilities detection #13351

Strengthen Linux kernel capabilities detection #13351

szubersk commented Apr 20, 2022

behlendorf left a comment

Strengthen Linux kernel capabilities detection #13351

Strengthen Linux kernel capabilities detection #13351

Conversation

szubersk commented Apr 20, 2022

Motivation and Context

Description

How Has This Been Tested?

Types of changes

Checklist:

behlendorf left a comment

Choose a reason for hiding this comment