libzfs: add keylocation=https://, backed by fetch(3) or libcurl

[nabijaczleweli]

Motivation and Context

#9947

Description

zfs create -o encryption=on -o keylocation=https://nabijaczleweli.xyz/pgp.txt -o keyformat=passphrase zest/test__
zfs load-key -nL https://nabijaczleweli.xyz/pgp.txt zest/test__

Also needs ZTS entries, but not today.
Might also be best to install libcurl-dev on the buildds. The GitHub ones have it, see openzfs/zfs-buildbot#224 for buildbot

The key difference from #9543 is that it only allows https:// and not whatever's on tap. This is better for security (no plaintext passwords), and, once you extend this thought to libcurl, you could, theoretically, download keys off IMAP, telnet, or CIFS. This cannot be backed by fetch(3), nor should it. Anything more complex and/or not required for feature parity with Solaris is really better served by something better adjusted for it (like #11731 or facsimiles thereof).

How Has This Been Tested?

Ran'er through libcurl-dynamic, libcurl-static (well, it's still the .so, but linked directly in), and none.
Also fetch(3) ex situ, rest should be covered by buildds.

I haven't tried this on FreeBSD, but it builds, though I'm not perfectly sure of the error path there. Should be easy enough to test for a FreeBSD user. Works.

I'm also not sure of the availability of the libfetch.so soname, but that is also a question for a FreeBSD enjoyer. Went with libfetch.so.6, available since 8.0-RELEASE.

No back-end: http://build.zfsonlinux.org/builders/Debian%2010%20x86_64%20%28TEST%29/builds/7659/steps/shell_4/logs/log
404: http://build.zfsonlinux.org/builders/FreeBSD%20stable%2F13%20amd64%20%28TEST%29/builds/608/steps/shell_4/logs/log

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv) – only to the internal opaque struct libzfs_handle
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied. – CI take my hand
• All commit messages are properly formatted and contain Signed-off-by.

[nabijaczleweli]

abigail has decided that make storeabi is not a valid way to generate an ABI again, dunno what I'd do about this

[gmelikov]

What's about backward compatibility? Look's like we need a feature flag here. Hope that I've missed something and we don't.

[nabijaczleweli]

When a raw replication stream is zfs-received, you get "cannot set property for 'filling/test__': invalid keylocation", and it falls back to keylocation=prompt, but everything else works as desired.

Importing a pool with a keylocation=https:// dataset works as expected, but doing zfs load-key pool/dset returns "Key load error: URI scheme is not supported".

So this behaves as expected and shouldn't need special a feature gate. Plus, AFAIUI, features are only for on-disk format changes, and this remains compatible and rectangularly in userspace.

[behlendorf]

@jasonbking @kithrup since this builds on the changes from #10218 would you mind reviewing it.

[nabijaczleweli]

From an administrative PoV: what should the https:// URLs in the ZTS point at? I have them point at raw files on the master branch (diverged in the second commit to a previous SHA from here), but it may be preferable to host them elsewhere maybe?

[behlendorf]

what should the https:// URLs in the ZTS point at?

One thing we could do is split the change which adds the PASSPHRASE, HEXKEY, and RAWKEY in to its own commit. Then merge that independently to the master branch and reference that commit form the test cases. That way even in the unlikely case we change these files on the master branch it won't break the test cases in previous releases.

One other minor concern is that the test suite may be run in an environment without network access. In which case it'd be nice if these tests didn't just fail. We can skip the tests by calling log_unsupported if for some reason the URI is unreachable, then add these tests to the maybe section of tests/test-runner/bin/zts-report.py as 'SKIP' (instead of 'FAIL') to indicate if they're skipped don't consider the ZTS run a failure.

[nabijaczleweli]

Right, gated the bits that use the network on ping github.com and added the https://-specific test as maybe SKIP with a bail-out at the top. It's not perfect, of course, since some otherwise-funxional networks block ICMP, but it's the best I can test for with commands.cfg I think.

Regarding the URL bit – I was mostly thinking about using some URL under OpenZFS' namespace and/or control, like openzfs.org/tests/key/{PASSPHRASE,HEXKEY,RAWKEY} or zfsonlinux.org/… (the whois of which suggests they're both owned by the laboratory).
If you'd rather stick with GitHub URLs, then using a separate commit and FFing this branch will work just as well, yeah.

[nabijaczleweli]

These seem to get skipped on GHA, so that works as expected.

checkstyle fails due to an abigail classic:

  [C]'const pool_config_ops_t libzfs_config_ops' was changed to 'pool_config_ops_t libzfs_config_ops' at libzutil.h:59:1:
    type of variable changed:
     entity changed from 'const pool_config_ops_t' to compatible type 'typedef pool_config_ops_t' at libzutil.h:54:1
       'const const pool_config_ops' changed to 'const pool_config_ops'