Introduce experimental "rpm-ostree jigdo"

Tracking issue: coreos#1081

To briefly recap: Let's experiment with doing ostree-in-RPM, basically the
"compose" process injects additional data (SELinux labels for example) in an
"ostree image" RPM, like `fedora-atomic-host-27.8-1.x86_64.rpm`. That "ostree
image" RPM will contain the OSTree commit+metadata, and tell us what RPMs we
need need to download. For updates, like `yum update` we only download changed
RPMs, plus the new "oirpm". But SELinux labeling, depsolving, etc. are still
done server side, and we still have a reliable OSTree commit checksum.

This is a lot like [Jigdo](http://atterer.org/jigdo/)

Here we fully demonstrate the concept working end-to-end; we use the
"traditional" `compose tree` to commit a bunch of RPMs to an OSTree repo, which
has a checksum, version etc. Then the new `ex commit2jigdo` generates the
"oirpm". This is the "server side" operation. Next simulating the client side,
`jigdo2commit` takes the OIRPM and uses it and downloads the "jigdo set" RPMs,
fully regenerating *bit for bit* the final OSTree commit.

If you want to play with this, I'd take a look at the `test-jigdo.sh`; from
there you can find other useful bits like the example `fedora-atomic-host.spec`
file (though the canonical copy of this will likely land in the
[fedora-atomic](http://pagure.io/fedora-atomic) manifest git repo.

Makefile-libpriv.am

@@ -45,6 +45,9 @@ librpmostreepriv_la_SOURCES = \
 	src/libpriv/rpmostree-rpm-util.h \
 	src/libpriv/rpmostree-importer.c \
 	src/libpriv/rpmostree-importer.h \
+	src/libpriv/rpmostree-jigdo-assembler.c \
+	src/libpriv/rpmostree-jigdo-assembler.h \
+	src/libpriv/rpmostree-jigdo-core.h \
 	src/libpriv/rpmostree-unpacker-core.c \
 	src/libpriv/rpmostree-unpacker-core.h \
 	src/libpriv/rpmostree-output.c \
@@ -55,6 +58,8 @@ librpmostreepriv_la_SOURCES = \
 	src/libpriv/rpmostree-editor.h \
 	src/libpriv/libsd-locale-util.c \
 	src/libpriv/libsd-locale-util.h \
+	src/libpriv/rpmostree-libarchive-input-stream.c \
+	src/libpriv/rpmostree-libarchive-input-stream.h \
 	$(NULL)
 
 librpmostreepriv_la_CFLAGS = \

Makefile-rpm-ostree.am

@@ -37,6 +37,8 @@ rpm_ostree_SOURCES = src/app/main.c \
 	src/app/rpmostree-builtin-status.c \
 	src/app/rpmostree-builtin-ex.c \
 	src/app/rpmostree-builtin-container.c \
+	src/app/rpmostree-ex-builtin-commit2jigdo.c \
+	src/app/rpmostree-ex-builtin-jigdo2commit.c \
 	src/app/rpmostree-builtin-db.c \
 	src/app/rpmostree-builtin-start-daemon.c \
 	src/app/rpmostree-db-builtin-diff.c \

design/jigdo.md

@@ -0,0 +1,97 @@
+Introducing rpm-ostree jigdo
+--------
+
+In the rpm-ostree project, we're blending an image system (libostree)
+with a package system (libdnf).  The goal is to gain the
+advantages of both.  However, the dual nature also brings overhead;
+this proposal aims to reduce some of that by adding a new "jigdo"
+model to rpm-ostree that makes more operations use the libdnf side.
+
+To do this, we're reviving an old idea: The [http://atterer.org/jigdo/](Jigdo)
+approach to reassembling large "images" by downloading component packages.  (We're
+not using the code, just the idea).
+
+In this approach, we're still maintaining the "image" model of libostree. When
+one deploys an OSTree commit, it will reliably be bit-for-bit identical. It will
+have a checksum and a version number. There will be *no* dependency resolution
+on the client by default, etc.
+
+The change is that we always use libdnf to download RPM packages as they exist
+today, storing any additional data inside a new "ostree-image" RPM. In this
+proposal, rather than using ostree branches, the system tracks an "ostree-image"
+RPM that behaves a bit like a "metapackage".
+
+Why?
+----
+
+The "dual" nature of the system appears in many ways; users and administrators
+effectively need to understand and manage both systems.
+
+An example is when one needs to mirror content. While libostree does support
+mirroring, and projects like Pulp make use of it, support is not as widespread
+as mirroring for RPM. And mirroring is absolutely critical for many
+organizations that don't want to depend on Internet availability.
+
+Related to this is the mapping of libostree "branches" and rpm-md repos. In
+Fedora we offer multiple branches for Atomic Host, such as
+`fedora/27/x86_64/atomic-host` as well as
+`fedora/27/x86_64/testing/atomic-host`, where the latter is equivalent to `yum
+--enablerepo=updates-testing update`. In many ways, I believe the way we're
+exposing as OSTree branches is actually nicer - it's very clear when you're on
+the testing branch.
+
+However, it's also very *different* from the yum/dnf model. Once package
+layering is involved (and for a lot of small scale use cases it will be,
+particularly for desktop systems), the libostree side is something that many
+users and administrators have to learn *in addition* to their previous "mental model"
+of how the libdnf/yum/rpm side works with `/etc/yum.repos.d` etc.
+
+Finally, for network efficiency; on the wire, libostree has two formats, and the
+intention is that most updates hit the network-efficient static delta path, but
+there are various cases where this doesn't happen, such as if one is skipping a
+few updates, or today when rebasing between branches. In practice, as soon as
+one involves libdnf, the repodata is already large enough that it's not worth
+trying to optimize fetching content over simply redownloading changed RPMs.
+
+(Aside: people doing custom systems tend to like the network efficiency of "pure
+ ostree" where one doesn't pay the "repodata cost" and we will continue to
+ support that.)
+
+How?
+----
+
+We've already stated that a primary design goal is to preserve the "image"
+functionality by default. Further, let's assume that we have an OSTree commit,
+and we want to point it at a set of RPMs to use as the jigdo source. The source
+OSTree commit can have modified, added to, or removed data from the RPM set, and
+we will support that. Examples of additional data are the initramfs and RPM
+database.
+
+We're hence treating the RPM set as just data blobs; again, no dependency
+resolution, `%post` scripts or the like will be executed on the client. Or again
+to state this more strongly, installation will still result in an OSTree commit
+with checksum that is bit-for-bit identical.
+
+A simple approach is to scan over the set of files in the RPMs, then the set
+of files in the OSTree commit, and add RPMs which contain files in the OSTree
+commit to our "jigdo set".
+
+However, a major complication is SELinux labeling. It turns out that in a lot of
+cases, doing SELinux labeling is expensive; there are thousands of regular
+expressions involved. However, RPM packages themselves don't contain labels;
+instead the labeling is stored in the `selinux-policy-targeted` package, and
+further complicating things is that there are also other packages that add
+labeling configuration such as `container-selinux`. In other words there's a
+circular dependency: packages have labels, but labels are contained in packages.
+We go to great lengths to handle this in rpm-ostree for package layering, and we
+need to do the same for jigdo.
+
+We can address this by having our OIRPM contain a mapping of (package, file
+path) to a set of extended attributes (including the key `security.selinux`
+one).
+
+At this point, if we add in the new objects such as the metadata objects from
+the OSTree commit and all new content objects that aren't part of a package,
+we'll have our OIRPM. (There is
+some [further complexity](https://pagure.io/fedora-atomic/issue/94) around
+handling the initramfs and SELinux labeling that we'll omit for now).

src/app/rpmostree-builtin-ex.c

@@ -30,6 +30,10 @@ static RpmOstreeCommand ex_subcommands[] = {
     "Assemble local unprivileged containers", rpmostree_builtin_container },
   { "kargs", 0,
     "Query or Modify the kernel arguments", rpmostree_ex_builtin_kargs },
+  { "commit2jigdo", RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD,
+    "Convert an OSTree commit into a rpm-ostree jigdo format", rpmostree_ex_builtin_commit2jigdo },
+  { "jigdo2commit", RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD,
+    "Convert an rpm-ostree jigdo into an OSTree commit", rpmostree_ex_builtin_jigdo2commit },
   { NULL, 0, NULL, NULL }
 };
 

src/app/rpmostree-compose-builtins.h

@@ -30,6 +30,8 @@ gboolean rpmostree_compose_builtin_tree (int argc, char **argv, RpmOstreeCommand
 gboolean rpmostree_compose_builtin_install (int argc, char **argv, RpmOstreeCommandInvocation *invocation, GCancellable *cancellable, GError **error);
 gboolean rpmostree_compose_builtin_postprocess (int argc, char **argv, RpmOstreeCommandInvocation *invocation, GCancellable *cancellable, GError **error);
 gboolean rpmostree_compose_builtin_commit (int argc, char **argv, RpmOstreeCommandInvocation *invocation, GCancellable *cancellable, GError **error);
+gboolean rpmostree_compose_builtin_commit2jigdo (int argc, char **argv, RpmOstreeCommandInvocation *invocation, GCancellable *cancellable, GError **error);
+gboolean rpmostree_compose_builtin_jigdo2commit (int argc, char **argv, RpmOstreeCommandInvocation *invocation, GCancellable *cancellable, GError **error);
 
 G_END_DECLS