cosa/mantle integration

[ajeddeloh]

As per discussion, we should look into moving the mantle bits to their own container. They aren't used to build the image; only for uploading, releasing, testing, etc.

Splitting them out would have the following advantages:

• Logically separating image/ostree creation and uploading/testing/releasing
• Reducing the number of container rebuilds since new commits to mantle will not affect this container
• Reduction in container size

There are of course a few disadvantages:

• Now we need two containers to do a release
• Total container size will likely increase (since there are two needed)

[dustymabe]

so I think this proposal has interesting overlap with the discussion in #52 . What I was proposing there is that we keep including the tools from mantle inside the coreos-assembler container, but we just pull them from another location (i.e. another build system takes care of them).

Carrying this discussion further:

Option A: This proposal #163. Split out mantle from COSA. We build mantle and create a container. The build system and local devs handle grabbing both mantle container and coreos-assembler container to perform all actions they needs to do.

Option B: Split out build of mantle from COSA (i.e. #52). The coreos-assembler container build pulls in pre-built binaries from mantle.

The advantages listed from the description:

• Logically separating image/ostree creation and uploading/testing/releasing

I think we can still achieve this goal with option B by just using separate verb in CoreOS Assembler

• Reducing the number of container rebuilds since new commits to mantle will not affect this container

With option B the number of rebuilds of the container would be increased. For the prod side of things this shouldn't be a big deal because the container build should be automated anyway. For the local dev side right now we do check rpm dependencies in the host and we could extend that to check that kola/ore/plume software was above a defined minimum release version. WDYT?

• Reduction in container size

With option B the container size would be slightly reduced from what it is today because we'd only include the binaries for mantle/ binaries, but we would still include the mantle binaries so not as much of a reduction, but then again you only need one container \o/ :)

Of course with option B (similar to option A) we'd need to figure out building the mantle pieces somewhere stable and adding them into the coreos-assembler container.

[ajeddeloh]

I'm personally still a fan of keeping them separate. Having the mantle container could also be useful in it's own right. On the CL side we could probably replace the CL jenkins job that rebuilds mantle every half hour if there's changes (@bgilbert @dm0- thoughts?). Since mantle is all golang and statically linked, the container should also be pretty small (assuming a multistage build).

I think we can still achieve this goal with option B by just using separate verb in CoreOS Assembler

I disagree. Instead of a container which does one thing, we'd have a container which does a few things. In general we ought to separate out things if we have the chance. I don't like bundling things just because we can.

With option B the number of rebuilds of the container would be increased. For the prod side of things this shouldn't be a big deal because the container build should be automated anyway. For the local dev side right now we do check rpm dependencies in the host and we could extend that to check that kola/ore/plume software was above a defined minimum release version. WDYT?

Sure it's automated, but that's still just burning CPU cycles for no reason. If we avoid extra logic for dep checking for local dev that's a win in my book. Keep everything as simple as possible, otherwise things have a tendency to explode in complexity.

Of course with option B (similar to option A) we'd need to figure out building the mantle pieces somewhere stable and adding them into the coreos-assembler container.

Building mantle into a container automatically would be pretty trivial. Just set up a dockerfile and hook the repo up to quay. Just building the artifacts straight would probably be harder.

[dustymabe]

I think we can still achieve this goal with option B by just using separate verb in CoreOS Assembler

I disagree. Instead of a container which does one thing, we'd have a container which does a few things. In general we ought to separate out things if we have the chance. I don't like bundling things just because we can.

but by that same logic we'd need a new container for everything we want to do within CoreOS Assembler. i.e. we'd need a container for rpm-ostree/ostree. A container for image build. A container for generating a ostree-in-container piece. I feel like part of the point of CoreOS Assembler is the gluing together of everything you need to build/test/release.

[ajeddeloh]

I'd argue assembling the image is coupled enough to be bundled.

If we do want a higher level abstraction for chaining everything together, I'm down, but it should chain the different stages (ie via multiple docker invocations). This would then be mirrored in the CI pipeline.

It also simplifies things like getting credentials into COSA since (at least for FCOS) it shouldn't need any credentials to build everything. The mantle container will still need them but at least then it will be clear what parts need what creds.

[dustymabe]

If we do want a higher level abstraction for chaining everything together, I'm down, but it should chain the different stages (ie via multiple docker invocations).

yeah. the idea in my head at least is that we have different stages of the pipeline call into coreos-assembler to do small pieces of the work.. For example

• stage 1 - coreos-assembler fetch, if no changes then exit here
• stage 2 - coreos-assembler build ostree
• stage 3 - coreos-assembler build qcow
• stage 4 - coreos-assembler kola arg1 arg2
• stage 5 - archive artifacts to public location or something

But when you're doing things locally you'd just run coreos-assembler build and it would do steps 1-3 for you and then you could run coreos-assembler kola to run tests if you want. The idea here is that if we make bake most of the logic into coreos-assembler then setting up a pipeline (using jenkins or a bash script or another build system) should be pretty easy to do so it doesn't matter what build system you choose.

[ajeddeloh]

I agree on roughly the same workflow (i.e. stages 1-5) but still think mantle ought to be it's own container. We've talked in #75 about having a top level entry point script that handles launching podman/docker. What if we had that script be the common bit and it used different containers for different steps if it needed to? In your example 1-3 would use the assembler container 4 would use the mantle container and 5 might also use the mantle container (e.g. for ore) or might use some other container or might even just use the aws cli.

[dustymabe]

I agree on roughly the same workflow (i.e. stages 1-5) but still think mantle ought to be it's own container.

I think we're mostly splitting hairs here. We could use the mantle container for step 4/5 (and others) but the question is.. is it better than, similar, or worse than including it. I think if the answer is similar or worse than then we should just include it in the assembler container. If the answer is better than then let's create a separate mantle container.

What if we had that script be the common bit and it used different containers for different steps if it needed to?

we could do that but that script is managed in the coreos-assembler repo, so IMHO might as well include mantle in the container :)

Either way I think either of these solutions work fine, we're both just trying to find the best solution. Also we can pivot on the choice we make now if it proves to be the wrong choice.

[cgwalters]

I'm not inherently opposed to multiple containers, but it (along with an external container-launching entrypoint) rather directly conflicts with my current workflow of doing most of my day-to-day work inside a single container.

[jlebon]

We could also build it in a separate container and just build COSA FROM it? That way you still get the single mantle container if you just want that but it's still integrated in the COSA container.

I agree that part of the awesomeness of COSA is that it's "context" aware. So e.g. coreos-assembler kola knows to look for builds in builds/latest. Wherever kola actually comes from, I think having "mantle wrapped" commands is super nice for local dev (and still leaves the door open for setting up your Jenkins however you want, e.g. if you want to run kola directly).

[ajeddeloh]

@cgwalters The issue is the need for nested containers, right?

@jlebon If anything I think we'd want to COPY --from=... rather than have the whole thing be FROM it. The containers I'm building (I should have a PR up today) for mantle will probably have 1 per tool. But that's beside my point. This context knowledge doesn't need to be baked into the container(s). I see the container(s) more as individual tools not as an integrated "do everything" tool. We can mount in the same directory to all of them so they all see builds/latest and all look there.

I really don't like integrating everything when it's not necessary.

[dustymabe]

@jlebon If anything I think we'd want to COPY --from=... rather than have the whole thing be FROM it

Yes I think that would be optimal and it fits precisely into what I was proposing in #52. Upstream container runtime can COPY --from an external image. I don't know if builah can do that yet.

https://docs.docker.com/develop/develop-images/multistage-build/#use-an-external-image-as-a-stage

[dustymabe]

possible fix in #297

[cgwalters]

Repurposing this one to be about mantle/cosa in general.

I still actually lean towards merging the projects, though it's hard to do nicely without destroying git history AFAIK.

[bgilbert]

Without commenting on the merits of merging the projects: note that the parents of a git merge don't need to have a common ancestor.

[darkmuggle]

Per the 2020.02.25 Cabal, we decided to move forward evaluating the feasibility of this.

• @cgwalters will attempt to merge the COSA and Mantle.
• Tiger team of @cgwalters @darkmuggle @arithx @jlebon and multi-arch to work closely together
• When doing feature work evaluate whether it should be written into the Mantle codebase.
• Considering retirement of Mantle's repos with the Container Linux EOL.