projects: add ORAS proposal #68
Conversation
|
@cyphar @ArangoGutierrez comments addressed! thanks |
|
@cyphar sorry for delay, comments addressed |
|
apart from the maintainer balance, LGTM |
|
@SteveLasker You probably want to squash the PR to a single commit for the proposal. |
|
@SteveLasker can you add the same voting markdown from #67 to the top comment so we can align these 2 proposals with the Friday vote deadline? Thanks! |
|
Just getting several cleanup items done, but added the voting to follow the dates committed. |
They're actually 2, one from @jdolitsky the other from me. These can get squashed on merge, no? |
proposals/oras.md
Outdated
|
|
||
| ## Frequently Asked Questions (FAQ) | ||
| *Q: Does this change the OCI Charter or Scope Table?* | ||
| A: Yes, this will add an additional row to the [OCI Scope Table][oci-scope-table], which does not currently include any mention of generic OCI image tooling. |
There was a problem hiding this comment.
This line needs to be updated now that there isn't an OCI Scope Table anymore (as I did in #67). In addition, ORAS should probably be referred to as an "OCI distribution tool"?
You should also answer some of the new questions I've added to #67 -- which effectively boils down to "what category of project is this (as defined by #76 and the still-draft #86)?". The reason this is important is that it's not quite clear whether this intends to be a Reference Implementation of something (whether that be distribution-spec as a client, or artifacts) or a Library, and the scope and usage of the project depends a fair bit on that categorisation. My view is that this appears to be a Reference Implementation (because while it is usable as a Library that's effectively an implementation detail -- umoci is also usable as a library, as is runc).
You also should explain how you plan to avoid the "runc issue" (conventions in a Reference Implementation that are within the scope of the Specification becoming a de-facto standard even though they weren't standardised and thus are impossible to change).
There was a problem hiding this comment.
Just updated FAQ section, stating ORAS a reference implementation of distribution spec
There was a problem hiding this comment.
We've also said it was a library, to be used by other tools. I suppose the question is whether it can be in both categories.
There was a problem hiding this comment.
@cyphar It looks like Josh's update addresses the feedback. Is there something else to clear the requested change?
|
I've squashed Steve's commit which fixed DCO check |
Signed-off-by: Josh Dolitsky <[email protected]>
|
I could be swayed either way (delay vote conclusion by one week or try and get people to vote in the next 3+ hours with @SteveLasker's reasonable point about ongoing discussion/PRs to clarify any open questions). I think the issue is at this point I assume the vote may fail given 2 participants will be in the middle of their night at vote close. I would like everyone on the TOB to feel comfortable that any larger questions are put to rest (with total agreement that there will always be nuance and changes over time that can be handled post-vote), and it feels like trying to get a complete vote will effectively have to be rushed at this point. It is a bit troublesome that after months of discussion most questions have come up in the 48 hours before the vote, but that's life in open source! :) Agree with @cyphar that given a lot of questions were raised on both proposals running them in parallel may have added to the difficulty there. Thoughts? Opinions? |
|
With no downside to extending, and in the spirit to make everyone comfortable with their questions, I'd vote to extend the vote a week to make it imminent to cause focus, but time to iterate. |
|
I think extending the vote another week to settle on the questions makes sense. |
|
👍 on extending the vote, and I don't think we need to vote to extend the vote (that'd get a bit silly pretty quickly) -- I think @estesp as TOB Chair can just announce that the vote will be extended since he called it in the first place. |
|
I've extended the vote to next Friday.
|
|
Thanks @SteveLasker; I did pass the extension by Amy and Chris A. today to make sure there were no governance issues I was unaware of with a decision to extend. I will send a note to the TOB list just to make sure it is clear. |
|
What are the remaining questions from TOB members? We need to finalize questions today (and a maximum of maybe 24 hrs from now IMHO) so that answers can be provided and we give the ORAS experts at least a day to provide answers. I haven't seen any substantial feedback or responses since we extended the vote, but we need a conscious acknowledgement from TOB members that their specific questions are answered and they are all comfortable to make a voting decision. |
| @@ -26,6 +26,7 @@ https://groups.google.com/a/opencontainers.org/forum/#!forum/tob (tob@opencontai | |||
| * [Image Format Spec](proposals/image-format) | |||
| * [SELinux](proposals/selinux.md) | |||
| * [Tools](proposals/tools.md) | |||
| * [ORAS](proposals/oras.md) | |||
There was a problem hiding this comment.
@jdolitsky if you could rebase on master to pick up the conflict in this list (umoci was merged) that would be good so this is mergeable when necessary.
There was a problem hiding this comment.
What are the remaining questions from TOB members? We need to finalize questions today (and a maximum of maybe 24 hrs from now IMHO) so that answers can be provided and we give the ORAS experts at least a day to provide answers.
Similar to how we approved OCI Artifacts, we based the TOB vote on the merits of adopting the project. Did the intent of the project meet the OCI charter? After it was adopted, we spent a looong period of time reviewing the details of the project as it went through the review process.
We've received great feedback on details of the ORAS implementation, in how it generates annotations by default, or the packages it references.
These are great issues we will be addressing.
The scope of ORAS remains the same.
- Enable developers to focus on building their new
artifact-thing. - Leverage the existing infrastructure every cloud offers for securing content from dev-->production.
- Let developers and users focus on managing one registry source for content, without having to build a YASS - (Yet Another Storage Solution).
- Enable quick prototyping with an ORAS binary
- Be a reference client implementation of the OCI distribution-spec and OCI Artifacts so developers can easily write
artifact-thing push / pullwithout having to know anything about registry implementations.
What ORAS isn't:
- A container runtime client
The vote:
The vote here agrees to merits of ORAS being a reference client implementation of the OCI distribution-spec and OCI Artifacts
Separately, we will continue to address the concerns for references and defaults. As ORAS exists as a project today, we can agree to address the set of issues before transferring ownership.
|
|
||
| ### ORAS Details ### | ||
|
|
||
| ORAS is a CLI that can publish arbitrary content to an OCI registry, with special features for setting mediatypes on manifest configs and on content. |
There was a problem hiding this comment.
ORAS is a CLI that can publish arbitrary content to an OCI registry, with special features for setting mediatypes on manifest configs and on content.
ORAS is both a library for inclusion in other artifact CLIs and CLI that can be used directly to publish arbitrary content to an OCI registry. ORAS provides features for setting manifest.config.mediaType, to identify the artifact type, and layer.mediaType to identify the content.
There was a problem hiding this comment.
LGTM
I am voting yes on this proposal both in the sense of @SteveLasker's "does the spirit of the proposal fit with the intentions of the OCI" as well as my personal opinion that ORAS is a valuable library that provides a strong basic building block on which both development and future experimentation around the distribution-spec and the artifacts project can be done without requiring that individuals and/or projects scrape together those foundational steps (fiddling with configuration of connection/authentication to a registry; putting together an OCI object with the right format and media types, etc. etc.) on their own. We can debate about whether it is small enough or doesn't rely on certain things we wouldn't prefer going forward, in my opinion.
|
Sorry for not sending this earlier, but I've been struggling to get ORAS to work so I can properly review it. Namely, unless I'm mistaken it appears ORAS cannot actually pull OCI images at all: % oras pull docker.io/library/debian:latest -ao oci_store
Downloaded empty artifact
Pulled docker.io/library/debian:latest
Digest: sha256:46d659005ca1151087efa997f1039ae45a7bf7a2cbbe2d17d3dcbda632a3ee9a
% ls oci_store
ls: cannot access 'oci_store': No such file or directoryThe digest is correct, but I don't know where the blobs are supposed to have gone (I found that ORAS does support using an OCI image-spec layout as of oras-project/oras#108 -- though that really should be the default IMHO). I imagine this can't be intentional, hence why I've put off replying until I got it working (I assumed that I was doing something wrong). But with the vote deadline being so close now, it's probably better to post my incomplete opinion on this proposal. Regarding "library" or "reference implementation", this project can only be one of the two. A reference implementation may provide a library interface (as However, @SteveLasker has discounted this project being a library (I'm not sure if you saw this @estesp), and instead saying that it is a reference implementation:
Which means it's very critical that I ask whether there are specific plans to make
I don't think all of the above needs to be addressed before the project is accepted, but there should be broad agreement that this is what ORAS should aim towards in the future. Without the above points settled, users will have to be told "yeah we have a reference implementation of the spec -- ORAS -- but if you actually want to download OCI images use skopeo" which is a very split-brain message. Again, sorry for springing this on you so late in the process but I do think this is important to clear up. |
|
Thanks Aleksa, ORAS is NOT intended to be yet another runtime-container tool. It's not focused on the runtime aspects of the image-spec. It's really focused pushing additional content into the distribution-spec, using the manifest format for defining a thing, and the layers that make up that thing. See Artifacts Scope We can certainly figure out how to fix the default OCI Image pull experience. However, the conversation for vote should be focused on: Enabling new uses of the distribution spec. Enabling the community to build new things they need to distribution.
In addition to reference implementation and library, perhaps we're talking about the tool category. Let me address the specific topics raised for details:
ORAS isn't intended to account for container-runtimes. It's intended to account for file based artifact types, including directories. The recent conversation on annotations will address this. I don't know how runtime images are tar'd up, but if they have a filename or directory, ORAS would support this model. Right now, I'd consider it a non-supported scenario, or a bug we could support if it were felt to be important. Remember, the ORAS binary is a prototyping binary for building your custom
I'd suggest we need a bit more clarity on what makes a library vs. reference implementation. Rather than require an artifact author to start with a go project, adding all the libraries, we provide a client reference implementation of the distribution-spec, which supports arbitrary artifact types. When the artifact author is ready, they can create a go project and include the same libraries to build their So, this could be a reference implementation that has the libraries to copy/paste/edit your own.
Noted above, we can enable this, if we feel it's important. While most currently use runtime, ORAS and Artifacts are all about expanding what people are doing tomorrow. ORAS was not intended to be a generic container-runtime tool. We have docker and others for this ORAS focuses on leveraging registries for new artifact types.
ORAS is not intended to default to oci-runtime images. See: [https://github.com/oras-project/oras/issues/115](ORAS mediaTypes should default to unknown #115) It's everything else ORAS enables. Consider ORAS a prototyping tool. When you create Honestly, I'd like to remove the
These points are focusing on the image-runtime being the priority. ORAS is about all the other artifact types. |
|
I think you're conflating the term "container runtime" with quite a few other things, which is understandable given the term's more generic use within the wider industry. However, within OCI a container runtime is a very specific thing -- it is a program or service which implements the OCI Runtime Specification. Images are not part of the container runtime, so when you said:
I was a little confused, because nobody should expect that to be the case (it's not going to spawn containers based on OCI runtime-spec bundles, so of course it isn't a container runtime). I just want to make sure that we're understanding each others' concerns and using a term like "container runtime" is a bit confusing when it has a specific meaning OCI but isn't being used that way is confusing (at least to me). In particular, this line also seems a bit confusing to me (again, "runtime image" is a term that I think you're using to refer to the OCI image-spec?):
Unless I'm mistaken, this is already implemented by oras-project/oras#108. If that mode of operation was the default, or was used based on the media-type (if it matched the OCI image-spec ones) then I would be totally okay with this because it would do what IMHO most users would expect. I think you might be talking about the But I guess your point is that ORAS has nothing at all to do with OCI images directly, it just so happens that you can in theory (though apparently not practically -- see above) use it with OCI images. That's fine. But distribution-spec explicitly does have a lot to do with OCI images. Yes, you can store other artefact types (and that's perfectly fine) but the primary usage people have today (and likely will always have) is related to OCI images. I don't think it's unreasonable to ask "why doesn't that work at all with ORAS right now?". I'm genuinely not trying to be difficult or anything, I just want to make sure we don't end up in a situation where we have a project in the OCI that explicitly wishes to not primarily operate on OCI images (which is fine), but at the same time is defined as a "reference implementation" of a specification which most people wish to use with OCI images. Because in that situation, if a future TOB had to consider adding a new project that does allow you to push/pull OCI images it would almost certainly be rejected because "hey, we already have ORAS". I'm not even sure if the problem is related to the "bucket" we put ORAS in -- when someone comes to the OCI and says "oh, I want to use this neat OCI thing -- let me go download an image to go run a container" our answer to that is going to be "well, we have ORAS but it doesn't actually do that -- you need to go elsewhere or write your own tool based on ORAS". That doesn't seem like a reasonable path forward for me.
But Docker isn't being proposed for inclusion into the OCI. ORAS is. If we just say that ORAS is a "reference implementation of the artefacts spec" then that's one thing, but having a project that looks and smells like a tool that could be used to download OCI images (but can't) without having a project that can seems like a bit of a bad idea -- we'd be ignoring what the vast majority of users actually want to do. |
|
Lots of good thoughts, that I didn't read all. It's the end of the day, so for now, I'll say:
I'll review tomorrow in detail and respond. I'm not sure how that impacts the voting timeline, but we're having the right conversation to narrow it in. |
|
@cyphar at this hour in my timezone there is way too much text to respond to for completeness, but let me summarize that it seems like there is a huge disconnect when the ORAS proposal goes out of its way to use phrases like "custom content", "artifacts", "other types of content" and consistently never talks about implementing any functionality related to the image-spec in reference to ORAS, yet almost the entire review revolves around (drum-roll please): ORAS doesn't seem to work with the image-spec! Yes, exactly. We already have plenty of tools for that all over the container ecosystem. ORAS is a library to handle the client-side interactions with any given distribution-spec implementation, and provides a common library of functionality on which people can implement and explore the ideas and media types being hashed out in the OCI artifacts project. It has no default purpose, nor would it make sense for it to handle pulling OCI image-spec bundles as it was never designed or even recommended as a tool for that purpose. It would take about 15 minutes of code to use ORAS to make an image-spec "compliant" client (I've actually done this with ORAS and a custom image handler function which properly walks the image-spec), but given that wasn't the purpose, I'm confused as to why that has to be the "default mode" of ORAS when it wasn't designed to be a container image client. |
|
I really should try to make more comments more brief, sorry about that. I do now better understand that ORAS explicitly does not wish to be related to OCI images. That's fine, but if we're going to throw around the word "reference implementation" in the context of a distribution-spec client I do want to make sure we agree what that actually means. While most of this text pre-dates the artefacts-spec, the distribution-spec explicitly talks about OCI images and their storage. It seems very odd to me that we would say that "a reference implementation of a distribution-spec client" doesn't actually do anything with OCI images. All I want to avoid is that we end up in a situation where we:
If we can avoid that (even if it's just saying "we don't discount the addition of other projects that implement OCI image-spec specific download features"), then I'd be fine with this proposal. Honestly I'd even be okay with a tiny I'm not sure if I'm missing something, but am I really the only person who finds it strange that you cannot download an OCI image from an OCI distribution-spec repo with a tool that we are discussing adding to the OCI as a reference implementation of the OCI distribution-spec?
This was my impression as well, looking at the examples and parts of the implementation. That's why I'm finding this all a little bit strange, as though I'm missing some greater context. |
|
Thanks @estesp, and @cyphar However, downloading an oci-image is needed for air-gapped environments where someone wants to download content, doesn't have or want the docker client as it requires a daemon, and needs to send the content through some gateway to the air-gapped environment. We're seeing this in IoT scenarios that implement Purdue networks. However, oci-image is not the primary scenario and should not be the default for push because:
To summarize:
|
…osal Signed-off-by: jdolitsky <[email protected]>
|
I feel this has been filibustered :) In my opinion, the value of ORAS is not related to images or the image-spec, but rather that it can publish other types of content to registries. It plugs into registries powered by Docker distribution, regardless of OCI specs in play. To that end, I can see CNCF as an alternative home for ORAS, considering it's built on containerd and used for Helm and Open Policy Agent (all CNCF projects). On the other hand, that might also signal that it has no use outside the "cloud-native" realm. In any case, ORAS can change its course to better represent its employer. It's certainly operating in a gray area right now. |
|
agreed. I tried to follow, but I think once the words began to spill: "when it rains, it pours". As to CNCF vs OCI, this lends to the funny relationship that exists between these two groups already. As for pulling OCI images, that should be expected behaviour. |
|
Should this be closed? Does opencontainers want oras? Otherwise I offer 3000 dogecoin for it |
So, the action item is to move some of the libraries to oci/articacts and streamline the ORAS project. That said, everyone has been pretty busy with Notary v2, Docker TOS and other priorities that we haven't gotten back to this. My belief is the Notary v2 implementation will require these changes and it will become a forcing function. For the sake of cleanliness, I'd be fine with closing this and creating a new issue when we're ready. |
|
Hi all - I've added to this week's call agenda the possibility of continuing with this effort. Perhaps we can discuss what requirements people would require to see in oras to make it a suitable opencontainers project? cc @cyphar @samuelkarp |
|
We had some action items from the last discussion:
It would be good to capture the specific steps to come to a consensus on what's required as I believe we agreed it fits the charter of OCI, but needed some cleanup. |
Continuation of #66
Vote Total (Closes 2020-06-19) (please make sure you also leave an LGTM comment):
/cc @opencontainers/tob