Create Spec Core Devs / Remove ZIC #44
Conversation
|
I’m in support of this. I’d hate my involvement in ZIC to block progress on Zarr. Thanks @rabernat! |
|
This sounds like a good plan. Thanks for nominating me. |
|
That is also fine with me. |
|
Thanks for kicking off this discussion, @rabernat. I'm definitely looking forward to getting more feedback on this topic from the wider implementation community.
Seconding this and also where/if necessary, apologies from my side where things haven't been merged quickly enough.
This decoupling would certainly be my priority. As we (i.e., @jhamman 🙌🏽) did with the zarr-python core team, we need to clearly be able to move things forward.
I'm not immediate on board with this statement. I would definitely agree with "not a particularly active body" and I would suggest we try to fix that in a more incremental way. I'd say one of our (my) mistakes was not to require some form of regular buy in. Let's say I was being overly optimistic. My concern with not having the explicit involvement of other implementations is that it becomes easy to lose sight of the bigger picture (and the impact of changes). Minimally, I'd suggest that we invite existing ZIC members to the new team with the understanding that there are time requirements. From @manzt and @constantinpape, I'm sensing they would say, "thanks but not thanks" but I'd like to hear that from everyone. Alternatively, we consider having two bodies -- the core team for moving the day to day work forward and then the ZIC for reviews and general sign offs. (And we can still consider culling the ZIC based on active implementations.)
I agree that all those statements are true but disagree that it is necessarily the optimal choice for the final sign off on major changes. In my mind, there is an additional burden on spec development that isn't taken into account by this PR, especially if we approach a situation where
💯
For me this is conditional on the above. If there's some mechanism for the wider picture in place, then I agree with this wholeheartedly and would encourage anyone who is interested to get involved and get added! 🎉
Agreed, but I'd push for more of a compromise for the first iteration of improvements. |
|
Thanks for this @rabernat. I find the recent lack of movement on spec changes concerning, and agree that we a fix for that. Regarding the ZIC, it seems as though the majority of the members of the ZIC do not participate in zarr spec decision making. I think the list of spec maintainers should be drawn from the set of people motivated to maintain the spec, not from a list of zarr implementation authors. If we follow the proposal in this PR and get an active collection of spec maintainers, i.e. people with review / commit privileges to |
|
Hi @rabernat, FWIW I would have no objection to decoupling a group of core spec developers from the implementation council. Having a group of core spec devs able to move forwards more rapidly would be great. It kind-of goes without saying that the core spec devs should try to maintain interoperability between implementations as much as possible, and so should be consulting with and considering the views of implementers carefully. I think it would be worth stating that explicitly, though, for the record. I'd also be in favour of keeping the ZIC at least as a group of people who are involved in actively maintaining implementations, and who should be notified and consulted on any new specs or spec changes, and whose views should be carefully considered by the core spec dev group. |
|
Just to add, it also goes without saying that the core spec devs should be trying to maintain interoperability between data and implementations, i.e., should also be consulting with and carefully considering the views of data producers. |
|
This seems to be stalling. From my perspective, removing the ZIC is a nice simplification but is not required to set up the spec core dev team. Perhaps we separate the two issues and move forward with the core team. Getting the spec moving again is the highest priority at this time. |
|
IIRC, at the last ZSC meeting, @joshmoore volunteered to revise this proposal to remove the changes to the ZIC, in line with what @jhamman has proposed above. Josh, are you still planning to do this? Feel free to push to my branch directly. |
|
@rabernat: "YRC" 😄 Apologies for not having gotten to this sooner. I still intended to push something to your branch. In the way of an outline of the points I remember and would like to work into this text:
The major piece of the puzzle that I think we haven't clarified (or at least, hasn't been clear enough for me to write up here) is: what is the unit of interaction with the ZIC? i.e. when is the main branch sufficiently different that we should take a step back and confirm with other implementations if developments are going in a mutually beneficial direction or might cause unforeseen issues in a particular implementation. Ultimately, this may come down to a definition of what future versions of the spec look like. To @jhamman's point: you are right that separating the two would allow us to more quickly merge this PR, set up the team, and allow PRs to be merged, but it also puts us in a situation where things will be merged without a definition of the release process and versioning scheme. Thoughts welcome on how we can move both quickly and safely in this regard. |
I see the mandate of the new spec dev team to sustainably evolve the spec with an emphasis on interoperability. As part of that mandate, the ZIC would morph into an advisory board to the spec dev team. For major changes, the ZIC would be asked to provide feedback (review-style). It would be up to the spec dev team to decide what warrants a major change. I think this would work quite naturally if representatives of the major implementations are in the spec dev team already. |
Speaking for myself (as a ZIC member), I think the spec core dev team should be responsible for listening to the voices of all implementation developers (and implementation users), whether they are on some list or not. This seems like part of the definition of "core spec dev" -- It's hard to imagine spec maintenance and development proceeding sustainably without looping in implementations. Implementation developers should also probably keep notified of changes in the zarr-specs repository via the various mechanism provided by github, but if people have other communication preferences we should adapt spec development accordingly. I don't see any problem with this PR as-is, and even if this PR is amended to preserve the ZIC I still don't see the question "how should members of the ZIC learn about spec changes" as a blocker for the central content of this PR. It would be helpful to hear from the other members of the ZIC: @zarr-developers/implementation-council any thoughts here? |
Noting that this call did not generate any response from anyone on Implementation Council. 🦗 Alas, I am the Zarr-Python rep on the implementation council and will therefore share my perspective. I am much more excited about having a functioning spec process than maintaining my membership on the implementation council. No surprise here but I would be in favor of doing away with the ZIC. |
I completely agree with this statement from @alimanfoo. However, to make this more explicit I have proposed additions to this PR. @joshmoore do those proposed changes assuage your concerns? If not, could you explain in concrete terms how we can unblock this PR? And to your concern that we do not have a well-defined versioning scheme for the zarr specifications: I agree that this is an important issue, but in my mind resolving such issues is exactly the job of the proposed spec dev team. |
|
As a member of the Zarr Steering Council, I approve these proposed changes (with the additions proposed by @d-v-b):
|
|
Sorry, @rabernat, quick question, are you asking about #44 (comment) or the PR as it currently stands? |
Co-authored-by: Davis Bennett <[email protected]>
|
So for some background, we had about 2 hours of chatting over this at some of last week's meetings (community & ZEP). Thanks to @d-v-b for writing up his summary and @rabernat for adding it in here. I definitely believe it's an improvement over the simple deletion. As it now stands my largest question is likely the definition of "stakeholders" along with specifics about timelines for feedback, etc. If we slightly update it to kick off these processes when there's a major change, then essentially we're saying "zarr-spec devs will follow ZEP0 for major changes". This brings us to @normanrz's version in #44 (comment) which I also liked:
Here, though, I would appreciate a definition of "major" (especially in the absence of a well-defined versioning mechanism). My working definition is something like: "anything that changes the on-disk format OR anything that requires an implementation to be modified". Ultimately, my primary goal is to make sure that even those who are not actively watching this repo (and certainly, GitHub mentions are not doing the trick as things stand) still have a stake in this, in my opinion, most critical of Zarr assets. I agree (from the meeting's last week) that if we had a wider zarr-spec core dev team that that would largely achieve the same thing the ZIC is intended for, but I assume there will remain a set of people who's input we value who will not be able to keep up with the day-to-day GitHub communication. For them, I expect that clearly documented decision points, communicated by whatever means reliably works would be worth our time. To move iteratively (so I'm not further blocking things), I'd propose:
|
My proposal was that the zarr-spec core dev team decides what a "major" change is on a case-by-case basis instead of preformulating a formal definition. |
|
Sorry, @normanrz, I understood that and was trying to say that's how I would alter your recommendation. |
|
While we wait on responses, a few additional thoughts from a recent chat with @normanrz. He mentioned the idea of a "release train" for zarr-specs on a monthly or quarterly releases. This resonated with me since it addresses one of the biggest issues I see is the need to balance development speed and long-term stability. There are groups around the world who care about the stability and are failing to keep up with the speed (of conversations, minimally). I could imagine including here in the spec-team definition something like the following:
Honestly, if we could make something like this work, I'd be happy to skip many of the other meetings (ZEP, community) to make more time for this. There are a couple of huge decisions coming out of recent, hard discussions
|
Who exactly are these groups? If we are going to shape our development cadence around their needs, we have to know who they are, and ideally hear from them. Otherwise it's unclear what development style would work. |
|
The most recent conversations with implementers that I've had are with:
especially regarding the versioning issue(s) I've mentioned. But I think it holds for several of the other "active" implementations:
With his schedule @jbms might benefit at the moment as well. I'd be interested to hear what @meggart says, as well. Thinking more widely & outside the specific implementer group, I could imagine interest/benefit for:
Obviously, then questions about voting rights (if there's even a vote), etc. would need answering. I would hope though that that regular communication channels could even help find new people. (e.g., depending on what we choose, I could see posting the invitations/summaries on image.sc each quarter.) |
At the latest Zarr Steering Council meeting, we discussed the fact that steering council review has been a bottleneck on development, particularly where evolving the spec is involved. Currently the SC needs to review and approve spec changes, which has been very slow to happen (zarr-developers/zarr-specs#292). There are two ways we intend to fix this:
This PR addresses the latter by creating "Zarr Spec Core Devs". It also removes the ZIC, which has not turned out to be a particularly useful body.
Here I'm proposing that the spec be run by a group of active core devs, the same way the implementations themselves are run. I think this is the simplest and most effective approach. This will allow us to move faster and get to an up-or-down decision on spec changes within a reasonable timeframe.
This PR needs to be accompanied by a ZEP which changes the spec evolution process. You can expect that soon. In the meantime, I wanted to start the discussion here.
A key question is, who will be the Spec Core Devs. I'd propose the following list to start. After an initial nomination from the SC, the spec core devs are free to evolve their membership according to the CORE_DEV_GUIDE like all other Zarr projects.
cc @zarr-developers/steering-council, @zarr-developers/implementation-council for review.
Overall, the vibe here is that we have created much too much process and governance structure in Zarr, and now it is holding us back. The aim is to simplify and become more agile.