Re-envisioning the CRUD/Snapshot split

[wseltzer]

Responding to a piece of #344

Splitting CR-phase publications into periodic Snapshots (which, as currently, require Director's approval and have IPR implications) and potentially continuous Update Drafts (which can be published by the WG directly and have no IPR implications, like a WD) in between the snapshots.

I believe this change makes the wrong tradeoff. Adding extra process hurdles to the publication of a snapshot will not always induce groups to seek/respond to wide review, rather, it may encourage them simply to avoid snapshot publication, as they have heretofore avoided formal CR and PR transitions.

Instead, I propose that (as in the Evergreen model), snapshots be minted quasi-automatically from CR updates, and that we incorporate the calls for and responses to wide review into the CR update directly.

In particular, I'd recommend that review groups be empowered to put a warning signal (or OK) in the status section of drafts that have unresolved issues rising to the level of objection (e.g. with a traffic signal metaphor: red light, yellow light, green light). If a draft has an unresolved red light for too long (such as 2 years), the Director or Council (in director-free) must adjudicate the disagreement and either address the concern, overrule it, or direct the group to stop publication.

Rationale: This mechanism helps to align incentives among the WG participants and wide reviewers. After all, both groups want the CR Update to reflect reality and the needs of horizontal groups. The patent considerations addressed by Snapshots are secondary to most of them, and thus serve as the wrong lever to encourage reviews and response to them. Instead, we keep snapshots easy to generate and primarily legal-focused, while putting signals of horizontal review status prominently on the CR Update/Working Draft document that developers and implementers are encouraged to follow.

[fantasai]

we incorporate the calls for and responses to wide review into the CR update directly.

Wide review is supposed to happen continuously, and this is already described in the Process. But the HRGs tend to find that groups forget about it until they're confronted with a Process step that checks for it. I don't think removing those checks will result in the wide review happening more continuously--that opportunity and encouragement is already there, and some groups are taking it and other groups are not. What I expect will happen is that those groups which are not taking that opportunity will simply never bother.

Tying together the IPR and other process checks is something we have done for a long time. It has several advantages:

• It is familiar, well-established practice, and the least change from the current Process, since it is how all CR publications operate now, making the transition to the new Process more straightforward and understandable.
• It puts all formal checks together in one package, which makes it clearer what's happening and when.
• It makes provides a framework for a WG to organize its work around coherent sets of changes if it wants to, which helps reviewers (legal, horizontal, and otherwise) who need to do a lot of context-switching. I wouldn't, for example, publish a snapshot when I've only addressed half of the related issues filed against a particular feature, because reviewing half of those fixes now and half six months from now wouldn't be an efficient use of the reviewers' efforts.

If an HRG wants to set up a traffic light dashboard, and let WGs use that as evidence when requesting update approval instead of explicitly requesting a review, that's fine, and allowable by the process. The Process doesn't require that any particular mechanism be used to indicate that the HRGs review has been satisfied, and quite frankly, I don't think it should. That's practice to be worked out by the various HRGs and the Director, and may well evolve over time as they figure out what works and what doesn't.

[wseltzer]

Tying together the IPR and other process checks is something we have done for a long time.

I draw the opposite conclusion from that -- it's one of the hurdles to WG agility, that we're trying to fix with Process updates.

[fantasai]

I draw the opposite conclusion from that -- it's one of the hurdles to WG agility, that we're trying to fix with Process updates.

I don't think the WGs’ agility is held back all that much by IPR concerns. Most WGs generally do whatever they can in terms of spec development wherever it's convenient, and are only concerned with IPR in the abstract. The main hurdles are not being able to publish or modify their work without excessive process, and the CRUD addition to the Process addresses that problem directly.

[AutomatedTester]

The Browser Testing and Tools Working Group were pleased with what was presented by @plehegar at TPAC 2019. We think this is a statisfactory approach.

[jakearchibald]

I wanted to leave a few comments about the general perspective from the service worker working group.

We have at least 3 documents in the wild:

https://www.w3.org/TR/service-workers-1/
https://w3c.github.io/ServiceWorker/v1/
https://w3c.github.io/ServiceWorker/

This has been confusing for users and implementers, who have found themselves looking at the 'wrong' document. We've added warnings to try and mitigate the issue, but it's a bandaid.

It's been a huge maintenance burden for editors, who've had to maintain two branches. This makes us afraid to make larger changes, which would benefit users and implementers, as diverging the two branches further increases the burden, and creates more opportunity for error.

We were under the impression that this cost to users, implementers, and editors was essential, as it was the only way to fulfil legal obligations. Organisations like the WHATWG showed us this isn't the case.

I'm really pleased to see the W3C react to this, and attempt to modernise. This effort is essential, and urgent, for the service worker group to continue to function within the W3C.

The equivalent process at the WHATWG involves us pressing a button once a month. That's ideal from our point of view.

[brewerj]

We have been improving the agility of the horizontal review process, through increased systematization and more checklists and resources to support streamlined reviews.

If at the same time we remove one of the levers to incentivize people to resolve the results of those reviews in a timely way, I think that that would be a step back on W3C's Web for All commitments.

I question whether a yellow or red flag on a CRUD would discourage many implementers from using a CRUD. For technologies that are taken up quickly into workplaces, education, healthcare, etc., with unresolved accessibility issues, this can significantly dislocate the lives of people with disabilities in a digital environment.

Unless a stronger mechanism is available to get quick resolution to issues raised in horizontal reviews, I think that continuing to tie together the IPR and other process checks is our best option.

[css-meeting-bot]

The Revising W3C Process CG just discussed Types of CR.

The full IRC log of that discussion

<fantasai> Topic: Types of CR
<fantasai> github: https://github.com//issues/346
<fantasai> florian: We have these two types of CRs which are analogous to what's happening in WHATWG: snapshots vs updates
<fantasai> florian: The difference with WHATWG model is snapshot is not only for lawyers, it also has Director's review, which checks for wide review, horizontal review etc.
<fantasai> florian: We discussed it's good to tie carrot and stick
<jeff> ... without verification the work does not get done
<jeff> ... good idea to tie carrot to the stick
<jeff> ... this is intentional we discussed many times
<jeff> ... many people agreed, but Wendy does not.
<jeff> David: People should just post to the issue.
<jeff> Mike: I agree with Wendy. Cannot close this.
<jeff> q?
<fantasai> i/... without/scribenick: jeff/
<fantasai> ScribeNick: fantasai
<jeff> David: 345 and 346 for next call.

[jeffjaffe]

I agree with Fantasai which reflects the current text in the Everblue branch. I think this was also the sense of the AB in the November F2F.

I also it is late in the process to consider Wendy's proposal. I view this as a substantial change which will add delay at a time that we are trying to reach closure. I might have felt differently if this issue were accompanied with a Pull Request so I could see all of the implications. Lacking that, I would prefer that we close this issue at the next Process CG call.

[michaelchampion]

I support @wseltzer 's position here and don't think this issue should be closed without further input from stakeholders and Process CG or AB discussion.

One stakeholder comments in #346 (comment) that their WG wants something as similar as possible to the WHATWG process/patent policy. WHATWG treats snapshots as being an artifact of the patent policy, not the spec development process. I understand the motivation to combine the "carrot" of patent commitments with the "stick" of insisting that horizontal reviews be completed before something that could be considered a living standard is published. But that stick can be used to bar the gate as well as to herd the cats through it, defeating the purpose of the Continuous Specification Development reforms.

Clearly a W3C version of a Living Standards process should rely on the consensus of a working group rather than the judgment of an editor to decide when a draft spec has enough internal consistency, implementer interest, and external support to publish as a CR. Likewise having the patent exclusion period triggered by a WG decision rather than an arbitrary date on the calendar makes sense in a W3C-flavored Living Standards process. But adding lots more bureaucracy and new opportunities for gatekeeping seems unlikely to get the support from stakeholders such as Service Worker WG that you are seeking.

I don't agree with the comment #346 (comment) that it's too late to consider this proposal. The baseline is the current process, not some branch of the Everteal proposal. Adding references to the (hopefully) new patent policy about CR exclusion opportunities and the resulting patent commitments, and a bit of explanation about how WGs MAY use this mechanism to do Continuous Specification Development seems like a lot less work / review / consensus building than pushing through the Everteal language that @wseltzer suggests will add "hurdles to WG agility, that we're trying to fix with Process updates."

[jeffjaffe]

@michaelchampion in terms of AB discussion, this was discussed in the last AB F2F meeting. Wendy is in the process of finalizing the member-readable minutes.

[wseltzer]

and my in-thread comments were made before that meeting.

[dwsinger]

I agree with the comment. I continue to believe that connecting procedural and IPR progress means that it'll cause problems for both. But the AB did not agree and want to try, I fear.

[michaelchampion]

To me this particular issue feels more like an implementation detail (which is in the Process CG's remit) than a strategic direction (which is in the AB's remit).

Whatever ... it's the AB's right to decide what to delegate to the CG or not. But it's increasingly unclear what the Process CG's role is if such decisions are made without attempting to build consensus with the wider community of stakeholders here.

And considering that the AB meeting was 3 weeks ago, it would have been more respectful of those who are not on the AB to close this GitHub issue as soon as the AB had decided it. That would have prevented those of us who disagree from wasting our time arguing the point.

[chaals]

I'm late to this party, but I agree with @michaelchampion and @wseltzer (and I think @dwsinger although it isn't clear which comment he agrees with so I may be confusing his position on this). I don't think we should just close this issue with the current status quo proposal based on timed snapshots.

(I think we should be working harder on a simple versioning approach, which makes it easier to point to the top of the version tree as a stable properly reviewed new version - which is probably what many people who do point to a dated version would be OK referencing, although some use cases will still want to be able to point to the specific dated version. This means simplifying the whole process of taking a new version of the same specification along the Rec track).

[jeffjaffe]

@michaelchampion is raising a fair process point about the relative roles of the AB and the process CG being unclear. fwiw, David has previously raised other aspects of this issue, and in response it is already on the list of agenda topics for the next AB F2F.

[fantasai]

@michaelchampion You wrote “ The baseline is the current process, not some branch of the Everteal proposal.”

Indeed, the baseline is the current Process. And the current P2020 proposal is much closer to the current Process than what's being proposed here. The P2020 proposal simply renames “Candidate Recommendation” to “Candidate Recommendation Snapshot” (and adds the ability to publish “Candidate Recommendation Drafts” to represent intermediary EDs between snapshots), whereas the proposal here to make the snapshots purely an IPR step is substantially different from the current Process because it removes the requirements and approvals that currently apply to a CR.

I think if we want to consider removing those requirements and approvals from CRs, we should consider that for a future revision of the Process. If nothing else, it's a bigger change from the current Process, and therefore a larger difference for the AC to approve, and P2020 is already a large change. But also, deferring to a later revision of the Process lets us try out the lesser change that the current P2020 proposal represents and fine tune it based on the actual experiences of the various WGs, including HRGs as well as WGs. (Also, as @jeffjaffe points out, it is quite late to adopt this proposal, given that it's significantly different from what was presented at TPAC.)

With respect to the Service Workers comment in #346 (comment) Afaict the biggest issue Service Workers has and insists on resolving that they have multiple branches and multiple publication points for their spec. 90% of their comment was a complaint about that. They want one single specification of truth. We are solving that with the Candidate Recommendation Drafts and continuous IPR commitments already.

The other 10% was an offhand comment about what their ideal process was, which we are approaching but not meeting exactly because our objective is not to only satisfy Service Workers editors, but also other stakeholders at W3C.

@chaals The status quo (whether you're referring to current Process or the 2020 proposal) isn't timed snapshots, so I'm not entirely sure what your comment means.

[frivoal]

I agree with @jeff and @fantasai. This suggestion would be a radical departure form the Process we know, much more than everything else in everblue/teal. If the snapshots are automatic, and the drafts published in-between have no checks other external to the group, then all enforcement of the formal criteria for being at CR fly out the window, including wide review, formally addressing comments, timely addressing of formal objections, etc.

This isn't just changing how easy it is to get to and maintain a CR, it is changing what a CR is, from something that has a reasonable degree of commonality across the W3C through formal rules enforced by a "director" review, to whatever each individual group wants it to be (and our groups are much more diverse than what the WHATWG is dealing with).

What we have presented at the TPAC was not a redefinition of the maturities on the REC track. Actually not changing the meaning of the various maturities was presented as an explicit design goal.

More recently, the AB resolved to accept the Everteal proposal, modulo some fine tuning (which this isn't).

I think this is time we let this rest. I don't really want to go back 6 months a restart discussing whether evergreen (which this effectively is) is a better idea than everblue/teal.

[plehegar]

I concur that, if we remove the the current linkage between snapshot and horizontal review checks, then there is nothing to force Groups to address horizontal reviews, nor an incentive to even request horizontal reviews. This is already a little discomfort that those checks may not happen for 24 months (the longer time period a Group can continue to publish a CRUD without publishing a snapshot).

[michaelchampion]

Speaking only for myself, it's clear that the disagreements here are more fundamental than issue 346. @plehegar frames it clearly:

if we remove the the current linkage between snapshot and horizontal review checks, then there is nothing to force Groups to address horizontal reviews, nor an incentive to even request horizontal reviews

That and other AB/W3M comments on this issue seem to assume several things that I don't think are true:

1. It assumes an adversarial relationship between spec writers and horizontal reviewers. In my experience, spec writers and implementers put a very high value on I18N, A11Y, and security. They have no interest in shipping products that don't work for a very broad range of consumers, or have vulnerabilities. They treasure advice from real experts that helps prevent such problems early in the development cycle.
2. It assumes patent commitments are a valuable "carrot." In my (recent) experience, patent commitments from other W3C members are nice to have but not business-critical; real patent threats come from trolls who don't participate in standards.
3. It assumes the process can "force" people to do things that have more cost than benefit. In my experience, people will stop investing in advancing a spec's formal maturity once it's "good enough." It's the outside world -- real implementers and web developers -- that determine whether a spec is "good enough", and resources such as CanIUse and MDN (not W3.org/TR) are where they go for authoritative guidance.

Given the AB's decision, just close this issue. I don't want to distract you from your chosen path now that you've decided not to seek Process CG consensus on it, and keeping this issue open makes it an attractive nuisance for dissenters like me.

If I were still on the AB or AC, I'd keep arguing for the points Wendy makes early in this thread: "keep snapshots easy to generate and primarily legal-focused, while putting signals of horizontal review status prominently on the CR Update/Working Draft document" and to LOWER the "hurdles to WG agility, that we're trying to fix with Process updates." But I've handed off those roles to people who are better at picking their battles than I am.

[caribouW3]

In the current process2020 proposal, the CR drafts will just be drafts, like we had WDs between CRs in the past (no longer trendy, WGs only have EDs between CRs nowadays, but this created the issue that people would not see TR publications for too long). The good thing: it's avoiding the sense of "going back to earlier stage in the process". The bad thing: some CRs will be very drafty and unreviewed (by HR), depending on groups, so that maturity level is basically an "unstable release". Wendy's proposal is somewhat fixing this, IMHO.

[dwsinger]

intending to close as WONTFIX on the Jan 22 Process CG call. Please comment if we are missing something. Note that CRUD is expected to be called CRD, which is better.

[css-meeting-bot]

The Revising W3C Process CG just discussed Rethinking CR Snapshot/Drafts, and agreed to the following:

• RESOLVED: Mark #346 closed on Jan 22nd call

The full IRC log of that discussion

<fantasai> Topic: Rethinking CR Snapshot/Drafts
<fantasai> github: https://github.com//issues/346
<wseltzer> q+
<fantasai> florian: I would like to close this as WONTFIX, because, roughly speaking, this is rejecting EverTeal and adopting EverGreen
<fantasai> wseltzer: I do not object
<fantasai> wseltzer: Whether or not there would have been an alternate direction to go, nobody else is pushing strongly
<dsinger> I lost this argument at the AB, so though I agree with Wendy, it’s probably too late
<fantasai> florian: I would like also to mention that if later we realize this is a better direction to go in, we can make that change
<fantasai> plh: ???
<fantasai> dsinger: I lost this argument at the AB, so while I agree with Wendy, I don't think it's productive to keep pushing on it
<fantasai> dsinger: However a long discussion a month ago, a lot of contributors, good to review to make sure we didn't miss anything
<fantasai> dsinger: ...
<plh> s/???/I do believe it's a significant change and changing later will be difficult, but no alternatives/
<fantasai> florian: Suggest to close it, and if anyone thinks a side discussion yielded a relevant issue, we open a new issue for that
<fantasai> dsinger: I'd like to leave it open until the next call, with a comment that we plan to close it on the Jan 22nd call
<fantasai> florian: If we were to accept this, we'd have to do a major rework of everything
<fantasai> florian: so I don't think we can accept this
<fantasai> dsinger: agree
<fantasai> dsinger: Just check for comments
<fantasai> dsinger: Would like to address issue of naming?
<fantasai> fantasai: Currently renamed to "Candidate Recommendation Draft"
<fantasai> fantasai: Hooks in nicely to Patent Policy, which needs us to define it as a "working draft" for the resign clause to trigger correclty
<fantasai> RESOLVED: Mark #346 closed on Jan 22nd call
<dsinger> zakim, q?
<Zakim> I see wseltzer on the speaker queue
<dsinger> ack ws
<fantasai> jeff, that's cool, file an issue? :)

[wseltzer]

As the one who opened this issue, I propose we close it as overtaken by events.

[dwsinger]

Anxiety was expressed on the Jan 29th call, leaving open for conversation

[michaelchampion]

As I recall, it would be "closed" for Process 2020 but remain open for 2021, correct?

[dwsinger]

Correct. I would therefore like to remove Agenda+ and assign it to the 2021 Milestone.

[frivoal]

Correct. I would therefore like to remove Agenda+ and assign it to the 2021 Milestone.

Done. Agenda+ was just a leftover from last time.

[jeffjaffe]

I recommend this be closed. We are not going to re-envision this split in 2021; we just created this split.

We might instead streamline the text of the Process document which is a different issue.

[css-meeting-bot]

The Revising W3C Process CG just discussed CRUD/Snapshot Split, and agreed to the following:

• RESOLVED: Close

The full IRC log of that discussion

<fantasai> Topic: CRUD/Snapshot Split
<fantasai> dsinger_: Can we finally close this issue?
<fantasai> wseltzer: Please do!
<fantasai> RESOLVED: Close
<fantasai> github: https://github.com//issues/346