Proposes changes to wide/horizontal review

[LJWatson]

Related discussion in #130
Proposed text also here https://www.w3.org/wiki/Wide_and_Horizontal_Review

[LJWatson]

It is likely that there are references that will need adding to this. I erred on the side of caution because I couldn't immediately see a discernable pattern to follow.

Where the Horizontal Review Groups are defined there is no referece to a WG/IG responsible fo security review because there isn't one. I understand from @plehegar that we are close to the point where all horizontal review requests will be managed and tracked on Github, so I suggest the definition of Horizontal Review Groups be aupdated when that happens.

[wseltzer]

All of this looks like good guidance. I'm uncomfortable adding more detail to Process.

[nigelmegitt]

This is moving in a really helpful direction. I've added quite a lot of comments, most of which are details.

The one that really stands out to me as a "we must fix this before proceeding" one is the 60 day minimum period before CR transition request one. I don't think it says what I guess was intended, or what I think it should say.

It would also be really helpful to enumerate precisely the HR documentation requirements, because that makes them easier to meet and test, and because we can, since we are in control of the HR process.

[nigelmegitt]

I like this "must" and I'd like it to be even clearer, to say:

• the notification happens after FPWD publication, and is not a pre-requisite to enable publication.
• who has the action to send the notification? I think it is staff, and in practice they achieve the action by means of an automated tool.

[LJWatson]

I don't think the Process needs to say who sends it or how, just that it must be sent. I'm also reticent about making anything dependent on tooling (especially tooling we may not have).

[nigelmegitt]

I'm definitely not proposing that we enshrine any tool usage! I do think that where actions are required it is important to define in some way the scope of who has those actions. It could be as broad as the WG, I guess.

The flip side of this is, this Process wording requires a thing to happen, but what would be the result if it did not happen? Apparently nothing. We could add an evidence requirement that it was done as part of what is needed to request transition to CR, I guess.

[frivoal]

There's a little bit of technology dependency in relying on announcements to a particular mailing list, but that's relatively minor, easily fixable the day we want to move to something else than email, and pre-existing in the test, so let's leave that aside.

I do think it would be good to say whose responsibility it is to make that announcement, not to luck us into a particular mode of operation, but to be clear whose responsible. For example, if we say it's the Team that's in charge, then using an automated tool is fine, so is sending a mail manually, and so is noticing that the Editor already did it and leaving it there. But to avoid situations where everybody agrees it should be done, but all think that it's someone else's job, having someone be accountable is good.

I'd go with something like:

When a [=First Public Working Draft=] of a specification is published
the Team must ensure that a notification is sent to [email protected].

[nigelmegitt]

I like that suggestion @frivoal .

[LJWatson]

I've updated the text in both the wide and horizontal review sections to be:

When the First Public Working Draft of a specification has been published, the Team must ensure that a notification is sent to >[email protected].>

Hopefully this clarifies who has the responsibility, and also that the notification is expected once the FPWD has been published.

[nigelmegitt]

I'd like this to say that the review should be solicited from FPWD at the latest. This wording suggests that it's okay to solicit review in a piecemeal fashion giving different stakeholders longer or shorter review periods. That might be okay but I think the SHOULD ought to prompt WGs to solicit feedback sooner rather than later, and to treat all stakeholders similarly while giving the freedom to make local exceptions if that is appropriate.

Should we allow consideration of WR sought on pre-FPWD drafts to count? Since HR is part of WR and the requirements in this proposal include stating that HR needs to begin before entering Rec Track I think the implication is probably yes, in which case we should say that explicitly.

[LJWatson]

Is there a reason you worry that groups could be offered different turnaround times at this particular point, as opposed to any other?

I'm hesitant to say much about wat happens before FPWD because more often than not these days that happens in a CG and outside of the Process' remit. It's tricky though.

[cwilso]

I do know that several HR groups have expressed a desire to participate in reviews in those early stages, though - e.g. the Privacy Interest Group wants to (and does) review WICG incubations.

[dwsinger]

"As a specification is being prepared for, and then moved along, the Recommendation track…" ?

[nigelmegitt]

Is there a reason you worry that groups could be offered different turnaround times at this particular point, as opposed to any other?

I think if it looks like W3C is treating different groups differently without some good reason then it would reflect badly on W3C.

I'm only picking it up here because this is the change being proposed. I suppose I could try to review the whole Process for it and open a separate issue, but I wasn't particularly aware of it as a problem elsewhere before.

[LJWatson]

I've updated the text to use @dwsinger's suggestion:

As a specification is being prepared for, and moved along, the Recommendation Track, the group responsible for the specification must solicit a horizontal review from each of the horizontal review groups (in many cases, submitting a self-review to the reviewing group will satisfy this requirement).>

[nigelmegitt]

This new wording works for me.

[nigelmegitt]

Proposed alternative: The Groups and external organisations documented in the Working Group's charter:

• widen "Working Group" to "Group" since charters can list CGs, IGs and BGs for example
• clarify that these other organisations can be external (they may have formal liaisons with W3C but we don't need to go that far here).

[LJWatson]

I've changed the bullet to:

The Groups and external organisations documented in the Working Group's charter or with which the Working Group has relevant liaisons>

[nigelmegitt]

propose to add including organisations with relevant liaisons to the Working Group to be super clear that liaison organisations should be solicited.

[frivoal]

I agree in principle, but isn't that redundant with "external organisations documented in the Working Group" as you suggested in an earlier point? Or do we have liaisons that aren't listed in the charter?

[nigelmegitt]

Indeed, there is nothing to tie the listed liaisons to the groups listed in the Charter for a WG. It is possible for liaisons to be established with a WG mid-Charter-period, for example.

[nigelmegitt]

I would mark this as resolved but I do not seem to have the rights to do that on this pull request.

[nigelmegitt]

Is this "may include" intended to be a closed whitelist of the only appropriate kinds of evidence, or an open list of which these are two examples?

[LJWatson]

The latter.

[nigelmegitt]

Thanks @LJWatson it would be good to find an editorial way to make that clearer.

[nigelmegitt]

Again, this is a post-publication action for staff, not a pre-publication requirement, right?

[LJWatson]

As before.

[nigelmegitt]

@LJWatson likewise!

[nigelmegitt]

This is a big problem, because it puts a 2 month brake on all transitions. We should not be adding new delays into the process. This delay is present with this proposed wording even if all the Horizontal Reviews are completed earlier. If what is intended is that HR groups get at least 60 days to review all specifications, then it could be reworded as:

Working Groups must not request transition to Candidate Recommendation either until at least 60 days have passed before Horizontal Review has been solicited or until all Horizontal Reviews have been completed, whichever is the earlier.

I'm sure someone could word it even better while keeping the same sense.

A different/additional option is to move this requirement from being stated here like this to being a permitted evidence response when requesting transition to CR, i.e. a WG would be allowed to cite "no response" and provide evidence that they sent the HR request >60 days before the transition request. I've included this in a proposal for a different list of allowed evidence below.

[LJWatson]

I don't understand the first part of your "either"?

I can see the logic in the second part, but I'm not sure it entirely resolves the whole problem. namely that WG could cajole HR groups into responding to a timetable they can't meet, then using the lack of HR response as an easy pass on heading to transition without it.

[nigelmegitt]

WG could cajole HR groups into responding to a timetable they can't meet, then using the lack of HR response as an easy pass on heading to transition without it.

Ah, right, thanks, there's a point I missed, which is that an HR review may have begun within 60 days but not yet be completed, (or in a "we went as far as we could" state). The more I think about it the more my second option of dropping this requirement and just adding it into the requirements for transition request is an easier fix, because my proposed list doesn't include "HR review started but we're still working on getting to resolution", and it's clearer.

[LJWatson]

I've updated the text based on @dwsinger's suggestion:

Transition to Candidate Recommendation must not be requested unless either the Horizontal Review Groups have been given at least 60 days notice to review the specification or the Horizontal Review Groups have completed their reviews.>

[nigelmegitt]

We need to say somewhere that a well-formed HR review includes a timeframe for response, which should either be predefined or have been negotiated out-of-band by staff and/or Chairs.

[nigelmegitt]

This include list doesn't tally up with the must-do list above. If the must-do list has been completed then the evidence is exactly enumerable, for each HR group, as:

1. documentation that the HR group did not consider a review to be necessary OR
2. documentation that at least 60 days have elapsed since the HR request was made, in case a 'no response' is being cited OR
3. documentation that the HR group completed their review and raised no issues OR
4. documentation of both:
   4.1. the list of successfully resolved HR group's issues AND
   4.2 the list of all remaining unresolved HR issues including, for each of them, evidence that a determined effort was made to find a resolution

[frivoal]

The must part of this sentence is redundant with the entry criterion of relevant stages on the REC track, which already call for this review.

[LJWatson]

I've updated the list per @nigelmegitt's suggestion, but I'm not sure what you mean in your comment @frivoal?

[nigelmegitt]

It would be helpful to add here some words to the effect that WR comments received on a spec that is in CR or later should be noted and resolved where possible but that it could be reasonable for WGs to defer those to a future iteration of the specification itself.

I say this because looking at the delta relative to the previous text here, there was a strong sense that WR feedback before CR is somehow weightier than WR feedback during CR or PR, or even Rec. This sense was borne out of the fact that WR must be demonstrated in order to transition to CR but further WR need not be solicited after CR.

My concern here is that WGs should be allowed to make good progress through the maturity levels without being pushed into some kind of circular pattern whenever any comment is received on a spec, especially one that has already made it to CR.

[LJWatson]

I agree we do not want specs being held up. Equally I'm not sure how to put something as nuanced as you suggest into the Process in a way that is useful - sometimes there will be things caught at CR that are a showstopper.

Based on suggestions elsewhere in this thread, the following was updated to include the "heads up" note:

Horizontal Review Groups should be notified of all new and significantly revised sections of a specification that has already been reviewed. Note: failure to do so may result in the transition to Candidate Recommendation being delayed.>

Does that help with your concern? If not could you suggest some succinct words/placement that would?

[nigelmegitt]

That's great, thank you @LJWatson .

[nigelmegitt]

All of this looks like good guidance. I'm uncomfortable adding more detail to Process.

I'm uncomfortable with the lack of detail in the current Process :-)

[LJWatson]

Thanks @nigelmegitt these are all really helpful comments. When more people have had a chance to review I'll start making edits, in the meantime I've responded to a few of your comments for clarification/more information.

[dwsinger]

I think this is overall very nice. But the message of "do it early, enable it to be continuous" is undermined by the two sentences that use the word "solicit", which I think sends the wrong message of 'late' and 'episodic' review.

[dwsinger]

I think this needs to be re-phrased so as not to imply that solicitation of review can be done once, but that the HR groups have adequate notice and opportunity to review.

[nigelmegitt]

The topic of soliciting review seems important to me.

As a WG Chair, given that the HR requirement is fixed and known, I want the admin task of requesting review to be minimal. It needs to happen automatically at defined stages, and manually when the WG makes a significant change that would warrant a re-review.

The automatic request should be as lightweight as possible. I certainly don't want to be engaging in multiple 1-1 conversations with HR group Chairs/leads to agree timescales for specific reviews, if it can be avoided. I suspect that if that seems like it would be onerous to me for one WG, it would be an order of magnitude worse for the HR groups!

I know @plehegar is presenting something about this at the upcoming AC, but I'm not sure how far it will go and how much it should be written into the Process.

[LJWatson]

I think the sentiment @dwsinger is aiming for is expressed in this sentence:

Horizontal review should be sought continuously as a
specification matures, so that accessibility, internationalisation, privacy,
security, and technical architecture are considered throughout.>

But the remainder of the text is more explicit about the minimum that needs to be done, which I think is where @nigemlmegitt isaiming.

[dwsinger]

I think this needs to be re-phrased; you cannot ask for a CR transition any sooner that 60 days after the latest HR group was made aware of the specification and the opportunity to comment.

[dwsinger]

whoops, you can only ask for a CR transition if either (a) horizontal groups have been given at least 60 days or (b) they have completed their reviews. (To complete the transition, you also need to address the issues raised, of course).

The way I phrased it before put a mandatory 60-day delay, which is silly if the HR groups have completed their review earlier.

[frivoal]

I don't think either Léonie's text or David's quite work.

One (mis)interpretation of the text is that the HR solicitation sent at FPWD counts, and thus this boils down to CR must not be sooner than 60 days after FPWD. I doubt that's the intent. Similarly, if you get another round of HR, then change everything in the spec, and try to go to CR, it shouldn't work, regardless of how many days ago it was, since the review is now outdated. If earlier reviews don't count, to send a request for review 60days before CR without delaying CR by 60 days, you need to know when CR is, and not change anything of significance between requesting the review and going for CR. In the general case at least, that date isn't known, since the CR entrance criteria are quality driven, not date driven. As long as you have blocking issues, you cannot enter CR and you generally don't know how long it will take to solve them (nor whether additional blocking issues will be found in the meanwhile).

I think we should instead go with something like:

In addition to continuous cooperation with HR groups and to the notification sent to [email protected] upon entering FPWD, Working Groups should explicitly notify HR groups when when major changes occur in a specification which has already been reviewed.
Note: failure to do so early may cause delays in publishing a CR: reviews take time, and having received Wide Review, which includes Horizontal Review, is a mandatory criterion for publishing a CR. Allowing for 60 days or more is recommended.

[nigelmegitt]

I like this wording @frivoal and note that there's a key point here around which the precise wording hinges:

1. are HR groups being given a maximum of 60 days to make some sort of response, otherwise WGs can proceed? or
2. are WGs being required to give evidence of completed HR regardless of how long it takes?

I got the sense before that the intent is 1, which I'd be happy with as a Chair, because it lets my group make progress, but I think 2 would also be reasonable as long as we don't get a situation where WGs are blocked waiting for HRs to be completed by HR groups that are completely snowed under. In this latter situation, I think we need some pragmatism to allow something to give.

[dwsinger]

This is OK. I don't like the word "solicit", as previously noted. And I wanted to give groups the ability to progress if the HR groups don't respond; i.e. you can progress if they have responded and you dealt with it, or you've given them at least 60 days.

[frivoal]

by the time they become a [=W3C Recommendation=].

How about

before compatibility with existing deployments make potential issues impractical to address,
and a fortiori by the time they become a [=W3C Recommendation=].

[LJWatson]

I'm hesitant to bring compatibility with existing deployments into it, but I like the idea I think you're aiming at, so I've updated both the wide and horizontal review intro paragraphs to include:

The goal is the early identification and resolution of issues in order to make the specification as robust as possible before it becomes a [=W3C Recommendation=].>

And:

The objective of horizontal review is the early identification and resolution of issues in order to make sure specifications are robust in terms of accessibility, internationalisation,
privacy, security, and technical architecture by the time they become a [=W3C Recommendation=].>

[frivoal]

While these last 3 bullets are true for issues raised during HR, they're not limited to HR, and are also true for issues raised by anybody at any time. If this needs to be clarified, it should be clarified in the "Advancement on the Recommendation Track" section. But redundant requirements with slightly different phrasing are not good, so I'd remove theses 3 bullet points from here.

[LJWatson]

Happy to remove them but I wasn't sure if you meant they should be moved to the general section, replace what's already in that section, or just be outright deleted?

[dwsinger]

There is so much discussion and proposed change here. Is anyone else feeling that they can no longer be sure what the result of addressing it all would look like?

[LJWatson]

I was hoping we'd worked our way through the worst of the discussion on the original issue, when the proposed text was in the wiki and easier to read and edit. I'm now not convinced this was ready for PR, so I'm tempted to take the feedback here as best I can, edit the wiki version, kill this PR and return discussion to the original issue.

[wseltzer]

Recommend we not add to the process.

[nigelmegitt]

I was hoping we'd worked our way through the worst of the discussion on the original issue, when the proposed text was in the wiki and easier to read and edit. I'm now not convinced this was ready for PR, so I'm tempted to take the feedback here as best I can, edit the wiki version, kill this PR and return discussion to the original issue.

@LJWatson ach, that's a shame. It looks to me as though there are concrete proposals on many of the comments such that they could easily be resolved, and we'd only be left with a few outstanding ones. The good thing about having the PR rather than the wiki entry is that the result and the context can be seen.

(by the way, shouldn't we have pr-preview enabled here?)

[nigelmegitt]

(by the way, shouldn't we have pr-preview enabled here?)

opened as #417.

[dwsinger]

I would like to have short text in the process that is congruent with a standing document of some sort that gives the sought-for guidance but doesn't clog the process and is more easily updated. I think the process text could be both better and shorter...

[LJWatson]

@nigelmegitt once I started ging through the comments it turns out you were right - it wasn't nearly so gnarly as it looked.

I think I've either addressed all the comments or have asked for more information.

Having prev would definitely be useful though.

[LJWatson]

I have also updated the version on the wiki for easier readability.

[nigelmegitt]

Thank you for going through all the comments @LJWatson - I really appreciate it.

[dwsinger]

I think this is a great improvement over the existing text

[nigelmegitt]

Seems like losing the sense of this section is a shame; I thought this process text was very useful in framing how WGs think about their wide review. I wonder if, in the way that @frivoal suggested in https://github.com/w3c/w3process/pull/401/files#r426429326 this can be refactored somehow so that the definition of documentation for demonstrating due process has been applied to review comments can be moved to a place where it clearly applies both to all wide review comments including horizontal review comments.

[LJWatson]

Moving it makes sense. I'm just not sure of the mechanics of doing it in a way that does not get lost. if @frivoal can make it happen, since his Github fu is likely better than mine) that would e good.

[TzviyaSiegman]

Thanks @LJWatson. This looks really good.

[samuelweiler]

I would prefer to see:

• The team/Director free to specify who must be asked for HR and how.
• Groups required to show proof of having asked, according to the above process, within a specified minimum of lead time. (Right now, they sometimes don't ask.)
• The Director instructed to take a hard line on "you must ask for review". (Again, right now, the Director does not always confirm that they have asked.)

I really don't like:

• Naming HR groups in the Process. What happens if we want to change who does review or how? What happens if a charter expires? I note that when the WebSec IG closed, we specified a process for Security reviews to be coordinated by team - so we have seen a recent example of such a change - and this could easily change again. I don't want it baked into Process.
• To the extent that areas are named, not naming security.
• Separate [email protected] and [email protected] lists. This doc doesn't make clear why they're separate. Again, this should not be baked into Process.

We are evolving the tools used for this, and we need the space to continue that evolution. This PR is too much over-specification.

[pes10k]

Wanted to mention my support for including HR in policy (wearing both my Brave AC hat, and my PING co-chair hat)

[dwsinger]

@samuelweiler I'd be fine with separating this into short text for the process linked to a longer, more easily updated, explainer and guide. On naming groups: I agree, until I think of a chair who needs to know: who am I supposed to ask, and how? They need a list, or something that indirects to a list.

I completely agree that the process needs to give clear requirements and express principle, the guidance needs to be helpful, and the tooling badly needs a revamp, and that the process needs to stay flexible to accommodate improvements in tooling and guidance.

[samuelweiler]

On naming groups: I agree, until I think of a chair who needs to know: who am I supposed to ask, and how?

@dwsinger We have a document that answers that question. If you will humor me: I challenge you to find it. I think you will find it easily and quickly. Please let me know if that is not the case and/or if you find other, conflicting documents. (I've been trying to kill off all the others.)

I completely agree that the process needs to give clear requirements and express principle, the guidance needs to be helpful, and the tooling badly needs a revamp, and that the process needs to stay flexible to accommodate improvements in tooling and guidance.

Have you used the new issue tracker tooling? Take a look at: https://w3c.github.io/horizontal-issue-tracker/?repo=w3cping/tracking-issues

[TzviyaSiegman]

I think it's important to list the groups do HR in the Process. I hear @samuelweiler's point, but how often does the group that does HR change? That happened once. Is this really an ongoing concern? I think the greater concern is that WGs don't know who to contact or how to get started on HR.

[dwsinger]

Next year we have a goal of streamlining the process document. I would like to use the opportunity to move guidance, background, explanatory and other narrative material OUT but have it LINKED, clearly not part of the normative process. This seems like an excellent candidate to try out that approach.

[samuelweiler]

I think it's important to list the groups do HR in the Process. I hear @samuelweiler's point, but how often does the group that does HR change? That happened once. Is this really an ongoing concern? I think the greater concern is that WGs don't know who to contact or how to get started on HR.

Yes, it is an ongoing concern. Hopefully we will improve the security review process and change it again. I don't want to wait for a Process cycle to implement such fixes. (A change happened once in the last year. I'm new enough that I can't enumerate all the changes over time, but given that PING was first chartered less than ten years ago, I'm sure there have been more changes within human memory.)

You say WGs don't know how to get started. Where would you like that information to be found?

(I agree with you that some WGs are lost. I tend to think that adding redundancy leads to stale instructions - which actually makes the problem worse over time. So I would rather have a single authoritative source of information. As above, I don't want it to be the Process. If it is the Process, I want to remove the other source(s).)

[nigelmegitt]

You say WGs don't know how to get started. Where would you like that information to be found?

I'd rather a push model than a pull model. When FPWD is published, tools begin the HR process. The "WGs have to go hunting" model when every WG has to do it for every Rec track document is a massive waste of time.

[LJWatson]

@TzviyaSiegman

I think the greater concern is that WGs don't know who to contact or how to get started on HR.>

The overwhelming feedback I had from WG chairs, spec editors, team contacts and others during the user research that eventually led to this PR, was that people do not know what HR is, what it includes, when it is needed, or how to go about it.

I had a conversation just yesterday with an extremely experienced spec editor of many years asking me these very questions. When I described the way HR is managed at present they were not impressed, and I cannot fault them.

[LJWatson]

@samuelweiler

I note that when the WebSec
IG closed, we specified a process for Security reviews to be coordinated by team - so we have seen a recent example of such a change - and this could easily
change again. I don't want it baked into Process.>

This is exactly why we need to document this.

As WG chair I was not told of any "new process" for security HR. Only a month or so ago when I asked, for the umpteenth time, if there was yet a group within W3C with responsibility for doing security HR, I was told (again) that emailing you was the "process".

I still do not know where this is documented. Neither do any of the other chairs and editors I have spoken to about HR over the last 18 months. This is the extreme example, but the same is true for HR as a whole.

To the extent that areas are named, not naming security.>

If you can let me know which group is responsible for security HR at W3C it is a moment's work to add it.

[samuelweiler]

I still do not know where this is documented

As I asked last week: where do you want it documented (or linked from)?

(I hear your frustration. I'm frustrated, too. r12a and I spent some notable time cleaning up the documentation, and I don't want that to have been for nothing. I dropped some links to it in some obvious places where people might look or get confused by finding a similarly-named but inappropriate doc. Tell me where you want to find it.)

If you can let me know which group is responsible for security HR at W3C it is a moment's work to add it.

For good or ill, we have a process; we don't have a "group".

[dwsinger]

I want the process document shorter and cleaner and the guidance material linked from it, clearly as supporting material, see #356 . I assume review wiki is the document Sam alluded to above...

[samuelweiler]

I assume review wiki is the document Sam alluded to above...

Thank you for taking my challenge! I hope finding it was quick and easy.

[dwsinger]

Once you told me it existed, a search on the W3C site was easy. But I needed to be prompted to seek...

[frivoal]

A few comments on the wiki text

1

• The requesting Working Group MUST acknowledge receipt of each issue, then give each issue due consideration before providing an appropriate response;
• If the Working Group and the horizontal review group do not agree, they SHOULD make a determined effort to find agreement;
• When the Working Group requests transition of a specification, evidence of this process MUST be provided and any issues that remain unresolved MUST be indicated.

I don't disagree with what these points say, but:

• I believe they are redundant with the requirement to formally address issues combined with the definition of consensus.
• WGs always should respond to issue like that, even outside of horizontal review.

It is possible that due to some nuance of phrasing, these redundant requirements end up being slightly different, but that's not helpful, and makes the Process longer / more complicated than it should be than it should be.

My recommendation is to:

• remove these 3 bullets from the proposed text about HR
• if necessary, open a separate issue, not specific to HR, about how WGs ought to respond to issues reported to them. I think the current text is adequate, so I don't plan on opening that issue myself, but if someone sees a problem with it, we should fix it in all contexts, not just for HR.

2

When a Working Group requests horizontal review:

• The horizontal review group MUST acknowledge receipt of the request and indicate whether the review will be conducted;
• The horizontal review group SHOULD review the specification and raise relevant issues within the agreed timeframe;

This puts emphasis on the notion that HR is a periodic discreet event, all while we're calling for a more continuous process.
Let's take the example of the i18n group's review of css-text: over a period of multiple years, the csswg has solicited input from i18n on a large number of issues. The i18n wg has itself opened a large number of issues. (large number = multiple tens). We've worked together to resolve them, to good effect. To me, this is horizontal review gone well. This did not involve a formal call for review, to which there could be a formal acknowledgement. There were a bunch of nudges and communications along the way to help that liaison work well, but that's different.

I suggest deleting that text.

3

Similar comment about:

Transition to Candidate Recommendation must not be requested unless either the Horizontal Review groups have been given at least 60 days notice to review the specification or the Horizontal Review Groups have completed their reviews.

This is problematic for multiple reasons:

• it too emphasizes that HR is a discrete thing
• It highlights HR as a thing that happens late in the cycle
• CRs are fundamentally quality driven, not date driven, so while you can guess, you cannot know when you'll transition to CR until you're actually ready to do it. This means:
  • If you wait until the spec text has settled before you call for HR, it adds a mandatory 60 day period to CR transitions
  • you can't really know how far away CR is, so "at least 60 days before CR" could end up being "300 days before CR" if you guessed wrong.
    • If HR is a discrete thing and you don't wait until the spec text settles, then whatever change gets made between "maybe 60 days, maybe 300 days" before CR and CR don't get reviewed.
    • if HR is continuous (which it should be), a notice "maybe 60 days, maybe 300 days" before CR is has no particular significance.

I suggest deleting that text, and replacing it with some statement about the necessity to continuously work with HR groups (although we already have some statements to that effect, maybe there is no need to repeat). In combination with the paragraph immediately above and below that text, I think it still makes it clear that you cannot just skip HR, and that there are consequences for doing it poorly, without introducing the problems listed above.

4

Depending on where we end up with the above points, some tweaks to the final list ("The Director MUST consider…") might be in order too, but let's get back to that once we've talked about the rest.

[frivoal]

Given that #130 was closed , should this be too?

[dwsinger]

I can't tell. This is a PR, but along the way we said we had a goal of removing background/explanatory material out of the Process, and linking to it. Did we adopt any of this text? Should we adopt it externally and link to it?

@LJWatson comments?

[LJWatson]

I suggest we close this. I still do not think the Process is clear enough about what is meant by wide and horizontal review, but having gone around the "it should be in the Process, no it should be in /Guide, no it can't be in /Guide because it needs to be normative" circle several times, I'm at a loss as to how to resolve the matter.