Add a proposal for the Interop 2024 process

[gsnedders]

To set out the goals behind this:

• Create a clear process that’s easy to execute
• Invite proposals from anyone interested, with clear documentation of the project’s purpose and process — so potential proposers understand the project, its values, and the requirements for successful proposals
• Evaluate and assess proposals in a realistic and doable fashion, greatly reducing the amount of time needed to do so
• Reach a result where the Focus Areas and Investigations chosen support the mission of the Interop Project

---

This is long, and clearly not totally complete—but we think this serves as a good starting point for future discussion. We think that setting out, explicitly, the scope and the goals for the Interop Project (which is—really—out of scope of the broader Interop Team charter, as the Interop Team will own things above and beyond the project) serves as a good introduction for what the process should achieve.

[jgraham]

So, broadly that's what we did the first year for webcompat, and then we switched in 2023. One reason was that sometimes the web compat proposals have been very small; possibly literally just one bug. At that level it seems like a lot of overhead to file multiple issues. On the other hand the process in 2023 wasn't perfect either and there was some confusion about what was included. so I don't object to changing back, but my concern is that people end up filtering out individual proposals because they're "too small" when in fact the intent was to group them all along. This particularly applies if there's a fixed size for the priority list below.

[jensimmons]

Changing to be a different example, since, right, Web Compat is likely an exception to this guideline.

[jgraham]

Are multiple test failures actually a requirement? If there's a compat issue that's encapsulated in a single bug I don't think I should have to write a new test to to reach the arbitary requirement of "multiple failures" (in general I agree with the idea that final focus areas will have multiple test failures).

[jensimmons]

The word multiple can easily be stricken. I don't think we intended to exclude something that might only have one test failure. More that we were just assuming anything that someone would proposed does have multiple failures.

[jgraham]

It's unclear to me what the value of not eliminating if anyone proposes elimination is. Or at least: if someone makes it clear that they will veto a proposal in later rounds it may as well be eliminated early.

[chrishtr]

As I see it, the value is twofold:

• Preventing this phase from taking too long
• Opportunity to convince those who propose elimination otherwise in the next round, with additional evidence

[jgraham]

My supposition is that if you're proposing elimination early you're pretty sure that the proposal won't reach consensus (e.g. because there's a negative standards position). Given that a lot of the process is about reducing the overhead I'd personally err on the side of believing people if they say that something isn't worth carrying forward.

I agree that there could also be a situation in which people think that maybe something should be eliminated, but are more open to extended discussion in the next phases. In that case we should of course keep the proposal in.

Maybe the wording is fine, as long as people understand that there is basically no advantage to retaining a proposal that a specific org clearly indicates it's against at this stage.

[bkardell]

Preventing this phase from taking too long

@chrishtr I'm confused by this part of your comment. Doesn't this inherently involve this and all other phases taking longer than if we just said "ok, let's take that one off the table and work on the convincing somewhere else/for next year"? Presumably if someone is saying that at this phase, as @jgraham says it's for real reasons (like a negative and public standards position that we can point to in the disqualification notification) or something pretty significant (chrome, for example, said no to several things while in the process of redoing the layout engine because that makes it that much harder)... I think it seems pretty easy to present, have the group accept that and move on. This doesn't seem like the venue for convincing on all things, right?

[chrishtr]

I think it seems pretty easy to present, have the group accept that and move on

Nevertheless this will take time, and it's hard enough to make decisions already. The purpose of rounds 1 and 2 is about efficiency of easy decisions, so we should stick to easy decisions and not ones where there is any debate at all.

[gsnedders]

Yeah, as @chrishtr says—it's about making non-controversial decisions quickly.

It's also intended that the first round doesn't have too many subjective decisions—a spec either does or does not meet the criteria we've set (and I hope we don't get into disagreements here!), tests already show interoperability or they don't (we can of course get into a debate about whether the final 5% matters!), though "overly vague or broad" is much more so subjective.

@jgraham: was your original question intended to be "why not let people veto up front, for any reason"? if so—the current kinda short list of reasons for disqualification are there because they're largely what we practically did in prior years, and because things that are largely objective are likely easier to get consensus on.

[bkardell]

Is it possible that vagueness is causing us to be talking about completely different things? I think yes let's ask people to 'veto' here, and not at the end. That is what I am saying - I might be completely alone in this... But honestly, I also kind of dislike that we're even using words like "vote" and "veto" at all because they conjure up something that I don't honestly think is really happening here. I understand why, it's a kind of vote - but really what we're doing is more like using some joint worksheets to do complicated sorting and planning and negotiate (with what little power any of us actually has to do so) some plausible subset of things we all agree we could work on improving interoperability of.

Let's say two concrete examples:

• Somebody submits that we should work on interoperability of the is="" attribute. It's in the WHATWG HTML spec, which I think has always had the same definition "two in favor, none opposed". Except Apple is opposed, and has a public position many times stating as much, including a recent WebKit standards position. So... Is it disqualified in that case? Or does this require a "veto"? If the latter, we can say it now, why drag it out. It's just a particularly easy example, but it's not unique: People are going to submit things that have negative standards positions - is this the venue to debate it? I kind of think not.

• Somebody submits we should work on 'all the color stuff' a few years ago and it's pointed out that this requires much low level rework in order to actually bring the value, so it's not likely something that would actually improve this year and there are lots of other things where we think we could all agree and get stuff measurably done and shipped --- but hey we are actually working on all that low level stuff and totally support it! That isn't currently disqualified, and it's certainly not a "veto" - but it's a known known and it seems to me the group could choose to say "that seems totally reasonable, let's just exclude it" and doing that as early as possible just seems good... To me.

[jgraham]

Let's say someone proposes "Geolocation Sensor" for Interop 2024. It has a spec on a standards track at W3C, and tests. So per the criteria given we can't eliminate the proposal at this stage. But it also has multiple negative standards positions, and no evidence of implementations going ahead despite those standards positions.

It seems like a pretty safe bet to me that proposal is not going to reach consensus.

But per the process here, we have to take it forward in case we want to have a longer debate at some later stage. But it's not clear to me what form that debate could possibly take, and indeed debating proposals as a group isn't part of any of the later stages of the proposed process. So assuming at least one org is in favour in round 2, the proposal has to go all the way through to round 3, even though we had the information in round 1 to know it wasn't going to reach consensus (because a participant could have said so).

Obviously this isn't the most serious concern; eventually we'll incorporate the information that the proposal isn't going to reach consensus, and it won't get selected. But it is an inefficiency in a process that claims to be optimising for making the best use of everyone's time.

[foolip]

It would be great to keep round 1 relatively objective and efficient. As @gsnedders says, "things that are largely objective are likely easier to get consensus on." By only eliminating things that we all agree about, it also makes bookkeeping very easy, and we can do this in a single meeting as proposed.

I do agree there's value to eliminating early if one organization knows they'll veto it in round 3. How about we allow for vetoes also in round 2 to avoid the more time-consuming round 3 for those proposals?

Concretely: proposals move on from round 2 if they have some support and no objections.

[foolip]

Actually, strike that, I don't want to complicate things at this point with suggested changes to the process.

[jgraham]

I don't understand how this will work in detail.

Incoming proposals may be different sizes. Per the above text some proposals may be far too small to form a focus area on their own. So in order to do fixed-N selection I think we'd need to have already grouped the proposals into similar sized chunks at this point. Otherwise it's impossible to compare proposals like "fix one small but important bug" and e.g. "implement an entire new CSS layout mode".

Also it's unclear how to pick N. We probably need N large enough that we're guaranteed to get enough proposals that Interop 2024 is a reasonable sized project. But given overlaps and possible later vetos, that doesn't seem possible without allowing the process to be iterative. One point of view is "well even if we only end up with five focus areas, we'll know those are the most important ones". But I think that would be a mistake if we end up rejecting things that everyone does in fact end up working on just because Process.

[jensimmons]

N should be "slightly too large".

If you imagine we expect 35 proposals and 20 "open slots", then figuring out N does seem impossible. And honestly, this whole step could be skipped. Just take all 35 proposals to the next round.

But if you instead imagine we could get 150 proposals and have only 5 open slots... then this step makes more sense. It's a way to make a huge cut very quickly to get the number of proposals into a reasonable number. Why sweat the details over which is ranked 82nd and which is ranked 97th if we are only accepting 9 new areas?

We won't know how many open slots there are, but we could guess, for instance, it's 15 at most (probably less) (after grouping), as so... we could make N be... 35? Or 55? And cut the rest of the proposals in case many more than that come in.

We probably need N large enough that we're guaranteed to get enough proposals that Interop 2024 is a reasonable sized project.

Exactly. And it's fine if N is "too high". It's really just a protection against having way too many proposals in the later stage of the process, just in case the success of and attention on Interop 2021-23 brings us exponentially more proposals yet again this year.

[jgraham]

I think the fundamental problem with this as stated is that there's no way to pick N that works with proposals of very different sizes. For example it doesn't make any sense to have to pick between a proposal that represents a full focus area like "implement an entire new CSS layout mode" and one that's a small DOM bug that causes webcompat issues, and would be merged with many other proposals to form a full focus area.

Assuming we do get different sized proposals, it seems like we'd either need to collectively try to size/group proposals earlier in the process, or allow people to propose their own groups at this phase, or not fix N, but ask people to pick a number of proposals that they feel reflects a reasonable size for Interop 2024, and allow that to vary between participants.

(The other possibility is that we just don't get so many proposals that this stage is really needed).

[nairnandu]

+1 to jgraham's comment above about selecting 'N' being particularly tricky. Suggest each organization pick a reasonable number of proposals that are well-justified and that they can support.

[gsnedders]

One of the big questions that came up in the meetings in recent weeks was the question of when to do the grouping; it might be worth revisiting this after deciding when to do the grouping?

[jgraham]

How to do any grouping, and how to estimate/balance the overall size of the focus areas, seem like the most fundamental remaining issues. Previously we did that at the last minute by "feel", but that doesn't work well with the process set out here due to the ordering step.

Grouping before ordering doesn't work if that happens before the proposals have got detailed input from engineering teams, because we might group things that have very different levels of complexity or priority, and then be forced to assess the group as a whole on the basis of its worst elements.

But trying to order things of very different size and complexity also doesn't seem to work well (how does one tradeoff "fix a single test that represents a serious webcompat issue but is probably a few days of work and isn't going to be a focus area on its own" against "implement an entire new platform feature that could take multiple months of work"?).

[foolip]

The suggestion I like the best so far is this one from @jgraham:

ask people to pick a number of proposals that they feel reflects a reasonable size for Interop 2024

I think there's general agreement that we should not increase the number of focus areas, so at the very very most this could be 26.

This does require each organization to make some qualified guesses about what will make sense to group after round 3. In the case of web compat bugs this is easy. For some other proposals there might be disagreement, but the only effect of this is that a few more proposals move on to round 3.

Corner case consideration: Elsewhere I suggested we should also allow for vetos in round 2. If we do this, then there's no guarantee that we have N proposals after round 2. I think this is very unlikely, but if it does happen we could do another round of round 2 for the remaining slots. Presumably there would be no vetos that time.

[chrishtr]

Thanks for the detailed proposal! Most of my comments are nits and phrasing adjustments.

[chrishtr]

As I see it, the value is twofold:

• Preventing this phase from taking too long
• Opportunity to convince those who propose elimination otherwise in the next round, with additional evidence

[chrishtr]

Suggested change

NOTE: in the past, “Exclude” was a way to express many different kinds of sentiments — including: does not meet the minimum qualifications, or this is low priority to us. We expect “veto” to be used much less than “exclude”, because there are other ways to express those sentiments. Vetoing is specifically for objecting to a particular proposal being included *at all,* no matter how highly ranked other organizations might find it. It’s for expressing objection to a particular web technology, perhaps to something an organization has already expressed an objection to in the web standards process or otherwise has an objection to implementing.

NOTE: in the past, “Exclude” was a way to express many different kinds of sentiments — including: does not meet the minimum qualifications, or this is low priority to us. We expect “veto” to be used much less than “exclude”, because there are other ways to express those sentiments. Vetoing is specifically for objecting to a particular proposal being included *at all,* no matter how highly ranked other organizations might find it. It’s for expressing objection to a particular web technology, perhaps to something an organization has already expressed an objection to in the web standards process or otherwise has an objection to implementing. A veto should when possible come with a brief explanation to the other Interop Team members to help them understand the reasoning, and further encourage scoring rather than vetoes.

[jgraham]

My supposition is that if you're proposing elimination early you're pretty sure that the proposal won't reach consensus (e.g. because there's a negative standards position). Given that a lot of the process is about reducing the overhead I'd personally err on the side of believing people if they say that something isn't worth carrying forward.

I agree that there could also be a situation in which people think that maybe something should be eliminated, but are more open to extended discussion in the next phases. In that case we should of course keep the proposal in.

Maybe the wording is fine, as long as people understand that there is basically no advantage to retaining a proposal that a specific org clearly indicates it's against at this stage.

[jgraham]

I think the fundamental problem with this as stated is that there's no way to pick N that works with proposals of very different sizes. For example it doesn't make any sense to have to pick between a proposal that represents a full focus area like "implement an entire new CSS layout mode" and one that's a small DOM bug that causes webcompat issues, and would be merged with many other proposals to form a full focus area.

Assuming we do get different sized proposals, it seems like we'd either need to collectively try to size/group proposals earlier in the process, or allow people to propose their own groups at this phase, or not fix N, but ask people to pick a number of proposals that they feel reflects a reasonable size for Interop 2024, and allow that to vary between participants.

(The other possibility is that we just don't get so many proposals that this stage is really needed).

[jgraham]

One concern with this overall approach is that it doesn't provide an easy way to balance the overall composition of the project. For example it might be that we get two strong proposals for including, say, two different DOM Storage APIs in Interop 2024. In that situation there are a variety of theoretical and practical reasons why we might not want to put both in (e.g. concerns about too much churn, practical resource constraints, wanting the metric to be more balanced across the entire platform).

In the previous setup each org might choose the one proposal they thought was highest value and object to the other one. In the worst case this could lead to neither being included, which isn't ideal, but isn't worse than the status quo. In this process you might have some orgs rating one proposal high and the other low, and some doing the opposite, with the end result that both are included, even though that outcome isn't what anyone wanted.

I don't know to what extent a formal process change is needed here, vs just discussing any such concerns, but one can imagine at least trying to group the proposals by area of the platform and prioritising each separately.

[jensimmons]

Do you still have this concern, James? If so, any suggestions about how to address it? (I recommend reading the current language in the GDoc, and commenting there.)

[chrishtr]

Suggested change

	In this round, each organization ranks the proposals in order of importance from most to least. And submits any vetos they may wish to exercise.
	In this round, each organization ranks the proposals in order of importance from most to least. And submits any vetos they may wish to exercise, with a brief rationale for the benefit of the Interop team.

[jgraham]

I don't think the process can require a reason for exercising a veto. Of course I also hope in most cases people will be able to articulate reasoning, but nevertheless.

[foolip]

Indeed we can't require a reason, how about we make it clearly optional but encouraged:

Suggested change

	In this round, each organization ranks the proposals in order of importance from most to least. And submits any vetos they may wish to exercise.
	In this round, each organization ranks the proposals in order of importance from most to least. And submits any vetos they may wish to exercise, if possible with a brief rationale for the benefit of the Interop team.

[foolip]

Adding a date based on in-person discussion in #405.

[foolip]

Let me try to enumerate the remaining issues that we need to resolve here:

• Process for carrying over focus areas from 2023 into 2024. @jgraham suggested filing an issue for each focus area and discussing there if the remaining test failures are important or not. I think we should do that, and think it'll be easy-ish to agree about the carry over afterwards.
• In round 2, remove or define the limit N on the number of proposals each organization can pick. I suggest we remove it and go with "ask people to pick a number of proposals that they feel reflects a reasonable size for Interop 2024" as suggested by @jgraham. Also allow each org to group proposals as they think would make sense in this round.
• Define what's member-confidential, leaning on Create the Interop Team Charter #102. @gsnedders @jgraham can either of you suggest the wording you'd prefer?

Am I missing anything?

[gsnedders]

Am I missing anything?

• When in the process do we do the grouping into focus areas?

[foolip]

When in the process do we do the grouping into focus areas?

I'd suggest that each organization say in round 2 which proposals they're thinking will be grouped, of the proposals they're supporting. Then after round 3 we'll have to come to a joint team decision about it similar to in past years.

[foolip]

The only remaining work here for me to give an approving review is to remove the final section about carryover, or replace it with a short description of what we've concluded in meetings. I think something this would suffice:

All focus areas in Interop 2023 will be evaluated for carryover into Interop 2024. Only those focus areas which have consensus to continue will be part of Interop 2024. As part of this evaluation, the scope may also be changed, and focus areas can be merged or split when there is consensus to do so.

I don't think we need to say anything about the logistics (filing issues).

@gsnedders WDYT?

[jensimmons]

I've moved this draft to GDocs so that it's easier to edit:
https://docs.google.com/document/d/138F0bzFwywci9p5q7HMP4Rbi9E0iBjz4s0ZC6IE9324/edit?usp=sharing

[jensimmons]

Wait what? In public? We should debate this. Please see GDoc for more. https://docs.google.com/document/d/138F0bzFwywci9p5q7HMP4Rbi9E0iBjz4s0ZC6IE9324/edit#heading=h.jeelvxhy0wl3

[jgraham]

Suggested change

NOTE: in the past, “Exclude” was a way to express many different kinds of sentiments — including: does not meet the minimum qualifications, or this is low priority to us. We expect “veto” to be used much less than “exclude”, because there are other ways to express those sentiments. Vetoing is specifically for objecting to a particular proposal being included *at all,* no matter how highly ranked other organizations might find it. It’s for expressing objection to a particular web technology, perhaps to something an organization has already expressed an objection to in the web standards process or otherwise has an objection to implementing.

I still think this note, whilst non-normative, is nevertheless misleading and that it remains acceptable for participants to veto any proposal for any (public or private) reason. Although I agree with the sentiment that direct vetos are less likely in the proposed process, I don't think the Process should try to hint at the existence of legitimate and illegitimate categories of reasons to veto, particularly given that we don't have experience of how things will play out in practice, and so don't know what will actually be required to end up with a final result that everyone can approve.

(If this PR is merged, as per the previous meeting notes, I will reopen this as a new PR).

[jensimmons]

This has already been removed. Have you checked the Google Doc draft? That's where the current draft is. This PR is now irrelevant.

[jgraham]

This appears to still be placeholder text?

[jensimmons]

This has been addressed in the current draft, in GDocs.

[jgraham]

Ah, apologies, the meeting notes suggested that the PR was going to be merged, so I assumed this was the canonical version again.

[jensimmons]

I'm going to close this PR, because we've created a new version of the proposal, which is now at
#591