How do we reach consensus?

[arp242]

There are a few different options, in no particular order:

1. Explicitly require normalisation.
2. Explicitly forbid normalisation (#993).
3. Make normalisation implementation-dependent.
4. Change the spec so that normalisation is no longer needed for most cases (#990).
5. Backout Unicode keys for now (#979).

(Am I forgetting any?)

I DO NOT want to talk about the content of the options here, but rather about how we can reach some form of consensus (preferably unanimously)? A few people have said things like "I would prefer X, but I'm okay with Y too". Okay, so maybe there's an option that can work for everyone?

Personally I'd be okay with options 1, 3, 4, and 5. That's actually all but one. I've made it pretty clear what my preferred path is and for what reasons, but I'm basically okay with most options except what's in main now.

Perhaps one option might actually be acceptable enough for an overwhelming majority?

There are still details that could (IMHO should) be addressed is we go with 1 or 3, and my proposal also needs a bit more work, but the main issue is the broad direction to go in – details are things we can discuss/change later, and are not necessarily highly contentious. It's just the broad direction that is.

So I propose everyone lists the options they find "acceptable enough", and let's see what overlap there is?

Or if anyone has a better idea? Ranked-choice voting? Something else?

I'm also okay with @pradyunsg just choosing an option (which may be an option I dislike) and we'll go with that. Hell, at this point I'd almost be okay with a random stranger off the street just choosing an option and just going with that.

Quite frankly I feel all of this is no longer worth my time. Things have become unpleasant and at times even silly. We're at about 292 comments in a quick count, and it's not just blocking the release of TOML 1.1, it's also chasing people away (as we've seen yesterday) and setting bad blood among regular contributors.

It's the most-discussed issue in the history of TOML by a considerable margin. All for what's really a minor feature that a relatively small number of people have asked for...

[erbsland-dev]

Thank you for summarizing the situation and proposing a clear path forward to find some middle ground. I agree, this has become a protracted discussion and it's time to make a decision, not just for the release but also for the general well-being of the community.

As for the ways to reach consensus, I like your idea of everyone listing options they find "acceptable enough". It's a practical approach that might reveal a majority-aligned option.

Alternatively, if it's agreeable, we could indeed employ something like ranked-choice voting, as you've suggested. This could provide a more nuanced understanding of community preference.

However, given the volume of discussion we've had, I'd also be fine with @pradyunsg or any other trusted maintainer making an executive decision. At this point, even a 'random stranger off the street' scenario wouldn't be the worst thing, as you humorously noted. The priority now is to move on and continue with the more substantial aspects of TOML 1.1.

Regardless of the direction we choose, it should be one that minimizes future debate and allows us to refocus on other important issues. Time spent here is time not spent elsewhere, and we all have valuable contributions to make beyond this singular issue.

Let's decide on a method for reaching consensus and proceed. The community has more to gain from resolution and action than from endless debate.

For me personally, I fully agree on options 2, 3, and 5 and could accept 4.

[marzer]

(Am I forgetting any?)

Looks complete to me. Your summary of the situation is a good one.

At this stage I would enthusiastically accept options 2 or 4, grudgingly accept option 3, and see that option 5 is an unfortunate, but viable, last resort. I am very much looking forward to moving on from this topic in some fashion 😅

[ChristianSi]

@arp242 Aren't we already close to consensus here? I would certainly have thought so before you opened this issue. The discussion in #966 was long and (I think) fairly good. Initially some people (especially me and @eksortso) argued for normalization, but when we were persuaded to come over to the other camp, I think a clear consensus regarding "no normalization, at least not by default" has begun to emerge, weeks if not not months ago. So that's option 2 in your list, and #993 looks like a good way to closure.

Now you say that option 2 is the only one you're not OK with, which I must say surprises and baffles me a bit. I can't read it from your comments to #966, where you said, not so long ago, that normalization is 'not "on the table" as far as I'm concerned'. In fact, I can't remember reading a single comment of you clearly in favor in normalization, even at times when @eksortso and me still were so.

Now, all concerns need to be taken seriously, but I still wonder what your reservations against option 2, or specifically #993, are?

[arp242]

Now, all concerns need to be taken seriously, but I still wonder what your reservations against option 2, or specifically #993, are?

I explicitly said I don't want this thread to be about the content of the matter, but rather about how to move forward. And I have written about this over the period over a year now, and in the past week in particular, so I don't really know what to add anyway, beyond just repeating what's already been said.

[pradyunsg]

Thanks for filing this @arp242! I haven't yet found the time to catch up on all the things being discussed, so I appreciate that you've filed this. :)

I do think the way that this is going to end up going is that I'll have to end up making the call on this and we'll deal with the aftermath of that choice once the dust settles on the release.

---

I'll set aside some time in the week after the coming one (i.e. 9th Oct) to catch up on this discussion and make the final call here. I currently don't have a strong leaning towards any of the options proposed, though 5 is likely the safest last resort as @ChristianSi mentions.

[ChristianSi]

I mentioned what?? 5 is the option I would least like. In fact, I would leave the project if we release 1.1 without Unicode support in bare keys. We have promised bare keys in arbitrary languages, and it's time we deliver.

[arp242]

Why do you care so strongly about it?

[ChristianSi]

If I have strong feelings on this, it's probably in part because English is not my mother tongue. I have fought for extended bare keys since I arrived here, very long ago, and sometimes I feel like a clear minority under mostly anglophones with little understanding of, and not much sympathy for, other languages.

[arp242]

It's not a matter of "anglophones" or sympathy, it's a technical matter, with pros and cons. I'm not a native English speaker either, and previously LongTengDao argued against it, and he's Chinese. Making this "us vs. them" or a moral matter is probably not helpful. If you want to be angry at someone then be angry at the Unicode people for making this harder than it needs to be.

(Besides, you can't actually spell English with just ASCII either, or any other European language AFAIK (the only language I know that you can is Indonesian). Zoë, naïve, Sinn Féin, piñata, façade, cause célèbre, raison d'être, née, Motörhead, Māori, etc. all occur in English – many of these are loanwords, but that's kind of how language works).

[marzer]

I mentioned what??

@ChristianSi I think @pradyunsg might have got you and I mixed up; it was me who said I recognized that option 5 is viable as last resort (not that I want that at all, but it would resolve all the interminable disagreement and circular discussions about normalization, which has been going on far too long).

[ChristianSi]

It wouldn't resolve anything, since we would still have to decide what to do about the normalization of quoted keys (and string values too, BTW). So 5 by itself is not even a complete option, even then we would still have to decide whether to choose option 1 or 2. The only reason we haven't done so earlier is that nobody had brought the issue up.

[marzer]

I meant that it would 'resolve' it in the sense that we'd no longer need to focus on resolving the key normalization disagreement as a blocker for v1.1, and could focus on other other things, bringing unicode bare keys back after that. Kicking the can down the road, yes, hence it being a last resort, but at least the other good stuff in 1.1 would actually see a release.

and string values too, BTW

Please no. Save that for a different discussion/PR, because that is a whole new can of worms, and totally off-topic here IMO.

[ChristianSi]

@marzer It's not really a different issue, because if we decide to normalize keys, but not values, that would open a whole new can of worms, since what's a key in one table may well appear as value in another one. But I agree that that's to be discussed in the normalization issue, not in this meta-issue.

Also, that it doesn't talk about normalization is really a bug in the current spec, since (theoretically, at least) it leaves the community divided, with some parsers normalizing while others don't. So I don't think we should leave it unaddressed in TOML 1.1.

[marzer]

So I don't think we should leave it unaddressed in TOML 1.1.

To be clear: I'm not disagreeing with you on that - I would very much like to address it in 1.1 too. (#687 was my issue after all! I greatly sympathize!)

...but, I'm also saying that I recognise key normalization has been an utterly paralyzing topic for this group, and it threatens to block the work of everyone else in 1.1 if we can't figure something out.

The point of this meta-issue was to gauge current opinion, and one of my opinions is that option 5 sucks but might be necessary. That's all.

[ChristianSi]

key normalization has been an utterly paralyzing topic for this group

Was, or is? As far as I can see, we're have reached a nearly unanimous consensus that we want to forbid normalization, with @arp242 being the only one opposing. And even he isn't really arguing for normalization, as far as I can tell. So I honestly think the issue is more or less settled.

[marzer]

I don't know anymore, to be honest 😅 Lets wait to see how the chips fall and/or what @pradyunsg's read of it all is.

[arp242]

Was, or is? As far as I can see, we're have reached a nearly unanimous consensus that we want to forbid normalization, with @arp242 being the only one opposing. And even he isn't really arguing for normalization, as far as I can tell. So I honestly think the issue is more or less settled.

Well, that's an easy claim to make if you quite literally bully away the people who disagree. I don't want to be harsh here, but let's not mince words about what happened the other day either, or refuse to acknowledge what this means for "consensus", or the perception thereof. (And for what it's worth, although the apology is not mine to accept, I have no doubt it's genuine, nor am I claiming I'm always perfect either – related comment from a few days ago which applies here as well).

That case was very clear and marked, but I wonder how often this happened without people announcing it. Related: https://www.arp242.net/censorship.html – If I didn't have the vested interest of maintaining an implementation I would have given up long ago, because what do I care about some stupid specification? The ferocity with which some people have argued some things has been quite strong and off-putting at times (not just Unicode).

Very few people actually comment here in the first place, and combined with the above quite frankly I don't put much stock in "consensus" necessarily, which is a bit ironic since it's in the title here, but this was also an attempt to get some other people involved rather than the same handful.

It also doesn't help that some of the people with the strongest opinions also seem to be the people with the least investment in TOML. It's easy to argue for things if you're not the one dealing with the fallout.

I get the feeling some people see TOML as some sort of art project, whereas I see it as a practical tool to solve practical problems.

[pradyunsg]

I think @pradyunsg might have got you and I mixed up;

Indeed! Sorry @ChristianSi and @marzer for the confusion!

it's probably in part because English is not my mother tongue.

Likewise.

[ChristianSi]

@arp242:

Well, that's an easy claim to make if you quite literally bully away the people who disagree.

Indeed, if something like that happens it's very unfortunate of course. But I've read through #966 again and (leaving aside your somewhat hard-to-summarize position), the only people who had argued for normalization were @eksortso and me. And we were not bullied away, but rather won over by arguments and given time to think, I'd say. And in my case also by looking at how other standards (JSON etc.) handle these issues. You used to argue against normalization too, e.g. here. And even today, you don't exactly seem to be a friend of requiring normalization.

@wjordan felt hurt by something that @marzer had written, I suppose in the heat of the moment, and @marzer apologized later. Stuff like that is unfortunate and should of course be avoided if possible – but like you say, nobody is perfect. Anyway, @wjordan hadn't argued for normalization either, but simply suggested to optionally print a warning if non-normalized keys (not in NFC form) are encountered.

So looking at the entire discussion, only two people, @eksortso and me, had ever really and unambiguously argued for normalization, and we were not bullied away, but persuaded to come over to the other side. That's why I think that we've essentially reached consensus, or something very close to it.

[arp242]

It's not about "normalisation [yes/no]", it's about "is the current approach going to cause problems [yes/no]". That was ALWAYS my objection and was also wjordan's objection, and you're twisting that in to something else. Normalisation is one possible solution. If you know of others then feel free to offer them.

I expanded on why I think it will cause problems extensively for a long time, across multiple discussions, including a fairly long PR just last week. Yet here you say you have no idea what my argument even is. Ehhh??

You're not listening at all to what's being said.

[ChristianSi]

@arp242:

Yet here you say you have no idea what my argument even is.

I never said that. Please don't put words into my mouth.

But neither do I want to put words into your mouth, which is why I said that I can't summarize your position briefly. And indeed I don't think I could, or should even try to do that.

[uyha]

Maybe I missed the discussion somewhere, but was there a discussion about the option of forbidding the parsers from doing normalization and requiring that the input is already normalized (if needed) before being fed to the parsers?

I know that the "if needed" part is doing a lot of work here, but the context for whether normalization is needed or not is only available to the programs that are being used to produce TOML files, hence it should be the responsiblity of those programs to make sure the binary represenation of those TOML files are already what their users would want. With that in mind, the TOML spec for sure shouldn't say in whether normalization is desired or not, but only to enforces that no normalization can be done on the input, and puts up a big warning sign say if normalization is needed, do it beforehand.