Results of TC39 presentation

[domenic]

I'm sorry to say that the committee declined to accept this proposal as-is. In the end, the concern (largely driven by @erights, although others were sympathetic) was that escaping cannot be done in a way that is totally safe in all cases, even with the extended safe set. For example,

new RegExp("\\" + RegExp.escape("w"))

is a hazard. (It does not matter that "\\" by itself is not a valid regex fragment. The above does not error to indicate that; it just silently creates a bug.)

Note that even if you attempted to correct this by escaping all initial characters, you then have

new RegExp("\\\\" + RegExp.escape("w"))

as a bug. @erights called this the "even-odd problem."

The general feeling was that to be completely safe you need a context-dependent join operation. The feeling was then that if author code wants to do unsafe escaping, the function is easy to write, but if something is going to be standardized, it must be completely safe. The idea that other languages are not held to this standard did not convince them, I'm sorry to say.

The committee recognized that you might not be willing to do work on a different, more complicated proposal. But, if you were interested, they think that a template string tag solution would be the best to investigate. Such a solution would be able to use the context of each insertion to do a safe join operation between any two fragments, instead of depending on string concatenation. Template strings can also be twisted to work in dynamic situations (e.g. those that this proposal would cover via new RegExp(pieces.map(RegExp.escape).join("/"))) by directly calling the tag function, probably with an adapter to work through the awkwardness of the parameter form for template tags. So this would be strictly more powerful. This was also preferred (for reasons I don't really remember) to a building-block approach of e.g. RegExp.concat plus RegExp.escape (used as RegExp.concat("\\", RegExp.escape(x))).

I'm pretty disappointed by this, and am sorry you and others sunk so much work into it with such an outcome. But, what can we do.

[jdalton]

I think most regexp escape methods are pretty simple. At least the ones in Closure Lib, Dojo, Lodash, Prototype, MooTools, Mout, YUI, & escape-string-regexp are.

This looks like a case of taking something with clear utility, boxing it into a context it wasn't intended for, then dumping it because it didn't fit the box.

[WebReflection]

But, what can we do.

we can ignore this decision and agree on a single standard method we, developers, can use, pointing at the possible footgun in a comment within the shipped code.

We put such utility in a CDN and we live happily ever after.

[matthewrobb]

This is certainly disappointing but I'm glad the results were posted here in this way. I love reading meeting notes but it would be great to have this kind of update on each proposal after discussions.

I'm okay with a template tag but still, this proposal would end up needed within that implementation so why not bring it in?

Doing URL escaping in the browser doesn't save a developer from SQL injection if they did something like the above but no one questions the usefulness of those existing escape functions.

[WebReflection]

Doing URL escaping in the browser doesn't save a developer from SQL injection

exactly, same as escaping HTML, or using even eval ... if you want to hurt yourself feel free to do it.

String concatenation is a reality, RegExp can be created on the fly within a Function too ... what the hack is the point of avoiding methods focused on increasing security?

Between totally unsafe and "maybe some noob will do something silly like that" there is quite an abyss.

Disappointing to say the least. Thanks gosh I don't care about these decisions

[stuartpb]

...if something is going to be standardized, it must be completely safe.

Uh, then why is the RegExp constructor itself standardized?

RegExp.escape is a function that reduces the attack surface of an existing component - if you're savvy enough to know that you need to use it, you're savvy enough to know it has its limits. The alternative to this is going to be more users rolling their own broken versions, thinking that [(*.)] are the only characters they need to escape (because it's the only characters they've ever seen used).

Saying that "avoiding a false sense of security" is going to help anything here is just plain wrong.

There has to be somewhere the general public can complain directly to the people who voted against this. Right?

More discussion at http://discourse.wicg.io/t/regexp-escape-str/832

[stuartpb]

The trouble I see with this is that it all comes down to arguments based on the behaviors of a hypothetical "average" developer nobody involved in the discussion would have any direct experience with, almost by definition.

The point that I would make is, what real code do they see ending up broken, that would not have been broken without its presence? The TC's argument, I feel, is that not having a RegExp.escape provided by the language would somehow drive developers to look up the documentation once they encounter this problem, and become enlightened - but, from my experience, for the type of developer who would think on sight that RegExp.escape is some kind of panacea, that's not what would happen. I find "average developers" (the type who would not realize RegExp.escape would have corner cases) tend to operate more in one of two modes:

• Use something they've seen somebody else use before
• Try to figure out how to do it using other things they've seen used before

And, without being given a RegExp.escape, they would not become enlightened as to how to do it correctly. They would (and do) try to fix it using the exact problem in front of them, using only what they've seen before, with as little thought into other cases where it would be a problem as possible - and that's going to generate a hell of a lot more corner cases than RegExp.escape ever would.

[benjamingr]

Thanks for the update. This is disappointing, I'll read the meeting notes and figure out how to go forward.

I definitely don't think that a completely safe tag is that way forward though.

[erights]

Historical note: When Domenic first proposed this, we all thought that the maximally escaped form could be used reliably. While we thought that, I was in favor of it. The even-odd example took us all by surprise. I changed my mind only once I understood the unfixability of the even-odd problem.

[erights]

I definitely don't think that a completely safe tag is that way forward though.

Hi Benjamin, why not?

• First, if we had such a tag, it would subsume the functionality you desire:

RegExp.escape = (str) => RegExp.tag`${str}`;

• Second, it would help developers avoid the hazards.

I emphasize the word "help" above because the claim that we insist on something completely safe is not the way I would put it, because it leads to the misunderstandings seen earlier in this thread. Nothing will prevent developers from writing code unsafe in these regards. We're trying to make the safe way the easy way, so that a set of fairly straightforward design rules generally suffices to avoid these hazards.

The issue is simple. RegExp source text is a little mini programming language with its own escaping rules. We've already heard the following song before: Having functions that provide local escaping fit for some contexts combined with use of string append to compose together source text from fragments. This has led to the nightmare that pervades the net, of escaping errors and injection attacks in html, css, sql, and more. We took a memory safe language, JavaScript, and by use of these manual-escape-and-append patterns, we reiterated the buffer-overflow-like vulnerability at a higher level of abstraction.

Lisp, when embedded languages are composed of S-Expressions, is largely free of this entire class of vulnerability because they compose structure to create structure. All the relevant source text is first passed through a reliable reader and low-level universal parser before this composition happens. Lisp quasi-quote makes this especially pleasant. Of course, we do not have a universal textual syntax like S-Expressions that we're starting from. We created the template string mechanism (and E's earlier quali-literal mechanism) as a universal framework for solving the context-dependent escaping problem for embedded languages, each with their own surface syntax.

[erights]

Btw, in case it wasn't clear, I do not propose that the tag be named RegExp.tag. This is merely a standin for whatever the tag would actually be named.

[stuartpb]

Historical note: When Domenic first proposed this, we all thought that the maximally escaped form could be used reliably. While we thought that, I was in favor of it. The even-odd example took us all by surprise. I changed my mind only once I understood the unfixability of the even-odd problem.

I feel like you've let this flaw blindside you, then. The nature of this kind of channeling attack has been a known issue with escapement for years - while I'm not going to argue against the value of a better model that checks individual RegExp components for validity, I will argue that it's short-sighted to say such a solution is the only adequate one. The more an end-developer (particularly one who's busy and only wants lo learn what's necessary to do their job) has to learn and adapt to make their solution "correct", the more likely it is that they're going to disregard that "correctness" in favor of doing it the maybe-sometimes-wrong way they already understand.

RegExp.escape is something we already get. It's something we already want. It's something we already use, and already works with all the code we already wrote. It's something we already know how to use correctly to prevent channeling attacks - without it in the language, I'm more likely to implement a broken solution, because there's a lot more effort involved in going out and looking up how to polyfill this than in taking half-a-second, deciding "yeah, I'm pretty sure it doesn't matter", and not escaping the string at all. If I had a RegExp.escape at hand that I could take, check I'm not concatenating with anything that might include uncontrolled backslashes, and drop in with 13 keystrokes, this would be a non-issue.

[benjamingr]

@erights

Well, the thing is that RegExp is not the same sort of language as SQL or HTML in this regard, there have been zero "RegExp injection" attacks (in this sense), there is no security guarantee to uphold here. I've been very in favour of contextual escaping for languages where not doing so presents a security risk.

The problem is that making a tag severely complicates this proposal. I mind but am willing to do the extra work on my side, but this means it will much less likely get implemented in browsers, will take a lot more of the TC's time and so on.

So, while a safe RegExp.tag method is interesting and I'm willing to continue exploring it - I feel like it would require a lot of effort to solve a problem a tiny bit better and it would create a less simple API. There is no security risk here like SQLi or XSS - just the lack of support of truly esoteric use cases.

We have scanned the top million websites, have gone through tens of thousands of NPM packages and explored a total of over a billion lines of code. I did not see a single reasonable case that was not caught by the original "SyntaxCharacter" proposal.

This trade between pragmatism and theoretical soundness is something I think ES needs to do here, just like C#, Python, Java, Perl, Ruby and pretty much every other language did before it. Just because a theoretical edge case exists doesn't mean we have to entertain it or focus our efforts based on it.

[erights]

I see here two interesting arguments for advancing RegExp.escape

• Composing a dynamic number of strings, as in Domenic's example.
• The extra work and delay for getting RegExp.tag specced and implemented.

I understand the force of the first argument and agree that it needs to be addressed. But I do not accept the second argument. A short term fix that become a long term hazard is a common development strategy and often appears in libraries that become de facto standards. When no good long term solution is known at the time, they often become de jure standards as well.

However, when a better long term solution is known at the time, and subsumes all the functionality and convenience of the short term fix, I think the committee would be failing the community to admit the short term fix merely because the good long term solution is more work. Admitting it into the de jure standard means we're likely to be stuck with it forever. We're forever stuck with enough of these already.

On the convenience point, compare

RegExp.escape(str)

and

RegExp.tag`${str}`

Once both exist, is there any reason whatsoever to prefer writing the first to the second?

So, while a safe RegExp.tag method is interesting and I'm willing to continue exploring it

Thank you. I'll be happy to help.

[WebReflection]

I am really having hard time trying to understand what's the tag supposed to solve compared to escape ... this is valid ES6, right ?

new RegExp("\\\\" + RegExp.tag`${str}`)

How does the user that would apparently fail with the initial example, avoid failing with the tag too?

Thanks for any sort of clarification, I know TC39 has always reasons but this time I really can't figure out the what, the where, the when, and the how.

Best Regards

[benjamingr]

@erights

So, while a safe RegExp.tag method is interesting and I'm willing to continue exploring it
Thank you. I'll be happy to help.

I'd love that, I think it would be interesting to apply the same formal analysis techniques you use in your work to show that the escaping is sound. I'm not sure it's something I'd put in the world's most popular programming language or that it would make a good API for people - but if you could point me in the right direction and tell me where to start this sounds like an interesting thing to do regardless of whether or not we build a language API out of it.

---

The arguments for RegExp.escape as I see it are:

• It's a de-facto standard, as @jdalton pointed out there are libraries with tens of millions of downloads that do this (with the less-strict escape set).
• It's a real problem programmers have been having for years, and we can't up with non-contrived cases it does not solve (and boy we tried). Moreover exploring a huge number of code bases does not show anyone doing anything remotely close to "\\" + escape("w") - so it's a problem that exists in theory.
• It's a simpler API for users, a fixed escape set is easy to reason about and we're only claiming that escape(s) matches s which matches the vast majority of use cases.
• It's shimmable very easily, so everyone using non ES2015 code bases can use it. I get this might not a hold of weight for the TC but it means a lot to a lot of developers.
• Composing a dynamic number of strings. This is probably the most common use case for RegExp.escape actually - building a routing table or a search pattern for multiple strings.

An idea was brought up here before (I think by @cscott) to support a second argument in RegExp.escape allowing more escaped characters (this is the standard API in some other languages, namely PHP). This means that the RegExp.escaped (tag) proposal can be easily built on top of it. Would this meet your concerns about contextual escaping?

[benjamingr]

@WebReflection a tag solution would presumably look like:

new RegExp(RegExp.escaped`\\\\${"w"}`);

A cursor would walk through the RegExp, keep track of its state and choose not to escape the string in this case because it is aware there is no need for an escape. This fails for dynamic length regular expressions which are actually one of the primary use cases here (Presuming we don't introduce a whole context-free escaping language here with iteration :D).

[bergus]

@WebReflection the proposed solution would look like

 RegExp.tag `\\${str}`

instead of new RegExp("\\\\" + RegExp.tag${str})

[matthewrobb]

We created the template string mechanism (and E's earlier quali-literal mechanism) as a universal framework for solving the context-dependent escaping problem for embedded languages, each with their own surface syntax.

So why isn't regex literal syntax being deprecated? I understand and agree with the idea but this is cause diverging paths for how to use a feature with baked in syntax support. Rather than a tag (or in addition to) add an escaped-interpolation syntax to regex literals.

[erights]

So why isn't regex literal syntax being deprecated?

As far as I can tell, in the entire history of JavaScript after 1999 we have only succeeded at deprecating one bit of API: function.{caller,arguments} and arguments.{caller,callee}. And they are still dying a very slow death. I hope to outlive them, but we'll see.

Our inability to effectively deprecate is exactly why we should be careful about admitting short term fixes into the standard.

[erights]

I say "after 1999" because that was the year ES3 was ratified. Whether the statement is still true from earlier, I have no idea.

[stuartpb]

If the TC is looking to build something safe and Lisp-like into the language to work around the problems with RegExp I suggest adding a module for constructing a parsing expression grammar.

[WebReflection]

so we are all presuming that if it's a tag no user will ever do the mistake we are supposing the same user would do with escape ?

If the concern is a user that does this:

new RegExp("\\" + RegExp.escape(str))

why there is no concern on a user that could just do this instead?

new RegExp("\\" + RegExp.tag`${str}`)

It is a silly thing to do in the first case, I don't see why the second couldn't be as silly as the first one.

Whoever would write the first case intentionally will eventually write the second one based on tags too.

Whoever would move the text inside the tag could eventually do the same with RegExp.escape

So instead of
new RegExp("\\" + RegExp.escape(str))
a user could do
new RegExp(RegExp.escape('\\' + str))

I think the concern is a bit overrated in this thread, but I also believe TC39 won't change its mind so ... I will keep reading curious, and I will remain disappointed a part.

[erights]

If the TC is looking to build something safe and Lisp-like into the language to work around the problems with [...] I suggest adding a module for constructing a parsing expression grammar.

Funny you should mention that. I replaced your "RegExp" with "[...]" above because the issue is more general. We need to make it easier to define tags for parsing new template languages. See https://github.com/erights/quasiParserGenerator . But please keep in mind that this is currently just an experiment not yet ready for serious use.

[erights]

so we are all presuming that if it's a tag no user will ever do the mistake we are supposing the same user would do with escape ?

No. Earlier I wrote:

Nothing will prevent developers from writing code unsafe in these regards. We're trying to make the safe way the easy way, so that a set of fairly straightforward design rules generally suffices to avoid these hazards.

[erights]

I think it would be interesting to apply the same formal analysis techniques you use in your work to show that the escaping is sound.

Actually, all the formal work on these issues is @mikesamuel 's. Mike, care to comment?

[benjamingr]

@erights addressing the other issues raised in that post would be nice:

• It's a de-facto standard, as @jdalton pointed out there are libraries with tens of millions of downloads that do this (with the less-strict escape set).
• It's a real problem programmers have been having for years, and we can't up with non-contrived cases it does not solve (and boy we tried). Moreover exploring a huge number of code bases does not show anyone doing anything remotely close to "\\" + escape("w") - so it's a problem that exists in theory.
• It's a simpler API for users, a fixed escape set is easy to reason about and we're only claiming that escape(s) matches s which matches the vast majority of use cases.
• It's shimmable very easily, so everyone using non ES2015 code bases can use it. I get this might not a hold of weight for the TC but it means a lot to a lot of developers.
  • Composing a dynamic number of strings. This is probably the most common use case for RegExp.escape actually - building a routing table or a search pattern for multiple strings.

And the two parameter solution:

An idea was brought up here before (I think by @cscott) to support a second argument in RegExp.escape allowing more escaped characters (this is the standard API in some other languages, namely PHP). This means that the RegExp.escaped (tag) proposal can be easily built on top of it. Would this meet your concerns about contextual escaping?

[erights]

I disagree with

It's a simpler API for users

It is simpler to implement, but is no simpler for users.

From all the other points, it seems the de-facto process is successfully doing what you want. Given that, why do you need TC39 de-jure blessing?

I don't understand the two argument solution. Perhaps if you explained how the template tag could be easily built on top I would understand it better. As to whether this would change my mind, I'll wait until I understand before reevaluating.

[mikesamuel]

Sorry I'm late to this thread.

If people don't mind, I'd like to try to summarize the discussion so far to make sure we're on the same page.

1. Some language support for dynamically created RegExp would help avoid errors.
2. Some of the dynamic parts of RegExps come from untrusted inputs.
3. There is no single RegExp.escape function we could define that would make composing regexs significantly less error-prone.
4. A string template for RegExpr concatenation could take that into account.

It seems there are a number of unique contexts in regexps.

1. Character sets: /[...]/
2. After a \ which we could assume starts a back-reference: /(#+)bar\.../
3. Possible, in a flag set. /foo/g...
4. Inside a count: a{1,...}
5. Where a non-capturing group like (?:xyz) can appear

so we might escape x differently in RegExpfoo${x}bar differently from `RegExp`[a-zA-Z${x}].

And a number of kinds of inputs.
A. Integral or decimal string
B. String that is not an unsigned decimal number
C. A RegExp instance

For example, RegExpfoo${ /[abc]/ }bar might be different from `RegExp`foo${ "[abc]" }bar and since \b means something different in string literals vs regex literals, we might treat RegExpfoo$( "\b" } differently from `RegExp`foo${ /\b/ }.

I've probably missed some, but trying to fill in a matrix might help see what feels dodgy as far as the principle of least surprise.

MATRIX	String	Integral	RegExp
[...]	individual characters	as string form	if it's a range inline it
\...	consume \ and escape	back-reference	consume \ and inline
flags	append to flags	error	error
count	error	inline	error
inline	escape within (?:...)	as string	inline within (?:...)

[benjamingr]

@erights

From all the other points, it seems the de-facto process is successfully doing what you want. Given that, why do you need TC39 de-jure blessing?

Because that's a big part of what a specification body does - spec de-facto solutions de-jura. Why specify String.prototype.includes or Array.from or promises? Most of the specified API has good user-land solutions - that doesn't mean we shouldn't standardise them. Even if they're not "interesting" they are still immensely useful to developers.

I don't understand the two argument solution.

It's pretty simple, RegExp.escape(string, additionalEscapes) where additionalEscapes is either a string or a RegExp of additionally escaped code points. That has the benefit of standardising the de-facto standard and we can later build the RegExp.escaped tag on top of it.

[l-cornelius-dol]

I think this issue is fine to close now :]

Could not disagree more.

[benjamingr]

@lawrence-dol please enlighten me why oh why the "result of the TC39 presentation from 5 years ago" should remain open in my repo and telling people to instead check the issue pointing to a new TC39 champion wanting to pick this up and an alternative path of how we're going to get this (or something better anyway) is somehow bad?

[l-cornelius-dol]

@benjamingr : My apologies -- I thought I was on a TC39 thread here. I've been multitasking too many things this morning. For whatever it's worth, I strongly support your proposal, and it's unfathomable to me how it was rejected.

[coolaj86]

I fundamentally don't understand the issue that's being raised by "the even-odd problem":

new RegExp("\\" + RegExp.escape("w"))

// same as
new RegExp("\\w")

// same as
/\w/

/\w/.test("a") // true, as expected

Okay, that makes sense. If I only partially escape my input... why should I expect the input to be safe?

I should instead do:

new RegExp(RegExp.escape("\\") + RegExp.escape("w"))

// same as
new RegExp("\\\\w")

// same as
/\\w/

/\\w/.test("a") // false, as expected
/^\\w$/.test("\\w") // true, as expected

To me this seems perfectly intuitive - I can't ever use an odd number of \ because it's an escape character that must be escaped.

Maybe "intuitive" is a strong word... "rational", let's say.

What is the "problem" part of "the even-odd problem"?

Is it that some developers don't know that you have to escape \ in a string and therefore don't know that you also have to escape the escaped \ in a RegExp?

Is that you must always have an even number of \?
(that's just have escaping works, right... 🤷‍♂️)

If you have an odd number of \ then one of:

• you made a typo (off-by-one \)
• you intended for the next character in the input to be a special character (not likely)
• you're mixing escaped and unescaped inputs

What about this behavior is unexpected or unsafe?

I don't get it.

It seems to behave in a perfectly predictable way that anyone familiar with basic strings should be able to reason about.

Can someone explain what the problem is?

[msikma]

I don't get it.
It seems to behave in a perfectly predictable way that anyone familiar with basic strings should be able to reason about.
Can someone explain what the problem is?

There is no problem. I'm afraid there is no nice way of saying this, and I hate to be this confrontational, but this is entirely a consequence of @erights fundamentally misunderstanding what escape functions are, and sticking to his guns even after multiple people have tried to explain it to him.

A regex escape function is trivial to create, tons of languages have them, and there's a common npm package and even Stack Overflow code snippet that will do the job. That npm package has 67 million weekly downloads. All of these work perfectly fine and are used daily without a problem—the Python docs don't even mention the supposed deal-breaking issue brought up here because it's perfectly obvious. As you pointed out, escape functions necessarily have a scope in which they are expected to work, and if you use them improperly they'll fail in predictable and documentable ways. This is a categorical truth about any kind of escape function.

By the same logic we can say that there should be no encodeURI() since you can do decodeURI('%' + encodeURI('%')) and that's invalid. No one should be "surprised" by this. If you surround escaped output by characters that are escapable or can escape others, like slashes in this case, you are misusing the function.

Then after all of this was kindly pointed out by people who actually understand how this works, the excuse became that we need to look after the "average programmer" and restrict ourselves to just making a tagged template because it makes it harder to screw up. Even though it's obviously still trivial to misuse and it's clearly not the favored interface for a function like this, as evidenced by the myriad escape function libraries that are not tagged templates (can anyone even find one in the wild that is?)

Everybody here is being extremely charitable and walking on eggshells out in fear of upsetting anyone from the committee, who are clearly ready to torpedo a good proposal and set us all back multiple years (this issue was opened 7 years ago) if they happen to be feeling like it on that day, but there actually just is no technical explanation for why this still isn't even on track to become part of the standard.

The real explanation is that the committee has colossally failed in its duty due to a lack of technical understanding (and probably so due to one single individual), and this whole proposal is now a good example of how you should never assume that TC39 committee members know what they're talking about or are amenable to technical arguments.

[benjamingr]

@msikma this is currently blocked on work not on Mark if we're honest. That said he is still very much against RegExp.escape and I'm opposed to a template-tag version that is a lot less ergonomic and frankly what users are asking for.

Honestly the (time and emotional) cost of standardization is such that I might end up just implementing this in Node and ask browsers to add it as a WHATWG API and we'll circumvent TC39?

[ljharb]

@benjamingr please don’t do that; that is a much worse outcome than never having it at all.

[ljharb]

@msikma it is inaccurate that it’s one individual; there are multiple people on the committee who feel the same as Mark does.

Please avoid personal attacks; that’s not in keeping with our CoC.

[benjamingr]

@benjamingr please don’t do that; that is a much worse outcome than never having it at all.

I feel like if I go through here I will spend another 5 years waiting and the API people are actually asking for (.escape) won't happen. I fail, I fail the users who have been rooting for me, I fail the ecosystem.

On the other hand if I make a PR to Node and coordinate with WHATWG I get the API I've been asking for since 2015 in about a week. The interaction with WHATWG has been significantly easier than TC39, browsers would (most likely) also want this and Node sure does.

So please help me understand why I would rather go through TC39 for something that is borderline platform and not language anyway (in other languages and ecosystems it's half-and-half)?

I like you and (full honesty) I like Mark. Both of you have helped me in the past with a bunch of things when I needed help both in private and in public. What I don't like is the process and how skewed it is towards concerns that our users in Node frankly don't care about as you can see from this (and the other repo issues).

Please give me a reason to trust the committee on this or faith this will progress at some point.

[ljharb]

The presentation I made that reactivated the proposal at stage 1 is the entire committee giving consensus to reconsider it, which woudn't have happened if anyone's position remained immovable. I'm sorry I haven't made the time to do more work on it since then, but I can assure you that, at least, I am confident it has a real shot at advancement.

[benjamingr]

@ljharb ok, let me know how it goes