Proposal: TypeScript should be built on the solid foundation of JS

[joshgoebel]

Since TypeScript is purely a superset of Javascript (and all valid JS is also valid TS) and that we should merge our JS and TS grammars into a single grammar for ease of maintenance - this probably should have been done from the beginning, but water under the bridge. Right now many things require double the effort and there are strange differences between the grammars (like JS supports JSX but TS does not support TSX).

It also makes things harder for contributors... they need to do more work (perhaps double) be more aware of the bigger picture, etc...

Most recent example: #2562

We could literally make js and javascript as an alias of TypeScript or JS could be more of a "dummy" grammar that really just re-uses the TypeScript grammar under the hood. I'm not sure what the value there is though other than that lang.name would be "Javascript".

@egor-rogov Any objections?

[joshgoebel]

Or at the very least a 90% js-like foundation layer... and perhaps which language you pick injects a few more rules/keywords. But adding a new highlight feature should most often be something that could be done in the 90% shared layer, etc.

[joshgoebel]

From my research just now about the actual differences between JS and TS grammars (after consolidating as much as we can)

• TypeScript adds 'interface' mode
• TypeScript adds 'namespace' mode
• TypeScript adds various new keywords
• TypeScript adds class/mode keyword "implements"
• TypeScript adds decorator mode
• JS allows use asm which doesn't really make sense in TS world

[joshgoebel]

And some weird things like TS decreasing the relevance of 'function' an saying that TS is more commonly => which I'm not sure if that is wholly accurate... but I'm not sure that matters if we lost it. And of course if we simply merge the grammars completely it certainly doesn't matter.

[joshgoebel]

@egor-rogov @allejo So the question is can TS simply consume JS or do we need to maintain both on a common foundation or some sort? The simplest thing would be to just merge them into a single language and then js and javascript merely become aliases of TypeScript.

The "downside" would be that I suppose TS keywords MIGHT get highlighted in JS when used in other contexts:

var number = 32 ; // number is a type in TS

But there are different solutions for that... improving context, etc...

I'm kind of curious to try the "simplest solution" first [TS consumes JS] - unless you all strongly object - and then if that fails to work in the long-run to fall back to a more nuanced solution.

[allejo]

I haven't looked at the JS and TS grammars, but I'd be opposed to making JS an alias for TS. Like you mentioned, TS keywords could be highlighted in a JS context. I think the best way forward would be to make the TS grammar extend the JS grammar and add any extra TS keywords and syntaxes (e.g. generics) on top of that.

[joshgoebel]

I think it might be simpler to build JS on TS and remove keywords - even if that sounds counter intuitive. Our ability to make subtle changes to another language right now isn’t superb.

[allejo]

...that sounds horribly counter-intuitive but if that's the best way to do things right now, sure! One way or another, I'd opt for having one language be the base of the other; whether it's additions or removals.

[joshgoebel]

I might do this in two pieces then, first pull TS up to the JS spec - which needs to be done anyways to make sure they are both pretty much 'doing the same thing', then work on one requiring the other.

[egor-rogov]

I agree with @allejo that merging the two languages doesn't look right. I'd rather have two grammars: either one based on another, or both based on a common foundation.

[Unnvaldr]

End user expects from syntax highlighting to not have such petty problems as method/property name being highlighted as keyword or incomplete modifier list.
I made a quick solutions for those problems, but both highlight.js limitations and not being experienced with API stood on my way, so that's why it became "more complicated".

Definitely merging here isn't an option, features from TS doesn't work in JS, but making TS extend JS dictionary seems to be the only wise solution.

[joshgoebel]

What I meant by merging is that TS is.a strict super-set of JS... so that means TS should 99.9% of the time correctly highlight all valid JS code. The only exception being keywords that don’t exist in JS.

If someone had a ridiculously tight budget for download size using TS to highlight JS code would be a very workable decision with very few downsides for most code.

But I will build TS on top of JS and find a way to make that work.

[joshgoebel]

I made a quick solutions for those problems, but both highlight.js limitations and not being experienced with API stood on my way, so that's why it became "more complicated".

Yes, sadly the quick solution is often far from the correct or optimal solution. Sometimes because of inadequate tests (and many potential contributors don’t even run the tests) - so it’s not clear what was just broken. Or sometimes because of common pattern we use (or avoid) or connections across languages that aren’t explicit (like JS and TS intertwining).

Sadly, sometimes the answer is “there is no great way to do that yet at all” - and fixes that only half work or 80% work can often cause more trouble than they are worth... look at all the issues filed for edge cases just as an example.

I’m open to suggestions on how to improve these things at a big picture level, but it’s a complicated problem since parsers and grammars are by their very nature a complicated thing. And some things are made much harder by the technical deficiencies of either JS (no look-behind support) or our parsing engine itself.

For example the BEST way to not highlight keywords that are methods would be to use a look-behind in the $pattern key to simply say “not proceeded by .”. Unfortunately that’s not possible yet.

beginKeywords hard codes this behavior, but that has it’s own problems as it behaves entirely than the normal keyword behavior.

I think adding method dispatch highlighting might actually solve this particular problem (keywords as method names). See the related thread on improving the fidelity of our highlighting.