Support TypeScript/Flow type annotations #482
Comments
|
Could this be fixed with a custom readtable? |
No, because its basically impossible to tell from a read whether |
|
There are also other productions that would need to come in shortly thereafter, but this bug more or less blocks those.
All three have the same syntax between Flow and TypeScript (mod the nit with constructors). |
|
Yeah, I'm definitely open to this. I think we would want to make sure our handling of the type language is generic enough to deal with both TS/Flow etc. My main concern would be the added language complexity macro authors who want to handle functions/vars would need to deal with as @natefaubion pointed out. |
|
Any update on this? |
|
Work is continuing on the redesign (#485). Once that lands it will make sense to think about adding type annotation support. |
|
+1, imported from microsoft/TypeScript#4892 |
|
disnet on gitter: "as long as your macro forms “look” like normal ES6 code you can pass it through TS/babel first just fine". Trying out a |
|
@disnet, Sounds like you're talking about macros that mostly look like function applications, which would be the most boring ones. (Ie, macros where you're playing with control flow, which are not too relevant now that the syntactic weight of wrapping stuff in thunks is tiny.) |
|
We've released an alpha version of our TS visitors framework called tspoon . Tspoon should enable |
|
@amir-arad, IIUC, that project provides a hook into TS somewhere between parsing the input and producing the output. If that's correct, then it's kind of an option to implementing some macros on top of TS, so in theory it could be used to implement sweet.js for TS. But I'm sure that there's lots of things that sweet is using that would be anywhere from inconvenient to impossible to do with the TS representation, which is why I think that the right way to go would be to extend sweet.js to recognize TS syntax and spit it back out (for TS to process). |
Would you mind elaborating on this? Wasn't TS just a superset of ES6 anyway?
Well, if one were to count outstanding ES proposals (e.g. decorators, type annotations, though I forgot which of my Babel plugins added that), I believe TS actually does not really add much syntax over what has been proposed for ES already -- the extent of 'not much', AFAIK, being decorators on parameters. So in that sense, might those two roads not be closer to being one and the same? |
I know I wasn't the one you were asking, but I think Sparkler is a very good example of this. The TypeScript parser isn't particularly extensible, either (it's implemented in a giant closure).
You're correct in that TS doesn't add much syntax that isn't already either in or proposed for ES now. But type annotations are probably harder to parse than even the rest of the language. |
|
[@tycho01]
I'm talking about the technical level. The kind of things that a good macro expander needs from a representation of syntax can be very demanding on one hand, and subtle on the other. (And sweet.js is one of the very few rare cases of a good macro system.) I doubt that a generic exposure of syntax nodes would be as full as needed (but TBH I didn't look too deeply). [@isiahmeadows]
Right -- that when I figured that for the TS people to add macros would be wrong from all kinds of aspects. It requires certain skills that people who really care about syntax enough have (to know what hygiene means), but they probably don't. It's a whole layer of complexity that actually has very little with the goals of TS. And like Flow, it's coming from a neighborhood of people who don't care much about such things anyway. And then I realized that this can be just like the Typed Racket case, where the typed language doesn't do anything interesting at the syntax level (besides typechecking, of course), and instead it just expands the code completely before it starts. So especially since the TS and Flow worlds have settled on a very similar syntax, this should really go as an extension of sweet.js which could then be consumed by one of these. And in the typed racket case, there was some pressure from a few people to have some type information available to the expander, so it's possible to use that information to expand macros in different ways based on it. In that case there was certainly understanding of how useful this could be, but not enough human resources to do so (to intertwine typechecking with expanding macros). In the JS case, it would be interesting what happens but for now the "immediate" goal is to get people to notice sweet.js and stop the insanity of piles and piles of complete-source to complete-source "transpilers" and start doing the damn thing properly.
It's probably not too hard for an environment that knows about identifiers etc. The real problems will happen in case of slightly different parsings being done by these things (TS vs Flow), but even in that case, sweet.js could just stick to some agreed lcd. |
|
@elibarzilay knows what's up. The right thing to do is extend Sweet with support for TS/flow syntax. This actually isn't too hard, certainly not nearly the same difficulty as the rest of the macro system. Just need to add a few cases in the parser and the codegen to handle type annotations. The obvious concern with going down this path of supporting "downstream" languages in Sweet itself is that handling n languages in our parser (that really should just be macros) is untenable. I think TS/Flow are big and close enough to warrant this embrace and extend approach. Maybe eventually Sweet will get good enough for that other "e" :) |
|
I completely agree. I think that TS/Flow are exceptions since they are a language extension that is largely unrelated to macros. For other transformer libraries, it would be nice to start seeing them convert the code to go through Sweet as a much more logical choice as usual with macros being local transformation rules. When that becomes popular enough (and I might be idealistic here, but I'm convinced it will), then people will finally "discover"[*] that they can do some actual work as macros -- to the point of re-implementing TS/Flow on top of sweet. ([*] And this will definitely take some time. Even in the Scheme world, I remember talking to someone who was very surprised that in Racket we'd do something crazy like invoke GCC as part of the compilation -- in most people's minds, that's way beyond what a sane macro system should be able to do. They're just not used to separate compilation (Racket's phase separation) making things robust enough that anything goes.) |
|
Regarding TypeScript as "some babel modules + compilation errors" for a second, I'm getting the impression that, in a similar vein as you've mentioned, existing babel modules essentially represent ES-next syntax implementations that would be awesome to have available in Sweet as well. This leads me to a question, answers likely clear to you guys, but not so much to me as an ignorant but curious user.
If the additional information required would be trivial, perhaps it could become the norm for the community to provide implementations supporting both babel and sweet for future ES proposals. |
|
I was thinking: would it be a better idea to, instead of adding more and more things to the parser, allow an Acorn-style syntax extension mechanism? That might work better, since you can validate anything you come across. Maybe something like this? // Just validates.
syntaxdecl async = function (ctx) {
// ctx.expectCallable(true or undefined) -> include newlines
return ctx.expectCallable(true) && ctx.is("expr")
}
syntaxdecl inline `::` = function (ctx) {
return ctx.next("expr") &&
ctx.next(this) &&
ctx.next("expr") &&
ctx.is("expr")
} |
|
Right, so worth taking a moment to explain the philosophy of Sweet (and macro systems in general). Consider the following languages:
Sweet is a parser for yet another language: The goal of Sweet is to support the creation of all the languages listed above (and many more) via syntax transformations that you can compose and reason about (acorn’s plugin approach is a non-starter because it is neither composable nor reasonable). A non-goal of Sweet is to bake in support for every JS-adjacent language into the parser. That’s what macros are for. Practically, the kinds of syntax transformations Sweet currently supports are not powerful enough to implement all the languages listed above. That said, the plan is to make them powerful enough. Operators ( Basically, the roadmap is steal everything from Racket. What we’ve been discussing in this thread is extending Sweet to support the language |
|
@disnet Okay. I'm not that invested in that idea of extensible syntax, anyways.
😆 |
|
To expand on my earlier comment, if all ES-next proposals would require non-trivial reimplementations as macros, I suppose that means Sweet would essentially be competing with Babel for features added. If so, then adopting Sweet unfortunately for the time being becomes more of a trade-off rather than a straight-forward decision. Obviously, its potential is orders of magnitude bigger, so I definitely hope Sweet could overtake Babel as the go-to way of implementing new features. Until that time, there may be a threshold of momentum though. But then again, I guess for now the bigger step is still that Racket roadmap... |
|
@tycho01 Almost every ES proposal at the syntax level is possible to implement, and many already have with previous versions of Sweet. This includes ES6 classes in their entirety and arrow functions. Oh, and Babel isn't likely to disappear. Compilers are much better at semantic optimization and even implementation correctness than macro processors in terms of specifications. Good luck implementing generators in pure Sweet. 😉 |
|
Are there any progress with typescript support? |
|
@vegansk various background tasks have been completed but nothing directly on supporting TS/Flow in sweet. I'm definitely motivated since everything I write now is in flow (even sweet core!). My current focus is updating our internal AST to the latest version of Shift (so we can support async/await). Once that has been handled we will be in a good position to support types. If anyone wants to help out a good place to start is getting TS/Flow support added to Shift. We depend on shift codegen to render our AST so it will need to be extended to handle types. |
|
@disnet are you thinking just matching what Babylon does (maybe more granular |
|
@gabejohnson I haven't looked into it in any depth yet but yeah that would probably be my initial approach. |
|
@disnet I'd like to help with this. I looked at the Swift issue list but couldn't see any feature request related to type annotations. Should I create such a request or is there a better way of adding the support you need? |
|
@slotik no need to open a shift issue. I chatted with them (sorry, forgot to update this thread) and the shift project is uninterested in type annotations in their core AST and that's actually fine for our purposes; we are already extending the shift-ast-spec in a couple of places (syntax declarations and import As far as what we can do right now I'm sort of blocking anyone helping out with code. I'm in the middle of refactoring sweet-spec and sweet-spec-macro, which will have far reaching consequences in sweet-core. What would be super helpful though is taking a look at the shift-spec and proposing what nodes we should add/change to support annotations. |
|
@disnet I started to look at shift-spec and I made a couple of notes around the places that seem to need changes: If this looks like it could provide some value, I can try to suggest concrete modifications as well as a new set of nodes to express |
For reference, TS AST nodes. |
|
@slotik looks like a great start! Definitely take inspiration from the TS/flow AST nodes. I'll try and finish up my background work on sweet-spec soon. |
|
I think I made quite some progress already, though some loose ends remain: I'll try to finalize that as soon as possible. I've modeled most of the missing nodes the way they are modeled in TypeScript. I've omitted everything related to JSDoc and JSX - maybe that is OK? Anyway, it's a bit hard to tell whether what I've done is actually correct. If anyone dares to look, I'll be happy to adjust as needed as well as try to explain why I did what I did. |
|
@slotik awesome! Feel free to open a PR on sweet-spec with your proposed changes and I can try and do a detailed review. I literally just updated it to ES2017 (sweet-js/sweet-spec#3 and #740) so your changes should apply nicely. |
|
If one were to take the position TS has subsumed all use-cases of JS/Flow, might it become preferable to just add this in the TS compiler rather than maintaining a separate copy of much of the functionality their compiler has already implemented? I realize that isn't the position taken here, and fragmentation of efforts is sub-optimal as well. Edit: either way, note that parallel to Sweet's The overlap in functionality seems somewhat significant.
Edit: oh, TS's exposed |
heh, I definitely do not take that position but am 0% interested in getting into a flame war. TS is great! Flow is great! JS is great! Use whatever makes you happy.
The only "functionality" we're talking about adding to sweet here is the ability to transparently pass through type annotations. No type checking or other fanciness. This allows sweet to focus on doing it's one thing (expanding macros) and let other systems consume the output of sweet and do what they do best. In terms of feasibility of integrating sweet with other compilers like TS, this is...non-trivial. This has already been pointed out in this thread #482 (comment) so I won't bother expanding on it. |
|
I see. I don't know much about the intricacies of hygiene, but it appears that TS handles it in their But yeah, even if they allow passing custom transformers, it does appear that extensibility of e.g. the parser with new nodes would remain an issue there... I wonder if external handling of macros might come at a price of its own though. Some IDEs facilitate the user with type checks, which I imagine would break down if they weren't able to handle the macro part as well. The approach suggested here so far leaves me wonder how that might work in practice. |
We're talking about the "ability" to transparently pass through the annotations though, not the necessity. Right? These are still going to be reader tokens -> AST nodes. If that's the case, we're really talking about hardcoding something that could be implemented as a language by using a custom reader and some macros ( I'm not suggesting we don't support TS natively, but it's something to think about. This could be a good test case for exposed reader macros. |
Nope :) That's not the hygiene @elibarzilay was talking about; it's just I could be wrong of course and if someone wants to add a Racket-style macro expansion system to TS go for it! I'm not interested in tilting at that windmill though.
Regardless of where the macro expander is (internal or external to the type system), the macros must first be expanded. This seems to me fairly straightforward to handle with a plugin that first runs sweet to expand the macros and then run TS/Flow to type check. Sourcemaps can stitch everything together. |
|
Thanks for elaborating. |
Of course. The real issue is not reader macros, correct me if I'm wrong but I don't think we need readtables at all to handle type annotations, there's nothing lexically different about them. The real issue is the resulting AST and codegen which is why we need to hard code types for now. We want the output of sweet to be something the existing type systems can consume. Eventually we could implement type checking itself entirely in sweet as just a collection of macros that communicate type information during macro expansion just like typed racket. |
Well, that'd be quite the feat as well; their |
You're right. I don't think that TS/Flow do anything different with
Disregard those comments, I was having a moment. Adding the type nodes would actually be really useful for creating alternative annotation syntax (HM perhaps). |
|
@tycho01, the gensym functionality that you've referred to is important As for the TS functionality that you're talking about, I don't know much In my experience, people who are talking about "preprocessing" are I very much hope for a future where this is different, but it'll take a |
|
What's the current situation with TypeScript support? If I just want to add some simple macros to some TS files, is it currently possible to do |
|
@Wizek I think it's safe to say Sweet is mostly abandoned. No commits at all for 2 years, no recent PRs, no recent comments on issues from a developer. Thus, I think no TS support is forthcoming. |
|
Ah, thanks for the heads up @letharion! In that case I am quite glad I didn't sink much time into learning Sweet.js! I'm just now looking at https://github.com/kentcdodds/babel-plugin-macros; is that the closest alternative or are there others? |
|
No, that module is very very simple and doesn't provide anything like custom readtables, custom operators, etc. It just avoids you having to write boilerplate to make a babel compiler plugin. |
Edit: Missed a difference...
This has come up before (#393), but I feel it could get resurrected again, since the syntax appears to be stabilizing. TypeScript would be much easier to use with macros, and Flow has kept a very TypeScript-like syntax as well. Babel supports Flow type annotations officially, and keeps them in its internal AST for plugins to manipulate. People have attempted to use SweetJS as-is with them, with varying degrees of success (generally little). And Closure Compiler also appears to be going in the direction of allowing TypeScript annotations as well. Could this use case be supported?
The type syntax overlaps tremendously between the two, and they only really differ in a few areas, mostly in semantics:
{}vs Flow'smixedObjectAs for pure syntax, you could parse them identically except for TypeScript's
new () => Tvs Flow'sClass<T>.The text was updated successfully, but these errors were encountered: