Wasp TS SDK (TypeScript-based Wasp AST generation)

[faassen]

Description

Wasp has its own DSL. This DSL lets you declare how a Wasp app goes together.

The DSL has the benefit that it makes very clear what is being configured, and lets you configure an entire full stack web app as a whole.

The DSL has some drawbacks:

• No default editor support, it needs to be built

• Need to maintain parser, typechecker

• It integrates JS/TypeScript code, but has no understanding of TypeScript types. To bring those we need to invent something like a TS DSL anyway and somehow integrate them with a language server that does TS compiles and interprets its feedback (see also https://www.notion.so/wasp-lang/TypeScript-cf20d4648b1e4dda99a422a2eea59a05)

• No access to programming features like conditionals or looping

• For better feedback, we need to create and maintain our own language server for the DSL

As an alternative we could make available a TypeScript based frontend. This would generate the same appspec, but from TS. Advantages:

• Quite easy to implement, enabling rapid prototyping of DSL features and type-driven thinking about how things go together

• Programming features are available

• Instead of giving everything an explicit name, you can assign to constants and get it typechecked too

• People write JS/TS code anyway to build a Wasp app, so language is already familiar

• Knows about TypeScript code. Lots of advanced typechecking features are in relatively easy reach (including typed routes)

Drawbacks:

• Integration of other languages may appear less natural

• We give up a sense of the Wasp DSL as it own language

Technical challenges

While the TS implementation isn't particularly complex mostly, to enable inline functions and natural imports we actually need access to the source code of what's included later on in the generation process. To do this may require parsing the TypeScript and extracting that information into its own module (which can then be used by the generator).

As an intermediary solution to reach parity with existing Wasp we can make the user do this explicitly (and not allow any inline code) through a "jsRef" construct:

jsRef("./components", "MyComponent")

To allow type checking an optional third argument may be supplied that is the JS entity to refer to:

import {MyComponent} from "./components";

jsRef("./components", "MyComponent", MyComponent);

Later on we may introduce a facility to analyze imports and allow jsRefs to be generated automatically.

[Martinsos]

A quick go on pros/cons as I see it:

• Wasp DSL

  • Pros
    1. Clear, concise, cool. If needed, we can adapt it as much as we need to -> infinite freedom in ergonomics. It can be what we need it to be.
    2. Language agnostic, therefore plays into the idea that Wasp is above the implementation details of the web app and de-couples us from specific implementation language. On one hand, short term we will be bound to Typescript so this is not so valuable. On the other hand, it does possibly direct our thinking in a way that we would like to think, which is Wasp as an intuitive DSL.
    3. Infinite control over error messages.
  • Cons
    1. We are doing our own compiler, which is a complex thing.
    2. We are doing our own lsp, which is a complex thing.
    3. Integration with other tech like TS, Prisma and similar is not automatic and we need to think and work on it (regarding dev experience in their IDE).
    4. Stuff like package manager and modules is smth we will also have to implement on our own -> however, this is not urgent for now, in the short term.

• Wasp TS SDK

  • Pros
    1. Familiar to TS users. (What about JS users? Will TS scare them? Probably not, I think).
    2. Utilizes existing JS/TS ecosystem -> we get Turing complete language for building AST (although we don't need that), we get modules and package management out of the box (although not urgent in short term), we get TS IDE support (lsp, formatting probably -> this can be very useful!).
    3. We might be able to do tighter integration with TS, giving us stuff like more thorough type checking and possibly inline TS code. Although there are also mechanisms in Wasp to achieve it (imagine Wasp constructing TS code and then forwarding compile errors as Wasp compile errors), but they could also benefit from researching TS SDK.
  • Cons
    1. We don't have control over TS as a language, so stuff like error messages, some of the tooling, and similar, are out of our (full) control.
    2. Somewhat more verbose/complex than Wasp DSL, we loose that effect of "this is language JUST for this". How much people care about this -> hard to say at the moment.
    3. Might make it harder to later de-couple from TS/JS if we wish to do so, and will also affect our design in such way probably -> but on the other hand, we are short term ok with that, and long-term we don't know yet how much will we ever expand further than having JS/TS as base language.

Btw looping, conditionals, no-name declarations -> we can implement all of those in the Wasp lang if/when we will need them. We do get them out of the box in TS, but it is not super hard to do in Wasp.

So with all this, it is a very attractive option to explore this more. With current state of Wasp, it makes sense to keep the Wasp DSL we have and focus on web framework features, but as we progress, it would be wise to do more experimentation with TS SDK and see how far can we get with it, what can we achieve. If we get to the point where we see that TS SDK is a better option than Wasp DSL, and it makes no sense to maintain both, we can even drop Wasp DSL as a compiler frontend.

It would be interesting to first just get TS SDK to the level where it can do exactly the same thing as current Wasp DSL does. So, generate equivalent AppSpec.
At that point we can start using it for real, and we can also continue experimenting with it, making sure it follows up development of Wasp DSL, and also try to add additional features to it, improve it to be better than Wasp DSL. Also as we need to implement smth in Wasp DSL, will be able to consider how hard is it to do the same thing in TS SDK.
Finally, as I mentioned above, if TS SDK is obviously better than Wasp DSL, we can switch exclusively to it.

[faassen]

If we had a way to import the appspec from JSON we could open up the ground for more experiments. You could then create some custom build command that uses the ingested JSON.

[Martinsos]

If we had a way to import the appspec from JSON we could open up the ground for more experiments. You could then create some custom build command that uses the ingested JSON.

Yes this is an exciting idea! If we make AppSpec serializable to JSON, which shouldn't be super hard, then we can as you said both have other tools generate that JSON, but also other tools consume that JSON. It does limit us a bit in the sense that AppSpec needs to be JSON serializable, but restrictions are also good, at least it is worth giving them a try.

[Martinsos]

This initiative is getting started now with #2047 !