Replace T4 code generation with a C# source generator

[viceroypenguin]

@atifaziz I've noticed your recent updates on the t4 generation. I've started playing with the new Source Generators tools. I think these would be a good replacement for t4 (don't have to check the generated code into source control; automatically generated; can return to parallel builds). I don't mind doing the work to transition, but I wanted to know if you're interested before I started doing it.

[atifaziz]

Thanks for checking in with the idea! I had the same thought of potentially replacing T4 with source generation and I even have a working prototype. It was a quick weekend experiment, but not something I wanted to publish until I was ready to commit to its long-term maintenance. Anyway, I'd love to collaborate with others rather than do this is in complete isolation.

It's also a good idea to always look around if someone else is working on a similar effort, so I wanted to share that in my research, I came across scriban/scriban#272 by @MarkPflug. I was also pleased to learn that such a source generator wouldn't be an edge idea.

[viceroypenguin]

I just finished doing it here for another library, so I have a fairly decent idea now of how they work and the proper way to do things. I'd be happy to start from scratch or offer advice on what you have or whatever... :)

[MarkPflug]

Hello.
That PR against scriban was closed because it wasn't going to be accepted into that project. I still need to figure out what to do with it. It was basically ready for release I think, might need a little cleanup.

It was intended to be a direct replacement for T4, so rather than a .tt file you'd add a .stt witch uses the scriban templating syntax. It isn't necessarily as powerful as T4, but should cover the 99% that people typically need.

The advantages over T4 being that it doesn't put any extra wiring the .csproj, doesn't add .g.cs to your source tree. If all you intend to do is replace the templates in this project, it would probably be a better approach that writing a bespoke generator.

[MarkPflug]

Indeed, I was considering doing something similar for T4 templates using the https://github.com/mono/t4 implementation. I quickly ran into issues with that library though that I didn't want to deal with so moved to the scriban templates instead. It would be nice if there was a templating source generator package that allowed various template formats, so people could pick whichever templating solution provided the best fit for their needs. If it was a drop-in replacement for T4 I think it would be more accepted by the community than if they had to migrate to a new template language.

[atifaziz]

so I have a fairly decent idea now of how they work and the proper way to do things. I'd be happy to start from scratch or offer advice on what you have or whatever... :)

@viceroypenguin Always good to be in the company of someone with experience!

@MarkPflug Thanks for dropping in with all your input!

I thought I'd also share my thoughts and approach. T4 has a long history in the .NET space. It's not something I want to tackle and compete with. My fear is that the community/users will approach such a (pretending-to-replace) project with high expectations and there are a number of things that would be quite tricky to solve as the platform has changed. It would create a lot of maintenance and design headaches for the maintainers. I'd much rather start with something simple and humble in scope. It should do a thing or two and do them well. If designed correctly, a simple project can be extended by others where additional complexity lives in higher layers.

The T4 usage model is similar and yet different to source generators at the same time. A T4 template is generally used to generate code, but to do so, the template gets compiled into code and then executed to generate the output (source code). While this usually happens transparently, it would add a lot of bloat and slow down the source generator. I am afraid that it will lead to a poor experience in the IDE because you'll be compiling and executing code during source generation. You could certainly optimise it with caching and re-compiling on changes, but that should be secondary and not inherent to the execution model.

I haven't checked out Scriban's inner workings in detail, but I am guessing that it interprets the template code and so would be more suitable. However, asking people to get used to a whole new language isn't an investment everyone's going to want to make. Taking T4's approach of splicing an existing language like C# within a set of delimiters feels more fitting for what is ultimately a source generator. Eventually, I think a hybrid approach would be best, which is what I tried. My experimental source generator so far stops at producing the generator code. That is, the template is transpiled to C# code that's subsequently added to the current compilation, but it will only generate the actual output of the template when invoked through the project being compiled. It's good for projects that want fast, strong-typed and static generation at runtime without any dependency on the original template and its engine. It's not good for generating code for the host project. The requires the extra step of compiling and executing the C# from the tanspilation step I mentioned earlier.

Syntax-wise, the hybrid approach I took is:

• Use {{ and }} from Liquid rather than <# and #> from T4.
• Everything in between the delimiters is assumed to be C# code. Although technically source generators are a Roslyn feature and could eventually apply to VB and F#, I stuck with C# for simplicity for now.
• Borrow - from Liquid and ~ from Scriban to deal with whitespace and new-line trimming that's always been a nuisance to get right with T4 templates
• Remove the distinction between expressions, statements and control blocks. There's only {{ and }} and the block type is discerned through some simple rules (which is where the choice of C# comes in since you'd need to have some intelligence about the language being used).

[viceroypenguin]

@atifaziz So far as models for source generation, there are really only two that I've seen others use so far: Interpolated Strings (i.e. not very different than Program.cs in your existing Generations project; and syntax building.

Personally, I used the syntax generation in the one I wrote for l2db, mainly because I believe that it is technically faster, though I haven't actually benchmarked it.

I think Interpolated Strings is essentially what you're describing already, so I don't know that adding a templating engine on top of it makes a lot of sense, especially for the cases used in MoreLINQ.

I've done most of the work to adapt your Program.cs into a Source Generator, and I'll try to finish that up later this week and push it. Then I'll look at adapting each t4 into a separate generator. The main thing I'll have to figure out is how to generate the Extensions output for each of the other generators; I'll probably abstract the actual output into a public method that the other generators can use in theirs.

Let me know if there's anything that stands out that you don't like?

[atifaziz]

@viceroypenguin:

I think Interpolated Strings is essentially what you're describing already

Isn't all templating some form of interpolation?

In the end, whether you decide to do it with existing facilities of C#, such as string interpolation, or a separate templating language/engine comes down to the balance that works for you. If the code being generated is thin then interpolation might be fine, but if it's the bulk with some small driving logic then templating might be a better fit. My goal was to solve the problem more generally rather than build something specific for MoreLINQ or as a replacement of T4. If it could help here then it would be a bonus but I have several projects where I have a large body of text with some placeholders that need replacing at runtime and those simple cases would have been the first target. I did take the idea a little further to see how easy it could serve as replacement for T4.

Of course, the one thing everyone's vastly enjoyed with T4 templates (and which can't be underestimated) is the IDE integration and the quick inner loop you get with just saving the template and seeing the generated results side-by-side and instantaneously. It would be a shame to lose that!

I've done most of the work to adapt your Program.cs into a Source Generator, and I'll try to finish that up later this week and push it.

Cool and I am guessing you're talking about the ExtensionsGenerator project? If so, the only thing that would make me hesitate using a source generator version as a complete replacement is if it extends the build times considerably. The public API changes rarely so generating all the extensions on each build might be a bit of an overkill. I'd also like to call out a comment from @MarkPflug on dotnet/roslyn#49921 that's worth keeping in mind about IDE interactions:

However, even with the caching enabled the ServiceHub.RoslynCodeAnalysisService.exe process is spinning up every time I type in the IDE,

What's more, right now the Extensions*.g.cs files weigh in at 370K+ so you'd be committing a 740K+ (UCS-16) string to memory each time, which will end up in the Large Object Heap and won't get collected so often. These might be some unfounded concerns so only benchmarking will tell, but I'd hate to see you put a lot of effort towards it only to be disappointed that it didn't get pulled in so just wanted to be upfront about some of the hurdles that might be in the path for a quality solution. If you're happy to try and just learn from the experience then I'd love hear about your findings. If my concerns are indeed unfounded then I'd be happy to take in your work!

Today, the wrapper extensions don't get generated unless you do a build from the CLI via the build scripts (build.cmd or build.sh). There is an assumption that someone adding a new API will run the scripts to refresh the wrapper extensions before checking in their code. If the person forgets, we detect it during the CI build and fail the build if things go out of sync. It's not great, but it's good enough. Swapping out with a solution based on source generator will only optimise this very rare path.

All my remarks so far also extend to replacing T4 templates with source generators, except I'd say that maintaining the inner loop there will be vital. There is no real burning problem with dotnet t4 right now that needs solving so it's your call how much time & energy you wish to invest for what might risk being a very marginal benefit.

[atifaziz]

@viceroypenguin, @MarkPflug FWIW, I have somewhat cleaned up and shared my experiment (Transplator) with very sparse documentation at the moment. Needless to say, it's work in progress, 🚧 and I can't say if and when I'd ever be ready to commit to a 1.0; I am in no real hurry whatsoever. 😊 That said, I'd love to get your input and collaborate if it piques your interest. As I said, I am interested in keeping this very simple and small instead of trying to compete or achieve feature parity with T4.