Put all (most) templates into the generator together for better developers' experience

[lmgyuan]

According to #1212, we propose to incorporate all the templates into the generator repository. Right now, templates are separated in different repositories (HTML templates, nodejs template, e.g.)

Why separation of templates from generator is not a good idea: Separation of templates causes an issue for both maintainers and developers. Because templates are not part of the generator repository, template developers sometimes need to do workarounds to address generator issues they encounter during development since they don't have access to make changes to generator and test them locally. Workarounds could lead to unexpected bugs in the codes, therefore making our codes harder to maintain.

Why integration of templates into generator would help: Putting templates into the generator repository will allow developers to test changes locally in the generator repository, making the development easier and reducing the number of workarounds in the template codes.

Would template developers still have ownership of their codes: Yes! We would implements championships that Modelina repository uses to allow champions to have their areas of responsibility where they can merge and accept pull requests. Template developers would have their templates as their areas of responsibility so they can still merge and accept pull requests for their templates after being integrated into the generator repository.

[lmgyuan]

Anyone's comments are welcome! Let's discuss this and see what we should do about it!

[derberg]

pinging maintainers that maintain different templates - for visibility

@mr-nuno @connil @kaushik-rishi @AGurlhosur @dalelane @dan-r @KieranM1999 @lewis-relph @JEFFLUFC @Tenischev @CameronRushton @anandsunderraman @smoya @emilianozublena @magicmatatjahu @jonaslagoni

[dalelane]

The objective sounds reasonable to me - simplifying the dev environment/experience is a good goal.

Have you thought about how to handle releases? Do you think they should similarly be combined, or kept separate (separate releases for each generator, although in the same repo)?

[jonaslagoni]

I am slowly pulling away from the generator and offering an alternative, but it sounds like a good idea, but don't count my vote 👍

[lmgyuan]

@dalelane That's a great question. I think a separate release with necessary tests before any release might be a good option. It means that we should allow each template maintainer to have their own release for flexibility, but we should create tests to make sure that their new releases can run perfectly with our generator functionalities. What do you think? Happy to discuss. : )

[lmgyuan]

@Gmin2 and I had a conversation about this and we think that it's a good idea to keep the discussion open. If there is no objection in the next month, we can create a pull request for this after that. @derberg let me know if you have some other thoughts on this.

[derberg]

To be honest, I was thinking that like in case of recently merged "filters" (that are released as part of generator by default, and do not need to be released separately) we could integrate "templates" in the same way, that they are by default part of the package (no npm install in the background fetching from npm). Of course we would still support people to manage their own templates separately, even in private registries - but this would be something like an "extension" or "customization" functionality.

From CLI API perspective we would have asyncapi generate models (like we have now, powered by modelina) and asyncapi generate fromTemplate the old way for people having their own templates and the new command would be powered by generator and include only official generators from generator repo -> asyncapi generate nodejs client or asyncapi generate java server --framework=quarkus

Does it make sense? We probably need some PoC first 🤔

[Tenischev]

Hi all, sorry, but I couldn't agree with proposed changes.
As thread owner do not show disadvantages, let me share what I see.

• We have an API between parser/generator and template. API is developed to separate components making them more independent and easy to change. For me it's not clear why we should ignore that and make components heavy coupled.
• Huge mono-repo. More disk space to consume. More resource requirements to work with repo as modern IDE intend to index files.
• More complicated issue tracking. Having issues for different, unrelated parts in one space will make a mess by numbers.
• Sophisticated code owning, test-release system.

Also, I would like to discuss this:

template developers sometimes need to do workarounds to address generator issues they encounter during development since they don't have access to make changes to generator and test them locally.

Why not create an issue for generator repo and fix it with MR? As I may see the generator is open-source.

[derberg]

Hey @Tenischev

We have an API between parser/generator and template. API is developed to separate components making them more independent and easy to change. For me it's not clear why we should ignore that and make components heavy coupled.

why do you think we make components heavy coupled? parser is staying outside? and if someone wants to stay with template outside, it is also allowed

Huge mono-repo. More disk space to consume. More resource requirements to work with repo as modern IDE intend to index files.

the truth is that in reality mono-repo means less code and less disk space as there is more space for reusability. We already have monorepo in generator -> it was proposed in October 2023, no objections, and things are already in motion - so far so good, no issues on performance

More complicated issue tracking. Having issues for different, unrelated parts in one space will make a mess by numbers.

we are not the only ones in that field. There are multiple projects doing monorepo, or are so complex that they need 20 or more maintainers - all of that is easily handled by labels and automation through issue comment commands

Sophisticated code owning, test-release system.

we did it already, it is really not that complex, and the advantage is that people are working closer. Modelina already has approach with champions and we can do the same or do what was done in website with triagers and committers that enables us to invite people to collaborate closer on early stage, as triagers that will do initial reviews and assign issues to proper groups.

Why not create an issue for generator repo and fix it with MR? As I may see the generator is open-source.

definitely, that would be lovely, but as long as this is optional, it is not perfect solution. Also people don't do it because of bad faith, but also because of some lack of knowledge.

---

it is critical to simplify things - like a refresh on approach to speed up things

first we are deprecating cli and nunjucks -> #1255

the best would be if we forget about the name "template" and rename it to something like "plugin/extension/customization", so we still allow people to do things on their own, behind firewalls or stuff like that. By default generator would have templates inside, like "generators", organized from perspective of what user wants to generate, like for example "client java" - so user do not have to research all the templates, look for their repositories, often go to not only readme but also code, to understand what they will get.

For me main motivation is "community" - to get all code/docs generation fans together closer.

As a side effect, with react render engine I believe that by working together in one place on all these generators, we will automatically start working on a library of react components that template/extension developers will use as dependency and also speed-up their work on templates

For now, we are 1 year after v3 release, and only 3 templates support it, and we need to do something to improve the development flow/speed

[derberg]

sharing a context, why issue was closed. The discussion did not continue and we jumped into the PoC work under #1269 so this discussion could be closed - and actual PoC it was better to separate

[derberg]

fyi proof of concept is merged https://github.com/asyncapi/generator/tree/master/packages
still lots of work to do.

@dalelane atm, how I see it long term is that templates that are part of generator will most probably not be versioned at all, and included inside generator - but we will see

@Tenischev the PoC showed that with turborepo that handles monorepo setup, we will be able to easily make conditional testing, and CI should not take long to run. Also with first simple example of template I already proved that there is a lot of code reusability across templates - expecially with extracting info from AsyncAPI documents - so long term it will really be better if main templates are developed inside generator and we reuse on regular basis