Add E2E tests

[Jakob-em]

As @danielrwolff suggested we should add some proper e2e tests to improve the current CI setup.

[danielrwolff]

I'm thinking Cypress is the way to go here, since it supports all frontend frameworks and doesn't have the disadvantages that Selenium has.

Are you thinking of only having these tests in the CI pipeline? It might be beneficial to actually include them as part of the templates, as they can be separated completely from the src folder. E2E tests can technically behave as their own application, completely decoupled from the application being developed (with their own package.json, and other config files). The template for this can then behave like the backend template - identical for all frameworks.

If we just include a simple E2E testing application adjacent to the backend/frontend folders, we can execute it during the CI pipeline to verify things are working, and it also gives the users of Goxygen a starting point for writing their own E2E tests, if they want to.

[Jakob-em]

I've heard about cypress and since then ever wanted to try it 👍

I agree with you that including them in the generated project is a good starting point for new projects.

[Jakob-em]

Cypress has also good support for github actions: https://github.com/cypress-io/github-action

[danielrwolff]

I finally got a Dev machine up and running, so I'll take a look into this today. I've also also wanted to learn Cypress!

[Jakob-em]

@Shpota Whats your opinion towards adding e2e tests to the generated project?

[Shpota]

I think it is important to distinguish testing Goxygen and testing the functionality of an application produced by Goxygen.

What Jokob implemented is simple yet functional end to end tests of Goxygen. Their task is to ensure the artifacts produced by Goxygen are of a good quality.

If we have end to end tests in templates they would serve as tests of a particular application produced by Goxygen. Their primary task is not to check how Goxygen works but to check how the generated application works.

The former should not be used in testing Goxygen (although they technically are able to prove that Goxygen produced correct artifacts). The issue is not only in the separation of concerns. Imagine a situation when templates are broken in the way that running an E2E suite starts no tests. E2E tests in templates are themselves a subject of testing.

I hope I managed to explain my point clear as it is hard to see the difference when both the tested application and its artifacts are source code.

Having this in mind, I think we should test Goxygen without relying on the code inside templates.

At the same time, I believe it is perfectly valid to have E2E tests in place in the generated applications. They might be used by the end-users. But I wouldn't mix them with testing Goxygen.

As for Cypress, I have no opinion on its usage in end to end tests. I am not an expert in this area. What I think is that it might be useful to investigate some common ways of approaching E2E testing in any particular single page application framework. It might be that there are frameworks/approaches that are a de facto standard in those areas. Users might be surprised to see something different from what they are used to.

Let me know if this makes sense. What do you think?

[danielrwolff]

Makes sense to me.

Either way, Cypress can probably still be used to verify that Goxygen is working, without having to be a template. Whether or not it's worth replacing what Jakob already has done is another story. At this stage in the game, it probably isn't worth it if it doesn't really change the current outcome, and doesn't benefit the users in the end.