Living Documentation

[Code-Grump]

Having seen a few mentions of the SpecFlow Living Docs plugin, it seems like there's a demand for an equivalent feature in Reqnroll.

What I'd like to understand is:

• Does it matter when this output is produced?
• Should it be the output of a build, or a test-run?
• Would running a stand-alone tool to extract this information be preferrable to something built into Visual Studio?
• What do we think should be the output?

I can think of many ways to approach this, so knowing the constraints is probably the easiest way to find a path forward.

[gasparnagy]

Thx for taking care of this.

I don't think there are single answers for these questions (there are so many situation and needs), but let me share my thoughts based the usages of SpecFlow+ LivingDoc I have seen.

Does it matter when this output is produced?

The most common usage is to produce the report at the same time of the test execution or directly after in CI (or locally). SpecFlow+ LivingDoc CLI enforced a separate step (generated the data during test execution and run the HTML gen after), but I know that many teams tried to merge this by calling the HTML gen from an AfterTestRun hook.

Should it be the output of a build, or a test-run?

I think the most common usage is to use it with test-run, but I have seen usages where they were only interested in the nice rendering of the feature file and not interested in the test result. So both, but more commonly related to test-run.

Would running a stand-alone tool to extract this information be preferrable to something built into Visual Studio?

Yes, definitely, as it is going to be used mostly as part of the CI. I would even say a "stand-alone command line tool" (no UI).

What do we think should be the output?

I think there are diverse answers for this. The most simple and common usage would require a single-file active HTML (maybe multi-file HTML is also acceptable, but more problematic to share). But there are teams who use this as part of their QA process, where they might need to produce other output (PDF, Word, ???). But I think this latter is more the special case.

When I say active HTML, I mainly mean that something where you can browse the folders and feature files, search, etc. within the offline HTML.

Besides these questions, there are plenty of "nice-to-haves" that I have heard of. These are not necessarily part of the primary goal (or a goal at all), but good to think about them when we set the design. (SpecFlow+ LivinDoc supported some of these.)

• Customizable HTML output (templating?)
• Have summary controls (charts) in the result
• Non-HTML output (PDF, Word, JSON)
• Seamless integration of the result document to common ALM systems, like Azure DevOps, Jira, GitHub
• Manage result history

[tulavalle]

Hi @gasparnagy! How is the live documentation part going? Publishing the HTML extension on Azure is a prerequisite for us to finally migrate from Specflow to Reqnroll and take advantage of all the improvements made. I must admit, I'm very excited about this.

I would like to understand the status of this request, if there is anything in progress, any planning, and if https://github.com/praticandobdd and I can actively contribute (if so, what contribution would add value), especially in terms of development and testing. We have a rich environment for testing this entire process, as well as experience to share with this community.

[gasparnagy]

@clrudolphi is currently working on the cucumber messages support that is a prerequisite of this (see #233), but we realized that it is a bigger task than we (or at least I) expected. Once this is done, we can work on the HTML generation.

In principle we could work on the HTML generation even before, because we can get sample Cucumber Messages reports from other Cucumber implementations. But to start with that, we would need some design concept at least. My preference would be to use React (like the old one did), so that we could reuse something from https://github.com/cucumber/react-components. But that's all I have and I'm not a React expert. If you (or anyone else) is fluent in React, you could start working on a prototype...

[a-korevaar]

Having a LivingDoc equivalent feature would allow us to migrate from SpecFlow to Reqnroll. I would gladly contribute to the development or testing of such a feature.

[Code-Grump]

Can you tell us what you need the output to be? Any technical constraints?

[a-korevaar]

We're currently using the SpecFlow Azure DevOps extension and the CLI to generate LivingDoc in HTML format and publish it externally. I have added some custom code to the build pipeline to include images (mainly screen mockups) as well.

For us, the integration in Azure DevOps is a nice to have, as developers are focused on the feature and step files in their local repositories and product owners and stakeholders use the published documentation.

The HTML format is ok, but it would be nice to have other options such as markdown or pdf as well.

I'm not familiar with Cucumber reports but it looks interesting. We are for the most part bound to on-premises solutions though.

[Code-Grump]

We are for the most part bound to on-premises solutions though.

This is good to know as a constraint. I'm certain it's a common one too.

[gasparnagy]

I agree. Test results are highly confidential in many environments I worked with and storing any test result data introduces a lot of complication (GDPR, etc.).

[adrichem]

I love this is getting attention! LivingDocs reports are the only useful source of test results for us. Especially the test runner mixes output from concurrently running scenarios.

1. We need to generate and store results in our CI/CD pipelines .My org needs flexibility in deciding where/when this work is done as we have different projects using different setups in their pipelines. Some are YAML, some are classic. Some are build- some are release. Some pipelines only run tests and have no 'real' branch associated with them. Having LivingDoc invoked by shell script ticks all our boxes w.r.t. integration into our devops.
2. Having HTML output is critical for us as we include the HTML into viewers configured in DevOps on the pipelines..
3. Its very nice LivingDocs contains the markdown from the scenarios. This helps communicate why tests do what they do.
4. When running locally we sometimes manually create a report (both for test results but also to show the content of scenario to someone in a nice format). So ability to execute multi platform is needed.
5. The Integration with Azure DevOps is not something we use yet. We are entering a situation where its going to be used. Even then not always.....for end-to-end tests there is no 1-to-1 correlation between test results and a specific branch.
6. We need LivingDoc 's ability to filter on outcomes.
7. The analysis on step-definitions for us adds no value as they are shared over multiple projects. Its normal for a step-definition to not be used in a test run.
8. We use the ISpecFlowOutputHelper to include OpenTelemetry TraceId's and links to logfiles (Blobstorage / S3 etc etc) into the LivingDoc report.

[gasparnagy]

Thx. The "filter on outcomes" is especially a good point that I did not think of. Also good to know your feedback about the analysis on step-definitions. I had the same feeling.

[Digbigpig]

Hi I'd like to chime in
In the past i've had a great experience using specflow and specrunner+. I'm now setting up a new test automation solution and realized we are without a reporting feature in reqnroll. Like others have mentioned, it could be a reason to delay reqnroll adoption over specflow.

The main feature we need is the ability to generate an html report that can be easily navigated and parsed. We'd like it to support navigation/ filtering between test categories. We'd like for easy screenshot integration into the report. Being able to see the step, the timing, and console output for each test is an expected feature.

I come from using spec runner and that was everything I needed. The livingdoc stuff looks cool and has nice features but some of it seems overkill for my general use case, however I never used it besides failing to integrate it with reqnroll :)

PS reqnroll sounds closer to rick-roll than rock n roll to me🕺

[gasparnagy]

There is no realistic chance that we will be able to redo the SpecFlow+ Runner. What would be good though is to make the livingdoc generator HTML "template" more extensible, so one could make a more "classic test report" kind of template for it (we can even put it to the product itself).

[brajpal-jangir]

Thanks for bringing this in discussion!

We used CLI command to generate reports. These are integrated in DevOps pipelines.

• In the active development phase, we generate living doc without results to let Product owner and other stakeholders review the test coverage. Living doc also have links to UserStory (from TFS) to validate the requirement.
• In times of GA, we generate living doc with results and archive them as test metrics as part of GA.

I also tried to use SpecFlow+LivingDoc DevOps extension, but it has some limitations and that’s where I switched to use CLI. Few limitations which I came across are

• We have multiple Specflow Projects in GitRepo and it’s not possible to publish multiple Living doc for single git branch. It will always override the previous uploaded document for same git branch.
• We also came across the issue when Living document HTML file size is more and it failed to publish document on DevOps. I reported this issue but am not able to find the link now.

[MathiasLopes]

I've found a way to generate LivingDoc HTML from Reqnroll unit tests. I've made a GitHub project to explain it, here's the link if you're interested:
https://github.com/MathiasLopes/ReqnrollTemporaryLivingDoc

[Insire]

Hi @Insire , I did another project for proof of concept. The report was generated successfully, both with tests passing and failing in the same execution, and with all tests passing (in this case, I ran it with 1 because for testing I removed the other one that I implemented to fail and do my proof of concept). Here is a print of the csproj, test executed and livingdo c command used to generate the documentation:

But I'm having trouble publishing this via pipe release so that I can display the generated HTML in Azure, it could even be on a wiki page. What I need to solve is that stakeholders have access to the documentation in a centralized way in Azure, like the livingdoc extension did (before it was discontinued and refused authentication). One alternative I thought of is to generate the report via the Livingdoc CLI command line in the release pipe yml after running the tests, save it in a folder based on the repository and branch in S3, and somehow retrieve this HTML via the S3 URL and display this report on a wiki page, I don't know if by pointing to a given URL or via "embed" to take advantage of the wiki's own widget that enables this display of embed URLs, at least temporarily until we have an extension for Reqnroll that can do this. Anyway, has anyone had this problem and solved it, even if temporarily in some way? If so, do you have a tutorial or step-by-step guide?

We are using the azure devops specflow extension which seems to pick up the livingdoc html file, when i upload it has artifact during CI/CD build. I am already manually creating the html file via the specflow cli and upload it to the current build via cake, which uses this under the hood.

[tulavalle]

And is it possible to authenticate with the Microsoft account from the livingdoc extension? I was able to do it until the end of December, but not anymore:

[RajeshPoola]

They pulled the plug on Specflow https://support-hub.tricentis.com/open?id=post&number=NEW0001432

[tulavalle]

Yes @RajeshPoola. That's why I came to change this idea of ​​what you're doing to view the post-deploy report.

[RajeshPoola]

@tulavalle They are currently not allowing new signups or logins. Mine is still working as I signed up a long time ago. However, if my browser or Azure forces me to reauthenticate, it could become a problem for me too.

[ShanmukhaGajula]

Yes I do.. thanks for the input. One question how can i add base64 image to the reporter?

…

On Wed, Aug 14, 2024, 11:41 AM dominik-niebuhr ***@***.***> wrote: @MathiasLopes <https://github.com/MathiasLopes> , small correction here. I gave the command as livingdoc feature-folder "mypath to features folder" --output-type HTML -t bin/Debug/net8.0/TestExecution.json instead of livingdoc feature-folder . --output-type HTML -t bin/Debug/net8.0/TestExecution.json but after removing the path it worked. Thanks again for the workaround. Appreciate your inputs. Have you got the test results to show up? — Reply to this email directly, view it on GitHub <#196 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFBWBLKU5SQDZMCEN5KKEYLZRLYHTAVCNFSM6AAAAABKFOZSCGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMZTGM2DQNI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Pressacco]

Status Update

As I understand it, the current game-plan is:

• to be compatible with the Cucumber eco-system by supporting Cucumber messages
  • in order to integrate with tools like Pickles, Report Portal, etc.
• which is being tracked by Feature/Cucumber Messages #233

yes?

[clrudolphi]

Yes. Support for Cucumber Messages is the first step.