Plugin not working in run mode triggered via module API

[jwohlfahrt]

Description

We were using this plugin with version 6.x.x in conjunction with cypress's module API run mode (starting cypress programmatically), and were able to upload results without issue. After bumping to version 7.3.0, and updating to new plugin invocation, I'm no longer seeing results uploaded at the end of test runs, but the plugin still authenticates fine with our x-ray instance at beginning of the run.

I did see this change pop up in the change logs for an early version of 7.x.x: #334. I'm curious if this update made it so that running the plugin via the module API run mode no longer works. I know the README mentioned this only worked via CLI, so just curious if that is the crux of the issue. It was nice being able to use this with the module API, as we call cypress this way in our CICD runs to ease the burden of passing a bunch of env variables inline to the CLI. Thanks!

Cypress version

12.17.4

Plugin version

7.3.0

Jira/Xray type

Cloud

Configuration

No response

Acknowledgements

• I have made sure not to include any sensitive data in screenshots or obfuscated all occurrences.
• I have made sure not to include any sensitive data in descriptions or replaced all values with placeholders.
• I have made sure not to include any sensitive data in code snippets or replaced all values with placeholders.

[csvtuda]

Hi @jwohlfahrt,

that's interesting, I did not know about the module API. I just tried it out myself using 7.3.0 and it works fine, including the results upload at the end of the test runs. This is my minimal module API configuration:

const cypress = require('cypress');

cypress.run({
    env: {
        XRAY_CLIENT_SECRET : "xyz",
        XRAY_CLIENT_ID : "abc",
        JIRA_API_TOKEN : "zyx",
        JIRA_USERNAME : "[email protected]",
        JIRA_PROJECT_KEY : "CYP",
        JIRA_URL : "https://example.atlassian.net"
    } 
})

---

I'm no longer seeing results uploaded at the end of test runs

Situations where the plugin won't upload any results typically are:

• the PR you've found is the culprit
  • can be detected through this log message
• xray.uploadResults is false for some reason
  • can be detected through this log message
• the Cypress results have status failed, e.g. if there are no spec files to run
  • can be detected through this log message
• you have registered another event listener to the after:run event after the call to configureXrayPlugin, which will overwrite the plugin's event callback
  • cannot really be detected, but I suggest using something like cypress-on-fix if you're using more than one plugin

Other than that, the plugin will run as normal, i.e. it will always try to upload stuff unless all tests are broken. But in that case, the plugin should also log a lot of errors or warnings.

Are there no log messages at all after the initial authentication (which - just in case - happens outside the after:run callback)?

[jwohlfahrt]

Correct, I'm not seeing any logs for the above situations in our CI/CD console output.

I did try testing out different versions of 7.x.x, and it seems my issue pops up in 7.2.0 specifically. At 7.2.0, whenever I run my test suite locally (via module API), I get an error about multiple test execution issue types:

/Users/jwohlfahrt/git/cypress-xray-integration-poc/reports/0ed134e0-f5c6-40f4-85e9-efddfd01b20c.html
│ Cypress Xray Plugin │ WARNING │ Encountered problems during plugin execution!
│ Cypress Xray Plugin │ WARNING │ No test results were uploaded
│ Cypress Xray Plugin │ ERROR   │ Failed to upload Cypress execution results.
│ Cypress Xray Plugin │ ERROR   │   
│ Cypress Xray Plugin │ ERROR   │     Caused by: Failed to retrieve Jira issue type information of test execution issue type called: Test Execution
│ Cypress Xray Plugin │ ERROR   │   
│ Cypress Xray Plugin │ ERROR   │     There are multiple issue types with this name, make sure to only make a single one available in project *****:

Whats odd is that I don't get this warning in CI/CD. Instead, its like the hooks don't run at all, but maybe something is failing silently in that env. I'm still trying to track down why the API thinks we have dupe issues, because the project we are integrating with only returns one Test Execution type when viewing project details via Jira API.

7.1.0 works as expected, and X-Ray API gets called appropriately. I'm going to keep debugging but wanted to update on what I'm seeing so far. Thanks!

[jwohlfahrt]

@csvtuda So the multiple issue types error was the issue at hand here. For some reason, in CI/CD, this error did not get logged, but when I resolved the dupe issue, upload worked as expected. Given that, I think we can close this issue, unless you wanted to investigate lack of logs. Appreciate the quick replies and investigation!

[csvtuda]

Glad to hear it's working now! The only explanation for the missing logs I can offer based on the information provided is that the plugin logs warnings and errors to stderr instead of stdout. So maybe the CI/CD pipeline doesn't show stderr messages? Highly unlikely though. I think I'll file this bug under let's just hope it never happens again 😃