Unable to generate dynamic tests from data fetched async

[jean-moldovan]

Hello!

I think this is somewhat related to https://stackoverflow.com/questions/48578761/run-tests-on-dynamic-file-in-cypress-io and #835.

In my scenario I need to fetch test data from the api and generate tests from that data. If I define data as a local var (no async) it works all fine, but if I need to prefetch it from the api (async) I get: No tests found in your file:

describe('Foo', function () {
  axios.post(url, { ... })
    .then(res => {
      const { data } = res
      console.log(data) // <- data is available here, request went fine
      data.forEach(item => {
        it('should generate test', function () {
          console.log(item)
        })
      })
    })
})

Thanks!

[jean-moldovan]

Related mocha thread: mochajs/mocha#2221

[jean-moldovan]

@chrisbreiding , Hello and sorry to bother. I see that there was some work being done in order to upgrade mocha to the latest version #2703. Is this going to enable support for --delay flag or maybe it's something that needs to be worked on separately?

This seems like a big blocker for my team, is there a workaround that we can use in the meantime? Thanks!

[chrisbreiding]

Support for the --delay flag will need separate work.

I created a proposal to outline what needs to be done.

As far as a workaround, the only thing I can think of is to utilize the Preprocessor API to modify the spec file on the fly and write the necessary data into the file, so you can synchronously iterate over it in the spec file to create the tests.

[thatwpdeveloper]

@jean-moldovan did you manage to get this working? I am trying to do the same, but using cy.request instead of axios.

[AttackOnMorty]

@thatwpdeveloper you can use the Preprocessor API to fetch the data before the test cases run

[thatwpdeveloper]

@AttackOnMorty Thanks!

[jean-moldovan]

Looks like this is something that confuses a lot of people, so I'll post a workaround here:
NOTE: I have tested it on:

    "@cypress/browserify-preprocessor": "^1.1.2",
    "cypress": "3.1.4",

No guarantee this would work the same on the latest versions.

In my /plugins I have index.js file which basically reads the data from the api and then prepends it as a variable to the test spec.

Simplified example:

const fs = require('fs')
const path = require('path')
const axios = require('axios')
const browserify = require('@cypress/browserify-preprocessor')

module.exports = (on, config) => {
  const baseSpecFilePath = path.resolve(__dirname, '../integration/mytest.template.js')
  const runSpecFilePath = path.resolve(__dirname, '../integration/mytest.spec.js')

  // https://github.com/cypress-io/cypress/issues/3114#issuecomment-459481491
  on('file:preprocessor', (file) => {
    if (!file.filePath.includes('mytest')) {
      return browserify()(file)
    }

    return new Promise((resolve, reject) => {
      axios.get('/users')
        .then(res => {
          const testdata = JSON.stringify(res.data)

          fs.readFile(baseSpecFilePath, { encoding: 'utf-8' }, (err, data) => {
            if (!err) {
              const stream = fs.createWriteStream(runSpecFilePath)

              stream.once('open', () => {
                stream.write(`var testData = ${testdata}\n`)
                stream.write(`\n`)
                stream.write(data)
                stream.end()
                resolve(runSpecFilePath)
              })
            } else {
              throw err
            }
          })
        })
        .catch(err => reject(err))
    })
  })
}

Basically what happens is:

1. We have our spec in mytest.template.js. It uses testData variable. testData needs to be injected as a string via preprocessor API.

describe('my case', function () {
  testData.forEach((item, i) => {
    it(`should work for ${item}`, function () {
      // item is available here
    })
  })
})

2. We read the test data from the API, read the contents of the spec template and kind of merge both together into the new spec file, where testData would be available.

[dwelle]

This could be implemented without using Mocha --delay --- if Cypress allowed us to export a promise/async function from the spec file, and awaited its resolution.

module.exports = async () => {
    const data = await /* ... */;

    describe('test', () => {
        data.forEach( item => {
            it(item.name, () => {
                /* ... */
            });
        });
    });
};

This way there'd be no need for extra API. The above pattern is kinda universal and intuitive, and just by looking at it everyone knows what's going on.

Though, more work would be needed on Cypress part in order to change how files are being added to Mocha instance.

[adg29]

This could be implemented without using Mocha --delay --- if Cypress allowed us to export a promise/async function from the spec file, and awaited its resolution.

module.exports = async () => {
    const data = await /* ... */;

    describe('test', () => {
        data.forEach( item => {
            it(item.name, () => {
                /* ... */
            });
        });
    });
};

This way there'd be no need for extra API. The above pattern is kinda universal and intuitive, and just by looking at it everyone knows what's going on.

Though, more work would be needed on Cypress part in order to change how files are being added to Mocha instance.

You're right, exporting a promise/async function and awaiting its resolution would be more intuitive. Question but where can I dig into the cypress mechanism for adding files to the mocha instance? Once I understand that I would be glad to make an effort at an implementation sketch.

[Kwon11]

Can someone help me understand how to modify @jean-moldovan 's solution to not break import/exports in the mytest.spec.js file?

The recipe in the cypress repo does not address what happens if you modify the preprocessor and I cannot figure out how to run the resulting file (runSpecFilePath) through browserify to avoid the "cannot use import statement outside a module" error

EDIT:
I've decided to just write the JSON to a separate file, and expect it to be available for import in "mytest.template.js" which for me is just "mytest.spec.js".

I am now running into an issue where the test seems to refresh, and the preprocessor runs repeatedly once i start running the test in the browser. I am looking to debug this now but i no longer need help resolving the original issue i asked for.

Looks like i need to understand the effects of

This callback function can and will be called multiple times with the same filePath.

The callback function is called any time a file is requested by the browser. This happens on each run of the tests.

EDIT (for posterity):
if you make a fetch call each time and then overwrite the "data" file that you are creating, then each time the preprocessor function is called for that file you will cause cypress to reload with the updated file.

on('file:preprocessor', (file) => {

    if (!file.filePath.includes('mytest')) {
      return browserify()(file);
    }
    if (!fetched) {
      fetched = true;
      fetching = new Promise((resolve, reject) => {
        fetchFunction(fetchConfig)
          .then((initialData) => {
            const stringifiedInitialData = JSON.stringify(initialData);
            // eslint-disable-next-line no-undef
            fs.writeFile(path.resolve(__dirname, '../support/data.json'), stringifiedMasterEligibility, (err) => {
              if (err) {
                reject(err);
              }
              resolve(browserify()(file));
            });
          });
      });
      return fetching;
    }
    return fetching;
  });```

[soldmtrieu]

Does anyone have an updated solution or a simpler workaround for this?

I'm also running into "No tests found" when using async fetch in my describe block...

[leonfs]

Yes, the simplest solution we found was to fetch the data before running the tests. Just run a pre-run step in your npm start that fetches and formats all the data required. When you need to run your tests, just require them in; ideally, it should be a simple JSON file you created with a very simple JS script.

I hope it helps!

[soldmtrieu]

Thanks @leonfs !

What a game changer!

Ended up creating separate JS set up scripts and executing them with cy.exec() in a before() statement.

Had so much trouble trying to fit into the restrictions of how to do it within cypress, I trapped myself into that box.

This "Think outside the box" approach provides way more flexibility and freedom 👍

Kudos again!

[MattHowdy]

I created a separated script that fetches the data from the API and saves the results as JSON in the fixture folder. I run that script in the config file. I require that fixture in the suite file and I loop through it to generate the tests.