Remove regex check for test files

[bdwain]

Are you requesting a feature or reporting a bug?

request a feature

What is the current behavior?

There are regexes that each test file is run against, and if they do not match, you get an error

No tests to run. Either the test files contain no tests or the filter function is too restrictive.

What is the desired behavior?

Can this regex test be disabled and instead look for tests when actually running/parsing the code? I've found valid use cases where those regexes might not match.

In my project, I've broken up my app into several "subapp" libraries, and each subapp exports a suite of tests, which are run in the main repo's testcafe suite.

for example:

library test file

export default function libraryTests(){
  fixture('Library tests').page('localhost:1234');

  test('Does some stuff', async t => {
    //await t.expect(....); //expect something
  });
}

main test file

import {libraryTest} from 'library';

libraryTest();

currently, this fails when i try to run the tests pointing at the main test file. However, it works if i add a commented out fixture and test. Ideally, I would not need to add a commented out fixture and test just to satisfy the regex.

[kirovboris]

Hi @bdwain,

In your example, simple parsing of the main test file won't find the fixture as well.
Besides, it'll be much slower than regexp. If we start parsing all dependencies, it can require much time, because it should also parse all your modules/libraries/node_modules... which were imported in the test file.

We can't get fixture/test names during script execution, because it can cause side effects. E.g. you have a .js file to setup test database script in a directory with tests. We can't execute it to check if it's a fixture file. So we need static file analysis anyway.

[bdwain]

Could the regex just be removed and not replaced then? It seems like it’s only used to filter out source files that have no tests and fixtures. But you could just assume there are tests in each file and error if there are not. My example works if I just add a commented out test, so it seems like actually checking is unnecessary.

…

On Jan 26, 2018, 6:47 AM -0600, Boris Kirov ***@***.***>, wrote: Hi @bdwain, In your example, simple parsing of the main test file won't find the fixture as well. Besides, it'll be much slower than regexp. If we start parsing all dependencies, it can require much time, because it should also parse all your modules/libraries/node_modules... which were imported in the test file. We can't get fixture/test names during script execution, because it can cause side effects. E.g. you have a .js file to setup test database script in a directory with tests. We can't execute it to check if it's a fixture file. So we need static file analysis anyway. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[kirovboris]

It seems like it’s only used to filter out source files that have no tests and fixtures.

Yes, it's correct.

But you could just assume there are tests in each file and error if there are not.

As I said before, we can't run every file by glob patterns like test/**/*.js because of side effects in non test files.
For now, we suppose a user doesn't have to concern whether the passed files have code with side effects or not, e.g. you can just run

testcafe chrome tests/**/*.js

instead of

testcafe chrome tests/e2e/fixtures/**/*.js

It can matter if you're going to add testcafe tests to an existing project with a complex directory structure.
If you need a single entry point for tests in your libraries, you can implement a nodejs runner using TestCafe programming API and get rid of export statements in your fixtures.

[bdwain]

I am already using the node api. Could the regex check be disabled with an option then? This way it's a new behavior that only affects people who opt-in. something like

runner.run({runAllFiles: true})

[kirovboris]

Unfortunately, we wouldn't like to add an option that can cause unexpected effects on a user's computer. As a workaround, you can regroup tests or change a directory structure.

[bdwain]

@kirovboris It would not be an unexpected effect. The entire point of the option is to run every file in the user's regex, so if they enable it, they should know what it does (and it would presumably be explained in the docs).

This would prevent needing a hacky workaround to share suites across different files, so I really think it's something that should be addressed.

[EvanNGoldstein]

Any update on this? In order to share suites across files I have to add something like

fixture `A fixture`;

test('', () => {});

to my files, which is extremely hack-y...

[bxt]

I also think these regexes should be removed for two reasons:

• Like mentioned before it prevents you from putting test case generators into a library file. The tests will just stop running without any notice. This left me wondering with no indication what happended.
• Also if you do rely on the behavior of the regexes and have e.g. some database cleanup code in a JS file, once a developer puts in test. and fixture. into them (even in a comment) they are suddenly run. This is I think very unexpected and might cause data loss etc.

I also do not agree with @kirovboris's previous points:

As I said before, we can't run every file by glob patterns like test/**/*.js because of side effects in non test files.

Most people probably name their tests files something like foo.spec.js anyways, so one could use test/**/*.spec.js then. There are also other tools which need extra configuration just for testcafe (see e.g. #1735) and those also use such a globs, so there would be no changes required in most cases.

For now I will put a comment like this into every test file:

// test( fixture( <- workaround for https://github.com/DevExpress/testcafe/issues/2074

Which unlike Evan's solution does not require any actual code but still is very hack-y.

[bdwain]

agreed with @bxt . bump. This issue really should be addressed.

[kirovboris]

Hi @bdwain @bxt,

I agree that the current implementation can cause problems with test generators.
We decided to add an option that enables the behavior you described in the upcoming releases.
Later, when we release a major update, we will enable this behavior by default.

[bdwain]

awesome! thanks!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs or feature requests. For TestCafe API, usage and configuration inquiries, we recommend asking them on StackOverflow.