Use jest reporters to report more events

[aaronabramov]

since we use a single instance of each reporter to report events from all projects/runners i though it would be helpful to report some more things, not only onTestRunStart, onTestResult, onTestRunFinish, etc, but also things like onProcessStart, onContextReady and other things.
I want to introduce some logging and performance monitoring, but there's no easy way to know how long each step of jest takes right now

[palmerj3]

Would also like to add this to the feature request #4500

TL;DR - add a callback option in reporters for when an individual test starts and completes.

[aaronabramov]

@palmerj3 what is your usecase?
this is not really possible in Jest because we run every test file in a separate process, and reporting each test() or it() means that we'll have to send a lot of data between two processes, which will introduce a lot of overhead

[palmerj3]

@aaronabramov my main use case is to introduce some VCR tech into a testing suite and I want to be able to isolate VCR files per test.

I'm able to do this with jasmine reporters but I get the feeling that jasmine reporters isn't the way to go since you've forked jasmine and created your own reporter API. I would worry about the long-term sustainability of that approach if I were to base my solution off jasmine reporters.

[aaronabramov]

@palmerj3 VCR you mean recording and replaying network requests?
it seems similar to what snapshots do and yeah, in this case it should probably live inside the worker/test code and it looks more like "hooks" rather than reporters.

we always had and will probably have two reporting mechanisms.

1. Everything that happens within the test file (jasmine right now)
2. Everything that happens outside the test file (aggregated test results per file + reports for the whole test process)

both of them are in incomplete (probably 80% done) state and unfortunately i don't have time right now to focus on them :(

[aaronabramov]

@palmerj3 i'm pretty sure that is what you want https://github.com/facebook/jest/blob/master/packages/jest-circus/src/event_handler.js#L20-L81
with ability to extend it same way we did for snapshots https://github.com/facebook/jest/blob/master/packages/jest-circus/src/legacy_code_todo_rewrite/jest_adapter_init.js#L43

[palmerj3]

@aaronabramov yes! If you have any suggestions or if there is something I can do to help build these things I'm more than happy to put in the time.

[segrey]

How can a reporter access addEventHandler from jest-circus/src/state.js? @palmerj3 Have you managed to access individual test starts and completes? I'm looking for a way to have a more responsive test run visualization. Unfortunately, onTestStart/onTestResult target a test file, and I'd like to have more fine-grained events.

[palmerj3]

@segrey jest reporters don't have access to that information but jasmine reporters do. So if you use the jasmine reporter API then you can access per-test details.

[segrey]

@palmerj3 Thanks for the quick reply! IIUC, a jasmine reporter should be specified via --setupTestFrameworkScriptFile and path to it has to be added to transformIgnorePatterns. Since passing these cli options might overwrite configuration from package.json/jest.conf.js, it's needed to merge configurations properly. Please let me know if I missed something and there is a simpler approach.

It'd be great to have such a capability in reporter API. @aaronabramov What do you think about optional reporting each test() or it()? AFAIK, karma sends each test result from browser to karma server.

[segrey]

Is there a way to pass a setup script via --setupTestFrameworkScriptFile, but ensure that files imported by the setup script will not be transformed during testing. Currently, the files are not mocked during testing according to https://facebook.github.io/jest/docs/en/cli.html#setuptestframeworkscriptfile-file.

[segrey]

Just discovered a way to configure setupTestFrameworkScriptFile and update transformIgnorePatterns programmatically. (Since transformIgnorePatterns can only be specified in configs, tools can't modify it.)

// my-custom-reporter.js
class MyCustomReporter {
  onTestStart(test) {
    const setupPath = require.resolve('./my-jasmine-setup.js');
    const dirname = require('path').dirname;
    test.context.config = Object.assign({}, test.context.config, {
      setupTestFrameworkScriptFile: setupPath,
      // ignore transforming all files in this folder
      transformIgnorePatterns: test.context.config.transformIgnorePatterns.concat([dirname(setupPath)])
    });
  }
}
module.exports = MyCustomReporter;

Then, jest --reporters=./my-custom-reporter.js
@aaronabramov Is it a legitimate way of configuring Jest?

[palmerj3]

@segrey I most likely don't understand the full picture of what you're attempting. But wouldn't it be a lot easier to simply put all the imports for your setup script in a dir like tools/jest/jasmineSetup and then add <rootDir>/tools/jest/jasmineSetup/**/*.js to transformIgnorePatterns?

[segrey]

@palmerj3 Sorry for being unclear. My case is developing a tool for Jest (IntelliJ IDEA / WebStorm Jest test runner). Running jest tests brings a test tree in UI. As tests are running, the tree is being updated: new tests are added as running nodes, passed tests are shown as green test nodes, failed - as red ones. The ealier the test node is added to the tree or updated, the better. However, currently tests results are available after all tests are run in a file (onTestResult event). So, I'm attempting to have a test tree reflecting current test run status better than now. AFAIU, the only way to have events for each test() or it() is to setup a jasmine reporter. Unfortunately, it's impossible to modify transformIgnorePatterns in user's config.

[SimenB]

@segrey can you open up a separate issue for setting transformIgnorePatterns through a CLI flag?
All config should be possible through CLI flags, IMO. And if this flag in particular is a blocker, we should fix it.

[segrey]

@SimenB Desired behavior of --transformIgnorePatterns would be to add a pattern to already defined in config, not overwriting existing ones as it works for other cli options. I can file the issue (definitely not a blocker), but maybe there should be filed a different issue - extending TestRunner API to optionally report results of each it()?

[SimenB]

add a pattern to already defined in config, not overwriting existing ones as it works for other cli options

Ah, interesting. That would be weird from a CLI API standpoint, wouldn't it? Not sure how

extending TestRunner API to optionally report results of each it()?

You can use testResultsProcessor to get each it already, is that enough?

---

To circle back to your original question:

IIUC, a jasmine reporter should be specified via --setupTestFrameworkScriptFile and path to it has to be added to transformIgnorePatterns.

Should be just reporters config. jest-circus is not in use, yet.

[segrey]

That would be weird from a CLI API standpoint, wouldn't it?

@SimenB Probably you're right. The motivation is to not overwrite transformIgnorePatterns defined in configs. It can also be achieved with running jest --showConfig at first to find out transformIgnorePatterns from configs, then pass all transformIgnorePatterns via CLI. This approach feels a bit complicated to me.

Currently, the integration uses --testResultsProcessor for jest < 20 and --reporters for jest >= 20. Yes, it's possible to get results for each it, the problem is that results are available after all tests in a test file are completed. It'd be great to report it results once the test is completed. In this case it wouldn't be needed to setup a jasmine reporter which is not great for long-term support.

[zaqqaz]

@aaronabramov is still in progress and do we have any updates ?

[bogomya]

@aaronabramov It would be nice to have a lifecycle hook like jasmine's specDone in Jest reporter API. It can be especially useful for the report generation tools, which will be able to attach some additional information to any it block in a report.

[SimenB]

We're currently doubling down on getting jest-circus out the door (see the commits from today and yesterday), which will make this work pretty much out of the box.

[patrickhulce]

@SimenB is there a way to give this a try today with JEST_CIRCUS=1 or something similar? The events in https://github.com/facebook/jest/blob/master/packages/jest-circus/src/event_handler.js#L20-L81 look great, but it doesn't look like they'll suddenly be emitted to reporters.

Happy to help work on this a bit if necessary, live individual test case reporting ala #6225 is a priority for me :)

[SimenB]

To run with circus, just install jest@23 and jest-circus@23, then invoke jest with JEST_CIRCUS=1 jest 🙂

But if I understood @aaronabramov correctly in #6225 we need a way to bubble the result out of circus (which runs inside the vm) up into reporters (which runs outside of it), and some way of namespacing the event (as tests run in parallel).

I guess keying by runner and test file path or something should be enough. Or just generate a uuid somewhere, but I think a deterministic key is best

[aaronabramov]

it's even more complicated than this :)
currently we have:
master process -> worker process -> vm

we run circus outside the vm and only return to back to the worker when we completed the whole test run.
after that we transform results and return it back to the master process via IPC (reproters live in the master process)

getting granular events back to reporters will be a pretty challenging task. it'll also affect performance

[patrickhulce]

Thanks for the pointers I was able to get a hacked up version going (master...patrickhulce:report_progress), so looks very doable with circus, awesome! :)

Last time I tried a PR to a FB project I got caught up in the CLA process with my employer, but let me know if I can help with anything else!

[aaronabramov]

@patrickhulce this is awesome!

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 7 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.