feat: Add GitHub Actions Reporter #11320

ockham · 2021-04-20T21:30:29Z

Summary

Based on @mariusGundersen's #10741, which is in turn based on @stefanbuck's stefanbuck/jest-matcher#2.

Add a Jest reporter to output a format that will be interpreted by GitHub Actions as annotations -- i.e. any errors reported by Jest will show up in a PR's diff below the offending line. This should make for significantly improved Developer Experience, since developers no longer have to scan CI logs for errors, but instead will be notified of them at the location where they happen.

All credit really goes to @stefanbuck and @mariusGundersen. I've only added a simple test (as was requested by @SimenB over at #10741, which seems to have gone stale) based on SummaryReporter's, ran the reporter through eslint, and made a few minor fixes.

This still needs a bit of tweaking (e.g. of the output format), hence setting the PR to draft status.

Closes #10741

Test plan

Run

yarn test packages/jest-reporters/src/__tests__/GithubActionsReporter.test.js

facebook-github-bot · 2021-04-20T21:30:34Z

Hi @ockham!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

stefanbuck · 2021-04-21T06:19:48Z

Thanks for picking this up again. I'm not really familiar with the Jest code base, but from what I can tell this looks good. Just one thing, when running Jest in another CI enviroment than GitHub Actions we might don't want to print such logs. Luckily there is a env variable for that.

GITHUB_ACTIONS Always set to true when GitHub Actions is running the workflow. You can use this variable to differentiate when tests are being run locally or by GitHub Actions.

Just for the recorder / issue watchers, the initial feature request was made in #10309

ockham · 2021-04-21T16:44:39Z

BTW, giving this some real-world testing over at WordPress/gutenberg#31041. Already found a few minor issues through that 😬

stefanbuck · 2021-04-21T17:34:22Z

Amazing, let me know if I can help.

ockham · 2021-04-21T18:15:44Z

Amazing, let me know if I can help.

Thank you! I think I'll polish the other PR first, and will then carry my modifications back over here 🙂

ockham · 2021-04-23T13:16:11Z

I've merged the real-world PR (WordPress/gutenberg#31041) -- it contains a few screenshots to showcase how the annotations work.

I've carried over a few changes (including the GITHUB_ACTIONS check). I think I'd like to open it for review now -- it probably needs some more polish (and maybe test coverage?), but I'd appreciate some pointers to know what to add 🙂

As for coloring the output: The Jest failure messages clearly include ANSI color codes (see test fixture), so it seems that they get stripped off at a later stage (possibly by GitHub Actions?).

At a closer look, this doesn't seem to be limited to this reporter -- Jest would normally also color its overall test results, but those colors never seem to make it to the GH Actions workflow run console:

So I won't be investigating the missing colors anymore for now, since it seems like the problem isn't specific to this reporter (but to how GH Actions process Jest output). (I've Googled a bit and found e.g. #3877 -- indicating that we might try to use the --colors cmd line arg to force them; but that that might also end up being ignored, since that option is passed through to chalk. Elsewhere, I found information on the heuristics that chalk uses to determine whether an output stream supports colors or not -- maybe that's what doesn't work so well with GH Actions.)

ockham · 2021-04-23T13:23:01Z

So I won't be investigating the missing colors anymore for now, since it seems like the problem isn't specific to this reporter (but to how GH Actions process Jest output). (I've Googled a bit and found e.g. #3877 -- indicating that we might try to use the --colors cmd line arg to force them; but that that might also end up being ignored, since that option is passed through to chalk. Elsewhere, I found information on the heuristics that chalk uses to determine whether an output stream supports colors or not -- maybe that's what doesn't work so well with GH Actions.)

~~chalk/supports-color@79d1303 🤔 Present in chalk >= v7.1.0~~

~~vs. https://github.com/facebook/jest/blob/a28f14a9dbd4621eb2e848034cdfdfec28ff9866/package.json#L29~~

~~So maybe all that's needed is a version bump for chalk?~~ 🤔

Edit: Ignore 🙄 GH Actions support was added in chalk/color-supports v7.1.0 (rather than chalk itself). And chalk has been using that version of color-supports apparently since v3.0.0

SimenB · 2022-02-23T15:24:38Z

Hey @ockham! I lost track of this. What's the current state? 🙂 Would love to ship this in Jest 28!

From my own look (beyond the boring stuff like rebase, changelog, and signing the CLA) this looks close to complete

SimenB · 2022-02-23T15:28:24Z

packages/jest-reporters/src/GithubActionsReporter.ts

+    _contexts?: Set<Context>,
+    aggregatedResults?: AggregatedResult,
+  ): void {
+    if (!process.env.GITHUB_ACTIONS) {


I think we should use ci-info instead of manually checking for the env. should make the test cleaner as well as you can use jest.mock

packages/jest-reporters/src/GithubActionsReporter.ts

ockham · 2022-02-25T11:39:57Z

Hey @ockham! I lost track of this. What's the current state? 🙂 Would love to ship this in Jest 28!

From my own look (beyond the boring stuff like rebase, changelog, and signing the CLA) this looks close to complete

Hi @SimenB! I'd love if we could get this into Jest 28. Unfortunately, I'm going to be a bit busy next week. Can you tell me when's the deadline for getting this PR merged to make it into v28?

In any case, I'll prod our legal dept to put me on our company CLA list. I'll also try to polish the PR a bit tomorrow if I manage to find some time 😅

SimenB · 2022-02-25T12:15:12Z

Can you tell me when's the deadline for getting this PR merged to make it into v28?

Eh, hard to say. Couple of weeks, probably. But if we make sure not to activate it by default, then we can land it outside of a major and people can use it (via github-actions in config, similar to default?) if they want

ockham · 2022-02-26T17:44:10Z

Can you tell me when's the deadline for getting this PR merged to make it into v28?

Eh, hard to say. Couple of weeks, probably. But if we make sure not to activate it by default, then we can land it outside of a major and people can use it (via github-actions in config, similar to default?) if they want

I've pushed a few more commits to address the feedback I've gotten (thanks!). I'll be out for most of next week, but I should be able to pick up work again around March 7. If that's soon enough (or the PR is good enough as-is 😅), we might still get it merged in time for the major 🤞

packages/jest-reporters/src/GithubActionsReporter.ts

Co-authored-by: Simen Bekkhus <[email protected]>

ockham · 2022-03-14T13:00:14Z

@SimenB 👋 Just checking in to say that I addressed pretty much all feedback (and had a question or two).

My company's legal team is in touch with Meta's to add me to the corporate CLA, I hope there will be some progress soon.

LMK if there's anything else you need from me! 🙂

facebook-github-bot · 2022-03-23T05:49:01Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

SimenB

(gh forces me to comment here, but the feedback in is the threads 🙂)

packages/jest-reporters/src/GitHubActionsReporter.ts

SimenB

thanks!

stefanbuck · 2022-03-30T08:44:57Z

AMAZING 😍 Thanks @ockham, @mariusGundersen and everyone else involved for putting time and effort to make this happen.

Co-authored-by: Marius Gundersen <[email protected]> Co-authored-by: Stefan Buck <[email protected]>

This reverts commit b20c324.

SimenB · 2022-04-05T15:14:26Z

https://github.com/facebook/jest/releases/tag/v28.0.0-alpha.8

SimenB · 2022-04-05T15:18:26Z

that said, no real way of activating it - we should fix that 🙂

github-actions · 2022-05-06T00:08:46Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

ockham mentioned this pull request Apr 20, 2021

feat: Added GithubActionsReporter #10741

Closed

ockham mentioned this pull request Apr 21, 2021

Tests: Adds reporter that shows JS test errors on GitHub when executing GitHub workflows WordPress/gutenberg#31041

Merged

ockham marked this pull request as ready for review April 26, 2021 19:09

SimenB reviewed Feb 23, 2022

View reviewed changes

xt0rted reviewed Feb 23, 2022

View reviewed changes

packages/jest-reporters/src/GithubActionsReporter.ts Outdated Show resolved Hide resolved

ockham added 8 commits February 25, 2022 11:43

Add GitHub Actions Reporter

4b441a1

Use plural to match filename

6b299e2

More expressive test failure message

b6cb64b

Fix line and col by making the RegEx less greedy

48e2778

Disable outside of GitHub Actions

195062d

Fix test description

808bd5c

Add test case for non-GHA context

5c4884f

Whitespace

97a7b1c

ockham force-pushed the add/github-actions-reporter branch from 08a3a39 to 97a7b1c Compare February 25, 2022 11:44

Use ci-info to check for GHA

fc6dd30

ockham added 3 commits February 26, 2022 12:46

Escape all 3 chars that require it

6289d6a

Update snapshot

5ab275e

Add Changelog entry

ec6eb0d

xt0rted reviewed Feb 26, 2022

View reviewed changes

packages/jest-reporters/src/GithubActionsReporter.ts Outdated Show resolved Hide resolved

ockham and others added 4 commits March 7, 2022 14:45

Remove now-obsolete snapshot

20a6562

Capitalize H in GithubActionsReporter

d8c0454

Co-authored-by: Simen Bekkhus <[email protected]>

More capital H's in GitHub

c7c4ac0

Use Node's flatMap

bb4bbe6

facebook-github-bot added the cla signed label Mar 23, 2022

ockham requested a review from SimenB March 23, 2022 13:50

SimenB requested changes Mar 24, 2022

View reviewed changes

ockham added 3 commits March 28, 2022 16:47

Drop ci-info check for GHA

e4d5c01

Strip ANSI codes

1ae84fb

Update test to reflect removal of GHA check

8b5414f

ockham requested a review from SimenB March 28, 2022 16:59

SimenB reviewed Mar 30, 2022

View reviewed changes

packages/jest-reporters/src/GitHubActionsReporter.ts Show resolved Hide resolved

Merge branch 'main' into add/github-actions-reporter

c963d7a

SimenB reviewed Mar 30, 2022

View reviewed changes

packages/jest-reporters/src/GitHubActionsReporter.ts Show resolved Hide resolved

Update packages/jest-reporters/src/GitHubActionsReporter.ts

0461d49

SimenB approved these changes Mar 30, 2022

View reviewed changes

SimenB added this to the Jest 28 milestone Mar 30, 2022

SimenB merged commit b9af518 into jestjs:main Mar 30, 2022

ockham deleted the add/github-actions-reporter branch March 30, 2022 08:13

stefanbuck mentioned this pull request Mar 30, 2022

Show failing tests in the GitHub UI using Problem Matchers #10309

Closed

F3n67u pushed a commit to F3n67u/jest that referenced this pull request Apr 1, 2022

feat: Add GitHub Actions Reporter (jestjs#11320)

b20c324

Co-authored-by: Marius Gundersen <[email protected]> Co-authored-by: Stefan Buck <[email protected]>

F3n67u added a commit to F3n67u/jest that referenced this pull request Apr 1, 2022

Revert "feat: Add GitHub Actions Reporter (jestjs#11320)"

328803d

This reverts commit b20c324.

mrazauskas mentioned this pull request Apr 9, 2022

feat(jest-core): GitHubActionsReporter goes live #12658

Merged

github-actions bot locked as resolved and limited conversation to collaborators May 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add GitHub Actions Reporter #11320

feat: Add GitHub Actions Reporter #11320

ockham commented Apr 20, 2021 •

edited by SimenB

Loading

facebook-github-bot commented Apr 20, 2021

stefanbuck commented Apr 21, 2021 •

edited

Loading

ockham commented Apr 21, 2021

stefanbuck commented Apr 21, 2021

ockham commented Apr 21, 2021

ockham commented Apr 23, 2021

ockham commented Apr 23, 2021 •

edited

Loading

SimenB commented Feb 23, 2022 •

edited

Loading

SimenB Feb 23, 2022

ockham Feb 25, 2022

ockham commented Feb 25, 2022

SimenB commented Feb 25, 2022

ockham commented Feb 26, 2022

ockham commented Mar 14, 2022

facebook-github-bot commented Mar 23, 2022

SimenB left a comment

SimenB left a comment

stefanbuck commented Mar 30, 2022 •

edited

Loading

SimenB commented Apr 5, 2022

SimenB commented Apr 5, 2022

github-actions bot commented May 6, 2022

feat: Add GitHub Actions Reporter #11320

feat: Add GitHub Actions Reporter #11320

Conversation

ockham commented Apr 20, 2021 • edited by SimenB Loading

Summary

Test plan

facebook-github-bot commented Apr 20, 2021

Action Required

Process

stefanbuck commented Apr 21, 2021 • edited Loading

ockham commented Apr 21, 2021

stefanbuck commented Apr 21, 2021

ockham commented Apr 21, 2021

ockham commented Apr 23, 2021

ockham commented Apr 23, 2021 • edited Loading

SimenB commented Feb 23, 2022 • edited Loading

SimenB Feb 23, 2022

Choose a reason for hiding this comment

ockham Feb 25, 2022

Choose a reason for hiding this comment

ockham commented Feb 25, 2022

SimenB commented Feb 25, 2022

ockham commented Feb 26, 2022

ockham commented Mar 14, 2022

facebook-github-bot commented Mar 23, 2022

SimenB left a comment

Choose a reason for hiding this comment

SimenB left a comment

Choose a reason for hiding this comment

stefanbuck commented Mar 30, 2022 • edited Loading

SimenB commented Apr 5, 2022

SimenB commented Apr 5, 2022

github-actions bot commented May 6, 2022

ockham commented Apr 20, 2021 •

edited by SimenB

Loading

stefanbuck commented Apr 21, 2021 •

edited

Loading

ockham commented Apr 23, 2021 •

edited

Loading

SimenB commented Feb 23, 2022 •

edited

Loading

stefanbuck commented Mar 30, 2022 •

edited

Loading