Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core: Introduce Reporting API and STORY_COMPLETED Event #29608

Closed
wants to merge 2 commits into from

Conversation

valentinpalkovic
Copy link
Contributor

@valentinpalkovic valentinpalkovic commented Nov 13, 2024

Closes #

What I did

This PR introduces the Reporting API and a new STORY_COMPLETED event:

Key Changes

  • STORY_COMPLETED Event: Added a new STORY_COMPLETED event to signal when a story has completed all phases, including handling unhandled exceptions. Updated StoryRender to emit this event with relevant status (success or error) and details on unhandled exceptions, allowing for comprehensive reporting at the story level.
  • Reporter API: Introduced a ReporterAPI class that collects reports, including status, version, and violation results, which can be used to track and store accessibility (a11y) or other test results.
    Integrated ReporterAPI within the story context, allowing each story to accumulate and report on its interactions, particularly for test and reporting purposes.
  • Enhanced Story Context in Tests: Added ReporterAPI to composeStory and StoryStore to enable the story context to store reports consistently across different phases of story rendering.
    Updated test utilities to manage the story’s reports metadata, ensuring that each story’s test results can be accessed and verified.

Working on top of this csf canary:
ComponentDriven/csf#110

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

  • stories
  • unit tests
  • integration tests
  • end-to-end tests

Manual testing

This section is mandatory for all contributions. If you believe no manual test is necessary, please state so explicitly. Thanks!

Documentation

  • Add or update documentation reflecting your changes
  • If you are deprecating/removing a feature, make sure to update
    MIGRATION.MD

Checklist for Maintainers

  • When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli-storybook/src/sandbox-templates.ts

  • Make sure this PR contains one of the labels below:

    Available labels
    • bug: Internal changes that fixes incorrect behavior.
    • maintenance: User-facing maintenance tasks.
    • dependencies: Upgrading (sometimes downgrading) dependencies.
    • build: Internal-facing build tooling & test updates. Will not show up in release changelog.
    • cleanup: Minor cleanup style change. Will not show up in release changelog.
    • documentation: Documentation only changes. Will not show up in release changelog.
    • feature request: Introducing a new feature.
    • BREAKING CHANGE: Changes that break compatibility in some way with current major version.
    • other: Changes that don't fit in the above categories.

🦋 Canary release

This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the @storybookjs/core team here.

core team members can create a canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=<PR_NUMBER>

name before after diff z %
createSize 0 B 0 B 0 B - -
generateSize 78.2 MB 78.2 MB 0 B 1.26 0%
initSize 143 MB 143 MB 4.64 kB 1.41 0%
diffSize 65.2 MB 65.2 MB 4.64 kB 0.76 0%
buildSize 6.88 MB 6.88 MB 2.13 kB 32.68 0%
buildSbAddonsSize 1.51 MB 1.51 MB 1.11 kB 264.38 0.1%
buildSbCommonSize 195 kB 195 kB 0 B - 0%
buildSbManagerSize 1.9 MB 1.9 MB 94 B 3.13 0%
buildSbPreviewSize 271 kB 272 kB 931 B Infinity 0.3%
buildStaticSize 0 B 0 B 0 B - -
buildPrebuildSize 3.88 MB 3.88 MB 2.13 kB 32.68 0.1%
buildPreviewSize 3 MB 3 MB 0 B - 0%
testBuildSize 0 B 0 B 0 B - -
testBuildSbAddonsSize 0 B 0 B 0 B - -
testBuildSbCommonSize 0 B 0 B 0 B - -
testBuildSbManagerSize 0 B 0 B 0 B - -
testBuildSbPreviewSize 0 B 0 B 0 B - -
testBuildStaticSize 0 B 0 B 0 B - -
testBuildPrebuildSize 0 B 0 B 0 B - -
testBuildPreviewSize 0 B 0 B 0 B - -
name before after diff z %
createTime 6.7s 6.7s -16ms -1.19 -0.2%
generateTime 20s 19.7s -252ms -0.63 -1.3%
initTime 14.4s 14s -352ms -1.16 -2.5%
buildTime 8.3s 7.9s -444ms -0.8 -5.6%
testBuildTime 0ms 0ms 0ms - -
devPreviewResponsive 4.7s 5.5s 773ms -0.35 14%
devManagerResponsive 3s 3.6s 655ms 0.25 17.9%
devManagerHeaderVisible 496ms 512ms 16ms -0.81 3.1%
devManagerIndexVisible 527ms 549ms 22ms -1.04 4%
devStoryVisibleUncached 1s 994ms -68ms -0.49 -6.8%
devStoryVisible 526ms 548ms 22ms -0.91 4%
devAutodocsVisible 400ms 487ms 87ms -0.35 17.9%
devMDXVisible 436ms 481ms 45ms -0.46 9.4%
buildManagerHeaderVisible 483ms 538ms 55ms -0.41 10.2%
buildManagerIndexVisible 496ms 550ms 54ms -0.46 9.8%
buildStoryVisible 484ms 540ms 56ms -0.36 10.4%
buildAutodocsVisible 380ms 441ms 61ms -0.4 13.8%
buildMDXVisible 370ms 446ms 76ms -0.04 17%

Copy link

nx-cloud bot commented Nov 13, 2024

☁️ Nx Cloud Report

CI is running/has finished running commands for commit d8c2c7e. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution


✅ Successfully ran 2 targets

Sent with 💌 from NxCloud.

@valentinpalkovic valentinpalkovic self-assigned this Nov 14, 2024
@valentinpalkovic valentinpalkovic force-pushed the valentin/implement-reporting-api branch from 06f7d0f to 239c066 Compare November 14, 2024 10:31
@valentinpalkovic valentinpalkovic changed the title Core: Add Reporting API Core: Enhance Storybook Interactions and Reporting API with STORY_COMPLETED Event and Reporter API Integration Nov 14, 2024
@valentinpalkovic valentinpalkovic changed the title Core: Enhance Storybook Interactions and Reporting API with STORY_COMPLETED Event and Reporter API Integration Core: Introduce Reporting API and STORY_COMPLETED Event Nov 14, 2024
status: !ignoreUnhandledErrors && unhandledErrors.size > 0 ? 'error' : 'success',
reporters: context.reporting.reports,
})
);
} catch (err) {
this.phase = 'errored';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where phase errored handled

unhandledExceptions: !ignoreUnhandledErrors
? Array.from(unhandledErrors).map(serializeError)
: [],
status: !ignoreUnhandledErrors && unhandledErrors.size > 0 ? 'error' : 'success',
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

// if one of the status is error -> error
// if no error but at least one warning -> warning
// otherwise -> passed

@valentinpalkovic
Copy link
Contributor Author

Merged into valentin/unified-a11y-testing

@valentinpalkovic
Copy link
Contributor Author

Merged into valentin/unified-a11y-testing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants