Evaluate Cypress for Web deliverables #716
Comments
Folder StructureRecommended default though, this does not take into account a mono-repository. Will determine the proposed structure once get into the code to try this out. Pros (Cypress.io)Cypress.io is a free, open-source front-end testing tool. Cypress.io’s main selling point is its blazing speed. It achieves this speed by running inside the browser. This means that it runs in the same execution loop as your application’s source code (as opposed to Puppeteer and Selenium, which run in a different process than the browser process that renders the application code). This means that Cypress.io doesn’t need to support the control protocols that are required for inter-process communication. This also resolves any system or network latency that slows down sending and receiving commands and responses.
Cons
Resources
User scenariosA good example of how Jest/Cypress can be used together for functional testing. (in this example he's functionally testing the login/logout user experience in real time during development). 15 minutes for that part.I don't actually think we have a need for functional testing currently. But, hopefully this will help explain a good use case for Cypress. https://frontendmasters.com/courses/testing-react/
Spec |
|
Example Implementations |
|
Wiki To install locally $ npm install cypress --save-dev
$ npm install @cypress/webpack-preprocessor --save-devConfigure tsconfig.json to include cypress type "compilerOptions": {
"types": ["cypress"]
}Add new configuration file {
"projectId": "1csr87"
}Create an CYPRESS_RECORD_KEY=<record key>Update package.json scripts "scripts": {
"build:app": "webpack --progress",
"integration-tests": "webpack --output-filename out.js --entry ./cypress/integration/*.ts",
"integration-tests:open": "cypress open",
"integration-tests:run": "node build/run-cypress.js",
}Create a script file to execute both Webpack development servers. One for the App and one for Cypress testing. const spawn = require('child_process').spawn
const webpack = require('webpack');
const webpackConfig = require('../webpack.config');
const WebpackDevServer = require('webpack-dev-server');
const envs = require('dotenv').config();
const cypress = require('cypress');
// Validate environment variables are secure and properly set
if(envs.error) {
if(process.env.CYPRESS_RECORD_KEY.length == 0)
throw envs.error;
else
console.log('Need to add CYPRESS_RECORD_KEY to CI');
}
console.log(envs.parsed);
// Start Webpack development server for the application as required by Cypress
function startServer() {
return new Promise((resolve, reject) => {
new WebpackDevServer(webpack(webpackConfig), {})
.listen(7001, 'localhost', function(err) {
resolve()
}).on('error', (error) => {
reject(error)
})
})
}
var server = startServer();
server.then(function() {
setTimeout(() => {
const cypress = spawn("cypress", ["run", "--record", "--key", process.env.CYPRESS_RECORD_KEY], {stdio: "inherit"});
console.log(cypress.stdout);
}, 10000);
})Open Cypress while Web server is running for manual testing $ npm run integration-tests:open |
|
Ready for review @chrisdholt, @nicholasrice , @janechu |
|
@awentzel Thanks! A few questions:
|
|
|
@awentzel I notice that Cypress stores video files in the repo locally. Is that correct? That would add a ton of weight to the repo itself. Can we store it outside the repo as an alternative? If we want the video functionality and the only way to have it is stored in the repo - I'm not sure we want to include it. |
|
@chrisdholt It does store when it runs locally during development testing run and so would be included in the source. It does delete all of these files at the start of execution so it's doesn't yet do visual regressing diff testing. When running on the CI it would be available and stored on the CI with the history of that job. While these tend to be large files, the way GitHub is architected this is not a huge perf cost when uploading and downloading. The benefits I believe would outweigh the cost. There is a feature to to turn off video feature for local testing and only use that in CI. More details https://docs.cypress.io/guides/guides/screenshots-and-videos.html |
|
@janechu @chrisdholt @nicholasrice last chance to provide thoughts on this Proposal before it's approved. Thanks |
|
Don't we need approvals in order for this to be approved? |
|
Still curious if we need two webpack servers. Could we use one of the preprocessors instead? |
|
@awentzel does Cypress support multiple frameworks (angular, vue, polymer)? More of a question for future proofing. I think a great thing about our unit testing with Jest and Enzyme is that it supports multiple frameworks. If Cypress supports multiple frameworks I think that's a solid in support of it. Were there any alternatives that seemed to compete with Cypress outside of the Applitools investigation? Anything you can provide in terms of trade-offs, etc? |
|
@chrisdholt We're already using the preprocessor. Cypress is 100% JavaScript and no Selenium/WebDriver required. There is a link above for trade-offs. I reviewed the top 10 equivalents. This was the best. During implementation, it can be investigated if 2 webpacks are really required. Cypress performs better than all the other frameworks which is one of its key selling points. |
|
TestCAFE is probably the closest competitor. The documentation, API, and gui isn't quite as good as /cypress. which literally has some of the best documentation I've seen. Very well done project. It's performance is outstanding with clear fairly straightforward implementation that doesn't require a lot of code. It's my recommendation. |
|
I'm a little late jumping on here but I feel like it's a big miss to go with an E2E testing solution that misses the ability to check on safari and firefox. It's more of a E2ChromiumBasedBrowsers. I think we need to go for a solution that doesn't box us into an API but provides more flexibility. There are too many trade-offs for me with missing these browsers. |
|
Ability to support multiple browsers is in the pipeline and was a link I provided a while back to upvote the feature. If you haven't yet done so, you should. This is by far the best framework, most flexible, most robust, and most performant of anything on the market today. It's new, a non-selenium webdriver solution that allows us to test anything in the browser, even the network itself. We can still test other browsers using Applitools during the visual regression testing until then. This is a stepping stone and we should be focused on functional regression testing which we don't have today. This steps us into a future framework that is very well supported, outstanding documentation and is as close as anything gets to E2E. No other framework comes anywhere near this close to E2E. And knowing that E2E is in the works it puts us in a place where we can have a stable API for a while to come. |
|
cypress-io/cypress#310 to upvote cross-browser functionality. |
|
@awentzel so that issue has been open since Nov 2016. If/when they get to it is hazy at best, at the bottom of their issue on the topic it has been deemed not a priority. They have the edge in performance and ease of use, but that does not override what I consider to be a core requirement of actual cross browser testing. Edit: Additionally, this being a Microsoft project, I would argue that it's a must to support IE and Edge for E2E testing |
|
@janechu I actually agree with their assessment. It's most important that functionality is not broken and can be quickly and accurately tested. Cypress core focus is integrated regression testing of functionality with snapshots and videos. Which we don't have today. Applitool's core strength is testing across devices, across screens, and viewports. So if you believe that IS a requirement for FAST-DNA it can be accomplished here instead with ease and host all the images on their end. There is another proposal outstanding for Applitools. Both were intended to compliment one another given there isn't a single service out there today that does great E2E. Furthermore, almost all current and latest releases browsers are moving towards Chromium, Chrome, , Canary, and Electron (headless by default) most choosing Blink. I disagree that this is a Microsoft project. It's open source and thus open for anyone to use. Sure there are teams internal to Microsoft using this. But, there are teams outside Microsoft using as well. |
|
I did discover Chroma which sounds about as great as Cypress. Though it's Selenium based and takes a dependency on Storybook. We investigated Storybook and wouldn't fit into our architecture or philosophy for testing components. That said, i'm discussing with Chroma a partnership where they might build an SDK for us so we don't have to take a dependency on Storybook. Waiting to hear back from Chroma as the sales team is meeting with the engineering team to discuss further details. |
|
Recent email exchanges with Cypress have let to a discussion they that are working on an engine for FireFox and IE. I requested Edge as well. Chroma has updated an API for us to use their platform without the dependency on Chroma. So I'll be investigating Chroma against Cypress.io. Both are JavaScript driven not Selenium. |
Cypress provides a comprehensive testing suite with CLI/App/Dashboard that allows testing functional UI components. The thought here is that with this we can wire up functional testing with ease easy to maintain and review issues with the team from the portal. Applitools, may also still be used on top to test Visual UI scenarios or sites.
Deliverables:
Include a remote branch with example implementation for packages that have a UI/UX.
The text was updated successfully, but these errors were encountered: