From ff53d3e057d3fa0a7cb4f5cdf76c0f9dc741c84c Mon Sep 17 00:00:00 2001
From: Brian Mann <brian@cypress.io>
Date: Fri, 27 Oct 2017 22:00:21 -0400
Subject: [PATCH] Issue 176 (#181)

* WIP start on conditional testing core concept

* move around core concepts and guides

* WIP more conditional testing content

* fixes #176, beat conditional testing to death

* some content cleanups
---
 source/_data/sidebar.yml                      |   7 +-
 .../core-concepts/conditional-testing.md      | 446 ++++++++++++++++++
 .../core-concepts/introduction-to-cypress.md  |   8 +-
 .../launching-browsers.md                     |   0
 .../screenshots-and-videos.md                 |   0
 themes/cypress/languages/en.yml               |   1 +
 6 files changed, 454 insertions(+), 8 deletions(-)
 create mode 100644 source/guides/core-concepts/conditional-testing.md
 rename source/guides/{core-concepts => guides}/launching-browsers.md (100%)
 rename source/guides/{core-concepts => guides}/screenshots-and-videos.md (100%)

diff --git a/source/_data/sidebar.yml b/source/_data/sidebar.yml
index 5b0b307ffb..e2caa32bf3 100644
--- a/source/_data/sidebar.yml
+++ b/source/_data/sidebar.yml
@@ -10,11 +10,10 @@ guides:
   core-concepts:
     introduction-to-cypress: introduction-to-cypress.html
     writing-and-organizing-tests: writing-and-organizing-tests.html
-    overview-of-the-gui: overview-of-the-gui.html
     interacting-with-elements: interacting-with-elements.html
+    overview-of-the-gui: overview-of-the-gui.html
     aliases-and-references: aliases-and-references.html
-    screenshots-and-videos: screenshots-and-videos.html
-    launching-browsers: launching-browsers.html
+    conditional-testing: conditional-testing.html
   guides:
     command-line: command-line.html
     debugging: debugging.html
@@ -22,6 +21,8 @@ guides:
     continuous-integration: continuous-integration.html
     environment-variables: environment-variables.html
     stubs-spies-and-clocks: stubs-spies-and-clocks.html
+    screenshots-and-videos: screenshots-and-videos.html
+    launching-browsers: launching-browsers.html
     web-security: web-security.html
     reporters: reporters.html
   # advanced-cypress:
diff --git a/source/guides/core-concepts/conditional-testing.md b/source/guides/core-concepts/conditional-testing.md
new file mode 100644
index 0000000000..f7cdee9ba3
--- /dev/null
+++ b/source/guides/core-concepts/conditional-testing.md
@@ -0,0 +1,446 @@
+---
+title: Conditional Testing
+comments: false
+---
+
+Conditional testing refers to the common programming pattern:
+
+> If X, then Y, else Z
+
+Many of our users ask how to accomplish this seemingly simple idiom in Cypress.
+
+Here are some example use cases:
+
+- How do I do something different whether an element does or doesn't exist?
+- My application does A/B testing, how do I account for that?
+- My users receive a "welcome wizard", but existing ones don't. Can I always close the wizard in case it's shown, and ignore it when it's not?
+- Can I recover from failed Cypress commands like if a `cy.get()` doesn't find an element?
+- I'm trying to write dynamic tests that do something different based on text on the page.
+- I want to automatically find all `<a>` elements, and based on which ones I found, I want to check that each link works.
+
+The problem is - while first appearing simple, writing tests in this fashion often leads to flaky tests, random failures, and difficult to track down edge cases.
+
+Let's investigate why and how you can overcome these problems...
+
+# The Problem
+
+These days modern JS applications are highly dynamic and mutable. Their state, and the DOM are continuously changing over a period of time.
+
+The problem with **conditional testing** is that it can only be used when the state has stabilized. In modern day applications, knowing when state is stable is oftentimes impossible.
+
+To a human - if something changes 10ms, or 100ms from now, we may not even notice this change and assume the state was always the same.
+
+To a robot - even 10ms represents billions+ of clock cycles. The timescale difference is incredible.
+
+A human also has intuition. If you click a button and see a loading spinner, you'll assume the state is in flux and will automatically wait for it to finish.
+
+To a robot, they have no intuition - they'll do exactly as they're programmed to do.
+
+To illustrate this, let's take a very simple example of trying to conditionally test unstable state.
+
+## The DOM is unstable
+
+```js
+// your app code
+
+// random amount of time
+const random = Math.random() * 100
+
+// create a <button> element
+const btn = document.createElement('button')
+
+// attach it to the body
+document.body.appendChild(btn)
+
+setTimeout(() => {
+  // add the class active after an indeterminate amount of time
+  btn.setAttribute('class', 'active')
+}, random)
+```
+
+```js
+// your cypress test code
+it('does something different based on the class of the button', function () {
+  // RERUN THIS TEST OVER AND OVER AGAIN
+  // AND IT WILL SOMETIMES BE TRUE, AND
+  // SOMETIMES BE FALSE.
+
+  cy.get('button').then(($btn) => {
+    if ($btn.hasClass('active')) {
+      // do something if its active
+    } else {
+      // do something else
+    }
+  })
+})
+```
+
+Do you see the problem here? This test is non-deterministic. The `<button>` will sometimes have the class `active` and sometimes not. In **most** cases, you cannot rely on the state of the DOM to determine what you should conditionally do.
+
+This is the heart of flakiness and bad tests. At Cypress we've designed our API's to combat this type of flakiness at every step.
+
+# The Situations
+
+The **only** way to do conditional testing on the DOM is if you are 100% the state has "settled" and there is no possible way for it to change.
+
+This is it! In any other circumstance you will have flaky tests if you try to rely on the state of the DOM for conditional testing.
+
+Let's explore a few examples.
+
+## Server side rendering
+
+If your application is server side rendered without JavaScript that asynchronously modifies the DOM - congratulations, you can easily do conditional testing on the DOM!
+
+Why? Because if the DOM is not going to change after the `load` event occurs, then it can accurate represent a stable state of truth.
+
+You can safely skip down to the bottom where we provide examples of conditional testing.
+
+## Client side rendering
+
+However, in most modern applications these days - when the `load` event occurs, usually nothing has rendered on the screen. It's usually at this moment your scripts begin to load dynamic content, and then begin to render asynchronously.
+
+Unfortunately it is not possible for you to use the DOM to do conditional testing. To do this would require you to know with 100% guarantee that your application has finished all asynchronous rendering, and there are no pending network requests, setTimeouts, intervals, or async/await code.
+
+This is difficult to do (if not impossible) without making changes to your application. You could use a library like `zone.js`, but even that does not capture every async possibility.
+
+In other words, you cannot do conditionally testing safely if you want your tests to run 100% consistently.
+
+But do not fret - there are better workarounds to still achieve conditional testing **without** relying on the DOM. You just have to *anchor* yourself to another piece of truth which is not mutable.
+
+# The Strategies
+
+If you're unable to guarantee that the DOM is stable - don't worry, there are other ways you can do conditional testing or work around the problems inherent with it.
+
+You could:
+
+- Remove the need to ever do conditional testing
+- Force your application to behave deterministically
+- Check other sources of truth (like your server or database)
+- Embed data into other places (cookies / local storage) you could read off
+- Add data to the DOM that you can read off to know how to proceed
+
+Let's explore some examples of conditional testing that will pass or fail 100% of the time.
+
+## A/B Campaign
+
+In this example let's assume you visit your website and the content will be different based on which A/B campaign your server decides to send. Perhaps it's based on geo-location, IP Address, time of day, locale, or other factors that are difficult to control. How can you write tests in this manner?
+
+Easily: control which campaign gets sent, or provide a reliable means to know which one it is.
+
+***Use URL query params:***
+
+```js
+// tell your backend server which campaign you want sent
+// so you can deterministically know what it is ahead of time
+cy.visit('https://app.com?campaign=A')
+
+...
+
+cy.visit('https://app.com?campaign=B')
+
+...
+
+cy.visit('https://app.com?campaign=C')
+```
+
+Now there's not even a need to conditional testing since you're able to know ahead of time what the campaign is. Yes, this may require server side updates, but you have to make an untestable app testable if you want to test it!
+
+***Use the server:***
+
+Alternatively, if your server saves the campaign with a session - you could just ask your server to tell you which campaign you're on.
+
+```js
+cy
+  // this sends us the session cookies
+  .visit('https://app.com')
+
+  // assuming this sends us back
+  // the campaign information
+  .request('https://app.com/me')
+  .its('body.campaign')
+  .then((campaign) => {
+    // runs different cypress test code
+    // based on the type of campaign
+    return campaigns.test(campaign)
+  })
+```
+
+***Use Session Cookies:***
+
+Perhaps an even easier way to test this is if your server sent the campaign in a session cookie that you could read off.
+
+```js
+cy
+  .visit('https://app.com')
+  .getCookie('campaign')
+  .then((campaign) => {
+    return campaigns.test(campaign)
+  })
+```
+
+***Embed Data in the DOM:***
+
+Another valid strategy would be to embed data directly into the DOM - but do so in a way where this data is **always** present and queryable. It would have to be present 100% of the time, else this would not work.
+
+```js
+cy.get('html').should('have.attr', 'data-campaign').then((campaign) => {
+  return campaigns.test(campaign)
+})
+```
+
+## Welcome Wizard
+
+In this example, let's imagine you're running a bunch of tests, and each time you load your application, it may show a "Welcome Wizard" modal.
+
+In this situation, you want to close the Wizard when it's present, and ignore it if it's not.
+
+The problem with this - is that if the Wizard renders asynchronously (because it likely does) you cannot use the DOM to conditionally dismiss it.
+
+Once again - we'll need another reliable way to achieve this without involving the DOM.
+
+These patterns are pretty much the same as before:
+
+***Use the URL to control it:****
+
+```js
+// dont show the wizard
+cy.visit('https://app.com?wizard=0')
+```
+
+```js
+// show the wizard
+cy.visit('https://app.com?wizard=1')
+```
+
+We'd likely just need to update our client side code to check whether this query param is present. Now we know ahead of time whether it will or will not be shown.
+
+***Use Cookies to know ahead of time:***
+
+In the case where you cannot control it, you can still conditionally dismiss it **if** you know whether it's going to be shown.
+
+```js
+cy
+  .visit('https://app.com')
+  .getCookie('showWizard')
+  .then((val) => {
+    if (val) {
+      // dismiss the wizard conditionally by enqueuing these
+      // three additional commands
+      cy.get('#wizard').contains('Close').click()
+    }
+  })
+  .foo() // more commands here
+  .bar() // more commands here
+  .baz() // more commands here
+
+```
+
+***Use your server or database:***
+
+If you store and/or persist whether to show the wizard on the server, then just ask it.
+
+```js
+cy
+  .visit('https://app.com')
+  .request('https://app.com/me')
+  .its('body.showWizard')
+  .then((val) => {
+    if (val) {
+      // dismiss the wizard conditionally by enqueuing these
+      // three additional commands
+      cy.get('#wizard').contains('Close').click()
+    }
+  })
+  .foo() // more commands here
+  .bar() // more commands here
+  .baz() // more commands here
+
+```
+
+Alternatively, if you're creating users it might just be easier to create the user and simply set whether you want the wizard shown ahead of time then. That would avoid this check later.
+
+***Embed data in the DOM:***
+
+Another valid strategy would be to embed data directly into the DOM - but do so in a way where this data is **always** present and queryable. It would have to be present 100% of the time, else this would not work.
+
+```js
+cy.get('html').should('have.attr', 'data-wizard').then((wizard) => {
+  if (wizard) {
+    // dismiss the wizard conditionally by enqueuing these
+    // three additional commands
+    cy.get('#wizard').contains('Close').click()
+  }
+})
+.foo() // more commands here
+.bar() // more commands here
+.baz() // more commands here
+```
+
+## Element Existence
+
+In the case where you **are** trying to use the DOM to do conditional testing, you can utilize the ability to synchronously query for elements in Cypress to create control flow.
+
+{% note warning %}
+In the event you did not read a word above and skipped down here we'll reiterate it one more time:
+
+You cannot do conditional testing on the DOM unless you are either:
+
+- Server side rendering with no asynchronous JavaScript
+- Using client side JavaScrip that **only** ever does synchronous rendering
+
+It's crucial you understand how your application works else you will write flaky tests.
+{% endnote %}
+
+Let's imagine we have a scenario where our application may do two separate things that we're unable to control. In other words you tried every strategy above and for whatever reason you are unable to know ahead of time what you application will do.
+
+Testing this in Cypress is possible.
+
+```js
+// app code
+
+$('button').on('click', function(e) {
+  // do something synchronously randomly
+  if (Math.random() < .5) {
+    // append an input
+    $('<input />').appendTo($('body'))
+  } else {
+    // or append a textarea
+    $('<textarea />').appendTo($('body'))
+  }
+})
+```
+
+```js
+cy
+  // click the button causing the new
+  // elements to appear
+  .get('button').click()
+  .get('body').then(($body) => {
+    // synchronously query from body
+    // to find which element was created
+    if ($body.find('input')) {
+      // input was found, do something else here
+      return 'input'
+    }
+
+    // else assume it was textarea
+    return 'textarea'
+  })
+  .then((selector) => {
+    // selector is a string that represents
+    // the selector we could use to find it
+    cy.get(selector).type(`found the element by selector ${selector}`)
+  })
+```
+
+We'll reiterate one more time. Had the `<input>` or the `<textarea>` been rendered asynchronously, you could not use the pattern above. You would have to involve arbitrary delays which will not work in every situation, slow down your tests, still leave chances that your tests are flaky, and are an all around anti pattern.
+
+Cypress is built around creating **reliable tests**. The secret to writing good tests is to provide as much "state" and "facts" to Cypress and to "guard it" from issuing new commands until your application has reached the desired state it needs to proceed.
+
+Doing conditional testing adds a huge problem - that the test writer themselves are unsure what the given state will be. In those situations, the only reliable way to have accurate tests is to embed this dynamic state in a reliable and consistent way.
+
+If you are not sure if you've written a potentially flaky test, there is an easy way to figure it out. Repeat the test an excessive number of times, and then repeat by modifying the Dev Tools to throttle the Network and the CPU. This will create different loads that simulate different environments (like CI). If you've written a good test, it will pass or fail 100% of the time.
+
+```js
+Cypress._.times(100, (i) => {
+  it(`num ${i+1} - test the thing conditionally`, function () {
+    // do the conditional bits 100 times
+  })
+})
+```
+
+## Dynamic Text
+
+The pattern of doing something conditionally based on whether or not certain text is present is identical to element existence above.
+
+***Conditionally check whether an element has certain text:***
+
+```js
+cy
+  // this only works if there's 100% guarantee
+  // body has fully rendered without any pending changes
+  // to its state
+  .get('body').then(($body) => {
+    // synchronously ask for the body's text
+    // and do something based on whether it includes
+    // another string
+    if ($body.text().includes('some string')) {
+      // yup found it
+      cy.foo().bar()
+    } else {
+      // nope not here
+      cy.baz().quux()
+    }
+  })
+```
+
+# Error Recovery
+
+Many of our users ask how they can recover from failed commands.
+
+> If I had error handling, I could just try to find X and if X fails go find Y
+
+Because error handling is a common idiom in most programming languages, and especially in Node.js - it seems reasonable to expect to do that in Cypress.
+
+However, this is really the same question as asking to do conditional testing just wrapped up in a slightly different implementation detail.
+
+For instance you may want to do this:
+
+```js
+cy
+  .get('button').contains('hello')
+  .catch((err) => {
+    // oh no the button wasn't found
+    // (or something else failed)
+    cy.get('somethingElse').click()
+  })
+```
+
+If you've been reading along, then you should already have a grasp on why trying to implement conditional code with asynchronous rendering is impossible. If the test writer cannot accurately predict the given state of the system, then neither can Cypress. Error handling offers no additional proof this can be done deterministically.
+
+You should think of failed commands in Cypress akin to uncaught exceptions in server side code. It's not possible to try to recover in those scenarios because the system has transitioned to an unreliable state. Instead you generally always opt to crash and log. When Cypress fails the test - that's exactly what it's doing. Bailing out, skipping any remaining commands in the test, and logging out the failure.
+
+But... for the sake of the argument, let's imagine for a moment you did have error handling in Cypress.
+
+Enabling this would mean that for every single command, it would recover from errors, but only after each applicable command timeout was reached. Since timeouts start at 4 seconds (and exceed from there), this means that it would only fail after a long, long time.
+
+Let's reimagine our "Welcome Wizard" example from before.
+
+```js
+// great error recovery code
+function keepCalmAndCarryOn () {
+  cy.foo().bar().baz()
+}
+
+cy
+  .get('#wizard').contains('Close').click()
+  .catch((err) => {
+    // no problem, i guess the wizard didn't exist
+    // or something... no worries
+    keepCalmAndCarryOn()
+  })
+  .then(keepCalmAndCarryOn)
+
+```
+
+In the **best** case scenario, we've wasted at LEAST 4 seconds waiting on the `<#wizard>` element to possibly exist before we errored and continued on.
+
+But in the **worst** case scenario we have a situation where the `<#wizard>` **was** going to be rendered, but it didn't render within our given timeout. Let's assume this was due to a pending network request or websocket message or a queued timer, or anything else.
+
+In this situation, not only did we wait a long period of time, but when the `<#wizard>` element was eventually shown it's likely caused an error downstream on other commands.
+
+If you cannot accurately know the state of your application then no matter what programming idioms you have available - **you cannot write 100% deterministic tests**.
+
+Still not convinced?
+
+Not only is this an anti pattern, but its an actual logical fallacy.
+
+You may think to yourself... okay fine, but 4 seconds - man that's not enough. Network requests could be slow, let's bump it up to 1 minute!
+
+Even then, it's still possible a websocket message could come in... so 5 minutes!
+
+Even then, not enough, its possible a setTimeout could trigger... 60 minutes.
+
+Continually raising the timeout only beleaguers the point. As you approach infinity your confidence does continue to rise on the chances you could prove the desired state will be reached, but you can never prove it will. Instead you could theoretically be waiting for the heat death of the universe for a condition to come that's only a moment away from happening. There is no way to prove or disprove that it *may* conditionally happen.
+
+You, the test writer must know ahead of time what your application is programmed to do - or have an 100% confidence the state of a mutable object (like the DOM) has stabilized stable in order to write accurate conditional tests.
diff --git a/source/guides/core-concepts/introduction-to-cypress.md b/source/guides/core-concepts/introduction-to-cypress.md
index dc9b4e7cb3..75e1f444c0 100644
--- a/source/guides/core-concepts/introduction-to-cypress.md
+++ b/source/guides/core-concepts/introduction-to-cypress.md
@@ -542,17 +542,15 @@ In Cypress there is no built in error recovery from a failed command. A command
 
 You might be wondering:
 
-> How do I create control flow, using if/else? So that if an element does (or doesn't) exist, I choose what to do?
+> How do I create conditional control flow, using if/else? So that if an element does (or doesn't) exist, I choose what to do?
 
-The problem with this question is that this type of control flow ends up being non-deterministic. This means its impossible for a script (or robot), to follow it 100% consistently.
+The problem with this question is that this type of conditional control flow ends up being non-deterministic. This means its impossible for a script (or robot), to follow it 100% consistently.
 
 In general, there are only a handful of very specific situations where you *can* create control flow. Asking to recover from errors is actually just asking for another `if/else` control flow.
 
 With that said, as long as you are aware of the potential pit falls with control flow, it is possible to do this in Cypress!
 
-{% note info %}
-We're working on a new guide for showing you how to manage and create "control flow" in your tests.
-{% endnote %}
+You can read all about how to do {% url 'conditional testing' conditional-testing %} here.
 
 # Assertions
 
diff --git a/source/guides/core-concepts/launching-browsers.md b/source/guides/guides/launching-browsers.md
similarity index 100%
rename from source/guides/core-concepts/launching-browsers.md
rename to source/guides/guides/launching-browsers.md
diff --git a/source/guides/core-concepts/screenshots-and-videos.md b/source/guides/guides/screenshots-and-videos.md
similarity index 100%
rename from source/guides/core-concepts/screenshots-and-videos.md
rename to source/guides/guides/screenshots-and-videos.md
diff --git a/themes/cypress/languages/en.yml b/themes/cypress/languages/en.yml
index e08c38b582..34f9bbbf26 100644
--- a/themes/cypress/languages/en.yml
+++ b/themes/cypress/languages/en.yml
@@ -32,6 +32,7 @@ sidebar:
     introduction-to-cypress: Introduction to Cypress
     writing-and-organizing-tests: Writing and Organizing Tests
     overview-of-the-gui: Overview of the GUI
+    conditional-testing: Conditional Testing
     debugging: Debugging
     aliases-and-references: Aliases and References
     stubs-spies-and-clocks: Stubs, Spies, and Clocks