Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide a way for custom commands to retry on failed should() #3109

Closed
alexkrolick opened this issue Jan 10, 2019 · 24 comments
Closed

Provide a way for custom commands to retry on failed should() #3109

alexkrolick opened this issue Jan 10, 2019 · 24 comments
Assignees
Labels
pkg/driver This is due to an issue in the packages/driver directory topic: plugins ⚙️ type: enhancement Requested enhancement of existing feature

Comments

@alexkrolick
Copy link

alexkrolick commented Jan 10, 2019

Current behavior:

cy.get('#me').should('not.exist') // retries chain
cy.customGetById('me').should('not.exist') // does not retry

#1210 (comment)

Desired behavior:

cy.customGetById('me').should('not.exist') // retries custom get if "should" fails

Versions

Cypress 3.1.4

Related

@kuceb
Copy link
Contributor

kuceb commented Jan 25, 2019

@alexkrolick can you provide the code for customGetById? I'm not able reproduce this issue

@kuceb kuceb added the stage: needs information Not enough info to reproduce the issue label Jan 25, 2019
@alexkrolick
Copy link
Author

See testing-library/cypress-testing-library#30 - there are a number of custom commands

@kuceb
Copy link
Contributor

kuceb commented Jan 25, 2019

@alexkrolick I see, you're asking for plugin authors to have access to the retry mechanism, custom commands using cy.get under the hood will still work fine

@kuceb kuceb added topic: plugins ⚙️ and removed stage: needs information Not enough info to reproduce the issue labels Jan 25, 2019
@alexkrolick
Copy link
Author

Yes exactly

@lukeapage
Copy link
Contributor

lukeapage commented Jan 27, 2019

This workaround works:

cy.getByTestId = (id) => cy.get(`[data-test-id="${id}"]`);

@alexkrolick
Copy link
Author

The custom queries we are trying to support aren't wrappers around jQuery attribute selectors; some of them use multiple DOM traversals to match labels to inputs, for example. cy.get isn't a workaround unfortunately. See linked issues.

@jennifer-shehane jennifer-shehane added stage: proposal 💡 No work has been done of this issue type: enhancement Requested enhancement of existing feature pkg/driver This is due to an issue in the packages/driver directory labels Jan 30, 2019
@Lakitna
Copy link
Contributor

Lakitna commented Jan 30, 2019

I've been looking at this for a bit now and digging into the source code gave me the command verifyUpcomingAssertions. That's how you can wait for a should to be resolved.

However, this command is not documented and can, therefore, be a bit iffy to work with at the moment.

The way I was able to figure out how to use it was again by looking to the source of the default commands found here: https://github.com/cypress-io/cypress/tree/develop/packages/driver/src/cy/commands

When you get it working it works great, but expect to spend a lot of time tinkering. It helps to use the same basic structure that's used in default commands.

I'm at a point where I got it working perfectly except for the log.end() of the first upcoming assertion, the steps remains 'pending' (blue with spinner) when it should be 'passed' (green).

edit: I got it working now! The basic format should look something like the following. I left out logging, options etc for clarity.

Cypress.Commands.add('aThing', (element, options={}) => {
    /**
     * This function is recursively called untill the timeout passes or the upcomming
     * assertion passes. Keep this function as fast as possible.
     *
     * @return {Promise}
     */
    function resolveAThing() {
        // Resolve a thing
        const aThing = $(element).attr('aThing');

        // Test the upcomming assertion where aThing is the value used to assert.
        return cy.verifyUpcomingAssertions(aThing, options, {
            // When the upcoming assertion failes first onFail is called
            // onFail: () => {},
            // When onFail resolves onRetry is called
            onRetry: resolveAThing,
        });
    }

    return resolveAThing();
});

@kuceb
Copy link
Contributor

kuceb commented Feb 28, 2019

You can make an arbitrary function retry and pass along its return value using a small hack that combines should and invoke

Here's an example of a custom command that makes sure all h2's are in alphabetical order, and it retries:

cy.get('h2')
.should(($els) => $els.fn = () => {
  return $els.toArray().map(el => el.innerText)
})
.invoke('fn')
.should(headers => {
  const sortedHeaders = headers.concat().sort()
  expect(headers).deep.eq(sortedHeaders)
})

or

better yet, add a new command that does this called try:

Cypress.Commands.addAll({
  prevSubject: 'optional',
},
{
  try: (subject, fn) => {
    return cy.wrap({ try: () => fn(subject) }, { log: false })
    .invoke('try')

  },
})

and use it like:

cy.get('h2').try(($els) => {
  return $els.toArray().map(el => el.innerText)
})
.should(headers => {
  const sortedHeaders = headers.concat().sort()

  expect(headers).deep.eq(sortedHeaders)
})

without a cypress parent command:

cy.try(() => {
  const h2s = cy.state('document').querySelectorAll('h2')
  return Array.from(h2s).map(el => el.innerText)
}).should(headers => {
  const sortedHeaders = headers.concat().sort()
  expect(headers).deep.eq(sortedHeaders)
})

@Lakitna
Copy link
Contributor

Lakitna commented Mar 1, 2019

It's a nice idea to add a command that is basically then + retry. I've run into situation where it would be usefull. Though I don't think try is the best name. It implies error handling and that you need a catch to go with it.

Couple of ideas:

Name it retry

cy.get('foo')
    .retry((elem) => {
        // ...
    });

Name it thenTry

cy.get('foo')
    .thenTry((elem) => {
        // ...
    });

Overwrite then
When you overwrite then you can add an option to enable retrying and make it false by default, something like below. Though try might be one of the few commands that can't be overwritten, I'm not sure.

cy.get('foo')
    .then((elem) => {
        // ...
    }, {retry: true});

@bohendo
Copy link

bohendo commented Mar 26, 2019

What's the status on this? I've got a two-part cy.get that I'd like to retry until it resolves to a non-zero value. My first attempt:

const getBalance = () => {
  return cy.wrap(new Cypress.Promise((resolve, reject) => {
    cy.get('h1').children('span').invoke('text').then(whole => {
      cy.get('h3').children('span').invoke('text').then(fraction => {
        cy.log(`Got balance: ${whole}${fraction}`)
        resolve(`${whole}${fraction}`)
      })
    })
  }))
}

describe('Test', () => {
  it(`Should wait until balance is non-zero`, () => {
    // cy.wait(2000)
    getBalance().should('not.equal', '0.00')
  })
})

The above runs getBalance() once and then waits on the return value to pass the assertion. It fails via timeout even though my UI eventually has a non-zero balance because getBalance() isn't rerun when it's return value fails the assertion.

I can get the above test to pass by uncommenting the cy.wait(2000) but I'd rather not introduce any potential race conditions.

My second attempt, inspired by the conversation between @Lakitna and @bkucera above.

Cypress.Commands.addAll({ prevSubject: 'optional' }, {
  retry: (subject, fn) => {
    return cy.wrap({ retry: () => fn(subject) }, { log: false }).invoke('retry')
  },
})

describe('Test', () => {
  cy.retry(getBalance).should('not.equal', '0.00')
})

The above errors out with CypressError: Cypress detected that you returned a promise from a command while also invoking one or more cy commands in that promise. and I've tinkered with it for a while but can't figure out how to avoid this error while still testing what I want to test..

@Lakitna
Copy link
Contributor

Lakitna commented Mar 26, 2019

There is no official stance on this yet. But that won't stop us!

I've made a quick implementation of this idea using the retry name and tried to document it a bit so you can alter it. I only tested if it retries, I have no idea what its limitations are.

const _ = Cypress._;
const $ = Cypress.$;

/**
 * Basically `then`, but will retry if any upcoming assertion fails
 * @param {JQuery} subject
 * @param {function} fn
 * @param {boolean} [options.log=true]
 * Log to Cypress bar
 *
 * @return {*}
 */
Cypress.Commands.add('retry', {prevSubject: 'optional'}, (subject, fn, options={}) => {
    _.defaults(options, {
        log: true,
    });

    // Setup logging
    const consoleProps = {
        'Applied to': $(subject),
    };
    if (options.log) {
        options._log = Cypress.log({
            $el: $(subject),
            name: 'retry',
            message: fn.name,
            consoleProps: () => {
                return consoleProps;
            },
        });
    }

    /**
     * This function is recursively called untill timeout or the upcomming
     * assertion passes. Keep this function as fast as possible.
     *
     * @return {Promise}
     */
    function resolve() {
        const result = fn(subject);

        // Update logging
        if (options.log) {
            consoleProps.Yielded = result;
        }

        // Test the upcomming assertion where result is the value used to assert.
        return cy.verifyUpcomingAssertions(result, options, {
            // When the upcoming assertion failes first onFail is called
            // onFail: () => {},
            // When onFail resolves onRetry is called
            onRetry: resolve,
        });
    }

    return resolve();
});

The full extend of my tests:

it('retries', function() {
    let c = 0;
    cy.retry(() => ++c)
        .should('equal', 5);
});

@bohendo
Copy link

bohendo commented Mar 27, 2019

Thank you @Lakitna 🙏 An(other) example using cy.verifyUpcomingAssertions was just what I needed. I think cy.retry is a reserved command though (at least in v3.2.0) because if I try to add it, unrelated parts of my code start blowing up.

But I tinkered a bit, got it working & was able to remove the race condition (ie a cy.wait) I was relying on earlier. Hope the following example is useful for others who are in a similar situation:

// cy.resolve(fn).should(blah) will re-run the promise-returning fn until
// the value it resolves to passes the assertion
Cypress.Commands.add('resolve', { prevSubject: 'optional' }, (subject, fn, opts={}) => {
  const resolve = () => {
    fn(subject).then(res => cy.verifyUpcomingAssertions(res, opts, { onRetry: resolve }))
  }
  return resolve();
});

// an example function that returns a Cypress.Promise
const getBalance = () => {
  return cy.wrap(new Cypress.Promise((resolve, reject) => {
    cy.get('h1').children('span').invoke('text').then(whole => {
      cy.get('h3').children('span').invoke('text').then(fraction => {
        cy.log(`Got balance: ${whole}${fraction}`)
        resolve(`${whole}${fraction}`)
      })
    })
  }))
}

describe('Test', () => {
  it(`Should wait until balance is non-zero`, () => {
    cy.resolve(getBalance).should('not.contain', '0.00')
  })
})

@Lakitna
Copy link
Contributor

Lakitna commented Mar 27, 2019

You are correct, retry is an existing, undocumented cy command (see screenshot).

image

I would personally fall back to thentry until I made a way to overwrite then with a retry option.

@Lakitna
Copy link
Contributor

Lakitna commented Mar 28, 2019

Behold, the birth of the module cypress-commands!

I've made the extension on the then command and published it in a repo (and on npm) where I will add more commands in the future. I have a few commands laying on the shelf I could add.

For more details see the repo at https://github.com/Lakitna/cypress-commands

@jennifer-shehane
Copy link
Member

Hey @Lakitna, can you add your commands library to our docs on the plugins page?

@Manuel-Suarez-Abascal
Copy link
Contributor

Hi, I was using the @Lakitna example to implement my own retry mechanism to assert three different modals on our app. I'm using the verifyUpcomingAssertion method, but it seems it retries right away & I can't make it to wait until the assertions fail or pass. This is my code:

Cypress.Commands.add(
  'assertModal',
  ({ selector = '[data-test=popup-container]', headerTitle, shouldClose = true } = {}) => {
    const assertADSModal = () => {
      cy.get('[data-test=acmodal]')
      // a bunch of assertions here
      .........
    };

    const assertDefaultModal = () => {
      cy.get(selector)
       // a bunch of assertions here
      .........
    };

    const assertLegacyModal = () => {
      cy.get('.ui-dialog')
      // a bunch of assertions here
      ......... 
    };

    const triggerModalAssertions = () => {
      cy.get('body').then($body => {
        const isAnyDefaultModalVisible = $body.find(selector).length > 0;
        const isAnyAdsModalVisible = $body.find('[data-test=acmodal]').length > 0;
        const isAnyLegacyModalVisible = $body.find('.ui-dialog').length > 0;

        if (isAnyAdsModalVisible) {
          assertADSModal();
        }
        if (isAnyDefaultModalVisible) {
          assertDefaultModal();
        }
        if (isAnyLegacyModalVisible || selector === '.ui-dialog') {
          assertLegacyModal();
        }

        return cy.verifyUpcomingAssertions(
          $body,
          {},
          {
            onRetry: triggerModalAssertions,
          }
        );
      });
    };

    return triggerModalAssertions();
  }
);

How can I make the verifyUpcomingAssertions to wait until the assertion pass or fail until retries? I have tried with setTimeout & specify the amount of time, but it didn't work.

@pathurs
Copy link

pathurs commented Oct 2, 2020

For this problem, I ended up creating my own implementation of "retry-ability" (in TypeScript):

interface RetryOptions<T> extends Cypress.Timeoutable {
    interval: number;
    default?: T;
    throw: boolean;
}

export function retry<T>(checker: () => T, confirmer: (result: T) => boolean, options?: Partial<RetryOptions<T>>): Cypress.Chainable<T>;
export function retry<T>(checker: () => T, confirmer: (result: T) => boolean, originalOptions?: Partial<RetryOptions<T>>): Cypress.Chainable<T> {
    const options: RetryOptions<T> = {
        ...{ timeout: Cypress.config('defaultCommandTimeout'), interval: 200, throw: true },
        ...originalOptions
    };

    return cy.wrap(
        new Promise<T>((resolve, reject) => {
            const startTime = Date.now();
            const result = checker();

            if (confirmer(result)) {
                resolve(result);

                return;
            }

            const intervalId = setInterval(() => {
                const currentTime = Date.now();
                const endTime = startTime + options.timeout;

                if (currentTime >= endTime) {
                    if (options.throw) {
                        reject(new Error(`Timed out while retrying after ${options.timeout}ms`));
                    } else if ('default' in options) {
                        resolve(options.default);
                    } else {
                        resolve();
                    }

                    clearInterval(intervalId);

                    return;
                }

                const result = checker();

                if (confirmer(result)) {
                    resolve(result);

                    clearInterval(intervalId);

                    return;
                }
            }, options.interval);
        }),
        { log: false, timeout: options.timeout + options.interval }
    );
}

Usage Example:

retry(
    () => {
        const selector = 'your selector';

        return Cypress.$(selector);
    },
    elements => elements.length > 0,
    { throw: false, default: Cypress.$() }
);

@yktoo
Copy link

yktoo commented Jan 22, 2021

Here's my naive implementation (Typescript, but can easily be converted to Javascript):

const retryableResolver = (getter: (...x: any) => any, ...args) => {
    const end = Date.now() + config('defaultCommandTimeout');
    const resolve = () => Cypress.Promise
        .try(() => getter(...args))
        .then(value => Date.now() > end ? value : cy.verifyUpcomingAssertions(value, {}, {onRetry: resolve}));
    return resolve();
};

And then use it when adding a command, here's an example that collects texts of passed subject elements and returns them as a string array:

const texts = elements => elements.map((i, e) => $(e).text()).get();
Cypress.Commands.add(
    'texts',
    {prevSubject: 'element'},
    elements => retryableResolver(texts, elements));

@DanaGoyette
Copy link

Does anyone have a nice way to effectively replace cy.get with simpler logging, while keeping the element highlighting and all that? I want something like the following, but I want the logging behavior to be synchronous the same way as the real cy.get:

export function getByTestId<E extends HTMLElement>(
  testId: string,
  options?: Partial<Cypress.Loggable & Cypress.Timeoutable & Cypress.Withinable & Cypress.Shadow>
): Cypress.Chainable<JQuery<E>> {
  const props = {
    Subject: null as JQuery<E>,
    Yielded: null as unknown,
  }
  const logConf: Partial<Cypress.LogConfig> = {
    $el: undefined,
    name: 'getByTestId',
    displayName: 'get(testid)',
    message: testId,
    consoleProps: () => props,
  }
  const log = Cypress.log(logConf)
  return cy.get(`[data-testid=${testId}]`, {log: false})

All my attempts to retry either end up with an infinite loop, or failing due to element not existing when it's expected that the element doesn't exist.

@maxkoretskyi
Copy link

maxkoretskyi commented Jul 28, 2021

This could work, retrying for 5 times within the interval of 1s

Cypress.Commands.add('custom', () => {
  let index = 0;
  const promise = new Promise((resolve) => {
    const interval = setInterval(() => {
      if (index++ > 5) {
        clearInterval(interval);
        resolve(1);
      } else {
        console.log('retry attempt ' + index);
      }
    }, 1000);
  });

  return cy.wrap(promise, {timeout: 10000});
});

@egucciar
Copy link
Contributor

I am also unable to get the above solutions to work consistently. I have the following assertion:

    cy.retryGet('[data-cy="form-back"]').safeClick();

And the following code:

/**
 * allow for promise to verify upcoming assertions to resolve
 * this differs from the above commands by running the future assertions
 * and retrying until those assertions also pass
 *
 * Inspired by https://github.com/cypress-io/cypress/issues/3109#issuecomment-477037752
 */
Cypress.Commands.add(
  'resolve',
  { prevSubject: 'optional' },
  (subject, fn, opts = {}) => {
    const resolve = () => {
      fn(subject).then((res) =>
        cy.verifyUpcomingAssertions(res, opts, { onRetry: resolve })
      );
    };
    return resolve();
  }
);

/**
 * get selector with retryability on the selector
 * check that element exists and is not undefiend
 */
Cypress.Commands.add('retryGet', (selector) => {
  const fn = () =>
    cy.wrap(
      new Cypress.Promise((resolve) => {
        cy.get(selector).then(resolve);
      })
    );

  // When retryGet is used as the first command in a test,
  // There can be a race condition where cypress see no commands
  // due to the leading promsie. Fix by forcing one command
  // with no interactions just to register it into cypress
  cy.get(selector);

  return cy
    .resolve(fn)
    .should((el) => expect(Cypress.dom.isAttached(el)).to.eq(true))
    .should((el) => expect(el === undefined).to.eq(false));
});

// Retryable click selector to be used in tandem with retryGet
// https://github.com/testing-library/cypress-testing-library/issues/153#issuecomment-897867503
Cypress.Commands.add('safeClick', { prevSubject: 'element' }, ($element) => {
  const click = ($el) => $el.click();
  return cy
    .wrap($element)
    .pipe(click)
    .should(($el) => expect(Cypress.dom.isDetached($el)).to.be.false);
});

@jennifer-shehane, @Lakitna , @bohendo , is there any reason why the above code would fail like so?

image

I would expect since that final assertion failed, it should retry the whole chain, but it appears not to do that.

Another shot of the command log:

image

I'm hoping to get a solution without intervals or timeout using Cypress methods. Also, the only way our team has been avoiding this issue has been by adding cy.wait(500) before interacting with elements we presume to be rerendering based on flakey tests.

@cypress-bot cypress-bot bot added stage: backlog and removed stage: proposal 💡 No work has been done of this issue labels Apr 29, 2022
@BlueWinds
Copy link
Contributor

Hello! I just wanted to let you all know that we're adding this in Cypress 12.0.0. It turns out that "ability to retry commands" was pretty central to resolving #7306 , and as part of that effort, we're exposing the addQuery interface publicly.

This will be going out in a couple of weeks with Cypress 12, but if you want a bit of a preview, here's the PR we have open with cypress-testing-library to update them to use the new API: https://github.com/testing-library/cypress-testing-library/pull/238/files

As an easier way to get started, here's a preview of the API docs: https://deploy-preview-4835--cypress-docs.netlify.app/api/cypress-api/custom-queries, and the re-written guide on retry-ability that discusses queries vs. other commands: https://deploy-preview-4835--cypress-docs.netlify.app/guides/core-concepts/retry-ability#Commands-Queries-and-Assertions

The docs are still in review, but I'd welcome any comments or questions on them if people want to read it / try out the pre-release builds of Cypress 12 (latest as of now: b9d053e#comments).

@BlueWinds
Copy link
Contributor

Cypress 12.0.0 is going out today, which should address these needs. If there's still anything lacking once you take a look at Cypress.Commands.addQuery, please feel free to open a new issue and we can start some new discussions and look at improvements!

@mjhenkes
Copy link
Member

mjhenkes commented Dec 7, 2022

This was release in 12.0.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pkg/driver This is due to an issue in the packages/driver directory topic: plugins ⚙️ type: enhancement Requested enhancement of existing feature
Projects
None yet
Development

No branches or pull requests