Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky tests: changeSize and discoverWork in test-resources.js #16124

Closed
cathyxz opened this issue Jun 18, 2018 · 16 comments
Closed

Flaky tests: changeSize and discoverWork in test-resources.js #16124

cathyxz opened this issue Jun 18, 2018 · 16 comments
Assignees
@alabiaga
Labels
P1: High Priority Related to: Flaky Tests Type: Bug WG: runtime
Milestone
Backlog Bugs

Comments

@cathyxz
Copy link
Contributor

cathyxz commented Jun 18, 2018 •
edited
Loading

Per build: https://travis-ci.org/ampproject/amphtml/jobs/393815752
These are failing / flaky on Safari.
Error messages below:

DESCRIBE => Resources changeSize
  DESCRIBE => attemptChangeSize rules wrt viewport
    IT => should change size when document is invisible
      ✗ Attempted to wrap getVisibilityState which is already wrapped
      checkWrappedMethod@/tmp/node_modules/sinon/lib/sinon/util/core/wrap-method.js:39 <- /tmp/76e9ec9ec8ad21bbdd44820c0bbaddcd.browserify:50286:34
      wrapMethod@/tmp/node_modules/sinon/lib/sinon/util/core/wrap-method.js:85 <- /tmp/76e9ec9ec8ad21bbdd44820c0bbaddcd.browserify:50332:31
      stub@/tmp/node_modules/sinon/lib/sinon/collection.js:119 <- /tmp/76e9ec9ec8ad21bbdd44820c0bbaddcd.browserify:47592:38
      /tmp/test/functional/test-resources.js:1997:19 <- /tmp/76e9ec9ec8ad21bbdd44820c0bbaddcd.browserify:148925:19
      --------------
      wrapMethod@/tmp/node_modules/sinon/lib/sinon/util/core/wrap-method.js:110 <- /tmp/76e9ec9ec8ad21bbdd44820c0bbaddcd.browserify:50357:40
      stub@/tmp/node_modules/sinon/lib/sinon/collection.js:119 <- /tmp/76e9ec9ec8ad21bbdd44820c0bbaddcd.browserify:47592:38
      /tmp/test/functional/test-resources.js:1450:17 <- /tmp/76e9ec9ec8ad21bbdd44820c0bbaddcd.browserify:148335:17
●
  IT => "after each" hook for "should change size when document is invisible"
    ✗ Expected getRect([...]) at least once (never called)
    fail@/tmp/node_modules/sinon/lib/sinon/mock-expectation.js:281 <- /tmp/76e9ec9ec8ad21bbdd44820c0bbaddcd.browserify:48572:34
    verify@/tmp/node_modules/sinon/lib/sinon/mock.js:108 <- /tmp/76e9ec9ec8ad21bbdd44820c0bbaddcd.browserify:48689:33
    /tmp/test/functional/test-resources.js:1785:4 <- /tmp/76e9ec9ec8ad21bbdd44820c0bbaddcd.browserify:148717:24
●
IT => "after each" hook for "should change size when document is invisible"
  ✗ You likely forgot to restore sinon timers (installed via sandbox.useFakeTimers).
  /tmp/test/_init_tests.js:479:20 <- /tmp/76e9ec9ec8ad21bbdd44820c0bbaddcd.browserify:111859:20
[20:57:18] Safari 11.1.0 (Mac OS X 10.13.4): Executed 1760 of 3610 (Skipped 174) 4 FAILED

Also from the same test suite, these are flaky / failing on Windows Chrome.

DESCRIBE => Resources discoverWork
  IT => "after each" hook for "should render two screens when visible"
    ✗ Unexpected call: getRect() at setState@/tmp/src/finite-state-machine.js:73:6 <- /tmp/a3788620192f2b19616a0bb87a169728.browserify:66567:15
    Unexpected call: getRect() at setState@/tmp/src/finite-state-machine.js:73:6 <- /tmp/a3788620192f2b19616a0bb87a169728.browserify:66567:15
    Expectation met: getRect([...]) once
    fail@/tmp/node_modules/sinon/lib/sinon/mock-expectation.js:281 <- /tmp/a3788620192f2b19616a0bb87a169728.browserify:48590:34
    verify@/tmp/node_modules/sinon/lib/sinon/mock.js:108 <- /tmp/a3788620192f2b19616a0bb87a169728.browserify:48707:33
    /tmp/test/functional/test-resources.js:1116:4 <- /tmp/a3788620192f2b19616a0bb87a169728.browserify:148053:24
@cathyxz cathyxz changed the title Resources changeSize test suites fail / flake on Safari Resources changeSize and discoverWork tests fail / flake on Safari / Windows Jun 20, 2018
@dreamofabear dreamofabear self-assigned this Jun 29, 2018
@dreamofabear
Copy link

I've noticed the following happening occasionally:

DESCRIBE => Resources discoverWork
IT => "after each" hook for "should NOT rerender anything"
✗ Unexpected call: getRect() at setState@/tmp/src/finite-state-machine.js:73:6 <- /tmp/6a9de7f3c7561ff1cd02fe96d61e6c30.browserify:65031:15
Unexpected call: getRect() at setState@/tmp/src/finite-state-machine.js:73:6 <- /tmp/6a9de7f3c7561ff1cd02fe96d61e6c30.browserify:65031:15
Unexpected call: getRect() at setState@/tmp/src/finite-state-machine.js:73:6 <- /tmp/6a9de7f3c7561ff1cd02fe96d61e6c30.browserify:65031:15
Expectation met: getRect([...]) once
fail@/tmp/node_modules/sinon/lib/sinon/mock-expectation.js:281 <- /tmp/6a9de7f3c7561ff1cd02fe96d61e6c30.browserify:47053:34
verify@/tmp/node_modules/sinon/lib/sinon/mock.js:108 <- /tmp/6a9de7f3c7561ff1cd02fe96d61e6c30.browserify:47170:33
/tmp/test/functional/test-resources.js:1116:4 <- /tmp/6a9de7f3c7561ff1cd02fe96d61e6c30.browserify:147360:24

[23:54:04] Safari 11.1.0 (Mac OS X 10.13.4): Executed 3227 of 3553 (Skipped 301) 1 FAILED

https://travis-ci.org/ampproject/amphtml/jobs/396641161

DESCRIBE => Resources discoverWork
DESCRIBE => getResourcesInRect
IT => should wait until ready-scan
✗ expected true to be false
AssertionError@node_modules/chai/chai.js:9320:22
assert@node_modules/chai/chai.js:239:31
node_modules/chai/chai.js:1055:16
propertyGetter@node_modules/chai/chai.js:7784:33
doAsserterAsyncAndAddThen@/tmp/node_modules/chai-as-promised/lib/chai-as-promised.js:289 <- /tmp/5f3328cdaff637ae69bd0a2b905ebb73.browserify:26288:27
/tmp/node_modules/chai-as-promised/lib/chai-as-promised.js:280 <- /tmp/5f3328cdaff637ae69bd0a2b905ebb73.browserify:26279:68
overwritingPropertyGetter@node_modules/chai/chai.js:9027:41
get@[native code]
proxyGetter@node_modules/chai/chai.js:9128:25
/tmp/test/functional/test-resources.js:1555:48 <- /tmp/5f3328cdaff637ae69bd0a2b905ebb73.browserify:148470:48

[23:44:56] Safari 11.1.0 (Mac OS X 10.13.4): Executed 3280 of 3583 (Skipped 303) 1 FAILED

https://travis-ci.org/ampproject/amphtml/jobs/397579389

@erwinmombay erwinmombay added this to the Backlog Bugs milestone Aug 21, 2018
@calebcordry
Copy link
Member

These have also been acting up a bit today. cc/ @rsimha

https://travis-ci.org/ampproject/amphtml/jobs/431103300

DESCRIBE => Resources changeSize●●●●●●●●●●●●
  IT => "after each" hook for "should NOT resize if multi req above vp cannot adjust scroll"
    ✗ Unexpected call: getRect() at setState@/home/travis/build/ampproject/amphtml/src/finite-state-machine.js:73:6 <- /tmp/6e80e9d0aaa7ec09e9678985eb7d9043.browserify:52144:15
    Expectation met: getRect([...]) at least once
    Expectation met: getRect([...]) once
    fail@node_modules/sinon/pkg/sinon.js:1983:34
    verify@node_modules/sinon/pkg/sinon.js:2109:33
    /home/travis/build/ampproject/amphtml/test/functional/test-resources.js:1786:4 <- /tmp/6e80e9d0aaa7ec09e9678985eb7d9043.browserify:139495:24

@rsimha
Copy link
Contributor

rsimha commented Sep 20, 2018

Can we disable this test while it's fixed? It's currently blocking several unrelated PRs from being merged.

@dreamofabear
Copy link

Looks like different tests in the same describe block. I'd like to monitor the new breakages for a few days to see if there's a pattern first -- please post the failing tests in this issue.

@rsimha
Copy link
Contributor

rsimha commented Sep 20, 2018 •
edited
Loading

Good observation, @choumx. Here are some failures on master from today:
https://travis-ci.org/ampproject/amphtml/jobs/431130238#L1473
https://travis-ci.org/ampproject/amphtml/jobs/431103300#L2931
https://travis-ci.org/ampproject/amphtml/jobs/431041217#L1461

@dreamofabear dreamofabear mentioned this issue Jan 8, 2019
Closed
@dreamofabear dreamofabear changed the title Resources changeSize and discoverWork tests fail / flake on Safari / Windows Flaky tests: changeSize and discoverWork in test-resources.js Jan 8, 2019
@dreamofabear
Copy link

Aaron, would you mind taking a crack at these two flaky tests? I've noticed them a lot recently on master and PR builds.

Recent failure on master: https://travis-ci.org/ampproject/amphtml/jobs/477890970

DESCRIBE => Resources discoverWork
  IT => "after each" hook for "should measure unbuilt elements"
    ✗ Unexpected call: getRect() at setState@/home/travis/build/ampproject/amphtml/src/finite-state-machine.js:73:14 <- /tmp/73d59251fee8e676cde5e2679e872de7.browserify.js:60474:15
    Expectation met: getRect([...]) once
    fail@node_modules/sinon/pkg/sinon.js:2027:34
    verify@node_modules/sinon/pkg/sinon.js:2152:33
    /home/travis/build/ampproject/amphtml/test/unit/test-resources.js:1115:4 <- /tmp/73d59251fee8e676cde5e2679e872de7.browserify.js:148811:24

@rsimha
Copy link
Contributor

rsimha commented Jan 17, 2019 •
edited
Loading

This is still a top cause of flakiness. From today: https://travis-ci.org/ampproject/amphtml/jobs/480964219#L1524

Meanwhile, can we skip the flaky tests to keep master green?

/cc @alanorozco who is build cop

@alabiaga
Copy link
Contributor

I haven't looked at this yet but will by end of day today. Sorry for the delay. Thanks.

@alabiaga
Copy link
Contributor

I am in the process of upgrading my sauce lab account so I can run this in the environment the logs indicate it is failing in. @choumx posted a failure in a non sauce lab environment but given it's flakiness, I am unable to reproduce after several attempts. Looking at the failures, I think the easy fix (not necessarily the right fix) could just be removing the number of calls expected from the call viewport.getRect(). Some of the logs from the test failures provided indicate the state machine throwing this error so it seems like there is some unexpected state transition triggering this? I have a PR to test on saucelabs with the modified expectations which passes locally. Once I can reproduce the failures in sauce labs and run my changes and verify that my changes reduces or fixes the flakiness then I will send out the PR. I think that immediate fix is ok as it'll allow us to run the tests with hopefully reduced flakiness and test the resources functionality without having to comment them out. Investigating why getRect is getting called more than expected is difficult when I can't reproduce. Thanks

@alabiaga
Copy link
Contributor

cc\ @estherkim

@alabiaga alabiaga mentioned this issue Jan 24, 2019
Merged
@ampprojectbot
Copy link
Member

This is a high priority issue but it hasn't been updated in awhile. @alabiaga Do you have any updates?

alabiaga added a commit that referenced this issue Feb 13, 2019
@alabiaga
Make test-resource.js not flaky (#20514)
e13a531 
Remove call count expectations on getRect method call which is the culprit of the flakiness of the tests mentioned. This should be fine as it doesn't take away from what's being primarily tested as it occurs in the afterEach call. Note that this is not a permanent fix but a temporary one to alleviate the flakiness. I will keep #16124 open.

Tested locally and ran it several times in saucelabs.
estherkim added a commit to estherkim/amphtml that referenced this issue Feb 15, 2019
@estherkim
Squashed commit of the following:
e199707 
commit 08466820bf2c8c13e21abc5c26159d228edb345c
Author: Esther Kim <[email protected]>
Date:   Fri Feb 15 10:47:56 2019 -0500

    Whitespace

commit cdfd719
Author: Esther Kim <[email protected]>
Date:   Fri Feb 15 10:34:13 2019 -0500

    Stop timer logging

commit 4b63cd7
Author: Esther Kim <[email protected]>
Date:   Fri Feb 15 10:08:59 2019 -0500

    Typos

commit 7db4b28
Author: Esther Kim <[email protected]>
Date:   Thu Feb 14 18:12:37 2019 -0500

    Review comments

commit ced9c5d
Author: Esther Kim <[email protected]>
Date:   Thu Feb 14 18:01:12 2019 -0500

    Review comments

commit 8fa0e88
Merge: e65dd15 893ff3f
Author: Esther Kim <[email protected]>
Date:   Thu Feb 14 17:29:46 2019 -0500

    Merge branch 'master' into travis-build-matrix

commit e65dd15
Author: Esther Kim <[email protected]>
Date:   Thu Feb 14 17:28:35 2019 -0500

    Review comments

commit d69c38a
Author: Esther Kim <[email protected]>
Date:   Thu Feb 14 17:08:20 2019 -0500

    Update logging

commit 7e4f701
Author: Esther Kim <[email protected]>
Date:   Thu Feb 14 17:00:38 2019 -0500

    Review changes

commit 893ff3f
Author: Aaron Turner <[email protected]>
Date:   Thu Feb 14 13:49:16 2019 -0800

    Return 400 when recaptcha example fails (ampproject#20668)

commit 3f0d6c6
Author: Enrique Marroquin <[email protected]>
Date:   Thu Feb 14 16:37:15 2019 -0500

    [amp-story] Allow more parameters for pan/zoom animations ✨ (ampproject#20778)

    * allow more parameters in for pan/zoom animations.

    * change attribute names

    * types

    * fix test

    * assert values for params, change attribute naming.

    * change attribute names do scale-start, scale-end.

    * hoist defaults to constants

commit 35d6030
Author: Justin Ridgewell <[email protected]>
Date:   Thu Feb 14 16:02:00 2019 -0500

    🐛 Assert that values are never returned from vsync callbacks (ampproject#20836)

    * Assert that values are never returned from vsync callbacks

    Vsync does not propogate those values in any way. This is specifically targetting code like:

    ```js
    vsync.measure(() => {
      return el.getBoundingClientRect();
    });

    // or
    vsync.mutatePromise(() => {
      return someOtherPromise;
    });
    ```

    * Fix dev call

    * Lint

    * Don't use devAssert so this stays in production

    * Type checks

commit b9ba432
Author: Esther Kim <[email protected]>
Date:   Thu Feb 14 15:52:54 2019 -0500

    Fix nit, hopefully fix chrome start

commit 3cb87fa
Author: Esther Kim <[email protected]>
Date:   Thu Feb 14 15:08:42 2019 -0500

    bash

commit 5d9844e
Author: Esther Kim <[email protected]>
Date:   Thu Feb 14 14:54:40 2019 -0500

    rm

commit b027eb0
Author: Esther Kim <[email protected]>
Date:   Thu Feb 14 14:39:52 2019 -0500

    Review comments

commit fe27d76
Author: sohanirao <[email protected]>
Date:   Thu Feb 14 11:38:43 2019 -0800

    SwG Release 0.1.22.44 (ampproject#20834)

commit 736d506
Author: Esther Kim <[email protected]>
Date:   Thu Feb 14 14:25:37 2019 -0500

    Install chrome for dist test job, add EXTENSIONS_CSS_MAP to build artifact zip

commit a36ee5d
Author: delima02 <[email protected]>
Date:   Thu Feb 14 14:17:42 2019 -0500

    🐛 Guard against getBoundingClientRect to prevent unspecified errors in IE. (ampproject#20731)

    * Guard against `Element.getBoundingClientRect`.

    * Refactor solution.

    * Check if element exists in the DOM.

    * Fix solution using `Node.isConnected`.

    * Correct the default value of the desired DOMRect

    * Patch getBoundingClientRect for unsupported browsers. Alternative solution

    * Refactor patch.

    * Fix - use assigned variable rect, previously unused.

    * Fix imports.

    * Minor fixes.

    * Make the return type of getBoundingClientRect be consistent with usages

    * File overview comment and a minor fix.

commit e9acbd2
Author: Cathy Zhu <[email protected]>
Date:   Fri Feb 15 03:17:37 2019 +0800

    Undo space reservation after successful height change (ampproject#20856)

commit ccea3e2
Author: Nikita Beloglazov <[email protected]>
Date:   Thu Feb 14 11:12:16 2019 -0800

    Export few data types from content-recommendation.js (ampproject#20854)

    It allows users of that file to use the types. Given that we return objects of these types from exported functions - exporting types themselves makes sense.

    Tested:
      presubmit

commit 80f9fa9
Author: Sepand Parhami <[email protected]>
Date:   Thu Feb 14 11:10:11 2019 -0800

    🐛Do not perform static layout for cloned nodes with static layout. (ampproject#20724)

commit a751cc7
Author: keithwrightbos <[email protected]>
Date:   Thu Feb 14 13:47:48 2019 -0500

    AdSense Fast Fetch delay 3 viewport experiment - respect loading strategy (ampproject#20866)

    * initial commit

    * modify dep-check

    * truly fix dep issue

commit 806db59
Author: Carlos Vializ <[email protected]>
Date:   Thu Feb 14 13:21:26 2019 -0500

    ✨Implement autoexpand for amp-form textarea element (ampproject#20772)

    * Add expanding textarea feature to amp-form

    * Add autoshrink behavior in addition to autoexpand

    * reset

    * reset2 it works

    * Does not stay at top

    * Prevent scrollbar from flashing before max-height

    * default autoshrink. Prevent shrinking if the textarea hasn't yet expanded

    * Use class style

    * Add manual test

    * Cleanup

    * Review feedback. Remove autoshrink attr. Lazy-load feature.

    * Encapsulate all textarea logic, including installation.

    * Mark expensive property accesses as OK

    * Warn and forbid textarea with initial overflow

    * Add tests for confidence. Fix presubmit errors.

commit cb95582
Author: Esther Kim <[email protected]>
Date:   Thu Feb 14 11:51:37 2019 -0500

    Separate PR jobs

commit 4679a4c
Author: Gabriel Majoulet <[email protected]>
Date:   Thu Feb 14 10:52:06 2019 -0500

    Displaying more specific messages when users have to resize their window, + refactoring. (ampproject#20835)

commit b9814f5
Author: keithwrightbos <[email protected]>
Date:   Thu Feb 14 10:43:42 2019 -0500

    initial commit (ampproject#20865)

commit e570d46
Author: Hongfei Ding <[email protected]>
Date:   Wed Feb 13 22:15:49 2019 -0800

    amp-ad-exit: use HostService if exists. (ampproject#20357)

    * amp-ad-exit: use HostService if exists.

    * naming & docs

    * rename host-api to host-service

    * lint

commit ecc3daa
Author: Cathy Zhu <[email protected]>
Date:   Thu Feb 14 13:34:08 2019 +0800

    Make is-my-pr-in-production-yet more prominent in our docs (ampproject#20857)

commit 12d441c
Author: Alan Orozco <[email protected]>
Date:   Wed Feb 13 13:44:50 2019 -0800

    ✨auto-lightbox: Discard tap actions with invalid references (ampproject#20789)

    Exposes `ActionService.hasResolvableAction` to:

      - Discard tap action in auto-lightbox criteria when it does not resolve to a valid target.
      - Fix `amp-lightbox-gallery` so it will properly traverse up the tree to discard targets, and not to discard those dynamically set.

commit dd4037d
Author: Nikita Beloglazov <[email protected]>
Date:   Wed Feb 13 13:25:13 2019 -0800

    Improve Google matched content logic to reflect adsbygoogle.js. (ampproject#20624)

    The commit introduces core-shared.js file which supposed to be shared between AMP and internal Google adsbygoogle.js codebase. The file has no dependencies on environment to make importing these files in other projects easier.

    In particular this commit adds support for publisher settings as described in https://support.google.com/adsense/answer/7533385

    Tested:
      unit tests
      manually on fast festh and delayed fetch

commit 76a7c0f
Author: keithwrightbos <[email protected]>
Date:   Wed Feb 13 16:10:58 2019 -0500

    initial commit (ampproject#20837)

commit f1fb33b
Author: keithwrightbos <[email protected]>
Date:   Wed Feb 13 15:37:39 2019 -0500

    initial commit (ampproject#20833)

commit 6831b63
Author: Nikita Beloglazov <[email protected]>
Date:   Wed Feb 13 12:37:10 2019 -0800

    Cleanup adsense amp-auto-ads holdout experiment. Close ampproject#9247. (ampproject#20774)

    It has been running for a while and we no longer need it. If someone decides to look at results in future - they can rollback this commit and restart experiment.

    Tested:
      presubmit

commit 4f3cd0d
Author: William Chou <[email protected]>
Date:   Wed Feb 13 15:28:26 2019 -0500

    Don't send amp-bind state to untrusted viewers (ampproject#20822)

    * Only send state for history updates to trusted viewers.

    * Add and fix tests.

    * Fix type check.

    * Fix presubmit.

commit ab467bc
Author: Carlos Vializ <[email protected]>
Date:   Wed Feb 13 14:26:35 2019 -0500

    ✅ Resolve some flakiness in e2e tests (ampproject#20794)

    * Resolve some flakiness in e2e tests

    * Unskip passing test

commit e13a531
Author: Aaron Labiaga <[email protected]>
Date:   Wed Feb 13 13:35:59 2019 -0500

    Make test-resource.js not flaky (ampproject#20514)

    Remove call count expectations on getRect method call which is the culprit of the flakiness of the tests mentioned. This should be fine as it doesn't take away from what's being primarily tested as it occurs in the afterEach call. Note that this is not a permanent fix but a temporary one to alleviate the flakiness. I will keep ampproject#16124 open.

    Tested locally and ran it several times in saucelabs.

commit 4a7282f
Author: Carlos Vializ <[email protected]>
Date:   Wed Feb 13 12:01:23 2019 -0500

    Skip date picker visual diff tests for flakes (ampproject#20809)

commit 703ab02
Author: William Chou <[email protected]>
Date:   Wed Feb 13 11:53:05 2019 -0500

    Clean up 'user-error-reporting' (ampproject#20779)

    * Clean up user-error-reporting.

    * Fix lint.

commit 229d909
Author: Alan Orozco <[email protected]>
Date:   Wed Feb 13 02:44:33 2019 -0800

    🛑 <amp-video-iframe> integration safeguards (ampproject#20826)

    - Prefix all frame-level errors with `<amp-video-iframe>` to let authors know source of integration mistakes.
    - Fail to `adopt` window twice, notify that script tag should only be included once.
    - Display valid listener types when trying to `listenTo('invalidFramework')`.
    - Don't let author `listenTo` twice.

commit 84f701f
Author: Ali Ghassemi <[email protected]>
Date:   Tue Feb 12 23:21:22 2019 -0800

    ✅ amp-youtube: add amp=1 to the url (ampproject#20821)

commit b58b6fa
Author: Alan Orozco <[email protected]>
Date:   Tue Feb 12 23:03:58 2019 -0800

    ✨ Set #amp=1 fragment in amp-video-iframe src (ampproject#20823)

    Docs say we do it, but we don't, so let's do it!

    Also cleaned up some tests to be clearer and to use `async/await`.

commit f92f627
Author: Justin Ridgewell <[email protected]>
Date:   Tue Feb 12 20:24:26 2019 -0500

    Optimize scopeSelector (ampproject#20819)

    https://jsbench.github.io/#e00c46505b5b875d2dcd5894d6e46adc

commit 0298883
Author: Alan Orozco <[email protected]>
Date:   Tue Feb 12 16:57:41 2019 -0800

    🐛Reset min dimensions on undock (ampproject#20817)

    Otherwise the video can be oversized and clipped when docked and resizing the window.

commit bcce892
Author: Caleb Cordry <[email protected]>
Date:   Tue Feb 12 16:33:05 2019 -0800

    📖amp story ads RTC & Targeting docs (ampproject#20818)
@rsimha
Copy link
Contributor

rsimha commented Feb 27, 2019

Here's another data point (I'm build cop this week): #21151 (comment) (I haven't skipped any tests yet, will hold off until this issue is fixed.)

@dreamofabear
Copy link

#20514 should've fixed the traces in OP but looks like there are new flakes.

@alabiaga
Copy link
Contributor

alabiaga commented Mar 4, 2019 via email

@alabiaga
Copy link
Contributor

adding flake: https://travis-ci.org/ampproject/amphtml/jobs/503826554

noranazmy pushed a commit to noranazmy/amphtml that referenced this issue Mar 22, 2019
@alabiaga
Make test-resource.js not flaky (ampproject#20514)
c69c7e3 
Remove call count expectations on getRect method call which is the culprit of the flakiness of the tests mentioned. This should be fine as it doesn't take away from what's being primarily tested as it occurs in the afterEach call. Note that this is not a permanent fix but a temporary one to alleviate the flakiness. I will keep ampproject#16124 open.

Tested locally and ran it several times in saucelabs.
bramanudom pushed a commit to bramanudom/amphtml that referenced this issue Mar 22, 2019
@alabiaga @bramanudom
Make test-resource.js not flaky (ampproject#20514)
18bad2d 
Remove call count expectations on getRect method call which is the culprit of the flakiness of the tests mentioned. This should be fine as it doesn't take away from what's being primarily tested as it occurs in the afterEach call. Note that this is not a permanent fix but a temporary one to alleviate the flakiness. I will keep ampproject#16124 open.

Tested locally and ran it several times in saucelabs.
@dreamofabear
Copy link

Haven't seen this in awhile -- closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alabiaga alabiaga

Labels
P1: High Priority Related to: Flaky Tests Type: Bug WG: runtime
Projects
None yet
Milestone
Backlog Bugs
Development

No branches or pull requests
7 participants
@erwinmombay @alabiaga @cathyxz @calebcordry @dreamofabear @ampprojectbot @rsimha