Fix flaky tests #784
Fix flaky tests #784
Conversation
| @@ -6,6 +6,7 @@ jobs: | |||
| xcode: "9.4.1" | |||
| steps: | |||
| - checkout | |||
| - run: xcrun simctl list | |||
There was a problem hiding this comment.
Since I added xcrun simctl list we've miraculously never had the error again. If I remove it we get the missing simulator error sporadically. This could also help us debug it next time.
| [self setupFlushTimer]; | ||
| }); | ||
| } | ||
| self.flushTimer = [NSTimer timerWithTimeInterval:self.configuration.flushInterval |
There was a problem hiding this comment.
do we need to check which thread we are on?
There was a problem hiding this comment.
No, we manually add the timer in NSRunLoop.mainRunLoop so it'll always be scheduled to run on the main thread:
We needed to check before because we used scheduledTimerWithTimeInterval instead of timerWithTimeInterval
There was a problem hiding this comment.
We've had some issues around concurrency - particularly as the threads we initialize integrations is not guaranteed.
This causes some integrations and customers to rely on implementation details
that have caused a few issues in the past (#518, #714, #683) so I'd like to be a bit careful rolling this out. If you're confident in these changes as is - let's plan to cut this as a beta release to get a bit more testing through this change!
There was a problem hiding this comment.
Thanks for pointing this out! If customers are relying on this kind of implementation detail I think it’d be better to spend more time looking into the impact. Moving this to #785.
| } | ||
| else { | ||
| sent = -1 | ||
| waitUntil(timeout: 60) {done in |
There was a problem hiding this comment.
is this the timeout to check for the messages in webhook? if so, you may want to increase it to 3 minutes as 1 minute is sometimes not enough to find them in webhook (3 mins is what library-e2e-tester currently does).
There was a problem hiding this comment.
It isn't this one, but I pushed a new commit to increase the webhook timeout to 3 minutes - thanks!
|
Note: I pushed a commit to use |
Codecov Report
@@ Coverage Diff @@
## master #784 +/- ##
==========================================
- Coverage 85.7% 85.55% -0.15%
==========================================
Files 52 52
Lines 2672 2666 -6
==========================================
- Hits 2290 2281 -9
- Misses 382 385 +3 |
fathyb
force-pushed
the
fix/flaky-test
branch
from
627d3ab to
72065b8
Compare
fathyb
force-pushed
the
fix/flaky-test
branch
from
72065b8 to
98375d3
Compare
This PR fixes 3 common source of failure for our test :
Deadlock : sometimes the test runner freeze and needs a total clean to be run. We fix this by removing the need to call the main thread in SegmentIntegrationMoved to Fix flaky tests #784We ran 21+ successful and consecutive test iterations : https://circleci.com/gh/segmentio/workflows/analytics-ios/tree/fix%2Fflaky-test
Ref : LIB-543