Error Running Acceptance Tests

[mtmckenna]

Hello there,

I'm trying emberfire out, but I've run into a problem: I get an error every time I run an acceptance test. I'e pasted the error below. Any ideas?

Thank you!

FIREBASE WARNING: Exception was thrown by user callback. Error: Called stop() outside of a test context at Object.extend.stop (http://localhost:4200/assets/test-support.js:3000:10) at exports.default._emberTestingAdaptersAdapter.default.extend.asyncStart (http://localhost:4200/assets/vendor.js:52454:13) at asyncStart (http://localhost:4200/assets/vendor.js:43110:47) at Object.async (http://localhost:4200/assets/vendor.js:43124:7) at fulfill (http://localhost:4200/assets/vendor.js:66638:22) at handleMaybeThenable (http://localhost:4200/assets/vendor.js:66598:9) at resolve (http://localhost:4200/assets/vendor.js:66611:7) at resolvePromise (http://localhost:4200/assets/vendor.js:66755:9) at http://localhost:4200/assets/vendor.js:99926:11 at ns.wrapper.fb.api.Query.goog.defineClass.once.onceCallback (http://localhost:4200/assets/vendor.js:99115:43) firebase-debug.js:4798 Uncaught Error: Called stop() outside of a test contextextend.stop @ qunit.js:329exports.default._emberTestingAdaptersAdapter.default.extend.asyncStart @ ember.debug.js:41900asyncStart @ ember.debug.js:32556(anonymous function) @ ember.debug.js:32570fulfill @ ember.debug.js:56084handleMaybeThenable @ ember.debug.js:56044resolve @ ember.debug.js:56057resolvePromise @ ember.debug.js:56201(anonymous function) @ firebase.js:193ns.wrapper.fb.api.Query.goog.defineClass.once.onceCallback @ firebase-debug.js:12452ns.wrapper.fb.core.view.ValueEventRegistration.getEventRunner @ firebase-debug.js:5053ns.wrapper.fb.core.util.exceptionGuard @ firebase-debug.js:4793ns.wrapper.fb.core.view.EventList.raise @ firebase-debug.js:3407ns.wrapper.fb.core.view.EventQueue.raiseQueuedEventsMatchingPredicate_ @ firebase-debug.js:3379ns.wrapper.fb.core.view.EventQueue.raiseEventsForChangedPath @ firebase-debug.js:3367(anonymous function) @ firebase-debug.js:11666(anonymous function) @ firebase-debug.js:11094ns.wrapper.fb.core.PersistentConnection.goog.defineClass.onDataMessage_ @ firebase-debug.js:11250ns.wrapper.fb.realtime.Connection.onDataMessage_ @ firebase-debug.js:10821ns.wrapper.fb.realtime.Connection.onPrimaryMessageReceived_ @ firebase-debug.js:10815ns.wrapper.fb.realtime.Connection.connReceiver_ @ firebase-debug.js:10731ns.wrapper.fb.realtime.WebSocketConnection.appendFrame_ @ firebase-debug.js:10507ns.wrapper.fb.realtime.WebSocketConnection.handleIncomingFrame @ firebase-debug.js:10539ns.wrapper.fb.realtime.WebSocketConnection.open.mySock.onmessage @ firebase-debug.js:10464

Uncaught Error: Called stop() outside of a test contextextend.stop @ qunit.js:329exports.default._emberTestingAdaptersAdapter.default.extend.asyncStart @ ember.debug.js:41900asyncStart @ ember.debug.js:32556(anonymous function) @ ember.debug.js:32570fulfill @ ember.debug.js:56084handleMaybeThenable @ ember.debug.js:56044resolve @ ember.debug.js:56057resolvePromise @ ember.debug.js:56201(anonymous function) @ firebase.js:193ns.wrapper.fb.api.Query.goog.defineClass.once.onceCallback @ firebase-debug.js:12452ns.wrapper.fb.core.view.ValueEventRegistration.getEventRunner @ firebase-debug.js:5053ns.wrapper.fb.core.util.exceptionGuard @ firebase-debug.js:4793ns.wrapper.fb.core.view.EventList.raise @ firebase-debug.js:3407ns.wrapper.fb.core.view.EventQueue.raiseQueuedEventsMatchingPredicate_ @ firebase-debug.js:3379ns.wrapper.fb.core.view.EventQueue.raiseEventsForChangedPath @ firebase-debug.js:3367(anonymous function) @ firebase-debug.js:11666(anonymous function) @ firebase-debug.js:11094ns.wrapper.fb.core.PersistentConnection.goog.defineClass.onDataMessage_ @ firebase-debug.js:11250ns.wrapper.fb.realtime.Connection.onDataMessage_ @ firebase-debug.js:10821ns.wrapper.fb.realtime.Connection.onPrimaryMessageReceived_ @ firebase-debug.js:10815ns.wrapper.fb.realtime.Connection.connReceiver_ @ firebase-debug.js:10731ns.wrapper.fb.realtime.WebSocketConnection.appendFrame_ @ firebase-debug.js:10507ns.wrapper.fb.realtime.WebSocketConnection.handleIncomingFrame @ firebase-debug.js:10539ns.wrapper.fb.realtime.WebSocketConnection.open.mySock.onmessage @ firebase-debug.js:10464

[RayMangum]

Seeing the same error message inside an acceptance test for authentication. Not mocking; just trying to hit the Firebase API:

test('a user signs in with the correct password', function(assert) {
  visit('/signin');
  fillIn("input[type='email']", "[email protected]");
  fillIn("input[type='password']", "validpassword");
  click("button[type='submit']");

  andThen(function() {
    assert.equal(currentURL(), '/projects');
  });
});

Signing in and out works fine in development when accessing localhost:4200, but it fails with with the error message above in test.

[leifcr]

Might it be related to #251 ?

[tstirrat]

Error: Called stop() outside of a test context at Object.extend.stop seems to indicate that the acceptance tests are not waiting for firebase to retrieve its data.

Can someone make a MCVE for this?

[RayMangum]

I created a repo that should be able to reproduce the "Called stop() outside of test context" error message: https://github.com/RayMangum/empty. The README explains how to reproduce the behavior. The easiest way to see the error is to run ember test --server and look for the error message in Safari or Chrome. I hope this helps. If I'm doing anything wrong on my end, please let me know. Thanks in advance.

[tstirrat]

Thanks @RayMangum ! really appreciate you taking the time to make a minimal example.

[gabrielgrant]

I was getting this error, as well as Error: Assertion Failed: You can only unload a record which is not inFlight. when I try to create, check, and then delete a record.

In both cases, it seems the root cause is that the ember testing system doesn't know how to handle Firebase's asynchronicity. With most adapters, this isn't a problem, because the testing system instruments AJAX calls and ensures they have completed before proceeding, but this obviously doesn't work with Firebase's websockets communication.

To work around the issue, I've been using a custom test waiter. You can install it with ember install ember-cli-test-model-waiter

I've just submitted a pull request to explain this in the README

[gabrielgrant]

@RayMangum it actually looks like the problem you're seeing with login/logout won't be fixed with ember-cli-test-model-waiter: because the login/logout process doesn't use ED models, tracking them doesn't help.

I've just pushed a kludgy fix of tracking the login process manually, but this should really be rolled into emberfire

@tstirrat is there any straightforward way of tracking outstanding Firebase requests explicitly? If so, putting that into the firebase Ember service is probably the best way to fix all these issues (and avoid other problems if people access the service directly). But if not, wrapping the login process (possibly as a generic wrapper for non-ajax-based torii logins) and using the model waiter should at least fix this issue for the public emberfire API.

[gabrielgrant]

Just submitted a PR to address the login issue. @RayMangum's example passes the test suite with this updated version of emberfire.

[edborden]

I have been banging my head on my desk trying to get Emberfire to run acceptance tests. I understand there are a few hacks floating around, and I've tried all of them, along with manually inserting test waiters, wrapping calls in Ember.run, and more. Super frustrating. Eventually, I'm sure I'll find something that works, but this is going to continuously be causing me pain, I can see it. I think this has to be looked at as a major failing of the library, and hence the service itself if you are trying to use it with Ember.

[tstirrat]

@edborden can you clarify if you are having issues while using both ember-cli-test-model-waiter and also applying the auth waiter workaround in #330?

I'll get #330 cleaned up and submitted.

[edborden]

ember-cli-test-model-waiter has a few issues, probably related to Ember v2 vs v1. I submitted gabrielgrant/ember-cli-test-model-waiter#1, but I ran into more when I hacked past that. I gave up on that path for moment because it don't think it will cover store#query anyway.

I also tried a version of http://spin.atomicobject.com/2015/07/23/testing-asynchronous-behavior-ember/, but I'm still running into stop() outside of test context and async outside of runloop errors.

I think parts of the firebase adapter itself need to be wrapped in explicit Ember.run's. There also needs to be a specific module, like liquid-fire's transition-map (https://github.com/ef4/liquid-fire/blob/master/addon/transition-map.js), where Emberfire tracks its own activity and inserts test waiters dynamically.

[tstirrat]

store.query is difficult, because we don't have a discrete event at the end of the query that we can use to mark the requests as "resolved". You just get a stream of child_added events. However, we could probably implement a timer on these (in test mode only) and mark them resolved after a certain time.

I had considered baking test waiters in to the adapter code -- thanks for the liquid-fire example -- I'll work on something like that.

[tstirrat]

release 1.6.4 includes the test waiters