Throw exceptions for testing purposes

[nxcco]

It would be a very useful feature, regarding tests for exception handling, when you could throw custom exceptions on method calls (just like using when(...).thenThrow(...) in the mockito package).

[mayurdhurpate]

This would be very useful if we could somehow simulate Network and other exceptions to better structure the tests. I'm not sure whether considering the package is a fake and not a mock (see this issue: #34 ) whether this could come under its scope or not.

Not sure with so many nested modules in Firestore (e.g.
instance.collection('info').where("week", isEqualTo: week).getDocuments()), whether Mockito too would be a viable option for this or not.

@atn832 would love some insights on this. If this is indeed possible, I would be happy to try raise a PR.

[atn832]

It's a great idea!

Regarding the scope, it's indeed a bit fuzzy. If we build this, we could think of it as the fake behaving like the real Firestore for failures as well.

Since we don't use Mockito's Mocks (the fakes technically still inherit Mock but I'll remove that soon), I wonder how we could implement the feature cleanly though.

I'm also not familiar with the Firestore behaviors on failure. What are the most common cases where calls fail? Which cases would you like the library to be able to throw?

[mayurdhurpate]

Yes, currently we use cloud_firestore_mocks for actual implementation test and use Mockito for testing exceptions. Having same scenario in both packages would help test better.

Few common areas I could think of:

• Security Rules related issue, such as permission denied to access doc
• No internet error while performing get/set and getting data from listening to a stream
• Other server errors thrown due to excessive calls to 1 document etc.

Another issue here is that currently Firestore's exhaustive Exception handling documentation doesn't exist yet and a big revamp is in progress there: firebase/flutterfire#2582

[kmrosiek]

Is there any workaround for throwing exceptions right now?

[gaburielcasado]

@atn832

I had the same necessity. The DocumentReference.set method, for instance, can throw a FirebaseException that will contain a code. Here's a list of possible codes: https://firebase.google.com/docs/reference/kotlin/com/google/firebase/firestore/FirebaseFirestoreException.Code

In my opinion, we could simply have a field in the fake Firestore that receives a FirebaseException (for the set method for instance, it could be called setException. If the value is present, then upon DocumentReference.set being called, we simply throw the provided exception.

I would be up for implementing this for the DocumentReference's set method, then we can just replicate to the others as needed. Tell me if this sounds like a good idea and I'll do it.

[atn832]

We used that approach in firebase_auth_mocks, but it quickly became quite verbose. See:

• https://github.com/atn832/firebase_auth_mocks/blob/master/lib/src/auth_exceptions.dart
• https://github.com/atn832/firebase_auth_mocks/blob/d8dee460643cb263b40bc13e5ab786f05868428e/lib/src/firebase_auth_mocks_base.dart#L64-L109

Because of this, I was wary of going down the same path for fake_cloud_firestore, which is much larger project.

So I took a look at how Mockito does it, and went ahead and built and published a lightweight Mockito lookalike (https://pub.dev/packages/mock_exceptions). It allowed me to much more cleanly support throwing exceptions in fake_cloud_firestore on any kinds of conditions using the standard matchers.

I started with DocumentReference.set, just because it's the one you mentioned. Next I'll write a little how-to for others to contribute similar PRs to support mocking exceptions for other methods.

[atn832]

With fake_cloud_firestore 2.2.0, it is now possible to mock exceptions on DocumentReference.set with the whenCalling(invocation).on(document).thenThrow(exception). See an example of how to use it:

fake_cloud_firestore/test/fake_cloud_firestore_test.dart

Lines 46 to 63 in 7dc8bcf

	test('DocumentReference.set throws exceptions', () async {
	final instance = FakeFirebaseFirestore();
	final doc = instance.collection('users').doc(uid);
	whenCalling(Invocation.method(#set, null))
	.on(doc)
	.thenThrow(FirebaseException(plugin: 'firestore'));
	expect(() => doc.set({'name': 'Bob'}), throwsA(isA<FirebaseException>()));
	});
	test('DocumentReference.set throws exceptions on certain conditions',
	() async {
	final instance = FakeFirebaseFirestore();
	final doc = instance.collection('users').doc(uid);
	whenCalling(Invocation.method(#set, [
	{'name': 'Bob'}
	])).on(doc).thenThrow(FirebaseException(plugin: 'firestore'));
	expect(() => doc.set({'name': 'Alice'}), returnsNormally);
	expect(() => doc.set({'name': 'Bob'}), throwsA(isA<FirebaseException>()));
	});

Since supporting all the methods by myself would be unfeasible, I'll write down a little guide on how to support an exception for a method and encourage you to contribute PRs. If you are unable to contribute a PR, feel free to file a ticket for the specific exception and method you'd like to use so that we can keep track of it.

First, figure out if the exception you're working with should be faked or mocked. For example, throwing an exception when calling DocumentReference.get on a non-existing document or missing field should be faked because it should throw that exception automatically depending on the actual state of the FakeFirebaseFirestore. If you want to simulate an arbitrary server failure, it should be mocked. If we take a look at this list of errors, I'd say ALREADY_EXISTS, NOT_FOUND, INVALID_ARGUMENT should be faked. The others such as ABORTED, CANCELLED, DATA_LOSS, DEADLINE_EXCEEDED, UNAVAILABLE should be mocked.

If it should be faked, you can write something similar to how it's been done. Don't forget to include unit tests in your PR.

If it should be mocked, make something similar to PR-258:

1. Go to the method you want upgrade to throw exceptions. On the first line, add a call to maybeThrowException with the right Invocation settings. You can take a look at these examples in mock_exceptions. They cover most use cases: methods, getters, setters, positional parameters, named parameters, type parameters.
2. Write the unit tests.
   1. Find the place where the method's regular behavior is being tested. Since fake_cloud_firestore_test.dart contains the most tests, perhaps it is there.
   2. Add a new test to check that the upgraded method can indeed throw exceptions. One or two basic tests will suffice.
3. Send a pull request. I'll review it and if all is well, merge and eventually release an update.

Thank you all in advance, and thanks for your patience thus far!

[gaburielcasado]

Thank you so much @atn832 !
This really is a better solution.

So if I understood this correctly:
Exceptions that depend on the internal state of the FakeFirebaseFirestore instance will be faked.

Exceptions that go beyond its internal state, such as exceptions related to security rules and service availability will need to be mocked.

I'll contribute soon to the other methods if no one gets there before. Cheers!

[atn832]

After writing over 200 commits over the past week (cel-dart is the main engine for security rules), I just released a big update for Fake Cloud Firestore regarding security rules. You can now initialize FakeFirebaseFirestore with:

1. the security rules that you use in production in Firebase.
2. a Stream of sign-in events from Firebase Auth Mocks.

Then FakeFirebaseFirestore will automatically verify access on documents based on your rules and the sign-in state (signed in or not, uid, claims) in your tests. Later we'll implement access verification for collections and queries. It is quite advanced, so I encourage you to try it. You can learn how to use it in the README.

So now you may want to mock exceptions only to simulate service unavailability or quota errors.

[gaburielcasado]

@atn832 That's really awesome, I'll give it a go. Thanks a lot.

[wllslmn]

With fake_cloud_firestore 2.2.0, it is now possible to mock exceptions on DocumentReference.set with the whenCalling(invocation).on(document).thenThrow(exception). See an example of how to use it:

fake_cloud_firestore/test/fake_cloud_firestore_test.dart

Lines 46 to 63 in 7dc8bcf

	test('DocumentReference.set throws exceptions', () async {
	final instance = FakeFirebaseFirestore();
	final doc = instance.collection('users').doc(uid);
	whenCalling(Invocation.method(#set, null))
	.on(doc)
	.thenThrow(FirebaseException(plugin: 'firestore'));
	expect(() => doc.set({'name': 'Bob'}), throwsA(isA<FirebaseException>()));
	});
	test('DocumentReference.set throws exceptions on certain conditions',
	() async {
	final instance = FakeFirebaseFirestore();
	final doc = instance.collection('users').doc(uid);
	whenCalling(Invocation.method(#set, [
	{'name': 'Bob'}
	])).on(doc).thenThrow(FirebaseException(plugin: 'firestore'));
	expect(() => doc.set({'name': 'Alice'}), returnsNormally);
	expect(() => doc.set({'name': 'Bob'}), throwsA(isA<FirebaseException>()));
	});

Since supporting all the methods by myself would be unfeasible, I'll write down a little guide on how to support an exception for a method and encourage you to contribute PRs. If you are unable to contribute a PR, feel free to file a ticket for the specific exception and method you'd like to use so that we can keep track of it.

First, figure out if the exception you're working with should be faked or mocked. For example, throwing an exception when calling DocumentReference.get on a non-existing document or missing field should be faked because it should throw that exception automatically depending on the actual state of the FakeFirebaseFirestore. If you want to simulate an arbitrary server failure, it should be mocked. If we take a look at this list of errors, I'd say ALREADY_EXISTS, NOT_FOUND, INVALID_ARGUMENT should be faked. The others such as ABORTED, CANCELLED, DATA_LOSS, DEADLINE_EXCEEDED, UNAVAILABLE should be mocked.

If it should be faked, you can write something similar to how it's been done. Don't forget to include unit tests in your PR.

If it should be mocked, make something similar to PR-258:

1. Go to the method you want upgrade to throw exceptions. On the first line, add a call to maybeThrowException with the right Invocation settings. You can take a look at these examples in mock_exceptions. They cover most use cases: methods, getters, setters, positional parameters, named parameters, type parameters.

2. Write the unit tests.

   1. Find the place where the method's regular behavior is being tested. Since fake_cloud_firestore_test.dart contains the most tests, perhaps it is there.
   2. Add a new test to check that the upgraded method can indeed throw exceptions. One or two basic tests will suffice.

3. Send a pull request. I'll review it and if all is well, merge and eventually release an update.

Thank you all in advance, and thanks for your patience thus far!

@atn832 whenCalling doesn't seem to be defined... 🤔

[atn832]

@wllslmn
whenCalling is defined in the package mock_exceptions. If your project depends on fake_cloud_firestore v2.2.0 or higher, you will automatically have it as an indirect dependency. Just make sure you import it in the file that you need it:

import 'package:mock_exceptions/mock_exceptions.dart';

If you are using Visual Studio Code, it will automatically suggest that you add this import.

See fake_cloud_firestore's dependencies:

fake_cloud_firestore/pubspec.yaml

Line 19 in d89417b

mock_exceptions: ^0.8.2

[wllslmn]

@atn832 thank you!

[rajjeet]

any idea what's wrong with this code? I'm trying to mock fail on update

test('should return [DataFetchException] when call is unsuccessful',
        () async {
      var docRef = mockFirestoreClient.collection('users').doc();
      await docRef.set(
        AppUserModel.empty().copyWith(id: docRef.id).toMap(),
      );
      var tException = FirebaseException(
          plugin: 'firestore',
          message: 'unable to update',
          code: 'server-unavailable');
      whenCalling(Invocation.method(#update, [
        {'userType', 'worker'}
      ])).on(docRef).thenThrow(tException);
      expect(() => docRef.update({'userType': 'worker'}), throwsA(isA<FirebaseException>()));

getting this output

Expected: throws <Instance of 'FirebaseException'>
  Actual: <Closure: () => Future<void>>
   Which: returned a Future that emitted <null>

[atn832]

@rajjeet Looks like you set up whenCalling with a Set ({'userType', 'worker'}) instead of a Map ({'userType': 'worker'}). See this example:

fake_cloud_firestore/test/fake_cloud_firestore_test.dart

Lines 58 to 62 in 7dc8bcf

	whenCalling(Invocation.method(#set, [
	{'name': 'Bob'}
	])).on(doc).thenThrow(FirebaseException(plugin: 'firestore'));
	expect(() => doc.set({'name': 'Alice'}), returnsNormally);
	expect(() => doc.set({'name': 'Bob'}), throwsA(isA<FirebaseException>()));

[rajjeet]

@atn832 Thanks for catching that!

[cromueloliver02]

CollectionReference.add() isn't supported yet?

[kazerdira]

can anyone give me a hint with this please ( FakeFirebaseFirestore is the type of fakeFirestore )

const mail = AnswerMailModel.empty();

  // arrange
  final collection = fakeFirestore.collection('answers');
  whenCalling(
    Invocation.method(
      #add,
      [
        {
          'request': '',
          'answer': '',
          'requestId': '',
          'createdAt': '',
          'answerUserId': '',
          'requestUserId': '',
          'icon': 0,
        }
      ],
    ),
  ).on(collection).thenThrow(FirebaseException(plugin: 'firestore'));

  expect(
      () => collection.add({
            'request': '5',
            'answer': '',
            'requestId': '',
            'createdAt': '',
          }),
      throwsA(isA<FirebaseException>()));
});

i am getting this error 

Expected: throws <Instance of 'FirebaseException'>
Actual: <Closure: () => Future<DocumentReference<Map<String, dynamic>>>>
Which: returned a Future that emitted <Instance of 'MockDocumentReference<Map<String, dynamic>>'>

or it is all updated ? and changed now 

[atn832]

@kazerdira The way you set up your test was:

1. if someone calls collection.add with { 'request': '', ... }, then throw an exception.
2. then you call collection.add({ 'request': '5' ... }).

As you can see above, the parameters don't match. So no exception will be thrown. If you want to throw an exception next time someone calls collection.add, you can write it more simply like so:

  whenCalling(Invocation.method(#add, null))
    .on(collection).thenThrow(FirebaseException(plugin: 'firestore'));

If you're curious about to use whenCalling/thenThrow, you can read more here: https://pub.dev/packages/firebase_auth_mocks#throwing-exceptions.

Finally since this feature is essentially built, I'll close this thread. Otherwise all of the previous participants will keep getting notified for no reason. Whoever has any issues can create a new ticket.

[dvaruas]

HI @atn832,
I am struggling a bit with having an exception thrown for firebase collection and couldn't quite figure it out.

My setup is, I have a Flutter widget, where there is an "add" button. When I click on the button it calls a class method which simply executes the code below, and there is a viewing area which listens to Labels and creates a ListTile for each document entry in the Labels collection.

Future<void> add(LabelEntry entry) async {
    await firestore.collection('Labels').add(entry.toJson());
  }

The firestore var used above is the FirebaseFirestore which I plug-in during unit-testing with FakeFirebaseFirestore.

Now, my unit-test looks something like this -

testWidgets('failure', (widgetTester) async {
      final firestore = FakeFirebaseFirestore();
      final collectionRef = firestore.collection('Labels');

      whenCalling(Invocation.method(#add, null))
          .on(collectionRef)
          .thenThrow(FirebaseException(plugin: 'firestore'));
      
      // pump the widget in widgetTester
      await widgetTester.tap(find.text('add'));
      await widgetTester.pump();

       // --> this fails, since the click on the 'add' button adds this successfully
      expect(find.byType(ListTile), findsNothing);
    });

Would highly appreciate any help or hints. Thanks.