Flutter : FirebaseMessaging "onBackgroundMessage" with FFmpegKit

[ujwalparker]

Description
In an app (Flutter) where we have configured FirebaseMessaging (https://pub.dev/packages/firebase_messaging/) with the onBackgroundMessage handler the FFmpegKit plugin is not usable as there seems to be some conflict due to which the EventChannel which is created by this plugin is rendered null

Expected behavior
Methods like FFmpegKit.executeAsync or FFprobeKit.getMediaInformationAsync or others execute with a result.

Current behavior
The methods FFmpegKit.executeAsync or FFprobeKit.getMediaInformationAsync or others fail with the following sample error

W/ffmpeg-kit-flutter(32468): ResultHandler can not send event {FFmpegKitExecuteCallbackEvent={createTime=1635353254392, startTime=1635353254414, sessionId=1, type=1, command= -i /data/user/0/com.arre.voice/app_flutter/effects/drumroll.wav -i /data/user/0/com.arre.voice/app_flutter/effects/cheer.wav -i /data/user/0/com.arre.voice/app_flutter/effects/boo.wav -filter_complex amix=inputs=3:duration=longest -y /storage/emulated/0/Android/data/com.arre.voice/files/[#6d178].m4a}} on a null event sink.

To Reproduce
Just add FirebaseMessaging to the app and implement onBackgroundMessage handler.

Screenshots

Logs
successAsync gets called for any method but eventSink is render null thus the error message printed by the plugin
Log.w(LIBRARY_NAME, String.format("ResultHandler can not send event %s on a null event sink.", object));

Environment

• Platform: Flutter Android
• Architecture: arm64, and others too
• Version v4.5.0
• Source branch v4.5.0

Other
FirebaseMessaging has 3 handlers as below ( firebase_messaging: ^10.0.7)

• FirebaseMessaging.onMessage
• FirebaseMessaging.onMessageOpenedApp
• FirebaseMessaging.onBackgroundMessage

When you just do not set the "FirebaseMessaging.onBackgroundMessage" handler the plugin works fine.

[tanersener]

Can you provide more details about which ffmpeg-kit methods you're calling?

And, does calling another method like await FFmpegKitConfig.getFFmpegVersion() before execute methods makes a difference?

[ujwalparker]

@tanersener

Tried with following methods, most of them which generate a future session

• FFmpegKitConfig.getFFmpegVersion()
• FFmpegKit.executeAsync()
• FFprobeKit.getMediaInformationAsync()

Yes FFmpegKitConfig.getFFmpegVersion() works. with or without FCM background and prints the following
v4.5-dev-2008-g90da43557f

Put together a sample app for this use case with 3 branches

https://github.com/ujwalparker/audio-mixer/tree/ffmpeg-kit/issues-209-audio-mixer

• Includes the sample code. All methods implemented work expected.

https://github.com/ujwalparker/audio-mixer/tree/ffmpeg-kit/issues-209-fcm-added

• On Adding FCM only FFmpegKitConfig.getFFmpegVersion() works and the others do not. On the UI in sample app only the first button works and displays the response from the function call. Others fail with the null sink error

https://github.com/ujwalparker/audio-mixer/tree/ffmpeg-kit/issues-209-fcm-background-disabled

• By just disabling FirebaseMessaging.onBackgroundMessage - the functions all work

Only thing not executed is below lines of code

Future<void> _firebaseMessagingBackgroundHandler(RemoteMessage message) async { await Firebase.initializeApp(); print('Handling a background message ${message.messageId}'); }

You can have a look at the change sheet for the above 3 branches with changes made. (issues-209-fcm-added and issues-209-fcm-background-disabled)

[tanersener]

FFmpegKitFlutterPlugin supports both V1 and V2 Android embeddings. There are also logs printed during init and register methods. As far as I see, when FirebaseMessaging.onBackgroundMessage is enabled a second FFmpegKitFlutterPlugin is created. But that second instance is not properly registered; EventSink which is necessary to send back events is null. That's why you have this error.

I suggest asking this behaviour to Firebase devs. There are similar errors reported for other plugins when they are registered through FirebaseMessaging.onBackgroundMessage.

[ujwalparker]

@tanersener I do not think this has any interference with Firebase. I spent some time to understand if there is any conflict in the method channel naming etc. but could not find any.

I left the onBackgroundMessage handler active and tried to invoke FFprobeKit.getMediaInformationAsync using the "ffmpeg_kit_flutter_min" package this time (does not have relevance to changing the package - I was just trying to reduce the APK size - unrelated)

1. On app launch - I see that Firebase Messaging background service has initialized
   
2. On invoking FFprobeKit.getMediaInformationAsync, the FFmpegKitFlutterPlugin is invoked. The following logs also get printed
   I/flutter (25239): Loading ffmpeg-kit-flutter. D/ffmpeg-kit-flutter(25239): FFmpegKitFlutterPlugin started listening to events on io.flutter.plugin.common.EventChannel$IncomingStreamRequestHandler$EventSinkImplementation@f7b8302. I/flutter (25239): Loaded ffmpeg-kit-flutter-android-min-arm64-v8a-4.5.0.

but the execution fails

Why I say its unrelated - because the onListen() is called on the FFmpegKitFlutterPlugin and the log clearly prints the eventSink id. which mean that the successAsync was invoked by asyncMediaInformationSessionExecute and AsyncWriteToPipeTask also invoked.

I could only see one FFmpegKitFlutterPlugin instance created - and that was after the call was made to FFprobeKit.getMediaInformationAsync on a user interaction of a button.

Attached logs, 1st is after launch of app where Firebase plugin is initialized. 2nd on invoking getMediaInformationAsync

1. FlutterFirebaseMessagingBackgroundService.txt
2. FFprobeKit.getMediaInformationAsync.txt

Thoughts?

[tanersener]

Why don't you use the version we have on issue-209 branch, it has more detailed logs?

[ujwalparker]

Ok will use that and report back.

[ifananomali]

@tanersener @ujwalparker I got this error too when using FFmpeg with Firebase Messaging 'background handler'. Hope to see a solution for this.

[ujwalparker]

@tanersener those logs helped. And yes your initial comment about two instances of FFmpegKitFlutterPlugin objects created is right - but this is expected behavior IMO (but why? more on that towards the end)

Firebase Messaging background service triggers the second object creation and that is where the bug manifests.

The issue is in the way eventSink is used across and since it is reinitialized when the new FFmpegKitFlutterPlugin object gets created, When the FFmpegKitConfig methods are called they are sending the result back on a eventSink which is now null.

I have thought of a crude low impact fix which solves the issue by way of using a static eventSink - to see if this fixes the issue and it does fix - this is crude because this uses a singleton initializer just to use a static eventSink. Checkout the commit @ https://github.com/ujwalparker/ffmpeg-kit/commit/8cc4bd0608551b44142aa25d7173cb7fa7b40fe0

This is not a fix Firebase will accept or review IMO - although I find it weird on why implementation of the background callback triggers an initialization :P. Found an issue which was closed for lack of support on their repo.

I have not raised this as a PR as I would not call this as an efficient fix as I do not fully understand the architecture of this plugin. It might possibly be an easier fix from your POV. Let me know if with this you can see a possible solution which is not as crude as the commit.

Here is a gist with the above fix working with _firebaseMessagingBackgroundHandler enabled
https://gist.github.com/ujwalparker/5afb7b29611119fa90ed3cd38764f3df

cc. @ifananomali

[tanersener]

flutter plugin of ffmpeg-kit registers itself in the same way as the official flutter.dev plugins. None of those plugins utilise static channels, event sinks or plugins. I want to keep it in that way, if possible.

I made a minor change in the issue-209 branch which might help you in your case. Try that please. It's the best I can do at the moment.

[ujwalparker]

@tanersener the change on that branch also works !!. You can take this into the main branch as it will solve this issue for all who want to use this plugin and have FCM integrated (which should be like 90% developers :P)

[yanivshaked]

I'm also using FCM, and had the same error message. Using the branch:

  # FFmpeg Kit for Flutter. Supports Android, iOS and macOS platforms.
  ffmpeg_kit_flutter:
    git:
      url: https://github.com/tanersener/ffmpeg-kit.git
      path: "flutter/flutter"
      ref: "2cbc498"

the error message is fixed, but I'm getting null when calling getReturnCode:

    var convertion = await FFmpegKit.executeWithArgumentsAsync(
        [
          '-loglevel',
          'error',
          '-y',
          '-f',
          's16le',
          '-ac',
          '${AudioConstants.NumChannels}',
          '-ar',
          '${AudioConstants.SampleRate}',
          '-i',
          absoluteInputFile,
          '-c:a',
          'aac',
          '-q:a',
          '3',
          absoluteOutputFile,
        ]);
    returnCode = await convertion.getReturnCode();
    if (!(returnCode!.isValueSuccess())) {
      logger!.warn("convertToAAC with FFmpeg failed. absoluteInputFile = $absoluteInputFile, absoluteOutputFile = $absoluteOutputFile");
      return null;
    }
    return outputFile;

Log:

I/flutter (21865): Loading ffmpeg-kit-flutter.
D/ffmpeg-kit-flutter(21865): FFmpegKitFlutterPlugin com.arthenica.ffmpegkit.flutter.FFmpegKitFlutterPlugin@997c2bf started listening to events on io.flutter.plugin.common.EventChannel$IncomingStreamRequestHandler$EventSinkImplementation@b97950a.
I/flutter (21865): Loaded ffmpeg-kit-flutter-android-https-arm64-v8a-4.5.0.
E/flutter (21865): [ERROR:flutter/lib/ui/ui_dart_state.cc(209)] Unhandled Exception: Null check operator used on a null value
E/flutter (21865): #0      AudioCoreMobile.convertToAAC (package:tw_app/Services/AudioCore/AudioCoreMobile.dart:279:21)
E/flutter (21865): <asynchronous suspension>
E/flutter (21865): #1      AudioRecorder.stopAudioRecorder (package:tw_app/UI/Mobile/Widgets/General/AudioRecorder.dart:212:29)
E/flutter (21865): <asynchronous suspension>
E/flutter (21865): #2      AudioBarState._onAudioRecordTapped (package:tw_app/UI/Mobile/Widgets/Chat/InputBar/AudioBar.dart:1679:7)
E/flutter (21865): <asynchronous suspension>
E/flutter (21865): #3      AudioBarState.getAudioRecordProgressBarButtonsWidget.<anonymous closure> (package:tw_app/UI/Mobile/Widgets/Chat/InputBar/AudioBar.dart:2078:13)
E/flutter (21865): <asynchronous suspension>
E/flutter (21865):
I/ffmpeg-kit-flutter(21865): emitStatistics called on FFmpegKitFlutterPlugin com.arthenica.ffmpegkit.flutter.FFmpegKitFlutterPlugin@997c2bf
I/ffmpeg-kit-flutter(21865): emitStatistics called on FFmpegKitFlutterPlugin com.arthenica.ffmpegkit.flutter.FFmpegKitFlutterPlugin@997c2bf
I/ffmpeg-kit-flutter(21865): emitSession called on FFmpegKitFlutterPlugin com.arthenica.ffmpegkit.flutter.FFmpegKitFlutterPlugin@997c2bf

Note that from the logs it looks like the await did not really wait for the operation to complete.

@ujwalparker Did you check the return code?

[ujwalparker]

Yeah the callback did return success. I did not use the function you have used. Have you checked if the command works and does not have any issue?

[tanersener]

I updated the development-flutter branch with the change tested in issue-209.

Remember that the change we made fixes only the cases where Firebase is configured using onBackgroundMessage and ffmpeg-kit is called by a foreground view. If you want to call ffmpeg-kit from a background service/view/widget make sure that ffmpeg-kit is registered successfully first.

@yanivshaked returnCode can be null. See the documentation we added for the getReturnCode method.

[yanivshaked]

@yanivshaked returnCode can be null. See the documentation we added for the getReturnCode method.

Thanks. Indeed it can be null, but in my case it means that the operation did not proceed/perform as planned. If this is the first call to the ffmpeg-kit, either the session has not started or failed.
Do the last 3 lines in my log imply that the operation is successfully completed?

[tanersener]

@yanivshaked Those three lines show that two statistics entries were produced and a session was completed natively. And those 3 events were forwarded through the event channel. But we can't see whether execution was successful or not in the logs. You have the session object, why don't you use the other methods to understand the problem?

[yanivshaked]

@tanersener I have added the following inquires to the session object:

var logs = await conversion.getAllLogs();
print("Number of logs: ${logs.length}");
for (var log in logs) {
    print("Log: Level:${log.getLevel()} SessionId:${log.getSessionId()} Message:${log.getMessage()}");
}
print("Failed StackTrace is ${await conversion.getFailStackTrace()}");
print("Ouput is ${await conversion.getOutput()}");
returnCode = await conversion.getReturnCode();
print("Return code is $returnCode");

And got the following:

I/flutter (17584): Loaded ffmpeg-kit-flutter-android-https-arm64-v8a-4.5.0.
I/flutter (17584): Number of logs: 0
I/flutter (17584): Failed StackTrace is null
I/flutter (17584): Ouput is
I/flutter (17584): Return code is null
I/ffmpeg-kit-flutter(17584): emitStatistics called on FFmpegKitFlutterPlugin com.arthenica.ffmpegkit.flutter.FFmpegKitFlutterPlugin@8bc6066
I/ffmpeg-kit-flutter(17584): emitStatistics called on FFmpegKitFlutterPlugin com.arthenica.ffmpegkit.flutter.FFmpegKitFlutterPlugin@8bc6066
I/ffmpeg-kit-flutter(17584): emitSession called on FFmpegKitFlutterPlugin com.arthenica.ffmpegkit.flutter.FFmpegKitFlutterPlugin@8bc6066

But when adding a short delay of 2 seconds (await Future.delayed(Duration(seconds: 2));), after the call to FFmpegKit.executeWithArgumentsAsync and before the inquires, problem is solved, and I am getting:

I/flutter (17584): Loaded ffmpeg-kit-flutter-android-https-arm64-v8a-4.5.0.
I/ffmpeg-kit-flutter(17584): emitStatistics called on FFmpegKitFlutterPlugin com.arthenica.ffmpegkit.flutter.FFmpegKitFlutterPlugin@8bc6066
I/ffmpeg-kit-flutter(17584): emitStatistics called on FFmpegKitFlutterPlugin com.arthenica.ffmpegkit.flutter.FFmpegKitFlutterPlugin@8bc6066
I/ffmpeg-kit-flutter(17584): emitSession called on FFmpegKitFlutterPlugin com.arthenica.ffmpegkit.flutter.FFmpegKitFlutterPlugin@8bc6066
I/flutter (17584): Number of logs: 0
I/flutter (17584): Failed StackTrace is null
I/flutter (17584): Ouput is
I/flutter (17584): Return code is 0

This show that the await operation does not really wait for the operation to complete, you are probably using some async operation which is not awaited. What do you suggest?

[yanivshaked]

If you look at FFmpegKitFlutterPlugin.java:

    protected void asyncFFmpegSessionExecute(@NonNull final Integer sessionId, @NonNull final Result result) {
        final Session session = FFmpegKitConfig.getSession(sessionId.longValue());
        if (session == null) {
            resultHandler.errorAsync(result, "SESSION_NOT_FOUND", "Session not found.");
        } else {
            if (session instanceof FFmpegSession) {
                FFmpegKitConfig.asyncFFmpegExecute((FFmpegSession) session);
                resultHandler.successAsync(result, null);
            } else {
                resultHandler.errorAsync(result, "NOT_FFMPEG_SESSION", "A session is found but it does not have the correct type.");
            }
        }
    }

asyncFFmpegExecute (Line 997) starts execution of ffmpeg operation, but does not wait for its completion?

[tanersener]

@yanivshaked I think there is a misunderstanding about async methods we have for flutter under FFmpegKit and FFprobeKit. All async execute methods we have under those classes are responsible of only starting an execution. That's why they are called async. They return immediately and having await there doesn't change anything.

If you want be notified about the completion then you can provide an ExecuteCallback to the method call. Or, creating a wait until session.getState() is completed or failed method is also possible. This was a design choice rather than a bug.

[yanivshaked]

Ok, that makes sense, in light of the code. Thank you for clarifying.
Adding:

  Future<ReturnCode?> waitForSessionCompletion(FFmpegSession conversion) async {
    SessionState? state;
    while (state != SessionState.completed && state != SessionState.failed) {
      state = await conversion.getState();
    }
    return await conversion.getReturnCode();
  }

seems to be working.

[naveenlearner38]

I am also facing the same issue while working with FirebaseMessaging. Is there any way to solve this issue?

[shilangyu]

Are there any workarounds available other than using the dev branch and building the package manually (we are using LTS)? If not, what is the the ETA for the fix to be released? Thank you

[tanersener]

We are publishing the dates for the releases under the Projects page. It is January 2022 for the next one.

[mytasksandprojects]

Ok, that makes sense, in light of the code. Thank you for clarifying. Adding:

  Future<ReturnCode?> waitForSessionCompletion(FFmpegSession conversion) async {
    SessionState? state;
    while (state != SessionState.completed && state != SessionState.failed) {
      state = await conversion.getState();
    }
    return await conversion.getReturnCode();
  }

seems to be working.

Where to add this function??

[shilangyu]

If anyone cannot wait till January (we cannot), I forked the repo to expose the pre built packages (no other changes were made). You can reference it in pubspec.yaml:

  ffmpeg_kit_flutter:
    git:
      url: https://github.com/shilangyu/ffmpeg-kit
      ref: development-flutter
      path: prebuilt/bundle-flutter-lts/default

and replacing path: to whatever variant you need

[herrmayr]

Ok, that makes sense, in light of the code. Thank you for clarifying. Adding:

  Future<ReturnCode?> waitForSessionCompletion(FFmpegSession conversion) async {
    SessionState? state;
    while (state != SessionState.completed && state != SessionState.failed) {
      state = await conversion.getState();
    }
    return await conversion.getReturnCode();
  }

seems to be working.

Where to add this function??

I'm also wondering. Would you mind elaborating so we can learn how to "wait" for the execution to finish?

[mytasksandprojects]

Ok, that makes sense, in light of the code. Thank you for clarifying. Adding:

  Future<ReturnCode?> waitForSessionCompletion(FFmpegSession conversion) async {
    SessionState? state;
    while (state != SessionState.completed && state != SessionState.failed) {
      state = await conversion.getState();
    }
    return await conversion.getReturnCode();
  }

seems to be working.

Where to add this function??

I'm also wondering. Would you mind elaborating so we can learn how to "wait" for the execution to finish?

Why you taking like this i asked because i don't know 😕

[herrmayr]

Ok, that makes sense, in light of the code. Thank you for clarifying. Adding:

  Future<ReturnCode?> waitForSessionCompletion(FFmpegSession conversion) async {
    SessionState? state;
    while (state != SessionState.completed && state != SessionState.failed) {
      state = await conversion.getState();
    }
    return await conversion.getReturnCode();
  }

seems to be working.

Where to add this function??

I'm also wondering. Would you mind elaborating so we can learn how to "wait" for the execution to finish?

Why you taking like this i asked because i don't know 😕

I was hoping @yanivshaked could help us learn by explaining how exactly he did it!

[yanivshaked]

Sorry for my late answer.

Since the FFmpegKit.executeAsync method only starts the operation but does not wait for its completion, you can do the following:

1. Add the following method anywhere in your project. (I'm using an abstraction layer over FFmpegKit operations):

  Future<ReturnCode?> waitForSessionCompletion(FFmpegSession conversion) async {
    SessionState? state;
    while (state != SessionState.completed && state != SessionState.failed) {
      state = await conversion.getState();
    }
    return await conversion.getReturnCode();
  }

2. When you want to use FFmpegKit.executeAsync and wait for the operation to complete, simply use the return value from FFmpegKit.executeAsync and pass it to waitForSessionCompletion, for example:

    var ffmpegSession =
        await FFmpegKit.executeAsync('-y -loglevel warning -f concat -safe 0 -i $viaFilePath -c:a aac \"$absoluteOutputFile\"');
    var returnCode = await waitForSessionCompletion(ffmpegSession);
    if (returnCode == null || !(returnCode.isValueSuccess())) {
      logger!.warn(
          "concatenateAudio with FFmpeg failed. returnCode=$returnCode log=${await ffmpegSession.getAllLogsAsString()}, viaFilePath = $viaFilePath, fileContext = $fileContent, absoluteOutputFile = $absoluteOutputFile");
      return false;
    }
    return true;

I hope this helps, let me know if you need more clarifications.

[herrmayr]

Hi @yanivshaked ,
thank you very much for your very helpful reply! Now I understand how it works - genius solution, thank you!

In the meantime, I have solved in in a far less elegant way, which also works and maybe it's interesting for someone because it's quite simple and low-key, so it might also be helpful for some people:

I use a variable and a do-while-loop to check if the execution has finished yet:

bool _finished = false;

FFmpegKit.executeAsync(ffmpegCommand, (Session session) async {
_finished = true;
};

do {
    await Future.delayed(Duration(milliseconds: 50));
  } while (!_finished); 
 return _finished;

[yanivshaked]

Two comments:

1. @herrmayr, your solution missed out the fact that SessionState has several different values:

enum SessionState { created, running, failed, completed }

So your code doesn't recognize the completion event correctly, and will stop when created or running is sent.

2. Looking back at the method I have suggested to use:

 Future<ReturnCode?> waitForSessionCompletion(FFmpegSession conversion) async {
    SessionState? state;
    while (state != SessionState.completed && state != SessionState.failed) {
      state = await conversion.getState();
    }
    return await conversion.getReturnCode();
  }

It is not bullet proof against situations in which the FFmpegKit.executeAsync crashes and doesn't return any value. The while loop should be protected with a timeout to make sure we don't wait here forever. I suggest the following (didn't test the next code):

  Future<ReturnCode?> timedWaitForSessionCompletion(FFmpegSession ffmpegSession, int timeoutMsec) async {
    var fTimer = Future.delayed(Duration(milliseconds: timeoutMsec), () => null);
    var fOperation = waitForSessionCompletion(ffmpegSession);
    return await Future.any([fTimer, fOperation]);
  }

This method wraps the operation of waitForSessionCompletion with timeout, making sure we don't get lost here.
@tanersener What do you think about this one?

[tanersener]

@yanivshaked I think I'll implement the execute() methods in Flutter. So, you guys won't have to use additional methods to wait the completion. I still don't understand the issue with ExecuteCallback by the way. I'm using it nicely.

Anyway, I haven't tested your updated waitForSessionCompletion but it seems like it times out as expected.

It is not bullet proof against situations in which the FFmpegKit.executeAsync crashes and doesn't return any value.

Let me know if there are cases where FFmpegKit.executeAsync crashes or returns null.

[tanersener]

This issue is fixed in the v4.5.1 release.

execute methods are also implemented in the new release. You don't have to wait until the session is completed.

FFmpegKit.execute(commad).then((session) {
    // called when command has executed
);