[camera_avfoundation] enable more than 30 fps #7394

misos1 · 2024-08-12T17:19:15Z

Camera plugin was crashing when I tried to set fps to 60 on most media presets (except maybe on 1280x720, although tested device supports 60 fps for up to 1920x1440 and can do 240 fps on 1280x720) because when is activeVideoMinFrameDuration and activeVideoMaxFrameDuration set to fps outside of what active format supports it throws exception. Now it tries to find a format which supports fps closest to wanted fps and clamps it if it cannot be set to exact value to prevent crashes. It searches for formats with the exact same resolution. For example in format list it can be like "1920x1080 { 3- 30 fps}", "1920x1080 { 3- 60 fps}" and "1920x1080 { 6-120 fps}, but when setting sessionPreset then "1920x1080 { 3- 30 fps}" is selected by default. On the tested device there are 2 "media subtypes" 420f and 420v for each format where the first is denoted as "supports wide color" in debug description and the system has tendency to choose this one. So it tries to preserve the media subtype to what is preferred by the system and this is also added to highestResolutionFormatForCaptureDevice (with lower priority than max resolution/fps). Also there was nested lockForConfiguration and unlockForConfiguration when using ResolutionPreset.max together with setting up fps.

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

stuartmorgan · 2024-08-21T11:46:30Z

@misos1 Could you split this into two different PRs? Having non-trivial changes to two unrelated plugins in the same PR makes review much more difficult.

hellohuanlin · 2024-08-27T18:10:50Z

packages/camera/camera_avfoundation/CHANGELOG.md


+* Adds possibility to use any supported FPS and fixes crash when using unsupported FPS.


*Adds ability to...

hellohuanlin · 2024-08-27T18:14:27Z

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m

+      double bestFrameRate = [self frameRateForFormat:bestFormat closestTo:targetFrameRate];
+      double minDistance = fabs(bestFrameRate - targetFrameRate);
+      int bestSubTypeScore = 1;
+      for (AVCaptureDeviceFormat *format in _captureDevice.formats) {


can you also move this to a helper function?

stuartmorgan · 2024-08-30T18:02:23Z

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m

@@ -211,20 +211,50 @@ - (instancetype)initWithMediaSettings:(FCPPlatformMediaSettings *)mediaSettings
  [_motionManager startAccelerometerUpdates];

  if (_mediaSettings.framesPerSecond) {
+    [_mediaSettingsAVWrapper beginConfigurationForSession:_videoCaptureSession];
+
+    // Possible values for presets are hard-coded in FLT interface having


Why is all this code moving out of the locked section; is that safe?

I did not see anything saying that the order of beginConfiguration and lockForConfiguration is important or that setting the session preset must be under that lock. There is example where it is used in that order: https://developer.apple.com/documentation/avfoundation/avcapturedevice/1389221-activeformat?language=objc

I moved setCaptureSessionPreset outside of lock because it can itself call lockForConfiguration and unlockForConfiguration and I did not see anything about whether that lock is recursive.

Seems lockForConfiguration is actually recursive from my observation (I could not get mentioned exceptions until now) as locking twice and unlocking once does not cause exceptions but unlocking twice yes. So I can revert this change (?).

Also it seems not all commitConfiguration calls go through _mediaSettingsAVWrapper.

stuartmorgan · 2024-08-30T18:05:06Z

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m

-        [_captureDevice unlockForConfiguration];
-        return nil;
+      // find the format which frame rate ranges are closest to the wanted frame rate
+      CMVideoDimensions targetRes = self.videoDimensionsForFormat(_captureDevice.activeFormat);


You've added several calls to self in this init method. The fact that there was already such a call is a problem, and that should not be made worse by adding more.

If there are general, stateless utilities that you want to call from init, they should be turned into freestanding static functions.

stuartmorgan · 2024-08-30T18:06:24Z

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m

-        [_videoCaptureSession commitConfiguration];
-        [_captureDevice unlockForConfiguration];
-        return nil;
+      // find the format which frame rate ranges are closest to the wanted frame rate


Comments need to follow standard sentence rules. https://google.github.io/styleguide/objcguide.html#comments

stuartmorgan · 2024-08-30T18:06:58Z

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m

-        [_captureDevice unlockForConfiguration];
-        return nil;
+      // find the format which frame rate ranges are closest to the wanted frame rate
+      CMVideoDimensions targetRes = self.videoDimensionsForFormat(_captureDevice.activeFormat);


Please don't abbreviate variable names. https://google.github.io/styleguide/objcguide.html#naming

stuartmorgan · 2024-08-30T18:09:44Z

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m

+        FourCharCode subType = CMFormatDescriptionGetMediaSubType(format.formatDescription);
+        int subTypeScore = subType == preferredSubType ? 1 : 0;
+        if (distance < minDistance ||
+            (distance == minDistance && subTypeScore > bestSubTypeScore)) {


I'm confused by this condition. bestSubTypeScore is initialized to 1, and subTypeScore is always either 0 or 1. How could this last piece of the condition ever be true?

I'm not clear on what this score even is; this whole algorithm needs some explanatory comment at the beginning.

if distance < minDistance then bestSubTypeScore can be overwritten by 0. This line explains it: subTypeScore = subType == preferredSubType ? 1 : 0, the score of the subtype is 1 when it is the preferred subtype and 0 otherwise. This is how to choose by minimum distance and then from formats which have equal distance choose that one with preferred subtype, it gives to subtypes some order. Instead of subTypeScore > bestSubTypeScore there could be something like bestSubType != preferredSubType && subType == preferredSubType but that makes it less clear as it deviates from simple way how to order something by multiple properties current.A < other.A || (current.A == other.A && current.B < other.B).

but that makes it less clear

I strongly disagree that abstracting the thing that matters—whether the current type is a preferred type—behind a completely arbitrary numeric value makes this code easier to understand. The code you have there that you described as "less clear" makes sense to me when I read it, whereas I did not understand what this score was, or why something that could only have two possible values was being called a score in the first place.

stuartmorgan · 2024-08-30T18:12:17Z

packages/camera/camera_avfoundation/CHANGELOG.md


+* Adds possibility to use any supported FPS and fixes crash when using unsupported FPS.


Where is the new test that triggers the codepath that used to crash?

When the real captureDevice.activeVideoMinFrameDuration (or ...Max...) is set to value outside of what is supported by active format it immediately throws an objc exception. For tests, mocked wrappers are used so this cannot be triggered. But the test testSettings_ShouldSelectFormatWhichSupports60FPS tries to set 60 fps which actually triggered this crash on my device for media presets other than 1280x720.

stuartmorgan · 2024-09-17T11:06:18Z

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m

@@ -251,6 +254,55 @@ - (instancetype)initWithMediaSettings:(FCPPlatformMediaSettings *)mediaSettings
  return self;
 }

+static void selectBestFormatForRequestedFrameRate(


These helpers should be defined above their call sites; I'm suprised this is even compiling.

stuartmorgan · 2024-09-17T11:07:51Z

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m

@@ -251,6 +254,55 @@ - (instancetype)initWithMediaSettings:(FCPPlatformMediaSettings *)mediaSettings
  return self;
 }

+static void selectBestFormatForRequestedFrameRate(


Per the Obj-C style guide, all declarations require declaration comments explaining them.

In this case, that should include covering what "select" means exactly, given that this doesn't return anything. Presumably it's modifying one or more arguments, but I can't tell without reading the implementation.

stuartmorgan · 2024-09-17T11:08:13Z

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m

+  mediaSettings.framesPerSecond = @(bestFrameRate);
+}
+
+static double bestFrameRateForFormat(AVCaptureDeviceFormat *format, double targetFrameRate) {


This also needs a comment (e.g., explaining what "best" means).

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m

stuartmorgan

Minor nits, but otherwise the code LGTM, thanks!

I'm not very familiar with the camera APIs, so I'll defer to @hellohuanlin to review the behavior itself.

stuartmorgan · 2024-09-26T19:01:42Z

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m

+
+// Finds format with same resolution as current activeFormat for which bestFrameRateForFormat
+// returned frame rate closest to mediaSettings.framesPerSecond. Preferred are formats with the
+// same subtype as current activeFormat. Sets this format as activeFormat and also updates


How about "Sets this format as the activeFormat in captureDevice and [...]"; I wasn't sure what activeFormat was referring to without reading the implementation.

stuartmorgan · 2024-09-26T19:06:04Z

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m

+      isBestSubTypePreferred = isSubTypePreferred;
+    }
+  }
+  if (![bestFormat isEqual:captureDevice.activeFormat]) {


Is there evidence in docs or actual testing that this check is necessary? Standard convention for setters in Obj-C is for them to explicitly handle set-to-same-value as a no-op, so I would expect that we could just unconditionally call the setter and get exactly the same behavior.

I will need to check this, I think it changes sessionPreset to AVCaptureSessionPresetInputPriority even if format is the same as it was (if I remember correctly), please wait with merging until I can get into this and other comments left.

Yes it changes sessionPreset so the idea is to not change it if the format does not need any change. Although maybe it actually should be changed to prevent the session from changing to an active format which no longer supports fps set by the client: "When a client sets the session preset to anything other than AVCaptureSessionPresetInputPriority, the session resumes responsibility for configuring inputs and outputs, and is free to change its inputs' activeFormat as needed."

hellohuanlin · 2024-10-01T19:42:14Z

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m

@@ -537,23 +580,32 @@ - (BOOL)setCaptureSessionPreset:(FCPPlatformResolutionPreset)resolutionPreset
        return NO;
      }
  }
+  CMVideoDimensions size = self.videoDimensionsForFormat(_captureDevice.activeFormat);
+  _previewSize = CGSizeMake(size.width, size.height);
  _audioCaptureSession.sessionPreset = _videoCaptureSession.sessionPreset;
  return YES;
 }

 /// Finds the highest available resolution in terms of pixel count for the given device.


Nit: "... if same pixel count, use the subtype as the tie breaker."

stuartmorgan · 2024-11-05T20:39:34Z

Sorry, this slipped through the cracks on getting landed.

(@hellohuanlin Please make sure to add the autosubmit label when doing the last review on a non-member PR.)

…ages#7394)

flutter/packages@7219431...bb5a258 2024-11-06 [email protected] [ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs (flutter/packages#7430) 2024-11-05 [email protected] Remove use_modular_headers! from Podfiles (flutter/packages#7796) 2024-11-05 [email protected] [camera_avfoundation] enable more than 30 fps (flutter/packages#7394) 2024-11-05 [email protected] Roll Flutter from 8591d0c to 29d40f7 (25 revisions) (flutter/packages#8027) 2024-11-05 [email protected] [ci] Add vector_graphics and flutter_svg to autolabeler (flutter/packages#8025) 2024-11-05 [email protected] [vector_graphics_compiler] wasm compatibility (flutter/packages#8021) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC [email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

* main: (1187 commits) [various] Update example app minSdkVersions (flutter#8035) [go_router] Activate leak testing (flutter#7546) [in_app_purchase_storekit] Add restore purchases and receipts (flutter#7964) [interactive_media_ads] Adds remaining methods for internal wrapper of the Android native `BaseManager` (flutter#7943) [google_sign_in/google_identity_services] Clear-up documentation of callbacks in various APIs and uses of those APIs (flutter#8029) [flutter_svg] wasm compatibility (flutter#8014) Applied Gradle Plugins Declaratively for Multiple Plugin Example Apps (Part 2) (flutter#8019) Roll Flutter from 29d40f7f6826 to 73546b3b71a7 (20 revisions) (flutter#8028) [ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs (flutter#7430) Remove use_modular_headers! from Podfiles (flutter#7796) [camera_avfoundation] enable more than 30 fps (flutter#7394) Roll Flutter from 8591d0c16a6c to 29d40f7f6826 (25 revisions) (flutter#8027) [ci] Add vector_graphics and flutter_svg to autolabeler (flutter#8025) [vector_graphics_compiler] wasm compatibility (flutter#8021) [vector_graphics*] Relax dependency constraints of vector_graphics, vector_graphics_codec, vector_graphics_compiler, flutter_svg (flutter#8018) [various] Add `missing_code_block_language_in_doc_comment` lint to flutter/packages. (flutter#6473) [various] Update example apps to Kotlin 1.9.0 (flutter#7998) [go_router] add current state getter (flutter#7651) Applied Gradle Plugins Declaratively for Multiple Plugin Example Apps (flutter#7968) Roll Flutter from f86b77721524 to 8591d0c16a6c (16 revisions) (flutter#8015) ... # Conflicts: # packages/quick_actions/quick_actions/CHANGELOG.md # packages/quick_actions/quick_actions_ios/CHANGELOG.md # packages/quick_actions/quick_actions_platform_interface/CHANGELOG.md

github-actions bot added p: camera platform-ios platform-macos p: video_player labels Aug 12, 2024

misos1 force-pushed the more_fps branch 8 times, most recently from 75250e0 to 3f0e949 Compare August 14, 2024 17:43

enable more than 30 fps

6c769f9

misos1 force-pushed the more_fps branch from 3f0e949 to 6c769f9 Compare August 14, 2024 18:35

misos1 marked this pull request as ready for review August 14, 2024 18:46

misos1 requested a review from hellohuanlin as a code owner August 14, 2024 18:46

misos1 added 2 commits August 14, 2024 21:11

Merge branch 'main' into more_fps

4920305

correctly set _previewSize

4f1483f

misos1 added 2 commits August 21, 2024 17:50

Merge branch 'main' into more_fps

6c210c9

split camera changes from more_fps

30a4856

misos1 changed the title ~~[video_player_avfoundation, camera_avfoundation] enable more than 30 fps~~ [camera_avfoundation] enable more than 30 fps Aug 21, 2024

github-actions bot removed the p: video_player label Aug 21, 2024

hellohuanlin reviewed Aug 27, 2024

View reviewed changes

hellohuanlin requested a review from stuartmorgan August 27, 2024 18:15

stuartmorgan requested changes Aug 30, 2024

View reviewed changes

extract format selection into function and more

5e1f249

misos1 force-pushed the more_fps branch from 8de5b00 to dd6ca09 Compare September 5, 2024 19:19

revert reordering lockDevice with beginConfigurationForSession

5007358

misos1 force-pushed the more_fps branch from dd6ca09 to 5007358 Compare September 5, 2024 19:21

misos1 added 2 commits September 5, 2024 21:31

fix format

65f8113

fix format

27868df

misos1 requested review from stuartmorgan and hellohuanlin September 5, 2024 19:52

stuartmorgan requested changes Sep 17, 2024

View reviewed changes

misos1 added 2 commits September 19, 2024 19:33

add comments and small changes

3983273

fix format

1d8f05a

misos1 requested a review from stuartmorgan September 19, 2024 18:04

stuartmorgan approved these changes Sep 26, 2024

View reviewed changes

hellohuanlin reviewed Oct 1, 2024

View reviewed changes

hellohuanlin approved these changes Oct 1, 2024

View reviewed changes

misos1 added 3 commits October 12, 2024 21:08

change comments

cc632ec

Merge branch 'main' into more_fps

f36d2dd

change activeFormat always if framesPerSecond is set

0125d77

stuartmorgan added the autosubmit Merge PR when tree becomes green via auto submit App label Nov 5, 2024

auto-submit bot merged commit 719bd84 into flutter:main Nov 5, 2024
76 checks passed

engine-flutter-autoroll mentioned this pull request Nov 6, 2024

Roll Packages from 721943194945 to bb5a25815cae (6 revisions) flutter/flutter#158267

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 6, 2024

719bd8434 [camera_avfoundation] enable more than 30 fps (flutter/pack…

0e50717

…ages#7394)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[camera_avfoundation] enable more than 30 fps #7394

[camera_avfoundation] enable more than 30 fps #7394

misos1 commented Aug 12, 2024 •

edited

Loading

stuartmorgan commented Aug 21, 2024

hellohuanlin Aug 27, 2024

hellohuanlin Aug 27, 2024

stuartmorgan Aug 30, 2024

misos1 Sep 4, 2024 •

edited

Loading

misos1 Sep 5, 2024

stuartmorgan Aug 30, 2024

stuartmorgan Aug 30, 2024

stuartmorgan Aug 30, 2024

stuartmorgan Aug 30, 2024

misos1 Aug 30, 2024 •

edited

Loading

stuartmorgan Aug 30, 2024

stuartmorgan Aug 30, 2024

misos1 Sep 4, 2024

stuartmorgan Sep 17, 2024

stuartmorgan Sep 17, 2024

stuartmorgan Sep 17, 2024

stuartmorgan left a comment

stuartmorgan Sep 26, 2024

stuartmorgan Sep 26, 2024

misos1 Oct 1, 2024 •

edited

Loading

misos1 Oct 12, 2024

hellohuanlin Oct 1, 2024

stuartmorgan commented Nov 5, 2024


		* Adds possibility to use any supported FPS and fixes crash when using unsupported FPS.

[camera_avfoundation] enable more than 30 fps #7394

[camera_avfoundation] enable more than 30 fps #7394

Conversation

misos1 commented Aug 12, 2024 • edited Loading

Pre-launch Checklist

stuartmorgan commented Aug 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

misos1 Sep 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

misos1 Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuartmorgan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

misos1 Oct 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuartmorgan commented Nov 5, 2024

misos1 commented Aug 12, 2024 •

edited

Loading

misos1 Sep 4, 2024 •

edited

Loading

misos1 Aug 30, 2024 •

edited

Loading

misos1 Oct 1, 2024 •

edited

Loading