control_transfer returned error, index: 768, error: Broken pipe, number: 32 and then repetitive Out of frame resources! Exception

[juamar]

---

Required Info
Camera Model	D415
Firmware Version	I don't know
Operating System & Version	Win 10
Platform	Android (ARM64!)
SDK Version	2.50.0
Language	C#
Segment	Smartphone

Issue Description

I keep on making Librealsense work on a Xamarin Android application. This time i want to report some issue i'm having, in order to check with community if there is a way to improve my situation.

I'm trying to develop a nice and simple HelloWorld application as coded in the examples from the Android Wrapper but in Xamarin which calculates distance information from an object in front of the camera. I'm using the Android AAR library by using Xamarin Android Library Binder. I was facing an issue the previous days, I was just being able to retrieve the first 15 frames from the camera (StreamType.Depth). After that, i was catching the following exception:

Frame didn't arrive within 5000

Seems to be that the Pipeline object is not able to receive more Frames. Executing the test with the Plain Simple Android Example App was working perfectly, so it may be some software problem, rather than a Hardware problem, as i have seen in previous Issues. Also, i'm not doing anything weird in my code, the logic it's almost copy paste from the example, so i don't know what can be wrong.

Nowadays, as a workarround, i've made some PipeManager, so when we reach 15 frames, I replace the current Pipeline with a new one, and i keep switching them using Threads in a way that no Pipeline's management is perceibed for the user.

So before i was having this:

private void stream0()
    {
        int i = 0;
        Pipeline pipe = new Pipeline();
        pipe.Start();

        while (!mStreamingThread.StopRequest)
        {
            using (FrameSet frames = pipe.WaitForFrames())
            {

                using (Frame f = frames.First(StreamType.Depth))
                {
                    DepthFrame depth = (DepthFrame)f.As(Extension.DepthFrame);
                    float deptValue = depth.GetDistance(depth.Width / 2, depth.Height / 2);

                    i++;
                    this.RunOnUiThread(() => realsenseFragment.PrintToTest("Distance: " + deptValue + "\n" + i));
                }
            }
            Thread.Sleep(1000 / 30);
        }
        pipe.Stop();
    }

And now i have this, with my Custom pipe's manager:

private void stream2()
    {
        int i = 0;
        PipeManager pipeManager = new PipeManager();

        while (!mStreamingThread.StopRequest)
        {
            using (FrameSet frames = pipeManager.ActualPipeline.WaitForFrames())
            {
                using (Frame f = frames.First(StreamType.Depth))
                {
                    DepthFrame depth = (DepthFrame)f.As(Extension.DepthFrame);
                    float deptValue = depth.GetDistance(depth.Width / 2, depth.Height / 2);

                    i++;
                    this.RunOnUiThread(() => realsenseFragment.PrintToTest("Distance: " + deptValue + "\n" + i));
                }
            }
            //Thread.Sleep(1000 / 30);
        }
        pipeManager.Stop();
    }

This is the Logcat generated from an execution from my app using stream0 method:

Forwarding debugger port 8862
Detecting existing process
> am start -a "android.intent.action.MAIN" -c "android.intent.category.LAUNCHER" -n "com.concatel.mauiExample/crc646bee10ba88d1aa8b.MainActivity"
> Starting: Intent { act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] cmp=com.concatel.mauiExample/crc646bee10ba88d1aa8b.MainActivity }
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/mauiExample.dll
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/Mono.Android.dll [External]
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/System.Runtime.dll [External]
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/Java.Interop.dll [External]
[zygote64] Late-enabling -Xcheck:jni
[debug-app-helper] Checking if libmonodroid was unpacked to /data/app/com.concatel.mauiExample-JU1GGpkWQKlnLNVxLQp0tg==/lib/arm64/libmonodroid.so
[debug-app-helper] Native libs extracted to /data/app/com.concatel.mauiExample-JU1GGpkWQKlnLNVxLQp0tg==/lib/arm64, assuming application/android:extractNativeLibs == true
[debug-app-helper] Setting up for DSO lookup in app data directories
[debug-app-helper] Added filesystem DSO lookup location: /data/app/com.concatel.mauiExample-JU1GGpkWQKlnLNVxLQp0tg==/lib/arm64
[debug-app-helper] Using runtime path: /data/app/com.concatel.mauiExample-JU1GGpkWQKlnLNVxLQp0tg==/lib/arm64
[debug-app-helper] checking directory: `/data/user/0/com.concatel.mauiExample/files/.__override__/lib`
[debug-app-helper] directory does not exist: `/data/user/0/com.concatel.mauiExample/files/.__override__/lib`
[debug-app-helper] Checking whether Mono runtime exists at: /data/user/0/com.concatel.mauiExample/files/.__override__/libmonosgen-2.0.so
[debug-app-helper] Checking whether Mono runtime exists at: /data/app/com.concatel.mauiExample-JU1GGpkWQKlnLNVxLQp0tg==/lib/arm64/libmonosgen-2.0.so
[debug-app-helper] Mono runtime found at: /data/app/com.concatel.mauiExample-JU1GGpkWQKlnLNVxLQp0tg==/lib/arm64/libmonosgen-2.0.so
[zygote64] Attempt to remove non-JNI local reference, dumping thread
[DOTNET] JNI_OnLoad: JNI_OnLoad in pal_jni.c
[DOTNET] GetOptionalMethod: optional method setApplicationProtocols ([Ljava/lang/String;)V was not found
[DOTNET] GetOptionalMethod: optional method getApplicationProtocol ()Ljava/lang/String; was not found
[monodroid] Creating public update directory: `/data/user/0/com.concatel.mauiExample/files/.__override__`
[zygote64] Attempt to remove non-JNI local reference, dumping thread
[monodroid-debug] Trying to initialize the debugger with options: --debugger-agent=transport=dt_socket,loglevel=0,address=127.0.0.1:8862,server=y,embedding=1
[monodroid-assembly] open_from_bundles: failed to load assembly mauiExample.dll
[monodroid-gc] GREF GC Threshold: 46080
[monodroid-assembly] open_from_bundles: failed to load assembly Mono.Android.dll
[monodroid-assembly] open_from_bundles: failed to load assembly System.Runtime.dll
[monodroid-assembly] open_from_bundles: failed to load assembly Java.Interop.dll
[monodroid-assembly] open_from_bundles: failed to load assembly System.Collections.dll
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/System.Collections.dll [External]
[monodroid-assembly] open_from_bundles: failed to load assembly System.Threading.dll
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/System.Threading.dll [External]
[monodroid-assembly] open_from_bundles: failed to load assembly System.Runtime.InteropServices.dll
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/System.Runtime.InteropServices.dll [External]
[monodroid-assembly] open_from_bundles: failed to load assembly System.Threading.Thread.dll
[monodroid-assembly] open_from_bundles: failed to load assembly System.Diagnostics.StackTrace.dll
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/System.Threading.Thread.dll [External]
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/System.Diagnostics.StackTrace.dll [External]
[zygote64] Attempt to remove non-JNI local reference, dumping thread
[monodroid-assembly] open_from_bundles: failed to load assembly Xamarin.AndroidX.AppCompat.dll
[monodroid-assembly] open_from_bundles: failed to load assembly Xamarin.AndroidX.Fragment.dll
[monodroid-assembly] open_from_bundles: failed to load assembly Xamarin.AndroidX.Activity.dll
[monodroid-assembly] open_from_bundles: failed to load assembly Xamarin.AndroidX.Core.dll
[monodroid-assembly] open_from_bundles: failed to load assembly mscorlib.dll
[monodroid-assembly] open_from_bundles: failed to load assembly Xamarin.AndroidX.Lifecycle.Common.dll
[monodroid-assembly] open_from_bundles: failed to load assembly Xamarin.AndroidX.Lifecycle.ViewModel.dll
[monodroid-assembly] open_from_bundles: failed to load assembly Xamarin.AndroidX.SavedState.dll
[monodroid-assembly] open_from_bundles: failed to load assembly Xamarin.AndroidX.DrawerLayout.dll
[monodroid-assembly] open_from_bundles: failed to load assembly Xamarin.AndroidX.CustomView.dll
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/Xamarin.AndroidX.AppCompat.dll [External]
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/Xamarin.AndroidX.Fragment.dll [External]
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/Xamarin.AndroidX.Activity.dll [External]
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/Xamarin.AndroidX.Core.dll [External]
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/mscorlib.dll [External]
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/Xamarin.AndroidX.Lifecycle.Common.dll [External]
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/Xamarin.AndroidX.Lifecycle.ViewModel.dll [External]
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/Xamarin.AndroidX.SavedState.dll [External]
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/Xamarin.AndroidX.DrawerLayout.dll [External]
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/Xamarin.AndroidX.CustomView.dll [External]
[monodroid-assembly] open_from_bundles: failed to load assembly Xamarin.AndroidX.Loader.dll
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/Xamarin.AndroidX.Loader.dll [External]
[monodroid-assembly] open_from_bundles: failed to load assembly System.Reflection.Emit.Lightweight.dll
[monodroid-assembly] open_from_bundles: failed to load assembly System.Reflection.Emit.ILGeneration.dll
[monodroid-assembly] open_from_bundles: failed to load assembly System.Reflection.Primitives.dll
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/System.Reflection.Emit.Lightweight.dll [External]
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/System.Reflection.Emit.ILGeneration.dll [External]
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/System.Reflection.Primitives.dll [External]
[monodroid-assembly] open_from_bundles: failed to load assembly IntelRealSenseBinding.dll
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/IntelRealSenseBinding.dll
Resolved pending breakpoint at 'MainActivity.cs:222,1' to void mauiExample.MainActivity.<StartStreaming>b__12_0 () [0x0000e].
Resolved pending breakpoint at 'MainActivity.cs:147,1' to void mauiExample.MainActivity.stream () [0x0003a].
Resolved pending breakpoint at 'MainActivity.cs:170,1' to void mauiExample.MainActivity.stream () [0x0012b].
[monodroid-assembly] open_from_bundles: failed to load assembly Xamarin.AndroidX.Lifecycle.LiveData.Core.dll
[monodroid-assembly] open_from_bundles: failed to load assembly System.Console.dll
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/Xamarin.AndroidX.Lifecycle.LiveData.Core.dll [External]
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/System.Console.dll [External]
[librs UsbUtilities] hasUsbPermission
[librs MessagesHandler] handleMessage: realsense device attached
[librs Enumerator] notifyOnAttach
[librs DeviceWatcher] Adding device: /dev/bus/usb/001/002
[librs DeviceWatcher] Device: /dev/bus/usb/001/002 added successfully
[monodroid-assembly] open_from_bundles: failed to load assembly Xamarin.AndroidX.ConstraintLayout.dll
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/Xamarin.AndroidX.ConstraintLayout.dll [External]
[monodroid-assembly] open_from_bundles: failed to load assembly DocumentData.dll
[monodroid-assembly] open_from_bundles: failed to load assembly netstandard.dll
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/DocumentData.dll
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/netstandard.dll [External]
[tel.mauiExample] type=1400 audit(0.0:903): avc: denied { lock } for path="/system/usr/share/zoneinfo/tzdata" dev="mmcblk1p11" ino=366 scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:system_file:s0 tclass=file permissive=1
[OpenGLRenderer] HWUI GL Pipeline
[mali_so] [File] : hardware/arm/maliT760/driver/product/base/src/mali_base_kbase.c; [Line] : 947; [Func] : base_context_deal_with_version_affairs_rk_ext;
[mali_so] arm_release_ver of this mali_so is 'r18p0-01rel0', rk_so_ver is '11@0'.
[mali_so] [File] : hardware/arm/maliT760/driver/product/base/src/mali_base_kbase.c; [Line] : 914; [Func] : determine_policy_against_patch_for_sf_memory_leak;
[mali_so] cmdline : com.concatel.mauiExample, we should GO THROUGH patch_for_sf_memory_leak.
[mali_so] [File] : hardware/arm/maliT760/driver/product/base/src/mali_base_kbase.c; [Line] : 955; [Func] : base_context_deal_with_version_affairs_rk_ext;
[mali_so] current process is NOT sf, to bail out.
[com.concatel.mauiExample] android::hardware::configstore::V1_0::ISurfaceFlingerConfigs::hasWideColorDisplay retrieved: 0
[OpenGLRenderer] Initialized EGL, version 1.4
[OpenGLRenderer] Swap behavior 2
[mali_winsys] EGLint new_window_surface(egl_winsys_display *, void *, EGLSurface, EGLConfig, egl_winsys_surface **, egl_color_buffer_format *, EGLBoolean) returns 0x3000
Resolved pending breakpoint at 'RealsenseFragment.cs:62,1' to void mauiExample.Fragments.RealsenseFragment.OnPause () [0x00001].
[librs] Found 1 RealSense devices (mask 0xfe)
Thread started: <Thread Pool> #2
Thread started: .NET ThreadPool Gate #3
[Choreographer] Skipped 30 frames!  The application may be doing too much work on its main thread.
Thread started: <Thread Pool> #4
[monodroid-assembly] open_from_bundles: failed to load assembly System.Linq.dll
Loaded assembly: /data/data/com.concatel.mauiExample/files/.__override__/System.Linq.dll [External]
[librs] Found 1 RealSense devices (mask 0xfe)
[librs] Found 1 RealSense devices (mask 0xfe)
[librs] Request: Z16 Depth, 
[librs] Resolved to: Z16 Depth, 
[librs] endpoint 130 read buffer size: 1844224
[librs] USB pipe 130 reset successfully
[librs] control_transfer returned error, index: 768, error: Broken pipe, number: 32
[librs] Request: RGB8 Color, 
[librs] Resolved to: YUYV Color, 
[librs] endpoint 132 read buffer size: 1844224
[librs] USB pipe 132 reset successfully
[librs] Frame Callback Color #0 overdue. (FPS: 30, max duration: 32.2581 ms)
[librs] control_transfer returned error, index: 768, error: Broken pipe, number: 32
[librs] control_transfer returned error, index: 768, error: Broken pipe, number: 32
[librs] Failed to allocate composite frame
[librs] Dropped frame. alloc_frame(...) returned nullptr
[librs] Exception was thrown during callback: Out of frame resources!
[librs] Dropped frame. alloc_frame(...) returned nullptr
[librs] Exception was thrown during callback: Out of frame resources!
[librs] Dropped frame. alloc_frame(...) returned nullptr
[librs] Exception was thrown during callback: Out of frame resources!
[librs] Dropped frame. alloc_frame(...) returned nullptr
[librs] Exception was thrown during callback: Out of frame resources!
[librs] Dropped frame. alloc_frame(...) returned nullptr
[librs] Exception was thrown during callback: Out of frame resources!
[librs] Dropped frame. alloc_frame(...) returned nullptr
[librs] Exception was thrown during callback: Out of frame resources!
[librs] Dropped frame. alloc_frame(...) returned nullptr
[librs] Exception was thrown during callback: Out of frame resources!
[librs] Dropped frame. alloc_frame(...) returned nullptr
[librs] Exception was thrown during callback: Out of frame resources!
... //And then it just repeats itself

I know what i'm doing it's not supposed to be supported by intel, but i was expecting this to work.

BR

[MartyG-RealSense]

Hi @juamar This issue with a 15 frame limit before an error occurs is typically found on Python applications, though I can envisage how it could occur in C# too. Information about it can be found at #946

Usually, the solution is to use the SDK's Keep() instruction to store the frames in memory. A RealSense team member discusses in #7201 about using Keep() in C#.

A C# discussion about object pooling at #8334 is also an interesting read.

[MartyG-RealSense]

Hi @juamar Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.

[juamar]

Hi after almost a month i returned to this. Sorry i did not replied before. I just wanted to update this because i resolved this in a different way.

As we are working with MAUI and we cannot work wirh .NET's Intel Realsense Wrapper, we are working with a Xamarin Java Binding to make the Intel Realsense SDK work.

The thing is, even thought the library is we are using is the java one, we were facing the same problem metioned on issue #7201 but java library does not have Keep() method. So we are managing dispose and close manually (instead of using using statement). So managing frames and FrameSets like this solved this issue:

   private void stream0()
   {
       int i = 0;
       Pipeline pipe = new Pipeline();
       pipe.Start();

       while (!mStreamingThread.StopRequest)
       {
           FrameSet frames = pipe.WaitForFrames();
               Frame f = frames.First(StreamType.Depth);

                   DepthFrame depth = (DepthFrame)f.As(Extension.DepthFrame);
                   float deptValue = depth.GetDistance(depth.Width / 2, depth.Height / 2);                    

                   i++;
                   this.RunOnUiThread(() => realsenseFragment.PrintToTest("Distance: " + deptValue + "\n" + i));
                   f.Close();
                   f.Dispose();

               frames.Close();
               frames.Dispose();

           Thread.Sleep(1000 / 30);
       }
       pipe.Stop();
   }

Thank you very much @MartyG-RealSense!

[MartyG-RealSense]

It's great news that you achieved a solution, @juamar - thanks so much for sharing it with the RealSense community :)

[juamar]

Also, when trying to align Depth and Color frames, we can do this:

    private void stream0()
    {
        int i = 0;
        Config _cfg = new Config();
        _cfg.EnableStream(StreamType.Depth, -1, 1280, 720, StreamFormat.Z16, 30);
        _cfg.EnableStream(StreamType.Color, -1, 1280, 720, StreamFormat.Bgr8, 30);

        Pipeline pipe = new Pipeline();
        pipe.Start(_cfg);

        while (!mStreamingThread.StopRequest)
        {
            Align AlignFilter = new Align(StreamType.Color);
            FrameSet frames = pipe.WaitForFrames();
                //frames = (FrameSet)frames.ApplyFilter(AlignFilter);
                FrameSet framesAligned = AlignFilter.Process(frames);
                Frame f = frames.First(StreamType.Depth);

                    DepthFrame depth = (DepthFrame)f.As(Extension.DepthFrame);
                    float deptValue = depth.GetDistance(depth.Width / 2, depth.Height / 2);                    

                    i++;
                    this.RunOnUiThread(() => realsenseFragment.PrintToTest("Distance: " + deptValue + "\n" + i));
                    f.Close();
                    f.Dispose();

                Frame f2 = frames.First(StreamType.Color);
                    VideoFrame frame = (VideoFrame)f2.As(Extension.VideoFrame);
                    Console.WriteLine(frame.Height);
                    f2.Close();
                    f2.Dispose();
                
                framesAligned.Close();
                framesAligned.Dispose();

            frames.Close();
            frames.Dispose();
            AlignFilter.Close();
            AlignFilter.Dispose();

            Thread.Sleep(1000);
        }
        pipe.Stop();
    }

You must close and dispose both FrameSets, the original (frames in my example) and framesAligned. framesAligned = frames.ApplyFilter() was giving me problems when disposing both, as runtime was telling me that frames was already disposed when first dispossing framesAligned.

Note: No need for Align object to be instanciated on every FrameSet capture.