Camera frame rate for capturing images is exceptionally low

[m1baldwin]

I have recently been testing Airsim + Unreal on a gaming desktop to see if I can get better performance than what I was able to get on a laptop. For reference, this is the desktop:

Alienware Aurora Ryzen™ Edition R10 Gaming Desktop
https://www.dell.com/en-us/shop/cty/pdp/spd/alienware-aurora-r10-desktop/wdryzr10mx?configurationid=8e4e7c7e-3794-4eef-abb4-173cdc0a3004

As part of my work, I am interested in flying a drone along a pre-programmed path, with the images from the cameras being captured at a high frame rate, ideally 30fps, which would recreate the video feed of a camera attached to a real drone. I am using this video for evaluation of different VSLAM/VO/VIO algorithms in the simulated environment with the goal of validating an approach before deploying it to a real environment. I have a stereo camera pair instantiated with individual resolutions of 848x800.

The issue I am having is that despite trying multiple different approaches, the only way I can get a decent frame rate of image capture is by running the simulation at a very low clock speed, like 0.1. I have tried using the built in recording tool in the Unreal editor, and even some custom python scripting which will record the pose of the robot and the camera image, so that I have ground truth information for each recorded frame.

My current simulation approach looks like this:

1. set the clock speed to 0.1 in settings
2. Set the display type to NoDisplay in settings
3. Start the simulation
4. Start the logging script, which records images at and pose of the drone. The recording script is done in a way that the rate limit of the capture loop will be defined by how fast the image capturing calls will allow it to run. I am using something of the form of the following to get the images, following the example script provided in the airsim source:
   responses = client.simGetImages([
   airsim.ImageRequest("front_left_custom", airsim.ImageType.Scene, False, True),
   airsim.ImageRequest("front_right_custom", airsim.ImageType.Scene, False, True)])
5. Allow the simulation to run, and halt the capturing script when the simulation is complete.

What I am seeing right now is that the performance of this logging program doesn't improve much, even when run on more powerful hardware. I would have expected with enough CPU, GPU, and RAM available, and with no display showing, that the image capture could run close to real time at a high frame rate.

Is there maybe something I am missing here? I have seen similar performance when trying to get the images over ros topic too. This is very easily done in gazebo so I am surprised that I am having so much trouble with airsim, which I expected to be much better. I took a scan of the issues on git and I am seeing some others mentioning similar issues.

[rajat2004]

Low frame rates are definitely a problem. Just some general ideas on improving it

Try using binaries, I've generally found that binaries are faster than running in Editor. Not sure if you're already using them, for your custom env or even the normal ones, once you have any changes in Airsim (such as the fetching the latest master) tested and working, packaging the project is pretty easy. See UE4's page, there's also a setting Full Rebuild you would want to disable if packaging frequently. First time packaging will take time though

You aren't using a high-resolution camera infrequently from the description, if you were, then trying out #2881 would be a very good thing. But mostly not applicable here

There's another PR #2713 which has some binaries linked you can try, that improves FPS quite a bit, but crashes. And currently, Depth images aren't working, need to spend some time on that

As discussed int the other issue #2892, maybe see if using uncompressed images helps

NoDisplay mode definitely increases the FPS a lot
I haven't played with ClockSpeed much, will test and see how it changes the FPS

Do you have a settings file which I can try out?
Will post if anything else comes to my mind

[m1baldwin]

Here are the contents of my settings file:

{
  "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
  "SettingsVersion": 1.2,
  "SimMode": "Multirotor",
  "ViewMode": "NoDisplay",
  "ClockSpeed": 1.0,
  "Recording": {
    "RecordOnMove": false,
    "RecordInterval": 0.05,
    "Cameras": [
      {
        "CameraName": "0",
        "ImageType": 0,
        "PixelsAsFloat": false,
        "Compress": true
      },
      {
        "CameraName": "1",
        "ImageType": 0,
        "PixelsAsFloat": false,
        "Compress": true
      }
    ]
  },
  "CameraDefaults": {
    "CaptureSettings": [
      {
        "ImageType": 0,
        "Width": 848,
        "Height": 800,
        "FOV_Degrees": 90,
        "AutoExposureSpeed": 100,
        "AutoExposureBias": 0,
        "AutoExposureMaxBrightness": 0.64,
        "AutoExposureMinBrightness": 0.03,
        "MotionBlurAmount": 0,
        "TargetGamma": 1.0,
        "ProjectionMode": "",
        "OrthoWidth": 5.12
      }
    ]
  },
  "Vehicles": {
    "drone_1": {
      "VehicleType": "SimpleFlight",
      "DefaultVehicleState": "Armed",
      "EnableCollisionPassthrogh": false,
      "EnableCollisions": true,
      "AllowAPIAlways": true,
      "RC": {
        "RemoteControlID": 0,
        "AllowAPIWhenDisconnected": false
      },
      "Sensors": {
        "Imu": {
          "SensorType": 2,
          "Enabled": true
        }
      },
      "Cameras": {
        "front_left_custom": {
          "CaptureSettings": [
            {
              "PublishToRos": 1,
              "ImageType": 0,
              "Width": 848,
              "Height": 800,
              "FOV_Degrees": 90,
              "TargetGamma": 1.5
            }
          ],
          "X": 0.50,
          "Y": -0.06,
          "Z": 0.10,
          "Pitch": 0.0,
          "Roll": 0.0,
          "Yaw": 0.0
        },
        "front_right_custom": {
          "CaptureSettings": [
            {
              "PublishToRos": 1,
              "ImageType": 0,
              "Width": 848,
              "Height": 800,
              "FOV_Degrees": 90,
              "TargetGamma": 1.5
            }
          ],
          "X": 0.50,
          "Y": 0.06,
          "Z": 0.10,
          "Pitch": 0.0,
          "Roll": 0.0,
          "Yaw": 0.0
        }
      }
    }
  }
}

Just want to go over the points you mentioned:

1. I will take a look at the pre-built binary option. I am mostly interested in the custom environment I have, so I will try with that and see if I can get it to work. I only got this machine I referenced above this week; before this, my laptop would crash whenever I tried building anything :)

Also, I just found that the performance does improve a bit if I build the lighting. I hadn't built it for some time, so maybe there is some benefit to building it before testing.

2. I've actually tried PR Faster Image Capture #2713 before - the pre-built binary anyway that you shared - and I did get some speed up. I would need to pull this and recompile the plugin and add it to my current project to test (windows). This is next on my todo list. I don't have need for a depth camera at the moment, so I think this will work for me.

3. I have been reducing clockspeed to get a "virtual" fps closer to a real camera, at the cost of the simulation taking much, much longer. I don't see a huge gain in the frame rate I can issue the getImages api command at when running at lower clock speed; the predominant benefit is really just that the simulation slows down.

I suppose one other performance improvement might be to separate the image request and the saving of the png into separate threads. I notice a decrease in the speed when I have these in sequential order. Should actually be pretty similar to the usual issue with getting the video from a webcam using opencv.

[m1baldwin]

Also, strangely now when I try with PR #2713, I get an error when trying to convert the uncompressed into a numpy array

(base) C:\Users\eia-l\Documents\SoftwareProjects\AirSim\PythonClient\computer_vision>python test_numpy.py
Connected!
Client Ver:1 (Min Req: 1), Server Ver:1 (Min Req: 1)

Len: 2764800, height: 800, width: 864
test_numpy.py:19: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead
img1d = np.fromstring(response.image_data_uint8, dtype=np.uint8)
Traceback (most recent call last):
File "test_numpy.py", line 22, in
img_rgb = img1d.reshape(response.height, response.width, 3)
ValueError: cannot reshape array of size 2764800 into shape (800,864,3)

And when I try to request and record the compressed image, even though it saves the image cannot be open, because the file format is supposedly unsupported

This worked for me before when I tried on a linux machine using opengl, but on my windows machine now it is not working. I didn't use opengl on windows

[rajat2004]

Oh yeah, sorry, the PR changes the format from RGB to RGBA, so for reshaping you'll need to use img1d.reshape(response.height, response.width, 4). I started using -1 in channels so I could keep testing without changing it all over.
The compressed image won't work, that part of the code actually got removed in the earlier commits, and needs to be added back

[m1baldwin]

Okay, thanks for the heads up. That allowed me to test with it.

[clintlombard]

I'm also seeing slow image capturing, with +-5 Hz @ 480p for both OpenGL and Vulkan. @rajat2004 I'm trying your suggestion of packaging the binaries in Linux, but I can't seem to get it working. I followed AirSim's guide, and have also tried using the ue4cli to no avail. I can compile everything by when trying to run the resulting .sh, I get the following error:

Plugin 'AirSim' failed to load because module 'AirSim' could not be found. Please ensure the plugin is properly installed, otherwise consider disabling the plugin for this project.

I'm using the following versions:
AirSim: 1.3.1-linux
UE4: 4.25.3

[rajat2004]

@clintlombard You're getting 5Hz at 480p? That's too low
I haven't yet used the command-line to package envs, or used UE from source inside docker, so can't say much. Instructions seem okay

[stale]

This issue has been automatically marked as stale because it has not had activity from the community in the last year. It will be closed if no further activity occurs within 20 days.