Difference in image when using sim.getImages to retrieve a compressed and uncompressed image.

[m1baldwin]

I have a simple script which I am using to test the difference in the compressed and uncompressed images retrieved using the image api getImages. I based the code on the example code shown in the airsim documentation.

This is my script:

import numpy as np
import airsim
import os

client = airsim.MultirotorClient()
client.confirmConnection()

filename = "test_uncompressed"
filename2= "test_compressed"
responses = client.simGetImages([airsim.ImageRequest("front_left_custom", airsim.ImageType.Scene, False, False), 
                                 airsim.ImageRequest("front_left_custom", airsim.ImageType.Scene, False, True)])

response = responses[0]



# get numpy array
img1d = np.fromstring(response.image_data_uint8, dtype=np.uint8) 

# reshape array to 4 channel image array H X W X 4
img_rgb = img1d.reshape(response.height, response.width, 3)

# original image is fliped vertically
img_rgb = np.flipud(img_rgb)

# write to png 
airsim.write_png(os.path.normpath(filename + '.png'), img_rgb) 
airsim.write_file(os.path.normpath(filename2 + '.png'), responses[1].image_data_uint8) 

When I run it, I get the following two images. They are from the same camera, and should be of the same thing. However, the image generated from the compressed image is okay, while the image from the uncompressed image request seems completely wrong.

I am running Unreal+ Airsim on Windows 10 with an Nvidia GeForce RTX 2080ti graphics card.

[rajat2004]

Hmm, I recently tested this and was working correctly.
Don't think this line img_rgb = np.flipud(img_rgb) is required

[m1baldwin]

That is strange, because if I don't include that line then the image looks very similar, but clearly upside down.

Any ideas on additional tests? Is there a way to run using opengl on windows, rather than vulkan?

[rajat2004]

@m1baldwin The problem seems to be in airsim.write_png method, that's storing the image incorrectly, and I'm unable to open it due to bad adaptive filter value on Linux. Using OpenCV to save the image works, and I have been using that instead of the utility function so hadn't noticed the problem.
My modified script -

import numpy as np
import airsim
import os
import cv2

client = airsim.MultirotorClient()
client.confirmConnection()

filename = "test_uncompressed"
filename2= "test_compressed"
responses = client.simGetImages([airsim.ImageRequest("front_center", airsim.ImageType.Scene, False, False), 
                                 airsim.ImageRequest("front_center", airsim.ImageType.Scene, False, True)])

response = responses[0]
print(f"Len: {len(response.image_data_uint8)}, height: {response.height}, width: {response.width}")

# get numpy array
img1d = np.fromstring(response.image_data_uint8, dtype=np.uint8) 

# reshape array to 3 channel image array H X W X 3
img_rgb = img1d.reshape(response.height, response.width, 3)

# Display image for confirmation
cv2.imshow("Image", img_rgb)
cv2.waitKey(0)

# original image is fliped vertically
# img_rgb = np.flipud(img_rgb)

# write to png 
# airsim.write_png(os.path.normpath(filename + '.png'), img_rgb)
# Write uncompressed image
ret = cv2.imwrite(os.path.normpath(filename + '.png'), img_rgb)
print(ret)

airsim.write_file(os.path.normpath(filename2 + '.png'), responses[1].image_data_uint8) 

The utility function should be changed to use cv2

Edit: included in #2683

[m1baldwin]

@rajat2004 Thank you, this worked for me.

Just wondering, what is the expected rate I should be able to continuously request images from the camera in the simulation? I am interested in recording frames and recombining them into a video to simulate a video feed for post-processing. But, I am getting very low frame rate as described here #2891

I suppose, rather than requesting the compressed image I could request the uncompressed image using the above methods to get a higher frame rate?

[rajat2004]

Not sure about the expected frame rate, would depend on the machine, resolution etc. But yeah, fetching uncompressed image would definitely speed it up a bit
I'll post on the other thread, to keep all details related to a topic in one place

[m1baldwin]

@rajat2004 sure, I will follow up in the other issue.