d435i imu-camera calibration problem

[lishanggui]

Required Info
Camera Model	D435i
Firmware Version	5.12.6
Operating System & Version	Ubuntu 16.04
Kernel Version (Linux Only)	(e.g. 4.15.0)
Platform	PC
SDK Version	2.36.0
Language	C++

I used two methods to calibrate imu-camera:

1. Calibrate the original realsense image and write the calibration result into the device. Read the camera internal parameters of the 640x480 corrected infra binocular image from the device, and calibrate it with imu;
2. Calibrate the realsense 640x480 corrected infra binocular image, and the calibration parameters obtained are calibrated with imu;
   At the same time, I tried different camera models (pinhole-radtan, pinhole-equi) in kalibr for imu and dual target positioning. The reprojection error of the image obtained was very large. As shown below:
   

But when I calibrate imu-camera for mynteye camera with the same method，The reprojection error of the image obtained was very small. As shown below:

May I ask if there is a problem with my calibration and how can I get better calibration results?

[MartyG-RealSense]

Hi @lishanggui There is an official Python-based calibration tool for the IMU of the D435i.

https://github.com/IntelRealSense/librealsense/tree/master/tools/rs-imu-calibration
https://www.intelrealsense.com/wp-content/uploads/2019/07/Intel_RealSense_Depth_D435i_IMU_Calibration.pdf

The official IMU calibration tool includes a system for correcting for known inaccuracies in the IMU (as the IMU component used in RealSense cameras does not have its own internal calibration). Without this correction, the acceleration values of the D435i may not settle at near zero even when the device is stationary, and the value of acceleration under gravity will not approach the ideal value of 9.8 (9.81 being the gravity force of the Earth).

So unless your own custom IMU calibration tool can include a similar correction mechanism, it is not likely to perform as well as the official calibration tool.

The discussion in the link below may be useful to you regarding how the official IMU calibrator operates:

#5177

[lishanggui]

Thank you very much. I have already calibrated imu like your said， before calibrating imu-camera. After calibration, the acceleration measured by placing the camera in 6 different directions is close to 9.8, as shown in the figure below.
file:///home/cti/%E5%9B%BE%E7%89%87/623809722.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1222965985.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1661764911.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1874259703.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1944418116.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/623809722.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1222965985.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1661764911.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1874259703.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1944418116.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/623809722.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1222965985.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1661764911.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1874259703.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1944418116.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/623809722.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1222965985.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1661764911.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1874259703.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1944418116.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/623809722.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1222965985.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1661764911.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1874259703.jpg
file:///home/cti/%E5%9B%BE%E7%89%87/1944418116.jpg

file:///home/cti/%E5%9B%BE%E7%89%87/webwxgetmsgimg.jpg

At the same time, I also have the imu noise, the imu noise data is as follows:
accelerometer_noise_density: 1.9697734375312265e-02 #Noise density (continuous-time)
accelerometer_random_walk: 8.2738217768213147e-04 #random walk
#Gyroscopes
gyroscope_noise_density: 1.7633550954006261e-03 #Noise density (continuous-time)
gyroscope_random_walk: 1.7151225519702968e-05 #random walk

But when I calibrate the imu-camera, the result is as mentioned above, the image reprojection error is very large.

[MartyG-RealSense]

There was a recent case of another RealSense user who felt that their D435i IMU was too noisy.

IntelRealSense/realsense-ros#1286

In that case, I highlighted how the SDK's rs-motion D435i example program can change the balance of theta calculations to bias towards the gyro or the accelerometer depending on whether the alpha value is set low or high.

IntelRealSense/realsense-ros#1286 (comment)

[lishanggui]

I'm very sorry that I read the link you sent, but I still don't quite understand. Do you mean to let me change the alpha parameter in rs-motion? Will changing this parameter affect imu-camera calibration? From my calibration results, it should not be caused by too noisy. I see that some anther people got similar values for d435i noise . Do you need me to provide other experimental data?

[MartyG-RealSense]

The reason for the links was to demonstrate that (a) there are others who also consider the IMU noisy, or over-sensitive; and (b) show examples of how code in programs such as rs-motion can be used to improve IMU results.

You are welcome to provide further test data to illustrate your belief that the data is too noisy.

[xzwang12345]

I have the same problem as lishanggui,how to solve this problem? what's alpha value range can improve the result in rs-motion program?

[lishanggui]

@MartyG-RealSense After imu calibration, the acceleration measured by placing the camera in 6 different directions is close to 9.8, as shown in the below figure：

When I ran the rs-motion program for the first time, the imu model rotates randomly, even if the camera was hold still.

It usually takes several seconds that the model finally become stable. When I was doing a rotating exercise, the results displayed were not correct.

After I run rs-motion several times, the results displayed are correct,when I rotate the realsense d435i.

After that, I used the kalibr program to recalibrate the infra binocular camera（follow method two） and imu-camera. The image reprojection error obtained after imu-camera calibration is shown in the figure below

the imu noise data is as follows:
accelerometer_noise_density: 1.9697734375312265e-02 #Noise density (continuous-time)
accelerometer_random_walk: 8.2738217768213147e-04 #random walk
#Gyroscopes
gyroscope_noise_density: 1.7633550954006261e-03 #Noise density (continuous-time)
gyroscope_random_walk: 1.7151225519702968e-05 #random walk

The above is the result of my experiment, the image reprojection error obtained after the imu-camera calibration is still very large. Please help find the cause of the problem？

[MartyG-RealSense]

Hi @lishanggui I would say that the results of your data from kalibr reflect how the IMU data is inherently noisy, as a RealSense team member confirms in the link below.

#4391 (comment)

RealSense programs such as rs-motion benefit from corrective mechanisms to stabilize the IMU data. As kalibr is not a RealSense SDK program, it can only see the raw, noisy version of the data that has not been corrected ("fixed", as the official RealSense IMU calibration tool would describe it).

[lishanggui]

@MartyG-RealSense I used the official tool to calibrate the imu. I showed the rs-motion display results as shown above, but I am very sorry,I don’t know how to use rs-motion to stabilize the IMU data. Do you mean I need to refer to the rs-motion program for further processing of the obtained imu data, or change the alpha value in rs-motion to stabilize the IMU data?

[xzwang12345]

hi @MartyG-RealSense, do you mean we need to correct imu data as rs-motion? Is the realsense-ros program do this correct?

[MartyG-RealSense]

As well as the ability of rs-motion to correct the IMU data, the RealSense Viewer can also perform corrections. Once an IMU calibration has been written to the camera by the official IMU calibration tool, a Motion Correction option appears in the Motion Module options.

#4751 (comment)

Librealsense programs do not have an effect on the camera's behavior in ROS though. For example, if you made a change in the RealSense Viewer program then those changes would not be reflected in how the camera behaved in the RealSense ROS wrapper. Likewise, I am not aware of a means to perform IMU data correction in the RealSense ROS wrapper in the same way that programs such as rs-motion and the Viewer can.

Some RealSense users have created custom frequency timings for the IMU to try to modify its behavior, such as defining a rate in ROS:

#6115

[lishanggui]

Hi @MartyG-RealSense As you can see in the figure below,

after running realsense-viewer, the corrected data is close to 9.8. At the same time, I run rs-camera.launch of realsense-ros, and the imu data obtained from it is the same as the corrected data.

I used the imu data obtained in rs-camera.launch to further perform imu-camera calibration.

[MartyG-RealSense]

@lishanggui Ultimately, the IMU in RealSense devices is what it is. It does not have an internal calibrator in its hardware. You can try to achieve as good a calibration with the official software tool as possible, and you can apply corrections to the raw IMU data with SDK programming.

It is sometimes possible to squeeze improvement out of systems if you apply further custom processing to the data, as you seem to be doing in ROS. You are welcome to share your improvement techniques with the RealSense ROS developers with a "pull request" with a view to merging your techniques into the wrapper. At this current point in time though, I do not think that there is any further advice that I can offer to improve IMU noise, unfortunately.

[lishanggui]

Hi @MartyG-RealSense From the above experiment, the corrected imu is close to 9.8. In my opinion, this is a perfect imu calibration result? I want to ask, how did you come to the conclusion that there is the imu calibration problem ？

[MartyG-RealSense]

@lishanggui In your kalibr information above, you said that "the image reprojection error obtained after the imu-camera calibration is still very large. Please help find the cause of the problem". This suggested that you were not satisfied with the IMU data.

After you performed additional processing on the IMU data in ROS after launch, are you now satisfied with the IMU results, please?

[lishanggui]

Hi @MartyG-RealSense I have not yet got a satisfactory result. Do you think I should perform the imu calibration experiment several times according to the official method? But I have repeated this experiment many times and I haven't seen any improvement.

[MartyG-RealSense]

It is best to assume that you will never achieve perfect results due to the variables in the calibration process. For example, in steps 2 and 4 of the calibration, where you hold the camera vertically by the connector end or the other end instead of placing it on a totally flat surface, it is difficult to maintain the camera at a completely level orientation. This will have an influence on the final calibration result.

It is therefore best to decide on a level of accuracy that is "good enough" for you. I believe that the most important result is the Y-Accel, and getting it as close to the ideal of 9.80 as possible.

[MartyG-RealSense]

Hi @lishanggui Do you still require assistance with this case, please? Thanks!

[lishanggui]

@MartyG-RealSense I'm very sorry,after repeating it many times, I still can't get the good result of imu-camera. But after the imu calibration, the acceleration is close to the acceleration of gravity (the local acceleration of gravity is 9.7925). I don’t know if anyone else has the same problem as me.

[MartyG-RealSense]

No need for apologies. :)

I don't think you can get any better in terms of the Y-acceleration results being so close to 9.80.

In regard to the accuracy of X and Z: if it is not causing problems for your particular application then I would not be too concerned about it. For example, whilst over-sensitivity on a particular axis might be a problem for a D435i that was on a flying drone, on a slower-moving project on the ground it may not matter as much.

[lishanggui]

After imu calibration, the acceleration when placed in other different directions is as follows：

The imu Calibration result is good, but I am very curious why I still cannot get a better imu-camera calibration result. The reprojection error analysis after imu-camera calibration(kalibr tool) is not perfect.

[MartyG-RealSense]

The IMU component in RealSense cameras does not have an internal calibrator. The software calibration tool does its best to compensate for known error characteristics in the component. What you are getting may be as good as it is going to get.

[MartyG-RealSense]

Hi @lishanggui Do you require further assistance please, or can this case be closed? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.

[lishanggui]

Hi @MartyG-RealSense Use the imu calibration result to calibrate imu-camera, the result always get the result shown above. I haven't been able to find any other way to solve the problem.

[MartyG-RealSense]

Unfortunately this discussion has already supplied all of the advice that I can give. There is nothing else I can suggest that would improve the IMU data further if it is being accessed in a raw form outside of the corrective software algorithms provided by the RealSense SDK.

[lishanggui]

@MartyG-RealSense thanks!
But I have anther t265 issue at #7199. can any people help me？

The sensor of T265 (including IMU) is calibrated on the production line, so no further calibration process is required (unlike the IMU on D435i). However, realsense-viewer shows that the acceleration will vary from 8 to 10 when placed at rest in different directions. Does this have an impact on vio and how to solve it?

[MartyG-RealSense]

I am not able to assist with that question as somebody else on the RealSense team is responsible for handling T265 questions. I recommend tagging the name RealSenseSupport in a comment in #7199 (not on this case).

[lishanggui]

@MartyG-RealSense Ok，thank you for your so much help!