realsense d435i problems

[maokp]

Thank you for your great work. I can run the car demenstration and Kitti GPS Fusion perfectly. But when I run it with realsense d435i,I get the following error.

$rosrun vins vins_node ~/catkin_ws/src/VINS-Fusion/config/realsense_d435i/realsense_stereo_imu_config.yaml
[ INFO] [1556271031.868824114]: init begins
config_file: realsense_stereo_imu_config.yaml
USE_IMU: 1
IMU_TOPIC: /camera/imu
result path /home/dji/output//vio.csv
[ WARN] [1556271031.872582533]:  Optimize extrinsic param around initial guess!
camera number 2
[ INFO] [1556271031.872806431]: Unsynchronized sensors, online estimate time offset, initial td: 0
[ INFO] [1556271031.872823934]: ROW: 480 COL: 640 
 exitrinsic cam 0
-0.00575863 -0.00404633    0.999975
  -0.999983 -0.00102246 -0.00576281
 0.00104575   -0.999991 -0.00404037
 0.0203293 0.00793252 0.00285598
 exitrinsic cam 1
-0.00100218 0.000363135    0.999999
  -0.999992 -0.00383034 -0.00100078
 0.00382998   -0.999993 0.000366971
 0.0152858 -0.0524358 0.00869313
set g     0     0 9.805
[ INFO] [1556271031.872958979]: reading paramerter of camera realsense_stereo_imu_config.yaml/left.yaml
[ INFO] [1556271031.873021335]: reading paramerter of camera realsense_stereo_imu_config.yaml/right.yaml
MULTIPLE_THREAD is 1
[ WARN] [1556271031.873101001]: waiting for image and imu...
vins_node: /usr/include/boost/smart_ptr/shared_ptr.hpp:648: typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = camodocal::Camera; typename boost::detail::sp_member_access<T>::type = camodocal::Camera*]: Assertion `px != 0' failed.
Aborted (core dumped)

ROS node was launched with following comands:

roslaunch vins vins_rviz.launch
rosrun loop_fusion loop_fusion_node ~/catkin_ws/src/VINS-Fusion/config/realsense_d435i/realsense_stereo_imu_config.yaml
rosrun vins vins_node ~/catkin_ws/src/VINS-Fusion/config/realsense_d435i/realsense_stereo_imu_config.yaml

I have checked the /camera/infra1/image_rect_raw,/camera/infra2/image_rect_rawand /camera/imutopics, these nodes looked fine.

Any idea about the error would be appreciate.

[pjrambo]

Could you please upload your camera intrinsic parameters' yaml file and your vins config file. It seems that there is something wrong with your config file.

[maokp]

The left camera and right camera instrinsic parameters' yaml files

%YAML:1.0
---
model_type: PINHOLE
camera_name: camera
image_width: 640
image_height: 480
distortion_parameters:
   k1: 0.0
   k2: 0.0
   p1: 0.0
   p2: 0.0
projection_parameters:
   fx: 385.7544860839844
   fy: 385.7544860839844
   cx: 323.1204833984375
   cy: 236.7432098388672

%YAML:1.0
---
model_type: PINHOLE
camera_name: camera
image_width: 640
image_height: 480
distortion_parameters:
   k1: 0.0
   k2: 0.0
   p1: 0.0
   p2: 0.0
projection_parameters:
   fx: 385.7544860839844
   fy: 385.7544860839844
   cx: 323.1204833984375
   cy: 236.7432098388672

And here is realsense_stereo_imu_config.yaml

%YAML:1.0

#common parameters
#support: 1 imu 1 cam; 1 imu 2 cam: 2 cam; 
imu: 1         
num_of_cam: 2  

imu_topic: "/camera/imu"
image0_topic: "/camera/infra1/image_rect_raw"
image1_topic: "/camera/infra2/image_rect_raw"
output_path: "/home/dji/output/"

cam0_calib: "left.yaml"
cam1_calib: "right.yaml"
image_width: 640
image_height: 480
   

# Extrinsic parameter between IMU and Camera.
estimate_extrinsic: 1   # 0  Have an accurate extrinsic parameters. We will trust the following imu^R_cam, imu^T_cam, don't change it.
                        # 1  Have an initial guess about extrinsic parameters. We will optimize around your initial guess.

body_T_cam0: !!opencv-matrix
   rows: 4
   cols: 4
   dt: d
   data: [ -5.7586305857286746e-03, -4.0463318787729019e-03,
       9.9997523237933461e-01, 2.0329267950355900e-02,
       -9.9998287214160420e-01, -1.0224590553211677e-03,
       -5.7628118925283633e-03, 7.9325209639615653e-03,
       1.0457519809151661e-03, -9.9999129084997906e-01,
       -4.0403746097850135e-03, 2.8559824645148020e-03, 0., 0., 0., 1. ]

body_T_cam1: !!opencv-matrix
   rows: 4
   cols: 4
   dt: d
   data: [ -1.0021770212322867e-03, 3.6313480322730518e-04,
       9.9999943188700535e-01, 1.5285779565991807e-02,
       -9.9999216342926500e-01, -3.8303422615924010e-03,
       -1.0007788055728661e-03, -5.2435791444330505e-02,
       3.8299766679101843e-03, -9.9999259827824449e-01,
       3.6697063849344680e-04, 8.6931302450199057e-03, 0., 0., 0., 1. ]

#Multiple thread support
multiple_thread: 1

#feature traker paprameters
max_cnt: 150            # max feature number in feature tracking
min_dist: 30            # min distance between two features 
freq: 10                # frequence (Hz) of publish tracking result. At least 10Hz for good estimation. If set 0, the frequence will be same as raw image 
F_threshold: 1.0        # ransac threshold (pixel)
show_track: 0           # publish tracking image as topic
flow_back: 1            # perform forward and backward optical flow to improve feature tracking accuracy

#optimization parameters
max_solver_time: 0.04  # max solver itration time (ms), to guarantee real time
max_num_iterations: 8   # max solver itrations, to guarantee real time
keyframe_parallax: 10.0 # keyframe selection threshold (pixel)

#imu parameters       The more accurate parameters you provide, the better performance
acc_n: 0.1          # accelerometer measurement noise standard deviation. #0.2   0.04
gyr_n: 0.01         # gyroscope measurement noise standard deviation.     #0.05  0.004
acc_w: 0.001         # accelerometer bias random work noise standard deviation.  #0.002
gyr_w: 0.0001       # gyroscope bias random work noise standard deviation.     #4.0e-5
g_norm: 9.805         # gravity magnitude

#unsynchronization parameters
estimate_td: 1                      # online estimate time offset between camera and imu
td: 0.00                             # initial value of time offset. unit: s. readed image clock + td = real image clock (IMU clock)

#loop closure parameters
load_previous_pose_graph: 0        # load and reuse previous pose graph; load from 'pose_graph_save_path'
pose_graph_save_path: "/home/dji/output/pose_graph/" # save and load path
save_image: 0                   # save image in pose graph for visualization prupose; you can close this function by setting 0 

by the way, I didn't change the configuration files here.

[pjrambo]

What is the resolution of your image?

[maokp]

It seems like the problem of vins configuration files.

I changed the estimate_td to 0, the programs work fine. But I don't know why.

[engcang]

I had similar Error using Intel D435i

$ rosrun vins vins_node d435i_stereo.yaml
[ INFO] [1556627880.348420802]: init begins
config_file: d435i_stereo.yaml
USE_IMU: 1
IMU_TOPIC: /camera/imu
result path ~/output//vio.csv
[ WARN] [1556627880.359646018]: fix extrinsic param
camera number 2
[ INFO] [1556627880.362078990]: Unsynchronized sensors, online estimate time offset, initial td: -0.072
[ INFO] [1556627880.362172561]: ROW: 480 COL: 640
exitrinsic cam 0
-0.00407513 -0.999991 0.00153372
-0.0060896 -0.00150888 -0.99998
0.999973 -0.00408439 -0.00608339
0.00237912 -0.0156927 0.000632511
exitrinsic cam 1
-0.00256499 -0.999995 0.00175985
-0.00633838 -0.00174356 -0.999978
0.999977 -0.00257608 -0.00633388
-0.0472626 -0.0156555 0.00062436
set g 0 0 9.80338
[ INFO] [1556627880.362701570]: reading paramerter of camera d435i_stereo.yaml/left.yaml
[ INFO] [1556627880.362908072]: reading paramerter of camera d435i_stereo.yaml/right.yaml
MULTIPLE_THREAD is 6
[ WARN] [1556627880.363108047]: waiting for image and imu...
vins_node: /usr/include/boost/smart_ptr/shared_ptr.hpp:734: typename boost::detail::sp_member_access::type boost::shared_ptr::operator->() const [with T = camodocal::Camera; typename boost::detail::sp_member_access::type = camodocal::Camera*]: Assertion `px != 0' failed.
Aborted (core dumped)

And I solved it by installing stable version of Ceres-solver and Eigen.
Surely after that, did 'catkin build' again.

Eigen :
$ wget -O eigen.zip http://bitbucket.org/eigen/eigen/get/3.3.7.zip #check version
$ unzip eigen.zip
$ mkdir eigen-build && cd eigen-build
$ cmake ../eigen_source_folder_name && sudo make install

Ceres :
$ sudo apt-get install -y cmake libgoogle-glog-dev libatlas-base-dev libsuitesparse-dev
$ wget http://ceres-solver.org/ceres-solver-1.14.0.tar.gz
$ tar zxf ceres-solver-1.14.0.tar.gz
$ mkdir ceres-bin && cd ceres-bin
$ cmake ../ceres-solver-1.14.0 && make -j8
$ make test
$ sudo make install

[pjrambo]

Do you have any bags from D435I, I can not reproduce the error as you said.

[pjrambo]

Here are some suggestions for those who are using D435I

1. Check the realsense launch file to make sure the resolution of the image is consistent with the VINS-FUSION config file.
2. The camera intrinsic parameters are obtained from /camera/infra*/camera_info provided by our realsense. You'd better check yours. If you want to get a good result, you are recommended to calibrate the cameras yourself.
3. The rs_camera.launch is used for the ros wrapper based on realsense SDK 2.18.1. For the higher version, some variables' name are changed.
4. Do not use the SDK 2.20(both 2.18 and 2.19 are fine), the IMU data will get stuck.
5. For a stable estimation result, turn off the IR projector of it.

[engcang]

I turned off IR projector using realsense-viewer and .json file from it
and used Kalibr to calibrate it (also tried with your configure file @pjrambo it was fine)
For me, using Xavier from jetson, Jetpack 4.1.1 SDK version is v2.17.1 and ros wrapper version is v2.1.3

even though i solved it by rebuild catkin workspace,
but sometimes i reboot the xavier, same error occur again

well i didnt save any .bag file but have youtube clip :
https://youtu.be/7yMDqiO2A2Q
https://youtu.be/b3l1TeNKyeU

[zxh2015]

I have the same problem evenif changed the estimate_td to 0, but it doesn't work

[zgxsin]

Here are some suggestions for those who are using D435I

1. Check the realsense launch file to make sure the resolution of the image is consistent with the VINS-FUSION config file.
2. The camera intrinsic parameters are obtained from /camera/infra*/camera_info provided by our realsense. You'd better check yours. If you want to get a good result, you are recommended to calibrate the cameras yourself.
3. The rs_camera.launch is used for the ros wrapper based on realsense SDK 2.18.1. For the higher version, some variables' name are changed.
4. Do not use the SDK 2.20(both 2.18 and 2.19 are fine), the IMU data will get stuck.
5. For a stable estimation result, turn off the IR projector of it.

Hi @pjrambo , regarding your provided calibration result for realsense d435i, i found it inconsistent with that from Tf tree. You might check here . Am i doing it wrong?

[haithink]

I also encounttered this problem. But in some cases, there was not this problem And the only difference is the version of some thirdparty libraries such as boost or opencv.

[BarzelS]

Did someone managed to run the VINS-FUSION flow correctly with the D435i?
I'm encountering all kinds of problem along the way:

1. With which opencv version are you compiling it?
2. Did you perform any prior calibration or is it possible to just use the provided files in:
   config/realsense_d435i/realsense_stereo_imu_config.yaml
   config/realsense_d435i/right.yaml
   config/realsense_d435i/left.yaml
   Do I need to perform specific calibration for my D435i?
   How do I do it?
   Which files do I need to change?
3. How were the results for you when using VINS-FUSION with the d435i??

[engcang]

@shirbarzel

Did someone managed to run the VINS-FUSION flow correctly with the D435i?
I'm encountering all kinds of problem along the way:

1. With which opencv version are you compiling it?
2. Did you perform any prior calibration or is it possible to just use the provided files in:
   config/realsense_d435i/realsense_stereo_imu_config.yaml
   config/realsense_d435i/right.yaml
   config/realsense_d435i/left.yaml
   Do I need to perform specific calibration for my D435i?
   How do I do it?
   Which files do I need to change?
3. How were the results for you when using VINS-FUSION with the d435i??

1. OpenCV version 3.4.1 is recommeded according to the authors. and same for me. I used the version.
2. Well I tried with the provided files in config/realsense_D435i and It worked quite well.
   But I usually use it after calibration usgin Kalibr
3. I attached the links of my youtube clips here.

Also let me attach the link of my setup's repository

https://github.com/engcang/VINS-application/tree/Intel-D435i

https://youtu.be/7yMDqiO2A2Q
https://youtu.be/b3l1TeNKyeU
https://youtu.be/sfj1kxMVeMU

Also tested ROVIO with d435i

https://youtu.be/_o2KwT8jJN0

[BarzelS]

@shirbarzel

Did someone managed to run the VINS-FUSION flow correctly with the D435i?
I'm encountering all kinds of problem along the way:

1. With which opencv version are you compiling it?
2. Did you perform any prior calibration or is it possible to just use the provided files in:
   config/realsense_d435i/realsense_stereo_imu_config.yaml
   config/realsense_d435i/right.yaml
   config/realsense_d435i/left.yaml
   Do I need to perform specific calibration for my D435i?
   How do I do it?
   Which files do I need to change?
3. How were the results for you when using VINS-FUSION with the d435i??

1. OpenCV version 3.4.1 is recommeded according to the authors. and same for me. I used the version.
2. Well I tried with the provided files in config/realsense_D435i and It worked quite well.
   But I usually use it after calibration usgin Kalibr
3. I attached the links of my youtube clips here.

Also let me attach the link of my setup's repository

https://github.com/engcang/VINS-application/tree/Intel-D435i

https://youtu.be/7yMDqiO2A2Q
https://youtu.be/b3l1TeNKyeU
https://youtu.be/sfj1kxMVeMU

Also tested ROVIO with d435i

https://youtu.be/_o2KwT8jJN0

1. Thanks @engcang

2. Can you please elaborate more on the process you did with Kalibr?
   a. Is it only used for updating the 3 files:
   config/realsense_d435i/realsense_stereo_imu_config.yaml
   config/realsense_d435i/right.yaml
   config/realsense_d435i/left.yaml
   or it also for other things in the system?
   b. Can you please explain or refer me to the exact process you did with the Kalibr, there are a lot of tutorials in their github.
   c. How do I update the files that are related to VINS in accordance with the calibration I did with Kalibr

3. Currently when I'm using the 3 config yaml files I'm getting large drift without any movement of the camera, like this:
   
   Maybe it's related somehow to my realsense-ros and librealsense versions?
   Have you tested it with any newer versions? (mine is realsense-ROS v2.2.13 librealsense v2.34.1)
   Maybe something changed with the imu-cameras transformations?

Update:
4. I started to perform the kalibr_calibrate_cameras and I came up with some questions:
a. which camera model did you used? (pinhole equi?) why? How did you knew what is the right model?
b. Did you used it with both /camera/infra1/image_rect_raw and infra2?
c. In which fps did you record the bag? kalibr ask for 4hz but the realsense only accept 6hz for the infra?
d. Did you close the emitter before the record?

Regarding the kalibr_calibrate_imu_camera:
How did you get the parameters of "imu_adis*.yaml" file? like accelerometer noise density and so on?
Is there any bias I need to add?
This my data when the camera is not in movement:
angular_velocity:
x: -0.00346277328208
y: -0.0103952223435
z: -0.00348881399259
angular_velocity_covariance: [0.01, 0.0, 0.0, 0.0, 0.01, 0.0, 0.0, 0.0, 0.01]
linear_acceleration:
x: 1.6682462695
y: -9.64356536365
z: -0.612127494984

Is it OK that the linear acceleration in x is high even without any movement?
BTW is it possible that my linear_acceleration in y is actually the acceleration in z and the opposite? is it the same in your case?

Attached bag file of my imu data:
imu.zip

[engcang]

@shirbarzel

Did someone managed to run the VINS-FUSION flow correctly with the D435i?
I'm encountering all kinds of problem along the way:

1. With which opencv version are you compiling it?
2. Did you perform any prior calibration or is it possible to just use the provided files in:
   config/realsense_d435i/realsense_stereo_imu_config.yaml
   config/realsense_d435i/right.yaml
   config/realsense_d435i/left.yaml
   Do I need to perform specific calibration for my D435i?
   How do I do it?
   Which files do I need to change?
3. How were the results for you when using VINS-FUSION with the d435i??

1. OpenCV version 3.4.1 is recommeded according to the authors. and same for me. I used the version.
2. Well I tried with the provided files in config/realsense_D435i and It worked quite well.
   But I usually use it after calibration usgin Kalibr
3. I attached the links of my youtube clips here.

Also let me attach the link of my setup's repository
https://github.com/engcang/VINS-application/tree/Intel-D435i
https://youtu.be/7yMDqiO2A2Q
https://youtu.be/b3l1TeNKyeU
https://youtu.be/sfj1kxMVeMU
Also tested ROVIO with d435i
https://youtu.be/_o2KwT8jJN0

1. Thanks @engcang
2. Can you please elaborate more on the process you did with Kalibr?
   a. Is it only used for updating the 3 files:
   config/realsense_d435i/realsense_stereo_imu_config.yaml
   config/realsense_d435i/right.yaml
   config/realsense_d435i/left.yaml
   or it also for other things in the system?
   b. Can you please explain or refer me to the exact process you did with the Kalibr, there are a lot of tutorials in their github.
   c. How do I update the files that are related to VINS in accordance with the calibration I did with Kalibr
3. Currently when I'm using the 3 config yaml files I'm getting large drift without any movement of the camera, like this:
   
   Maybe it's related somehow to my realsense-ros and librealsense versions?
   Have you tested it with any newer versions? (mine is realsense-ROS v2.2.13 librealsense v2.34.1)
   Maybe something changed with the imu-cameras transformations?

Update:
4. I started to perform the kalibr_calibrate_cameras and I came up with some questions:
a. which camera model did you used? (pinhole equi?) why? How did you knew what is the right model?
b. Did you used it with both /camera/infra1/image_rect_raw and infra2?
c. In which fps did you record the bag? kalibr ask for 4hz but the realsense only accept 6hz for the infra?
d. Did you close the emitter before the record?

Regarding the kalibr_calibrate_imu_camera:
How did you get the parameters of "imu_adis*.yaml" file? like accelerometer noise density and so on?
Is there any bias I need to add?
This my data when the camera is not in movement:
angular_velocity:
x: -0.00346277328208
y: -0.0103952223435
z: -0.00348881399259
angular_velocity_covariance: [0.01, 0.0, 0.0, 0.0, 0.01, 0.0, 0.0, 0.0, 0.01]
linear_acceleration:
x: 1.6682462695
y: -9.64356536365
z: -0.612127494984

Is it OK that the linear acceleration in x is high even without any movement?
BTW is it possible that my linear_acceleration in y is actually the acceleration in z and the opposite? is it the same in your case?

Attached bag file of my imu data:
imu.zip

.
.
.
.
.
.
.
.
.
.
.

heyyyyy
why don't you first check my repository? as I linked above.
there I wrote explanation and also there are my configurations files.....
https://github.com/engcang/vins-application/tree/Intel-D435i
.
.
.
and From your question number 3 and 4,
I strongly recommend you to "calibrate IMU of Intel Realsense D435i" first, before using Kalibr
The link of manual for IMU calibration of d435i from intel, the official one, is also provided in my github repo.....

https://github.com/engcang/vins-application/tree/Intel-D435i#-imu-calibration-recommended-here

I just googled imu parameters. Also for the camera model.

[BarzelS]

@shirbarzel

Did someone managed to run the VINS-FUSION flow correctly with the D435i?
I'm encountering all kinds of problem along the way:

1. With which opencv version are you compiling it?
2. Did you perform any prior calibration or is it possible to just use the provided files in:
   config/realsense_d435i/realsense_stereo_imu_config.yaml
   config/realsense_d435i/right.yaml
   config/realsense_d435i/left.yaml
   Do I need to perform specific calibration for my D435i?
   How do I do it?
   Which files do I need to change?
3. How were the results for you when using VINS-FUSION with the d435i??

1. OpenCV version 3.4.1 is recommeded according to the authors. and same for me. I used the version.
2. Well I tried with the provided files in config/realsense_D435i and It worked quite well.
   But I usually use it after calibration usgin Kalibr
3. I attached the links of my youtube clips here.

Also let me attach the link of my setup's repository

https://github.com/engcang/VINS-application/tree/Intel-D435i

https://youtu.be/7yMDqiO2A2Q
https://youtu.be/b3l1TeNKyeU
https://youtu.be/sfj1kxMVeMU

Also tested ROVIO with d435i

https://youtu.be/_o2KwT8jJN0

I did everything you said here and followed your instrucstions in your repo, thd only thing I did different is that Im using new vdrsions of librealsense and I think thats the main reason because I think intel changed the frame /camera/imu is published

[engcang]

@shirbarzel

Did someone managed to run the VINS-FUSION flow correctly with the D435i?
I'm encountering all kinds of problem along the way:

1. With which opencv version are you compiling it?
2. Did you perform any prior calibration or is it possible to just use the provided files in:
   config/realsense_d435i/realsense_stereo_imu_config.yaml
   config/realsense_d435i/right.yaml
   config/realsense_d435i/left.yaml
   Do I need to perform specific calibration for my D435i?
   How do I do it?
   Which files do I need to change?
3. How were the results for you when using VINS-FUSION with the d435i??

1. OpenCV version 3.4.1 is recommeded according to the authors. and same for me. I used the version.
2. Well I tried with the provided files in config/realsense_D435i and It worked quite well.
   But I usually use it after calibration usgin Kalibr
3. I attached the links of my youtube clips here.

Also let me attach the link of my setup's repository
https://github.com/engcang/VINS-application/tree/Intel-D435i
https://youtu.be/7yMDqiO2A2Q
https://youtu.be/b3l1TeNKyeU
https://youtu.be/sfj1kxMVeMU
Also tested ROVIO with d435i
https://youtu.be/_o2KwT8jJN0

I did everything you said here and followed your instrucstions in your repo, thd only thing I did different is that Im using new vdrsions of librealsense and I think thats the main reason because I think intel changed the frame /camera/imu is published

..
.
.
.
.
.
.
.
.
.

Did you calibrate the IMU of d435i using official librealsense python calibration tool?
your IMU data looks very weird

[BarzelS]

Yes, using the rs-imu-calibration tool, I think that its something related to the frame my imu data is published which I suspect that it has changed, in which frame your Imu data is published?

I saw that in older recordings of the D435i the imu frame_id was camera_imu_frame and for some reason intel-realsense changed it to camera_imu_optical_frame

Maybe you can try to run it with your camera but with the new realsense versions:
RealSense ROS v2.2.15
LibRealSense v2.36.0 ?

[engcang]

Yes, using the rs-imu-calibration tool, I think that its something related to the frame my imu data is published which I suspect that it has changed, in which frame your Imu data is published?

I saw that in older recordings of the D435i the imu frame_id was camera_imu_frame and for some reason intel-realsense changed it to camera_imu_optical_frame

Maybe you can try to run it with your camera but with the new realsense versions:
RealSense ROS v2.2.15
LibRealSense v2.36.0 ?

Why dont you just use older version of LibRealsense then? You can download it from old-release legacy
and also for realsense ros version, using github's commit option.

[BarzelS]

Yes, using the rs-imu-calibration tool, I think that its something related to the frame my imu data is published which I suspect that it has changed, in which frame your Imu data is published?
I saw that in older recordings of the D435i the imu frame_id was camera_imu_frame and for some reason intel-realsense changed it to camera_imu_optical_frame
Maybe you can try to run it with your camera but with the new realsense versions:
RealSense ROS v2.2.15
LibRealSense v2.36.0 ?

Why dont you just use older version of LibRealsense then? You can download it from old-release legacy
and also for realsense ros version, using github's commit option.

I tried to install but it doesnt support my linux kernel version

[engcang]

Yes, using the rs-imu-calibration tool, I think that its something related to the frame my imu data is published which I suspect that it has changed, in which frame your Imu data is published?
I saw that in older recordings of the D435i the imu frame_id was camera_imu_frame and for some reason intel-realsense changed it to camera_imu_optical_frame
Maybe you can try to run it with your camera but with the new realsense versions:
RealSense ROS v2.2.15
LibRealSense v2.36.0 ?

Why dont you just use older version of LibRealsense then? You can download it from old-release legacy
and also for realsense ros version, using github's commit option.

I tried to install but it doesnt support my linux kernel version

Okay. Then why do not you downgrade your kernel version??
You know what, people downgrade their kernel versions to use D435i on Jetson Xavier or TX2 board.
Since we are not working in manufacturer company of the camera or the Ubuntu,
we should find the suitable versions all the time, man.
.
.
And I do not understand why you are suspecting the frame id. that is just for frame id. that does not affect the VINS-Fusion algorithm, you know man.
Let's say something changed because of the frame id, then it should be reflected into your .yaml configuration file as extrinsic calibration between IMU and cameras. But your IMU data looks weird as I told you before.
So I recommended you to re-calibrate your camera's IMU sensor.
.
check your IMU data again and attach here
linear acceleartion data should be around (0, 0, -9.8) when put on the flat surface stable.
And again, ok, let's say something changed because of frame id, but anyway the linear accelration data should be around (0,0, -9.8) or just in different axes but same values, you know man.
.
.
Did you test your VINS-Fusion using only stereo camera but not using IMU first?
if it does not diverge, then IMU must be wrong, but it diverges even withoug IMU, then your camera parameters are also wrong.

[BarzelS]

Thanks!

1. Its a good idea to test vins-fusion only with stereo I will try that. Didnt know that the it has support for it.
2. Actually my linear acceleration data is not (0, 0, -9.8) even on a flat surface stable. I think thats my main reason for this drift. Even after using the calibration tool of intel it still not correct.

[lishanggui]

Hi@engcang ，What is the reprojection error of the image obtained when you calibrate imu-camera using kalibr? I used the official method to calibrate imu, and at the same time used imu-utils to calibrate the noise. After calibrating the infra binocular camera and imu-camera with kalibr, the reprojection error I got was larger (imu-camera calibration). And I can calibrate more perfect results by using mynteye camera.

I used librealsense SDK2.36 and realsense-ros2.2.15 to test the process. At the same time, I also used previous version tests (including SDK2.19) and failed to get good results.
The experimental results are shown in the following link（IntelRealSense/librealsense#7103）

[engcang]

Hi@engcang ，What is the reprojection error of the image obtained when you calibrate imu-camera using kalibr? I used the official method to calibrate imu, and at the same time used imu-utils to calibrate the noise. After calibrating the infra binocular camera and imu-camera with kalibr, the reprojection error I got was larger (imu-camera calibration). And I can calibrate more perfect results by using mynteye camera.

I used librealsense SDK2.36 and realsense-ros2.2.15 to test the process. At the same time, I also used previous version tests (including SDK2.19) and failed to get good results.
The experimental results are shown in the following link（IntelRealSense/librealsense#7103）

Hi @lishanggui

Please refer here : https://github.com/engcang/vins-application/tree/Intel-D435i/kalibr_d435i
I attached all the results of Kalibr using D435i in that folder.

And your camera reprojection error seems ok for me.
But I did not know that IMU is so noisy as you attached the gif.... wow
I am not sure whether that is normal or not.

[xzwang12345]

Hi @engcang, I encountered the same problem as lishanggui when i calibration imu-cam with realsense d435i ,What is the realsense sdk version and firmware version you used? Thanks.

[engcang]

Hi @engcang, I encountered the same problem as lishanggui when i calibration imu-cam with realsense d435i ,What is the realsense sdk version and firmware version you used? Thanks.

Hi. @xzwang12345 . why don't you check my github repo I linked above? I think I already wrote the details there
Thanks.

[liming-jing]

I met the same problem. i don't know how to solve this problem?

vins_node: /usr/include/boost/smart_ptr/shared_ptr.hpp:734: typename boost::detail::sp_member_access::type boost::shared_ptr::operator->() const [with T = camodocal::Camera; typename boost::detail::sp_member_access::type = camodocal::Camera*]: Assertion `px != 0' failed.

please , did you solve this problem? can you help me?

[FPSychotic]

vins_node: /usr/include/boost/smart_ptr/shared_ptr.hpp:734: typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = camodocal::Camera; typename boost::detail::sp_member_access<T>::type = camodocal::Camera*]: Assertion px != 0' failed.`
anyone fixed it?

[celik02]

In my case the problem was number representation. After using scientific number representation in cam0_pinhole.yaml file the issue solved. By scientific, I mean x.xxxxxxe+xxx

I hope that helps

[jgcat]

vins_node: /usr/include/boost/smart_ptr/shared_ptr.hpp:734: typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = camodocal::Camera; typename boost::detail::sp_member_access<T>::type = camodocal::Camera*]: Assertion px != 0' failed.` anyone fixed it?

hello,did you fix it?