What is the coordinate convention for the saved dataparser_transforms.json ?

[AruniRC]

It is not clear if the dataparser_transforms.json "transform" matrix is in OpenCV or OpenGL convention, if the data pre-processing, say, was done using COLMAP (which would be using OpenCV).

This makes it hard to correctly apply this transformation, if I am trying to convert camera poses in the input coordinate system (ie. whatever COLMAP gives) into the coordinate system used by the trained Nerf.

[lyg1597]

Hope this answer helps with the relationship between colmap coordinate frame and trained nerf coordinate frame.
#3488 (comment)

[AruniRC]

Thank you for the response @lyg1597 .

However, I think one more bit of information is needed to correctly apply the transform you pointed out: we still do not know the coordinate convention of the transformation matrix and scale factor that are found in dataparser_transforms.json. That is, is it saved in the OpenCV format (which colmap uses) or the OpenGL format (which nerfstudio uses).

[lyg1597]

I am not that familar with the difference/conversion between OpenCV and OpenGL convention, but I can describe what I am doing. The steps that I am using is:

1. Collecct images and camera poses from gazebo simulator.
2. Use script from Problem using ORB-SLAM2 results instead of COLMAP for the data processing step #3101 (comment) (thanks for your solution) to perform the first coordinate transformation.
3. Apply the second coordinate transformation I described in the previous comment Transform conversion from colmap transform.json to camera_path.json #3488 (comment).
4. Render images at the resulting camera pose.

The images that I am getting from the last step looks exactly (except nerf artifect) the same as the images directly from the simulator. Hope this is helpful for your problem.

[AruniRC]

@lyg1597 - this is super helpful, thank you for the details. The step 4 that you mention is basically calling ns-render from the cmd line on the camera_path.json you generated in step 3?

[lyg1597]

No, I am using my own renderer because I want minimum code that can give me single images. But all functions I am using is from nerfstudio/ns-render.