Not sure if binning should be considered in IPM, or be converted to image coordinates prior

[ijnek]

Currently, binning is considered within IPM such that the input point can be a "binned" pixel coordinate (I'm not sure what the correct terminology here is).

I'm wondering if it is necessary to consider the binning inside IPM, or outside IPM by ensuring pixels are converted to those that match the original width x height of the image.

[Flova]

We are using this feature at Bit-Bots. The official image proc software binning node does it like that, and there is also support for that in the camera_info message. It only changes the binning params in the camera_info keeping it otherwise the same. I think the idea is that roi, binning, etc don't change the original camera calibration matrix and instead provide an additional transformation that needs to be applied. This makes it a bit harder for us, as we need to consider this, but it also makes the camera_info cleaner, as the same camera always has the same camera matrix. I think we should use the default way of handling this instead of e.g. creating an additional node that creates "unbinned" camera_infos.

Or did you mean "outside" of the projection, but still inside the ipm libary? Because I think we need to handle the binning param in the camera_info ether way. You could also scale the pixels instead of the camera_matrix, but this is the exact same thing only the other way around, and I see no benefits in changing the implementation. The binning param is available ether way at this stage due to the CameraInfo message being passed to the projection function. The presence of the complete CameraInfo msg type will also be helpful in the future if we consider the distortion matrix in the projection.

[ijnek]

I see, thinking twice I think we should handle binning_x and binning_y inside IPM, and project_point() should be called with "binned" pixels like you currently are doing.

For consistency, would it make sense to handle ROIs from CameraInfo aswell? (Related to #25)

[Flova]

It makes sense to handle ROIs but their implementation is currently not my to priory. We should open an issue for this.