data model for camera orientation

[mbredif]

We need a image orientation data model that would be general enough to support :

• individual images
• unassembled panoramic images (rigid multi camera rig mounted on a moving acquisition vector)
• assembled panoramic images (cubemaps, equirectangular images...)
• optional distorsion

Here is a proposed data model encoded in JSON, modified from the camera calibration of itowns v1.

The main idea is to list the transform in the order they are applied to transform a point from the world coordinate system (eg: Lambert 93) to the image pixel coordinate system ([0,width]x[0,height] with integer coordinates at pixel corners ).

{
"id": 300,
"size" : [2048, 2048],
"transfo" : 
    [
        {
            "description":"MMS trajectory",
            "type" : "affine",
            "trajectory": 1
        },
        {   
            "description":"itownsWay",
            "type" : "linear",
            "mat3" : [
                0, 0, 1,
                1, 0, 0,
                0,-1, 0
            ]
        },
        {   
            "description":"position",
            "type" : "translation",
            "vec3" : [ 0.145711, 0.0008142, 0.867 ]
        },
        {   
            "description":"rotation",
            "type" : "linear",
            "mat3" : [ 
                0.000735648, -0.00148884, -0.999999,
                0.999998, 0.00205265, 0.000732591,
                0.00205156, -0.999997 , 0.00149035
            ]
        },
        {   
            "description":"JMM",
            "type" : "linear",
            "mat3": [ 
                 0, -1, 0,
                 0,  0, 1,
                -1,  0, 0
            ]
        },
        {   
            "description":"rot0, rot90, rot180, rot270",
            "type" : "linear",
            "orientation": 3
        },
        {   
            "description":"photgramme_image",
            "type" : "linear",
            "mat3" : [ 
                1, 0, 0,
                0,-1, 0,
                0, 0,-1
            ]
        },
        {   
            "description":"projection",
            "type" : "linear",
            "mat3" : [ 
                1150.66785706630299, 0, 1030.29197487242254,
                0, 1150.66785706630299, 1023.03935469545331,
                0, 0, 1
            ]
        },
        {
            "description":"disto357",
            "type" : "distortion",
            "pps": [ 1042.178, 1020.435 ],
            "poly357": [-1.33791587603751819E-7, 3.47540977328314388E-14, -4.44103985918888078E-21]
        }
    ]
}

The exemple above explicits all convention change matrices so that they would no longer be necessary.

What itowns would have to do project such a texture is :

• check that transforms array is composed by a set of affine transforms (affine, linear, rotation, translation...), followed. These may be coalesced to a single affine transform and sent as a uniform to the fragment shader (as a mat3 + vec3). This coalescing may be done for each camera at initialization for consecutive affine transforms that are not trajectories. The final coalescing needs to be done when the trajectories are all interpolated.
• if there is a mask, add the masking fragment shader chunk and mask sampler uniform
• if there is a distorsion, add the corresponding fragment shader chunk and uniforms
• finally the fragment shader applies the perspective and image dimension divides to get the texture coordinates to texture at.

@vpicavet , @gmaillet ?

[mbredif]

here is an implementation on itowns v1 to read such a calibration file : https://github.com/iTowns/itowns/tree/transfo_list

[vpicavet]

LGTM
Comments are not supported in JSON.
And we have to decide is if we want in the API to have just IDs, resource address or URLs. @ldgeo or @tbroyer an opinion about this ?

[vpicavet]

To have a level3 REST API, it is probably better to use URIs instead of ids :

{
            "description":"MMS trajectory",
            "type" : "affine",
            "trajectory": '/traj/1'
        },

[gchoqueux]

@mbredif, @gliegard Where is the specification of the image/camera orientation ?

[mgermerie]

@gchoqueux @mbredif, would you know what is left to do regarding this issue ?