[JOSS Review] Enhance plotting ability

[sreschechtko]

The package would be more useful if it included ability to plot the 3D reconstruction in synchrony with at least one camera view. This functionality exists in the demo, but it is not implemented in the actual package, which is confusing. WIthout synchronized video it is difficult to verify the performance of the reconstruction.

[SwathiSheshadri]

A very good point, thank you! We have updated the simple_3Ddata_plotter.m file to now show 3D reconstructed data alongside user selected 2D tracked video. The path to the video the user wishes to visualize alongside the 3D reconstructed results has to be entered in the config file. Visualization of more than one 2D tracked videos alongside 3D reconstructed data is also possible using pose3d. This feature is included as a helper function named make_illustrative_movie.m and can be used optionally.

[sreschechtko]

Thank you for adding this functionality. I think it would be better implemented by having the option to input a path to an untracked video and then to plot the 2d points on this video since the program already has access to the tracked points in 2d. At present, if someone does not have 2d tracked videos saved, they would have to go back into another application to produce these videos.

Additionally, I think the error thrown by the plotter when no video (or images) is present but plotting tracked video option is on is difficult to interpret because it just says the file doesn't exist at this path and requires multiple layers of trackback. Please raise a more specific error.

[SwathiSheshadri]

Thank you for this feature suggestion. The plotting function has been updated to allow feature annotation along with plotting video frames/images used for 2D tracking recorded from primary camera. Since, primary camera is typically positioned to have the best view, pose3d plots movies or images recorded from primary camera next to 3D reconstructed results. I took this approach to reduce the number of user inputs required when filling the config file. Alternatively, make_illustrative_movie.m helper function is now updated to allow users to flexibly select any number of perspectives they want to visualize alongside 3D reconstructed results. Furthermore, error handling in the config file has been upgraded to deal with missing video/image path entries in the config file.

[sreschechtko]

Thanks for the updates; could you explain in more detail how make_illustrative_movie is supposed to be used? Is it intended to be a standalone function or is it called by another helper?

[SwathiSheshadri]

make_illustrative_movie is a standalone function which can be run to visualize 3D reconstructed results alongside any number of 2D videos recorded during the experiment. This can be used by editing the make_illustrative_movie.m function to include the config file that was used to do 3D reconstruction and then providing paths to the videos and .csv files holding 2D tracked videos for the respective videos. Successful execution of make_illustrative_movie.m function, plots all frames from the selected videos, alongside 3D reconstructed results and saves the results in .avi format in a subfolder called Videos in your experiment folder.

[SwathiSheshadri]

Thanks for following up on this issue Sasha. I was looking into make_illustrative_movie.m function and found that I had hardcoded the axis limits! Since it worked for our data, it passed all the tests I did! This function is now updated to automatically set axis limits based on the maximum and minimum values in the dataset it is used on. Please download the latest version of make_illustrative_movie.m function from the repository and let me know if you have any other question/suggestion for this function.

[sreschechtko]

Thanks for the update. I got it to work! I feel like it would make more sense to just automatically generate the video using this script instead of the simple 3D data plotter, but maybe you're of a different opinion 😃

[SwathiSheshadri]

Thanks for verifying that the update works! Super happy to hear back from you :)). I programmed simple 3D data plotter for quick visualization purposes, since the movie generation with make_illustrative_movie.m can take a long while especially if users have high data volumes coming from a large number of cameras, high resolution images, higher frame rate etc,. Also, from our recording setup we tried out the visualization from 5 cameras, and it was too overwhelming to view all 5 2D movies alongside 3D reconstruction results at once.
I will add a pointer in the readme file to direct users to try out the make_illustrative_movie.m file after running the main function and verifying the results. Thanks again so much for your very helpful feedback and suggestions :)).