Implementation of Quasi Dense Stereo algorithm.

modules/README.md

@@ -54,7 +54,7 @@ $ cmake -D OPENCV_EXTRA_MODULES_PATH=<opencv_contrib>/modules -D BUILD_opencv_<r
 
 - **sfm**: Structure from Motion -- This module contains algorithms to perform 3d reconstruction from 2d images. The core of the module is a light version of Libmv.
 
-- **stereo**: Stereo Correspondence -- Stereo matching done with different descriptors: Census / CS-Census / MCT / BRIEF / MV.
+- **stereo**: Stereo Correspondence -- Stereo matching done with different descriptors: Census / CS-Census / MCT / BRIEF / MV and dense stereo correspondence using Quasi Dense Stereo method.
 
 - **structured_light**: Structured Light Use -- How to generate and project gray code patterns and use them to find dense depth in a scene.
 

modules/stereo/CMakeLists.txt

@@ -1,2 +1,2 @@
 set(the_description "Stereo Correspondence")
-ocv_define_module(stereo opencv_imgproc opencv_features2d opencv_core opencv_calib3d)
+ocv_define_module(stereo opencv_imgproc opencv_features2d opencv_core opencv_calib3d opencv_tracking opencv_video)

modules/stereo/README.md

@@ -2,3 +2,10 @@ Stereo Correspondence with different descriptors
 ================================================
 
 Stereo matching done with different descriptors: Census / CS-Census / MCT / BRIEF / MV.
+
+Quasi Dense Stereo
+======================
+
+Quasi Dense Stereo is method for performing dense stereo matching.
+The code uses pyramidal Lucas-Kanade with Shi-Tomasi features to get the initial seed correspondences.
+Then these seeds are propagated by using mentioned growing scheme.

modules/stereo/doc/stereo.bib

@@ -0,0 +1,36 @@
+@InProceedings{Stoyanov2010,
+author="Stoyanov, Danail
+and Scarzanella, Marco Visentini
+and Pratt, Philip
+and Yang, Guang-Zhong",
+editor="Jiang, Tianzi
+and Navab, Nassir
+and Pluim, Josien P. W.
+and Viergever, Max A.",
+title="Real-Time Stereo Reconstruction in Robotically Assisted Minimally Invasive Surgery",
+booktitle="Medical Image Computing and Computer-Assisted Intervention (MICCAI 2010)",
+year="2010",
+publisher="Springer Berlin Heidelberg",
+address="Berlin, Heidelberg",
+pages="275--282",
+abstract="The recovery of 3D tissue structure and morphology during robotic assisted surgery is an important step towards accurate deployment of surgical guidance and control techniques in minimally invasive therapies. In this article, we present a novel stereo reconstruction algorithm that propagates disparity information around a set of candidate feature matches. This has the advantage of avoiding problems with specular highlights, occlusions from instruments and view dependent illumination bias. Furthermore, the algorithm can be used with any feature matching strategy allowing the propagation of depth in very disparate views. Validation is provided for a phantom model with known geometry and this data is available online in order to establish a structured validation scheme in the field. The practical value of the proposed method is further demonstrated by reconstructions on various in vivo images of robotic assisted procedures, which are also available to the community.",
+isbn="978-3-642-15705-9"
+}
+
+
+
+
+@article{Lhuillier2000,
+abstract = {A new robust dense matching algorithm is introduced. The algorithm$\backslash$nstarts from matching the most textured points, then a match propagation$\backslash$nalgorithm is developed with the best first strategy to dense matching.$\backslash$nNext, the matching map is regularised by using the local geometric$\backslash$nconstraints encoded by planar affine applications and by using the$\backslash$nglobal geometric constraint encoded by the fundamental matrix. Two most$\backslash$ndistinctive features are a match propagation strategy developed by$\backslash$nanalogy to region growing and a successive regularisation by local and$\backslash$nglobal geometric constraints. The algorithm is efficient, robust and can$\backslash$ncope with wide disparity. The algorithm is demonstrated on many real$\backslash$nimage pairs, and applications on image interpolation and a creation of$\backslash$nnovel views are also presented},
+author = {Lhuillier, Maxime and Quan, Long},
+doi = {10.1109/ICPR.2000.905620},
+file = {:home/dimitrisps/Desktop/ucl/papers/quasiDenseMatching.pdf:pdf},
+isbn = {0-7695-0750-6},
+issn = {10514651},
+journal = {Proceedings-International Conference on Pattern Recognition},
+number = {1},
+pages = {968--972},
+title = {{Robust dense matching using local and global geometric constraints}},
+volume = {15},
+year = {2000}
+}

modules/stereo/include/opencv2/stereo.hpp

@@ -49,6 +49,7 @@
 #include "opencv2/core/affine.hpp"
 #include "opencv2/stereo/descriptor.hpp"
 #include "opencv2/stereo/matching.hpp"
+#include <opencv2/stereo/quasi_dense_stereo.hpp>
 
 /**
 @defgroup stereo Stereo Correspondance Algorithms

modules/stereo/include/opencv2/stereo/quasi_dense_stereo.hpp

@@ -0,0 +1,234 @@
+/*
+By downloading, copying, installing or using the software you agree to this license.
+If you do not agree to this license, do not download, install,
+copy or use the software.
+
+
+                          License Agreement
+               For Open Source Computer Vision Library
+                       (3-clause BSD License)
+
+Copyright (C) 2015-2018, OpenCV Foundation, all rights reserved.
+Third party copyrights are property of their respective owners.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+  * Redistributions of source code must retain the above copyright notice,
+    this list of conditions and the following disclaimer.
+
+  * Redistributions in binary form must reproduce the above copyright notice,
+    this list of conditions and the following disclaimer in the documentation
+    and/or other materials provided with the distribution.
+
+  * Neither the names of the copyright holders nor the names of the contributors
+    may be used to endorse or promote products derived from this software
+    without specific prior written permission.
+
+This software is provided by the copyright holders and contributors "as is" and
+any express or implied warranties, including, but not limited to, the implied
+warranties of merchantability and fitness for a particular purpose are disclaimed.
+In no event shall copyright holders or contributors be liable for any direct,
+indirect, incidental, special, exemplary, or consequential damages
+(including, but not limited to, procurement of substitute goods or services;
+loss of use, data, or profits; or business interruption) however caused
+and on any theory of liability, whether in contract, strict liability,
+or tort (including negligence or otherwise) arising in any way out of
+the use of this software, even if advised of the possibility of such damage.
+ */
+
+
+//__OPENCV_QUASI_DENSE_STEREO_H__
+#ifndef __OPENCV_QUASI_DENSE_STEREO_H__
+#define __OPENCV_QUASI_DENSE_STEREO_H__
+
+
+
+#include <opencv2/core.hpp>
+#include <queue>
+
+
+
+namespace cv
+{
+namespace stereo
+{
+/** \addtogroup stereo
+ *  @{
+ */
+
+
+// A basic match structure
+struct CV_EXPORTS Match
+{
+    cv::Point2i p0;
+    cv::Point2i p1;
+    float	corr;
+
+    bool operator < (const Match & rhs) const//fixme  may be used uninitialized in this function
+    {
+        return this->corr < rhs.corr;
+    }
+};
+struct CV_EXPORTS PropagationParameters
+{
+    int	corrWinSizeX;			// similarity window
+    int	corrWinSizeY;
+
+    int borderX;					// border to ignore
+    int borderY;
+
+    //matching
+    float correlationThreshold;	// correlation threshold
+    float textrureThreshold;		// texture threshold
+
+    int	  neighborhoodSize;		// neighborhood size
+    int	  disparityGradient;	// disparity gradient threshold
+
+    // Parameters for LK flow algorithm
+    int lkTemplateSize;
+    int lkPyrLvl;
+    int lkTermParam1;
+    float lkTermParam2;
+
+    // Parameters for GFT algorithm.
+    float gftQualityThres;
+    int gftMinSeperationDist;
+    int gftMaxNumFeatures;
+
+};
+
+typedef std::priority_queue<Match, std::vector<Match>, std::less<Match> > t_matchPriorityQueue;
+
+
+/**
+ * @brief Class containing the methods needed for Quasi Dense Stereo computation.
+ *
+ * This module contains the code to perform quasi dense stereo matching.
+ * The method initially starts with a sparse 3D reconstruction based on feature matching across a
+ * stereo image pair and subsequently propagates the structure into neighboring image regions.
+ * To obtain initial seed correspondences, the algorithm locates Shi and Tomashi features in the
+ * left image of the stereo pair and then tracks them using pyramidal Lucas-Kanade in the right image.
+ * To densify the sparse correspondences, the algorithm computes the zero-mean normalized
+ * cross-correlation (ZNCC) in small patches around every seed pair and uses it as a quality metric
+ * for each match. In this code, we introduce a custom structure to store the location and ZNCC value
+ * of correspondences called "Match". Seed Matches are stored in a priority queue sorted according to
+ * their ZNCC value, allowing for the best quality Match to be readily available. The algorithm pops
+ * Matches and uses them to extract new matches around them. This is done by considering a small
+ * neighboring area around each Seed and retrieving correspondences above a certain texture threshold
+ * that are not previously computed. New matches are stored in the seed priority queue and used as seeds.
+ * The propagation process ends when no additional matches can be retrieved.
+ *
+ *
+ * @sa This code represents the work presented in @cite Stoyanov2010.
+ * If this code is useful for your work please cite @cite Stoyanov2010.
+ *
+ * Also the original growing scheme idea is described in @cite Lhuillier2000
+ *
+ */
+
+class CV_EXPORTS QuasiDenseStereo
+{
+public:
+    /**
+     * @brief destructor
+     * Method to free all the memory allocated by matrices and vectors in this class.
+     */
+    virtual ~QuasiDenseStereo() = 0;
+
+
+    /**
+     * @brief Load a file containing the configuration parameters of the class.
+     * @param[in] filepath The location of the .YAML file containing the configuration parameters.
+     * @note default value is an empty string in which case the default parameters will be loaded.
+     * @retval 1: If the path is not empty and the program loaded the parameters successfully.
+     * @retval 0: If the path is empty and the program loaded default parameters.
+     * @retval -1: If the file location is not valid or the program could not open the file and
+     * loaded default parameters from defaults.hpp.
+     * @note The method is automatically called in the constructor and configures the class.
+     * @note Loading different parameters will have an effect on the output. This is useful for tuning
+     * in case of video processing.
+     * @sa loadParameters
+     */
+    virtual int loadParameters(cv::String filepath="") = 0;
+
+
+    /**
+     * @brief Save a file containing all the configuration parameters the class is currently set to.
+     * @param[in] filepath The location to store the parameters file.
+     * @note Calling this method with no arguments will result in storing class parameters to a file
+     * names "qds_parameters.yaml" in the root project folder.
+     * @note This method can be used to generate a template file for tuning the class.
+     * @sa loadParameters
+     */
+    virtual int saveParameters(cv::String filepath="./qds_parameters.yaml") = 0;
+
+
+    /**
+     * @brief Get The sparse corresponding points.
+     * @param[out] sMatches A vector containing all sparse correspondences.
+     * @note The method clears the sMatches vector.
+     * @note The returned Match elements inside the sMatches vector, do not use corr member.
+     */
+    virtual void getSparseMatches(std::vector<stereo::Match> &sMatches) = 0;
+
+
+    /**
+     * @brief Get The dense corresponding points.
+     * @param[out] dMatches A vector containing all dense matches.
+     * @note The method clears the dMatches vector.
+     * @note The returned Match elements inside the sMatches vector, do not use corr member.
+     */
+    virtual void getDenseMatches(std::vector<stereo::Match> &dMatches) = 0;
+
+
+
+    /**
+     * @brief Main process of the algorithm. This method computes the sparse seeds and then densifies them.
+     *
+     * Initially input images are converted to gray-scale and then the sparseMatching method
+     * is called to obtain the sparse stereo. Finally quasiDenseMatching is called to densify the corresponding
+     * points.
+     * @param[in] imgLeft The left Channel of a stereo image pair.
+     * @param[in] imgRight The right Channel of a stereo image pair.
+     * @note If input images are in color, the method assumes that are BGR and converts them to grayscale.
+     * @sa sparseMatching
+     * @sa quasiDenseMatching
+     */
+    virtual void process(const cv::Mat &imgLeft ,const cv::Mat &imgRight) = 0;
+
+
+    /**
+     * @brief Specify pixel coordinates in the left image and get its corresponding location in the right image.
+     * @param[in] x The x pixel coordinate in the left image channel.
+     * @param[in] y The y pixel coordinate in the left image channel.
+     * @retval cv::Point(x, y) The location of the corresponding pixel in the right image.
+     * @retval cv::Point(0, 0) (NO_MATCH)  if no match is found in the right image for the specified pixel location in the left image.
+     * @note This method should be always called after process, otherwise the matches will not be correct.
+     */
+    virtual cv::Point2f getMatch(const int x, const int y) = 0;
+
+
+    /**
+     * @brief Compute and return the disparity map based on the correspondences found in the "process" method.
+     * @param[in] disparityLvls The level of detail in output disparity image.
+     * @note Default level is 50
+     * @return cv::Mat containing a the disparity image in grayscale.
+     * @sa computeDisparity
+     * @sa quantizeDisparity
+     */
+    virtual cv::Mat getDisparity(uint8_t disparityLvls=50) = 0;
+
+
+    static cv::Ptr<QuasiDenseStereo> create(cv::Size monoImgSize, cv::String paramFilepath ="");
+
+
+    PropagationParameters Param;
+};
+
+} //namespace cv
+} //namespace stereo
+
+/** @}*/
+
+#endif // __OPENCV_QUASI_DENSE_STEREO_H__

modules/stereo/samples/dense_disparity.cpp

@@ -0,0 +1,58 @@
+#include <opencv2/core.hpp>
+#include <opencv2/highgui.hpp>
+#include <fstream>
+#include <opencv2/stereo.hpp>
+
+
+
+
+using namespace cv;
+using namespace std;
+
+
+int main()
+{
+
+    Mat rightImg, leftImg;
+
+    //Read video meta-data to determine the correct frame size for initialization.
+    leftImg = imread("./imgLeft.png", IMREAD_COLOR);
+    rightImg = imread("./imgRight.png", IMREAD_COLOR);
+    cv::Size frameSize = leftImg.size();
+    // Initialize qds and start process.
+    Ptr<stereo::QuasiDenseStereo> stereo = stereo::QuasiDenseStereo::create(frameSize);
+
+    uint8_t displvl = 80;					// Number of disparity levels
+    cv::Mat disp;
+
+    // Compute dense stereo.
+    stereo->process(leftImg, rightImg);
+
+    // Compute disparity between left and right channel of current frame.
+    disp = stereo->getDisparity(displvl);
+
+    vector<stereo::Match> matches;
+    stereo->getDenseMatches(matches);
+
+    // Create three windows and show images.
+    cv::namedWindow("right channel");
+    cv::namedWindow("left channel");
+    cv::namedWindow("disparity map");
+    cv::imshow("disparity map", disp);
+    cv::imshow("left channel", leftImg);
+    cv::imshow("right channel", rightImg);
+
+    std::ofstream dense("./dense.txt", std::ios::out);
+
+    for (uint i=0; i< matches.size(); i++)
+    {
+        dense << matches[i].p0 << matches[i].p1 << endl;
+    }
+    dense.close();
+
+
+
+    cv::waitKey(0);
+
+    return 0;
+}

modules/stereo/samples/export_param_file.cpp

@@ -0,0 +1,18 @@
+#include <opencv2/core.hpp>
+#include <opencv2/stereo.hpp>
+
+using namespace cv;
+using namespace std;
+
+int main(int argc, char* argv[])
+{
+    std::string parameterFileLocation = "./parameters.yaml";
+    if (argc > 1)
+        parameterFileLocation = argv[1];
+
+
+    Ptr<stereo::QuasiDenseStereo> stereo =  stereo::QuasiDenseStereo::create(cv::Size(5,5));
+    stereo->saveParameters(parameterFileLocation);
+
+    return 0;
+}