cv::aruco::detectMarkers consumes all available memory when CORNER_REFINE_CONTOUR

[sebasth]

System information (version)

• OpenCV => 4.2.0 and 4.5.0
• Operating System / Platform => Ubuntu 18.04.5
• Compiler => GCC 7.5.0

Detailed description

cv::aruco::detectMarkers consumes all available memory when CORNER_REFINE_CONTOUR is used and camera distortion parameter are provided. This does not happen if no distortion parameters are provided. This happens in both C++ and in Python. Example program below which demonstrates the issue.

I have tested this with with OpenCV versions 4.2.0 and 4.5.0.

Steps to reproduce

Example creates aruco marker and places it in the middle of the image. Dummy camera matrix (focal length is image width) is used, but the issue also present when real images calibrated camera values are used.

import cv2 as cv
import numpy as np

aruco_dict = cv.aruco.getPredefinedDictionary(cv.aruco.DICT_4X4_100)
# create 100x100 aruco marker in center of 1280x720 image
sz = 100
w, h = 1280, 720

marker = cv.aruco.drawMarker(aruco_dict, 0, sz)
im = np.zeros([h, w, 3], dtype=np.uint8)
im += 255
im[((h-sz)//2):((h+sz)//2), ((w-sz)//2):((w+sz)//2)] = marker[..., np.newaxis]

camera_matrix = np.identity(3, dtype=np.float)
camera_matrix[0, 0] = w
camera_matrix[1, 1] = w
camera_matrix[0, 2] = w/2
camera_matrix[1, 2] = h/2
dist_coeffs = None

print("camera matrix: \n", camera_matrix)
print("distortion coefficients: ", dist_coeffs)

params = cv.aruco.DetectorParameters_create()
params.cornerRefinementMethod = cv.aruco.CORNER_REFINE_CONTOUR

# works
corners, ids, _ = cv.aruco.detectMarkers(im, aruco_dict, None, None,
                                         params, None, camera_matrix, dist_coeffs)
print("corners:\n", corners)

print("\nwith distortion cofficients")

# consumes all available memory (OOM)
dist_coeffs = np.zeros([5], dtype=np.float)
print("distortion cofficients: ", dist_coeffs)
corners, ids, _ = cv.aruco.detectMarkers(im, aruco_dict, None, None,
                                         params, None, camera_matrix, dist_coeffs)

Output

camera matrix: 
 [[  1.28000000e+03   0.00000000e+00   6.40000000e+02]
 [  0.00000000e+00   1.28000000e+03   3.60000000e+02]
 [  0.00000000e+00   0.00000000e+00   1.00000000e+00]]
distortion coefficients:  None
corners:
 [array([[[ 590.00054932,  309.99969482],
        [ 688.99957275,  310.00030518],
        [ 689.00036621,  408.99914551],
        [ 589.99951172,  409.00085449]]], dtype=float32)]

with distortion cofficients
distortion cofficients:  [ 0.  0.  0.  0.  0.]

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues,
  answers.opencv.org, Stack Overflow, etc and have not found solution
• I updated to latest OpenCV version and the issue is still there
• There is reproducer code and related data files: videos, images, onnx, etc

[sebasth]

Bug possibly in aruco.cpp:840, branch is taken when both camera matrix and distortion coefficients are provided.

[catree]

I have converted the Python code in C++ to be able to run it with a debugger:

    Ptr<aruco::Dictionary> aruco_dict =
        aruco::getPredefinedDictionary(aruco::PREDEFINED_DICTIONARY_NAME(aruco::DICT_4X4_100));

    // create 100x100 aruco marker in center of 1280x720 image
    int sz = 100;
    int w = 1280, h = 720;

    Mat marker;
    aruco::drawMarker(aruco_dict, 0, sz, marker);
    cout << "marker: " << marker.rows << "x" << marker.cols << " ; channels: " << marker.channels() << endl;

    Mat im(h, w, CV_8UC3, cv::Scalar(255,255,255));

    for (int i = (h - sz) / 2; i < (h + sz) / 2; i++) {
        for (int j = (w - sz) / 2; j < (w + sz) / 2; j++) {
            im.at<Vec3b>(i, j)[0] = marker.at<uchar>(i - (h - sz) / 2, j - (w - sz) / 2);
            im.at<Vec3b>(i, j)[1] = marker.at<uchar>(i - (h - sz) / 2, j - (w - sz) / 2);
            im.at<Vec3b>(i, j)[2] = marker.at<uchar>(i - (h - sz) / 2, j - (w - sz) / 2);
        }
    }

    Mat camera_matrix = Mat::eye(3, 3, CV_64FC1);
    camera_matrix.at<double>(0, 0) = w;
    camera_matrix.at<double>(1, 1) = h;
    camera_matrix.at<double>(0, 2) = w/2;
    camera_matrix.at<double>(1, 2) = h/2;

    Ptr<aruco::DetectorParameters> params = aruco::DetectorParameters::create();
    params->cornerRefinementMethod = aruco::CORNER_REFINE_CONTOUR;

    Mat dist_coeffs = Mat::zeros(5, 1, CV_64FC1);

    vector< int > markerIds;
    vector< vector< Point2f > > markerCorners, rejectedMarkers;
    aruco::detectMarkers(im, aruco_dict, markerCorners, markerIds, params, rejectedMarkers, camera_matrix, dist_coeffs);

Looks like there is an infinite loop here:

opencv_contrib/modules/aruco/src/aruco.cpp

Lines 805 to 810 in ae4d5d3

	// saves extra group into corresponding
	if( !cntPts[4].empty() ){
	for( unsigned int i=0; i < cntPts[4].size() ; i++ )
	cntPts[group].push_back(cntPts[4].at(i));
	cntPts[4].clear();
	}

When you print the variables, you have:

cntPts[0]: 0
cntPts[1]: 0
cntPts[2]: 0
cntPts[3]: 0
cntPts[4]: 396
group: 4

So you keep adding points in the vector indefinitely when group=4:

opencv_contrib/modules/aruco/src/aruco.cpp

Lines 807 to 808 in ae4d5d3

	for( unsigned int i=0; i < cntPts[4].size() ; i++ )
	cntPts[group].push_back(cntPts[4].at(i));

This is just a quick debugging and I am not interested into debugging it any further. I don't like Aruco and prefer AprilTag.

[esopalumpa]

Problem probably is that points in nContours are undistorted into contour2f

	vector<Point2f> contour2f(nContours.begin(), nContours.end());

	if(!camMatrix.empty() && !distCoeff.empty()){
		undistortPoints(contour2f, contour2f, camMatrix, distCoeff);
	}

and later compared to the not-undistorted points in nCorners.

	for ( unsigned int i =0; i < nContours.size(); i++ ) {
		for(unsigned int j=0; j<4; j++){
			if ( nCorners[j] == contour2f[i] ){
				cornerIndex[j] = i;
				group=j;
			}
		}
		cntPts[group].push_back(contour2f[i]);
	}

which never yields true.

That’s in _refineCandidateLines()

[alalek]

@catree Thank you for initial investigation!
Code with infinite loop comes from #973

Added a quick workaround to eliminate infinite loop: #2793 (does not close this issue)

@sebasth @esopalumpa Feel free to prepare PR with complete fix for this issue.