Aruco detection is worsened since OpenCV 4.1.2

[stefanopini]

System information (version)

• OpenCV => 4.2.0.34 (opencv-contrib-python)
• Operating System / Platform => Windows 10 64bit / Python 3.6.4
• Compiler => Package installed with pip

Detailed description

The last version of the Aruco detection algorithm ( see pull request #2236 ), which is included in OpenCV 4.1.2 and above, has worse performance than the previous one when the outer border of black markers is small.
The algorithm completely fails to find black markers whose outer white border is quite small (just few pixels) with both real and synthetic images. The old algorithm finds them.
I tried to change some DetectorParameters (e.g. minMarkerPerimeterRate, minCornerDistanceRate and markerBorderBits) without success.

Steps to reproduce

Please see the attached reference Python code and image.
Code is tested with OpenCV 4.1.1.26 (8/8 markers found) and 4.2.0.34 (2/8 markers found) with Python 3.6.4. Corresponding results are reported.

aruco-detection-issue.zip

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues,
  answers.opencv.org, Stack Overflow, etc and have not found solution
• I updated to latest OpenCV version (available through PyPI) and the issue is still there
• There is reproducer code and related data files: videos, images, onnx, etc

[paroj]

@szk1509

[szk1509]

Hi @stefanopini, you can reduce the minimum marker distance rate (in this case to 0.025), which allows to recognize all the markers at the bottom of the image provided, i.e. the synthetic markers.

[stefanopini]

Hi @szk1509 , thank you for spotting it.
However, it seems to be just a workaround: the issue doesn't seem related to the distance between markers (it seems to be caused by the small/missing outer white border only from OpenCV 4.1.2) and it is still unable to detect half of the markers.

I looked again at your pull request (#2236 improving the accuracy with white markers), but I didn't find what may cause the misdetection of markers without external white border.
May it be something related to this part of the code?

opencv_contrib/modules/aruco/src/aruco.cpp

Lines 249 to 322 in 93f910e

	if(distSq < minMarkerDistancePixels * minMarkerDistancePixels) {
	// i and j are not related to a group
	if(candGroup[i]<0 && candGroup[j]<0){
	// mark candidates with their corresponding group number
	candGroup[i] = candGroup[j] = (int)groupedCandidates.size();
	// create group
	vector<unsigned int> grouped;
	grouped.push_back(i);
	grouped.push_back(j);
	groupedCandidates.push_back( grouped );
	}
	// i is related to a group
	else if(candGroup[i] > -1 && candGroup[j] == -1){
	int group = candGroup[i];
	candGroup[j] = group;
	// add to group
	groupedCandidates[group].push_back( j );
	}
	// j is related to a group
	else if(candGroup[j] > -1 && candGroup[i] == -1){
	int group = candGroup[j];
	candGroup[i] = group;
	// add to group
	groupedCandidates[group].push_back( i );
	}
	}
	}
	}
	}
	// save possible candidates
	candidatesSetOut.clear();
	contoursSetOut.clear();
	vector< vector< Point2f > > biggerCandidates;
	vector< vector< Point > > biggerContours;
	vector< vector< Point2f > > smallerCandidates;
	vector< vector< Point > > smallerContours;
	// save possible candidates
	for( unsigned int i = 0; i < groupedCandidates.size(); i++ ) {
	int smallerIdx = groupedCandidates[i][0];
	int biggerIdx = -1;
	// evaluate group elements
	for( unsigned int j = 1; j < groupedCandidates[i].size(); j++ ) {
	size_t currPerim = contoursIn[ groupedCandidates[i][j] ].size();
	// check if current contour is bigger
	if ( biggerIdx < 0 )
	biggerIdx = groupedCandidates[i][j];
	else if(currPerim >= contoursIn[ biggerIdx ].size())
	biggerIdx = groupedCandidates[i][j];
	// check if current contour is smaller
	if(currPerim < contoursIn[ smallerIdx ].size() && detectInvertedMarker)
	smallerIdx = groupedCandidates[i][j];
	}
	// add contours und candidates
	if(biggerIdx > -1){
	biggerCandidates.push_back(candidatesIn[biggerIdx]);
	biggerContours.push_back(contoursIn[biggerIdx]);
	if( detectInvertedMarker ){
	smallerCandidates.push_back(candidatesIn[smallerIdx]);
	smallerContours.push_back(contoursIn[smallerIdx]);
	}
	}
	}

[szk1509]

Hi @stefanopini , I'm glad, I can help :)
I see what you mean, I find the name of the parameter actually not very sharp. I mean, the parameter does not compare the distance between the markers, but attempt to group the nearest candidates of the same marker.
As you can see on the top row of the tested image, the markers have an almost non-existing white border. I don't think that it is the normal use-case, since the detection of this kind of markers could lead to more false positives.
On the other hand, the improvement show, on my tests, an increase of the precision, so i'm not sure if we can consider it as a worsening.

[daanvV]

I am experiencing a similar issue.

I have a video file consisting of roughly 1000 frames, showing an Aruco marker in an outdoor location with bright sunlight. Every now and then a slight shadow passes over the marker.

In opencv-contrib-python 4.1.1.26 the marker was detected in every single frame.

In opencv-contrib-python 4.3.0.36 the marker is detected in only half the frames (see top image). The frames in which the marker is not detected is when the shadow passes over the marker (see bottom image). For the parameters I'm using the default values.

It took me a while to realise that the problem was the module version, quite frustrating.

[stefanopini]

@szk1509 I know that is not the normal use case, but I am wondering if there's a way to combine the two things: ability to detect the markers with almost non-existing white border and better detection of inverted markers.
Besides, I'd like to understand what causes the marker misdetection in this (borderline) use case. I looked at the code changes and didn't find it.

Looking at the example reported by @daanvV (thank you for posting it), it looks like the misdetection does not only happen when the white border is extremely small/non-existent, but also when the white border present a slightly hue variation (in the example, it has a blueish tint).
This may be a common issue in real-world situations and may require further investigation.

[daanvV]

Any updates on this, @szk1509? Recently I was doing a new install of python on my machine and it seems that it's no longer possible to install opencv-contrib-python 4.1.1.26 via pip or poetry, so I have a big problem here since the Aruco detection of the most recent opencv-contrib versions is insufficient for my purposes.
Any help would be much appreciated. :)

[szk1509]

Hi @stefanopini and @daanvV, sorry I haven't answered so far. Due to some personal issues and since I'm working on my thesis I haven't been able to continue working on this topic as much as I would like :(
I also comprehend your feedback, so feel free to contribute. I would like to hear from you again. :)

[stefanopini]

Hi @szk1509 !
Thank you for your update. I hope you're ok and I wish you good luck with your thesis! :)

In this moment, I don't have a lot of time (nor much competence btw) to work on this issue, but I'll try to continue investigating its causes and when it occurs.
I'll keep you posted.

[paroj]

#2900 maybe?

[stefanopini]

Thank you @paroj for the reference to the pull request!

I tested the code compiling opencv 4.5.1 with the opencv_contrib commit a124db8 and the issue is still there unfortunately.
I'm not sure if the approach I followed (compile a stable opencv version with the merge commit of opencv_contrib) is the correct one though.

[jason-cso]

Any news on this issue?

[AleksandrPanov]

I am experiencing a similar issue.

I have a video file consisting of roughly 1000 frames, showing an Aruco marker in an outdoor location with bright sunlight. Every now and then a slight shadow passes over the marker.

In opencv-contrib-python 4.1.1.26 the marker was detected in every single frame.

In opencv-contrib-python 4.3.0.36 the marker is detected in only half the frames (see top image). The frames in which the marker is not detected is when the shadow passes over the marker (see bottom image). For the parameters I'm using the default values.

It took me a while to realise that the problem was the module version, quite frustrating.

PR #3201 fixes marker detection on your image. But you need to set detectorParams->minMarkerDistanceRate as 0.049 or less (Otherwise, the correct inner contour will be removed from candidates).

[AleksandrPanov]

System information (version)

• OpenCV => 4.2.0.34 (opencv-contrib-python)
• Operating System / Platform => Windows 10 64bit / Python 3.6.4
• Compiler => Package installed with pip

Detailed description

The last version of the Aruco detection algorithm ( see pull request #2236 ), which is included in OpenCV 4.1.2 and above, has worse performance than the previous one when the outer border of black markers is small. The algorithm completely fails to find black markers whose outer white border is quite small (just few pixels) with both real and synthetic images. The old algorithm finds them. I tried to change some DetectorParameters (e.g. minMarkerPerimeterRate, minCornerDistanceRate and markerBorderBits) without success.

Steps to reproduce

Please see the attached reference Python code and image. Code is tested with OpenCV 4.1.1.26 (8/8 markers found) and 4.2.0.34 (2/8 markers found) with Python 3.6.4. Corresponding results are reported.

aruco-detection-issue.zip

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues,
  answers.opencv.org, Stack Overflow, etc and have not found solution
• I updated to latest OpenCV version (available through PyPI) and the issue is still there
• There is reproducer code and related data files: videos, images, onnx, etc

PR #3201 fixes marker detection on your image. But you need to set detectorParams->minMarkerDistanceRate as 0.026 or less (Otherwise, the correct inner contour will be removed from candidates).

[stefanopini]

Amazing! Thanks a lot @AleksandrPanov for fixing this!