Fabio Bellavia* · Zhenjun Zhao* · Luca Morelli · Fabio Remondino
👉 📖 paper on arXiv 📖 👈
Planar clusters assigned by MOP+MiHo, outlier matches are marked
with black diamonds, NCC keypoint shifts are not shown.
MOP+MiHo+NCC is a modular, non-deep method designed to filter and refine image matches. This approach enhances the quality of image matching by utilizing multiple techniques:
- Multiple Overlapping Planes (MOP) removes outlier matches while jointly clustering inlier matches into planes by an iterative RANSAC-based strategy;
- Middle Homography (MiHo) improves planar homography from MOP by minimizing relative patch distortion in the plane reprojection;
- Normalized Cross Correlation (NCC) refines keypoint positions on patches after planar transformation.
MOP+MiHo consistently improves match quality when used a pre-processing step for RANSAC, without negatively affecting the final results in any case. MOP+MiHo+NCC introduces additional improvements in the case of corner-like matches, which are common in methods such as Key.Net or SuperPoint. However, for blob-like matches, which are more typical of methods like SIFT, MOP+MiHo+NCC unfortunately tends to degrade performance.
Comparative evaluation is available here.
Unlike other benchmarks, it assumes no camera intrinsics are available, which reflects a more general and realistic scenario. In the tables, cv2.findEssentialMat
. AUCs considering metric instead of angular translation errors are reported as
As better RANSAC, MAGSAC is employed with the thresholds of 1 and 0.75 px, indicated respectively by
For installing the code, create a Python virtual environment and after its activation run pip install -r requirements.txt
. If you are going to run only MOP+MiHo+NCC and not the full benchmark, edit requirements.txt
as described within its comments. Current requirement file has been tested on Ubuntu 24.04, the previous one which was tested on Ubuntu 20.04 can be found in the data/other
folder. Notice that in the last revision the pydegensac
and pymulticonsensus
python modules cannot be installed (until now), so they have been removed. Corresponding filter modules will return no matches.
For the basic usage, check the comments and run miho_ncc_demo.py
. Notice that in the code MOP is denoted as miho_unduplex
while MOP+MiHo as miho_duplex
. In case you get Qt window displaying errors, reinstall again PyQt (pip install pyqt5
or pip install pyqt6
depending on your system, this is a OpenCV issue) or try to uncomment matplotlib.use('tkagg')
in src/miho.py
src/miho_other.py
after having installed Tkinter (the python3-tk
package in Ubuntu distributions). Decreasing the max_iters
params for ransac_middle_args
in get_avg_hom
, e.g. to 1500, 1000, or 500, can improve the speed at the expense of the accuracy; this is not recommended for SIFT or Key.Net when one wants to push up the recall because their matches are more contaminated by outliers than other matching pipelines. Note that in the experiments the NNR thresholds for these pipelines are quite high (0.95 and 0.99, respectively) for an effective evaluation of the match filtering. An analysis on the performance variations is reported here.
To run the full benchmark, use run_bench.py
and generate result tables with save_bench.py
. If you want to limit the methods or datasets being compared, you can comment out specific lines in the pipes
, pipe_heads
, pipe_ransacs
and benchmark_data
sections of the code. For RANSAC ablation, use the corresponding scripts run_ransac_ablation.py
and save_ransac_ablation.py
. For running times and max_iters
ablation, use the corresponding scripts run_bench_time.py
and save_bench_time.py
. Visual matches can be inspected using run_showcase.py
and NCC patches can be compared through patch_demo.py
. Additional scripts like intrinsiscs_bench.py
and intrinsics_other_bench.py
provide more detailed reports on camera intrinsics, while corr_bench.py
generates the correlation between different measurement errors. All detailed results are available in the data/results
directory.
See the paper Image Matching Filtering and Refinement by Planes and Beyond for details about the method and for the benchmark setup and results. The initial idea of this approach can be found instead in the paper Progressive keypoint localization and refinement in image matching presented at FAPER2023.