Camera calibration

[NikTechT]

This is an amazing project! Thanks for sharing. I have a question regarding the camera calibration. How did you calculate the intrinsic matrix, distortion and rotation. Did you follow this: https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html or did you write your own code?

[jyjblrd]

Hi, thanks for the kind words!

I did follow that tutorial, and then put the intrinsic matrix and distortion parameters in the file camera-params.json. This is also where you define the number of cameras in your setup.

[gumby0q]

Hi,
that's really magnificent project :)

I want to add a few my notes about calibration, because that point is not obvious especially for new ppl in that field

I made/adapted two raw scripts for that process based on next sources:

• https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html
• https://learnopencv.com/camera-calibration-using-opencv/

A note: I have used custom M12 lenses with focal length of 2.8 mm so I definitely needed these numbers.

1. Since there is crappy resolution in pseye camera using 9x6 chess board on a4 paper is quite bad idea. I achieved more or less good results with cropped board to 6x5 and printed it on good a4 not glossy photo paper. I will attach it here
   
2. To collect images i just run the script and 'danced' with chessboard in front of the camera.

from pseyepy import Camera, Display
import cv2  # OpenCV for image handling
import time

cam_index = 1

# Initialize camera
# cams = Camera(fps=60, resolution=Camera.RES_LARGE)  # wrong camera mode
cams = Camera(fps=90, resolution=Camera.RES_SMALL)  # camera mode which is used in main mocap logic

for i in range(30):  # Capture 10 images
    frame, timestamp = cams.read()
    filename = f'cam_{cam_index}/image_{i}.jpg'
    cv2.imwrite(filename, frame)
    time.sleep(0.5)  # Wait a second between captures

# Clean up
cams.end()

3. I built opencv without UI module, so I just saved images to folder, and then run the next script to recognize chessboard points.

#!/usr/bin/env python
 
import cv2
import numpy as np
import os
import glob
 
cam_images_folder_name = 'cam_1'
cam_images_folder_name_calibrated = f'{cam_images_folder_name}_c'
# cam_images_folder_name = 'cam_1'

# Defining the dimensions of checkerboard
# CHECKERBOARD = (6,9)
CHECKERBOARD = (5,6)
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
 
# Creating vector to store vectors of 3D points for each checkerboard image
objpoints = []
# Creating vector to store vectors of 2D points for each checkerboard image
imgpoints = [] 
 
 
# Defining the world coordinates for 3D points
objp = np.zeros((1, CHECKERBOARD[0] * CHECKERBOARD[1], 3), np.float32)
objp[0,:,:2] = np.mgrid[0:CHECKERBOARD[0], 0:CHECKERBOARD[1]].T.reshape(-1, 2)
prev_img_shape = None
 
# Extracting path of individual image stored in a given directory
images = glob.glob(f'./{cam_images_folder_name}/*.jpg')
for fname in images:
    img = cv2.imread(fname)
    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
    # Find the chess board corners
    # If desired number of corners are found in the image then ret = true
    # ret, corners = cv2.findChessboardCorners(gray, CHECKERBOARD, cv2.CALIB_CB_ADAPTIVE_THRESH + cv2.CALIB_CB_FAST_CHECK + cv2.CALIB_CB_NORMALIZE_IMAGE)
    ret, corners = cv2.findChessboardCorners(gray, CHECKERBOARD, cv2.CALIB_CB_ADAPTIVE_THRESH + cv2.CALIB_CB_NORMALIZE_IMAGE)
     
    """
    If desired number of corner are detected,
    we refine the pixel coordinates and display 
    them on the images of checker board
    """
    if ret == True:
        objpoints.append(objp)
        # refining pixel coordinates for given 2d points.
        corners2 = cv2.cornerSubPix(gray, corners, (11,11),(-1,-1), criteria)
         
        imgpoints.append(corners2)
 
        # Draw and display the corners
        img = cv2.drawChessboardCorners(img, CHECKERBOARD, corners2, ret)
     
    # cv2.imshow('img',img)
    # cv2.waitKey(0)
    
    new_frame_name = cam_images_folder_name_calibrated + '/' + os.path.basename(fname)
    # print(new_frame_name)
    cv2.imwrite(new_frame_name, img)

 
# cv2.destroyAllWindows()
 
h,w = img.shape[:2]
 
"""
Performing camera calibration by 
passing the value of known 3D points (objpoints)
and corresponding pixel coordinates of the 
detected corners (imgpoints)
"""
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1], None, None)
 
print("Camera matrix : \n")
print(mtx)
print("dist : \n")
print(dist)

4. After run of the script from point 3 have checked if the chessboard recognized correctly and if some images was bad i just removed these samples and run it again. Also if few images are not recognized at all that's ok

I wasn't able to get reproducible numbers of camera parameters, they are always fluctuating even if collections of samples are ok.
For example, I collected 3 bunches of samples and got three different numbers (especially for distortion).

# Camera matrix : 
# [[293.37352356   0.         148.83826395]
#  [  0.         293.62691058 111.79537397]
#  [  0.           0.           1.        ]]
# dist : 
# [[-0.45485019  0.2550664   0.00066252  0.00063356 -0.09096037]]

# Camera matrix : 
# [[293.27414483   0.         147.12726117]
#  [  0.         293.30479926 110.98823837]
#  [  0.           0.           1.        ]]
# dist : 
# [[-4.47296210e-01  2.18968662e-01 -1.38218013e-04  1.25328277e-03
#   -4.54514945e-02]]

# Camera matrix : 
# [[293.72298031   0.         149.37364445]
#  [  0.         293.91614053 110.97211794]
#  [  0.           0.           1.        ]]
# dist : 
# [[-4.51543547e-01  2.30999426e-01 -3.70807888e-04  1.38167804e-04
#   -2.23855337e-02]]

Anyway I took the most "middle" result on just my eye and camera pose calibration algorithm worked after that!

Also I assume that it is better to have IR filter while doing calibration, because the image are stuffed with IR light and it's to grey which makes a bad contrast of black color, which lead to issues with chessboard. Any thoughts about that?
For instance one of my samples looks like that:

[jyjblrd]

Thanks! This is a great overview.

I don't think it should make a difference if the checkerboard is grey in the image, as long as opencv can find the checkerboard. It probably would be best to not use the IR filter, as some printer inks may not be visible in IR lighting.

[cultist123]

Do you mean performing camera calibration first and then physically modifying the camera to add an infrared filter? This is such a small detail that it requires me to disassemble the camera again and remove the filter just to complete the calibration task.

Did you use an infrared floodlight along with the infrared filter? My camera can't capture anything other than the light itself, as shown in the image below. How did you resolve this issue?

[gumby0q]

On the images with chess boards from the post these are cameras without factory IR filter. Just raw lenses.
After calibration add the light filter made from floppy disk alternative.
Also a tip: it is more convenient to mount light flood filter on the front of the lenses so you don't need to disassemble it every time

[d4saher]

Hi @gumby0q,

Thanks for sharing your notes and code for calibration. I'm currently trying to calibrate my cameras, but I'm having trouble downloading the chessboard you shared. I used a different one, but I'm unsure if the square size in mm matters for the calibration process.

Here it says:
"This consideration helps us to find only X,Y values. Now for X,Y values, we can simply pass the points as (0,0), (1,0), (2,0), ... which denotes the location of points. In this case, the results we get will be in the scale of size of chess board square. But if we know the square size, (say 30 mm), we can pass the values as (0,0), (30,0), (60,0), ... . Thus, we get the results in mm. (In this case, we don't know square size since we didn't take those images, so we pass in terms of square size)."

This explanation is a bit confusing for me, and right now I can't get a correct camera pose with the chessboard I am using (5x6 inner corners and 34mm each square). Could you please clarify if the square size is important and, if so, what size I should be using?

Additionally, I used your code to write another script that automates the task a bit more. You can find the code here.

Thank you in advance!

[gumby0q]

Hi,

• to download image just click on it with right mouse button and select "save image as"
• as far as I saw we do not need to handle the squares size or know it in mm. The algorithm should just calculate distortion from detected points on the chessboard image. Anyway you need to cover as much as you can screen area with chessboard image to have better samples
• about the code and samples. I ran images collecting and calibration screens separately. After taking samples and calibration run I checked the parsed images (images with color lines and points). And if found bad sample with wrong chessboard detection I just removed it from the source samples and ran calibration again. Also i have made whole procedure several times and only then took params with similar numbers
• I found a mistake in images collection. Sorry that my bad, provided example with mistake :( . Camera has several modes, and it should be calibrated with params which are used in mocap code.
  https://github.com/d4saher/Low-Cost-Mocap/blob/4dc950b4b7a9894f7def26be912155aeaa8fccf1/camera_calibration/calibrate-cameras.py#L20
  that line is incorrect :(
  put next line instead. everything should work then

cams = Camera(fps=90, resolution=Camera.RES_SMALL)

[d4saher]

Hi,

• Now I could download it; previous attempts said that the image was private.
• Good to know, thanks.
• I ran the calibration again using separate scripts and got these matrices:

[
    {
        "intrinsic_matrix": [[269.61269233,   0, 158.13850671],
                            [  0,         269.52402037, 124.48516327],
                            [  0,           0,           1        ]],    
        "distortion_coef": [-0.08418665,  -0.02910327,  0.00144069,  0.0025247,  0.52680835],
        "rotation": 0
    },
    {
        "intrinsic_matrix": [[272.23749492,   0, 153.96547061],
                            [  0,         271.85716949, 125.5030155],
                            [  0,           0,           1        ]],    
        "distortion_coef": [-0.12387721,  0.26769884,  0.00316347,  0.00239143,  -0.30745277],
        "rotation": 0
    },
    {
        "intrinsic_matrix": [[271.68875067,   0, 165.80213766],
                            [  0,         271.61911612, 111.30482265],
                            [  0,           0,           1        ]],    
        "distortion_coef": [-0.12749098,  0.44302503,  -0.00295966,  -0.00186736,  -0.81783704],
        "rotation": 0
    },
    {
        "intrinsic_matrix": [[270.10424478,   0, 149.82845498],
                            [  0,         269.75951418, 114.44848971],
                            [  0,           0,           1        ]],    
        "distortion_coef": [-8.40609099e-02,  -1.05115945e-01,  -4.00725641e-03,  -7.63121941e-04,  8.28819771e-01],
        "rotation": 0
    }
]

Which seems to be more coherent. However, when I capture the points and run pose calibrations, most of the time I get this error:

File "/usr/lib/python3/dist-packages/flask_socketio/__init__.py", line 751, in _handle_event
    ret = handler(*args)
  File "/home/d4saher/tb-tracker/Low-Cost-Mocap/computer_code/api/index.py", line 249, in calculate_camera_pose
    E = cv.sfm.essentialFromFundamental(F, cameras.get_camera_params(0)["intrinsic_matrix"], cameras.get_camera_params(1)["intrinsic_matrix"])
cv2.error: OpenCV(4.10.0-dev) /home/d4saher/tb-tracker/opencv_contrib/modules/sfm/src/fundamental.cpp:440: error: (-215:Assertion failed) F.cols == 3 && F.rows == 3 && F.size() == _K1.size() && F.size() == _K2.size() && (depth == CV_32F || depth == CV_64F) in function 'essentialFromFundamental'

It looks like F is None and it fails. I want to do more tests on the camera calibration, but I don't know why it is failing. Maybe the points that I get for pose are not good enough? I moved around the room getting more or less 30 points. Sometimes it completes the calculation, but the cameras are in random locations.

• My fault, I didn't check twice. I had hoped that this would fix the previous issue, but it still fails. Has anyone else encountered this error?

Thank you!

[d4saher]

Hi,

I wanted to update that I finally managed to get it working by capturing more points, around 70. Let's see if now I can set the scale, floor, and origin correctly.

Thanks for your help and attention!

[high993]

Hey just had a question regarding the resolution, I'm repurposing part of this project and wanted to know if the calibration process is possible with the cams = Camera(fps=90, resolution=Camera.RES_LARGE). The larger resolution (640 by 480) makes more sense for my project, and I'm guessing that doing the calibration like this would yield more accurate results. Did you ever get it working this way or did you settle for the Camera.RES_SMALL?

[gumby0q]

@high993 use the same camera parameters during calibration as you use for your project