Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rescale colmap keypoints. #817

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

tiantianxiangshang629
Copy link
Collaborator

@tiantianxiangshang629 tiantianxiangshang629 commented Nov 11, 2024

The purpose of this PR is to enable running GTSFM with feature extraction results from COLMAP's front end. COLMAP front-end results are typically stored in a database format. The colmap_correspondence_generator.py library aim to load COLMAP's feature extraction results directly from this database.

However, there is a discrepancy in how COLMAP handles feature extraction and storage:

  • COLMAP extracts features on a resized image (default resolution: 1200x900) and then scales the feature points back to the original image size before storing them in the database.
  • This creates a mismatch if GTSFM assumes that the feature points correspond to the resized image resolution instead of the original resolution.

Key Issues Addressed by this PR:

  • The original code assumes that:
    • COLMAP extracts features at a resolution of 1013x760.
    • Feature points for this resolution are stored in the database.
    • This assumption is incorrect, as COLMAP operates on a different resized resolution by default (1200x900).
  • This PR resizes the feature points stored in COLMAP's database (scaled to the original image size) to match the GTSFM resized resolution (1013x760).

Important Notes:

  • GTSFM extracts features using a resized image resolution of 1013x760, while COLMAP’s default feature extraction is performed on images resized to 1200x900.
  • To use GTSFM's backend directly with COLMAP's front end, the feature extraction process in COLMAP must also be performed using the 1013x760 resolution. This ensures compatibility between the feature extraction results from COLMAP and GTSFM.

@travisdriver
Copy link
Collaborator

Do the results when loading in the COLMAP database now align with what you get when running GTSfM with the COLMAP frontend (i.e., running with the colmap_front_end.yaml config)?

@tiantianxiangshang629
Copy link
Collaborator Author

Do the results when loading in the COLMAP database now align with what you get when running GTSfM with the COLMAP frontend (i.e., running with the colmap_front_end.yaml config)?

@travisdriver Hi Travis, apologies for the delayed response; I haven’t checked this account in a while. Are you asking whether the results from running GTSfM with the COLMAP frontend align with the results from running the full end-to-end COLMAP pipeline?

@tiantianxiangshang629 tiantianxiangshang629 marked this pull request as ready for review January 13, 2025 15:47
)
return Keypoints(coordinates=coordinates, scales=None, responses=None)

def _read_image_ids_and_keypoints(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you using the same formatting settings? please refer to https://github.com/borglab/gtsfm/blob/master/CONTRIBUTING.md#python-style

Copy link
Collaborator

@akshay-krishnan akshay-krishnan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are still some format changes that are undesirable. I have highlighted 3 of them. You can be sure the right formatting settings are being used if you reformat and those changes get reverted.

two_view_geometry = self._pycolmap_db.read_two_view_geometry(colmap_i1, colmap_i2)
two_view_geometry = self._pycolmap_db.read_two_view_geometry(
colmap_i1, colmap_i2
)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

undesirable format change.


# Only read matches if we have an essential or a fundamental matrix
if two_view_geometry.config != 2 and two_view_geometry.config != 3:
continue

# Note(Ayush): the matches we are loading are actually post verification
corr_idxs[(i1, i2)] = np.array(two_view_geometry.inlier_matches, dtype=np.int32)
corr_idxs[(i1, i2)] = np.array(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

undesirable format change.

@@ -107,7 +160,9 @@ def generate_correspondences(
# Note: we will end up reading verified correspondences from the colmap DB.
images_actual = client.gather(images)

gtsfm_id_to_pycolmap_id, keypoints = self._read_image_ids_and_keypoints(images_actual)
gtsfm_id_to_pycolmap_id, keypoints = self._read_image_ids_and_keypoints(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

undesirable format change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants