project 4: IMAGE WARPING, MOSAICING, and FEATURE MATCHING for AUTOSTITCHING

Introduction

In this project I try to perform image rectification and mosaicing using homographies.

Here's the images Im working with

Image 1

Home 1

Image 2

Home 2

Image 3

Home 3

Image 4

Home 4

Image 1

Bart 1

Image 2

Bart 2

Image 3

Bart 3

Image 4

Bart 4

Image 1

wework 1

Image 2

wework 2

Image 3

wework 3

Image 4

wework 4

Recover homographies

Firstly, we need to define correspondnces between pictures(I used the given tool: https://cal-cs180.github.io/fa23/hw/proj3/tool.html). Then, using these keypoints, we want to recover homographies: Here's how I did it. We have: \( pts1 = \{(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)\} \), \( pts2 = \{(x'_1, y'_1), (x'_2, y'_2), \dots, (x'_n, y'_n)\} \). To compute a homography matrix \( H \), For each pair of points \( (x_i, y_i) \) from \( pts1 \) and \( (x'_i, y'_i) \) from \( pts2 \), we create: \[ M = \begin{bmatrix} x_1 & y_1 & 1 & 0 & 0 & 0 & -x_1 x'_1 & -y_1 x'_1 \\ 0 & 0 & 0 & x_1 & y_1 & 1 & -x_1 y'_1 & -y_1 y'_1 \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\ x_n & y_n & 1 & 0 & 0 & 0 & -x_n x'_n & -y_n x'_n \\ 0 & 0 & 0 & x_n & y_n & 1 & -x_n y'_n & -y_n y'_n \end{bmatrix} \] and then flatten points from points 2 into a vector\[ \mathbf{res\_coords} = \begin{bmatrix} x'_1 \\ y'_1 \\ \vdots \\ x'_n \\ y'_n \end{bmatrix}\] Then, solve \( M \mathbf{h} = \mathbf{res\_coords} \), where \( \mathbf{h} \) is: \[ \mathbf{h} = (h_1, h_2, h_3, h_4, h_5, h_6, h_7, h_8) \]

Warping images

To warp images, I create interpolation functions for each channel using RectBivariateSpline. Then I apply inverse homography matrix to the grid of result coords, for each channel interpolate the pixel values, and return the result. In my function, I add the desired shape for the output as an input.

Image rectification

For this part, I create a separate list of destination points that basically has all corners, and then define keypoints as corners of either portrait or poster.

original image

warped

original image

warped

Mosaic

Initially I tried mosaicing two images.Here's what I did:

1. Compute homographies

2. Determine the bounding box of the panorama

3. Shift images so that all coordinates are positive

4. Warp images

5. Create masks

6. Blend warped images

For two images I got this

Then, I tried doing the same thing with 4 pics, the only thing I changed was multiplying homographies

Harris Interest Point Detector

Running the skeleton code on my pictures gives me this result:

initial

skeleton code applied

Adaptive Non-Maximal Suppression

in this part, I implemented ANMS to reduce the number of dots of potential keypoints. The result is below

initial

skeleton code applied

Feature descriptors

Here's I extract feature descriptors(40x40 windows of 8x8 patches) and implement feature matching to find the best correspondences. more formally: match_descriptors matches feature descriptors between two images by finding two euclidean distance of each pair and apply ratio test for reliable matches. extract_descriptors: extract a normalized descriptor for each point by downsampling images, also make it robust to brightness and contrast changes.

Results are below

here's the resulst with two pics:

manual

automatic

here's the resulst with four pics:

this is manual and automatic next to each other:

manual

automatic

manual

automatic

manual

automatic

Reflection

I think the maing problem here was bad pictures (shaky hands) and the fact that I tried to align to the first picture rather than the either second or third one. It was fun to learn how to manipulate pictures and see something that was hard to see initially like in one of the lectures with floor mosaic. Can't wait for the second part, where we wouldn't have to manually mark correspondences. For the part B it took me forever to center towards the third picturesm because I don't use the same homographies structure as I did before.