L09

Lecture 9¶

Announcements¶

Midterm exam coming up a week from today!
- Covers material through today
- You can bring one double-sided 8.5x11" sheet of hand-written notes.
P2 is out! All necessary material will be wrapped up by Tuesday.

Goals¶

Know how to find a least-squares best-fit transformation for:
- homography (with caveats)
Understand the Random Sample Consensus (RANSAC) algorithm
Know how to resample images using forward and inverse warping (and why the latter is usually preferred)
Understand how to perform bilinear interpolation

In [15]:

# boilerplate setup
%load_ext autoreload
%autoreload 2

%matplotlib inline

import os
import sys

src_path = os.path.abspath("../src")
if (src_path not in sys.path):
    sys.path.insert(0, src_path)

# Library imports
import numpy as np
import imageio.v3 as imageio
import matplotlib.pyplot as plt
import skimage as skim
import cv2

# codebase imports
import util
import filtering
import features
import geometry

The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

Plan¶

Homography fitting
Outlier robustness: RANSAC
Warping: forward and inverse
Bilinear interpolation

Recall our definition of the optimal transformation for a given set of correspondences is the one that minimizes the sum of squared residuals: $$ \min_T \sum_i||(T\mathbf{p}_i - \mathbf{p}_i')||^2 $$

Homework Problem 1¶

Write down the $x$ and $y$ residuals for a pair of corresponding points $(x, y)$ in image 1 and $(x', y')$ in image 2 under a homography (projective) motion model. Assume the homography matrix is parameterized as $$ \begin{bmatrix} a & b & c\\ d & e & f\\ g & h & 1 \end{bmatrix} $$

Whiteboard:

homography residuals
roadblocks
duct-tape fixes

Whiteboard: solving homogeneous least squares systems: $$ \min_\mathbf{x} ||A\mathbf{x}|| $$ subject to $$ ||x|| = 1 $$

TL;DM (too long; didn't math):

Decompose $A$ using the SVD: $$ U_{m \times m}, \Sigma_{m\times n}, V^T_{n \times n} = \mathrm{SVD}(A_{m \times n}) $$
The optimal vector $x^* = $ the row of $V^T$ (column of $V$) corresponding to the smallest element of $\Sigma$ (which is diagonal)
Usually your linear algebra library will order things so that $\Sigma$'s elements are in descending order, so in practice $x^* = $ the last row of $V^T$ is the optimal $x*$

Next up: Robustness to outliers¶

RANSAC: RAndom SAmple Consensus¶

Finding a transformation is a model fitting problem. A simple model fitting problem that we'll use as analogy is line fitting (in fact, this is what linear least squares is doing for us, it's just fitting higher-dimensional lines).

Problem statement, for now: Given a set of points with some outliers, find the line that fits the non-outliers best.

Key Idea:

“All good matches are alike; every bad match is bad in its own way.”

-Tolstoy, as misquoted by Alyosha Efros

Observation: If I have a candidate model, I can tell how good it is by measuring how many points "agree" on that model.

Algorithm, take 1:

for every possible line:
   count how many points are inliers to that line
return the line with the most inliers

Runtime: O($\infty$)

Algorithm, take 2:

for every line that goes through two of the given points:
   count how many points are inliers to that line
return the line with the most inliers

Runtime: O(n^3)

Algorithm, take 3: RANSAC - see whiteboard notes

Homework Problems 2-4¶

In the inner loop of RANSAC, how many points are used to fit a candidate model if you are fitting a line to a set of 2D points?
In the inner loop of RANSAC, how many pairs of corresponding points are used to fit a candidate model if you are fitting a translation to a set of correspondences?
In the inner loop of RANSAC, how many pairs of corresponding points are used to fit a candidate model if you are fitting a homography to a set of correspondences?

Homework Problem 5 (preview)¶

In this problem, we'll analyze the RANSAC algorithm to help us understand how to decide how many iterations to run ($K$). Suppose we are fitting some model that requires a minimal set of $s$ points to fully determine (e.g., $s=4$ matches for a homography, $s=2$ points for a line). We also know (or have assumed) that the data has an inlier ratio of $r = \frac{\text{\# inliers}}{\text{\# data points}}$; in other words, the probability that a randomly sampled point from the dataset as a probability of $r$ of being an inlier.

Warping: Forward and Inverse¶

See whiteboard notes.

Forward warping:

for x, y in src:
  x', y' = T(x, y)
  dst[x', y'] = src[x, y]

Inverse warping:

Tinv = inv(T)
for (x', y') in dst:
  x, y = Tinv(x', y')
  dst[x', y'] = src[x, y]

Homework Problem (for next week - we didn't get to this today)¶

Complete the following function with Python-esque pseudocode (or working code in the lecture codebase), which performs inverse warping with nearest-neighbor sampling in the source image.

def warp(img, tx, dsize=None)
    """ Warp img using tx, a matrix representing a geometric transformation.
    Pre: tx is 3x3 (or some upper-left slice of a 3x3 matrix). img is grayscale.
    Returns: an output image of shape dsize with the warped img"""
    H, W = img.shape[:2]

    # turn a 2x2 or 2x3 tx into a full 3x3 matrix
    txH, txW = tx.shape
    M = np.eye(3)
    M[:txH,:txW] = tx

    Minv = np.linalg.inv(M)
    
    if dsize is None:
        DH, DW = (H, W)
    else:
        DH, DW = dsize[::-1]
    out = np.zeros((DH, DW))

    # your code here
    
    return out

In [ ]:

y1 = imageio.imread("../data/yos1.jpg").astype(np.float32) / 255
y1 = skim.color.rgb2gray(y1)