L03_576

Announcements¶

I made a Discord server for discussion and Q&A. Link on Canvas, please use your real name as your server nickname. I'll try to respond to questions in q-and-a when possible.
Feedback for P01 and P02 came back yesterday. I added a few notes on logistics to the syllabus, just so they're written down somewhere:
- The problems for each lecture will be posted in the Schedule table by the start of each class.
- Please typset your answers and submit to Canvas in PDF format by Sunday night of the week they are assigned.
- Since these are "ungraded", the individual Canvas assignments will be scored out of 0; I will assign a 0/0 and provide feedback to any submissions.
- If you wish to submit or or resubmit anytime after 7am Monday morning after the deadline, please submit via Canvas then send me an email to let me know that you've done so. I will give timely feedback on late and re-submissions on a best-effort basis.
Project 1 is out today. We'll talk about it a bit in class on Thursday.
476-lectures repo - rather than branching nonsense, I'll just commit L??_476.ipynb and L??_576.ipynb

Goals¶

Know how to compute image derivatives using convolution filters
Understand how to derive and apply the the Sobel edge detection filter
Have an intuitive understanding of spatial frequency in images
Know the meaning and construction of "low pass" and "high pass" filters
Know how to make images smaller:
- The naive way via subsampling (and why this is bad)
- The principled way by downsampling with prefiltering (and why this is better)
Know how to make images bigger

In [2]:

# boilerplate setup
%load_ext autoreload
%autoreload 2

%matplotlib inline

import os
import sys

src_path = os.path.abspath("../src")
if (src_path not in sys.path):
    sys.path.insert(0, src_path)

# Library imports
import numpy as np
import imageio.v3 as imageio
import matplotlib.pyplot as plt
import skimage as skim
import cv2

# codebase imports
import util
import filtering

The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

Differentiating Images¶

Images are functions; differentiation is an operator. What does it mean?

Since they're functions of two variables, a single-valued output needs to be a partial derivative:

Horizontal ($x$) derivative: $\frac{\partial}{\partial x} f(x, y)$
Vertical ($y$) derivative: $\frac{\partial}{\partial y} f(x, y)$

We have discrete (i.e., sampled) images, so we need to approximate this with finite differences. Let's design a convolution kernel that accomplishes this.

Whiteboard: calculus reminder

Homework Problem 1¶

Consider the following two candidate horizontal derivative filters. $$ \begin{bmatrix} 1 & -1 & 0\\ \end{bmatrix} $$

$$ \begin{bmatrix} 1 & 0 & -1\\ \end{bmatrix} $$

Why is the negative number to the right of the positive one?
If we wanted to accurately calculate the slope with correct scale, how would we need to modify the above kernels?
What are the relative merits of each of these filters?

In [3]:

bikes = imageio.imread("../data/bikesgray.jpg").astype(np.float32) / 255.0
bikes = skim.transform.rescale(bikes, 0.5, anti_aliasing=True)
bikes = bikes + np.random.randn(*bikes.shape) * 0.05

util.imshow_gray(bikes)

Look at filtering.py

I tweaked our filter function to handle non-square kernels.
I added a convolve that just flips the kernel then runs filter

Let's look at the intensities along a single scanline. This one is a vertical scanline that crosses the brick pattern.

In [6]:

plt.plot(bikes[:,100])

Out[6]:

[<matplotlib.lines.Line2D at 0x7f5b9e963ed0>]

In [7]:

plt.plot(by[:,100])

Out[7]:

[<matplotlib.lines.Line2D at 0x7f5b9e9d2110>]

This motivates an idea: blur the noise so that the real edges stick out!

Why use 2 filters when you could use just 1?

Homework Problem 2¶

Compute the following convolution, which results in a new filter kernel, and describe the effect of this new kernel in words. $$ \begin{bmatrix} 1 & 2 & 1\\ 2 & 4 & 2\\ 1 & 2 & 1 \end{bmatrix} * \begin{bmatrix} 0 & 0 & 0\\ 1 & 0 & -1\\ 0 & 0 & 0 \end{bmatrix} = \begin{bmatrix} \ & \ & \ \\ \ & \ & \ \\ \hspace{1em} & \hspace{1em} & \hspace{1em} \end{bmatrix} $$

Let's check our work:

In [8]:

blur = np.array([
    [1, 2, 1],
    [2, 4, 2],
    [1, 2, 1]], dtype=np.float32)

dx = np.array([
    [0, 0,  0],
    [1, 0, -1],
    [0, 0,  0]], dtype=np.float32)

# check our answer
xsobel = filtering.convolve(blur, dx)
ysobel = xsobel.T
xsobel

Out[8]:

array([[ 2.,  0., -2.],
       [ 4.,  0., -4.],
       [ 2.,  0., -2.]], dtype=float32)

For whatever reason, this is more often written scaled down by 1/2:

In [9]:

xsobel /= 2
ysobel = xsobel.T
xsobel, ysobel

Out[9]:

(array([[ 1.,  0., -1.],
        [ 2.,  0., -2.],
        [ 1.,  0., -1.]], dtype=float32),
 array([[ 1.,  2.,  1.],
        [ 0.,  0.,  0.],
        [-1., -2., -1.]], dtype=float32))

In [11]:

plt.plot(by[:,100])

Out[11]:

[<matplotlib.lines.Line2D at 0x7f5b86313ed0>]

From sobel to edge detection¶

Direction-independent edge detector? First pass: gradient magnitude

$$ \Delta f = \begin{bmatrix} \frac{\partial}{\partial x} f \\ \frac{\partial}{\partial y} f \end{bmatrix} $$

Edge strength: gradient magnitude $||\Delta f||$

In [12]:

plt.imshow(np.sqrt(bx ** 2 + by**2), cmap="gray")

Out[12]:

<matplotlib.image.AxesImage at 0x7f5b8619b6d0>

This is useful enough that I wrote filtering.grad_mag to make it simple to do.

Classical fancier method: Canny Edge Detector

Convert to grayscale
Blur with 5x5 $\sigma=1.4$ Gaussian
Calculate gradient magnitude
Apply non-maximum suppression
Threshold with 2 cutoffs for strong edges, weak edges, and non-edges
Preserve only weak edges that connect to strong edges

Spatial Frequency¶

Whiteboard - intuition (only) for the Fourier decomposition

Introducing: the Frequencyometer(TM)¶

A very imprecise way of talking about what spatial frequencies are present in an image.

In [13]:

beans = imageio.imread("../data/beans.jpg").astype(np.float32) / 255.0
bg = skim.color.rgb2gray(beans) # grayscale beans

plt.imshow(beans)

Out[13]:

<matplotlib.image.AxesImage at 0x7f5b86311990>

In [14]:

plt.imshow(beans[400:500, 100:200])

Out[14]:

<matplotlib.image.AxesImage at 0x7f5b862f18d0>

In [15]:

plt.imshow(beans[20:120, 500:600])

Out[15]:

<matplotlib.image.AxesImage at 0x7f5b9e901850>

In [16]:

plt.imshow(beans[420:480, 550:600])

Out[16]:

<matplotlib.image.AxesImage at 0x7f5b85876250>

Definitions: "Low-Pass" and "High-Pass" filters¶

Low-pass: allows low frequencies to pass through unaffected, i.e., attenuates high frequencies.

In other words: blur!

High-pass: allows high frequencies to pass through unaffected, i.e., attenuates low frequencies.

In other words: derivative, (with slight but common terminology abuse) sobel, or sharpening

Question that didn't make it onto the homework: in what sense is Sobel not truly a high-pass filter?

Homework Problems 3-6¶

(3) Using the language of "low-" and "high-frequency" image content, explain why sharpening is not the inverse of blurring, and what it accomplishes instead.

(4) Consider the original image of beans on the left, and the processed version on the right. Describe what has changed in terms of frequency content.

(5) What's the maximum frequency (expressed in full periods per pixel) representable in a 1D image (i.e., a row of pixels)? What does such an image look like?

(6) What's the minimum frequency representable in a 1D image? What does such an image look like?

Break, if we haven't already taken one!¶

Downsampling¶

My image is too big to fit on my screen. For example, suppose beans is 600x600, but I want to display the image in 300x300 pixels. What should I do?

In [21]:

bricks = imageio.imread("../data/bricks.jpg").astype(np.float32) / 255.0
plt.imshow(bricks[::4,::4,:])

Out[21]:

<matplotlib.image.AxesImage at 0x7f5b85274910>

In [24]:

plt.imshow(checker[:, 1::2], vmin=0, vmax=1, cmap="gray") # force color scale to [0,1] range

Out[24]:

<matplotlib.image.AxesImage at 0x7f5b854218d0>

Homework Problem 7¶

If you walked far away from the above image until you couldn't distinguish individual pixels, what would it look like?

Whiteboard: downsampling freqeuncyometer

Upsampling¶

My image is too small for my screen. For example, suppose beans is 150x150, but I want to display the image in 600x600 pixels. What should I do?

See naive version preimplemented in filtering.up_2x

Whiteboard: Filtering view of upsampling

In [41]:

# todo: implement reconstruction filtering version in up_2x
# - Gaussian reconstruction filter
# - Linear reconstruction filter
util.imshow_truesize(filtering.up_4x(beans150, interp="linear"))

Lecture 3¶