Project 1: Hybrid Images

Key Information

Implementation Details

This project is intended to familiarize you with Python, NumPy and image filtering. Once you have created an image filtering function, it is relatively straightforward to construct hybrid images.

This project requires you to implement 5 functions, each of which builds on the previous ones:

1. cross_correlation_2d
2. convolve_2d
3. gaussian_blur_kernel_2d
4. low_pass
5. high_pass

Image Filtering. Image filtering (or convolution) is a fundamental image processing tool. See chapter 3.2 of Szeliski and the lecture materials to learn about image filtering (specifically linear filtering). Numpy has numerous built in and efficient functions to perform image filtering, but you will be writing your own such function from scratch for this assignment. More specifically, you will implement cross_correlation_2d, followed by convolve_2d which would use cross_correlation_2d.

Gaussian Blur. As you have seen in the lectures, there are a few different way to blur an image, for example taking an unweighted average of the neighboring pixels. Gaussian blur is a special kind of weighted averaging of neighboring pixels, and is described in the lecture slides. To implement Gaussian blur, you will implement a function gaussian_blur_kernel_2d that produces a kernel of a given height and width which can then be passed to convolve_2d from above, along with an image, to produce a blurred version of the image.

High and Low Pass Filters.Recall that a low pass filter is one that removed the fine details from an image (or, really, any signal), whereas a high pass filter only retails the fine details, and gets rid of the coarse details from an image. Thus, using Gaussian blurring as described above, implement high_pass and low_pass functions.

Hybrid Images. A hybrid image is the sum of a low-pass filtered version of the one image and a high-pass filtered version of a second image. There is a free parameter, which can be tuned for each image pair, which controls how much high frequency to remove from the first image and how much low frequency to leave in the second image. This is called the "cutoff-frequency". In the paper it is suggested to use two cutoff frequencies (one tuned for each image) and you are free to try that, as well. In the starter code, the cutoff frequency is controlled by changing the standard deviation (sigma) of the Gausian filter used in constructing the hybrid images. We provide you with the code for creating a hybrid image, using the functions described above.

Notes on efficiency cross_correlation_2d is computationally intensive: filtering an image of size M x N with a kernel of size K x K is an O(MNK²) operation. For arbitrary kernels, this is unavoidable without using Fourier domain tricks. However, numpy's array processing routines are highly optimized and allow for huge speedups of array operations relative to Python for loops, which must be executed line by line by the Python interpreter. As usual, focus on getting a correct solution first; then, see if you can eliminate some of the nested loops using numpy functions. For full credit, write cross_correlation_2d with only one pair of for loops that loop over the kernel; see the rubric for details on the points assigned for efficiency. Please don't sacrifice readability: make sure your approach is well-commented if you're making any nontrivial optimizations.

Testing. test.py contains a test suite that may help you debug your code.

Forbidden functions. For just this assignment, you are forbidden from using any Numpy, Scipy, OpenCV, or other preimplemented functions for filtering. This limitation will be lifted in future assignments, but for now, you should use for loops or Numpy vectorization to apply a kernel to each pixel in the image. The bulk of your code will be in cross_correlation_2d, and gaussian_blur_kernel_2d, with the other functions using these functions either directly or through one of the other functions you implement.

We have provided a GUI in gui.py, to help you debug your image filtering algorithm. To see a pre-labeled version of the sample images run:

We provide you with a pair of images that need to be aligned using the GUI. The code for alignment uses an affine transform to map the eyes to eyes and nose to nose, etc. as you specify on the UI. We encourage you to create additional examples (e.g. change of expression, morph between different objects, change over time, etc.). See the hybrid images project page for some inspiration. The project page also contains materials from their Siggraph presentation.

For the example shown at the top of the page, the two original images look like this:

The low-pass (blurred) and high-pass versions of these images look like this:

Adding the high and low frequencies together gives you the image at the top of this page. If you're having trouble seeing the multiple interpretations of the image, a useful way to visualize the effect is by progressively downsampling the hybrid image as is done below:

Python and Numpy Tutorials

We will use python programming language for all assignments in this course. In particular, we will make heavy use of the Numpy package for scientific computing. If you are not farmilar with python and numpy, the following websites provide very good tutorials for them. If you have any questions related to python and numpy, please ask on Piazza or visit office hours.

• Python and Numpy Tutorial from Stanford CS231n
• Official Numpy Quick Tutorial
• Numpy for Matlab users
• Short Python Tutorial

Submission

Code:

Artifact:

• left.png
• right.png
• hybrid.png

• Give the high pass and low pass filter parameters (kernel size and kernel sigma) and Mix-in ratio.
• Specify which image was used for the high frequencies and which was used for low frequency.
• Optionally you can add comments on something interesting or different you did in the project.

Late sumbission:

To submit your work late, you must push your changes via git (as usual) then send me an email stating that you have submitted the assignment late. The timestamp of the email, which must be sent after your final changes are pushed to git, will be used as the submission time. It is your responsibility to keep track of your slip day balance - no exceptions will be made for accounting errors on your part. If you are submitting the assignment late, you must submit the artifact at the same time as the code.

If you submit late but don't email me, the latest code on GitHub as of the submission deadline will be the version that is graded.