Lecture 20 - Linear Regression Plus Tricks; Generalization, Continued

Announcements:

• There was no Quiz 7; there will be a Quiz 8
• FP proposal feedback out to 7 of 11 groups; remaining 4 coming later today.
• FP presentations next week T, W, and F if needed.
• Faculty Candidate talks today and tomorrow - Hsiang-Jen Hong:
  • Monday 2/27 4pm CF 105 - Research Talk: Making the Bitcoin Network More Secure and Scalable
  • Tuesday 2/28 4pm CF 025 - Teaching Demo: SSL/TLS: From Basics to Best Practices

Goals

Linear Regression Plus Tricks:

• Understand the setup of the linear regression problem in linear algebra notation.
• Know how to fit nonlinear functions with linear regression using feature expansions
• Know about some of the limitations and failure modes of linear regression:
  • Outliers
  • Linearly dependent columns

Generalization, Continued:

• Understand how Occam's Razor applies to mathematical modeling and prediction tasks
• Know why and how to create separate validation and test sets to evaluate a model
• Know how cross-validation works and why you might want to use it.

Linear Regression

• In general, linear regression models map input $x \in \mathbb{R}^D$ to output $y \in \mathbb{R}^C$
  • If $D=C=1 \rightarrow$ simple linear regression
  • If $D>1, C=1 \rightarrow$ multiple linear regression
  • If $D \geq 1, C>1 \rightarrow$ multivariate linear regression

Multiple linear regression

$$ h(x) = w^T x + b = \sum_{i=1}^D w_i x_i + b$$

• Input: $x \in \mathbb{R}^D$
• Weights: $w \in \mathbb{R}^D$
• Bias: $b \in \mathbb{R}$
• Unfortunately, still cannot teach you how to train them (mathematically)

Multivariate linear regression

$$ h(x) = W^T x + b $$

• Input: $x \in \mathbb{R}^D$
• Weights: $W \in \mathbb{R}^{D \times C}$
• Bias: $b \in \mathbb{R}^C$
• This is functionally the same as doing $C$ different multiple linear regression models, one for each output element
• Unfortunately, still cannot teach you how to train them (mathematically)