In [3]:
sns.scatterplot(x="Height", y="Leg", data=bm, s=3)
Out[3]:
<AxesSubplot:xlabel='Height', ylabel='Leg'>
import pandas as pd
import numpy as np
import seaborn as sns
resample (and a couple other neat tricks) on the weather data is linked from yesterday's lecture.What do we mean by correlation?
So far, an informal definition: Variables are correlated if one appears to predict the other, or if an increase or decrease in one is consistently associated with an increase or decrease in the other.
import pandas as pd
import seaborn as sns
data_url = "https://fw.cs.wwu.edu/~wehrwes/courses/data311_21f/data/NHANES/NHANES.csv"
cols_renamed = {"SEQN": "SEQN",
"RIAGENDR": "Gender", # 1 = M, 2 = F
"RIDAGEYR": "Age", # years
"BMXWT": "Weight", # kg
"BMXHT": "Height", # cm
"BMXLEG": "Leg", # cm
"BMXARML": "Arm", # cm
"BMXARMC": "Arm Cir", # cm
"BMXWAIST": "Waist Cir"} # cm
bm = pd.read_csv(data_url)
bm = bm.rename(cols_renamed, axis='columns')
bm = bm.drop("SEQN", axis='columns')
bm = bm[bm["Age"] >= 21]
sns.scatterplot(x="Height", y="Leg", data=bm, s=3)
<AxesSubplot:xlabel='Height', ylabel='Leg'>
sns.scatterplot(x="Height", y="Arm Cir", data=bm, s=3)
<AxesSubplot:xlabel='Height', ylabel='Arm Cir'>
sns.scatterplot(x="Age", y="Weight", data=bm, s=3)
<AxesSubplot:xlabel='Age', ylabel='Weight'>
sns.scatterplot(x="Age", y="Arm Cir", data=bm[bm["Age"] >= 60], s=3)
<AxesSubplot:xlabel='Age', ylabel='Arm Cir'>
Variables are correlated if one appears to predict the other, or if an increase or decrease in one is consistently associated with an increase or decrease in the other.
Suppose you open up a dataset, make a jointplot and find that two variables A and B are highly correlated. Without knowing any more than that, what might have given rise to this correlation?
A is liquid precip per minute, B is number of people wearing raincoats.
What happened? A caused B.
A is a measure of lung health; B is number of cigarettes smoked per day.
What happened? B caused A
A is the ranking of the college a person graduated from; B is their score on the GRE (an SAT-like exam sometimes used for graduate school admissions).
What happened? C caused B and A
A is height in centimeters; B is height in feet
What happened? A is B.
A is total firefox downloads and B is membership in the Wicca religion.
What happened?
More (real) examples: https://www.tylervigen.com/spurious-correlations
It's easy to find ridiculous correlations that are obviously not causal and have a good laugh about it.
It's still easy to see correlation and jump to the conclusion that there is a causation at work, (even if you understand intellectually that correlation does not imply causation!)
It's also vastly easier to find correlations than to prove causation - this is a big part of what makes science hard.
A fun activity: google "study finds link" and look at some of the news articles that result.
From my second result, way down the page:
[The scientists] added that randomized intervention trials, as well as long-term objective measurements of physical activity in prospective studies, are also needed to assess the validity and causality of the association they reported.
Title text: "Correlation doesn't imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing 'look over there'."
We'd like a number (a statistic, really) that captures the degree of correlation between two variables.
Here's one such statistic, called the Pearson Correlation Coefficient: $$r(X, Y) = \frac{\sum_{i=1}^n (X_i - \bar{X}) (Y_i - \bar{Y})} {\sqrt{\sum_{i=1}^n (X_i-\bar{X})^2}\sqrt{\sum_{i=1}^n(Y_i-\bar{Y})^2}}$$
where $\bar{X}$ and $\bar{Y}$ are the means $X$ and $Y$, respectively.
On the one hand, yikes! On the other hand, this is made of mostly things we've seen before. Notice that the denominator is the product of the two variables' standard deviations! $$r(X, Y) = \frac{\sum_{i=1}^n (X_i - \bar{X}) (Y_i - \bar{Y})} {\sigma(X) \sigma(Y)}$$
And the numerator is... well, it's sort of like a variance, except instead of squaring the distance of one varaible to its mean, we multiply $X_i$'s distance from the mean by $Y_i$'s distance from the mean. This is a nifty enough thing that it also has its own name: covariance. So we can equivalently write
$$r(X, Y) = \frac{\mathrm{Cov}(X,Y)}{\sigma(X) \sigma(Y)}$$Note: You might notice that the denominators are not quite standard deviations: they're missing the $1/(n-1)$ normalizer. Covariance alone is also usually defined with a $1/(n-1)$, but by leaving it out on the top and bottom we get the same result because the $1/(n-1)$s would cancel each other out.
Of course pandas has a function to calculate $r$ for you:
bm["Height"].corr(bm["Leg"]) # pandas
0.7887364110516857
You can also just compute all the correlations among all columns of a dataframe:
bm.corr() # pandas-er
| Gender | Age | Weight | Height | Leg | Arm | Arm Cir | Waist Cir | |
|---|---|---|---|---|---|---|---|---|
| Gender | 1.000000 | -0.024645 | -0.245535 | -0.680140 | -0.546402 | -0.556222 | -0.153541 | -0.112250 |
| Age | -0.024645 | 1.000000 | -0.080169 | -0.140594 | -0.264969 | 0.019138 | -0.123960 | 0.144533 |
| Weight | -0.245535 | -0.080169 | 1.000000 | 0.446089 | 0.334135 | 0.566939 | 0.891091 | 0.896244 |
| Height | -0.680140 | -0.140594 | 0.446089 | 1.000000 | 0.788736 | 0.803174 | 0.262247 | 0.201539 |
| Leg | -0.546402 | -0.264969 | 0.334135 | 0.788736 | 1.000000 | 0.653224 | 0.207673 | 0.063750 |
| Arm | -0.556222 | 0.019138 | 0.566939 | 0.803174 | 0.653224 | 1.000000 | 0.432323 | 0.397757 |
| Arm Cir | -0.153541 | -0.123960 | 0.891091 | 0.262247 | 0.207673 | 0.432323 | 1.000000 | 0.809686 |
| Waist Cir | -0.112250 | 0.144533 | 0.896244 | 0.201539 | 0.063750 | 0.397757 | 0.809686 | 1.000000 |
def corrplot(c1, c2, df, sz=2):
r = df[c1].corr(df[c2]) # Pearson
p = df[c1].corr(df[c2], method="spearman")
sns.scatterplot(x=c1, y=c2, data=df, s=sz).set(title=f"r={r:.02}; p={p:.02}")
corrplot("Height", "Leg", bm)
corrplot("Height", "Arm Cir", bm)
corrplot("Age", "Height", bm)
Let's look at a synthetic example to make a couple points here:
import numpy as np
df = pd.DataFrame()
x1 = np.arange(0, 10, .1)
y_ideal = np.arange(6, 10, 0.04)
noise = np.random.randn(*y_ideal.shape) * .2
y1 = y_ideal + noise
df["x1"] = x1
df["y1"] = y1
corrplot("x1", "y1", df, sz=8)
x2 = np.arange(0, 10, .1)
y2_ideal = np.arange(6, 10, 0.04)
y2_ideal[50] = 38
noise2 = np.random.randn(*y2_ideal.shape) * .5
y2 = y2_ideal + noise2
df["x2"] = x2
df["y2"] = y2
corrplot("x2", "y2", df, sz=20)
x3 = np.concatenate((np.arange(0, 5, .1), np.arange(5, 6, 0.02)))
y3 = np.concatenate((np.arange(6, 8, 0.04), np.arange(8, 12, 0.08)))
df["x3"] = x3
df["y3"] = y3
corrplot("x3", "y3", df, sz=8)
x4 = np.arange(0, 100)
y4 = np.arange(6, 106)**6
df["x4"] = x4
df["y4"] = y4
corrplot("x4", "y4", df, sz=8)
Notice that Pearson correlation ($r$) is concerned with how well the varaibles are modeled by a linear relationship. Deviations from that result in lower $r$. Also outliers have an outsided effect on $r$.
Another measure of correlation that is less susceptible to these is called the Spearman Rank Correlation Coefficient.
The intuition is to look at the relative positioning of a value of $x$ within its fellow $x$s and see how that compares to the corresponding $y$'s position among $y$s. Let $rank(x_i)$ be the position of $x_i$ in sorted order (i.e., 1st for the smallest, $n$ for the largest). Then if the $x$'s and $y$'s are perfectly "rank-correlated", then $rank(x_i)$ will equal $rank(y_i)$ for all $i$. A measure of how far the dataset deviates from this ideal is: $$\rho = 1 - \frac{6 \sum_{i=1}^n (rank(x_i) - rank(y_i))^2}{n(n^2-1)}$$
The details of this formula aren't too interesting; the factor of 6 in the numerator and the entire denominator are just there to make the sure the reusult lies in the range from -1 to 1.
TODO: Add the Spearman Correlation Coefficient to the plots above and see how the two measures compare for our synthetic examples.