Lecture 15

Get Yer Data - Discussion

Correlation (does not imply causation)

import pandas as pd

Announcements:

• Lab 2 grades out. Good overall! Some general comments:

  • I spent a lot of time and gave detailed feedback. Please read it.
  • Pretty much never loop over rows of a dataframe. There's pretty much always a better way using pandas functions.
  • If you have (say) 18 or 12 dataframes or subsets thereof, please loop over them! Please write functions to perform processing that you need to do multiple times!
    • Digging for new pandas functions is often worth it, e.g.: df.resample()
  • Use the notebook format to your advantage, and consider your reader when organizing your cells.

• Week 3 survey - themes:

  • Many requests for more pictures of cats (harder to put cats in dataframes than ndarrays, but I'll see what I can do...)
  • Probability is confusing (it is famously true that probability is unintuitive) - quiz 3 explainer video?
  • Finding data was the hard part of lab 3.

Goals:

• Discuss the results of your investigations of the data some internet company has collected about you.
• Understand the distinction between correlation and causation, and be wary of the dangers of assuming that correlation implies causation.

Data Ethics - Get Yer Data

df = pd.DataFrame(index=range(1,6))

Poll time!

1. On a scale from 1 (completely unsurprised) to 5 (shocked), how surprised were you by the data?

df["Surprised?"] = [2, 15, 10, 1, 0]
df["Surprised?"].plot.bar()

<AxesSubplot:>

1. On a scale from 1 (unperturbed) to 5 (fully spooked), how creeped out were you by the amount or types of data?

df["Creeped out?"] = [5, 6, 11, 3, 1]
df["Creeped out?"].plot.bar()

<AxesSubplot:>

1. On a scale from 1 (completely bogus) to 5 (perfectly accurate), how accurate did you find the data to be?

df["Accurate?"] = [0, 1, 4, 23, 2]
df["Accurate?"].plot.bar()

<AxesSubplot:>

Discussion in groups of three:

1. Discuss any particularly notable findings that pertain to the questions you were asked to write about:

   1. How did your data compare to what you expected? Was there anything surprising, or creepy, or just plain strange? Describe the types of data that you see.
   2. How comprehensive was your download? Are you able to determine whether the company gave you everything they had, or were they more selective?
   3. What kinds of data science questions could someone answer about you based solely on this data? What kinds of data science questions could someone with access to millions of records like yours answer?
   4. Are you comfortable with the extent and/or accuracy of data collected? Does the company have controls for opting out of collection of the sorts of data you’d rather they not have? If not - or if the company suddenly decided tomorrow to remove those controls - what should our society do about this?

2. Were your reactions similar or different? Was this due to your attitudes, or due to differences in the data?

Decide on one most interesting thing (from any of the above discussion) to share with the class. The person who woke up latest this morning will be your group's spokesperson.

Discussion as a class:

1. Is there a problem here?
2. If so, how should society solve it?