{ "cells": [ { "cell_type": "markdown", "id": "34a0fb3c-33ba-4c84-ad50-2a8ff172f4e5", "metadata": {}, "source": [ "# L12 - Data Collection and Structured Data 2\n", "## Scraping Ethics; APIs; Reshaping and Merging" ] }, { "attachments": {}, "cell_type": "markdown", "id": "886fc079-d927-4ced-9256-89ed446b2ad5", "metadata": {}, "source": [ "#### Announcements\n", "\n", "* New seats, new friends!" ] }, { "cell_type": "code", "execution_count": null, "id": "6fd99cd1-2dff-4c1c-af07-62f0488e8402", "metadata": {}, "outputs": [], "source": [ "import random\n", "random.seed(311)\n", "datafolk = \"Alli Keira Malik Erika Narina Sebastian Josh Dylan Haden Zach Maven Marcus Finnley\".split()\n", "random.shuffle(datafolk)\n", "print(datafolk[:4])\n", "print(datafolk[4:9])\n", "print(datafolk[9:])" ] }, { "cell_type": "markdown", "id": "feb586e3-727c-49cf-b64d-72abc9e68c08", "metadata": {}, "source": [ "* Project proposal due Sunday\n", "* Ethics 2 due Monday\n", "* Code along with me again today!" ] }, { "cell_type": "markdown", "id": "d97c96b3-71a3-4e1f-bda9-b9fedafe9e84", "metadata": {}, "source": [ "#### Goals:\n", "* Understand how to responsibly and ethically use web scraping to collect datasets.\n", "* Know how to make basic usage of an API to fetch data\n", "* Know how to reshape tables from long to wide and wide to long format\n", "* Know how to join tables using left, right, inner, and outer joins." ] }, { "cell_type": "markdown", "id": "9c9ac8cf-7caf-42cc-bc8f-9702bcead49c", "metadata": { "id": "M2XMbzBEqaIi" }, "source": [ "# Scraping Ethics\n", "\n", "* What does it cost you to scrape a website?\n", "* What does it cost the person/company/entity of the website being scraped?\n", "\n", "* Should anyone be able to scrape anything?\n", "\n", "* Should we do a Data Ethics reading on web scraping in the AI era?\n", "\n", "# Scraping Etiquette\n", "\n", "Things to keep in mind:\n", "* First, ask: can you get the data without scraping?\n", " * If the service provides downloadable datasets or an API, use these instead of scraping.\n", "* Scraping public data from websites is generally OK (but I'm not a lawyer and this is not legal advice).\n", "* Most websites will have **Terms of Service** or Terms of Use. Violating these may be illegal (but I am not a lawyer and this is not legal advice).\n", " * Most sites will also have a `robots.txt` which specifies how and what non-human users may access. Example: \n", "* **Don't redistribute** data without permission.\n", "* **Rate limit** your scraping requests - wait *at least* 1 second between requests (for a typical webpage); ideally more like 5-10 seconds is better.\n", " * `robots.txt` may specify a rate limit; respect this\n", " * If you don't rate limit, you are indistinguishable from a denial-of-service attack.\n", " * If pages are large or involve database queries on the backend, it may be polite to wait longer between queries.\n", "* Always **save results** instead of re-requesting.\n", " * Save early before too much analysis is done in case you want to change your analysis." ] }, { "cell_type": "markdown", "id": "dab1848c-6278-43c2-8740-81e885b5e423", "metadata": {}, "source": [ "## Using APIs\n", "\n", "Scraping is a workaround; APIs (**application programming interfaces**) are designed to respond to programmatic requests for data.\n", "\n", "There are many different sorts of APIs, but the gist of how to use them is:\n", "* Construct a URL that encodes the parameters of your request\n", "* Visit that URL in code (e.g. via Python's `requests`) or some other tool (`curl` unix command)\n", " * You can use a web browser, but the response generally isn't a displayable webpage.\n", "* Get back a response in some kind of structured data format - often JSON or XML." ] }, { "cell_type": "markdown", "id": "593227d7-fddd-4feb-8165-17d93e6f42b9", "metadata": {}, "source": [ "### API Demo\n", "OpenWeatherMap.org has an API with a generous free usage tier - you can make up to 1000 requests per day for free.\n", "\n", "#### API Keys and Security\n", "To use it, you need an API key - you can register and get your own key at . \n", "\n", "API keys should be treated like passwords - they are a secret; if I give you mine, you can burn through my 1000-request quota as quick as you like.\n", "\n", "Even worse, many (most?) APIs are paid, either by subscription with limits or per-call. Leaking your API key could be equivalent to leaking your credit card!\n", "\n", "I've stored mine in a file to avoid including it plaintext in this notebook:" ] }, { "cell_type": "code", "execution_count": null, "id": "3ca64ade-0a55-44ac-8cd6-0868dea3c67b", "metadata": {}, "outputs": [], "source": [ "API_KEY = open(\"/cluster/home/wehrwes/Documents/openweathermap_api_key.txt\").read().strip()\n", "BASE_URL = \"http://api.openweathermap.org/data/2.5/weather\"" ] }, { "cell_type": "code", "execution_count": null, "id": "7d206ac0-62a2-40eb-ba63-32923f39d29c", "metadata": {}, "outputs": [], "source": [ "import time\n", "import json\n", "import requests\n", "import pandas as pd\n", "import seaborn as sns" ] }, { "cell_type": "markdown", "id": "87b48d3d-ba8d-4b2b-aaaf-2d107583165b", "metadata": {}, "source": [ "#### A Basic API request with Python" ] }, { "cell_type": "code", "execution_count": null, "id": "ad3f49af-05ad-4b64-9c80-c244ef37b9fb", "metadata": {}, "outputs": [], "source": [ "def get_weather_json(city):\n", " params = {\n", " 'q': city,\n", " 'appid': API_KEY,\n", " 'units': 'metric'\n", " }\n", " return requests.get(BASE_URL, params=params).json()" ] }, { "cell_type": "markdown", "id": "cce16915-bf4c-454b-82dc-6ef1210820af", "metadata": {}, "source": [ "Let's try it out:" ] }, { "cell_type": "code", "execution_count": null, "id": "163a6061-518f-4c76-be76-31ec1927dc1c", "metadata": {}, "outputs": [], "source": [ "seattle = get_weather_json(\"Seattle\")\n", "print(seattle)" ] }, { "cell_type": "markdown", "id": "d975fd96-174f-477c-a49e-015cfc19e4b1", "metadata": {}, "source": [ "This is an example of another structured data format called **JSON** (JavaScript Object Notation).\n", "\n", "Similar to a Python dictionary, a JSON object represents a collection of key-value pairs. They can be nested to represent hierarchical structure similar to what we saw with XML; many people find the syntax to be a little nicer than XML.\n", "\n", "The response came back from the API as a string, but `requests` has already parsed it into a python `dict` for us. Python has a `json` module that works with JSON to help manipulate them; here we'll use it to pretty-print the response:" ] }, { "cell_type": "code", "execution_count": null, "id": "07bc66d3-a4b5-44cf-9037-821deb0de2a1", "metadata": {}, "outputs": [], "source": [ "print(json.dumps(seattle, indent=2))" ] }, { "cell_type": "code", "execution_count": null, "id": "cf224551-c22b-4478-bf3a-f68ccb385ede", "metadata": {}, "outputs": [], "source": [ "with open(\"seattle.json\", \"w\") as f:\n", " json.dump(seattle, f)" ] }, { "cell_type": "code", "execution_count": null, "id": "6db8a360-941b-4f7a-9f02-f33def27cf5f", "metadata": {}, "outputs": [], "source": [ "# if you didn't make the API call, use this to load up a sample JSON result:\n", "seattle = requests.get(\"https://facultyweb.cs.wwu.edu/~wehrwes/courses/data311_26s/lectures/L12/seattle_json.csv\").json()" ] }, { "cell_type": "markdown", "id": "c54c6615-9524-416d-a0a3-702352efeaf8", "metadata": {}, "source": [ "Since it's just a `dict`, we can access things using familiar dictionary indexing syntax:" ] }, { "cell_type": "code", "execution_count": null, "id": "4e61f3c9-2127-4cb7-b0f7-c3ef52e7ad7a", "metadata": {}, "outputs": [], "source": [ "# get the temperature\n", "seattle[\"main\"][\"temp\"]" ] }, { "cell_type": "markdown", "id": "393a61f4-ddab-45b5-8682-578a0873027e", "metadata": {}, "source": [ "**Exercise**: get the wind speed" ] }, { "cell_type": "code", "execution_count": null, "id": "ca065177-048d-4656-be70-49e0d24b049f", "metadata": {}, "outputs": [], "source": [ "seattle[\"wind\"][\"speed\"]" ] }, { "cell_type": "markdown", "id": "5d816b28-ee03-4e77-a528-2e8aaedcbb6c", "metadata": {}, "source": [ "Now let's use this to build a little dataset of weather for a few cities:" ] }, { "cell_type": "code", "execution_count": null, "id": "bed2d526-18ab-4293-ae7c-de6d97dda095", "metadata": {}, "outputs": [], "source": [ "cities = [\"Seattle\", \"Los Angeles\", \"London\", \"Paris\", \"Tokyo\", \"Sydney\", \"New York\"]\n", "\n", "# Fetch current weather for multiple cities\n", "data = {}\n", "for city in cities:\n", " data[city] = get_weather_json(city)\n", "\n", " time.sleep(1)" ] }, { "cell_type": "code", "execution_count": null, "id": "974b92f3-981c-4964-a66d-fa93a2c79030", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "b7e75e88-0fab-45f9-aed1-acf8bb7e3c7d", "metadata": {}, "outputs": [], "source": [ "data_table = []\n", "for city, resp in data.items():\n", " data_table.append({\n", " 'city': city,\n", " 'temp': resp['main']['temp'],\n", " 'feels_like': resp['main']['feels_like'],\n", " 'humidity': resp['main']['humidity']\n", " })" ] }, { "cell_type": "code", "execution_count": null, "id": "a0525410-5be1-4d4a-af87-4bbc3765df79", "metadata": {}, "outputs": [], "source": [ "df_orig = pd.DataFrame(data_table)\n", "df_orig" ] }, { "cell_type": "markdown", "id": "15a699e4-c60c-41cd-ab71-02f231193c97", "metadata": {}, "source": [ "## Merging Tables" ] }, { "cell_type": "markdown", "id": "60c031d7-a8e6-4e86-8d9f-af829103b988", "metadata": {}, "source": [ "Sometimes you have multiple tables with complementary information - this will happen in Lab 5, where we'll scrape two different sites for different info about (some of) the same movies.\n", "\n", "How do we combine tables together? `pd.merge`.\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "id": "1287561d-efe3-4caf-9124-8ebf6817d9bb", "metadata": {}, "outputs": [], "source": [ "employees = pd.DataFrame({\n", " 'emp_id': [1, 2, 3, 4],\n", " 'name': ['Alice', 'Bob', 'Charlie', 'Diana'],\n", " 'dept_id': [10, 20, 10, 30]\n", "})\n", "employees" ] }, { "cell_type": "code", "execution_count": null, "id": "9c332eca-613b-4603-ab96-788a26cf2b25", "metadata": {}, "outputs": [], "source": [ "departments = pd.DataFrame({\n", " 'dept_id': [10, 20, 40],\n", " 'dept_name': ['Engineering', 'Sales', 'Marketing']\n", "})\n", "departments" ] }, { "cell_type": "markdown", "id": "c8c5e8ee-5120-4191-a399-db0eb1e05c96", "metadata": {}, "source": [ "### Inner Join\n" ] }, { "cell_type": "code", "execution_count": null, "id": "3798eda7-4450-4739-8423-2ada86644eb5", "metadata": {}, "outputs": [], "source": [ "employees.merge(departments, on='dept_id', how='inner')" ] }, { "cell_type": "markdown", "id": "7f01e6f6-ef16-4b71-828b-08c82487c4f7", "metadata": {}, "source": [ "**Exercise 1** (worksheet): Describe what an **inner join** does." ] }, { "cell_type": "markdown", "id": "6ffce2b6-6011-4ff2-8252-5a84ea78841c", "metadata": {}, "source": [ "### Left Join and Right Join" ] }, { "cell_type": "code", "execution_count": null, "id": "d328be1d-3e6e-47ac-b49e-67b48b27e460", "metadata": {}, "outputs": [], "source": [ "# Left join:\n", "employees.merge(departments, on='dept_id', how='left')" ] }, { "cell_type": "code", "execution_count": null, "id": "db2a756b-bff6-4a27-b312-9c1657d7e21c", "metadata": {}, "outputs": [], "source": [ "# Right join:\n", "employees.merge(departments, on='dept_id', how='right')" ] }, { "cell_type": "markdown", "id": "e94459d0-58a7-490e-91cd-d4bb5643cb9c", "metadata": {}, "source": [ "**Exercise**: Describe what **left** and **right joins** do." ] }, { "cell_type": "markdown", "id": "b84151d2-fad5-4900-ad83-a63cae3b9567", "metadata": {}, "source": [ "### Outer join" ] }, { "cell_type": "code", "execution_count": null, "id": "4880aa79-82d2-494a-b34b-1e7456c9532d", "metadata": {}, "outputs": [], "source": [ "employees.merge(departments, on='dept_id', how='outer')" ] }, { "cell_type": "markdown", "id": "f3993cf6-8f4a-4b55-a030-a75b85598ef0", "metadata": {}, "source": [ "**Exercise**: Describe what an **outer join** does." ] }, { "cell_type": "markdown", "id": "67b9caf5-ad7c-4f92-b5ef-19eca4eabce2", "metadata": {}, "source": [ "**Exercise**: I have two CSV files from my CSCI 141 class; one is from the start-of-quarter survey, and one is the final gradebook.\n", "\n", "On the start-of-quarter survey (in a dataframe `survey`), students reported how many months of programming experience they had. Some students added the class late, and are not present in the survey results.\n", "\n", "The gradebook (in a DataFrame `grades`) has all the grades for the quarter. Some students who dropped the course after the first week are not included are not included in the gradebook.\n", "\n", "Both files have the following columns:\n", "* Name (in Last, First format)\n", "* W number\n", "* Email address\n", "\n", "I want to analyze the relationship between programming experience and final grades, so I can run something like\n", "\n", "```\n", "sns.scatter(data=combined, x=\"Months Experience\", y=\"Final Grade\")\n", "```\n", "\n", "How should I construct the `combined` dataframe that will enable this?" ] }, { "cell_type": "markdown", "id": "7807a177-563e-4d16-adb5-12f527ed0913", "metadata": {}, "source": [ "## Reshaping Data Tables: Long format vs Wide Format" ] }, { "cell_type": "markdown", "id": "fe75d975-e85e-4485-a44b-4a89db6b12a9", "metadata": {}, "source": [ "When dealing with tabular data, there's a notion of \"wide\" vs \"long\" format.\n", "\n", "The table we've built is considered **wide** because each property occupies a column:" ] }, { "cell_type": "code", "execution_count": null, "id": "d5ec2289-bd92-42fd-a006-188fb999d11a", "metadata": {}, "outputs": [], "source": [ "df_wide = df_orig.set_index('city')\n", "df_wide" ] }, { "cell_type": "markdown", "id": "c0a518de-6d19-4876-a2d5-e937a52457d0", "metadata": {}, "source": [ "If we want to plot one column, for example, this is great:" ] }, { "cell_type": "code", "execution_count": null, "id": "d47b6777-e2af-4b21-ab26-ba3edd9096fb", "metadata": {}, "outputs": [], "source": [ "sns.barplot(df_wide[\"temp\"]);" ] }, { "cell_type": "markdown", "id": "8bd44a20-0ca5-4b15-9b3c-f51c34002924", "metadata": {}, "source": [ "If we want to plot them all alongside each other, Seaborn doesn't do what we really want:" ] }, { "cell_type": "code", "execution_count": null, "id": "c249ad2a-fbbf-455f-bdce-5b30ba5e611d", "metadata": {}, "outputs": [], "source": [ "sns.barplot(data=df_wide);" ] }, { "cell_type": "markdown", "id": "003d0b3c-2486-4930-9ed3-2f903b0ce928", "metadata": {}, "source": [ "Sometimes it's useful to convert to **long** format, where the values all live in one column and the properties (formerly column names) are a categorical column.\n", "\n", "In pandas, we do this with `melt`.\n", "\n", "Here, the parameters are:\n", "* `id_vars`, the thing we want to keep as a per-row thing; this can be more than one column!\n", "* `var_name` is the name of the categorical column that will contain the other column names\n", "* `value_name` is the name of the numerical column with the values of the properties in the `var_name` column.\n", "\n", "Sound confusing? Example:" ] }, { "cell_type": "code", "execution_count": null, "id": "9de8c392-1a96-4454-b91f-480c2dc3d255", "metadata": {}, "outputs": [], "source": [ "df_long = df_orig.melt(id_vars=['city'],\n", " value_vars=[\"temp\", \"feels_like\", \"humidity\"],\n", " var_name='metric', value_name='value')\n", "df_long" ] }, { "cell_type": "markdown", "id": "ae5090f7-0e85-4d50-9b8c-c5d0e36a0a07", "metadata": {}, "source": [ "Why would we want this? it might be more flexible for certain kinds of analysis (think group_by->aggregate, maybe)?\n", "\n", "Also, Seaborn:" ] }, { "cell_type": "code", "execution_count": null, "id": "c8729a83-2d26-444b-9434-5b4424fe94d6", "metadata": {}, "outputs": [], "source": [ "sns.barplot(data=df_long, x='city', y='value', hue='metric');" ] }, { "cell_type": "markdown", "id": "9fdcd445-e0d1-4c76-88fd-3e418e7d9fcb", "metadata": {}, "source": [ "If you have long data and want wide data, you can use **pivot**:" ] }, { "cell_type": "code", "execution_count": null, "id": "aa239662-d77a-4125-a5bf-a77081a1f481", "metadata": {}, "outputs": [], "source": [ "# long to wide\n", "df_long.pivot(index=\"city\", columns=\"metric\", values=\"value\")" ] }, { "cell_type": "markdown", "id": "25ad921e-cef1-433d-a000-e5eaf312181b", "metadata": {}, "source": [ "### Exercise: tips dataset" ] }, { "cell_type": "code", "execution_count": null, "id": "df39c9fe-f74c-402f-bb7c-2acd61a512a9", "metadata": {}, "outputs": [], "source": [ "tips = sns.load_dataset('tips')\n", "tips.head()" ] }, { "cell_type": "markdown", "id": "3d4032b5-2d9f-4bf2-8c1f-9345b034aa1b", "metadata": {}, "source": [ "**Exercise**: Use `melt` to stack `total_bill` and `tip` into one column so you can plot both on the same chart with different colors. Leave the other columns alone (keep them as id_vars). To be compatible with the plotting command below, use `amount_type` for the variable name and `dollars` for the value name." ] }, { "cell_type": "code", "execution_count": null, "id": "bfbf524f-0681-4e63-84c4-fa014b7bf25f", "metadata": {}, "outputs": [], "source": [ "tips_long = # ...\n", "tips_long" ] }, { "cell_type": "markdown", "id": "6a394141-5dc6-450a-b58e-c8f6c83f5db1", "metadata": {}, "source": [ "Here's a plot that shows total bill and tip side-by-side, per-day, and aggregated over all meals:" ] }, { "cell_type": "code", "execution_count": null, "id": "6e02f6fc-5b86-45cf-bed3-a14c60549137", "metadata": {}, "outputs": [], "source": [ "sns.barplot(data=tips_long, x='day', y='dollars', hue='amount_type');" ] }, { "cell_type": "markdown", "id": "19b42087-2a80-4dc1-86d5-fa3dec8c1998", "metadata": {}, "source": [ "### Exercise / Example: Flights dataset" ] }, { "cell_type": "code", "execution_count": null, "id": "9c29f2eb-5e8f-45db-8cc3-d67c282f519a", "metadata": {}, "outputs": [], "source": [ "flights = sns.load_dataset('flights')\n", "flights" ] }, { "cell_type": "markdown", "id": "06d5df73-5601-4c51-9b5f-10346b85bb35", "metadata": {}, "source": [ "This dataset comes in **long** format!" ] }, { "cell_type": "code", "execution_count": null, "id": "8c242668-38ed-42de-95e9-cd077713ac2f", "metadata": {}, "outputs": [], "source": [ "sns.lineplot(data=flights, x=\"month\", y=\"passengers\", hue=\"year\");" ] }, { "cell_type": "markdown", "id": "aa382640-c446-42a7-9fc1-0b1b2929c914", "metadata": {}, "source": [ "**Exercise**: Convert it to **wide** format using `pivot`. In particular, we want a table with `year` as the index, and one column per `month`, such that each table cell is the number of flights in the given year (row) and month (column)." ] }, { "cell_type": "code", "execution_count": null, "id": "9d0037a2-1ac3-49ad-980d-5967f5c57de0", "metadata": {}, "outputs": [], "source": [ "# flights_wide = ...\n", "flights_wide" ] }, { "cell_type": "markdown", "id": "6b3ccb4b-0caa-4335-8fe8-06fd89aed885", "metadata": {}, "source": [ "With flights_wide as described above, we can plot the table as a heatmap:" ] }, { "cell_type": "code", "execution_count": null, "id": "68c94e0e-1da3-417a-904d-20da0ab9b1eb", "metadata": {}, "outputs": [], "source": [ "sns.heatmap(flights_wide, annot=True, fmt='d');" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.13" } }, "nbformat": 4, "nbformat_minor": 5 }