{ "cells": [ { "cell_type": "markdown", "id": "1ad710c2", "metadata": { "id": "1ad710c2" }, "source": [ "# L11 - Data Collection and Structured Data 1\n", "## HTML, XML, and Web Scraping" ] }, { "cell_type": "markdown", "id": "1d52ae3b", "metadata": { "id": "1d52ae3b" }, "source": [ "#### Announcements:\n", "\n", "* Ethics 2 out\n", " * Read an article and answer some questions by class Monday 5/4\n", " * In-class data analysis activity, done in pairs; submit by 10pm Monday 5/11\n", "* (draft) Project writeup is posted. Start thinking about topics to propose! You're encouraged to run ideas by me and/or Nick before writing up your proposal.\n", "* Feel free to code along with today's lecture - and there are a couple exercises interspersed!" ] }, { "cell_type": "markdown", "id": "EFgHfIcpLvQZ", "metadata": { "id": "EFgHfIcpLvQZ" }, "source": [ "#### Goals:\n", "* Know the basics of how to read HTML\n", "* Understand the basic purpose and structure of XML\n", "* Know how to get the HTML of a webpage in Python using `requests`\n", "* Know how to get information out of raw HTML by parsing it using the Beautiful Soup library." ] }, { "cell_type": "markdown", "id": "646f31cc", "metadata": { "id": "646f31cc" }, "source": [ "## HTML - HyperText Markup Language\n", "\n", "HTML is one pervasive example of **structured data**.\n", "\n", "This is the language that web page content is written in. Some basic facts about HTML:\n", "* It's not really a \"programming language\" per se - more like Markdown than like Python.\n", "* It's basically a way to assign structure to the webpage content.\n", "* Basic units of HTML:\n", " * **tags**: `Contents`\n", " * Example: `h1`: `

This is the largest heading

`\n", " * **attributes**, such as `href` attribute of the `a` tag:\n", " * `` stands for \"anchor\" and really means \"link\"\n", " * `Click here to go to example dot com` becomes [Click here to go to example dot com](https://example.com).\n", " * Comments are written `` \n", "* HTML is (delightfully) \"boring\": \n", " * HTML does *not* specify the appearance, formatting, or even layout of the page elements. This is done using a different language called Cascading Style Sheets (CSS).\n", " * Much of the fancy dynamic and interactive page content you encounter on real websites is implemented using JavaScript (a \"real\" programming language)." ] }, { "cell_type": "markdown", "id": "10ddbd95", "metadata": { "id": "10ddbd95" }, "source": [ "#### Basic Elements\n", "\n", "Instead of reading the following list, let's look at the source for a couple of webpages and see what we find:\n", "* https://fw.cs.wwu.edu/~wehrwes/\n", "* https://facultyweb.cs.wwu.edu/~wehrwes/courses/data311_26s/\n", "\n", "Ways to view HTML:\n", "* Right click > View Page Source\n", "* Open developer tools (varies by browser)" ] }, { "cell_type": "markdown", "id": "f6018ecc", "metadata": { "id": "f6018ecc" }, "source": [ "Some common HTML elements to know about:\n", "\n", "*Note*: If Jupyter sees HTML amongst your Markdown, it will render it like HTML - that's why I'm able to show you both the code and how it renders in the examples below.\n", "* The whole page is enclosed in an `html` tag\n", "* The body of the page is enclosed in a `body` tag\n", "* `h1`...`h6` are headings\n", " *

Heading 1

\n", " *

Heading 2

\n", " * ...\n", " *
Heading 6
\n", "* `p` is for paragraph\n", " *

Paragraph 1

Paragraph 2

\n", "* `div` is a general-purpose (and by default invisible) container for blocks of page content\n", " *
This stuff lives in a div.
\n", "* `span` is a general-purpose container for snippets of text content\n", " * This stuff lives in a span, but this stuff does not.\n", "* Tables allow you to lay out information in tabular format.\n", " ```html\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Heading 1 \n", " Heading 2 \n", "
Row 1, Column 1Row 1, Column 2
Row 2, Column 1Row 2, Column 2
\n", " ```\n", "renders to:\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Heading 1 \n", " Heading 2 \n", "
Row 1, Column 1Row 1, Column 2
Row 2, Column 1Row 2, Column 2
" ] }, { "cell_type": "markdown", "id": "52547e5c-bdbd-4b69-83c4-4d966171efc4", "metadata": {}, "source": [ "## XML\n", "\n", "Example XML document:\n", "```xml\n", "\n", "\n", " \n", " The Great Gatsby\n", " \n", " F. Scott\n", " Fitzgerald\n", " \n", " 1925\n", " 978-0-7432-7356-5\n", " \n", " Fiction\n", " Classic\n", " \n", " \n", " \n", " 1984\n", " \n", " George\n", " Orwell\n", " \n", " 1949\n", " 978-0-452-28423-4\n", " \n", " Dystopian\n", " Science Fiction\n", " \n", " \n", "\n", "```\n", "\n", "* XML is HTML's more general (and not particularly well-liked) cousin.\n", "* HTML-like, but not document-specific\n", "* Meant to represent whatever structured data you like\n", "* Doesn't even have the \"default\" presentation that HTML has via browsers\n", "* You can define a \"schema\" that narrows down what a \"valid\" XML document looks like for your particular use case.\n", " * For example, below: maybe you insist a `` can contain only a sequence of ``s\n", " * Documents can then be automatically error checked against the schema, then parsed much more easily\n", "\n", "One major (and largely successful) competitor to XML is JSON - we'll see this next time." ] }, { "cell_type": "code", "execution_count": null, "id": "48b3be23-8490-4f95-8233-e323cf5997ab", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "62236313", "metadata": { "id": "62236313" }, "source": [ "## Web Scraping\n", "#### So you want some data, but you can only find it buried in some webpage.\n", "\n", "Game plan:\n", "* Use `requests` to get the HTML code for a webpage given its URL\n", "* Use `beautifulsoup4` to parse the resulting HTML and extract the data we want from it.\n" ] }, { "cell_type": "markdown", "id": "8888e109-2a6a-4155-8f13-ab0d560c7db1", "metadata": {}, "source": [ "### Demo" ] }, { "cell_type": "code", "execution_count": 1, "id": "febba039", "metadata": { "executionInfo": { "elapsed": 164, "status": "ok", "timestamp": 1675881161960, "user": { "displayName": "Scott Wehrwein", "userId": "11327482518794216604" }, "user_tz": 480 }, "id": "febba039" }, "outputs": [], "source": [ "import requests\n", "import bs4\n", "import pandas as pd" ] }, { "cell_type": "code", "execution_count": 2, "id": "00c342bf", "metadata": { "id": "00c342bf" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " DATA 311 - Fundamentals of Data Science\n", "