{ "cells": [ { "cell_type": "markdown", "id": "5886e925", "metadata": { "id": "5886e925" }, "source": [ "# Lecture 9 - Preprocessing and Cleaning: Text Normalization and Natural Language Processing" ] }, { "cell_type": "code", "execution_count": null, "id": "b788b5bb", "metadata": { "executionInfo": { "elapsed": 1224, "status": "ok", "timestamp": 1675189729193, "user": { "displayName": "Scott Wehrwein", "userId": "11327482518794216604" }, "user_tz": 480 }, "id": "b788b5bb" }, "outputs": [], "source": [ "import numpy as np\n", "import seaborn as sns\n", "import pandas as pd\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "id": "f95f47be", "metadata": { "id": "f95f47be" }, "source": [ "# Announcements:\n", "* Data Ethics 2 - next wednesday, apparently! I'll get this out Soon.\n", "\n", "\n", "# Goals:\n", "* Know the meaning and purpose of some basic text normalization operations (from natual language processing):\n", " * Sentence tokenization\n", " * Lowercasing, contractions, punctuation, canonicalization\n", " * Stemming\n", " * Lemmatization\n", " * Stopword removal\n", "* Get some hands-on practice using the above" ] }, { "cell_type": "markdown", "id": "KTrujBZBZx7h", "metadata": { "id": "KTrujBZBZx7h" }, "source": [ "# Start of Quarter Survey" ] }, { "cell_type": "markdown", "id": "fCfbKAhJ66Pu", "metadata": { "id": "fCfbKAhJ66Pu" }, "source": [ "Responses to the survey prompt (from Fall, when I had a bigger dataset): \n", "> Name one hobby or activity you enjoy outside of school." ] }, { "cell_type": "code", "execution_count": 94, "id": "nrLemP4vaCdn", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 424 }, "executionInfo": { "elapsed": 661, "status": "ok", "timestamp": 1675190787375, "user": { "displayName": "Scott Wehrwein", "userId": "11327482518794216604" }, "user_tz": 480 }, "id": "nrLemP4vaCdn", "outputId": "d372113e-8d8c-4125-da61-81f5fcbbf31e" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
0
0I love riding my bike, recently I have been en...
1I like hanging out with friends by going on wa...
2I love reading eastern fantasy/cultivation nov...
3Hiking!
4I love singing! It's technically in-school, bu...
5I like video games
6I manage a home media server in my downtime, i...
7Gaming
8I enjoy long distance running! I've been doing...
9Rock climbing
10Volleyball
11playing bass guitar
12I like to play the guitar and to cook
13Video Games
14I enjoy biking and hiking outdoors, and readin...
15Basketball
16Mountain biking
17Reading
18Archery. Painting. Reading.
19I really love baseball. My parents and I watch...
20Reading
21Hiking
22rock climbing
23Golf and Swiming
24Lifting!
25I enjoy working out, hanging out with friends,...
\n", "
" ], "text/plain": [ " 0\n", "0 I love riding my bike, recently I have been en...\n", "1 I like hanging out with friends by going on wa...\n", "2 I love reading eastern fantasy/cultivation nov...\n", "3 Hiking!\n", "4 I love singing! It's technically in-school, bu...\n", "5 I like video games\n", "6 I manage a home media server in my downtime, i...\n", "7 Gaming\n", "8 I enjoy long distance running! I've been doing...\n", "9 Rock climbing \n", "10 Volleyball\n", "11 playing bass guitar\n", "12 I like to play the guitar and to cook\n", "13 Video Games \n", "14 I enjoy biking and hiking outdoors, and readin...\n", "15 Basketball\n", "16 Mountain biking\n", "17 Reading\n", "18 Archery. Painting. Reading. \n", "19 I really love baseball. My parents and I watch...\n", "20 Reading\n", "21 Hiking\n", "22 rock climbing\n", "23 Golf and Swiming\n", "24 Lifting!\n", "25 I enjoy working out, hanging out with friends,..." ] }, "execution_count": 94, "metadata": {}, "output_type": "execute_result" } ], "source": [ "hob = pd.read_csv(\"https://facultyweb.cs.wwu.edu/~wehrwes/courses/data311_25f/lectures/L09/hobbies.csv\", header=None)\n", "hob" ] }, { "cell_type": "markdown", "id": "c9a9867d-d1c0-471c-a946-e47272ba4654", "metadata": {}, "source": [ "#### What I'd like to do:" ] }, { "cell_type": "code", "execution_count": 95, "id": "cc738b49-1704-4641-95f5-faf6986c73db", "metadata": {}, "outputs": [ { "ename": "TypeError", "evalue": "no numeric data to plot", "output_type": "error", "traceback": [ "\u001b[31m---------------------------------------------------------------------------\u001b[39m", "\u001b[31mTypeError\u001b[39m Traceback (most recent call last)", "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[95]\u001b[39m\u001b[32m, line 1\u001b[39m\n\u001b[32m----> \u001b[39m\u001b[32m1\u001b[39m hob[\u001b[32m0\u001b[39m].plot.hist()\n", "\u001b[36mFile \u001b[39m\u001b[32m/opt/miniforge/lib/python3.12/site-packages/pandas/plotting/_core.py:1694\u001b[39m, in \u001b[36mPlotAccessor.hist\u001b[39m\u001b[34m(self, by, bins, **kwargs)\u001b[39m\n\u001b[32m 1639\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mhist\u001b[39m(\n\u001b[32m 1640\u001b[39m \u001b[38;5;28mself\u001b[39m, by: IndexLabel | \u001b[38;5;28;01mNone\u001b[39;00m = \u001b[38;5;28;01mNone\u001b[39;00m, bins: \u001b[38;5;28mint\u001b[39m = \u001b[32m10\u001b[39m, **kwargs\n\u001b[32m 1641\u001b[39m ) -> PlotAccessor:\n\u001b[32m 1642\u001b[39m \u001b[38;5;250m \u001b[39m\u001b[33;03m\"\"\"\u001b[39;00m\n\u001b[32m 1643\u001b[39m \u001b[33;03m Draw one histogram of the DataFrame's columns.\u001b[39;00m\n\u001b[32m 1644\u001b[39m \n\u001b[32m (...)\u001b[39m\u001b[32m 1692\u001b[39m \u001b[33;03m >>> ax = df.plot.hist(column=[\"age\"], by=\"gender\", figsize=(10, 8))\u001b[39;00m\n\u001b[32m 1693\u001b[39m \u001b[33;03m \"\"\"\u001b[39;00m\n\u001b[32m-> \u001b[39m\u001b[32m1694\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mkind\u001b[49m\u001b[43m=\u001b[49m\u001b[33;43m\"\u001b[39;49m\u001b[33;43mhist\u001b[39;49m\u001b[33;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mby\u001b[49m\u001b[43m=\u001b[49m\u001b[43mby\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbins\u001b[49m\u001b[43m=\u001b[49m\u001b[43mbins\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n", "\u001b[36mFile \u001b[39m\u001b[32m/opt/miniforge/lib/python3.12/site-packages/pandas/plotting/_core.py:1185\u001b[39m, in \u001b[36mPlotAccessor.__call__\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m 1182\u001b[39m label_name = label_kw \u001b[38;5;129;01mor\u001b[39;00m data.columns\n\u001b[32m 1183\u001b[39m data.columns = label_name\n\u001b[32m-> \u001b[39m\u001b[32m1185\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mplot_backend\u001b[49m\u001b[43m.\u001b[49m\u001b[43mplot\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdata\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mkind\u001b[49m\u001b[43m=\u001b[49m\u001b[43mkind\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n", "\u001b[36mFile \u001b[39m\u001b[32m/opt/miniforge/lib/python3.12/site-packages/pandas/plotting/_matplotlib/__init__.py:71\u001b[39m, in \u001b[36mplot\u001b[39m\u001b[34m(data, kind, **kwargs)\u001b[39m\n\u001b[32m 69\u001b[39m kwargs[\u001b[33m\"\u001b[39m\u001b[33max\u001b[39m\u001b[33m\"\u001b[39m] = \u001b[38;5;28mgetattr\u001b[39m(ax, \u001b[33m\"\u001b[39m\u001b[33mleft_ax\u001b[39m\u001b[33m\"\u001b[39m, ax)\n\u001b[32m 70\u001b[39m plot_obj = PLOT_CLASSES[kind](data, **kwargs)\n\u001b[32m---> \u001b[39m\u001b[32m71\u001b[39m \u001b[43mplot_obj\u001b[49m\u001b[43m.\u001b[49m\u001b[43mgenerate\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m 72\u001b[39m plt.draw_if_interactive()\n\u001b[32m 73\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m plot_obj.result\n", "\u001b[36mFile \u001b[39m\u001b[32m/opt/miniforge/lib/python3.12/site-packages/pandas/plotting/_matplotlib/core.py:516\u001b[39m, in \u001b[36mMPLPlot.generate\u001b[39m\u001b[34m(self)\u001b[39m\n\u001b[32m 514\u001b[39m \u001b[38;5;129m@final\u001b[39m\n\u001b[32m 515\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mgenerate\u001b[39m(\u001b[38;5;28mself\u001b[39m) -> \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[32m--> \u001b[39m\u001b[32m516\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_compute_plot_data\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m 517\u001b[39m fig = \u001b[38;5;28mself\u001b[39m.fig\n\u001b[32m 518\u001b[39m \u001b[38;5;28mself\u001b[39m._make_plot(fig)\n", "\u001b[36mFile \u001b[39m\u001b[32m/opt/miniforge/lib/python3.12/site-packages/pandas/plotting/_matplotlib/core.py:716\u001b[39m, in \u001b[36mMPLPlot._compute_plot_data\u001b[39m\u001b[34m(self)\u001b[39m\n\u001b[32m 714\u001b[39m \u001b[38;5;66;03m# no non-numeric frames or series allowed\u001b[39;00m\n\u001b[32m 715\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m is_empty:\n\u001b[32m--> \u001b[39m\u001b[32m716\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mTypeError\u001b[39;00m(\u001b[33m\"\u001b[39m\u001b[33mno numeric data to plot\u001b[39m\u001b[33m\"\u001b[39m)\n\u001b[32m 718\u001b[39m \u001b[38;5;28mself\u001b[39m.data = numeric_data.apply(\u001b[38;5;28mtype\u001b[39m(\u001b[38;5;28mself\u001b[39m)._convert_to_ndarray)\n", "\u001b[31mTypeError\u001b[39m: no numeric data to plot" ] } ], "source": [ "hob[0].plot.hist()" ] }, { "cell_type": "markdown", "id": "099bcadb-cd13-499a-8e87-314a972221e8", "metadata": {}, "source": [ "## Text Normalization" ] }, { "cell_type": "markdown", "id": "bf9085a0-5655-4e53-b0b5-af55ba6de114", "metadata": { "id": "9rxAbh4yTjHz" }, "source": [ "**Text normalization**: transforming text into standard or canonical forms.\n", "\n", "Often needed to convert text data into tabular data.\n", "\n", "Tools for text normalization?\n", "- Built-in Python string processin functions (e.g., `strip`, `lower`)\n", "- Python regular expressions (e.g., find and replace)\n", "- Linux commandline tool `sed` (stream editor) or `tr` (translate)\n", "- NLP toolkits; e.g., `spacy`, `nltk` (support tokenizing, stemming, lemmatizing, etc.)\n", "\n", "As a rule, natural language presents many challenges for seeming simple tasks." ] }, { "cell_type": "markdown", "id": "1f824a98-170d-40b8-a85f-1a498002434b", "metadata": {}, "source": [ "**String functions - quick demo:**" ] }, { "cell_type": "code", "execution_count": 96, "id": "7cc7dd71-a895-42e7-aaeb-466726cf9274", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['8', '6', '12', '5', '4']" ] }, "execution_count": 96, "metadata": {}, "output_type": "execute_result" } ], "source": [ "hours_responses = [\n", " \"8\",\n", " \"6.5\",\n", " \"12 hours\",\n", " \"5h\",\n", " \" 4 \"\n", "]\n", "\n", "import re\n", "\n", "def normalize(s):\n", " s = re.search(\"\\d+\", s)\n", " return s[0]\n", "\n", "[normalize(s) for s in hours_responses]" ] }, { "cell_type": "code", "execution_count": 97, "id": "2ed46f0c-eba0-44ce-930e-6b83eee3da02", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "I\n", "enjoy\n", "working\n", "out,\n", "hanging\n", "out\n", "with\n", "friends,\n", "and\n", "walking.\n" ] } ], "source": [ "words = hob.iloc[-1].item().split()\n", "print(\"\\n\".join(words))" ] }, { "cell_type": "code", "execution_count": 98, "id": "fb773851-ae0e-4206-9705-e7d4ebf2d54c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "i\n", "enjoy\n", "working\n", "out\n", "hanging\n", "out\n", "with\n", "friends\n", "and\n", "walking\n" ] } ], "source": [ "words = [w.strip(\"., \").lower() for w in words]\n", "print(\"\\n\".join(words))" ] }, { "cell_type": "markdown", "id": "6063741a-e3a4-453d-b1d9-4d6d6aa4853e", "metadata": { "id": "9rxAbh4yTjHz" }, "source": [ "#### Tokenization\n", "\n", "Roughly defined: splitting a string into linguistically meaningful pieces.\n", "\n", "For each of the following, think of an example piece of text where the naive approach does not give the desired result.\n", "\n", "Word tokenization: break into word-ish pieces\n", " - Naive: split on spaces. `str.split(' ')`" ] }, { "cell_type": "code", "execution_count": null, "id": "34cf05ad-1f57-4382-aa78-ab0a38fef985", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "f2683ea4-57d1-4c30-aa8e-8787d82d9ee0", "metadata": { "id": "9rxAbh4yTjHz" }, "source": [ " - **Failure cases**:\n", " - \" The paragraph starts here.\"\n", " - \"The paragraphstarts here.\"\n", " - The pargraph starts here---yet it's not over yet!\"\n", " - \"I like rock-climbing.\"" ] }, { "cell_type": "markdown", "id": "a52f7876-da70-40e0-9f60-0839d3c37fa9", "metadata": { "id": "9rxAbh4yTjHz" }, "source": [ "- **Sentence tokenization**: breaking up paragraphs (or larger) into sentences.\n", " - Naive: split on periods. `str.split('. ')`" ] }, { "cell_type": "code", "execution_count": null, "id": "a1826123-303e-4205-9063-c6203c4fba53", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "fa4d3ea6-30bf-48db-944e-b017095beed6", "metadata": { "id": "9rxAbh4yTjHz" }, "source": [ " - **Failure cases**:\n", " - \"Who thought this was a good idea? I didn't.\"\n", " - \"Oh no...\"\n", " - \"Mr. Rogers is the coolest.\"\n", " - \"J.R.R. Tolkien wrote some stuff.\"\n", " - The sentence ends with \"a quote.\"" ] }, { "cell_type": "markdown", "id": "ae1f3e3e-52f6-4a83-aad8-29380daa392b", "metadata": { "id": "9rxAbh4yTjHz" }, "source": [ "- **Lowercasing**\n", " - Naive: `str.lower()`" ] }, { "cell_type": "code", "execution_count": null, "id": "41e07d99-9375-4f66-a2a5-7ed24e1054fd", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "9rxAbh4yTjHz", "metadata": { "id": "9rxAbh4yTjHz" }, "source": [ " - **Failure cases**:\n", " - Joy vs joy, River vs river\n", " - \"I have a 20 MBps internet connection.\"\n", " - lol vs LOL\n", " - PIN vs pin" ] }, { "cell_type": "markdown", "id": "DfnAMn841PYP", "metadata": { "id": "DfnAMn841PYP" }, "source": [ "### Other Text Normalization Operations\n", "\n", "- **Expand contractions**; instead of tokenizing, you could preprocess these away\n", " - the tokenizer expanded `doesn't` to `does, n't`\n", " - Could instead first expand it to \"does not\"\n", "- **Canonicalize language variants**; e.g., color vs colour\n", "- **Converting numerical representation** of words to numbers\n", " - \"two and a half\" -> 2.5\n", " - four million\" -> 4,000,000 or 4000000\n", "- **Stripping accents or unicode characters** \n", " - E.g., résumé to resume.\n" ] }, { "cell_type": "markdown", "id": "b10a3f10-b4a9-488f-932a-aaf1797b1cf0", "metadata": {}, "source": [ "### Tokenization - Example:" ] }, { "cell_type": "code", "execution_count": 100, "id": "bd7ecd60-b9cf-4f6c-a602-75bc21fa72d7", "metadata": {}, "outputs": [], "source": [ "import spacy\n", "\n", "nlp = spacy.load('en_core_web_sm')" ] }, { "cell_type": "code", "execution_count": 103, "id": "8ebd6be7-fabb-4981-8961-2c49016fa7c3", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Dr.,\n", " Wehrwein,\n", " does,\n", " n't,\n", " work,\n", " for,\n", " the,\n", " F.B.I.,\n", " His,\n", " résumé,\n", " would,\n", " n't,\n", " qualify,\n", " him,\n", " for,\n", " such,\n", " a,\n", " job,\n", " .]" ] }, "execution_count": 103, "metadata": {}, "output_type": "execute_result" } ], "source": [ "\n", "text = \"Dr. Wehrwein doesn't work for the F.B.I. His résumé wouldn't qualify him for such a job.\"\n", "\n", "ans = nlp(text)\n", "list(ans)" ] }, { "cell_type": "code", "execution_count": 104, "id": "ff3a355c-9954-42e7-8a4c-36892e3ac8ea", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Dr. Wehrwein doesn't work for the F.B.I.,\n", " His résumé wouldn't qualify him for such a job.]" ] }, "execution_count": 104, "metadata": {}, "output_type": "execute_result" } ], "source": [ "list(ans.sents) # sentence tokenization" ] }, { "cell_type": "markdown", "id": "2832bd0e-6029-4235-a258-0f74ed0d5c30", "metadata": { "id": "vsogEjsk5xgi" }, "source": [ "- Stripping punctuation\n", " - If it isn't important for your task, could strip all punctuation out.\n", " - Beware of side effects; e.g., 192.168.1.1 -> 19216811." ] }, { "cell_type": "code", "execution_count": 105, "id": "95871115-c590-4c85-9d76-9d10efc84f1e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Localhost,\n", " is,\n", " 127.0.0.1,\n", " ,,\n", " whereas,\n", " your,\n", " home,\n", " router,\n", " is,\n", " traditionally,\n", " configured,\n", " to,\n", " be,\n", " 192.168.0.1,\n", " .]" ] }, "execution_count": 105, "metadata": {}, "output_type": "execute_result" } ], "source": [ "localhost = \"Localhost is 127.0.0.1, whereas your home router is traditionally configured to be 192.168.0.1.\"\n", "ans = nlp(localhost)\n", "list(ans)\n" ] }, { "cell_type": "code", "execution_count": 108, "id": "7295fc37-8938-42d9-9d70-74e52aff5202", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 108, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tok = list(ans)\n", "tok[-1].is_punct" ] }, { "cell_type": "markdown", "id": "vsogEjsk5xgi", "metadata": { "id": "vsogEjsk5xgi" }, "source": [ "- Removing stopwords\n", " - Stopwords: common function words like \"to\" \"in\" \"the\"\n", " - For some tasks they aren't important or relevant \n", " - E.g., topic detection" ] }, { "cell_type": "code", "execution_count": 109, "id": "059ddfc8-6751-4568-8459-427b51d2fd37", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Localhost, 127.0.0.1, home, router, traditionally, configured, 192.168.0.1]" ] }, "execution_count": 109, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tok = [t for t in ans if (not t.is_stop and not t.is_punct)]\n", "tok" ] }, { "cell_type": "code", "execution_count": null, "id": "b09f757a-6700-4e53-bc23-2e83f0da57c6", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "YQvpojcX1bO5", "metadata": { "id": "YQvpojcX1bO5" }, "source": [ "\n", "#### Stemming\n", "Convert words to word stem (even if the stem itself isn't a whole word). \n", "- E.g., argue, argued, argues, arguing all replaced by argu.\n", "- Works without knowing the part of speech.\n", "\n", "#### Lemmatization\n", "Like stemming, but attempts to infer part of speech and use custom rules based on part of speech.\n", "\n", "#### Part-of-speech tagging" ] }, { "cell_type": "code", "execution_count": 127, "id": "9672e84c-5b19-4eaa-9e20-bd45da760355", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'I love riding my bike, recently I have been enjoying riding my dirt jumper (type of bike made for dirt jumps) to the bike park at the Civic sport complex area.\\xa0'" ] }, "execution_count": 127, "metadata": {}, "output_type": "execute_result" } ], "source": [ "hobby = hob.iloc[0,0]\n", "hobby" ] }, { "cell_type": "code", "execution_count": 128, "id": "2cbhbcLB2oo0", "metadata": { "id": "2cbhbcLB2oo0" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TokenLemmaPOS
0loveloveVERB
1ridingridVERB
2bikebikeNOUN
3recentlyrecentlyADV
4enjoyingenjoyVERB
5ridingridVERB
6dirtdirtNOUN
7jumperjumperNOUN
8typetypeNOUN
9bikebikeNOUN
10dirtdirtNOUN
11jumpsjumpNOUN
12bikebikeNOUN
13parkparkNOUN
14CivicCivicPROPN
15sportsportNOUN
16complexcomplexADJ
17areaareaNOUN
18SPACE
\n", "
" ], "text/plain": [ " Token Lemma POS\n", "0 love love VERB\n", "1 riding rid VERB\n", "2 bike bike NOUN\n", "3 recently recently ADV\n", "4 enjoying enjoy VERB\n", "5 riding rid VERB\n", "6 dirt dirt NOUN\n", "7 jumper jumper NOUN\n", "8 type type NOUN\n", "9 bike bike NOUN\n", "10 dirt dirt NOUN\n", "11 jumps jump NOUN\n", "12 bike bike NOUN\n", "13 park park NOUN\n", "14 Civic Civic PROPN\n", "15 sport sport NOUN\n", "16 complex complex ADJ\n", "17 area area NOUN\n", "18     SPACE" ] }, "execution_count": 128, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ans = nlp(hobby)\n", "tok = [t for t in ans if (not t.is_stop and not t.is_punct)]\n", "pd.DataFrame({\"Token\" : [t for t in tok],\n", " \"Lemma\" : [t.lemma_ for t in tok], \n", " \"POS\" : [t.pos_ for t in tok]})" ] }, { "cell_type": "code", "execution_count": 123, "id": "0ba9d426-b159-4755-9867-3ef9e7b9961d", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'VERB'" ] }, "execution_count": 123, "metadata": {}, "output_type": "execute_result" } ], "source": [ "text = \"I got rid of my shoes and went riding.\"\n", "riding = nlp(text)\n", "riding[2].lemma_" ] }, { "cell_type": "code", "execution_count": 124, "id": "453c20b4-3d1a-4d19-8881-0c48ab335a01", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'rid'" ] }, "execution_count": 124, "metadata": {}, "output_type": "execute_result" } ], "source": [ "riding[-2].lemma_" ] }, { "cell_type": "code", "execution_count": 125, "id": "b3beae22-e931-42c3-9463-6fcd5e55aaa4", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "False" ] }, "execution_count": 125, "metadata": {}, "output_type": "execute_result" } ], "source": [ "riding[-2] == riding[2]" ] }, { "cell_type": "markdown", "id": "-Td4g0l76XBJ", "metadata": { "id": "-Td4g0l76XBJ" }, "source": [ "#### Noun phrase parsing" ] }, { "cell_type": "code", "execution_count": 129, "id": "CFAADOsv6dAP", "metadata": { "id": "CFAADOsv6dAP" }, "outputs": [ { "data": { "text/plain": [ "[I,\n", " my bike,\n", " I,\n", " my dirt jumper,\n", " type,\n", " bike,\n", " dirt jumps,\n", " the bike park,\n", " the Civic sport complex area]" ] }, "execution_count": 129, "metadata": {}, "output_type": "execute_result" } ], "source": [ "list(ans.noun_chunks)" ] }, { "cell_type": "markdown", "id": "_F_gGlFi6t6N", "metadata": { "id": "_F_gGlFi6t6N" }, "source": [ "#### Named entity recognition" ] }, { "cell_type": "code", "execution_count": 130, "id": "8b565d1f-a38b-4095-8b86-c8e0ff936559", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(Civic,)" ] }, "execution_count": 130, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ans.ents" ] }, { "cell_type": "code", "execution_count": 131, "id": "4dca7414-692b-45e7-b11f-d13d41a6ef2e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(Jude Law, New York City, Air Force One, JFK)" ] }, "execution_count": 131, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nlp(\"Jude Law visited New York City. Air Force One happened to be parked at JFK.\").ents" ] }, { "cell_type": "code", "execution_count": 132, "id": "98f0438d-6eed-4f62-836a-6d8615611805", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(new york city, air force one, jfk)" ] }, "execution_count": 132, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nlp(\"Jude Law visited New York City. Air Force One happened to be parked at JFK.\".lower()).ents" ] }, { "cell_type": "code", "execution_count": 133, "id": "6631e428-34ae-49d2-9122-9b19f30f7bab", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(big bird, new york city, air force one, jfk)" ] }, "execution_count": 133, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nlp(\"Big Bird visited New York City. Air Force One happened to be parked at JFK.\".lower()).ents" ] }, { "cell_type": "markdown", "id": "DrAMaLRa7Vky", "metadata": { "id": "DrAMaLRa7Vky" }, "source": [ "#### Sentiment analysis" ] }, { "cell_type": "code", "execution_count": 134, "id": "d3f267cf-28db-4fbc-8e14-e87501b8e834", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 134, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from spacytextblob.spacytextblob import SpacyTextBlob\n", "nlp.add_pipe(\"spacytextblob\")\n" ] }, { "cell_type": "code", "execution_count": 135, "id": "70b56b33-06a3-4569-bb4d-3a7bf779ebad", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.7\n", "-0.5\n" ] } ], "source": [ "yay = nlp(\"Today is a good day.\")\n", "boo = nlp(\"I'm feeling sad.\")\n", "\n", "print(yay._.blob.polarity)\n", "print(boo._.blob.polarity)" ] }, { "cell_type": "code", "execution_count": 136, "id": "be73f4c4-a22d-424a-8c3b-303c7487889c", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAHpCAYAAACmzsSXAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjgsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvwVt1zgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAI7VJREFUeJzt3QtUVWX+//EvhiI6iOGltEC0EjQvWN4qR6W8ZnabmTIvOZqNXcyUxpJl/tSmIptJnSI1XWlWQ7qazJxqvFQqmZdEJcu8lgrLNPIG4gUV9n89z1qHvygwgnDO95z9fq31rHPOvhyevc+BD8/ez95PkOM4jgAAAHWq+LoCAACgeIQ0AABKEdIAAChFSAMAoBQhDQCAUoQ0AABKEdIAACgV8CFtLgPPycmxjwAA+JOAD+njx49LeHi4fQQAwJ8EfEgDAOCvCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAJQK9nUF4F8yMjLk0KFD4lZ169aVqKgoX1cDgEsQ0ihTQMfGNpNTp066dq+FhtaQ7du3EdQAvIKQxiUzLWgT0B2GTpBaDaJdt+dyDuyV9XMm2f1AaxqANxDSKDMT0BFRMew5AKhkdBwDAEApQhoAAKUIaQAAlCKkAQBQipAGAEApQhoAAKUIaQAAlCKkAQBQipAGAEApQhoAAKUIaQAAlCKkAQBQipAGAEApQhoAAKUIaQAAlCKkAQBQipAGAEApQhoAAKV8GtKpqanSt29fadiwoQQFBcmiRYsuWmbbtm1y9913S3h4uISFhUnHjh0lIyPDJ/UFAMA1IX3ixAlp3bq1JCcnFzv/p59+kk6dOklsbKysXLlSvvvuOxk/frxUr17d63UFAMDbgsWHevfubUtJxo0bJ3feeae8+uqrhdOaNGlS6nvm5eXZ4pGTk1NBtQUAwLvUnpMuKCiQzz77TJo2bSo9e/aU+vXrS4cOHYo9JH6+pKQke2jcUyIjI71WZwAAXBHSWVlZkpubK6+88or06tVLli1bJvfdd5/cf//9smrVqhLXS0xMlOzs7MKSmZnp1XoDABAQh7v/V0vauOeee2T06NH2eVxcnKxZs0ZmzpwpXbp0KXa9kJAQWwAA8HdqW9J169aV4OBgad68eZHpzZo1o3c3AMAV1IZ0tWrVpF27drJjx44i03fu3CmNGjXyWb0AAHDF4W5zznn37t2Fr/fs2SPp6ekSEREhUVFRMmbMGHnwwQelc+fOEh8fL0uWLJH//Oc/9nIsAAACnU9DOi0tzYavR0JCgn0cPHiwvPPOO7ajmDn/bHpsjxw5UmJiYuSjjz6y104DABDofBrSXbt2FcdxSl1m6NChtgAA4DZqz0kDAOB2hDQAAEoR0gAAKEVIAwCgFCENAIBShDQAAEoR0gAAKEVIAwCgFCENAIBShDQAAEoR0gAAKEVIAwCgFCENAIBShDQAAEoR0gAAKEVIAwCgFCENAIBShDQAAEoR0gAAKEVIAwCgFCENAIBShDQAAEoR0gAAKEVIAwCgFCENAIBShDQAAEoR0gAAKEVIAwCgFCENAIBShDQAAEoR0gAAKEVIAwCgFCENAIBShDQAAEoR0gAAKEVIAwCgFCENAIBSPg3p1NRU6du3rzRs2FCCgoJk0aJFJS47fPhwu8y0adO8WkcAAFwZ0idOnJDWrVtLcnJyqcuZ8F6/fr0NcwAA3CLYlz+8d+/etpRm//79MmLECFm6dKn06dPnf75nXl6eLR45OTkVUlcAALxN9TnpgoICGTRokIwZM0ZuvPHGS1onKSlJwsPDC0tkZGSl1xMAANeF9OTJkyU4OFhGjhx5yeskJiZKdnZ2YcnMzKzUOgIAEJCHu0uzceNG+ec//ymbNm2yHcYuVUhIiC0AAPg7tS3pr7/+WrKysiQqKsq2pk3Zt2+fPPPMMxIdHe3r6gEA4N6WtDkX3a1btyLTevbsaacPGTLEZ/UCAMAVIZ2bmyu7d+8ufL1nzx5JT0+XiIgI24KuU6dOkeWrVq0qV199tcTExPigtgAAuCik09LSJD4+vvB1QkKCfRw8eLC88847PqwZAAAuD+muXbuK4ziXvPzevXsrtT4AAGiituMYAABuR0gDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAASvk0pFNTU6Vv377SsGFDCQoKkkWLFhXOO3v2rDz33HPSsmVLqVmzpl3m4Ycfll9++cWXVQYAwB0hfeLECWndurUkJydfNO/kyZOyadMmGT9+vH1cuHCh7Ny5U+6++26f1BUAAG8LFh/q3bu3LcUJDw+X5cuXF5n2xhtvSPv27SUjI0OioqK8VEsAAFwY0mWVnZ1tD4vXrl27xGXy8vJs8cjJyfFS7QAAcGnHsdOnT8vYsWOlf//+UqtWrRKXS0pKsq1wT4mMjPRqPQEAcFVIm05k/fr1k4KCApk+fXqpyyYmJtoWt6dkZmZ6rZ4AALjqcLcJ6AceeED27NkjX331VamtaCMkJMQWAAD8XbA/BPSuXbtkxYoVUqdOHV9XCQAAd4R0bm6u7N69u/C1aS2np6dLRESEvS76j3/8o7386tNPP5X8/Hw5ePCgXc7Mr1atmg9rDgBAgId0WlqaxMfHF75OSEiwj4MHD5aJEyfK4sWL7eu4uLgi65lWddeuXb1cWwAAXBTSJmgdxylxfmnzAAAIdH7RuxsAADcipAEAUIqQBgBAKUIaAAClCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAJTyaUinpqZK3759pWHDhhIUFCSLFi0qMt9xHJk4caKdHxoaKl27dpWtW7f6rL4AALgmpE+cOCGtW7eW5OTkYue/+uqrMmXKFDt/w4YNcvXVV0v37t3l+PHjXq8rAADeFiw+1Lt3b1uKY1rR06ZNk3Hjxsn9999vp82bN0+uuuoqSUlJkeHDh3u5tgAAeJfac9J79uyRgwcPSo8ePQqnhYSESJcuXWTNmjUlrpeXlyc5OTlFCgAA/khtSJuANkzL+XzmtWdecZKSkiQ8PLywREZGVnpdAQBwVUh7mA5lFx4Gv3Da+RITEyU7O7uwZGZmeqGWAAAE2Dnp0phOYoZpNTdo0KBwelZW1kWt6/OZQ+KmAADg79S2pBs3bmyDevny5YXTzpw5I6tWrZJbb73Vp3UDACDgW9K5ubmye/fuIp3F0tPTJSIiQqKiomTUqFHy8ssvyw033GCLeV6jRg3p37+/L6sNAEDgh3RaWprEx8cXvk5ISLCPgwcPlnfeeUeeffZZOXXqlDzxxBNy9OhR6dChgyxbtkzCwsJ8WGsAAFwQ0uYOYqYjWElMBzFzxzFTAABwG7XnpAEAcLtyhXSTJk3k8OHDF00/duyYnQcAAHwU0nv37pX8/Pxi7/a1f//+CqgWAAAo0znpxYsXFz5funSpvaOXhwntL7/8UqKjo9mrAAB4O6Tvvffewg5dpgf2+apWrWoD+rXXXquIegEA4HplCumCgoLCG42YoSPr1q3r+h0IAICqS7DMTUcAAIDS66TN+WdTzL20PS1sjzlz5lRE3QAAcLVyhfSkSZPkhRdekLZt29rBL0oblQoAAHgxpGfOnGlv2zlo0KBy/lgAAFAp10mb0agYiQoAAIUhPWzYMElJSan42gAAgMs73H369GmZNWuWfPHFF9KqVSt7jfT5pkyZUp63BQAAlxvSW7Zskbi4OPv8hx9+KDKPTmQAAPgwpFesWFFBPx4AAJSEoSoBAAiklnR8fHyph7W/+uqry6kTAAAob0h7zkd7nD17VtLT0+356QsH3gAAAF4M6alTpxY7feLEiZKbm1vOqgAAgEo7Jz1w4EDu2w0AgMaQXrt2rVSvXr0i3xIAANcq1+Hu+++/v8hrx3HkwIEDkpaWJuPHj6+ougEA4GrlCunw8PAir6tUqSIxMTF2ZKwePXpUVN0AAHC1coX03LlzK74mAADg8kPaY+PGjbJt2zZ7zXTz5s2lTZs2l/N2AADgckM6KytL+vXrJytXrpTatWvbc9LZ2dn2Jifz58+XevXqledtAQDA5fbufuqppyQnJ0e2bt0qR44ckaNHj9obmZhpI0eOLM9bAgCAimhJL1myxA5T2axZs8Jp5nD3m2++SccxAAB82ZIuKCi4aAxpw0wz8wAAgI9C+vbbb5enn35afvnll8Jp+/fvl9GjR8sdd9xRAdUCAADlCunk5GQ5fvy4REdHy3XXXSfXX3+9NG7c2E5744032KsAAPjqnHRkZKRs2rRJli9fLtu3b7e9u8056W7dulVEnQAAQFkPd5txok0Ym17cRvfu3W1Pb9Oju127dnLjjTfK119/zY4FAMDbIT1t2jR59NFHpVatWsXeKnT48OEyZcqUiqgXAACuV6aQ/u6776RXr14lzjf37TZ3IQMAAF4O6V9//bXYS688goOD5bfffquAagEAgDKF9DXXXCPff/99ifO3bNkiDRo0YK8CAODtkL7zzjvl//7v/+T06dMXzTt16pRMmDBB7rrrLqko586dk+eff95e3hUaGipNmjSxw2FywxQAgBuU6RIsE5gLFy6Upk2byogRI+wY0mYELDMSlrklaH5+vowbN67CKjd58mSZOXOmzJs3z/YcT0tLkyFDhthOauZmKgAABLIyhfRVV10la9askccff1wSExPt9dGGCeqePXvK9OnT7TIVZe3atXLPPfdInz597Gtz85QPPvjAhnVJ8vLybPHwXC4GAEDA33GsUaNG8vnnn8uhQ4dk/fr1sm7dOvvcTDMhWpE6deokX375pezcubOwd/nq1avtYfeSJCUl2Za2p5gbrwAA4Jo7jhlXXnmlvYFJZXruuefsONWxsbFyxRVX2MPpL730kjz00EMlrmNa+AkJCUVa0gQ1AMBVIe0NCxYskPfff19SUlLsOen09HQZNWqUNGzYUAYPHlzsOiEhIbYAAODvVIf0mDFjZOzYsdKvXz/7umXLlrJv3z57SLukkAYAwNWjYHnLyZMnpUqVolU0h725BAsA4AaqW9J9+/a156CjoqLs4e7Nmzfbe4MPHTrU11UDAMDdIW3Gph4/frw88cQTkpWVZc9Fm0E8zA1VAAAIdKpDOiwszI68ZQoAAG6j+pw0AABuRkgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAopT6k9+/fLwMHDpQ6depIjRo1JC4uTjZu3OjragEAUOmCRbGjR4/KbbfdJvHx8fLf//5X6tevLz/99JPUrl3b11UDAMDdIT158mSJjIyUuXPnFk6Ljo4udZ28vDxbPHJyciq1jgAAuPJw9+LFi6Vt27bypz/9ybai27RpI7Nnzy51naSkJAkPDy8sJuQBAPBHqkP6559/lhkzZsgNN9wgS5culccee0xGjhwp7777bonrJCYmSnZ2dmHJzMz0ap0BAHDF4e6CggLbkn755Zfta9OS3rp1qw3uhx9+uNh1QkJCbAEAwN+pbkk3aNBAmjdvXmRas2bNJCMjw2d1AgDAW1SHtOnZvWPHjiLTdu7cKY0aNfJZnQAA8BbVIT169GhZt26dPdy9e/duSUlJkVmzZsmTTz7p66oBAODukG7Xrp18/PHH8sEHH0iLFi3kb3/7m0ybNk0GDBjg66oBAODujmPGXXfdZQsAAG6juiUNAICbEdIAAChFSAMAoBQhDQCAUoQ0AABKEdIAAChFSAMAoBQhDQCAUoQ0AABKEdIAAChFSAMAoBQhDQCAUoQ0AABKEdIAAChFSAMAoBQhDQCAUoQ0AABKEdIAAChFSAMAoBQhDQCAUoQ0AABKEdIAAChFSAMAoBQhDQCAUoQ0AABKEdIAAChFSAMAoBQhDQCAUoQ0AABKEdIAAChFSAMAoBQhDQCAUoQ0AABKEdIAAChFSAMAoBQhDQCAUoQ0AABK+VVIJyUlSVBQkIwaNcrXVQEAoNL5TUhv2LBBZs2aJa1atfJ1VQAA8Ipg8QO5ubkyYMAAmT17trz44oulLpuXl2eLR05OjhdqCADukJGRIYcOHRK3qlu3rkRFRXnt5/lFSD/55JPSp08f6dat2/8MaXNIfNKkSV6rGwC4KaBjY5vJqVMnxa1CQ2vI9u3bvBbU6kN6/vz5smnTJnu4+1IkJiZKQkJCkZZ0ZGRkJdYQANzBtKBNQHcYOkFqNYgWt8k5sFfWz5lk9wMhLSKZmZny9NNPy7Jly6R69eqXtBNDQkJsAQBUDhPQEVEx7F63t6Q3btwoWVlZcvPNNxdOy8/Pl9TUVElOTrbnnq+44gqf1hEAAFeG9B133CHff/99kWlDhgyR2NhYee655whoAEBAUx3SYWFh0qJFiyLTatasKXXq1LloOgAAgcZvrpMGAMBtVLeki7Ny5UpfVwEAAK+gJQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKCU6pBOSkqSdu3aSVhYmNSvX1/uvfde2bFjh6+rBQCAV6gO6VWrVsmTTz4p69atk+XLl8u5c+ekR48ecuLECV9XDQCAShcsii1ZsqTI67lz59oW9caNG6Vz584+qxcAAOL2kL5Qdna2fYyIiChxmby8PFs8cnJyvFI3wC0yMjLk0KFD4lZ169aVqKgoX1cDLuE3Ie04jiQkJEinTp2kRYsWpZ7HnjRpklfrBrgpoGNjm8mpUyfFrUJDa8j27dsIaniF34T0iBEjZMuWLbJ69epSl0tMTLRhfn5LOjIy0gs1BAKfaUGbgO4wdILUahAtbpNzYK+snzPJ7gda0/AGvwjpp556ShYvXiypqaly7bXXlrpsSEiILQAqjwnoiKgYdjHg5pA2h7hNQH/88ceycuVKady4sa+rBACA16gOaXP5VUpKinzyySf2WumDBw/a6eHh4RIaGurr6gEA4N7rpGfMmGF7dHft2lUaNGhQWBYsWODrqgEA4O6WtDncDQCAW6luSQMA4GaENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKAUIQ0AgFKENAAAShHSAAAoRUgDAKBUsK8r4G8yMjLk0KFD4kbbtm3zdRVUcPN+cPO2n8+t+8Gt2+1LhHQZAzo2tpmcOnVS3Oxs3hlxo1PZh0UkSAYOHChux3fA3d8Bt37+vkBIl4FpQZuA7jB0gtRqEC1uc+D7tfLD4lly7tw5caOzJ4+LiCNx/Z+Teo1jxY34Drj7O+D2z98XCOlyMAEdERUjbpNzYK+vq6DC7+pHufLzN/gOuPs7wOfvfXQcAwBAKUIaAAClCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAJQipAEAUIqQBgBAKb8I6enTp0vjxo2levXqcvPNN8vXX3/t6yoBAFDp1If0ggULZNSoUTJu3DjZvHmz/P73v5fevXtLRkaGr6sGAIC7Q3rKlCnyyCOPyLBhw6RZs2Yybdo0iYyMlBkzZvi6agAAVKpgUezMmTOyceNGGTt2bJHpPXr0kDVr1hS7Tl5eni0e2dnZ9jEnJ+ey65Obm2sfj+zbIefyTonb5BzYZx+z9++SqsFB4jZu337D7fuA7Xf5538wozALKiJTjLCwMAkKKmVfOort37/fMVX85ptvikx/6aWXnKZNmxa7zoQJE+w6FPYB3wG+A3wH+A6I8n2QnZ1dag6qbkl7XPhfhuM4Jf7nkZiYKAkJCYWvCwoK5MiRI1KnTp3S/1tRzPzHZg7xZ2ZmSq1atSSQsa2Bic81cPHZXn5LujSqQ7pu3bpyxRVXyMGDB4tMz8rKkquuuqrYdUJCQmw5X+3atSUQmIAO9JD2YFsDE59r4OKzdWHHsWrVqtlLrpYvX15kunl96623+qxeAAB4g+qWtGEOXQ8aNEjatm0rt9xyi8yaNctefvXYY4/5umoAALg7pB988EE5fPiwvPDCC3LgwAFp0aKFfP7559KoUSNxC3P4fsKECRcdxg9EbGtg4nMNXHy2lSvI9B6r5J8BAAAC7Zw0AABuRkgDAKAUIQ0AgFKENAAAShHSCh09etRedhYeHm6LeX7s2LFS11m4cKH07NnT3gDG3FktPT1dAmXo0VWrVtnlzPJNmjSRmTNnir8oy7aaqxf69+8vMTExUqVKFTv6mz8py7aa72v37t2lXr169iYY5vLKpUuXSiBu6+rVq+W2226zdz0MDQ2V2NhYmTp1qviL8g4V/M0330hwcLDExcWJP5lehu1duXKl/Xt7Ydm+fXvFVagi77WNitGrVy+nRYsWzpo1a2wxz++6665S13n33XedSZMmObNnz7b3g928ebPKj2P+/PlO1apVbT1//PFH5+mnn3Zq1qzp7Nu3r9jlf/75Z6dGjRp2ObO8Wc+s/+9//9vRrqzbumfPHmfkyJHOvHnznLi4OLu8vyjrtpr5kydPdr799ltn586dTmJiol1/06ZNTqBtq9mmlJQU54cffrCf8XvvvWe/02+99ZYTaNvqcezYMadJkyZOjx49nNatWzv+Yn4Zt3fFihX27+2OHTucAwcOFJZz585VWJ0IaWXMF8N86OvWrSuctnbtWjtt+/bt/3N980dAc0i3b9/eeeyxx4pMi42NdcaOHVvs8s8++6ydf77hw4c7HTt2dLQr67aer0uXLn4V0pezrR7Nmze3/2i6YVvvu+8+Z+DAgU6gbuuDDz7oPP/883bAI38K6fZl3F5PSB89erTS6sThbmXWrl1rD3F36NChcFrHjh3ttJKG5/QXnqFHzVCjlzr0qNkfFy5vDuunpaXJ2bNnJZC21V9VxLaagXCOHz8uEREREujbunnzZrtsly5dJBC3de7cufLTTz/ZGzD5kzOX8dm2adNGGjRoIHfccYesWLHCXXcccxszmEj9+vUvmm6mXTjQiL85dOiQ5OfnXzQ4inld0raZ6cUtf+7cOft+5hcjULbVX1XEtr722mty4sQJeeCBByRQt/Xaa6+V3377zX53J06cKMOGDZNA29Zdu3bJ2LFj7Xlccz7anxwqx/aavz/mVtXm3HVeXp689957NqjNuerOnTtXSL38ay/6MfNLOWnSpFKX2bBhg30sbkjN0obn9DdlGXq0pOWLmx4I2+rPyrutH3zwgf39+OSTT4r9BzVQttUEV25urqxbt84G2fXXXy8PPfSQBMq2moAzHR/N37mmTZuKvwoqw2drOnma4mE6QJohhf/xj38Q0v5mxIgR0q9fv1KXiY6Oli1btsivv/560TzzH3hJw3P6i/IMPXr11VcXu7z5L930lg2kbfVXl7OtCxYskEceeUQ+/PBD6datmwTytpoew0bLli3t77j5x0RzSJd1W83pCnMayhzON3/vPKcxTMiZ39dly5bJ7bffLoH+O9uxY0d5//33K6xenJP24hfAXHpRWjFd/s1/YtnZ2fLtt98Wrrt+/Xo7zd+H5yzP0KNmf1y4vPllN6OiVa1aVbRy0zCr5d1W04L+85//LCkpKdKnTx/xBxX1uZrgModHA2lbzaV033//vb3801PMaIWmpWmen9/PJpA/282bN1fsabhK65KGy7oEq1WrVrZXtyktW7a86BKsmJgYZ+HChYWvDx8+bHt0f/bZZ7a3obmUwLw2lwNovMTh7bfftj3ZR40aZS9x2Lt3r51velEOGjTookuwRo8ebZc36/nbJViXuq2G+cxMufnmm53+/fvb51u3bnUCbVvNJUnBwcHOm2++WeTSFXPpTqBta3JysrN48WJ7qZkpc+bMcWrVquWMGzfOCcTv8Pn8rXf3/DJu79SpU52PP/7Yfq7mEjsz3/z9/eijjyqsToS0QiZwBwwY4ISFhdlinl/Yxd98EebOnVv42jw30y4s5pdEG/OHuVGjRk61atWcm266yVm1alXhvMGDB9vLj863cuVKp02bNnb56OhoZ8aMGY6/KOu2FvcZmvUDbVvN8+K21SwXaNv6+uuvOzfeeKP9Z9OEs/kuT58+3cnPz3cC8TvszyFd1u011/pfd911TvXq1Z0rr7zS6dSpk20oVSSGqgQAQCnOSQMAoBQhDQCAUoQ0AABKEdIAAChFSAMAoBQhDQCAUoQ0AABKEdIAAChFSAOwzPB6ZrSfY8eOXfYeMe+zaNEi9ixwmQhpIICYAStMQJpiBiBp0qSJ/PWvf7VjNXvTgQMHpHfv3vb53r17bX3MIAsAyobxpIEA06tXL5k7d66cPXvWjmE8bNgwG9IzZsyo9J995swZO5qQGWIUwOWjJQ0EmJCQEBuSkZGR0r9/fxkwYIA99GyGRhw5cqTUr1/fDovaqVMn2bBhQ4nvc/jwYTve8bXXXis1atSw4yCb4SXP17VrVzt2cEJCgh2OtXv37hcd7vaMo9ymTRs73ayTmppqW/oXjt37zDPPSOfOnSthrwD+iZAGAlxoaKhtVT/77LPy0Ucfybx582TTpk1y/fXXS8+ePeXIkSPFrnf69Gk7vu6nn34qP/zwg/zlL3+RQYMG2fHNz2feLzg4WL755ht56623Lnofz9joX3zxhT0MvnDhQhvE5lD8e++9V7jcuXPn5P3335chQ4ZU+D4A/BUhDQQwE5ApKSkSHx9vD3f//e9/t+eKmzdvLrNnz7YB/vbbbxe77jXXXGPPZ8fFxdlAfeqpp2yof/jhh0WWM2H/6quvSkxMjMTGxl70PvXq1bOPderUsS38iIgI+/qRRx6xh+U9PvvsMzl58qQ88MADFbwXAP9FSAMBxrR8f/e739lD2rfccotttZqANa3p2267rXA5c7i5ffv2sm3btmLfJz8/X1566SVp1aqVDVjznsuWLZOMjIwiy7Vt27bcndx2794t69ats6/nzJljA7pmzZrlej8gENFxDAgwnlazCeGGDRvax++++87OM+eEz+c4zkXTPF577TWZOnWqTJs2zZ6PNuE5atQo2znsfOUNVXNuvG/fvrY1bVrqn3/+ub0MDMD/R0saCDAmNM0h6EaNGtmANsxr0+t69erVhcuZlnVaWpo0a9as2PcxPcPvueceGThwoLRu3doG6a5du8pcH/NzPS3zC5me5/Pnz7fnsq+77roiLX0AhDTgmuB+/PHHZcyYMbJkyRL58ccf5dFHH7XngM254eKYYF++fLmsWbPGHhIfPnz4Rb2xL7XFbM59m5/766+/SnZ2duE8c447PDxcXnzxRTqMAcWgJQ24xCuvvCJ/+MMfbA/tm266yZ4PXrp0qVx55ZXFLj9+/Hi7nAlSc9mU6fR17733lvnnmp7fr7/+um0tm8PvpnXuUaVKFXtu2rSyH3744cvaPiAQBTnmpBQA+Ihp0ZsW9uLFi/kMgAvQcQyAT5jD3uZmKv/617/kk08+4VMAikFIA/AJc9jbXMdtznV77lQGoCgOdwMAoBQdxwAAUIqQBgBAKUIaAAClCGkAAJQipAEAUIqQBgBAKUIaAAClCGkAAESn/werEn+4fHQIjgAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "hob[\"Polarity\"] = hob[0].apply(lambda x: nlp(x)._.blob.polarity)\n", "\n", "sns.displot(data=hob, x=\"Polarity\");" ] }, { "cell_type": "code", "execution_count": 137, "id": "ec0730fb-4d64-4cbf-8321-4b7e6a67001b", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
0Polarity
0I love riding my bike, recently I have been en...0.175000
1I like hanging out with friends by going on wa...0.000000
2I love reading eastern fantasy/cultivation nov...0.500000
3Hiking!0.000000
4I love singing! It's technically in-school, bu...0.312500
5I like video games0.000000
6I manage a home media server in my downtime, i...0.500000
7Gaming0.000000
8I enjoy long distance running! I've been doing...0.171875
9Rock climbing0.000000
10Volleyball0.000000
11playing bass guitar-0.150000
12I like to play the guitar and to cook0.000000
13Video Games0.000000
14I enjoy biking and hiking outdoors, and readin...0.400000
15Basketball0.000000
16Mountain biking0.000000
17Reading0.000000
18Archery. Painting. Reading.0.000000
19I really love baseball. My parents and I watch...0.262500
20Reading0.000000
21Hiking0.000000
22rock climbing0.000000
23Golf and Swiming0.000000
24Lifting!0.000000
25I enjoy working out, hanging out with friends,...0.400000
\n", "
" ], "text/plain": [ " 0 Polarity\n", "0 I love riding my bike, recently I have been en... 0.175000\n", "1 I like hanging out with friends by going on wa... 0.000000\n", "2 I love reading eastern fantasy/cultivation nov... 0.500000\n", "3 Hiking! 0.000000\n", "4 I love singing! It's technically in-school, bu... 0.312500\n", "5 I like video games 0.000000\n", "6 I manage a home media server in my downtime, i... 0.500000\n", "7 Gaming 0.000000\n", "8 I enjoy long distance running! I've been doing... 0.171875\n", "9 Rock climbing  0.000000\n", "10 Volleyball 0.000000\n", "11 playing bass guitar -0.150000\n", "12 I like to play the guitar and to cook 0.000000\n", "13 Video Games  0.000000\n", "14 I enjoy biking and hiking outdoors, and readin... 0.400000\n", "15 Basketball 0.000000\n", "16 Mountain biking 0.000000\n", "17 Reading 0.000000\n", "18 Archery. Painting. Reading.  0.000000\n", "19 I really love baseball. My parents and I watch... 0.262500\n", "20 Reading 0.000000\n", "21 Hiking 0.000000\n", "22 rock climbing 0.000000\n", "23 Golf and Swiming 0.000000\n", "24 Lifting! 0.000000\n", "25 I enjoy working out, hanging out with friends,... 0.400000" ] }, "execution_count": 137, "metadata": {}, "output_type": "execute_result" } ], "source": [ "hob" ] }, { "cell_type": "code", "execution_count": null, "id": "1834f9eb-6a92-45f8-a826-51877f85e2b7", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.13" } }, "nbformat": 4, "nbformat_minor": 5 }