{ "cells": [ { "cell_type": "markdown", "id": "23319f0f-ad45-4135-b9bf-891881c152cd", "metadata": {}, "source": [ "# Lecture 21 - Multiclass Classification Example, end-to-end" ] }, { "attachments": {}, "cell_type": "markdown", "id": "3821aa77-6eb1-4f0c-8369-84ce46b2469b", "metadata": {}, "source": [ "#### Goals:\n", "\n", "* Cover no new concepts\n", "* Put a whole supervised learning system together end-to-end from scratch.\n", "* ![Image of the beans to be classified](beans.png)" ] }, { "cell_type": "markdown", "id": "b1a5fcd1-5b6a-4b8a-a49f-a4b2cc968ebc", "metadata": {}, "source": [ "**Our goal**: predict a dry bean's species given various geometric measures which have been computed from images of the bean via computer vision." ] }, { "cell_type": "code", "execution_count": 1, "id": "cbcf7404-b0be-44e4-9909-cd2412ab5875", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import seaborn as sns\n", "import matplotlib.pyplot as plt\n", "import sklearn" ] }, { "cell_type": "code", "execution_count": 2, "id": "778af6d5-6f8c-4582-805a-2f5d6bcf1fc8", "metadata": {}, "outputs": [], "source": [ "RANDOM_SEED = 42" ] }, { "cell_type": "markdown", "id": "7e72d66f-797c-4db8-88c2-790164066b7f", "metadata": {}, "source": [ "## 1: Load and Split the Data" ] }, { "cell_type": "code", "execution_count": 3, "id": "6981d2b2-1114-4d20-a86d-1cccdba907cb", "metadata": {}, "outputs": [], "source": [ "beans = pd.read_csv('/cluster/academic/DATA311/202620/drybean.csv')" ] }, { "cell_type": "code", "execution_count": 4, "id": "ce7a7b34-39cb-4427-b762-c9fa928c53b6", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
AreaPerimeterMajorAxisLengthMinorAxisLengthAspectRatioEccentricityConvexAreaEquivDiameterExtentSolidityRoundnessCompactnessShapeFactor1ShapeFactor2ShapeFactor3ShapeFactor4Class
028395610.291208.178117173.8887471.1971910.54981228715190.1410970.7639230.9888560.9580270.9133580.0073320.0031470.8342220.998724SEKER
128734638.018200.524796182.7344191.0973560.41178529172191.2727510.7839680.9849860.8870340.9538610.0069790.0035640.9098510.998430SEKER
229380624.110212.826130175.9311431.2097130.56272729690193.4109040.7781130.9895590.9478490.9087740.0072440.0030480.8258710.999066SEKER
330008645.884210.557999182.5165161.1536380.49861630724195.4670620.7826810.9766960.9039360.9283290.0070170.0032150.8617940.994199SEKER
430140620.134201.847882190.2792791.0607980.33368030417195.8965030.7730980.9908930.9848770.9705160.0066970.0036650.9419000.999166SEKER
......................................................
1360642097759.696288.721612185.9447051.5527280.76500242508231.5157990.7145740.9903310.9166030.8018650.0068580.0017490.6429880.998385DERMASON
1360742101757.499281.576392190.7131361.4764390.73570242494231.5267980.7999430.9907520.9220150.8222520.0066880.0018860.6760990.998219DERMASON
1360842139759.321281.539928191.1879791.4725820.73406542569231.6312610.7299320.9898990.9184240.8227300.0066810.0018880.6768840.996767DERMASON
1360942147763.779283.382636190.2757311.4893260.74105542667231.6532470.7053890.9878130.9079060.8174570.0067240.0018520.6682370.995222DERMASON
1361042159772.237295.142741182.2047161.6198410.78669342600231.6862230.7889620.9896480.8883800.7849970.0070010.0016400.6162210.998180DERMASON
\n", "

13611 rows × 17 columns

\n", "
" ], "text/plain": [ " Area Perimeter MajorAxisLength MinorAxisLength AspectRatio \\\n", "0 28395 610.291 208.178117 173.888747 1.197191 \n", "1 28734 638.018 200.524796 182.734419 1.097356 \n", "2 29380 624.110 212.826130 175.931143 1.209713 \n", "3 30008 645.884 210.557999 182.516516 1.153638 \n", "4 30140 620.134 201.847882 190.279279 1.060798 \n", "... ... ... ... ... ... \n", "13606 42097 759.696 288.721612 185.944705 1.552728 \n", "13607 42101 757.499 281.576392 190.713136 1.476439 \n", "13608 42139 759.321 281.539928 191.187979 1.472582 \n", "13609 42147 763.779 283.382636 190.275731 1.489326 \n", "13610 42159 772.237 295.142741 182.204716 1.619841 \n", "\n", " Eccentricity ConvexArea EquivDiameter Extent Solidity Roundness \\\n", "0 0.549812 28715 190.141097 0.763923 0.988856 0.958027 \n", "1 0.411785 29172 191.272751 0.783968 0.984986 0.887034 \n", "2 0.562727 29690 193.410904 0.778113 0.989559 0.947849 \n", "3 0.498616 30724 195.467062 0.782681 0.976696 0.903936 \n", "4 0.333680 30417 195.896503 0.773098 0.990893 0.984877 \n", "... ... ... ... ... ... ... \n", "13606 0.765002 42508 231.515799 0.714574 0.990331 0.916603 \n", "13607 0.735702 42494 231.526798 0.799943 0.990752 0.922015 \n", "13608 0.734065 42569 231.631261 0.729932 0.989899 0.918424 \n", "13609 0.741055 42667 231.653247 0.705389 0.987813 0.907906 \n", "13610 0.786693 42600 231.686223 0.788962 0.989648 0.888380 \n", "\n", " Compactness ShapeFactor1 ShapeFactor2 ShapeFactor3 ShapeFactor4 \\\n", "0 0.913358 0.007332 0.003147 0.834222 0.998724 \n", "1 0.953861 0.006979 0.003564 0.909851 0.998430 \n", "2 0.908774 0.007244 0.003048 0.825871 0.999066 \n", "3 0.928329 0.007017 0.003215 0.861794 0.994199 \n", "4 0.970516 0.006697 0.003665 0.941900 0.999166 \n", "... ... ... ... ... ... \n", "13606 0.801865 0.006858 0.001749 0.642988 0.998385 \n", "13607 0.822252 0.006688 0.001886 0.676099 0.998219 \n", "13608 0.822730 0.006681 0.001888 0.676884 0.996767 \n", "13609 0.817457 0.006724 0.001852 0.668237 0.995222 \n", "13610 0.784997 0.007001 0.001640 0.616221 0.998180 \n", "\n", " Class \n", "0 SEKER \n", "1 SEKER \n", "2 SEKER \n", "3 SEKER \n", "4 SEKER \n", "... ... \n", "13606 DERMASON \n", "13607 DERMASON \n", "13608 DERMASON \n", "13609 DERMASON \n", "13610 DERMASON \n", "\n", "[13611 rows x 17 columns]" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "beans" ] }, { "cell_type": "code", "execution_count": 5, "id": "13be69df-69be-4534-ad23-c23ed2bb14c6", "metadata": {}, "outputs": [], "source": [ "# separate features from labels\n", "features = beans.drop(columns=\"Class\")\n", "labels = beans[\"Class\"]" ] }, { "cell_type": "code", "execution_count": 6, "id": "169ab707-420a-4452-8454-03136e9ed3cd", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Class\n", "DERMASON 3546\n", "SIRA 2636\n", "SEKER 2027\n", "HOROZ 1928\n", "CALI 1630\n", "BARBUNYA 1322\n", "BOMBAY 522\n", "Name: count, dtype: int64" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "labels.value_counts()" ] }, { "cell_type": "code", "execution_count": 7, "id": "f5de8fbe-be21-4d69-b8e6-8ffa829e324a", "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "VAL_FRAC = 0.2\n", "TEST_FRAC = 0.2\n", "\n", "features_rest, features_test, labels_rest, labels_test = sklearn.model_selection.train_test_split(\n", " features, labels,\n", " test_size=TEST_FRAC,\n", " shuffle=True, # default\n", " stratify=labels,\n", " random_state=RANDOM_SEED,\n", ")\n", "\n", "features_train, features_val, labels_train, labels_val = sklearn.model_selection.train_test_split(\n", " features_rest, labels_rest,\n", " test_size = VAL_FRAC,\n", " shuffle=True,\n", " stratify=labels_rest,\n", " random_state=RANDOM_SEED,\n", ")" ] }, { "cell_type": "markdown", "id": "eac249e4-4311-452b-90c9-6d555b006d31", "metadata": {}, "source": [ "## 2: Baselines" ] }, { "cell_type": "code", "execution_count": 8, "id": "197d2869-11af-4518-b3ae-aa417828c736", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Class\n", "DERMASON 2269\n", "SIRA 1687\n", "SEKER 1297\n", "HOROZ 1234\n", "CALI 1043\n", "BARBUNYA 846\n", "BOMBAY 334\n", "Name: count, dtype: int64" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "labels_train.value_counts()" ] }, { "cell_type": "markdown", "id": "c5fef085-3f18-4df7-aba6-950e6555c6c4", "metadata": {}, "source": [ "If we predict the most common label (\"DERMASON\"), we'll get 2269 correct. Our accuracy will be:" ] }, { "cell_type": "code", "execution_count": 9, "id": "60f4527b-27fc-47cd-b283-71f6e36838ab", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "np.float64(0.26050516647531574)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "labels_train.value_counts().iloc[0] / labels_train.shape[0]" ] }, { "cell_type": "markdown", "id": "80467c38-0197-43cc-819f-ed54296fd5b9", "metadata": {}, "source": [ "I could also calculate random guessing (or random guessing with probabilities weighted by the the class balance. I don't expect any other baseline to beat the above." ] }, { "cell_type": "markdown", "id": "fdd61211-97ed-4673-a959-cd577e17c185", "metadata": {}, "source": [ "## 3. Feature Extraction" ] }, { "cell_type": "code", "execution_count": 31, "id": "1d745324-e9b3-4dec-ad09-956825b87640", "metadata": {}, "outputs": [], "source": [ "# Naively take features as they are:\n", "X_train = features_train.to_numpy()\n", "X_val = features_val.to_numpy()\n", "X_test = features_test.to_numpy()\n", "\n", "# Convert numerical features to z-scores:\n", "scaler = sklearn.preprocessing.StandardScaler().fit(features_train)\n", "X_train = scaler.transform(features_train)\n", "X_val = scaler.transform(features_val)\n", "X_test = scaler.transform(features_test)\n", "\n", "# Convert categorical labels to ordinal integers\n", "encoder = sklearn.preprocessing.LabelEncoder().fit(labels_train)\n", "y_train = encoder.transform(labels_train)\n", "y_val = encoder.transform(labels_val)\n", "y_test = encoder.transform(labels_test)\n" ] }, { "cell_type": "markdown", "id": "14981e30-c54d-4a8b-b54f-72ecfb95e2c6", "metadata": {}, "source": [ "## 4. Learn the Machine!" ] }, { "cell_type": "code", "execution_count": 42, "id": "6bef5851-e9f6-4979-867b-36001e6acfb4", "metadata": {}, "outputs": [], "source": [ "# Hyperparameters\n", "\n", "K = 15\n", "\n", "# simple metric choices: 'l1', 'l2'\n", "METRIC = 'l2'" ] }, { "cell_type": "code", "execution_count": 43, "id": "5829ff80-5065-49fb-a763-d08ad76b8b09", "metadata": {}, "outputs": [], "source": [ "from sklearn.neighbors import KNeighborsClassifier\n", "\n", "knn = KNeighborsClassifier(n_neighbors=K, metric=METRIC)\n", "\n", "knn.fit(X_train, y_train);" ] }, { "cell_type": "markdown", "id": "6e63567a-fbe4-4a9b-a78a-fa065e616b0e", "metadata": {}, "source": [ "## 5. Evaluate Performance" ] }, { "cell_type": "code", "execution_count": 44, "id": "14014b77-9d43-4556-82a9-3539bf948a26", "metadata": {}, "outputs": [], "source": [ "def evaluate(classifier, X_train, y_train, X_val, y_val):\n", " # Classification accuracy on training and validation sets:\n", "\n", " y_train_pred = knn.predict(X_train)\n", " y_val_pred = knn.predict(X_val)\n", "\n", " train_acc = (y_train_pred == y_train).sum() / y_train.shape[0]\n", " val_acc = (y_val_pred == y_val).sum() / y_val.shape[0]\n", " \n", " print(f\"Training accuracy: {train_acc:.4f} ({train_acc*100:.2f}%)\")\n", " print(f\"Validation accuracy: {val_acc:.4f} ({val_acc*100:.2f}%)\")" ] }, { "cell_type": "code", "execution_count": 45, "id": "533a412f-855d-48c0-a6a5-58efc1a348e7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training accuracy: 0.9323 (93.23%)\n", "Validation accuracy: 0.9311 (93.11%)\n" ] } ], "source": [ "evaluate(knn, X_train, y_train, X_val, y_val)" ] }, { "cell_type": "markdown", "id": "a9da3d5c-3db9-4cb3-83e7-ed350e38d538", "metadata": {}, "source": [ "## Where could we go from here?\n", "* Tune $K$\n", "* Try different distance metrics\n", "* Feature selection: does a subset of the features work better than using all of them?\n", "* Different models: go beyond KNN" ] }, { "cell_type": "code", "execution_count": 46, "id": "1c8a924e-df2a-4572-9c88-c0dacf0b4374", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(8710, 16)" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_train.shape" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.13" } }, "nbformat": 4, "nbformat_minor": 5 }