{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TP2 K Nearest Neigbors (KNN)\n",
    "\n",
    "\n",
    "### Table of Contents\n",
    "\n",
    "* [0. Warm-up](#chapter0)\n",
    "* [1. K-NN classification for  `Iris`](#chapter1)\n",
    "    * [1.1 Data Preparation](#section_1_1)\n",
    "    * [1.2 Writing a  K-NN classifier](#section_1_2)\n",
    "    * [1.3 `sklearn`'s K-NN classifier](#section_1_3)\n",
    "* [2. ($\\star$) Decision boundary for a  K-NN classifier](#chapter2)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Warm-up <a class=\"anchor\" id=\"chapter0\"></a>\n",
    "\n",
    "The table below  is a  dataset containing six observations, three covariates (features), and one dependent variable (label/target).\n",
    "\n",
    "\n",
    "| Observations | $X^1$ | $X^2$ |$X^3$ |$Y$ |\n",
    "| :-- | :-- | :-- | :-- | :-- |\n",
    "| 0 | 0 | 3 | 0 | Red |\n",
    "| 1 | 2 | 0 | 0| Red |\n",
    "| 2 | 0 | 1 | 3 | Red |\n",
    "| 3 | 0 | 1 | 2 | Green |\n",
    "| 4 | -1 | 0 | 1 | Green |\n",
    "| 5 | 1 | 1 | 1 | Red |\n",
    "\n",
    "We want to use this dataset to predict the color for a new data point $x = (x1, x2, x3) = (0, 0, 0)$ using a K-NN classifier. \n",
    "\n",
    "**Exercise 1** : answer the following questions. (You can do the calculation yourself or write a Python program.)\n",
    "\n",
    "(1) Calculate the euclidian  distance between each observation and  the point to be predicted $x = (x1, x2, x3) = (0, 0, 0)$.\n",
    "\n",
    "(2) What is the prediction when $K=1$ ?\n",
    "\n",
    "(2) What is the prediction when $K=3$ ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer for  Exercise 1**\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-19T11:00:08.826606Z",
     "start_time": "2025-03-19T11:00:08.811426Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:17.854652Z",
     "start_time": "2025-02-07T16:32:17.851755Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.0\n",
      "4.0\n",
      "10.000000000000002\n",
      "5.000000000000001\n",
      "2.0000000000000004\n",
      "2.9999999999999996\n",
      "The nearest neighbor is the observation [4]\n",
      "The predicted value is [4]\n",
      "The nearest neighbor is the observation [4 5 1]\n",
      "The predicted value is [4 5 1]\n"
     ]
    }
   ],
   "source": [
    "X0 = np.array([0, 0, 0])\n",
    "X1 = np.array([0, 3, 0])\n",
    "X2 = np.array([2, 0, 0])\n",
    "X3 = np.array([0, 1, 3])\n",
    "X4 = np.array([0, 1, 2])\n",
    "X5 = np.array([-1, 0, 1])\n",
    "X6 = np.array([1, 1, 1])\n",
    "X = np.array([X1, X2, X3, X4, X5, X6])\n",
    "\n",
    "distances = []\n",
    "for x in X:\n",
    "    dist = np.linalg.norm(x - X0) ** 2\n",
    "    distances.append(dist)\n",
    "    print(dist)\n",
    "\n",
    "for k in [1, 3]:\n",
    "    near = np.argsort(distances)[:k]\n",
    "    print(f\"The nearest neighbor is the observation {near}\")\n",
    "    print(f\"The predicted value is {near}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. K-NN classification for  `Iris`  <a class=\"anchor\" id=\"chapter1\"></a>"
   ]
  },
  {
   "attachments": {
    "Iris_setosa_versicolor_virginica.png": {
     "image/png": "[encoded data, 1068556 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this practical work, we will use a classic dataset called 'Iris' included in 'sklearn'.\n",
    "\n",
    "This dataset consists of 3 different types of irises (Setosa, Versicolor, and Virginica). The features are *sepal length*, *sepal width*, *petal length*, and *petal width*.\n",
    "\n",
    "![Iris_setosa_versicolor_virginica.png](attachment:Iris_setosa_versicolor_virginica.png)\n",
    "\n",
    "- The images of irises are taken from  https://fr.wikipedia.org/wiki/Iris_de_Fisher \n",
    "\n",
    "For more information about this dataset, see\n",
    "\n",
    "- https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html\n",
    "\n",
    "- https://en.wikipedia.org/wiki/Iris_flower_data_set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Data preparation  <a class=\"anchor\" id=\"section_1_1\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "First step : importing this dataset.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-19T11:00:12.701811Z",
     "start_time": "2025-03-19T11:00:12.595101Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "X = iris.data\n",
    "y = iris.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 2** : What are the  dimensions of `X` and of `y` ? What is the sample size ?  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:18.569068Z",
     "start_time": "2025-02-07T16:32:18.566479Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(150, 4)\n",
      "(150,)\n"
     ]
    }
   ],
   "source": [
    "# Answer for Exercise 2\n",
    "print(np.shape(X))\n",
    "print(np.shape(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will split `X` and `Y` into two parts: training set and test set using `train_test_split`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:18.644913Z",
     "start_time": "2025-02-07T16:32:18.609906Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.33, random_state=42\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 3** : What are the parameters  `test_size=0.33` and  `random_state=42` used for ? What are the dimensions of  `X_train`,`X_test`, `y_train` and `y_test`?\n",
    "\n",
    "- Indication : https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:18.656592Z",
     "start_time": "2025-02-07T16:32:18.654009Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(100, 4)\n",
      "(50, 4)\n",
      "(100,)\n",
      "(50,)\n"
     ]
    }
   ],
   "source": [
    "# Answer for Exercise 3\n",
    "print(np.shape(X_train))\n",
    "print(np.shape(X_test))\n",
    "print(np.shape(y_train))\n",
    "print(np.shape(y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Creating a K-NN classifier <a class=\"anchor\" id=\"section_1_2\"></a>\n",
    "\n",
    "Let's implement our own K-NN classifier from scratch before diving into the `KNeighborsClassifier` from `sklearn`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 4** : Write a  function that computes the  euclidian distance between `sample1` and `sample2`$\\in \\mathbb{R}^d$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:18.668040Z",
     "start_time": "2025-02-07T16:32:18.666273Z"
    }
   },
   "outputs": [],
   "source": [
    "def euc_dis(sample1, sample2):\n",
    "    return np.linalg.norm(sample1 - sample2, axis=1) ** 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 5** : Write a  classification function that predicts the class of a new input `X_new` based on the training data and the value of `K`. The function should implement the K-Nearest Neighbors algorithm to make predictions. \n",
    "\n",
    "Hint : \n",
    "\n",
    "- Step 1 : Calculate the  distance between `X_new` and each data point in `X_train` using the function `euc_dis` defined in Exercise 4. \n",
    "\n",
    "- Step 2 : Find the K nearest neighbors using\n",
    "  `np.argsort`.\n",
    "    - Ref : https://numpy.org/doc/stable/reference/generated/numpy.argsort.html . \n",
    "    \n",
    "- Step 3 : Return the most frequent label among those K neighbors. To do this, you can use  `collections - Counter - most_common()` \n",
    "    - Ref : https://docs.python.org/fr/3/library/collections.html . \n",
    "    - Check out the following cell for a practical demonstration of  Counter.most_common().  \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:18.698079Z",
     "start_time": "2025-02-07T16:32:18.695117Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The indices where the 4 smallest digits are located are \n",
      " [12 11  9 10]\n",
      "The 4 smallest digits are  [0.2 0.5 1.  1. ]\n",
      "\n",
      "\n",
      "In 'aabbbbccccccc', the frequencies of the letters are : \n",
      " [('c', 7), ('b', 4), ('a', 2)]\n",
      "In 'aabbbbccccccc', The letter that repeats the most is  c\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "#  np.argsort\n",
    "distance_ex = np.array([4, 4, 4, 3, 3, 3, 2, 2, 2, 1, 1, 0.5, 0.2])\n",
    "print(\n",
    "    \"The indices where the 4 smallest digits are located are \\n\",\n",
    "    np.argsort(distance_ex)[:4],\n",
    ")\n",
    "print(\"The 4 smallest digits are \", distance_ex[np.argsort(distance_ex)[:4]])\n",
    "\n",
    "print(\"\\n\")\n",
    "\n",
    "# counter.most_common()\n",
    "print(\n",
    "    \"In 'aabbbbccccccc', the frequencies of the letters are : \\n\",\n",
    "    Counter(\"aabbbbccccccc\").most_common(),\n",
    ")\n",
    "print(\n",
    "    \"In 'aabbbbccccccc', The letter that repeats the most is \",\n",
    "    Counter(\"aabbbbccccccc\").most_common()[0][0],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:18.727024Z",
     "start_time": "2025-02-07T16:32:18.725108Z"
    }
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "\n",
    "def knn_classifier(X_train, y_train, x_new, K):\n",
    "    distances = euc_dis(X_train, x_new)\n",
    "    nearest_neighbors = np.argsort(distances)[:K]\n",
    "    return Counter(y_train[nearest_neighbors]).most_common(1)[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**OPTIONAL Exercise (faster computation)** (**only for those of you who are comfortable with Python, or if you have already answered all the questions** in this notebook). \n",
    "\n",
    "In this exercise, we try to improve a little the previous code. We will use NumPy's parallel computing capabilities to make distance calculations faster and optimize the computation time of our classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Optional question 1** In order to use parallel computing, you will first create a function, named `euc_dis_mat`, that will compute the euclidian distance between the rows of two matrices.  \n",
    "Let us first remind you of a few facts (if needed) :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`a.sum()`  computes the sum of all the elements in the array `a `(whatever the dimension of a is...). If you want to compute the sum for each row or each column of a 2-dimensional array, you have to specify axis=1 or axis=0 :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:18.747162Z",
     "start_time": "2025-02-07T16:32:18.743867Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2],\n",
       "       [3, 4]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.array([[1, 2], [3, 4]])\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:18.803162Z",
     "start_time": "2025-02-07T16:32:18.798675Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.int64(10)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:18.862775Z",
     "start_time": "2025-02-07T16:32:18.859640Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3, 7])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.sum(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:18.936458Z",
     "start_time": "2025-02-07T16:32:18.932745Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 6])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.sum(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Even more optional** \n",
    "Obviously, this works for a 3(or more)-dimensional array. For instance :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:18.973366Z",
     "start_time": "2025-02-07T16:32:18.970454Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[0, 1],\n",
       "        [2, 3]],\n",
       "\n",
       "       [[4, 5],\n",
       "        [6, 7]]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = np.arange(8).reshape(2, 2, 2)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:19.006689Z",
     "start_time": "2025-02-07T16:32:19.003910Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[ 4,  6],\n",
       "        [ 8, 10]]),\n",
       " array([[ 2,  4],\n",
       "        [10, 12]]),\n",
       " array([[ 1,  5],\n",
       "        [ 9, 13]]),\n",
       " np.int64(28))"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.sum(axis=0), b.sum(axis=1), b.sum(axis=2), b.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comment : for a 3-dimensional array :\n",
    "- axis=0 means we sum the matrices. \n",
    "- axis=1 means we sum the rows of each matrix\n",
    "- axis=2 means we sum the columns of each matrix. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**End of even more optional**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:19.039310Z",
     "start_time": "2025-02-07T16:32:19.037538Z"
    }
   },
   "outputs": [],
   "source": [
    "def euc_dis_mat(sample1, sample2):\n",
    "    return np.linalg.norm(sample1 - sample2, axis=1) ** 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Optional question 2**\n",
    "Create a new and optimized classifier : it will be the same as before except it must use the function `euc_dist_mat`and compute the distances between the new value `x_new` and  the training values *simultaneously*.\n",
    "<details>\n",
    "<summary>\n",
    "    Click here for a first hint </summary> \n",
    "    Create a 2-dimensional array `X_new` that contains n rows all equal to `x_new`, where n=number of rows of `X_train`. (So `X_new`is just a 2-dimensional array (same shape as `X_train`)   and the rows of `X_new` are all the same.)  \n",
    "    </details>\n",
    "\n",
    "<details>   \n",
    "<summary>\n",
    "    Click here for a second hint </summary> \n",
    "    You can use `np.tile` \n",
    "        </details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:19.076151Z",
     "start_time": "2025-02-07T16:32:19.073174Z"
    }
   },
   "outputs": [],
   "source": [
    "def knn_class_2(X_train, y_train, x_new, K):\n",
    "    X_new = np.tile(x_new, (len(X_train), 1))\n",
    "    distances = euc_dis_mat(X_train, X_new)\n",
    "    k_neighbors = np.argsort(distances)[:K]\n",
    "    return Counter(y_train[k_neighbors]).most_common()[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**END OF OPTIONAL** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6** : Here we predict the labels for `X_test` using the  `knn_classifier` defined in Exercise 5 for K=3. Display the accuracy rate. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:19.114434Z",
     "start_time": "2025-02-07T16:32:19.109526Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The accuracy rate of our classifier is 98.0%\n",
      "The accuracy rate of our classifier is 98.0%\n"
     ]
    }
   ],
   "source": [
    "predictions = [knn_classifier(X_train, y_train, data, 3) for data in X_test]\n",
    "predictions2 = [knn_class_2(X_train, y_train, data, 3) for data in X_test]\n",
    "\n",
    "print(\n",
    "    f\"The accuracy rate of our classifier is {np.sum(predictions == y_test) / len(predictions) * 100}%\"\n",
    ")\n",
    "print(\n",
    "    f\"The accuracy rate of our classifier is {np.sum(predictions2 == y_test) / len(predictions2) * 100}%\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3  The K-NN classifier of `sklearn`  <a class=\"anchor\" id=\"section_1_3\"></a>\n",
    "\n",
    "\n",
    "Now we will directly use `KNeighborsClassifier` available in the `sklearn` package. \n",
    "\n",
    "- Ref : https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html\n",
    "\n",
    "We will use the value $K=3$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:19.215397Z",
     "start_time": "2025-02-07T16:32:19.148514Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "knn_classifier_2 = KNeighborsClassifier(n_neighbors=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 7** : Train the  `knn_classifier_2` model using `X_train` and `y_train`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:19.233667Z",
     "start_time": "2025-02-07T16:32:19.227889Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: #000;\n",
       "  --sklearn-color-text-muted: #666;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: flex;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "  align-items: start;\n",
       "  justify-content: space-between;\n",
       "  gap: 0.5em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label .caption {\n",
       "  font-size: 0.6rem;\n",
       "  font-weight: lighter;\n",
       "  color: var(--sklearn-color-text-muted);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 0.5em;\n",
       "  text-align: center;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>KNeighborsClassifier(n_neighbors=3)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow\"><div><div>KNeighborsClassifier</div></div><div><a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.6/modules/generated/sklearn.neighbors.KNeighborsClassifier.html\">?<span>Documentation for KNeighborsClassifier</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></div></label><div class=\"sk-toggleable__content fitted\"><pre>KNeighborsClassifier(n_neighbors=3)</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "KNeighborsClassifier(n_neighbors=3)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_classifier_2.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 8** : Display the accuracy rate of `knn_classifier_2` on the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:19.260546Z",
     "start_time": "2025-02-07T16:32:19.254828Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.98"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_classifier_2.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. ($\\star$) The decision boundary of the  K-NN classifier<a class=\"anchor\" id=\"chapter2\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we'll examine the decision boundary for the K-NN model in the context of binary classification. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 9** : What is the decision boundary ? \n",
    "\n",
    "Hint : https://en.wikipedia.org/wiki/Decision_boundary "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer for Exercise 9 :\n",
    " \n",
    " \n",
    " \n",
    " \n",
    " \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us examine  the decision boundary for a K-NN classifier on a simulated dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-19T11:00:20.853667Z",
     "start_time": "2025-03-19T11:00:20.714184Z"
    }
   },
   "outputs": [],
   "source": [
    "from itertools import product\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:19.575414Z",
     "start_time": "2025-02-07T16:32:19.570441Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(200, 2) (200,)\n"
     ]
    }
   ],
   "source": [
    "rng = np.random.default_rng(seed=12)\n",
    "num_observations = 100\n",
    "\n",
    "xx1 = rng.multivariate_normal([0, 0], [[1, 0.6], [0.6, 1]], num_observations)\n",
    "xx2 = rng.multivariate_normal([1, 2], [[1, 0.6], [0.6, 1]], num_observations)\n",
    "\n",
    "X2 = np.vstack((xx1, xx2)).astype(np.float32)\n",
    "Y2 = np.hstack((np.zeros(num_observations), np.ones(num_observations)))\n",
    "\n",
    "print(X2.shape, Y2.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:19.693282Z",
     "start_time": "2025-02-07T16:32:19.599201Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x12546dd60>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 72240 characters]",
      "text/plain": [
       "<Figure size 800x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 6))\n",
    "plt.scatter(X2[:, 0], X2[:, 1], c=Y2, alpha=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 10** : Define 6  K-NN classifiers with  K=1,3,5,7,9,11 and fit these 6 classifiers on  `(XX,yy)`. The 6 models will be contained in a list named  `KNNs`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:19.710747Z",
     "start_time": "2025-02-07T16:32:19.706881Z"
    }
   },
   "outputs": [],
   "source": [
    "nb_neighbors = [1, 3, 5, 7, 9, 11]\n",
    "\n",
    "KNNs = []\n",
    "\n",
    "for k in nb_neighbors:\n",
    "    knn_classifier_k = KNeighborsClassifier(n_neighbors=k)\n",
    "    knn_classifier_k.fit(X2, Y2)\n",
    "    KNNs.append(knn_classifier_k)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now visualize the  decision boundary of our 6   K-NN classifiers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:20.615872Z",
     "start_time": "2025-02-07T16:32:19.741060Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 343792 characters]",
      "text/plain": [
       "<Figure size 1500x1200 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_min, x_max = X2[:, 0].min() - 1, X2[:, 0].max() + 1\n",
    "y_min, y_max = X2[:, 1].min() - 1, X2[:, 1].max() + 1\n",
    "\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.1), np.arange(y_min, y_max, 0.1))\n",
    "f, axarr = plt.subplots(2, 3, sharex=\"col\", sharey=\"row\", figsize=(15, 12))\n",
    "\n",
    "for idx, clf, tt in zip(\n",
    "    product([0, 1, 2], [0, 1, 2]), KNNs, [f\"KNN (k={k})\" for k in nb_neighbors], strict=False\n",
    "):\n",
    "    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "    Z = Z.reshape(xx.shape)\n",
    "\n",
    "    axarr[idx[0], idx[1]].contourf(xx, yy, Z, alpha=0.4)\n",
    "    axarr[idx[0], idx[1]].scatter(X2[:, 0], X2[:, 1], c=Y2, s=20, edgecolor=\"k\")\n",
    "    axarr[idx[0], idx[1]].set_title(tt)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 11** : what comments do these six boundary decisions bring up?\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer for Exercise 11: the larger K is, ....\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. ($\\star$) MNIST dataset<a class=\"anchor\" id=\"chapter2\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST dataset is a dataset of 70,000 small square 28×28 pixel grayscale images of handwritten single digits between 0 and 9. Let us import the dataset  using sklearn : "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:23.064353Z",
     "start_time": "2025-02-07T16:32:20.623241Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_openml\n",
    "\n",
    "mnist = fetch_openml(\"mnist_784\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is contained in the attributes `data` and `target`: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:23.107290Z",
     "start_time": "2025-02-07T16:32:23.095904Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pixel1</th>\n",
       "      <th>pixel2</th>\n",
       "      <th>pixel3</th>\n",
       "      <th>pixel4</th>\n",
       "      <th>pixel5</th>\n",
       "      <th>pixel6</th>\n",
       "      <th>pixel7</th>\n",
       "      <th>pixel8</th>\n",
       "      <th>pixel9</th>\n",
       "      <th>pixel10</th>\n",
       "      <th>...</th>\n",
       "      <th>pixel775</th>\n",
       "      <th>pixel776</th>\n",
       "      <th>pixel777</th>\n",
       "      <th>pixel778</th>\n",
       "      <th>pixel779</th>\n",
       "      <th>pixel780</th>\n",
       "      <th>pixel781</th>\n",
       "      <th>pixel782</th>\n",
       "      <th>pixel783</th>\n",
       "      <th>pixel784</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69995</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69996</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69997</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69998</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69999</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>70000 rows × 784 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       pixel1  pixel2  pixel3  pixel4  pixel5  pixel6  pixel7  pixel8  pixel9  \\\n",
       "0           0       0       0       0       0       0       0       0       0   \n",
       "1           0       0       0       0       0       0       0       0       0   \n",
       "2           0       0       0       0       0       0       0       0       0   \n",
       "3           0       0       0       0       0       0       0       0       0   \n",
       "4           0       0       0       0       0       0       0       0       0   \n",
       "...       ...     ...     ...     ...     ...     ...     ...     ...     ...   \n",
       "69995       0       0       0       0       0       0       0       0       0   \n",
       "69996       0       0       0       0       0       0       0       0       0   \n",
       "69997       0       0       0       0       0       0       0       0       0   \n",
       "69998       0       0       0       0       0       0       0       0       0   \n",
       "69999       0       0       0       0       0       0       0       0       0   \n",
       "\n",
       "       pixel10  ...  pixel775  pixel776  pixel777  pixel778  pixel779  \\\n",
       "0            0  ...         0         0         0         0         0   \n",
       "1            0  ...         0         0         0         0         0   \n",
       "2            0  ...         0         0         0         0         0   \n",
       "3            0  ...         0         0         0         0         0   \n",
       "4            0  ...         0         0         0         0         0   \n",
       "...        ...  ...       ...       ...       ...       ...       ...   \n",
       "69995        0  ...         0         0         0         0         0   \n",
       "69996        0  ...         0         0         0         0         0   \n",
       "69997        0  ...         0         0         0         0         0   \n",
       "69998        0  ...         0         0         0         0         0   \n",
       "69999        0  ...         0         0         0         0         0   \n",
       "\n",
       "       pixel780  pixel781  pixel782  pixel783  pixel784  \n",
       "0             0         0         0         0         0  \n",
       "1             0         0         0         0         0  \n",
       "2             0         0         0         0         0  \n",
       "3             0         0         0         0         0  \n",
       "4             0         0         0         0         0  \n",
       "...         ...       ...       ...       ...       ...  \n",
       "69995         0         0         0         0         0  \n",
       "69996         0         0         0         0         0  \n",
       "69997         0         0         0         0         0  \n",
       "69998         0         0         0         0         0  \n",
       "69999         0         0         0         0         0  \n",
       "\n",
       "[70000 rows x 784 columns]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnist.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:23.292852Z",
     "start_time": "2025-02-07T16:32:23.277441Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        5\n",
       "1        0\n",
       "2        4\n",
       "3        1\n",
       "4        9\n",
       "        ..\n",
       "69995    2\n",
       "69996    3\n",
       "69997    4\n",
       "69998    5\n",
       "69999    6\n",
       "Name: class, Length: 70000, dtype: category\n",
       "Categories (10, object): ['0', '1', '2', '3', ..., '6', '7', '8', '9']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnist.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:23.505490Z",
     "start_time": "2025-02-07T16:32:23.500486Z"
    }
   },
   "outputs": [],
   "source": [
    "X, y = mnist.data, mnist.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:23.559082Z",
     "start_time": "2025-02-07T16:32:23.555630Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:23.690898Z",
     "start_time": "2025-02-07T16:32:23.686508Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So X is a dataframe  and y is a series. We will encounter those objects again later but for the moment, let us manipulate numpy arrays instead : "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:23.768402Z",
     "start_time": "2025-02-07T16:32:23.759338Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(numpy.ndarray, numpy.ndarray)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X, y = X.to_numpy(), y.to_numpy()\n",
    "type(X), type(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:23.842258Z",
     "start_time": "2025-02-07T16:32:23.831330Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((70000, 784), (70000,))"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape, y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to display the corresponding images, you must reshape the data into 28x28 arrays (instead of one-dimensional arrays with 784 elements). Let us display the first digit :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:23.969004Z",
     "start_time": "2025-02-07T16:32:23.905090Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 9688 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "first_figure = X[0].reshape(28, 28)\n",
    "plt.imshow(first_figure, cmap=plt.cm.gray_r, interpolation=\"nearest\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:24.460659Z",
     "start_time": "2025-02-07T16:32:24.455555Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'5'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we recognised a \"5\"\n",
    "y[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:32:25.246771Z",
     "start_time": "2025-02-07T16:32:24.668327Z"
    }
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.33, random_state=42\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 12**\n",
    "Display the accuracy rate on the test set : again we will use the K-NN classifier with K=3.\n",
    "\n",
    "Warning : do not use your own classifier as it will probably fail. Use sklearn's classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:33:20.674476Z",
     "start_time": "2025-02-07T16:32:25.251727Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score: 0.9693506493506493\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'5'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_classifier_3 = KNeighborsClassifier(n_neighbors=3)\n",
    "knn_classifier_3.fit(X_train, y_train)\n",
    "print(f\"Accuracy score: {knn_classifier_3.score(X_test, y_test)}\")\n",
    "predictions = knn_classifier_3.predict(X)\n",
    "predictions[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. ($\\star$) CIFAR10 dataset<a class=\"anchor\" id=\"chapter2\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The CIFAR-10 and CIFAR-100 datasets are labeled subsets of the 80 million tiny images dataset. CIFAR-10 and CIFAR-100 were created by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. \n",
    "The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. More information here  : https://www.cs.toronto.edu/~kriz/cifar.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-19T11:00:47.008627Z",
     "start_time": "2025-03-19T11:00:45.853619Z"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "(X_train, y_train), (X_test, y_test) = tf.keras.datasets.cifar10.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-19T11:00:47.016892Z",
     "start_time": "2025-03-19T11:00:47.014283Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-19T11:00:47.097843Z",
     "start_time": "2025-03-19T11:00:47.094336Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((50000, 32, 32, 3), (50000, 1))"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us reshape `y_train` into a one-dimensional array (instead of a 2 dimensional array with one column) using `ravel` (see the very example here if needed : https://numpy.org/doc/2.1/reference/generated/numpy.ravel.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us reshape `y_train` into a one-dimensional array instead of a 2 dimensional array using `ravel` (see the very example here if needed : https://numpy.org/doc/2.1/reference/generated/numpy.ravel.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-19T11:00:51.610775Z",
     "start_time": "2025-03-19T11:00:51.607269Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50000,)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = y_train.ravel()\n",
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-19T11:00:52.188886Z",
     "start_time": "2025-03-19T11:00:52.185299Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([6, 9, 9, 4, 1, 1, 2, 7, 8, 3], dtype=uint8)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-19T11:00:55.705062Z",
     "start_time": "2025-03-19T11:00:55.617852Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 16152 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure()\n",
    "plt.imshow(X_train[3], interpolation=\"nearest\")\n",
    "plt.show()  # the image is very blurred, do not worry..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-19T11:00:59.717372Z",
     "start_time": "2025-03-19T11:00:59.714673Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deer\n"
     ]
    }
   ],
   "source": [
    "class_names = [\n",
    "    \"plane\",\n",
    "    \"car\",\n",
    "    \"bird\",\n",
    "    \"cat\",\n",
    "    \"deer\",\n",
    "    \"dog\",\n",
    "    \"frog\",\n",
    "    \"horse\",\n",
    "    \"ship\",\n",
    "    \"truck\",\n",
    "]\n",
    "print(class_names[y[3]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each image is a 32x32x3 array that we turn into a $32*32*3$=3072 one-dimensional array using `reshape`, in order to be able to feed it to a KNN classifier.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-03-19T11:01:02.017921Z",
     "start_time": "2025-03-19T11:01:01.759177Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50000, 3072)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = X_train.reshape(X_train.shape[0], -1)\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 13** Same as Exercise 12. \n",
    "\n",
    "Warning : again, use Sklearn's classifier.  \n",
    "\n",
    "Can you explain the poor performance of the K-NN classifier on this dataset ? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2025-03-19T11:01:02.886276Z"
    },
    "jupyter": {
     "is_executing": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 3072) (10000,)\n"
     ]
    }
   ],
   "source": [
    "X1, y1 = X_test.reshape(X_test.shape[0], -1), y_test.ravel()\n",
    "print(X1.shape, y1.shape)\n",
    "\n",
    "knn_classifier_4 = KNeighborsClassifier(n_neighbors=3)\n",
    "knn_classifier_4.fit(X, y)\n",
    "knn_classifier_4.score(X1, y1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
