{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TP1 A first example of statistical learning\n",
    "\n",
    "\n",
    "### Table of Contents\n",
    "\n",
    "* [1. Linear regression](#chapter1)\n",
    "* [2. Polynomial regression](#chapter2)\n",
    "   * [2.1 `PolynomialFeatures` 2-dimensional features](#section2_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use  the following simulated data. "
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:32:45.876883Z",
     "start_time": "2025-01-22T09:32:45.601010Z"
    }
   },
   "source": [
    "from cProfile import label\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "rng = np.random.default_rng(seed=42)\n",
    "size = 100\n",
    "x = np.sort(rng.uniform(-5, 5, size))\n",
    "X = x.reshape(size, 1)  # See later (Question 8) for the reason why we reshape\n",
    "# X into a 2-dimensional array with the same size (i.e \n",
    "# same number of elements)\n",
    "y = 0.5 + x ** 2 + x + 2 * rng.standard_normal(size)"
   ],
   "outputs": [],
   "execution_count": 1
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us display the data : "
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:34:02.429428Z",
     "start_time": "2025-01-22T09:34:02.363995Z"
    }
   },
   "source": [
    "plt.scatter(x, y)\n",
    "plt.show()"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ],
      "image/png": "[encoded data, 18924 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "execution_count": 7
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1** : \n",
    "1. Which variable is the *feature/input/covariable*? \n",
    "2. Which variable is the  *label/outcome/target*? \n",
    "3. What is the dimension  $X$? What is the sample size? \n"
   ]
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:34:58.459483Z",
     "start_time": "2025-01-22T09:34:58.457153Z"
    }
   },
   "cell_type": "code",
   "source": "print(X.shape, x.shape)",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(100, 1) (100,)\n"
     ]
    }
   ],
   "execution_count": 8
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer for Exercise 1 :\n",
    "\n",
    "1. Feature : x\n",
    "\n",
    "2. Target : y\n",
    "\n",
    "3. The dimension of X is (100, 1) and the sample size is 100.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Linear regression <a class=\"anchor\" id=\"chapter1\"></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will first use linear regression to model the relationship between $x$ and $y$. That is, we are looking for $\\hat{a}$ and $\\hat{b}$ such that $\\hat{f}(x)=\\hat{a}\\cdot x+\\hat{b}$ is close to $y$ in the sense of quadratic loss.\n",
    "\n",
    "We will use the sklearn package  : `sklearn.linear_model.LinearRegression`. \n",
    "\n",
    "(Ref: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) <-- See the example on this page. The goal is to get used to sklearn syntax as we will use this package for all sessions (except for the neural networks). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 2** : Define a linear regression model, named  `lin_reg`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:38:51.198238Z",
     "start_time": "2025-01-22T09:38:51.195801Z"
    }
   },
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "lin_reg = LinearRegression()"
   ],
   "outputs": [],
   "execution_count": 14
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 3** : Train the linear regression model `lin_reg` on $X$, y. If the training is successful, `lin_reg` will store the model and you will be able to access the different results using its attributes. "
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:38:55.111182Z",
     "start_time": "2025-01-22T09:38:55.104393Z"
    }
   },
   "source": "lin_reg.fit(X, y)",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression()"
      ],
      "text/html": [
       "<style>#sk-container-id-2 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: #000;\n",
       "  --sklearn-color-text-muted: #666;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-2 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-2 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: flex;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "  align-items: start;\n",
       "  justify-content: space-between;\n",
       "  gap: 0.5em;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label .caption {\n",
       "  font-size: 0.6rem;\n",
       "  font-weight: lighter;\n",
       "  color: var(--sklearn-color-text-muted);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-2 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-2 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-2 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 0.5em;\n",
       "  text-align: center;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-2 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LinearRegression()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow\"><div><div>LinearRegression</div></div><div><a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.6/modules/generated/sklearn.linear_model.LinearRegression.html\">?<span>Documentation for LinearRegression</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></div></label><div class=\"sk-toggleable__content fitted\"><pre>LinearRegression()</pre></div> </div></div></div></div>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 16
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 4**: Try the following code and explain the problem. "
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:38:57.098642Z",
     "start_time": "2025-01-22T09:38:56.937883Z"
    }
   },
   "source": "lin_reg.fit(x, y)  # we used x instead of X",
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Expected 2D array, got 1D array instead:\narray=[-4.9263773  -4.77287927 -4.69182165 -4.56196234 -4.41697258 -4.36182744\n -4.12350081 -4.09952139 -4.05822652 -3.85469926 -3.81994098 -3.71886367\n -3.70078495 -3.60247516 -3.60203002 -3.47687897 -3.45710508 -3.38728221\n -3.3302708  -3.10528641 -3.05361292 -3.00091798 -2.85415327 -2.73090651\n -2.72761278 -2.66060514 -2.18616108 -2.11671896 -2.06406242 -1.98487911\n -1.96049902 -1.87633359 -1.74174642 -1.45474032 -1.38187389 -1.29540294\n -1.29201976 -1.18978774 -1.12521621 -0.93613139 -0.91471356 -0.63282611\n -0.62848081 -0.6112156  -0.56585801 -0.53843724 -0.49614062 -0.41084224\n -0.3812277  -0.33278996 -0.30444189 -0.28903794 -0.24295074  0.01044775\n  0.53579401  0.54584787  0.57032152  0.59207161  0.65236106  0.68741196\n  1.30282593  1.31664399  1.3471832   1.4386512   1.61916515  1.6431354\n  1.64850857  1.68402962  1.69813995  1.82495504  1.83048953  1.96320375\n  1.97368029  2.00265102  2.05165379  2.22359351  2.44762156  2.5808774\n  2.61139702  2.64998857  2.73956049  2.78383497  2.80729031  2.83898209\n  2.86064305  2.86924378  3.04764357  3.14020385  3.22761613  3.27631172\n  3.32259801  3.32678196  3.53403073  3.5859792   3.93121121  4.26764989\n  4.61897665  4.67509732  4.70698024  4.75622352].\nReshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.",
     "output_type": "error",
     "traceback": [
      "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[0;31mValueError\u001B[0m                                Traceback (most recent call last)",
      "Cell \u001B[0;32mIn[17], line 1\u001B[0m\n\u001B[0;32m----> 1\u001B[0m \u001B[43mlin_reg\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mfit\u001B[49m\u001B[43m(\u001B[49m\u001B[43mx\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43my\u001B[49m\u001B[43m)\u001B[49m  \u001B[38;5;66;03m# we used x instead of X\u001B[39;00m\n",
      "File \u001B[0;32m/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sklearn/base.py:1389\u001B[0m, in \u001B[0;36m_fit_context.<locals>.decorator.<locals>.wrapper\u001B[0;34m(estimator, *args, **kwargs)\u001B[0m\n\u001B[1;32m   1382\u001B[0m     estimator\u001B[38;5;241m.\u001B[39m_validate_params()\n\u001B[1;32m   1384\u001B[0m \u001B[38;5;28;01mwith\u001B[39;00m config_context(\n\u001B[1;32m   1385\u001B[0m     skip_parameter_validation\u001B[38;5;241m=\u001B[39m(\n\u001B[1;32m   1386\u001B[0m         prefer_skip_nested_validation \u001B[38;5;129;01mor\u001B[39;00m global_skip_validation\n\u001B[1;32m   1387\u001B[0m     )\n\u001B[1;32m   1388\u001B[0m ):\n\u001B[0;32m-> 1389\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[43mfit_method\u001B[49m\u001B[43m(\u001B[49m\u001B[43mestimator\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43margs\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43mkwargs\u001B[49m\u001B[43m)\u001B[49m\n",
      "File \u001B[0;32m/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sklearn/linear_model/_base.py:601\u001B[0m, in \u001B[0;36mLinearRegression.fit\u001B[0;34m(self, X, y, sample_weight)\u001B[0m\n\u001B[1;32m    597\u001B[0m n_jobs_ \u001B[38;5;241m=\u001B[39m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mn_jobs\n\u001B[1;32m    599\u001B[0m accept_sparse \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mFalse\u001B[39;00m \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mpositive \u001B[38;5;28;01melse\u001B[39;00m [\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mcsr\u001B[39m\u001B[38;5;124m\"\u001B[39m, \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mcsc\u001B[39m\u001B[38;5;124m\"\u001B[39m, \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mcoo\u001B[39m\u001B[38;5;124m\"\u001B[39m]\n\u001B[0;32m--> 601\u001B[0m X, y \u001B[38;5;241m=\u001B[39m \u001B[43mvalidate_data\u001B[49m\u001B[43m(\u001B[49m\n\u001B[1;32m    602\u001B[0m \u001B[43m    \u001B[49m\u001B[38;5;28;43mself\u001B[39;49m\u001B[43m,\u001B[49m\n\u001B[1;32m    603\u001B[0m \u001B[43m    \u001B[49m\u001B[43mX\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m    604\u001B[0m \u001B[43m    \u001B[49m\u001B[43my\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m    605\u001B[0m \u001B[43m    \u001B[49m\u001B[43maccept_sparse\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43maccept_sparse\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m    606\u001B[0m \u001B[43m    \u001B[49m\u001B[43my_numeric\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[38;5;28;43;01mTrue\u001B[39;49;00m\u001B[43m,\u001B[49m\n\u001B[1;32m    607\u001B[0m \u001B[43m    \u001B[49m\u001B[43mmulti_output\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[38;5;28;43;01mTrue\u001B[39;49;00m\u001B[43m,\u001B[49m\n\u001B[1;32m    608\u001B[0m \u001B[43m    \u001B[49m\u001B[43mforce_writeable\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[38;5;28;43;01mTrue\u001B[39;49;00m\u001B[43m,\u001B[49m\n\u001B[1;32m    609\u001B[0m \u001B[43m\u001B[49m\u001B[43m)\u001B[49m\n\u001B[1;32m    611\u001B[0m has_sw \u001B[38;5;241m=\u001B[39m sample_weight \u001B[38;5;129;01mis\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m \u001B[38;5;28;01mNone\u001B[39;00m\n\u001B[1;32m    612\u001B[0m \u001B[38;5;28;01mif\u001B[39;00m has_sw:\n",
      "File \u001B[0;32m/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sklearn/utils/validation.py:2961\u001B[0m, in \u001B[0;36mvalidate_data\u001B[0;34m(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)\u001B[0m\n\u001B[1;32m   2959\u001B[0m         y \u001B[38;5;241m=\u001B[39m check_array(y, input_name\u001B[38;5;241m=\u001B[39m\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124my\u001B[39m\u001B[38;5;124m\"\u001B[39m, \u001B[38;5;241m*\u001B[39m\u001B[38;5;241m*\u001B[39mcheck_y_params)\n\u001B[1;32m   2960\u001B[0m     \u001B[38;5;28;01melse\u001B[39;00m:\n\u001B[0;32m-> 2961\u001B[0m         X, y \u001B[38;5;241m=\u001B[39m \u001B[43mcheck_X_y\u001B[49m\u001B[43m(\u001B[49m\u001B[43mX\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43my\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43mcheck_params\u001B[49m\u001B[43m)\u001B[49m\n\u001B[1;32m   2962\u001B[0m     out \u001B[38;5;241m=\u001B[39m X, y\n\u001B[1;32m   2964\u001B[0m \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m no_val_X \u001B[38;5;129;01mand\u001B[39;00m check_params\u001B[38;5;241m.\u001B[39mget(\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mensure_2d\u001B[39m\u001B[38;5;124m\"\u001B[39m, \u001B[38;5;28;01mTrue\u001B[39;00m):\n",
      "File \u001B[0;32m/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sklearn/utils/validation.py:1370\u001B[0m, in \u001B[0;36mcheck_X_y\u001B[0;34m(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, force_all_finite, ensure_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)\u001B[0m\n\u001B[1;32m   1364\u001B[0m     \u001B[38;5;28;01mraise\u001B[39;00m \u001B[38;5;167;01mValueError\u001B[39;00m(\n\u001B[1;32m   1365\u001B[0m         \u001B[38;5;124mf\u001B[39m\u001B[38;5;124m\"\u001B[39m\u001B[38;5;132;01m{\u001B[39;00mestimator_name\u001B[38;5;132;01m}\u001B[39;00m\u001B[38;5;124m requires y to be passed, but the target y is None\u001B[39m\u001B[38;5;124m\"\u001B[39m\n\u001B[1;32m   1366\u001B[0m     )\n\u001B[1;32m   1368\u001B[0m ensure_all_finite \u001B[38;5;241m=\u001B[39m _deprecate_force_all_finite(force_all_finite, ensure_all_finite)\n\u001B[0;32m-> 1370\u001B[0m X \u001B[38;5;241m=\u001B[39m \u001B[43mcheck_array\u001B[49m\u001B[43m(\u001B[49m\n\u001B[1;32m   1371\u001B[0m \u001B[43m    \u001B[49m\u001B[43mX\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m   1372\u001B[0m \u001B[43m    \u001B[49m\u001B[43maccept_sparse\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43maccept_sparse\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m   1373\u001B[0m \u001B[43m    \u001B[49m\u001B[43maccept_large_sparse\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43maccept_large_sparse\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m   1374\u001B[0m \u001B[43m    \u001B[49m\u001B[43mdtype\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mdtype\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m   1375\u001B[0m \u001B[43m    \u001B[49m\u001B[43morder\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43morder\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m   1376\u001B[0m \u001B[43m    \u001B[49m\u001B[43mcopy\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mcopy\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m   1377\u001B[0m \u001B[43m    \u001B[49m\u001B[43mforce_writeable\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mforce_writeable\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m   1378\u001B[0m \u001B[43m    \u001B[49m\u001B[43mensure_all_finite\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mensure_all_finite\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m   1379\u001B[0m \u001B[43m    \u001B[49m\u001B[43mensure_2d\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mensure_2d\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m   1380\u001B[0m \u001B[43m    \u001B[49m\u001B[43mallow_nd\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mallow_nd\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m   1381\u001B[0m \u001B[43m    \u001B[49m\u001B[43mensure_min_samples\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mensure_min_samples\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m   1382\u001B[0m \u001B[43m    \u001B[49m\u001B[43mensure_min_features\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mensure_min_features\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m   1383\u001B[0m \u001B[43m    \u001B[49m\u001B[43mestimator\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mestimator\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m   1384\u001B[0m \u001B[43m    \u001B[49m\u001B[43minput_name\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[38;5;124;43m\"\u001B[39;49m\u001B[38;5;124;43mX\u001B[39;49m\u001B[38;5;124;43m\"\u001B[39;49m\u001B[43m,\u001B[49m\n\u001B[1;32m   1385\u001B[0m \u001B[43m\u001B[49m\u001B[43m)\u001B[49m\n\u001B[1;32m   1387\u001B[0m y \u001B[38;5;241m=\u001B[39m _check_y(y, multi_output\u001B[38;5;241m=\u001B[39mmulti_output, y_numeric\u001B[38;5;241m=\u001B[39my_numeric, estimator\u001B[38;5;241m=\u001B[39mestimator)\n\u001B[1;32m   1389\u001B[0m check_consistent_length(X, y)\n",
      "File \u001B[0;32m/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sklearn/utils/validation.py:1093\u001B[0m, in \u001B[0;36mcheck_array\u001B[0;34m(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, force_all_finite, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)\u001B[0m\n\u001B[1;32m   1086\u001B[0m         \u001B[38;5;28;01melse\u001B[39;00m:\n\u001B[1;32m   1087\u001B[0m             msg \u001B[38;5;241m=\u001B[39m (\n\u001B[1;32m   1088\u001B[0m                 \u001B[38;5;124mf\u001B[39m\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mExpected 2D array, got 1D array instead:\u001B[39m\u001B[38;5;130;01m\\n\u001B[39;00m\u001B[38;5;124marray=\u001B[39m\u001B[38;5;132;01m{\u001B[39;00marray\u001B[38;5;132;01m}\u001B[39;00m\u001B[38;5;124m.\u001B[39m\u001B[38;5;130;01m\\n\u001B[39;00m\u001B[38;5;124m\"\u001B[39m\n\u001B[1;32m   1089\u001B[0m                 \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mReshape your data either using array.reshape(-1, 1) if \u001B[39m\u001B[38;5;124m\"\u001B[39m\n\u001B[1;32m   1090\u001B[0m                 \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124myour data has a single feature or array.reshape(1, -1) \u001B[39m\u001B[38;5;124m\"\u001B[39m\n\u001B[1;32m   1091\u001B[0m                 \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mif it contains a single sample.\u001B[39m\u001B[38;5;124m\"\u001B[39m\n\u001B[1;32m   1092\u001B[0m             )\n\u001B[0;32m-> 1093\u001B[0m         \u001B[38;5;28;01mraise\u001B[39;00m \u001B[38;5;167;01mValueError\u001B[39;00m(msg)\n\u001B[1;32m   1095\u001B[0m \u001B[38;5;28;01mif\u001B[39;00m dtype_numeric \u001B[38;5;129;01mand\u001B[39;00m \u001B[38;5;28mhasattr\u001B[39m(array\u001B[38;5;241m.\u001B[39mdtype, \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mkind\u001B[39m\u001B[38;5;124m\"\u001B[39m) \u001B[38;5;129;01mand\u001B[39;00m array\u001B[38;5;241m.\u001B[39mdtype\u001B[38;5;241m.\u001B[39mkind \u001B[38;5;129;01min\u001B[39;00m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mUSV\u001B[39m\u001B[38;5;124m\"\u001B[39m:\n\u001B[1;32m   1096\u001B[0m     \u001B[38;5;28;01mraise\u001B[39;00m \u001B[38;5;167;01mValueError\u001B[39;00m(\n\u001B[1;32m   1097\u001B[0m         \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mdtype=\u001B[39m\u001B[38;5;124m'\u001B[39m\u001B[38;5;124mnumeric\u001B[39m\u001B[38;5;124m'\u001B[39m\u001B[38;5;124m is not compatible with arrays of bytes/strings.\u001B[39m\u001B[38;5;124m\"\u001B[39m\n\u001B[1;32m   1098\u001B[0m         \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mConvert your data to numeric values explicitly instead.\u001B[39m\u001B[38;5;124m\"\u001B[39m\n\u001B[1;32m   1099\u001B[0m     )\n",
      "\u001B[0;31mValueError\u001B[0m: Expected 2D array, got 1D array instead:\narray=[-4.9263773  -4.77287927 -4.69182165 -4.56196234 -4.41697258 -4.36182744\n -4.12350081 -4.09952139 -4.05822652 -3.85469926 -3.81994098 -3.71886367\n -3.70078495 -3.60247516 -3.60203002 -3.47687897 -3.45710508 -3.38728221\n -3.3302708  -3.10528641 -3.05361292 -3.00091798 -2.85415327 -2.73090651\n -2.72761278 -2.66060514 -2.18616108 -2.11671896 -2.06406242 -1.98487911\n -1.96049902 -1.87633359 -1.74174642 -1.45474032 -1.38187389 -1.29540294\n -1.29201976 -1.18978774 -1.12521621 -0.93613139 -0.91471356 -0.63282611\n -0.62848081 -0.6112156  -0.56585801 -0.53843724 -0.49614062 -0.41084224\n -0.3812277  -0.33278996 -0.30444189 -0.28903794 -0.24295074  0.01044775\n  0.53579401  0.54584787  0.57032152  0.59207161  0.65236106  0.68741196\n  1.30282593  1.31664399  1.3471832   1.4386512   1.61916515  1.6431354\n  1.64850857  1.68402962  1.69813995  1.82495504  1.83048953  1.96320375\n  1.97368029  2.00265102  2.05165379  2.22359351  2.44762156  2.5808774\n  2.61139702  2.64998857  2.73956049  2.78383497  2.80729031  2.83898209\n  2.86064305  2.86924378  3.04764357  3.14020385  3.22761613  3.27631172\n  3.32259801  3.32678196  3.53403073  3.5859792   3.93121121  4.26764989\n  4.61897665  4.67509732  4.70698024  4.75622352].\nReshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample."
     ]
    }
   ],
   "execution_count": 17
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same *problem* arises very often with sklearn : we will have to use `reshape` for one-dimensional features. Some extra info about `reshape`:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:39:52.478784Z",
     "start_time": "2025-01-22T09:39:52.472388Z"
    }
   },
   "source": [
    "a = np.arange(6).reshape(2, 3)\n",
    "# 3 ways to reshape the array a into a 3 x 2 array \n",
    "b = a.reshape(3, 2)\n",
    "b2 = a.reshape(3, -1)  # -1 means \"guess\" the number\n",
    "b3 = a.reshape(-1, 2)  # same\n",
    "# obviously reshape(4,-1) would not work here as 2 X 3=6 is not\n",
    "# divisible by 4...\n",
    "print(a)\n",
    "print(b)\n",
    "print(b2)\n",
    "print(b3)"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 1 2]\n",
      " [3 4 5]]\n",
      "[[0 1]\n",
      " [2 3]\n",
      " [4 5]]\n",
      "[[0 1]\n",
      " [2 3]\n",
      " [4 5]]\n",
      "[[0 1]\n",
      " [2 3]\n",
      " [4 5]]\n"
     ]
    }
   ],
   "execution_count": 20
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 5** : \n",
    "\n",
    "(1) Compute $\\hat{y}$ and the plot the line estimated by the model. \n",
    "<details> \n",
    "<summary>Click here for a hint</summary>\n",
    "You can use  `lin_reg.predict`. \n",
    "</details>\n",
    "\n",
    "(2) Predict the value of $y$ for  x=1. "
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:50:01.091807Z",
     "start_time": "2025-01-22T09:50:00.964206Z"
    }
   },
   "source": [
    "y_predict = lin_reg.predict(X)\n",
    "plt.plot(X, y_predict, color='r', label='Regression')\n",
    "plt.scatter(X, y, label='Sample')\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "y_1 = lin_reg.predict([[1]])\n",
    "print(\"The estimated value for x=1 is\", y_1)"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ],
      "image/png": "[encoded data, 25480 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The estimated value for x=1 is [8.54372371]\n"
     ]
    }
   ],
   "execution_count": 52
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6** : Display the coefficients $\\hat{a}$ and $\\hat{b}$ computed by  `lin_reg`. "
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:42:26.316701Z",
     "start_time": "2025-01-22T09:42:26.312959Z"
    }
   },
   "source": "print(\"The estimated coefficients are  a=\", lin_reg.coef_, \", b=\", lin_reg.intercept_)",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The estimated coefficients are  a= [0.6841386] , b= 7.8595851113619375\n"
     ]
    }
   ],
   "execution_count": 30
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 7** : Calculate the quadratic error  $\\sum_{i=1}^{100}(y_i-\\hat{y}_i)^2$."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:42:50.448815Z",
     "start_time": "2025-01-22T09:42:50.445550Z"
    }
   },
   "source": "print(np.sum((y - y_predict) ** 2))",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4906.471161398307\n"
     ]
    }
   ],
   "execution_count": 31
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 8** : What does the next line of code compute ?  (see the doc :  https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html.)"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:42:57.691432Z",
     "start_time": "2025-01-22T09:42:57.686810Z"
    }
   },
   "source": [
    "lin_reg.score(X, y)"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.06603572054935491"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 32
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "The output is the $R^2$ coefficient. It is far from 1, which means that the model is not perfect."
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.Polynomial regression <a class=\"anchor\" id=\"chapter2\"></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We seek 3 coefficients $\\hat{a}$, $\\hat{b}$ and $\\hat{c}$ such that $\\hat{f}(x)=\\hat{a}\\cdot x^{2}+\\hat{b}\\cdot x +\\hat{c}$ is close to  $y$ with respect to the quadratic loss. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 9** : Create a new 2-dimensional array named `X2`, of  dimension $100\\times$2, the first column of which is $x^2$, and the second column is $x$."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:44:45.265060Z",
     "start_time": "2025-01-22T09:44:45.261420Z"
    }
   },
   "source": [
    "X2 = np.array([x ** 2, x]).T\n",
    "print(X2.shape)"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(100, 2)\n"
     ]
    }
   ],
   "execution_count": 36
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 10** : Fit a new model, named `poly2_reg`, which will be the linear regression  of  `y` on `X2`  (which will correspond to  2nd degree polynomial regression of `y` on `X`)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:45:05.543859Z",
     "start_time": "2025-01-22T09:45:05.532828Z"
    }
   },
   "source": [
    "poly2_reg = LinearRegression()\n",
    "poly2_reg.fit(X2, y)"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression()"
      ],
      "text/html": [
       "<style>#sk-container-id-3 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: #000;\n",
       "  --sklearn-color-text-muted: #666;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-3 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-3 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-3 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: flex;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "  align-items: start;\n",
       "  justify-content: space-between;\n",
       "  gap: 0.5em;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 label.sk-toggleable__label .caption {\n",
       "  font-size: 0.6rem;\n",
       "  font-weight: lighter;\n",
       "  color: var(--sklearn-color-text-muted);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-3 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-3 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-3 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-3 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-3 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 0.5em;\n",
       "  text-align: center;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-3 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-3 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LinearRegression()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" checked><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow\"><div><div>LinearRegression</div></div><div><a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.6/modules/generated/sklearn.linear_model.LinearRegression.html\">?<span>Documentation for LinearRegression</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></div></label><div class=\"sk-toggleable__content fitted\"><pre>LinearRegression()</pre></div> </div></div></div></div>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 37
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 11** : \n",
    "\n",
    "(1) Plot the second degree polynomial estimated by `poly2_reg` \n",
    "<details> \n",
    "    <summary> Hint </summary> you can use`poly2_reg.predict`. \n",
    "</details>        \n",
    " \n",
    "\n",
    "(2) Predict the value of   $y$ for  x=1 using `poly2_reg`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:49:38.985508Z",
     "start_time": "2025-01-22T09:49:38.910274Z"
    }
   },
   "source": [
    "# Answer for Exercise 11\n",
    "y2_predict = poly2_reg.predict(X2)\n",
    "plt.scatter(x, y, label='Sample')\n",
    "plt.plot(x, y2_predict, color='r', label='Regression')\n",
    "\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "y2_1 = poly2_reg.predict([[1, 1]])\n",
    "print(\"The estimated value for x=1 is\", y2_1)  #add your code"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ],
      "image/png": "[encoded data, 34936 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The estimated value for x=1 is [2.51857814]\n"
     ]
    }
   ],
   "execution_count": 50
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 12** : Display the estimated coefficients $\\hat{a}$, $\\hat{b}$, $\\hat{c}$. "
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:48:21.636481Z",
     "start_time": "2025-01-22T09:48:21.632962Z"
    }
   },
   "source": [
    "print(\"The estimated coefficients are  a=\", poly2_reg.coef_[0], \", b=\", poly2_reg.coef_[1], \"and c=\",\n",
    "      poly2_reg.intercept_)"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The estimated coefficients are  a= 1.0030014670964114 , b= 1.0581583492954199 and c= 0.4574183201104409\n"
     ]
    }
   ],
   "execution_count": 45
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 13** : Display the quadratic error of `poly2_reg` and compare with the quadratic error related to `lin_reg`. "
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:48:26.904691Z",
     "start_time": "2025-01-22T09:48:26.902199Z"
    }
   },
   "source": "print(\"Quadratic error for the 2nd degree polynomial regression\", np.sum((y - y2_predict) ** 2))",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Quadratic error for the 2nd degree polynomial regression 381.2677328521117\n"
     ]
    }
   ],
   "execution_count": 46
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The creation of the array  $[X^2, X]$ from $X$ can also be performed directly by by `sklearn.preprocessing.PolynomialFeatures`\n",
    "\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html\n",
    "\n",
    "(Pay attention to  the order of the features in the output of  `PolynomialFeatures` : $X\\rightarrow [1, X, X^2]$ )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercice 13** : In the cell below, we show how to use `PolynomialFeatures`in our example. Examine the syntax and then create a model for a $3^{rd}$ degree polynomial. "
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:51:54.357008Z",
     "start_time": "2025-01-22T09:51:54.285099Z"
    }
   },
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "poly2 = PolynomialFeatures(degree=2)  # definition of the transformer\n",
    "# no calculation made yet\n",
    "poly2.fit(X)  # here you ask your transformer to make some calculations\n",
    "# no output yet (no assignment here)\n",
    "X2_bis = poly2.transform(X)  # here you ask your transformer for an output\n",
    "# (assignment 'X21_bis=...'). Your new array X2_bis now contains\n",
    "# $[1,X,X^2]$\n",
    "\n",
    "# Fitting a linear regression model as before (y on X2_bis)\n",
    "poly2_bis_reg = LinearRegression()\n",
    "poly2_bis_reg.fit(X2_bis, y)\n",
    "\n",
    "# Plotting the estimated polynomial\n",
    "plt.scatter(x, y)\n",
    "plt.plot(x, poly2_bis_reg.predict(X2_bis), color='r')\n",
    "plt.show()\n",
    "\n",
    "# estimated coefficients (Pay attention to the order)\n",
    "print(poly2_bis_reg.coef_, poly2_bis_reg.intercept_)\n",
    "print(\"The estimated coefficients are  a=\", poly2_bis_reg.coef_[2], \", b=\", poly2_bis_reg.coef_[1], \"and c=\",\n",
    "      poly2_bis_reg.intercept_)"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ],
      "image/png": "[encoded data, 30988 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.         1.05815835 1.00300147] 0.45741832011044625\n",
      "The estimated coefficients are  a= 1.0030014670964105 , b= 1.0581583492954199 and c= 0.45741832011044625\n"
     ]
    }
   ],
   "execution_count": 53
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:56:27.580084Z",
     "start_time": "2025-01-22T09:56:27.504624Z"
    }
   },
   "source": [
    "# Answer for Exercise 13 ($3^{rd}$ degree)\n",
    "poly3 = PolynomialFeatures(degree=3)\n",
    "poly3.fit(X)\n",
    "X3_bis = poly3.transform(X)\n",
    "\n",
    "poly3_reg = LinearRegression()\n",
    "poly3_reg.fit(X3_bis, y)\n",
    "\n",
    "plt.scatter(x, y, label='Sample')\n",
    "plt.plot(x, poly3_reg.predict(X3_bis), color='r', label='Regression')\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "print(poly3_reg.coef_, poly3_reg.intercept_)\n",
    "print(\"The estimated coefficients are  a=\", poly3_reg.coef_[3], \", b=\", poly3_reg.coef_[2], \", c=\", poly3_reg.coef_[1],\n",
    "      \"and d=\", poly3_reg.intercept_)"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ],
      "image/png": "[encoded data, 34896 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.          1.11490357  1.00197225 -0.0042044 ] 0.45683227934303705\n",
      "The estimated coefficients are  a= -0.00420440206943242 , b= 1.0019722513203924 , c= 1.1149035682964386 and d= 0.45683227934303705\n"
     ]
    }
   ],
   "execution_count": 61
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 `PolynomialFeatures` for 2-dimensional features <a class=\"anchor\" id=\"section2_1\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 14** : Examine the next 3 cells and their outputs. Foir an output  $X=(x_1,x_2)$, what do  the output of `PolynomialFeatures(2)` and the output of  `PolynomialFeatures(2, interaction_only=True)` correspond to?"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:57:11.033033Z",
     "start_time": "2025-01-22T09:57:11.029811Z"
    }
   },
   "source": [
    "XX = np.arange(6).reshape(3, 2)\n",
    "print(XX)"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 1]\n",
      " [2 3]\n",
      " [4 5]]\n"
     ]
    }
   ],
   "execution_count": 62
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:57:11.899363Z",
     "start_time": "2025-01-22T09:57:11.893990Z"
    }
   },
   "source": [
    "poly2 = PolynomialFeatures(2)\n",
    "poly2.fit_transform(XX)"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.,  0.,  1.,  0.,  0.,  1.],\n",
       "       [ 1.,  2.,  3.,  4.,  6.,  9.],\n",
       "       [ 1.,  4.,  5., 16., 20., 25.]])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 63
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-22T09:57:12.753351Z",
     "start_time": "2025-01-22T09:57:12.748167Z"
    }
   },
   "source": [
    "poly2_bis = PolynomialFeatures(2, interaction_only=True)\n",
    "poly2_bis.fit_transform(XX)"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.,  0.,  1.,  0.],\n",
       "       [ 1.,  2.,  3.,  6.],\n",
       "       [ 1.,  4.,  5., 20.]])"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 64
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "### Answer for Exercise 14\n",
    "The output of PolynomialFeatures(2) corresponds to $[1, x_1, x_2, x_1^2, x_1x_2, x_2^2]$\n",
    "\n",
    "The output of PolynomialFeatures(2, interaction_only=True) corresponds to $[1, x_1, x_2, x_1x_2]$"
   ]
  },
  {
   "metadata": {},
   "cell_type": "code",
   "outputs": [],
   "execution_count": null,
   "source": ""
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
