{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4398a857-3e6e-4244-bf6b-681f5a00b69d",
   "metadata": {},
   "source": [
    "# OPTIONAL notebook "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2551b5dc-fc25-40e8-bdc7-11d99f728e04",
   "metadata": {},
   "source": [
    "## 1. Dataframes et package Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "968bf85f-3030-4766-a0b9-c32bcaedc9f4",
   "metadata": {},
   "source": [
    " We can open the `iris` file to obtain a dataframe. To manipulate dataframes, we need the `pandas` package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "674da132-880c-433c-8048-857f640dc26f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:25:26.185851Z",
     "start_time": "2025-02-07T16:25:26.180354Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn import datasets\n",
    "\n",
    "iris = datasets.load_iris(as_frame=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "861a284d-8402-4031-bf9f-f76baa0fd689",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:25:27.562898Z",
     "start_time": "2025-02-07T16:25:27.559127Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(pandas.core.frame.DataFrame, pandas.core.series.Series)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = iris.data\n",
    "y = iris.target\n",
    "type(X), type(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28bc7626-2f3e-428a-86ca-172eb0fac524",
   "metadata": {},
   "source": [
    "X is a dataframe and y is a series.\n",
    "\n",
    "To answer the following questions, you can use this tutorial:  https://pandas.pydata.org/docs/user_guide/10min.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d32ef752-34c0-43e3-8128-937bdf0b4c6c",
   "metadata": {},
   "source": [
    "**Question 1** Use some functions described in the tutorial to get information about the dataframe X and the series y.\n",
    "Specifically, display the types of explanatory variables, the names of explanatory variables, and the number of flowers in each category (setosa, versicolor and virginica) in `y`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "301cabcd-d30c-45e1-9a90-b4fe6d6fba4a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:25:29.915708Z",
     "start_time": "2025-02-07T16:25:29.904109Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal length (cm)</th>\n",
       "      <th>sepal width (cm)</th>\n",
       "      <th>petal length (cm)</th>\n",
       "      <th>petal width (cm)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>150.000000</td>\n",
       "      <td>150.000000</td>\n",
       "      <td>150.000000</td>\n",
       "      <td>150.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>5.843333</td>\n",
       "      <td>3.057333</td>\n",
       "      <td>3.758000</td>\n",
       "      <td>1.199333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.828066</td>\n",
       "      <td>0.435866</td>\n",
       "      <td>1.765298</td>\n",
       "      <td>0.762238</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>4.300000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>5.100000</td>\n",
       "      <td>2.800000</td>\n",
       "      <td>1.600000</td>\n",
       "      <td>0.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5.800000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>4.350000</td>\n",
       "      <td>1.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.400000</td>\n",
       "      <td>3.300000</td>\n",
       "      <td>5.100000</td>\n",
       "      <td>1.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>7.900000</td>\n",
       "      <td>4.400000</td>\n",
       "      <td>6.900000</td>\n",
       "      <td>2.500000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       sepal length (cm)  sepal width (cm)  petal length (cm)  \\\n",
       "count         150.000000        150.000000         150.000000   \n",
       "mean            5.843333          3.057333           3.758000   \n",
       "std             0.828066          0.435866           1.765298   \n",
       "min             4.300000          2.000000           1.000000   \n",
       "25%             5.100000          2.800000           1.600000   \n",
       "50%             5.800000          3.000000           4.350000   \n",
       "75%             6.400000          3.300000           5.100000   \n",
       "max             7.900000          4.400000           6.900000   \n",
       "\n",
       "       petal width (cm)  \n",
       "count        150.000000  \n",
       "mean           1.199333  \n",
       "std            0.762238  \n",
       "min            0.100000  \n",
       "25%            0.300000  \n",
       "50%            1.300000  \n",
       "75%            1.800000  \n",
       "max            2.500000  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "00bab47d-d404-449c-85a5-f7aca4ce8088",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:25:31.806100Z",
     "start_time": "2025-02-07T16:25:31.800315Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 150 entries, 0 to 149\n",
      "Data columns (total 4 columns):\n",
      " #   Column             Non-Null Count  Dtype  \n",
      "---  ------             --------------  -----  \n",
      " 0   sepal length (cm)  150 non-null    float64\n",
      " 1   sepal width (cm)   150 non-null    float64\n",
      " 2   petal length (cm)  150 non-null    float64\n",
      " 3   petal width (cm)   150 non-null    float64\n",
      "dtypes: float64(4)\n",
      "memory usage: 4.8 KB\n"
     ]
    }
   ],
   "source": [
    "X.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "1c4d3a0b-107d-44b5-b515-a3c330d5033c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:25:32.460861Z",
     "start_time": "2025-02-07T16:25:32.454513Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal length (cm)</th>\n",
       "      <th>sepal width (cm)</th>\n",
       "      <th>petal length (cm)</th>\n",
       "      <th>petal width (cm)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)\n",
       "0                5.1               3.5                1.4               0.2\n",
       "1                4.9               3.0                1.4               0.2\n",
       "2                4.7               3.2                1.3               0.2\n",
       "3                4.6               3.1                1.5               0.2\n",
       "4                5.0               3.6                1.4               0.2"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "477009d822ba15c5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:25:55.088019Z",
     "start_time": "2025-02-07T16:25:55.085034Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "target\n",
       "0    50\n",
       "1    50\n",
       "2    50\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "469dd338-32f0-44dc-8c4a-697bea1a134a",
   "metadata": {},
   "source": [
    "# 2.  Confusion matrix, accuracy score and stratify"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "40d4f99e-903b-49ad-b75b-08ecff2de3ba",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:26:05.694663Z",
     "start_time": "2025-02-07T16:26:05.294132Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.33, random_state=42\n",
    ")\n",
    "knn_clf = KNeighborsClassifier(n_neighbors=5)\n",
    "knn_clf.fit(X_train, y_train)\n",
    "predictions = knn_clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c25e8ef-bf8a-46bc-abd1-854c70bbd724",
   "metadata": {},
   "source": [
    "**Question 2** Display the confusion matrix related to  test data (X_test,y_test), see https://en.wikipedia.org/wiki/Confusion_matrix \n",
    "Compute the accuracy score using this matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "9ba22bc9-67f8-4a70-b6f8-81296f8adf25",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:27:06.517615Z",
     "start_time": "2025-02-07T16:27:06.513166Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[19,  0,  0],\n",
       "       [ 0, 15,  0],\n",
       "       [ 0,  1, 15]])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "m = confusion_matrix(y_test, predictions)\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "08bfd440-3321-4fd7-bbd2-d59741bd6db3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:28:28.347817Z",
     "start_time": "2025-02-07T16:28:28.345306Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.98\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "accuracy = np.sum(np.diag(m)) / np.size(y_test)\n",
    "print(accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0dc9ba6-8c5b-4249-878b-587f4673c5cb",
   "metadata": {},
   "source": [
    "**Question 3** Display the accuracy score directly (without the confusing matrix). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "759594a1-95f5-4e32-9fee-ec59ed1204c3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:28:29.707471Z",
     "start_time": "2025-02-07T16:28:29.698956Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.98"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bac4659-bbc8-4982-8e0e-2a987fc0dc52",
   "metadata": {},
   "source": [
    "**Question 4** Display the number of individuals for each label in the series `y_test` and observe that the distribution is not uniform across labels, unlike in the original `y` data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "78ef5f41-8b7c-43bf-83e5-6b4d1323c889",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:28:35.305451Z",
     "start_time": "2025-02-07T16:28:35.301354Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "target\n",
       "0    19\n",
       "2    16\n",
       "1    15\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e43046-6780-4a0e-86f2-df692bbbb9b3",
   "metadata": {},
   "source": [
    "**Question 5** Use the `stratify` parameter in `train_test_split` to create a new split into training and test samples, ensuring that the distribution of each label in the test sample matches that of the initial sample (i.e., in this case, an equal number of individuals with labels y=0, y=1, and y=2). (Verify the result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "a662a29f-dd19-4329-96e4-8b16dfdefe7e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:28:56.644202Z",
     "start_time": "2025-02-07T16:28:56.638228Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "target\n",
       "2    17\n",
       "1    17\n",
       "0    16\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_strat, X_test_strat, y_train_strat, y_test_strat = train_test_split(\n",
    "    X, y, test_size=0.33, random_state=42, stratify=y\n",
    ")\n",
    "\n",
    "y_test_strat.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45c1d704-5cfc-4f8a-bb21-fe0a8ba2ad1b",
   "metadata": {},
   "source": [
    "# 3.Dataframes and graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1658432-eb32-4519-a61e-15a06190dbff",
   "metadata": {},
   "source": [
    "We will use a new dataset called housing.\n",
    "\n",
    "This dataset includes features such as population, median income, and median house price for each geographic unit (with a population of 600 to 3,000 people). In this case, the individuals (or data points) essentially represent neighborhoods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "6e63d373-f9e0-4cdb-9a8b-4ff98c78698b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:29:03.527485Z",
     "start_time": "2025-02-07T16:29:03.504708Z"
    }
   },
   "outputs": [],
   "source": [
    "housing = pd.read_csv(\"data/housing.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "062ce13c-1dbd-4747-94fc-f7a2091dd309",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:29:04.094282Z",
     "start_time": "2025-02-07T16:29:04.090499Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(housing)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5c52c1a-9db8-4f88-80ff-ed08ac64a2a1",
   "metadata": {},
   "source": [
    "**Question 6** Display the main information about the housing dataset. Notice that there are missing data. What is the type of each column?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "34616b7e-a96f-4457-b4f8-71f963731943",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:29:09.892510Z",
     "start_time": "2025-02-07T16:29:09.885789Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 20640 entries, 0 to 20639\n",
      "Data columns (total 10 columns):\n",
      " #   Column              Non-Null Count  Dtype  \n",
      "---  ------              --------------  -----  \n",
      " 0   longitude           20640 non-null  float64\n",
      " 1   latitude            20640 non-null  float64\n",
      " 2   housing_median_age  20640 non-null  float64\n",
      " 3   total_rooms         20640 non-null  float64\n",
      " 4   total_bedrooms      20433 non-null  float64\n",
      " 5   population          20640 non-null  float64\n",
      " 6   households          20640 non-null  float64\n",
      " 7   median_income       20640 non-null  float64\n",
      " 8   median_house_value  20640 non-null  float64\n",
      " 9   ocean_proximity     20640 non-null  object \n",
      "dtypes: float64(9), object(1)\n",
      "memory usage: 1.6+ MB\n"
     ]
    }
   ],
   "source": [
    "housing.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3971104e-23f6-470f-9920-da73caf68080",
   "metadata": {},
   "source": [
    "We can try to predict the variable `median_house_values` based on all other variables. \n",
    "\n",
    "We'll start with a geographical visualization of the data: (these are California data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "933c15e5-6d55-400b-b9c6-573178be085c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:29:14.574215Z",
     "start_time": "2025-02-07T16:29:14.025201Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: xlabel='longitude', ylabel='latitude'>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 53212 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "housing.plot(kind=\"scatter\", x=\"longitude\", y=\"latitude\", grid=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40e4e4e0-8d11-41ef-97ff-f0e4bda2827f",
   "metadata": {},
   "source": [
    "**Question 7** Use a transparency argument  to better visualize areas with high density.\n",
    "\n",
    "<details> \n",
    "    <summary> Hint\n",
    "        </summary> use the `alpha` parameter. \n",
    "        </details>\n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "533f7ca3-4995-49d4-ac49-25fd4805ba3a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:29:28.077026Z",
     "start_time": "2025-02-07T16:29:27.984881Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: xlabel='longitude', ylabel='latitude'>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 116144 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "housing.plot(kind=\"scatter\", x=\"longitude\", y=\"latitude\", alpha=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c62b1f12-a77b-4a77-8a1b-eb9e2657ee7b",
   "metadata": {},
   "source": [
    "**Question 8** Make the points, which are meant to represent neighborhoods, have a size proportional to the neighborhood's population and vary in color according to the median_house_value of the neighborhood. Display the legend and the colorbar. You can use the following example as a guide: https://matplotlib.org/stable/plot_types/basic/scatter_plot.html#sphx-glr-plot-types-basic-scatter-plot-py\n",
    "\n",
    "Hint: Use the population divided by 100 for the point sizes.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "e4ee49f0-9233-4c95-8378-1dea6e1d6ebb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:29:43.976610Z",
     "start_time": "2025-02-07T16:29:43.536874Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: xlabel='longitude', ylabel='latitude'>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 209372 characters]",
      "text/plain": [
       "<Figure size 1000x700 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "housing.plot(\n",
    "    kind=\"scatter\",\n",
    "    x=\"longitude\",\n",
    "    y=\"latitude\",\n",
    "    grid=True,\n",
    "    s=housing[\"population\"] / 100,\n",
    "    label=\"population\",\n",
    "    c=\"median_house_value\",\n",
    "    cmap=\"jet\",\n",
    "    colorbar=True,\n",
    "    legend=True,\n",
    "    figsize=(10, 7),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cb68a7a-e075-4101-9c46-0531f732f55c",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "651ae721-568b-4a6a-ac34-b7ee88bb2434",
   "metadata": {},
   "source": [
    "We can display scatter plots for all pairs of columns in a dataframe using the `scatter_matrix` function from the `pandas.plotting` package. Since there are 11 columns, we won't represent all 11^2=121 corresponding graphs, but only a few of them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "897973c9-c8d2-4398-9f75-666f79d04cd4",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:29:52.115929Z",
     "start_time": "2025-02-07T16:29:50.865923Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[<Axes: xlabel='median_house_value', ylabel='median_house_value'>,\n",
       "        <Axes: xlabel='median_income', ylabel='median_house_value'>,\n",
       "        <Axes: xlabel='total_rooms', ylabel='median_house_value'>,\n",
       "        <Axes: xlabel='housing_median_age', ylabel='median_house_value'>],\n",
       "       [<Axes: xlabel='median_house_value', ylabel='median_income'>,\n",
       "        <Axes: xlabel='median_income', ylabel='median_income'>,\n",
       "        <Axes: xlabel='total_rooms', ylabel='median_income'>,\n",
       "        <Axes: xlabel='housing_median_age', ylabel='median_income'>],\n",
       "       [<Axes: xlabel='median_house_value', ylabel='total_rooms'>,\n",
       "        <Axes: xlabel='median_income', ylabel='total_rooms'>,\n",
       "        <Axes: xlabel='total_rooms', ylabel='total_rooms'>,\n",
       "        <Axes: xlabel='housing_median_age', ylabel='total_rooms'>],\n",
       "       [<Axes: xlabel='median_house_value', ylabel='housing_median_age'>,\n",
       "        <Axes: xlabel='median_income', ylabel='housing_median_age'>,\n",
       "        <Axes: xlabel='total_rooms', ylabel='housing_median_age'>,\n",
       "        <Axes: xlabel='housing_median_age', ylabel='housing_median_age'>]],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 523848 characters]",
      "text/plain": [
       "<Figure size 1200x800 with 16 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pandas.plotting import scatter_matrix\n",
    "\n",
    "attributes = [\n",
    "    \"median_house_value\",\n",
    "    \"median_income\",\n",
    "    \"total_rooms\",\n",
    "    \"housing_median_age\",\n",
    "]\n",
    "scatter_matrix(housing[attributes], figsize=(12, 8))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f86ed4f7-47f6-4099-91d2-09aec4df561a",
   "metadata": {},
   "source": [
    "Looking at the diagonal, we can see the histograms for each column. By examining these graphs, it appears that the variable most correlated with median_house_value is median_income.\n",
    "\n",
    "**Question 9** Display the correlations between the target median_house_value and the other variables, in descending order. For the correlation matrix, use `housing.corr`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "fb419590-cd32-409c-b511-7b5fd7a617c4",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:30:27.259321Z",
     "start_time": "2025-02-07T16:30:27.250795Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>longitude</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.924664</td>\n",
       "      <td>-0.108197</td>\n",
       "      <td>0.044568</td>\n",
       "      <td>0.069608</td>\n",
       "      <td>0.099773</td>\n",
       "      <td>0.055310</td>\n",
       "      <td>-0.015176</td>\n",
       "      <td>-0.045967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>latitude</th>\n",
       "      <td>-0.924664</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.011173</td>\n",
       "      <td>-0.036100</td>\n",
       "      <td>-0.066983</td>\n",
       "      <td>-0.108785</td>\n",
       "      <td>-0.071035</td>\n",
       "      <td>-0.079809</td>\n",
       "      <td>-0.144160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>housing_median_age</th>\n",
       "      <td>-0.108197</td>\n",
       "      <td>0.011173</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.361262</td>\n",
       "      <td>-0.320451</td>\n",
       "      <td>-0.296244</td>\n",
       "      <td>-0.302916</td>\n",
       "      <td>-0.119034</td>\n",
       "      <td>0.105623</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>total_rooms</th>\n",
       "      <td>0.044568</td>\n",
       "      <td>-0.036100</td>\n",
       "      <td>-0.361262</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.930380</td>\n",
       "      <td>0.857126</td>\n",
       "      <td>0.918484</td>\n",
       "      <td>0.198050</td>\n",
       "      <td>0.134153</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>total_bedrooms</th>\n",
       "      <td>0.069608</td>\n",
       "      <td>-0.066983</td>\n",
       "      <td>-0.320451</td>\n",
       "      <td>0.930380</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.877747</td>\n",
       "      <td>0.979728</td>\n",
       "      <td>-0.007723</td>\n",
       "      <td>0.049686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>population</th>\n",
       "      <td>0.099773</td>\n",
       "      <td>-0.108785</td>\n",
       "      <td>-0.296244</td>\n",
       "      <td>0.857126</td>\n",
       "      <td>0.877747</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.907222</td>\n",
       "      <td>0.004834</td>\n",
       "      <td>-0.024650</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>households</th>\n",
       "      <td>0.055310</td>\n",
       "      <td>-0.071035</td>\n",
       "      <td>-0.302916</td>\n",
       "      <td>0.918484</td>\n",
       "      <td>0.979728</td>\n",
       "      <td>0.907222</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.013033</td>\n",
       "      <td>0.065843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>median_income</th>\n",
       "      <td>-0.015176</td>\n",
       "      <td>-0.079809</td>\n",
       "      <td>-0.119034</td>\n",
       "      <td>0.198050</td>\n",
       "      <td>-0.007723</td>\n",
       "      <td>0.004834</td>\n",
       "      <td>0.013033</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.688075</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>median_house_value</th>\n",
       "      <td>-0.045967</td>\n",
       "      <td>-0.144160</td>\n",
       "      <td>0.105623</td>\n",
       "      <td>0.134153</td>\n",
       "      <td>0.049686</td>\n",
       "      <td>-0.024650</td>\n",
       "      <td>0.065843</td>\n",
       "      <td>0.688075</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    longitude  latitude  housing_median_age  total_rooms  \\\n",
       "longitude            1.000000 -0.924664           -0.108197     0.044568   \n",
       "latitude            -0.924664  1.000000            0.011173    -0.036100   \n",
       "housing_median_age  -0.108197  0.011173            1.000000    -0.361262   \n",
       "total_rooms          0.044568 -0.036100           -0.361262     1.000000   \n",
       "total_bedrooms       0.069608 -0.066983           -0.320451     0.930380   \n",
       "population           0.099773 -0.108785           -0.296244     0.857126   \n",
       "households           0.055310 -0.071035           -0.302916     0.918484   \n",
       "median_income       -0.015176 -0.079809           -0.119034     0.198050   \n",
       "median_house_value  -0.045967 -0.144160            0.105623     0.134153   \n",
       "\n",
       "                    total_bedrooms  population  households  median_income  \\\n",
       "longitude                 0.069608    0.099773    0.055310      -0.015176   \n",
       "latitude                 -0.066983   -0.108785   -0.071035      -0.079809   \n",
       "housing_median_age       -0.320451   -0.296244   -0.302916      -0.119034   \n",
       "total_rooms               0.930380    0.857126    0.918484       0.198050   \n",
       "total_bedrooms            1.000000    0.877747    0.979728      -0.007723   \n",
       "population                0.877747    1.000000    0.907222       0.004834   \n",
       "households                0.979728    0.907222    1.000000       0.013033   \n",
       "median_income            -0.007723    0.004834    0.013033       1.000000   \n",
       "median_house_value        0.049686   -0.024650    0.065843       0.688075   \n",
       "\n",
       "                    median_house_value  \n",
       "longitude                    -0.045967  \n",
       "latitude                     -0.144160  \n",
       "housing_median_age            0.105623  \n",
       "total_rooms                   0.134153  \n",
       "total_bedrooms                0.049686  \n",
       "population                   -0.024650  \n",
       "households                    0.065843  \n",
       "median_income                 0.688075  \n",
       "median_house_value            1.000000  "
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corr = housing.corr(numeric_only=True)\n",
    "corr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "2a532ab6be85ff5d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:31:03.755612Z",
     "start_time": "2025-02-07T16:31:03.752852Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "median_house_value    1.000000\n",
      "median_income         0.688075\n",
      "total_rooms           0.134153\n",
      "housing_median_age    0.105623\n",
      "households            0.065843\n",
      "total_bedrooms        0.049686\n",
      "population           -0.024650\n",
      "longitude            -0.045967\n",
      "latitude             -0.144160\n",
      "Name: median_house_value, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print(corr[\"median_house_value\"].sort_values(ascending=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "d95b7506e92f33a8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-02-07T16:31:20.335946Z",
     "start_time": "2025-02-07T16:31:20.332747Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "median_house_value    1.000000\n",
      "median_income         0.688075\n",
      "latitude              0.144160\n",
      "total_rooms           0.134153\n",
      "housing_median_age    0.105623\n",
      "households            0.065843\n",
      "total_bedrooms        0.049686\n",
      "longitude             0.045967\n",
      "population            0.024650\n",
      "Name: median_house_value, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print(corr[\"median_house_value\"].abs().sort_values(ascending=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b30eccf2-d51f-45ff-aaf7-db44e4c41dfb",
   "metadata": {},
   "source": "## 3. Extra exercises on pandas"
  },
  {
   "cell_type": "markdown",
   "id": "4baa6183-ff69-450f-9824-6ad3d830f18d",
   "metadata": {},
   "source": [
    "Many exercises here : https://github.com/guipsamora/pandas_exercises/tree/master\n",
    "(with  solutions)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
