{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8797cb42",
   "metadata": {},
   "source": [
    "# Step 1 — Initial loading and exploration\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "391c54d2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(116, 10)\n",
      "   Age        BMI  Glucose  Insulin      HOMA   Leptin  Adiponectin  Resistin  \\\n",
      "0   48  23.500000       70    2.707  0.467409   8.8071     9.702400   7.99585   \n",
      "1   83  20.690495       92    3.115  0.706897   8.8438     5.429285   4.06405   \n",
      "2   82  23.124670       91    4.498  1.009651  17.9393    22.432040   9.27715   \n",
      "3   68  21.367521       77    3.226  0.612725   9.8827     7.169560  12.76600   \n",
      "4   86  21.111111       92    3.549  0.805386   6.6994     4.819240  10.57635   \n",
      "\n",
      "     MCP.1  Classification  \n",
      "0  417.114               1  \n",
      "1  468.786               1  \n",
      "2  554.697               1  \n",
      "3  928.220               1  \n",
      "4  773.920               1  \n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 116 entries, 0 to 115\n",
      "Data columns (total 10 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   Age             116 non-null    int64  \n",
      " 1   BMI             116 non-null    float64\n",
      " 2   Glucose         116 non-null    int64  \n",
      " 3   Insulin         116 non-null    float64\n",
      " 4   HOMA            116 non-null    float64\n",
      " 5   Leptin          116 non-null    float64\n",
      " 6   Adiponectin     116 non-null    float64\n",
      " 7   Resistin        116 non-null    float64\n",
      " 8   MCP.1           116 non-null    float64\n",
      " 9   Classification  116 non-null    int64  \n",
      "dtypes: float64(7), int64(3)\n",
      "memory usage: 9.2 KB\n",
      "The explanatory variables are: ['Age', 'BMI', 'Glucose', 'Insulin', 'HOMA', 'Leptin', 'Adiponectin', 'Resistin', 'MCP.1']\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 78036 characters]",
      "text/plain": [
       "<Figure size 1200x800 with 12 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Import libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# Load data\n",
    "data = pd.read_csv('breast+cancer+coimbra/dataR2.csv')\n",
    "\n",
    "# Data overview\n",
    "print(data.shape)\n",
    "print(data.head())\n",
    "data.info()\n",
    "\n",
    "# List of explanatory variables\n",
    "features = data.columns[:-1]\n",
    "print(\"The explanatory variables are:\", list(features))\n",
    "target = 'Classification'\n",
    "\n",
    "# Histograms of variables\n",
    "data.hist(bins=50, figsize=(12, 8))\n",
    "plt.suptitle(\"Histograms of variables\", fontsize=16)\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "1c07b059",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification\n",
      "2    0.551724\n",
      "1    0.448276\n",
      "Name: proportion, dtype: float64\n",
      "the dataset is not balanced\n"
     ]
    }
   ],
   "source": [
    "print(data[\"Classification\"].value_counts(normalize=True))\n",
    "print(\"the dataset is not balanced\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a62b9137",
   "metadata": {},
   "source": [
    "We use data from the *Breast Cancer Coimbra* dataset, which contains biometric measurements of 116 patients. The goal is to predict whether a patient has breast cancer or not, based on nine biological variables.\n",
    "\n",
    "### Dataset overview\n",
    "\n",
    "The dataset contains 116 observations and 10 variables, including:\n",
    "- **9 continuous explanatory variables**: `Age`, `BMI`, `Glucose`, `Insulin`, `HOMA`, `Leptin`, `Adiponectin`, `Resistin`, `MCP.1`.\n",
    "- **1 target variable**: `Classification`, which takes the value `2` if the patient has cancer (positive) and `1` otherwise (negative).\n",
    "\n",
    "There are no missing values in the data, allowing us to proceed directly with exploratory analysis.\n",
    "\n",
    "### Variable distributions\n",
    "\n",
    "The distribution of the variables was visualized using histograms:\n",
    "\n",
    "- Some variables exhibit **asymmetric distributions** with extreme values, notably `Insulin`, `HOMA`, `Leptin`, `Adiponectin`, `Resistin`, and `MCP.1`. This may motivate **data transformation** or **normalization** prior to modeling.\n",
    "- The `Classification` variable is **imbalanced**, with a majority of healthy patients (`2`). This is important information as it will affect the **choice of evaluation metrics**: simple accuracy will not be sufficient to assess model quality.\n",
    "\n",
    "We will now prepare the data for model training by separating features and target and performing a **stratified split** of the dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cb261cd",
   "metadata": {},
   "source": [
    "# Step 2 — Split the dataset into training and testing sets\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "3eddff01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: (92, 9) (92,)\n",
      "Test : (24, 9) (24,) \n",
      "\n",
      "Train class distribution: Classification\n",
      "1    0.554348\n",
      "0    0.445652\n",
      "Name: proportion, dtype: float64 \n",
      "\n",
      "Test class distribution : Classification\n",
      "1    0.541667\n",
      "0    0.458333\n",
      "Name: proportion, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Separate features and target variable\n",
    "X = data.drop(columns='Classification')\n",
    "y = data['Classification'].map({2: 1, 1: 0})\n",
    "# Mapping: 2 = patient with cancer (positive class), 1 = healthy patient (negative class)\n",
    "# We recode to: 1 → positive, 0 → negative\n",
    "\n",
    "# Stratified split to preserve class distribution\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y,\n",
    "    test_size=0.2,\n",
    "    random_state=42,\n",
    "    stratify=y)\n",
    "\n",
    "# Check shapes of the splits\n",
    "print(\"Train:\", X_train.shape, y_train.shape)\n",
    "print(\"Test :\", X_test.shape, y_test.shape, \"\\n\")\n",
    "\n",
    "print(\"Train class distribution:\", y_train.value_counts(normalize=True), \"\\n\")\n",
    "print(\"Test class distribution :\", y_test.value_counts(normalize=True))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2eec1d6e",
   "metadata": {},
   "source": [
    "We start by separating the explanatory variables (features) from the target variable (`Classification`), recoding the latter to facilitate result interpretation:\n",
    "- `1`: patient with cancer (positive class),\n",
    "- `0`: healthy patient (negative class).\n",
    "\n",
    "A **stratified split** was performed using the `train_test_split` function from `scikit-learn` to preserve the original class proportions in both training and testing sets. This is particularly important given the observed **class imbalance**.\n",
    "\n",
    "The resulting dataset sizes are:\n",
    "- **Training set**: 92 observations\n",
    "- **Test set**: 24 observations\n",
    "\n",
    "This corresponds to a classical 80% / 20% split.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02d2e734",
   "metadata": {},
   "source": [
    "# Step 3 — Visualization of distributions by class in the training dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "ac59107b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 119276 characters]",
      "text/plain": [
       "<Figure size 1500x1200 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 152692 characters]",
      "text/plain": [
       "<Figure size 1500x1200 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 248336 characters]",
      "text/plain": [
       "<Figure size 1500x1200 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Temporary merge for visualization\n",
    "train_data = X_train.copy()\n",
    "train_data['Classification'] = y_train\n",
    "\n",
    "variables = X_train.columns.tolist()\n",
    "n_cols = 3\n",
    "n_rows = int(np.ceil(len(variables) / n_cols))\n",
    "class_labels = {1: 'Patient with cancer', 0: 'Healthy patient'}\n",
    "\n",
    "# Histograms by class\n",
    "fig, axes = plt.subplots(n_rows, n_cols, figsize=(5 * n_cols, 4 * n_rows))\n",
    "axes = axes.flatten()\n",
    "\n",
    "for i, col in enumerate(variables):\n",
    "    for cls in [1, 0]:  # 1 = malade, 0 = sain\n",
    "        axes[i].hist(X_train[y_train == cls][col], bins=30, alpha=0.5, label=class_labels[cls])\n",
    "    axes[i].set_title(f\"{col} by class\")\n",
    "    axes[i].legend()\n",
    "\n",
    "# Supprimer axes inutilisés\n",
    "for j in range(i + 1, len(axes)):\n",
    "    fig.delaxes(axes[j])\n",
    "\n",
    "fig.suptitle(\"Histograms by class (before transformation)\", fontsize=16)\n",
    "plt.tight_layout(rect=[0, 0, 1, 0.97])\n",
    "plt.show()\n",
    "\n",
    "# Boxplots by class\n",
    "fig, axes = plt.subplots(n_rows, n_cols, figsize=(5 * n_cols, 4 * n_rows))\n",
    "axes = axes.flatten()\n",
    "\n",
    "for i, col in enumerate(variables):\n",
    "    sns.boxplot(x=y_train, y=X_train[col], ax=axes[i])\n",
    "    axes[i].set_title(f\"Boxplot of {col}\")\n",
    "    axes[i].set_xlabel(\"Class\")\n",
    "    axes[i].set_ylabel(col)\n",
    "    axes[i].set_xticks([0, 1])\n",
    "    axes[i].set_xticklabels([class_labels[0], class_labels[1]])\n",
    "\n",
    "for j in range(i + 1, len(axes)):\n",
    "    fig.delaxes(axes[j])\n",
    "\n",
    "fig.suptitle(\"Boxplots by class (before transformation)\", fontsize=16)\n",
    "plt.tight_layout(rect=[0, 0, 1, 0.97])\n",
    "plt.show()\n",
    "\n",
    "# Jittered scatter plots by class\n",
    "fig, axes = plt.subplots(n_rows, n_cols, figsize=(5 * n_cols, 4 * n_rows))\n",
    "axes = axes.flatten()\n",
    "\n",
    "for i, col in enumerate(variables):\n",
    "    for cls in [1, 0]:\n",
    "        axes[i].scatter(\n",
    "            X_train[y_train == cls][col],\n",
    "            np.random.normal(loc=cls, scale=0.02, size=X_train[y_train == cls][col].shape),\n",
    "            alpha=0.5,\n",
    "            label=class_labels[cls]\n",
    "        )\n",
    "    axes[i].set_title(f\"{col} (jittered scatter plot)\")\n",
    "    axes[i].set_ylabel(\"Class (jittered)\")\n",
    "    axes[i].legend()\n",
    "\n",
    "for j in range(i + 1, len(axes)):\n",
    "    fig.delaxes(axes[j])\n",
    "\n",
    "fig.suptitle(\"Jittered scatter plots by class (before transformation)\", fontsize=16)\n",
    "plt.tight_layout(rect=[0, 0, 1, 0.97])\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "066377f6",
   "metadata": {},
   "source": [
    "To better understand the differences between patients with cancer (Class 1) and healthy patients (Class 0), we represented the distributions of explanatory variables segmented by class. Three complementary types of visualizations were produced: histograms, boxplots, and jittered scatter plots.\n",
    "\n",
    "### Histograms by class\n",
    "\n",
    "Histograms allow us to observe the distribution of values within each class. Several variables show notable shifts between the two groups, notably:\n",
    "- **Age**: patients with cancer tend to be slightly older.\n",
    "- **Resistin**, **MCP.1**, and **Adiponectin** appear to exhibit density shifts between classes, suggesting potential discriminative power.\n",
    "- Conversely, variables such as **BMI** or **Leptin** show strong overlap between classes.\n",
    "\n",
    "These qualitative observations suggest that some variables may be informative, but **none alone allow a clear separation**.\n",
    "\n",
    "### Boxplots by class\n",
    "\n",
    "Boxplots provide a more precise visual assessment of distribution differences:\n",
    "- Most variables display **high dispersion** with numerous **outliers** visible in both classes, particularly for `Insulin`, `Leptin`, and `MCP.1`.\n",
    "- **Medians are close**, but some trends differ (e.g., `Resistin` is higher for class 1).\n",
    "\n",
    "These elements indicate **high intra-class variance**, which may affect the performance of models sensitive to unstandardized scales.\n",
    "\n",
    "### Jittered scatter plots\n",
    "\n",
    "These visualizations explicitly show vertical separation between classes (0 or 1):\n",
    "- No single variable fully separates the classes.\n",
    "- Some clusters appear (e.g., low values of `Insulin`, `HOMA`, `Adiponectin` associated with class 0), but with **significant overlap**.\n",
    "\n",
    "---\n",
    "\n",
    "### Summary of exploratory analysis\n",
    "\n",
    "- Variables seem to contain **partial signals**, but will need to be **combined in a multivariate model** to best exploit interactions.\n",
    "- The presence of **outliers**, **asymmetric distributions**, and **variables on very different scales** justifies:\n",
    "  - **prior standardization** of variables,\n",
    "  - and possibly a **logarithmic transformation** for some (e.g., `Insulin`, `Leptin`, `HOMA`).\n",
    "- Finally, the strong **inter-class overlap** implies that linear separation may be limited, motivating the use of nonlinear models.\n",
    "\n",
    "This exploratory step thus rigorously motivates the choices of **preprocessing** and **supervised modeling** to follow.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09e7c056",
   "metadata": {},
   "source": [
    "# Step 4 — Correlation matrix and scatter matrix (training set only)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "9ff82466",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 111040 characters]",
      "text/plain": [
       "<Figure size 1000x800 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 401548 characters]",
      "text/plain": [
       "<Figure size 1200x800 with 81 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pandas.plotting import scatter_matrix\n",
    "from matplotlib.colors import ListedColormap\n",
    "\n",
    "\n",
    "# Correlation matrix (on training data)\n",
    "corr_matrix = X_train.corr()\n",
    "\n",
    "plt.figure(figsize=(10, 8))\n",
    "sns.heatmap(corr_matrix, annot=True, fmt=\".2f\", cmap='coolwarm', square=True)\n",
    "plt.suptitle(\"Correlation matrix (X_train)\", fontsize=14)\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Combined DataFrame for colored visualization\n",
    "X_train_visu = X_train.copy()\n",
    "X_train_visu['Classification'] = y_train.values\n",
    "\n",
    "# Custom color map: 0 = Healthy (light red), 1 = Cancer (blue)\n",
    "custom_cmap = ListedColormap([\"#FFA07A\", \"#1E90FF\"])  # light salmon, dodger blue\n",
    "\n",
    "# Colored scatter matrix\n",
    "scatter_matrix(X_train_visu.drop(columns='Classification'),\n",
    "               figsize=(12, 8),\n",
    "               diagonal='kde',\n",
    "               c=X_train_visu['Classification'],\n",
    "               cmap=custom_cmap,\n",
    "               alpha=0.7)\n",
    "plt.suptitle(\"Colored scatter matrix (X_train)\", y=1.02)\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a358c6aa",
   "metadata": {},
   "source": [
    "To examine potential redundancy between variables and anticipate multicollinearity risks, we studied the linear relationships among explanatory variables using two complementary tools:\n",
    "\n",
    "### Correlation matrix\n",
    "\n",
    "The correlation matrix highlights linear dependencies between continuous variables. Notably, we observe:\n",
    "\n",
    "- A **very strong correlation** between `Insulin` and `HOMA` (ρ ≈ 0.93), which is expected since HOMA is a deterministic function of insulin and glucose (`HOMA = (Glucose × Insulin) / 405`).\n",
    "- Moderate correlations between:\n",
    "  - `Glucose` and `HOMA` (ρ ≈ 0.70),\n",
    "  - `BMI` and `Leptin` (ρ ≈ 0.60).\n",
    "- Other variable pairs show relatively low correlations (ρ < 0.4), suggesting that **most features provide complementary information**.\n",
    "\n",
    "This analysis motivates the use of **dimensionality reduction** or **feature selection**, especially with models sensitive to collinearity such as logistic regression.\n",
    "\n",
    "### Colored scatter matrix\n",
    "\n",
    "The scatter matrix, colored by class (`Classification`), offers a global view of pairwise interactions while highlighting group structures by class.\n",
    "\n",
    "- The **color scheme is adjusted** so that blue points represent patients with cancer and red points represent healthy individuals.\n",
    "- There is **substantial overlap** between the two classes in most 2D projections.\n",
    "- Some variable pairs—such as (`Resistin`, `MCP.1`) or (`Adiponectin`, `HOMA`)—show partial separation but no clearly discriminative boundary.\n",
    "\n",
    "---\n",
    "\n",
    "### Conclusion of this step\n",
    "\n",
    "- There is no **obvious linear separation** between classes based on individual variable pairs.\n",
    "- The dataset structure suggests that multivariate models may benefit from capturing **nonlinear** or **interaction-based** patterns.\n",
    "- **Standardization** is required before applying scale-sensitive algorithms such as SVM, k-NN, or PCA.\n",
    "\n",
    "We now proceed to the preprocessing step (data standardization) before comparing different supervised learning models.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "556f426b",
   "metadata": {},
   "source": [
    "# Step 5 — Logarithmic transformation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "af21af16",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 44992 characters]",
      "text/plain": [
       "<Figure size 1200x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 47788 characters]",
      "text/plain": [
       "<Figure size 1200x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 45488 characters]",
      "text/plain": [
       "<Figure size 1200x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 48652 characters]",
      "text/plain": [
       "<Figure size 1200x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 103904 characters]",
      "text/plain": [
       "<Figure size 1500x1200 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 130616 characters]",
      "text/plain": [
       "<Figure size 1500x1200 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 219492 characters]",
      "text/plain": [
       "<Figure size 1500x1200 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# List of variables to transform (asymmetric distributions)\n",
    "variables_log = ['Insulin', 'HOMA', 'MCP.1', 'Resistin']\n",
    "\n",
    "# Comparison before / after transformation (1 figure per variable)\n",
    "for col in variables_log:\n",
    "    fig, axes = plt.subplots(1, 2, figsize=(12, 4))\n",
    "\n",
    "    # Before transformation\n",
    "    axes[0].hist(X_train[col], bins=30, alpha=0.6, color='salmon', edgecolor='black')\n",
    "    axes[0].set_title(f\"{col} (before transformation)\")\n",
    "    axes[0].set_xlabel(col)\n",
    "    axes[0].set_ylabel(\"Frequency\")\n",
    "\n",
    "    # After log1p transformation\n",
    "    transformed = np.log1p(X_train[col])\n",
    "    axes[1].hist(transformed, bins=30, alpha=0.6, color='steelblue', edgecolor='black')\n",
    "    axes[1].set_title(f\"{col} after log({col} + 1) transformation\")\n",
    "    axes[1].set_xlabel(f\"log1p({col})\")\n",
    "    axes[1].set_ylabel(\"Frequency\")\n",
    "\n",
    "    plt.suptitle(f\"Comparison before / after transformation: {col}\", fontsize=14)\n",
    "    plt.tight_layout(rect=[0, 0, 1, 0.93])\n",
    "    plt.show()\n",
    "\n",
    "# Apply the transformation to X_train\n",
    "X_train_transformed = X_train.copy()\n",
    "for col in variables_log:\n",
    "    X_train_transformed[col + '_log'] = np.log1p(X_train[col])\n",
    "X_train_transformed = X_train_transformed.drop(columns=variables_log)\n",
    "\n",
    "# Update the list of transformed features\n",
    "features = X_train_transformed.columns.tolist()\n",
    "\n",
    "# Merge with y_train to facilitate visualization\n",
    "train_data2 = X_train_transformed.copy()\n",
    "train_data2['Classification'] = y_train.values\n",
    "\n",
    "# Histograms by class\n",
    "n_cols = 3\n",
    "n_rows = int(np.ceil(len(features) / n_cols))\n",
    "\n",
    "fig, axes = plt.subplots(n_rows, n_cols, figsize=(5 * n_cols, 4 * n_rows))\n",
    "axes = axes.flatten()\n",
    "\n",
    "for i, col in enumerate(features):\n",
    "    for cls in [0, 1]:\n",
    "        axes[i].hist(train_data2[train_data2['Classification'] == cls][col],\n",
    "                     bins=30, alpha=0.5, label=f\"Class {cls}\")\n",
    "    axes[i].set_title(f\"{col} by class\")\n",
    "    axes[i].legend()\n",
    "\n",
    "for j in range(i + 1, len(axes)):\n",
    "    fig.delaxes(axes[j])\n",
    "\n",
    "fig.suptitle(\"Histograms by class (after transformation)\", fontsize=16)\n",
    "plt.tight_layout(rect=[0, 0, 1, 0.97])\n",
    "plt.show()\n",
    "\n",
    "# Boxplots by class\n",
    "\n",
    "fig, axes = plt.subplots(n_rows, n_cols, figsize=(5 * n_cols, 4 * n_rows))\n",
    "axes = axes.flatten()\n",
    "\n",
    "for i, col in enumerate(features):\n",
    "    sns.boxplot(x='Classification', y=col, data=train_data2, ax=axes[i])\n",
    "    axes[i].set_title(f\"Boxplot of {col}\")\n",
    "    axes[i].set_xlabel(\"Class\")\n",
    "    axes[i].set_ylabel(col)\n",
    "\n",
    "for j in range(i + 1, len(axes)):\n",
    "    fig.delaxes(axes[j])\n",
    "\n",
    "fig.suptitle(\"Boxplots by class (after transformation)\", fontsize=16)\n",
    "plt.tight_layout(rect=[0, 0, 1, 0.97])\n",
    "plt.show()\n",
    "\n",
    "# Jittered scatter plots by class\n",
    "\n",
    "fig, axes = plt.subplots(n_rows, n_cols, figsize=(5 * n_cols, 4 * n_rows))\n",
    "axes = axes.flatten()\n",
    "\n",
    "for i, col in enumerate(features):\n",
    "    for cls in [0, 1]:\n",
    "        values = train_data2[train_data2['Classification'] == cls][col]\n",
    "        jittered_y = np.random.normal(loc=cls, scale=0.02, size=values.shape)\n",
    "        axes[i].scatter(values, jittered_y, alpha=0.5, label=f\"Class {cls}\")\n",
    "    axes[i].set_title(f\"{col} (jittered scatter plot)\")\n",
    "    axes[i].legend()\n",
    "\n",
    "for j in range(i + 1, len(axes)):\n",
    "    fig.delaxes(axes[j])\n",
    "\n",
    "fig.suptitle(\"Jittered scatter plots by class (after transformation)\", fontsize=16)\n",
    "plt.tight_layout(rect=[0, 0, 1, 0.97])\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c582a520",
   "metadata": {},
   "source": [
    "### Objective\n",
    "\n",
    "The initial exploratory analysis revealed a **marked skewness** (distribution heavily right-skewed) for several variables: `Insulin`, `HOMA`, `MCP.1`, and `Resistin`.  \n",
    "These distributions included extreme values that could hinder the performance of models sensitive to scale differences or non-normality (e.g., logistic regression, k-NN, SVM).\n",
    "\n",
    "To make these distributions more symmetric and reduce the impact of extreme values, we applied a **logarithmic transformation** of the form `log(x + 1)` using the `np.log1p` function.\n",
    "\n",
    "---\n",
    "\n",
    "### Comparison before / after transformation\n",
    "\n",
    "For each transformed variable, we compared the distribution **before and after transformation** using histograms:\n",
    "\n",
    "- The long right tails are **strongly reduced**.\n",
    "- The distributions become **more compact, less skewed**, and often closer to a Gaussian shape.\n",
    "- This improves data homogeneity and facilitates subsequent **standardization** steps.\n",
    "\n",
    "These visualizations provide a **clear justification** for the use of logarithmic transformation.\n",
    "\n",
    "---\n",
    "\n",
    "### Dataset update\n",
    "\n",
    "The new transformed variables (`Insulin_log`, `HOMA_log`, `MCP.1_log`, `Resistin_log`) were added to the dataset, and the original variables were removed to avoid redundancy.\n",
    "\n",
    "---\n",
    "\n",
    "### Effect of transformation on class-separated distributions\n",
    "\n",
    "To observe the effect of this transformation on the distribution of values within the two classes (Class 0: healthy, Class 1: cancer), we reproduced several visualizations on the transformed variables:\n",
    "\n",
    "#### • Histograms by class  \n",
    "These histograms illustrate relative densities within each class after transformation.  \n",
    "Some variables, notably `Insulin_log` and `Resistin_log`, still exhibit class-dependent differences, but within a **much more regular distribution**.\n",
    "\n",
    "#### • Boxplots by class  \n",
    "The log transformation clearly reduces the impact of **visible outliers** in the boxplots.  \n",
    "Medians are more representative, and **interquartile ranges are narrower and more stable**.\n",
    "\n",
    "#### • Jittered scatter plots  \n",
    "These visualizations confirm that the classes remain vertically separable, while the **horizontal dispersion is more controlled**, which benefits distance-based models.\n",
    "\n",
    "---\n",
    "\n",
    "### Conclusion\n",
    "\n",
    "Applying the logarithmic transformation made it possible to:\n",
    "- Reduce the effect of extreme values,\n",
    "- Stabilize variance,\n",
    "- Improve the geometric structure of the dataset in preparation for standardization and model training.\n",
    "\n",
    "These transformations constitute an **essential step in the preprocessing pipeline** prior to supervised learning.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "184a592b",
   "metadata": {},
   "source": [
    "# Step 6 — Standardization (StandardScaler)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "4b77fbe5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Age</th>\n",
       "      <td>-2.003229e-16</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BMI</th>\n",
       "      <td>-6.389816e-16</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Glucose</th>\n",
       "      <td>8.326673e-17</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Leptin</th>\n",
       "      <td>-5.068409e-17</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Adiponectin</th>\n",
       "      <td>8.839547e-17</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Insulin_log</th>\n",
       "      <td>-1.194697e-16</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HOMA_log</th>\n",
       "      <td>-2.256649e-16</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MCP.1_log</th>\n",
       "      <td>1.128324e-15</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Resistin_log</th>\n",
       "      <td>6.552729e-16</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      mean       std\n",
       "Age          -2.003229e-16  1.005479\n",
       "BMI          -6.389816e-16  1.005479\n",
       "Glucose       8.326673e-17  1.005479\n",
       "Leptin       -5.068409e-17  1.005479\n",
       "Adiponectin   8.839547e-17  1.005479\n",
       "Insulin_log  -1.194697e-16  1.005479\n",
       "HOMA_log     -2.256649e-16  1.005479\n",
       "MCP.1_log     1.128324e-15  1.005479\n",
       "Resistin_log  6.552729e-16  1.005479"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Initialize the scaler\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Fit the scaler on the transformed training set and transform\n",
    "X_train_scaled = pd.DataFrame(\n",
    "    scaler.fit_transform(X_train_transformed),\n",
    "    columns=X_train_transformed.columns,\n",
    "    index=X_train_transformed.index\n",
    ")\n",
    "\n",
    "# Display summary statistics of scaled data\n",
    "X_train_scaled.describe().T[['mean', 'std']]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e525eb2",
   "metadata": {},
   "source": [
    "### Objective\n",
    "\n",
    "After applying logarithmic transformation to asymmetric variables, we apply **standardization** to all explanatory variables in the training set.  \n",
    "This step aims to **bring all variables to the same scale** by centering them around zero and scaling them to unit variance.\n",
    "\n",
    "Standardization is particularly important for models sensitive to distances or scale differences, such as:\n",
    "- logistic regression (for more stable convergence),\n",
    "- support vector machines (SVM),\n",
    "- k-nearest neighbors (k-NN),\n",
    "- regularization methods (Ridge, Lasso),\n",
    "- and dimensionality reduction techniques (PCA, etc.).\n",
    "\n",
    "---\n",
    "### Method\n",
    "\n",
    "The transformation used is the classical Z-score standardization, defined as:\n",
    "\n",
    "$$\n",
    "X^{(i)}_{\\text{std}} = \\frac{X^{(i)} - \\mu^{(i)}}{\\sigma^{(i)}}\n",
    "$$\n",
    "\n",
    "where \\( \\mu^{(i)} \\) and \\( \\sigma^{(i)} \\) denote respectively the mean and standard deviation of variable \\( X^{(i)} \\) computed on the training set.\n",
    "\n",
    "We use the `StandardScaler` tool from `scikit-learn`, applied only on the training data (`fit_transform`) to avoid any data leakage.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d374b93f",
   "metadata": {},
   "source": [
    "# Step 7 — Preparation of X_test\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "28a4ffe0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy X_test\n",
    "X_test_transformed = X_test.copy()\n",
    "\n",
    "# Apply the same log-transformations\n",
    "for col in variables_log:\n",
    "    X_test_transformed[col + '_log'] = np.log1p(X_test_transformed[col])\n",
    "\n",
    "# Drop original columns\n",
    "X_test_transformed = X_test_transformed.drop(columns=variables_log)\n",
    "\n",
    "# Standardize using the scaler previously fitted on X_train\n",
    "X_test_scaled = pd.DataFrame(\n",
    "    scaler.transform(X_test_transformed),\n",
    "    columns=X_test_transformed.columns,\n",
    "    index=X_test_transformed.index\n",
    ")\n",
    "\n",
    "# Check that the columns of X_test are identical (and in the same order) as those of X_train to avoid prediction errors\n",
    "if list(X_train_scaled.columns) != list(X_test_scaled.columns):\n",
    "    print(\"Mismatch between train and test columns\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4a5689a",
   "metadata": {},
   "source": [
    "The transformations previously applied to the training set (logarithmic transformation + standardization) must be **reproduced exactly the same way** on the test set, without re-estimating any parameters.\n",
    "\n",
    "### Steps performed:\n",
    "1. **Logarithmic transformation** on the same variables (`Insulin`, `HOMA`, `MCP.1`, `Resistin`) using `log(x + 1)`;\n",
    "2. **Removal of raw variables** to avoid redundancy;\n",
    "3. **Standardization using the same parameters** (mean `μ`, standard deviation `σ`) learned from `X_train`, via `scaler.transform()`.\n",
    "\n",
    "We also verified that the **columns of `X_test_scaled` are exactly the same (and in the same order)** as those of `X_train_scaled`, ensuring compatibility with the learning models.\n",
    "\n",
    "---\n",
    "\n",
    "This step completes the preprocessing pipeline and allows us to proceed to **supervised modeling**.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fd9b3f0",
   "metadata": {},
   "source": [
    "# Step 8 — Modeling with LogisticRegression\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "d58417ad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation of simple model (logistic regression)\n",
      "Recall: 0.692\n",
      "F1-score: 0.75\n",
      "\n",
      "Classification report:\n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.69      0.82      0.75        11\n",
      "           1       0.82      0.69      0.75        13\n",
      "\n",
      "    accuracy                           0.75        24\n",
      "   macro avg       0.76      0.76      0.75        24\n",
      "weighted avg       0.76      0.75      0.75        24\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import accuracy_score, f1_score, classification_report,recall_score\n",
    "\n",
    "# Initialize and train the model without explicit regularization\n",
    "logreg_simple = LogisticRegression(random_state=42)\n",
    "logreg_simple.fit(X_train_scaled, y_train)\n",
    "\n",
    "# Predictions on the test set\n",
    "y_pred_simple = logreg_simple.predict(X_test_scaled)\n",
    "\n",
    "# Basic performance evaluation\n",
    "print(\"Evaluation of simple model (logistic regression)\")\n",
    "print(\"Recall:\", round(recall_score(y_test, y_pred_simple), 3)) \n",
    "print(\"F1-score:\", round(f1_score(y_test, y_pred_simple), 3))\n",
    "print(\"\\nClassification report:\\n\", classification_report(y_test, y_pred_simple))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76d3da3a",
   "metadata": {},
   "source": [
    "We begin by training a simple logistic regression model, without explicit regularization, on the preprocessed dataset (after logarithmic transformation and standardization).\n",
    "\n",
    "### Objective\n",
    "\n",
    "The goal is to predict the probability that a patient has cancer based on the nine explanatory variables.  \n",
    "Logistic regression is a widely used, interpretable linear model, particularly suited as a baseline for supervised classification tasks.\n",
    "\n",
    "### Results on the test set\n",
    "\n",
    "The model was evaluated on the test set after training on the standardized data. The results are as follows:\n",
    "\n",
    "- **Recall**: 0.692  \n",
    "- **F1-score**: 0.750\n",
    "\n",
    "### Classification report details:\n",
    "\n",
    "| Class        | Precision | Recall | F1-score | Support |\n",
    "|--------------|-----------|--------|----------|---------|\n",
    "| 0 (healthy)  | 0.69      | 0.82   | 0.75     | 11      |\n",
    "| 1 (cancer)   | 0.82      | 0.69   | 0.75     | 13      |\n",
    "\n",
    "- **Macro avg F1**: 0.75  \n",
    "- **Weighted avg F1**: 0.75\n",
    "\n",
    "These results show a reasonable trade-off between precision and recall, which is particularly important in a medical classification task.  \n",
    "However, this simple model does not yet account for variable interactions, non-linearity, or hyperparameter tuning—these aspects will be addressed in the following steps.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b7c1cdf",
   "metadata": {},
   "source": [
    "# Step 9 — Cross-validation on training data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "d8804f23",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10-fold cross-validation\n",
      "Mean F1-score : 0.777 ± 0.152\n",
      "Mean Recall : 0.800 ± 0.200\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 63432 characters]",
      "text/plain": [
       "<Figure size 800x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# 10-fold cross-validation on the simple model\n",
    "f1_scores = cross_val_score(logreg_simple, X_train_scaled, y_train, cv=10, scoring='f1')\n",
    "rec_scores = cross_val_score(logreg_simple, X_train_scaled, y_train, cv=10, scoring='recall')\n",
    "\n",
    "# Summary\n",
    "print(\"10-fold cross-validation\")\n",
    "print(f\"Mean F1-score : {f1_scores.mean():.3f} ± {f1_scores.std():.3f}\")\n",
    "print(f\"Mean Recall : {rec_scores.mean():.3f} ± {rec_scores.std():.3f}\")\n",
    "\n",
    "# Visualization of scores by fold\n",
    "folds = range(1, len(f1_scores) + 1)\n",
    "plt.figure(figsize=(8, 5))\n",
    "plt.plot(folds, f1_scores, marker='o', label='F1-score')\n",
    "plt.plot(folds, rec_scores, marker='s', label='Recall')\n",
    "plt.title(\"Scores by fold (cross-validation)\")\n",
    "plt.xlabel(\"Fold\")\n",
    "plt.ylabel(\"Score\")\n",
    "plt.ylim(0, 1)\n",
    "plt.grid(True)\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8407908d",
   "metadata": {},
   "source": [
    "To evaluate the stability of the logistic regression model and its generalization capacity, we performed a **10-fold cross-validation** (k-fold CV).\n",
    "\n",
    "Two main metrics were monitored: the **F1-score** (suitable for imbalanced class contexts) and **recall** (sensitivity or true positive rate).\n",
    "\n",
    "---\n",
    "\n",
    "### Cross-validation results\n",
    "\n",
    "- **Mean F1-score**: 0.777 ± 0.152  \n",
    "- **Mean Recall**: 0.800 ± 0.200\n",
    "\n",
    "---\n",
    "\n",
    "### Analysis\n",
    "\n",
    "The high variance in scores (standard deviations > 0.15) reflects a **sensitivity to fold composition**, likely due to the **small training set size** (92 observations).  \n",
    "This highlights the need for future **regularization**, **feature selection**, or **data augmentation** strategies to improve robustness.\n",
    "\n",
    "---\n",
    "\n",
    "### Visualization\n",
    "\n",
    "The figure below shows individual scores obtained on each fold.  \n",
    "There is **significant variability between folds**, especially in recall scores (ranging from 0.4 to 1.0), confirming the need to optimize the model to improve consistency and generalization.\n",
    "\n",
    "---\n",
    "\n",
    "This step establishes a **baseline** for the simple model’s performance before considering **hyperparameter optimization**.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c8f1ec9",
   "metadata": {},
   "source": [
    "# Step 10 — Hyperparameter Optimization (GridSearchCV)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "ba6b4fd9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters: {'C': 4.6415888336127775, 'penalty': 'l1', 'solver': 'liblinear'}\n",
      "Best F1-score (cross-val): 0.783\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "# Hyperparameter grid\n",
    "param_grid = {\n",
    "    'penalty': ['l1', 'l2'],\n",
    "    'C': np.logspace(-2, 4, 10),  # from 0.01 to 10000 (log scale)\n",
    "    'solver': ['liblinear']  # required for 'l1' penalty\n",
    "}\n",
    "\n",
    "# Initialize the model + cross-validation grid search\n",
    "logreg_grid = LogisticRegression(random_state=42, max_iter=1000)\n",
    "grid_search = GridSearchCV(logreg_grid, param_grid, cv=10, scoring='f1')\n",
    "grid_search.fit(X_train_scaled, y_train)\n",
    "\n",
    "# Best model\n",
    "best_logreg = grid_search.best_estimator_\n",
    "\n",
    "print(\"Best parameters:\", grid_search.best_params_)\n",
    "print(\"Best F1-score (cross-val):\", round(grid_search.best_score_, 3))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "090475a0",
   "metadata": {},
   "source": [
    "To improve the model’s performance, we implemented a **grid search (GridSearchCV)** to identify the best combination of hyperparameters for logistic regression.  \n",
    "The goal is to automatically select the optimal level of regularization to balance the model’s learning capacity and generalization.\n",
    "\n",
    "---\n",
    "\n",
    "### Hyperparameter grid tested\n",
    "\n",
    "The following combinations were evaluated using 10-fold cross-validation:\n",
    "\n",
    "- **Regularization type (`penalty`)**:  \n",
    "  `'l1'` (Lasso, encouraging sparsity) and `'l2'` (Ridge, classical quadratic regularization)\n",
    "- **Regularization coefficient (`C`)**:  \n",
    "  `[0.01, 0.1, 1, 10, 100, ...]`  \n",
    "  *(the larger `C`, the weaker the regularization; `C` is the inverse of the regularization strength)*\n",
    "- **Solver (`solver`)**:  \n",
    "  `'liblinear'`, chosen because it is the only solver compatible with `'l1'` regularization in `scikit-learn`.\n",
    "\n",
    "The evaluation metric used to guide selection was the **F1-score**, well suited to binary classification problems with class imbalance.\n",
    "\n",
    "---\n",
    "\n",
    "### Search results\n",
    "\n",
    "- **Best hyperparameters identified**:  \n",
    "  `{ 'C': 4.64, 'penalty': 'l1', 'solver': 'liblinear' }`\n",
    "\n",
    "- **Best mean F1-score in cross-validation (10 folds)**:  \n",
    "  **0.783**\n",
    "\n",
    "---\n",
    "\n",
    "### Analysis and interpretation\n",
    "\n",
    "The optimal model uses:\n",
    "\n",
    "- A **moderate L1 regularization** (`C = 4.64`), allowing some automatic zeroing of coefficients while limiting overfitting,\n",
    "- A **sparse model structure**, which may help with small datasets and enhances interpretability.\n",
    "\n",
    "The fact that the optimal `C` lies within the search grid (and not at the extreme) confirms that the model benefits from regularization.  \n",
    "This suggests that tuning regularization **has a measurable but moderate impact** on performance in this setting.\n",
    "\n",
    "---\n",
    "\n",
    "### Expected impact\n",
    "\n",
    "The regularized model will now be evaluated on the test set (step 11) to determine whether this optimization provides a **significant performance improvement** compared to the baseline logistic regression model from step 8.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57f03133",
   "metadata": {},
   "source": [
    "# Step 11 — Evaluation of the optimized model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "bd749019",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Evaluation on test set:\n",
      "Recall: 0.692\n",
      "F1-score: 0.75\n",
      "\n",
      "Classification Report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.69      0.82      0.75        11\n",
      "           1       0.82      0.69      0.75        13\n",
      "\n",
      "    accuracy                           0.75        24\n",
      "   macro avg       0.76      0.76      0.75        24\n",
      "weighted avg       0.76      0.75      0.75        24\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 19836 characters]",
      "text/plain": [
       "<Figure size 600x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 44248 characters]",
      "text/plain": [
       "<Figure size 800x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix, roc_curve, auc\n",
    "\n",
    "# Predictions\n",
    "y_pred_best = best_logreg.predict(X_test_scaled)\n",
    "\n",
    "# Overall evaluation\n",
    "print(\"\\nEvaluation on test set:\")\n",
    "print(\"Recall:\", round(recall_score(y_test, y_pred_best), 3))\n",
    "print(\"F1-score:\", round(f1_score(y_test, y_pred_best), 3))\n",
    "print(\"\\nClassification Report:\")\n",
    "print(classification_report(y_test, y_pred_best))\n",
    "\n",
    "# Confusion matrix\n",
    "cm = confusion_matrix(y_test, y_pred_best)\n",
    "plt.figure(figsize=(6, 5))\n",
    "sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', cbar=False,\n",
    "            xticklabels=['Negative', 'Positive'], yticklabels=['Negative', 'Positive'])\n",
    "plt.title(\"Confusion Matrix\")\n",
    "plt.xlabel(\"Predictions\")\n",
    "plt.ylabel(\"True labels\")\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# ROC curve and AUC\n",
    "y_proba_best = best_logreg.predict_proba(X_test_scaled)[:, 1]\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_proba_best)\n",
    "roc_auc = auc(fpr, tpr)\n",
    "\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(fpr, tpr, label=f'AUC = {roc_auc:.2f}', color='blue')\n",
    "plt.plot([0, 1], [0, 1], linestyle='--', color='red')\n",
    "plt.title(\"ROC Curve - Optimized Model\")\n",
    "plt.xlabel(\"False Positive Rate (FPR)\")\n",
    "plt.ylabel(\"True Positive Rate (TPR)\")\n",
    "plt.grid(True)\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bcbfdc8",
   "metadata": {},
   "source": [
    "After selecting the best logistic regression model by cross-validation (see Step 10), we evaluated it on the **test set**, which remained completely independent during training.\n",
    "\n",
    "---\n",
    "\n",
    "### Overall Results\n",
    "\n",
    "- **Recall**: 0.692  \n",
    "- **F1-score**: 0.750\n",
    "\n",
    "**Classification Report:**\n",
    "\n",
    "| Class        | Precision | Recall | F1-score | Support |\n",
    "|--------------|-----------|--------|----------|---------|\n",
    "| 0 (healthy)  | 0.69      | 0.82   | 0.75     | 11      |\n",
    "| 1 (cancer)   | 0.82      | 0.69   | 0.75     | 13      |\n",
    "| **Macro avg**| 0.76      | 0.76   | 0.75     | 24      |\n",
    "\n",
    "> The balance between precision and recall is good.  \n",
    "> Overall performance is comparable to that of the simple model (Step 8), indicating that regularization had limited impact in this context.\n",
    "\n",
    "---\n",
    "\n",
    "### Confusion Matrix\n",
    "\n",
    "|                | Predicted: 0 | Predicted: 1 |\n",
    "|----------------|--------------|--------------|\n",
    "| **True: 0**    | 9            | 2            |\n",
    "| **True: 1**    | 4            | 9            |\n",
    "\n",
    "- **9 true negatives** and **9 true positives** correctly identified  \n",
    "- **2 false positives** and **4 false negatives**\n",
    "\n",
    "The model shows a good ability to identify both classes, though several diseased patients were missed (false negatives), which is critical in medical applications.\n",
    "\n",
    "---\n",
    "\n",
    "### ROC Curve and AUC\n",
    "\n",
    "The ROC curve on the test set confirms a **good discriminative ability**, with an **area under the curve (AUC) of 0.79**.\n",
    "\n",
    "This means that on average, the model assigns a higher probability to a diseased individual than to a healthy one in **79% of cases**.  \n",
    "The ROC curve lies well above the diagonal (random classifier), validating the model’s usefulness.\n",
    "\n",
    "---\n",
    "\n",
    "### Conclusion\n",
    "\n",
    "The optimized model offers a **robust trade-off between performance and sparsity**, with:\n",
    "\n",
    "- **L1 regularization** that limits overfitting,\n",
    "- Good **test set performance**,\n",
    "- **Interpretable structure** (coefficients can be analyzed),\n",
    "- But overall results **comparable** to the simpler model from Step 8.\n",
    "\n",
    "This suggests that while logistic regression is a **reliable and interpretable baseline**, nonlinear models (e.g., SVM, tree-based methods) could be explored to improve performance further.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "918960cc",
   "metadata": {},
   "source": [
    "# Step 12 — Comparison of Logistic Regression Variants\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb288e88",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "e9c2a95f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                          Model  Recall  F1-score    AUC\n",
      "0                 Simple LogReg   0.692      0.75  0.783\n",
      "1  Optimized LogReg (L1, C=100)   0.692      0.75  0.790\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 51704 characters]",
      "text/plain": [
       "<Figure size 800x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 1000x600 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 45740 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 28952 characters]",
      "text/plain": [
       "<Figure size 700x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>y_test</th>\n",
       "      <th>Pred_simple</th>\n",
       "      <th>Pred_optimized</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>111</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>104</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     y_test  Pred_simple  Pred_optimized\n",
       "111       1            0               0\n",
       "104       1            0               0\n",
       "95        1            0               0\n",
       "38        0            1               1\n",
       "7         0            1               1\n",
       "58        1            0               0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "# 1. Compute predicted probabilities\n",
    "y_proba_simple = logreg_simple.predict_proba(X_test_scaled)[:, 1]\n",
    "y_proba_best = best_logreg.predict_proba(X_test_scaled)[:, 1]\n",
    "\n",
    "# 2. Summary of scores\n",
    "results = pd.DataFrame({\n",
    "    'Model': ['Simple LogReg', 'Optimized LogReg (L1, C=100)'],\n",
    "    'Recall': [\n",
    "        recall_score(y_test, y_pred_simple),\n",
    "        recall_score(y_test, y_pred_best)\n",
    "    ],\n",
    "    'F1-score': [\n",
    "        f1_score(y_test, y_pred_simple),\n",
    "        f1_score(y_test, y_pred_best)\n",
    "    ],\n",
    "    'AUC': [\n",
    "        roc_auc_score(y_test, y_proba_simple),\n",
    "        roc_auc_score(y_test, y_proba_best)\n",
    "    ]\n",
    "})\n",
    "print(results.round(3))\n",
    "\n",
    "# 3. Overlaid ROC curves\n",
    "fpr_simple, tpr_simple, _ = roc_curve(y_test, y_proba_simple)\n",
    "fpr_best, tpr_best, _ = roc_curve(y_test, y_proba_best)\n",
    "\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(fpr_simple, tpr_simple, '--', label=f'Simple (AUC={roc_auc_score(y_test, y_proba_simple):.2f})')\n",
    "plt.plot(fpr_best, tpr_best, '-', label=f'Optimized (AUC={roc_auc_score(y_test, y_proba_best):.2f})')\n",
    "plt.plot([0, 1], [0, 1], linestyle=':', color='grey')\n",
    "plt.title(\"ROC Curve Comparison\")\n",
    "plt.xlabel(\"False Positive Rate (FPR)\")\n",
    "plt.ylabel(\"True Positive Rate (TPR)\")\n",
    "plt.legend()\n",
    "plt.grid(True)\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# 4. Comparison of coefficients\n",
    "coef_df = pd.DataFrame({\n",
    "    'Feature': X_train_scaled.columns,\n",
    "    'Simple LogReg': logreg_simple.coef_[0],\n",
    "    'Optimized LogReg (L1)': best_logreg.coef_[0]\n",
    "}).set_index('Feature')\n",
    "\n",
    "plt.figure(figsize=(10, 6))\n",
    "coef_df.T.plot(kind='barh', colormap='viridis')\n",
    "plt.axvline(x=0, color='black', linestyle='--')\n",
    "plt.title(\"Comparison of Coefficients Between Models\")\n",
    "plt.xlabel(\"Coefficient\")\n",
    "plt.ylabel(\"Model / Feature\")\n",
    "plt.grid(True)\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# 5. Distribution of predicted probabilities (class 1)\n",
    "plt.figure(figsize=(7, 5))\n",
    "plt.hist(y_proba_simple, bins=10, alpha=0.5, label=\"Simple\", density=True)\n",
    "plt.hist(y_proba_best, bins=10, alpha=0.5, label=\"Optimized\", density=True)\n",
    "plt.title(\"Distribution of Predicted Probabilities (Positive Class)\")\n",
    "plt.xlabel(\"Predicted Probability\")\n",
    "plt.ylabel(\"Density\")\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# 6. Analysis of errors per individual\n",
    "errors = pd.DataFrame({\n",
    "    'y_test': y_test.values,\n",
    "    'Pred_simple': y_pred_simple,\n",
    "    'Pred_optimized': y_pred_best\n",
    "}, index=y_test.index)\n",
    "\n",
    "# Display only cases misclassified by at least one model\n",
    "diff = errors.query(\"Pred_simple != y_test or Pred_optimized != y_test\")\n",
    "display(diff)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a30c65a7",
   "metadata": {},
   "source": [
    "### Final Comparison Between Models\n",
    "\n",
    "We compared two versions of the logistic regression model:\n",
    "\n",
    "- A **simple model**, without explicit regularization (`penalty='l2'` by default),\n",
    "- An **optimized model**, with **L1 regularization** (`penalty='l1', C=100`), selected via cross-validation.\n",
    "\n",
    "---\n",
    "\n",
    "### Overall Performance on Test Set\n",
    "\n",
    "| Model                        | Recall | F1-score | AUC   |\n",
    "|-----------------------------|--------|----------|-------|\n",
    "| Simple LogReg               | 0.692  | 0.750    | 0.783 |\n",
    "| Optimized LogReg (L1, C=100)| 0.692  | 0.750    | 0.790 |\n",
    "\n",
    "Both models achieve **similar performance** on the test set, with identical recall and F1-scores, and a small difference in AUC.\n",
    "\n",
    "---\n",
    "\n",
    "### ROC Curve\n",
    "\n",
    "The ROC curves of both models are **nearly identical**, confirming equivalent discriminative power.  \n",
    "The L1-regularized model slightly improves AUC (0.790 vs. 0.783), but the difference remains marginal.\n",
    "\n",
    "---\n",
    "\n",
    "### Coefficients Comparison\n",
    "\n",
    "The L1-regularized model induces **sparser coefficients** by reducing or nullifying some features.  \n",
    "Notably:\n",
    "- `Insulin_log` and `HOMA_log` are downweighted,\n",
    "- `Glucose` and `BMI` remain strong contributors in both models.\n",
    "\n",
    "This illustrates the **interpretability and parsimony** offered by L1 regularization.\n",
    "\n",
    "---\n",
    "\n",
    "### Predicted Probabilities\n",
    "\n",
    "- Both models produce similar distributions of predicted probabilities for the positive class.\n",
    "- However, the L1 model yields **more extreme probabilities** (closer to 0 or 1), which may reflect **greater decision confidence**.\n",
    "\n",
    "---\n",
    "\n",
    "### Error Analysis\n",
    "\n",
    "We identified the observations for which the predictions **differ between models**.  \n",
    "Although most decisions agree, a few **borderline individuals** are treated differently, indicating **structural differences in decision boundaries**.\n",
    "\n",
    "---\n",
    "\n",
    "### Final Remarks\n",
    "\n",
    "| Model            | Key Strength                           |\n",
    "|------------------|-----------------------------------------|\n",
    "| Simple LogReg    | Slightly simpler, marginally faster     |\n",
    "| Optimized LogReg | Parsimony, better interpretability      |\n",
    "\n",
    "In practice:\n",
    "- Both models are viable,\n",
    "- The L1-regularized model is preferred when **feature selection, interpretability or generalization** are prioritized.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
