{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "963f8c8c",
   "metadata": {},
   "source": [
    "#                                       **EDA ANALYSIS**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8797cb42",
   "metadata": {},
   "source": [
    "# Step 1 — Initial loading and exploration\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "391c54d2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shape of the dataset is : (116, 10) \n",
      "\n",
      "The first 5 rows of the dataset are : \n",
      "    Age        BMI  Glucose  Insulin      HOMA   Leptin  Adiponectin  Resistin  \\\n",
      "0   48  23.500000       70    2.707  0.467409   8.8071     9.702400   7.99585   \n",
      "1   83  20.690495       92    3.115  0.706897   8.8438     5.429285   4.06405   \n",
      "2   82  23.124670       91    4.498  1.009651  17.9393    22.432040   9.27715   \n",
      "3   68  21.367521       77    3.226  0.612725   9.8827     7.169560  12.76600   \n",
      "4   86  21.111111       92    3.549  0.805386   6.6994     4.819240  10.57635   \n",
      "\n",
      "     MCP.1  Classification  \n",
      "0  417.114               1  \n",
      "1  468.786               1  \n",
      "2  554.697               1  \n",
      "3  928.220               1  \n",
      "4  773.920               1   \n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 116 entries, 0 to 115\n",
      "Data columns (total 10 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   Age             116 non-null    int64  \n",
      " 1   BMI             116 non-null    float64\n",
      " 2   Glucose         116 non-null    int64  \n",
      " 3   Insulin         116 non-null    float64\n",
      " 4   HOMA            116 non-null    float64\n",
      " 5   Leptin          116 non-null    float64\n",
      " 6   Adiponectin     116 non-null    float64\n",
      " 7   Resistin        116 non-null    float64\n",
      " 8   MCP.1           116 non-null    float64\n",
      " 9   Classification  116 non-null    int64  \n",
      "dtypes: float64(7), int64(3)\n",
      "memory usage: 9.2 KB\n",
      "The explanatory variables are: ['Age', 'BMI', 'Glucose', 'Insulin', 'HOMA', 'Leptin', 'Adiponectin', 'Resistin', 'MCP.1'] \n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 78036 characters]",
      "text/plain": [
       "<Figure size 1200x800 with 12 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Import libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# Load data\n",
    "data = pd.read_csv('breast+cancer+coimbra/dataR2.csv')\n",
    "\n",
    "# Data overview\n",
    "print(\"The shape of the dataset is :\",data.shape,\"\\n\")\n",
    "print(\"The first 5 rows of the dataset are : \\n\",data.head(),\"\\n\")\n",
    "data.info()\n",
    "\n",
    "# List of explanatory variables\n",
    "features = data.columns[:-1]\n",
    "print(\"The explanatory variables are:\", list(features),\"\\n\")\n",
    "target = 'Classification'\n",
    "\n",
    "# Histograms of variables\n",
    "data.hist(bins=50, figsize=(12, 8))\n",
    "plt.suptitle(\"Histograms of variables\", fontsize=16)\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "1c07b059",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification\n",
      "2    0.551724\n",
      "1    0.448276\n",
      "Name: proportion, dtype: float64\n",
      "the dataset is not balanced\n"
     ]
    }
   ],
   "source": [
    "#Proportion of classes\n",
    "print(data[\"Classification\"].value_counts(normalize=True))\n",
    "print(\"the dataset is not balanced\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "b794ea03",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of missing values in each column is:\n",
      "Age               0\n",
      "BMI               0\n",
      "Glucose           0\n",
      "Insulin           0\n",
      "HOMA              0\n",
      "Leptin            0\n",
      "Adiponectin       0\n",
      "Resistin          0\n",
      "MCP.1             0\n",
      "Classification    0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "#Check for missing values\n",
    "print(\"The number of missing values in each column is:\")\n",
    "print(data.isna().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a62b9137",
   "metadata": {},
   "source": [
    "We use data from the *Breast Cancer Coimbra* dataset, which contains biometric measurements of 116 patients. The objective is to **predict whether a patient has breast cancer** based on **nine biological markers**.\n",
    "\n",
    "### General Information\n",
    "\n",
    "- **Number of observations**: 116  \n",
    "- **Number of variables**: 10  \n",
    "  - **9 continuous explanatory variables**:  \n",
    "    `Age`, `BMI`, `Glucose`, `Insulin`, `HOMA`, `Leptin`, `Adiponectin`, `Resistin`, `MCP.1`  \n",
    "  - **1 binary target variable**: `Classification`  \n",
    "    - `2`: Patient has cancer (**positive class**)  \n",
    "    - `1`: Patient is healthy (**negative class**)  \n",
    "\n",
    "There are **no missing values** in the dataset, which allows us to proceed directly with exploratory analysis.\n",
    "\n",
    "### Distribution of Variables\n",
    "\n",
    "We visualized the distributions of all variables using **histograms**. This step helps detect irregularities or preprocessing needs.\n",
    "\n",
    "- Several variables show **strong asymmetry** and **long right tails**, notably:  \n",
    "  `Insulin`, `HOMA`, `Leptin`, `Adiponectin`, `Resistin`, and `MCP.1`.  \n",
    "  → These patterns may hinder model performance and justify a **logarithmic transformation**.\n",
    "\n",
    "- The **target variable is imbalanced**:  \n",
    "  Around **55%** of the patients are healthy, while **45%** have cancer.  \n",
    "  → This class imbalance will influence our **choice of evaluation metrics**: relying solely on accuracy would be misleading.\n",
    "\n",
    "---\n",
    "\n",
    "### Next Step\n",
    "\n",
    "To ensure robust training and fair evaluation, we will now:\n",
    "\n",
    "- Separate **features** from the **target** variable\n",
    "- Apply a **stratified train-test split** to preserve class proportions in both sets\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cb261cd",
   "metadata": {},
   "source": [
    "# Step 2 — Split the dataset into training and testing sets\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "3eddff01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: (92, 9) (92,)\n",
      "Test : (24, 9) (24,) \n",
      "\n",
      "Train class distribution: Classification\n",
      "1    0.554348\n",
      "0    0.445652\n",
      "Name: proportion, dtype: float64 \n",
      "\n",
      "Test class distribution : Classification\n",
      "1    0.541667\n",
      "0    0.458333\n",
      "Name: proportion, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Separate features and target variable\n",
    "X = data.drop(columns='Classification')\n",
    "y = data['Classification'].map({2: 1, 1: 0})\n",
    "# Mapping: 2 = patient with cancer (positive class), 1 = healthy patient (negative class)\n",
    "# We recode to: 1 → positive, 0 → negative\n",
    "\n",
    "# Stratified split to preserve class distribution\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y,\n",
    "    test_size=0.2,\n",
    "    random_state=42,\n",
    "    stratify=y)\n",
    "\n",
    "# Check shapes of the splits\n",
    "print(\"Train:\", X_train.shape, y_train.shape)\n",
    "print(\"Test :\", X_test.shape, y_test.shape, \"\\n\")\n",
    "\n",
    "print(\"Train class distribution:\", y_train.value_counts(normalize=True), \"\\n\")\n",
    "print(\"Test class distribution :\", y_test.value_counts(normalize=True))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2eec1d6e",
   "metadata": {},
   "source": [
    "To prepare for supervised modeling, we begin by separating:\n",
    "\n",
    "- **Explanatory variables** (features) → `X`  \n",
    "- **Target variable** (`Classification`) → `y`\n",
    "\n",
    "We recode the target to improve interpretability:\n",
    "\n",
    "- `1` → Patient has cancer (**positive class**)  \n",
    "- `0` → Patient is healthy (**negative class**)\n",
    "\n",
    "### Stratified Sampling\n",
    "\n",
    "To preserve the original class proportions in both training and testing sets, we perform a **stratified split** using the `train_test_split` function from `scikit-learn`.\n",
    "\n",
    "This is especially important given the observed **class imbalance** in the dataset.\n",
    "\n",
    "### Resulting Dataset Structure\n",
    "\n",
    "- **Training set**: 92 patients  \n",
    "- **Test set**: 24 patients  \n",
    "- **Split ratio**: 80% / 20%\n",
    "\n",
    "### Class Proportions (Post-Split)\n",
    "\n",
    "- **Training set**:  \n",
    "  ~55.4% class `1` (cancer), ~44.6% class `0` (healthy)\n",
    "\n",
    "- **Test set**:  \n",
    "  ~54.2% class `1` (cancer), ~45.8% class `0` (healthy)\n",
    "\n",
    "✅ The class distribution is successfully preserved across both sets.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02d2e734",
   "metadata": {},
   "source": [
    "# Step 3 — Visualization of distributions by class in the training dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "ac59107b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 119276 characters]",
      "text/plain": [
       "<Figure size 1500x1200 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 152692 characters]",
      "text/plain": [
       "<Figure size 1500x1200 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 249280 characters]",
      "text/plain": [
       "<Figure size 1500x1200 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Temporary merge for visualization\n",
    "train_data = X_train.copy()\n",
    "train_data['Classification'] = y_train\n",
    "\n",
    "variables = X_train.columns.tolist()\n",
    "n_cols = 3\n",
    "n_rows = int(np.ceil(len(variables) / n_cols))\n",
    "class_labels = {1: 'Patient with cancer', 0: 'Healthy patient'}\n",
    "\n",
    "# Histograms by class\n",
    "fig, axes = plt.subplots(n_rows, n_cols, figsize=(5 * n_cols, 4 * n_rows))\n",
    "axes = axes.flatten()\n",
    "\n",
    "for i, col in enumerate(variables):\n",
    "    for cls in [1, 0]:  # 1 = malade, 0 = sain\n",
    "        axes[i].hist(X_train[y_train == cls][col], bins=30, alpha=0.5, label=class_labels[cls])\n",
    "    axes[i].set_title(f\"{col} by class\")\n",
    "    axes[i].legend()\n",
    "\n",
    "# Supprimer axes inutilisés\n",
    "for j in range(i + 1, len(axes)):\n",
    "    fig.delaxes(axes[j])\n",
    "\n",
    "fig.suptitle(\"Histograms by class (before transformation)\", fontsize=16)\n",
    "plt.tight_layout(rect=[0, 0, 1, 0.97])\n",
    "plt.show()\n",
    "\n",
    "# Boxplots by class\n",
    "fig, axes = plt.subplots(n_rows, n_cols, figsize=(5 * n_cols, 4 * n_rows))\n",
    "axes = axes.flatten()\n",
    "\n",
    "for i, col in enumerate(variables):\n",
    "    sns.boxplot(x=y_train, y=X_train[col], ax=axes[i])\n",
    "    axes[i].set_title(f\"Boxplot of {col}\")\n",
    "    axes[i].set_xlabel(\"Class\")\n",
    "    axes[i].set_ylabel(col)\n",
    "    axes[i].set_xticks([0, 1])\n",
    "    axes[i].set_xticklabels([class_labels[0], class_labels[1]])\n",
    "\n",
    "for j in range(i + 1, len(axes)):\n",
    "    fig.delaxes(axes[j])\n",
    "\n",
    "fig.suptitle(\"Boxplots by class (before transformation)\", fontsize=16)\n",
    "plt.tight_layout(rect=[0, 0, 1, 0.97])\n",
    "plt.show()\n",
    "\n",
    "# Jittered scatter plots by class\n",
    "fig, axes = plt.subplots(n_rows, n_cols, figsize=(5 * n_cols, 4 * n_rows))\n",
    "axes = axes.flatten()\n",
    "\n",
    "for i, col in enumerate(variables):\n",
    "    for cls in [1, 0]:\n",
    "        axes[i].scatter(\n",
    "            X_train[y_train == cls][col],\n",
    "            np.random.normal(loc=cls, scale=0.02, size=X_train[y_train == cls][col].shape),\n",
    "            alpha=0.5,\n",
    "            label=class_labels[cls]\n",
    "        )\n",
    "    axes[i].set_title(f\"{col} (jittered scatter plot)\")\n",
    "    axes[i].set_ylabel(\"Class (jittered)\")\n",
    "    axes[i].legend()\n",
    "\n",
    "for j in range(i + 1, len(axes)):\n",
    "    fig.delaxes(axes[j])\n",
    "\n",
    "fig.suptitle(\"Jittered scatter plots by class (before transformation)\", fontsize=16)\n",
    "plt.tight_layout(rect=[0, 0, 1, 0.97])\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "072d85ad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Healthy patients:\n",
      "              Age        BMI     Glucose    Insulin       HOMA     Leptin  \\\n",
      "count  41.000000  41.000000   41.000000  41.000000  41.000000  41.000000   \n",
      "mean   59.195122  28.397908   87.804878   7.263829   1.627387  27.433273   \n",
      "std    18.911927   5.691456    9.762734   5.353097   1.344318  20.239726   \n",
      "min    24.000000  18.670000   60.000000   2.707000   0.467409   4.470000   \n",
      "25%    45.000000  23.000000   82.000000   4.090000   0.827271  10.280900   \n",
      "50%    66.000000  27.700000   87.000000   5.537000   1.145436  21.786300   \n",
      "75%    75.000000  32.500000   93.000000   8.079000   1.840410  35.891000   \n",
      "max    89.000000  38.578759  110.000000  26.211000   7.111918  83.482100   \n",
      "\n",
      "       Adiponectin   Resistin        MCP.1  \n",
      "count    41.000000  41.000000    41.000000  \n",
      "mean     10.641013  11.608922   522.124707  \n",
      "std       8.313134  12.414262   307.280571  \n",
      "min       2.194280   3.291750    45.843000  \n",
      "25%       5.100000   6.284450   263.499000  \n",
      "50%       7.931700   8.704480   530.410000  \n",
      "75%      10.567295  12.766000   695.754000  \n",
      "max      38.040000  82.100000  1256.083000  \n",
      "\n",
      "Cancer patients:\n",
      "              Age        BMI     Glucose    Insulin       HOMA     Leptin  \\\n",
      "count  51.000000  51.000000   51.000000  51.000000  51.000000  51.000000   \n",
      "mean   57.607843  27.329921  106.725490  11.819549   3.355671  25.502937   \n",
      "std    13.633897   4.670391   25.515547  10.201004   3.648005  18.657352   \n",
      "min    34.000000  18.370000   77.000000   2.432000   0.507936   6.333900   \n",
      "25%    45.500000  22.862850   92.000000   4.621500   1.131090  12.355850   \n",
      "50%    55.000000  27.887617  100.000000   8.150000   2.248594  19.065300   \n",
      "75%    68.500000  31.066529  112.000000  15.269500   4.117015  32.112650   \n",
      "max    86.000000  37.109375  201.000000  41.894000  20.630734  90.280000   \n",
      "\n",
      "       Adiponectin   Resistin        MCP.1  \n",
      "count    51.000000  51.000000    51.000000  \n",
      "mean     10.219945  16.582901   557.800706  \n",
      "std       6.475597  11.626857   375.401172  \n",
      "min       2.784910   3.210000    90.090000  \n",
      "25%       5.482135   8.022450   313.025000  \n",
      "50%       8.429960  14.769660   448.799000  \n",
      "75%      12.309480  21.705520   737.853000  \n",
      "max      33.750000  53.671700  1698.440000  \n"
     ]
    }
   ],
   "source": [
    "print(\"Healthy patients:\\n\", X_train[y_train == 0].describe())\n",
    "print(\"\\nCancer patients:\\n\", X_train[y_train == 1].describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "066377f6",
   "metadata": {},
   "source": [
    "To better understand the discriminative potential of each variable, we analyzed the distributions by class (healthy vs. cancer) using the training set. We combined three complementary types of visualizations: histograms, boxplots, and jittered scatter plots, followed by descriptive statistics for each class.\n",
    "\n",
    "---\n",
    "\n",
    "### Objective\n",
    "\n",
    "The goal of this step is to assess whether some features show different behaviors across the two classes:\n",
    "- Class `0` = healthy patients,\n",
    "- Class `1` = cancer patients.\n",
    "\n",
    "By visually comparing the distributions, we aim to identify features that could be informative for classification.\n",
    "\n",
    "---\n",
    "\n",
    "### Histograms by Class\n",
    "\n",
    "The histograms highlight the distribution of values for each variable, separated by class. Key observations:\n",
    "\n",
    "- `Glucose`, `Insulin`, `HOMA`, and `Resistin` show noticeable right-shifted distributions for class 1, suggesting higher values in cancer patients.\n",
    "- `Adiponectin` and `MCP.1` exhibit moderate class-dependent density shifts.\n",
    "- `BMI` and `Leptin` display strong overlaps between classes, suggesting limited discriminative power on their own.\n",
    "\n",
    "These preliminary shifts suggest that some variables may carry predictive signal, although no single feature is perfectly separating.\n",
    "\n",
    "---\n",
    "\n",
    "### Boxplots by Class\n",
    "\n",
    "Boxplots provide a clearer comparison of medians, dispersion, and outliers for each variable:\n",
    "\n",
    "- High dispersion is visible across most features, particularly for `Insulin`, `HOMA`, `Resistin`, and `MCP.1`, with long upper tails in class 1.\n",
    "- `Resistin` and `HOMA` show visibly higher medians in cancer patients.\n",
    "- Outliers are frequent in both classes, justifying the need for transformation or robust models.\n",
    "\n",
    "---\n",
    "\n",
    "### Jittered Scatter Plots\n",
    "\n",
    "To better visualize class separation (especially for categorical targets), we used jittered scatter plots:\n",
    "\n",
    "- The vertical axis encodes the class (0 or 1), while the horizontal axis shows the variable value.\n",
    "- These plots reveal:\n",
    "  - Some clusters (e.g., low `Insulin` and `Adiponectin` values for class 0),\n",
    "  - But substantial horizontal overlap remains across most features.\n",
    "\n",
    "These plots visually confirm that no single variable can fully separate the classes, underscoring the importance of multivariate modeling.\n",
    "\n",
    "---\n",
    "\n",
    "### Descriptive Statistics by Class\n",
    "\n",
    "We computed summary statistics separately for each class to numerically reinforce visual trends.\n",
    "\n",
    "#### Healthy patients (class 0)\n",
    "\n",
    "- Mean Glucose: 87.8\n",
    "- Mean Insulin: 7.3\n",
    "- Mean HOMA: 1.63\n",
    "- Mean Resistin: 11.6\n",
    "- High variability observed (standard deviations often > 5)\n",
    "\n",
    "#### Cancer patients (class 1)\n",
    "\n",
    "- Mean Glucose: 106.7\n",
    "- Mean Insulin: 11.8\n",
    "- Mean HOMA: 3.36\n",
    "- Mean Resistin: 16.6\n",
    "- The distributions are more skewed and dispersed than in class 0\n",
    "\n",
    "These numerical insights confirm what was seen in the plots: class 1 patients tend to have higher values and more variability in metabolic features.\n",
    "\n",
    "---\n",
    "\n",
    "### Summary of EDA Findings\n",
    "\n",
    "- Several features show partial class separation, including:  \n",
    "  `Glucose`, `Insulin`, `HOMA`, `Resistin`, `MCP.1`.\n",
    "- Most distributions are asymmetric, with long tails and outliers.\n",
    "- There is high intra-class variance, suggesting noise and overlap.\n",
    "- These findings motivate:\n",
    "  - A logarithmic transformation of skewed variables,\n",
    "  - Standardization to control for different scales,\n",
    "  - The use of regularized or nonlinear models to handle multicollinearity and partial separability.\n",
    "\n",
    "---\n",
    "\n",
    "### Conclusion\n",
    "\n",
    "This exploratory step confirms that:\n",
    "\n",
    "- The dataset contains informative but noisy features,\n",
    "- No feature alone is sufficient, but combinations of features are likely discriminative,\n",
    "- Visual and statistical patterns strongly support the preprocessing choices in the next steps:\n",
    "  - Log transformation,\n",
    "  - Standardization,\n",
    "  - Regularized classification models.\n",
    "\n",
    "We now proceed to Step 4 — correlation analysis.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09e7c056",
   "metadata": {},
   "source": [
    "# Step 4 — Correlation matrix and scatter matrix (training set only)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "9ff82466",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 111040 characters]",
      "text/plain": [
       "<Figure size 1000x800 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 401548 characters]",
      "text/plain": [
       "<Figure size 1200x800 with 81 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pandas.plotting import scatter_matrix\n",
    "from matplotlib.colors import ListedColormap\n",
    "\n",
    "\n",
    "# Correlation matrix (on training data)\n",
    "corr_matrix = X_train.corr()\n",
    "\n",
    "plt.figure(figsize=(10, 8))\n",
    "sns.heatmap(corr_matrix, annot=True, fmt=\".2f\", cmap='coolwarm', square=True)\n",
    "plt.suptitle(\"Correlation matrix (X_train)\", fontsize=14)\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Combined DataFrame for colored visualization\n",
    "X_train_visu = X_train.copy()\n",
    "X_train_visu['Classification'] = y_train.values\n",
    "\n",
    "# Custom color map: 0 = Healthy (light red), 1 = Cancer (blue)\n",
    "custom_cmap = ListedColormap([\"#FFA07A\", \"#1E90FF\"])  # light salmon, dodger blue\n",
    "\n",
    "# Colored scatter matrix\n",
    "scatter_matrix(X_train_visu.drop(columns='Classification'),\n",
    "               figsize=(12, 8),\n",
    "               diagonal='kde',\n",
    "               c=X_train_visu['Classification'],\n",
    "               cmap=custom_cmap,\n",
    "               alpha=0.7)\n",
    "plt.suptitle(\"Colored scatter matrix (X_train)\", y=1.02)\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a358c6aa",
   "metadata": {},
   "source": [
    "After inspecting individual variable distributions and class-dependent variations, we now explore **pairwise relationships between variables** to:\n",
    "\n",
    "- detect potential **redundancies** or **multicollinearity**,\n",
    "- assess the relevance of **dimensionality reduction or regularization** techniques,\n",
    "- and evaluate how well the different features contribute **complementary** information.\n",
    "\n",
    "We perform this analysis on the **training set only** to avoid any data leakage.\n",
    "\n",
    "---\n",
    "\n",
    "### Correlation Matrix (Pearson Coefficients)\n",
    "\n",
    "The **correlation matrix** below summarizes the **linear relationships** between continuous explanatory variables using Pearson correlation coefficients (ρ):\n",
    "\n",
    "- Values near **+1** indicate a **strong positive linear correlation**.\n",
    "- Values near **0** indicate **no linear correlation**.\n",
    "- Values near **–1** (none observed here) would indicate a **strong negative correlation**.\n",
    "\n",
    "#### Key observations:\n",
    "\n",
    "- **`Insulin` and `HOMA`** show an extremely high correlation (ρ ≈ 0.93). This is expected, as `HOMA` is a deterministic function of `Insulin` and `Glucose`:\n",
    "\n",
    "  $$\n",
    "  \\text{HOMA} = \\frac{\\text{Glucose} \\times \\text{Insulin}}{405}\n",
    "  $$\n",
    "\n",
    "- Moderate correlations include:\n",
    "  - `Glucose` and `HOMA` (ρ ≈ 0.67)\n",
    "  - `BMI` and `Leptin` (ρ ≈ 0.58)\n",
    "\n",
    "- All other variable pairs have correlations **below 0.4**, suggesting that most features carry **distinct and complementary information**.\n",
    "\n",
    "This supports the use of the full feature set, but also **justifies regularization** or **dimensionality reduction** (e.g., PCA) to address potential multicollinearity in models like logistic regression.\n",
    "\n",
    "---\n",
    "\n",
    "### Colored Scatter Matrix (Pairwise Interactions with Class Overlay)\n",
    "\n",
    "To further explore the structure of the data and how variables relate to class membership, we plot a **scatter matrix**, where each point is colored by its class:\n",
    "\n",
    "- **Blue**: Class 1 (patient with cancer)\n",
    "- **Red**: Class 0 (healthy patient)\n",
    "\n",
    "This visualization helps assess whether **pairs of variables** allow separation of the two groups.\n",
    "\n",
    "#### Key observations:\n",
    "\n",
    "- The `Insulin`–`HOMA` pair displays a nearly perfect linear relationship, confirming the analytical link.\n",
    "- Some variable combinations suggest **partial clustering** by class, such as:\n",
    "  - `Resistin` vs `MCP.1`\n",
    "  - `Adiponectin` vs `HOMA`\n",
    "  - `Glucose` vs `Insulin`\n",
    "- However, the **overlap remains substantial** in most 2D projections.\n",
    "\n",
    "These findings imply that:\n",
    "\n",
    "- **No individual variable pair** fully separates the classes.\n",
    "- Discriminative power is likely to emerge from **higher-dimensional or nonlinear** interactions.\n",
    "\n",
    "---\n",
    "\n",
    "### Summary of Step 4\n",
    "\n",
    "#### Insights:\n",
    "\n",
    "- A few strong linear relationships (notably `Insulin`–`HOMA`) warrant caution, especially in **unregularized linear models**.\n",
    "- Most variables provide **non-redundant information**, supporting their joint use.\n",
    "- The **absence of clear separation in 2D** reinforces the need for **multivariate and possibly nonlinear modeling** strategies.\n",
    "\n",
    "#### Consequences:\n",
    "\n",
    "- **Standardization** will be necessary before modeling to neutralize differences in scale.\n",
    "- We may apply:\n",
    "  - **L1 regularization** (to sparsify coefficients),\n",
    "  - or **PCA** (to decorrelate features),\n",
    "  - or continue with all variables if models used are robust to collinearity.\n",
    "\n",
    "We now proceed to **data transformation and scaling** before supervised modeling.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "556f426b",
   "metadata": {},
   "source": [
    "# Step 5 — Logarithmic transformation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "af21af16",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 44992 characters]",
      "text/plain": [
       "<Figure size 1200x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 47788 characters]",
      "text/plain": [
       "<Figure size 1200x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 45488 characters]",
      "text/plain": [
       "<Figure size 1200x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 48652 characters]",
      "text/plain": [
       "<Figure size 1200x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 103904 characters]",
      "text/plain": [
       "<Figure size 1500x1200 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 130616 characters]",
      "text/plain": [
       "<Figure size 1500x1200 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 220844 characters]",
      "text/plain": [
       "<Figure size 1500x1200 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# List of variables to transform (asymmetric distributions)\n",
    "variables_log = ['Insulin', 'HOMA', 'MCP.1', 'Resistin']\n",
    "\n",
    "# Comparison before / after transformation (1 figure per variable)\n",
    "for col in variables_log:\n",
    "    fig, axes = plt.subplots(1, 2, figsize=(12, 4))\n",
    "\n",
    "    # Before transformation\n",
    "    axes[0].hist(X_train[col], bins=30, alpha=0.6, color='salmon', edgecolor='black')\n",
    "    axes[0].set_title(f\"{col} (before transformation)\")\n",
    "    axes[0].set_xlabel(col)\n",
    "    axes[0].set_ylabel(\"Frequency\")\n",
    "\n",
    "    # After log1p transformation\n",
    "    transformed = np.log1p(X_train[col])\n",
    "    axes[1].hist(transformed, bins=30, alpha=0.6, color='steelblue', edgecolor='black')\n",
    "    axes[1].set_title(f\"{col} after log({col} + 1) transformation\")\n",
    "    axes[1].set_xlabel(f\"log1p({col})\")\n",
    "    axes[1].set_ylabel(\"Frequency\")\n",
    "\n",
    "    plt.suptitle(f\"Comparison before / after transformation: {col}\", fontsize=14)\n",
    "    plt.tight_layout(rect=[0, 0, 1, 0.93])\n",
    "    plt.show()\n",
    "\n",
    "# Apply the transformation to X_train\n",
    "X_train_transformed = X_train.copy()\n",
    "for col in variables_log:\n",
    "    X_train_transformed[col + '_log'] = np.log1p(X_train[col])\n",
    "X_train_transformed = X_train_transformed.drop(columns=variables_log)\n",
    "\n",
    "# Update the list of transformed features\n",
    "features = X_train_transformed.columns.tolist()\n",
    "\n",
    "# Merge with y_train to facilitate visualization\n",
    "train_data2 = X_train_transformed.copy()\n",
    "train_data2['Classification'] = y_train.values\n",
    "\n",
    "# Histograms by class\n",
    "n_cols = 3\n",
    "n_rows = int(np.ceil(len(features) / n_cols))\n",
    "\n",
    "fig, axes = plt.subplots(n_rows, n_cols, figsize=(5 * n_cols, 4 * n_rows))\n",
    "axes = axes.flatten()\n",
    "\n",
    "for i, col in enumerate(features):\n",
    "    for cls in [0, 1]:\n",
    "        axes[i].hist(train_data2[train_data2['Classification'] == cls][col],\n",
    "                     bins=30, alpha=0.5, label=f\"Class {cls}\")\n",
    "    axes[i].set_title(f\"{col} by class\")\n",
    "    axes[i].legend()\n",
    "\n",
    "for j in range(i + 1, len(axes)):\n",
    "    fig.delaxes(axes[j])\n",
    "\n",
    "fig.suptitle(\"Histograms by class (after transformation)\", fontsize=16)\n",
    "plt.tight_layout(rect=[0, 0, 1, 0.97])\n",
    "plt.show()\n",
    "\n",
    "# Boxplots by class\n",
    "\n",
    "fig, axes = plt.subplots(n_rows, n_cols, figsize=(5 * n_cols, 4 * n_rows))\n",
    "axes = axes.flatten()\n",
    "\n",
    "for i, col in enumerate(features):\n",
    "    sns.boxplot(x='Classification', y=col, data=train_data2, ax=axes[i])\n",
    "    axes[i].set_title(f\"Boxplot of {col}\")\n",
    "    axes[i].set_xlabel(\"Class\")\n",
    "    axes[i].set_ylabel(col)\n",
    "\n",
    "for j in range(i + 1, len(axes)):\n",
    "    fig.delaxes(axes[j])\n",
    "\n",
    "fig.suptitle(\"Boxplots by class (after transformation)\", fontsize=16)\n",
    "plt.tight_layout(rect=[0, 0, 1, 0.97])\n",
    "plt.show()\n",
    "\n",
    "# Jittered scatter plots by class\n",
    "\n",
    "fig, axes = plt.subplots(n_rows, n_cols, figsize=(5 * n_cols, 4 * n_rows))\n",
    "axes = axes.flatten()\n",
    "\n",
    "for i, col in enumerate(features):\n",
    "    for cls in [0, 1]:\n",
    "        values = train_data2[train_data2['Classification'] == cls][col]\n",
    "        jittered_y = np.random.normal(loc=cls, scale=0.02, size=values.shape)\n",
    "        axes[i].scatter(values, jittered_y, alpha=0.5, label=f\"Class {cls}\")\n",
    "    axes[i].set_title(f\"{col} (jittered scatter plot)\")\n",
    "    axes[i].legend()\n",
    "\n",
    "for j in range(i + 1, len(axes)):\n",
    "    fig.delaxes(axes[j])\n",
    "\n",
    "fig.suptitle(\"Jittered scatter plots by class (after transformation)\", fontsize=16)\n",
    "plt.tight_layout(rect=[0, 0, 1, 0.97])\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c582a520",
   "metadata": {},
   "source": [
    "#### Objective\n",
    "\n",
    "The initial exploratory analysis highlighted several variables exhibiting strong right-skewness and extreme values, notably:\n",
    "\n",
    "- `Insulin`,  \n",
    "- `HOMA`,  \n",
    "- `MCP.1`,  \n",
    "- `Resistin`.\n",
    "\n",
    "Such distributions may negatively impact the performance of machine learning models sensitive to scale differences or non-normality, including logistic regression, SVMs, and k-nearest neighbors. \n",
    "\n",
    "To address this, we apply a **logarithmic transformation** of the form `log(x + 1)` using NumPy’s `np.log1p` function. This transformation compresses large values, reduces skewness, and brings the distributions closer to Gaussian, which improves data regularity and supports model training.\n",
    "\n",
    "---\n",
    "\n",
    "#### Comparison Before / After Transformation\n",
    "\n",
    "The following plots compare the distribution of each targeted variable **before and after** applying the log transformation:\n",
    "\n",
    "- Right tails are **substantially reduced**.\n",
    "- Distributions become **more compact and symmetric**.\n",
    "- Variability is stabilized, which facilitates **standardization** and enhances interpretability.\n",
    "\n",
    "These graphical comparisons provide **a clear visual justification** for the use of log transformation on these variables.\n",
    "\n",
    "---\n",
    "\n",
    "#### Dataset Update\n",
    "\n",
    "- The transformed variables were renamed to `*_log` (e.g., `Insulin_log`) and **added** to the training dataset.\n",
    "- The original (raw) variables were **removed** to avoid redundancy and multicollinearity.\n",
    "- The list of features was updated accordingly.\n",
    "\n",
    "---\n",
    "\n",
    "#### Effect on Class-Separated Distributions\n",
    "\n",
    "We re-examined the transformed variables using three complementary plots (as done in the raw version):\n",
    "\n",
    "##### • Histograms by class\n",
    "\n",
    "Histograms show that after transformation:\n",
    "\n",
    "- Distributions are much more **uniform and comparable** between the two classes.\n",
    "- Differences remain observable, particularly for `Insulin_log`, `HOMA_log`, and `Resistin_log`, but are now embedded in **more regular shapes**.\n",
    "\n",
    "##### • Boxplots by class\n",
    "\n",
    "The boxplots reveal:\n",
    "\n",
    "- A **reduction in the number and intensity of outliers**.\n",
    "- Interquartile ranges (IQRs) are narrower and more symmetric.\n",
    "- Median differences between classes are more interpretable (e.g., `Insulin_log` and `Resistin_log`).\n",
    "\n",
    "##### • Jittered scatter plots\n",
    "\n",
    "These scatter plots help visualize how individual observations separate vertically across class labels:\n",
    "\n",
    "- The **horizontal spread is better controlled**, enhancing the applicability of distance-based models.\n",
    "- Some separation patterns (e.g., for `MCP.1_log`, `Resistin_log`) are still visible, but overlap remains significant — reinforcing the need for multivariate modeling.\n",
    "\n",
    "---\n",
    "\n",
    "#### Conclusion\n",
    "\n",
    "Applying a logarithmic transformation on skewed variables allowed us to:\n",
    "\n",
    "- **Reduce the influence of extreme values**,  \n",
    "- **Stabilize variance**,  \n",
    "- **Improve the regularity and symmetry** of variable distributions,  \n",
    "- **Facilitate downstream standardization and model fitting**.\n",
    "\n",
    "This transformation is thus considered an **essential preprocessing step** prior to standardization and supervised learning.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "184a592b",
   "metadata": {},
   "source": [
    "# Step 6 — Standardization (StandardScaler)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "4b77fbe5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Age</th>\n",
       "      <td>-2.003229e-16</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BMI</th>\n",
       "      <td>-6.389816e-16</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Glucose</th>\n",
       "      <td>8.326673e-17</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Leptin</th>\n",
       "      <td>-5.068409e-17</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Adiponectin</th>\n",
       "      <td>8.839547e-17</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Insulin_log</th>\n",
       "      <td>-1.194697e-16</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HOMA_log</th>\n",
       "      <td>-2.256649e-16</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MCP.1_log</th>\n",
       "      <td>1.128324e-15</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Resistin_log</th>\n",
       "      <td>6.552729e-16</td>\n",
       "      <td>1.005479</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      mean       std\n",
       "Age          -2.003229e-16  1.005479\n",
       "BMI          -6.389816e-16  1.005479\n",
       "Glucose       8.326673e-17  1.005479\n",
       "Leptin       -5.068409e-17  1.005479\n",
       "Adiponectin   8.839547e-17  1.005479\n",
       "Insulin_log  -1.194697e-16  1.005479\n",
       "HOMA_log     -2.256649e-16  1.005479\n",
       "MCP.1_log     1.128324e-15  1.005479\n",
       "Resistin_log  6.552729e-16  1.005479"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Initialize the scaler\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Fit the scaler on the transformed training set and transform\n",
    "X_train_scaled = pd.DataFrame(\n",
    "    scaler.fit_transform(X_train_transformed),\n",
    "    columns=X_train_transformed.columns,\n",
    "    index=X_train_transformed.index\n",
    ")\n",
    "\n",
    "# Display summary statistics of scaled data\n",
    "X_train_scaled.describe().T[['mean', 'std']]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e525eb2",
   "metadata": {},
   "source": [
    "### Objective\n",
    "\n",
    "After applying logarithmic transformation to asymmetric variables, we apply **standardization** to all explanatory variables in the training set.  \n",
    "This step aims to **bring all variables to the same scale** by centering them around zero and scaling them to unit variance.\n",
    "\n",
    "Standardization is particularly important for models sensitive to distances or scale differences, such as:\n",
    "- logistic regression (for more stable convergence),\n",
    "- support vector machines (SVM),\n",
    "- k-nearest neighbors (k-NN),\n",
    "- regularization methods (Ridge, Lasso),\n",
    "- and dimensionality reduction techniques (PCA, etc.).\n",
    "\n",
    "---\n",
    "### Method\n",
    "\n",
    "The transformation used is the classical Z-score standardization, defined as:\n",
    "\n",
    "$$\n",
    "X^{(i)}_{\\text{std}} = \\frac{X^{(i)} - \\mu^{(i)}}{\\sigma^{(i)}}\n",
    "$$\n",
    "\n",
    "where \\( \\mu^{(i)} \\) and \\( \\sigma^{(i)} \\) denote respectively the mean and standard deviation of variable \\( X^{(i)} \\) computed on the training set.\n",
    "\n",
    "We use the `StandardScaler` tool from `scikit-learn`, applied only on the training data (`fit_transform`) to avoid any data leakage.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d374b93f",
   "metadata": {},
   "source": [
    "# Step 7 — Preparation of X_test\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "28a4ffe0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy X_test\n",
    "X_test_transformed = X_test.copy()\n",
    "\n",
    "# Apply the same log-transformations\n",
    "for col in variables_log:\n",
    "    X_test_transformed[col + '_log'] = np.log1p(X_test_transformed[col])\n",
    "\n",
    "# Drop original columns\n",
    "X_test_transformed = X_test_transformed.drop(columns=variables_log)\n",
    "\n",
    "# Standardize using the scaler previously fitted on X_train\n",
    "X_test_scaled = pd.DataFrame(\n",
    "    scaler.transform(X_test_transformed),\n",
    "    columns=X_test_transformed.columns,\n",
    "    index=X_test_transformed.index\n",
    ")\n",
    "\n",
    "# Check that the columns of X_test are identical (and in the same order) as those of X_train to avoid prediction errors\n",
    "if list(X_train_scaled.columns) != list(X_test_scaled.columns):\n",
    "    print(\"Mismatch between train and test columns\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4a5689a",
   "metadata": {},
   "source": [
    "The transformations previously applied to the training set (logarithmic transformation + standardization) must be **reproduced exactly the same way** on the test set, without re-estimating any parameters.\n",
    "\n",
    "### Steps performed:\n",
    "1. **Logarithmic transformation** on the same variables (`Insulin`, `HOMA`, `MCP.1`, `Resistin`) using `log(x + 1)`;\n",
    "2. **Removal of raw variables** to avoid redundancy;\n",
    "3. **Standardization using the same parameters** (mean `μ`, standard deviation `σ`) learned from `X_train`, via `scaler.transform()`.\n",
    "\n",
    "We also verified that the **columns of `X_test_scaled` are exactly the same (and in the same order)** as those of `X_train_scaled`, ensuring compatibility with the learning models.\n",
    "\n",
    "---\n",
    "\n",
    "This step completes the preprocessing pipeline and allows us to proceed to **supervised modeling**.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
