{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Séance 1 - Bonus : Première régularisation d'un réseau de neurones\n",
    "\n",
    "Pour poursuivre le travail du TP, on se propose d'explorer une première manière de régulariser un réseau de neurones. Nous avons vu que nous pouvions atteindre de bonne performances, mais que cela entraînait fréquemment un sur-apprentissage. \n",
    "Une des premières manières de régulariser que l'on apprend en Machine Learning est de pénaliser une régression linéaire : c'est la régression Ridge. Le principe est de modifier la fonction de perte pour contraindre les poids appris à être *petit*. Il est possible de le faire couche par couche dans un réseau de neurones. Essayons !\n",
    "\n",
    "Commençons par importer et traiter les données."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set(style='whitegrid')\n",
    "\n",
    "import tensorflow as tf\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from tensorflow import keras\n",
    "\n",
    "(X_train_full, y_train_full), (X_test, y_test) = (keras.datasets.mnist.load_data())\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X_train_full, y_train_full, train_size=0.8)\n",
    "\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train.astype(np.float32).reshape(-1, 28 * 28)).reshape(-1, 28, 28)\n",
    "X_valid = scaler.transform(X_valid.astype(np.float32).reshape(-1, 28 * 28)).reshape(-1, 28, 28)\n",
    "X_test = scaler.transform(X_test.astype(np.float32).reshape(-1, 28 * 28)).reshape(-1, 28, 28)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pénalisation $L_2$\n",
    "\n",
    "La pénalisation à laquelle on s'intéresse ici est la pénalisation $L_2$. Si l'on considère un problème d'optimisation d'une fonction de perte $\\mathcal{L}$\n",
    "\n",
    "$$w^* = \\arg\\min_{w\\in\\mathbb{R}^d} \\mathcal{L}(w)$$\n",
    "\n",
    "Alors sa version pénalisée est la suivante avec $\\lambda > 0$ :\n",
    "\n",
    "$$w^* = \\arg\\min_{w\\in\\mathbb{R}^d} \\mathcal{L}(w) + \\lambda\\|w\\|^2$$\n",
    "\n",
    "Ce n'est plus exactement le même objectif puisqu'on force ici le vecteur solution $w^*$ a prendre de plus petite valeurs. Nous pouvons faire cela couche par couche.\n",
    "\n",
    "**Consigne** : Définir un modèle en ajoutant une pénalisation [$L_2$](https://keras.io/api/layers/regularizers/#l2-class) aux couches [`Dense`](https://keras.io/api/layers/core_layers/dense/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential(\n",
    "    [\n",
    "        keras.layers.Input(shape=[28, 28]),\n",
    "        keras.layers.Flatten(),\n",
    "        keras.layers.Dense(256, activation=\"relu\", kernel_regularizer=keras.regularizers.l2(0.001)),\n",
    "        keras.layers.Dense(128, activation=\"relu\", kernel_regularizer=keras.regularizers.l2(0.001)),\n",
    "        keras.layers.Dense(10, activation=\"softmax\"),\n",
    "    ]\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Consigne** : Lancer sur quelques époques le modèle pour valider qu'il fonctionne correctement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "\u001b[1m1500/1500\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 1ms/step - accuracy: 0.6287 - loss: 1.8807 - val_accuracy: 0.8105 - val_loss: 1.3201\n",
      "Epoch 2/5\n",
      "\u001b[1m1500/1500\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 983us/step - accuracy: 0.8471 - loss: 1.1539 - val_accuracy: 0.8680 - val_loss: 1.0520\n",
      "Epoch 3/5\n",
      "\u001b[1m1500/1500\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 988us/step - accuracy: 0.8835 - loss: 0.9892 - val_accuracy: 0.8895 - val_loss: 0.9567\n",
      "Epoch 4/5\n",
      "\u001b[1m1500/1500\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 994us/step - accuracy: 0.9001 - loss: 0.9158 - val_accuracy: 0.9024 - val_loss: 0.9055\n",
      "Epoch 5/5\n",
      "\u001b[1m1500/1500\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 1ms/step - accuracy: 0.9094 - loss: 0.8713 - val_accuracy: 0.9092 - val_loss: 0.8730\n"
     ]
    }
   ],
   "source": [
    "model.compile(\n",
    "    loss=\"sparse_categorical_crossentropy\",\n",
    "    optimizer=keras.optimizers.SGD(learning_rate=1e-3),\n",
    "    metrics=[\"accuracy\"],\n",
    ")\n",
    "\n",
    "epochs = 5\n",
    "batch_size = 32\n",
    "\n",
    "history = model.fit(\n",
    "    X_train,\n",
    "    y_train,\n",
    "    epochs=epochs,\n",
    "    batch_size=batch_size,\n",
    "    validation_data=(X_valid, y_valid),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "De la même manière que l'on souhaitais observer l'impact du learning rate sur l'entraînement, on souhaite mesurer l'apport de la régularisation au modèle. \n",
    "\n",
    "## Mesure de l'impact de la régularisation\n",
    "\n",
    "**Consigne** : Définir une fonction `get_model` qui prend en paramètre :\n",
    "* *lambda_l2* : float correspondant à la magnitude de la pénalisation $L_2$\n",
    "* *learning_rate* : float par défaut à $0.001$ correspond au learning rate de l'optimizer\n",
    "\n",
    "La fonction renvoie le modèle compilé."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_model(lambda_l2: float, learning_rate: float) -> keras.Model:\n",
    "    model = keras.models.Sequential(\n",
    "        [\n",
    "            keras.layers.Input(shape=[28, 28]),\n",
    "            keras.layers.Flatten(),\n",
    "            keras.layers.Dense(256, activation=\"relu\", kernel_regularizer=keras.regularizers.l2(lambda_l2)),\n",
    "            keras.layers.Dense(128, activation=\"relu\", kernel_regularizer=keras.regularizers.l2(lambda_l2)),\n",
    "            keras.layers.Dense(10, activation=\"softmax\"),\n",
    "        ]\n",
    "    )\n",
    "    model.compile(\n",
    "        loss=\"sparse_categorical_crossentropy\",\n",
    "        optimizer=keras.optimizers.SGD(learning_rate=learning_rate),\n",
    "        metrics=[\"accuracy\"],\n",
    "    )\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Consigne** : En s'inspirant du travail réalisé pour le learning rate, comparer différente valeur de régularisation. Commenter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_epochs = 30\n",
    "batch_size = 256\n",
    "learning_rates = [10 ** (-power) for power in range(1, 4)]\n",
    "lambda_l2_values = [0.001, 0.01, 0.1]\n",
    "\n",
    "results = []\n",
    "for learning_rate in learning_rates:\n",
    "    for lambda_l2 in lambda_l2_values:\n",
    "        model = get_model(lambda_l2, learning_rate)\n",
    "        history = model.fit(\n",
    "            X_train,\n",
    "            y_train,\n",
    "            epochs=n_epochs,\n",
    "            batch_size=batch_size,\n",
    "            validation_data=(X_valid, y_valid),\n",
    "            verbose=0,\n",
    "        )\n",
    "        results.append(\n",
    "            {\n",
    "                \"learning_rate\": learning_rate,\n",
    "                \"lambda_l2\": lambda_l2,\n",
    "                \"history\": pd.DataFrame(history.history),\n",
    "                \"n_epochs\": n_epochs,\n",
    "            }\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_results(results: list) -> None:\n",
    "    plt.figure(figsize=(12, 4))\n",
    "    colors = sns.color_palette(\"husl\", len(results))\n",
    "    for _, result in enumerate(results):\n",
    "        history_df = result[\"history\"]\n",
    "        learning_rate = result[\"learning_rate\"]\n",
    "        lambda_l2 = result[\"lambda_l2\"]\n",
    "\n",
    "        plt.subplot(1, 2, 1)\n",
    "        plt.plot(history_df[\"val_loss\"], label=f\"LR={learning_rate}, L2={lambda_l2}\", color=colors[_])\n",
    "        plt.plot(history_df[\"loss\"], linestyle=\"--\", color=colors[_])\n",
    "        plt.xlabel(\"Epochs\")\n",
    "        plt.ylabel(\"Loss\")\n",
    "        plt.legend()\n",
    "\n",
    "        plt.subplot(1, 2, 2)\n",
    "        plt.plot(history_df[\"val_accuracy\"], label=f\"LR={learning_rate}, L2={lambda_l2}\", color=colors[_])\n",
    "        plt.plot(history_df[\"accuracy\"], linestyle=\"--\", color=colors[_])\n",
    "        plt.xlabel(\"Epochs\")\n",
    "        plt.ylabel(\"Accuracy\")\n",
    "        plt.legend()\n",
    "\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 134896 characters]",
      "text/plain": [
       "<Figure size 1200x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_results(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "studies",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
