{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Séance 2 - Réseau de neurones convolutionnel\n",
    "\n",
    "On se propose de classifier les chiffres manuscrit du dataset [FashionMNIST](https://github.com/zalandoresearch/fashion-mnist) en définissant ses propres réseaux de neurones convolutionnel. L'objectif est de découvrir la manière d'entraîner ces algorithmes et observer en pratique les bases théoriques discutées en cours.\n",
    "\n",
    "## Exploration des données\n",
    "\n",
    "Commençons par importer les données."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "sns.set(style='whitegrid')\n",
    "\n",
    "from tensorflow import keras\n",
    "\n",
    "(X_train_full, y_train_full), (X_test, y_test) = (keras.datasets.fashion_mnist.load_data())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Consigne** : À l'aide de la fonction [`train_test_split`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html), séparer le jeu d'entraînement complet en un dataset d'entraînement et un dataset de validation. Afficher les tailles des datasets respectifs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(48000, 28, 28) (48000,)\n",
      "(12000, 28, 28) (12000,)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(\n",
    "    X_train_full, y_train_full, test_size=0.2, random_state=42\n",
    ")\n",
    "print(X_train.shape, y_train.shape)\n",
    "print(X_valid.shape, y_valid.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Les classes sont encore des nombres, mais ils correspondent à une catégorie. Pour mieux visualiser, nous allons faire un dictionnaire."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_map = {\n",
    "    0: \"t-shirt/top\",\n",
    "    1: \"trouser\",\n",
    "    2: \"pullover\",\n",
    "    3: \"dress\",\n",
    "    4: \"coat\",\n",
    "    5: \"sandal\",\n",
    "    6: \"shirt\",\n",
    "    7: \"sneaker\",\n",
    "    8: \"bag\",\n",
    "    9: \"ankle boot\",\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Consigne** : Afficher plusieurs images du dataset d'entraînement aléatoirement. On pourra utiliser la fonction [`imshow`](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.imshow.html) et le dictionnaire."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 74856 characters]",
      "text/plain": [
       "<Figure size 1000x1000 with 25 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10, 10))\n",
    "for i in range(25):\n",
    "    plt.subplot(5, 5, i + 1)\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])\n",
    "    plt.grid(False)\n",
    "    plt.imshow(X_train[i], cmap=plt.cm.binary)\n",
    "    plt.xlabel(label_map[y_train[i]])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Consigne** : Standardiser les données en utilisant la classe [`StandardScaler`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html). On commencera par applatir les images en utilisant la méthode [`reshape`](https://numpy.org/doc/stable/reference/generated/numpy.reshape.html), puis on applique le pré-processing et on termine par reformer la matrice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "scaler = StandardScaler()\n",
    "X_train_flat = X_train.reshape(X_train.shape[0], -1).astype(np.float32)\n",
    "X_valid_flat = X_valid.reshape(X_valid.shape[0], -1).astype(np.float32)\n",
    "X_test_flat = X_test.reshape(X_test.shape[0], -1).astype(np.float32)\n",
    "\n",
    "X_train = scaler.fit_transform(X_train_flat).reshape(X_train.shape)\n",
    "X_valid = scaler.transform(X_valid_flat).reshape(X_valid.shape)\n",
    "X_test = scaler.transform(X_test_flat).reshape(X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modélisation\n",
    "\n",
    "On veut définir le réseau suivant:\n",
    "* Deux convolutions avec 32 filtres 3x3 en conservant la taille. On utilisera la couche [`Conv2D`](https://keras.io/api/layers/convolution_layers/convolution2d/)\n",
    "* Une couche max pooling avec un filtre 2x2 et 2 de stride. On utilisera la couche [`MaxPool2D`](https://keras.io/api/layers/pooling_layers/max_pooling2d/)\n",
    "* Une couche [`Flatten`](https://keras.io/api/layers/reshaping_layers/flatten/) puis un réseau dense de 64 neurones\n",
    "* Une couche de sortie à 10 neurones\n",
    "\n",
    "**Consigne** : Définir le réseau souhaité. On sélectionnera la fonction d'activation et la distribution initiale des poids adaptées."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential(\n",
    "    [\n",
    "        keras.layers.Input(shape=(28, 28, 1)),\n",
    "        keras.layers.Conv2D(\n",
    "            filters=32, kernel_size=3, activation=\"relu\", padding=\"same\"\n",
    "        ),\n",
    "        keras.layers.Conv2D(\n",
    "            filters=32, kernel_size=3, activation=\"relu\", padding=\"same\"\n",
    "        ),\n",
    "        keras.layers.MaxPooling2D(pool_size=2, strides=2),\n",
    "        keras.layers.Flatten(),\n",
    "        keras.layers.Dense(units=64, activation=\"relu\"),\n",
    "        keras.layers.Dense(units=10, activation=\"softmax\"),\n",
    "    ]\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Consigne** : Calculer le nombre de paramètre du réseau de neurones à la main, puis vérifier avec la méthode [`summary`](https://keras.io/api/models/model/#summary-method)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"sequential_10\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel: \"sequential_10\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Layer (type)                    </span>┃<span style=\"font-weight: bold\"> Output Shape           </span>┃<span style=\"font-weight: bold\">       Param # </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ conv2d_16 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Conv2D</span>)              │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">28</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">28</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">32</span>)     │           <span style=\"color: #00af00; text-decoration-color: #00af00\">320</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ conv2d_17 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Conv2D</span>)              │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">28</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">28</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">32</span>)     │         <span style=\"color: #00af00; text-decoration-color: #00af00\">9,248</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ max_pooling2d_10 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">MaxPooling2D</span>) │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">14</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">14</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">32</span>)     │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ flatten_10 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Flatten</span>)            │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">6272</span>)           │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense_20 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)             │       <span style=\"color: #00af00; text-decoration-color: #00af00\">401,472</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense_21 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">10</span>)             │           <span style=\"color: #00af00; text-decoration-color: #00af00\">650</span> │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      Param #\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ conv2d_16 (\u001b[38;5;33mConv2D\u001b[0m)              │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m28\u001b[0m, \u001b[38;5;34m28\u001b[0m, \u001b[38;5;34m32\u001b[0m)     │           \u001b[38;5;34m320\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ conv2d_17 (\u001b[38;5;33mConv2D\u001b[0m)              │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m28\u001b[0m, \u001b[38;5;34m28\u001b[0m, \u001b[38;5;34m32\u001b[0m)     │         \u001b[38;5;34m9,248\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ max_pooling2d_10 (\u001b[38;5;33mMaxPooling2D\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m14\u001b[0m, \u001b[38;5;34m14\u001b[0m, \u001b[38;5;34m32\u001b[0m)     │             \u001b[38;5;34m0\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ flatten_10 (\u001b[38;5;33mFlatten\u001b[0m)            │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m6272\u001b[0m)           │             \u001b[38;5;34m0\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense_20 (\u001b[38;5;33mDense\u001b[0m)                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m64\u001b[0m)             │       \u001b[38;5;34m401,472\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense_21 (\u001b[38;5;33mDense\u001b[0m)                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m10\u001b[0m)             │           \u001b[38;5;34m650\u001b[0m │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">411,690</span> (1.57 MB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m411,690\u001b[0m (1.57 MB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">411,690</span> (1.57 MB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m411,690\u001b[0m (1.57 MB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Consigne** : Lancer l'entraînement avec les paramètres adaptés sur quelques époque pour vérifier son fonctionnement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "750/750 - 17s - 22ms/step - accuracy: 0.8602 - loss: 0.3948 - val_accuracy: 0.8903 - val_loss: 0.2993\n",
      "Epoch 2/5\n",
      "750/750 - 15s - 20ms/step - accuracy: 0.9103 - loss: 0.2474 - val_accuracy: 0.9101 - val_loss: 0.2474\n",
      "Epoch 3/5\n",
      "750/750 - 15s - 20ms/step - accuracy: 0.9290 - loss: 0.1945 - val_accuracy: 0.9115 - val_loss: 0.2397\n",
      "Epoch 4/5\n",
      "750/750 - 17s - 23ms/step - accuracy: 0.9442 - loss: 0.1528 - val_accuracy: 0.9158 - val_loss: 0.2382\n",
      "Epoch 5/5\n",
      "750/750 - 17s - 22ms/step - accuracy: 0.9566 - loss: 0.1183 - val_accuracy: 0.9103 - val_loss: 0.2720\n"
     ]
    }
   ],
   "source": [
    "model.compile(\n",
    "    optimizer=keras.optimizers.Adam(learning_rate=1e-3),\n",
    "    loss=\"sparse_categorical_crossentropy\",\n",
    "    metrics=[\"accuracy\"],\n",
    ")\n",
    "\n",
    "history = model.fit(\n",
    "    X_train,\n",
    "    y_train,\n",
    "    epochs=5,\n",
    "    batch_size=64,\n",
    "    validation_data=(X_valid, y_valid),\n",
    "    verbose=2,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Couche BatchNormalization\n",
    "\n",
    "On souhaite mesurer l'apport de la couche [`BatchNormalization`](https://keras.io/api/layers/normalization_layers/batch_normalization/) à un réseau de neurone. Pour cela, on se propose de faire une étude comparative sur le modèle que nous venons de définir. Nous nous proposons de placer la couche BatchNormalization uniquement entre les deux couches de convolution.\n",
    "\n",
    "**Consigne** : Définir une fonction `get_model` qui prend en paramètre:\n",
    "* *normalization*: un booléen indiquant si la couche [`BatchNormalization`](https://keras.io/api/layers/normalization_layers/batch_normalization/) doit être présente dans le modèle\n",
    "* *learning_rate*: un flottant correspondant au learning rate souhaité\n",
    "\n",
    "La fonction renvoie un modèle compilé."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_model(normalization: bool, learning_rate: float) -> keras.Model:\n",
    "    if normalization:\n",
    "        model = keras.models.Sequential(\n",
    "            [\n",
    "                keras.layers.Input(shape=(28, 28, 1)),\n",
    "                keras.layers.Conv2D(\n",
    "                    filters=32, kernel_size=3, activation=\"relu\", padding=\"same\"\n",
    "                ),\n",
    "                keras.layers.BatchNormalization(),\n",
    "                keras.layers.Conv2D(\n",
    "                    filters=32, kernel_size=3, activation=\"relu\", padding=\"same\"\n",
    "                ),\n",
    "                keras.layers.MaxPooling2D(pool_size=2, strides=2),\n",
    "                keras.layers.Flatten(),\n",
    "                keras.layers.Dense(units=64, activation=\"relu\"),\n",
    "                keras.layers.Dense(units=10, activation=\"softmax\"),\n",
    "            ]\n",
    "        )\n",
    "    else:\n",
    "        model = keras.models.Sequential(\n",
    "            [\n",
    "                keras.layers.Input(shape=(28, 28, 1)),\n",
    "                keras.layers.Conv2D(\n",
    "                    filters=32, kernel_size=3, activation=\"relu\", padding=\"same\"\n",
    "                ),\n",
    "                keras.layers.Conv2D(\n",
    "                    filters=32, kernel_size=3, activation=\"relu\", padding=\"same\"\n",
    "                ),\n",
    "                keras.layers.MaxPooling2D(pool_size=2, strides=2),\n",
    "                keras.layers.Flatten(),\n",
    "                keras.layers.Dense(units=64, activation=\"relu\"),\n",
    "                keras.layers.Dense(units=10, activation=\"softmax\"),\n",
    "            ]\n",
    "        )\n",
    "\n",
    "    model.compile(\n",
    "        optimizer=keras.optimizers.Adam(learning_rate=learning_rate),\n",
    "        loss=\"sparse_categorical_crossentropy\",\n",
    "        metrics=[\"accuracy\"],\n",
    "    )\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Consigne** : Combien de paramètre un modèle avec la couche de [`BatchNormalization`](https://keras.io/api/layers/normalization_layers/batch_normalization/) a-t-il ? Est-ce équivalent à un modèle sans [`BatchNormalization`](https://keras.io/api/layers/normalization_layers/batch_normalization/) ? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"sequential_11\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel: \"sequential_11\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Layer (type)                    </span>┃<span style=\"font-weight: bold\"> Output Shape           </span>┃<span style=\"font-weight: bold\">       Param # </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ conv2d_18 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Conv2D</span>)              │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">28</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">28</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">32</span>)     │           <span style=\"color: #00af00; text-decoration-color: #00af00\">320</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ batch_normalization_3           │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">28</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">28</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">32</span>)     │           <span style=\"color: #00af00; text-decoration-color: #00af00\">128</span> │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">BatchNormalization</span>)            │                        │               │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ conv2d_19 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Conv2D</span>)              │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">28</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">28</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">32</span>)     │         <span style=\"color: #00af00; text-decoration-color: #00af00\">9,248</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ max_pooling2d_11 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">MaxPooling2D</span>) │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">14</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">14</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">32</span>)     │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ flatten_11 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Flatten</span>)            │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">6272</span>)           │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense_22 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)             │       <span style=\"color: #00af00; text-decoration-color: #00af00\">401,472</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense_23 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">10</span>)             │           <span style=\"color: #00af00; text-decoration-color: #00af00\">650</span> │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      Param #\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ conv2d_18 (\u001b[38;5;33mConv2D\u001b[0m)              │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m28\u001b[0m, \u001b[38;5;34m28\u001b[0m, \u001b[38;5;34m32\u001b[0m)     │           \u001b[38;5;34m320\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ batch_normalization_3           │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m28\u001b[0m, \u001b[38;5;34m28\u001b[0m, \u001b[38;5;34m32\u001b[0m)     │           \u001b[38;5;34m128\u001b[0m │\n",
       "│ (\u001b[38;5;33mBatchNormalization\u001b[0m)            │                        │               │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ conv2d_19 (\u001b[38;5;33mConv2D\u001b[0m)              │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m28\u001b[0m, \u001b[38;5;34m28\u001b[0m, \u001b[38;5;34m32\u001b[0m)     │         \u001b[38;5;34m9,248\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ max_pooling2d_11 (\u001b[38;5;33mMaxPooling2D\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m14\u001b[0m, \u001b[38;5;34m14\u001b[0m, \u001b[38;5;34m32\u001b[0m)     │             \u001b[38;5;34m0\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ flatten_11 (\u001b[38;5;33mFlatten\u001b[0m)            │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m6272\u001b[0m)           │             \u001b[38;5;34m0\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense_22 (\u001b[38;5;33mDense\u001b[0m)                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m64\u001b[0m)             │       \u001b[38;5;34m401,472\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense_23 (\u001b[38;5;33mDense\u001b[0m)                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m10\u001b[0m)             │           \u001b[38;5;34m650\u001b[0m │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">411,818</span> (1.57 MB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m411,818\u001b[0m (1.57 MB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">411,754</span> (1.57 MB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m411,754\u001b[0m (1.57 MB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">64</span> (256.00 B)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m64\u001b[0m (256.00 B)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_normalized = get_model(normalization=True, learning_rate=1e-3)\n",
    "\n",
    "model_normalized.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pour s'affranchir un peu de l'aléatoire, nous proposons de lancer trois fois les deux types de modèles pour les comparer.\n",
    "\n",
    "**Consigne** : Écrire une boucle d'entraînement qui va stocker dans une liste les courbes d'apprentissage. Chaque élément de la liste correspondra à un dictionnaire avec pour clé:\n",
    "* *type*: le type du réseau (avec ou sans BatchNormalization)\n",
    "* *history*: l'historique d'apprentissage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m20s\u001b[0m 25ms/step - accuracy: 0.8519 - loss: 0.4220 - val_accuracy: 0.8890 - val_loss: 0.3176\n",
      "Epoch 2/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m19s\u001b[0m 25ms/step - accuracy: 0.9057 - loss: 0.2612 - val_accuracy: 0.9048 - val_loss: 0.2682\n",
      "Epoch 3/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m18s\u001b[0m 25ms/step - accuracy: 0.9229 - loss: 0.2076 - val_accuracy: 0.9071 - val_loss: 0.2575\n",
      "Epoch 4/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m18s\u001b[0m 24ms/step - accuracy: 0.9386 - loss: 0.1689 - val_accuracy: 0.9178 - val_loss: 0.2389\n",
      "Epoch 5/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m18s\u001b[0m 24ms/step - accuracy: 0.9499 - loss: 0.1365 - val_accuracy: 0.9142 - val_loss: 0.2515\n",
      "Epoch 1/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m20s\u001b[0m 25ms/step - accuracy: 0.8575 - loss: 0.4019 - val_accuracy: 0.8900 - val_loss: 0.2951\n",
      "Epoch 2/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m18s\u001b[0m 24ms/step - accuracy: 0.9098 - loss: 0.2486 - val_accuracy: 0.9018 - val_loss: 0.2785\n",
      "Epoch 3/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m18s\u001b[0m 24ms/step - accuracy: 0.9287 - loss: 0.1943 - val_accuracy: 0.9118 - val_loss: 0.2507\n",
      "Epoch 4/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m18s\u001b[0m 23ms/step - accuracy: 0.9410 - loss: 0.1596 - val_accuracy: 0.9111 - val_loss: 0.2514\n",
      "Epoch 5/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m20s\u001b[0m 27ms/step - accuracy: 0.9524 - loss: 0.1258 - val_accuracy: 0.9097 - val_loss: 0.2765\n",
      "Epoch 1/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m21s\u001b[0m 26ms/step - accuracy: 0.8556 - loss: 0.4125 - val_accuracy: 0.8885 - val_loss: 0.3105\n",
      "Epoch 2/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m18s\u001b[0m 24ms/step - accuracy: 0.9057 - loss: 0.2611 - val_accuracy: 0.9078 - val_loss: 0.2556\n",
      "Epoch 3/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m18s\u001b[0m 24ms/step - accuracy: 0.9241 - loss: 0.2071 - val_accuracy: 0.9078 - val_loss: 0.2501\n",
      "Epoch 4/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m19s\u001b[0m 26ms/step - accuracy: 0.9381 - loss: 0.1680 - val_accuracy: 0.9144 - val_loss: 0.2460\n",
      "Epoch 5/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m19s\u001b[0m 25ms/step - accuracy: 0.9491 - loss: 0.1378 - val_accuracy: 0.9134 - val_loss: 0.2607\n",
      "Epoch 1/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m18s\u001b[0m 23ms/step - accuracy: 0.8570 - loss: 0.4044 - val_accuracy: 0.8943 - val_loss: 0.2826\n",
      "Epoch 2/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m21s\u001b[0m 28ms/step - accuracy: 0.9096 - loss: 0.2515 - val_accuracy: 0.9045 - val_loss: 0.2609\n",
      "Epoch 3/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m19s\u001b[0m 26ms/step - accuracy: 0.9282 - loss: 0.1999 - val_accuracy: 0.9141 - val_loss: 0.2360\n",
      "Epoch 4/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m17s\u001b[0m 23ms/step - accuracy: 0.9417 - loss: 0.1589 - val_accuracy: 0.9187 - val_loss: 0.2385\n",
      "Epoch 5/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m15s\u001b[0m 21ms/step - accuracy: 0.9533 - loss: 0.1287 - val_accuracy: 0.9013 - val_loss: 0.2911\n",
      "Epoch 1/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m18s\u001b[0m 23ms/step - accuracy: 0.8619 - loss: 0.3881 - val_accuracy: 0.8992 - val_loss: 0.2768\n",
      "Epoch 2/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m19s\u001b[0m 25ms/step - accuracy: 0.9149 - loss: 0.2365 - val_accuracy: 0.8944 - val_loss: 0.2771\n",
      "Epoch 3/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m17s\u001b[0m 23ms/step - accuracy: 0.9308 - loss: 0.1882 - val_accuracy: 0.9162 - val_loss: 0.2368\n",
      "Epoch 4/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m18s\u001b[0m 24ms/step - accuracy: 0.9457 - loss: 0.1482 - val_accuracy: 0.9101 - val_loss: 0.2530\n",
      "Epoch 5/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m19s\u001b[0m 25ms/step - accuracy: 0.9575 - loss: 0.1167 - val_accuracy: 0.9150 - val_loss: 0.2544\n",
      "Epoch 1/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m18s\u001b[0m 23ms/step - accuracy: 0.8632 - loss: 0.3900 - val_accuracy: 0.8923 - val_loss: 0.2962\n",
      "Epoch 2/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m18s\u001b[0m 23ms/step - accuracy: 0.9131 - loss: 0.2449 - val_accuracy: 0.9139 - val_loss: 0.2346\n",
      "Epoch 3/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m17s\u001b[0m 23ms/step - accuracy: 0.9310 - loss: 0.1897 - val_accuracy: 0.9169 - val_loss: 0.2320\n",
      "Epoch 4/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m17s\u001b[0m 22ms/step - accuracy: 0.9446 - loss: 0.1501 - val_accuracy: 0.9151 - val_loss: 0.2353\n",
      "Epoch 5/5\n",
      "\u001b[1m750/750\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m17s\u001b[0m 22ms/step - accuracy: 0.9560 - loss: 0.1180 - val_accuracy: 0.9145 - val_loss: 0.2734\n"
     ]
    }
   ],
   "source": [
    "training_curves = []\n",
    "\n",
    "num_trainings = 3\n",
    "epochs = 5\n",
    "batch_size = 64\n",
    "\n",
    "for normalized in [True, False]:\n",
    "    for _ in range(num_trainings):\n",
    "        model = get_model(normalization=normalized, learning_rate=1e-3)\n",
    "        history = model.fit(\n",
    "            X_train,\n",
    "            y_train,\n",
    "            epochs=epochs,\n",
    "            batch_size=batch_size,\n",
    "            validation_data=(X_valid, y_valid),\n",
    "        )\n",
    "        training_curves.append({\n",
    "            'history': history,\n",
    "            'normalization': normalized\n",
    "        })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Il faut maintenant visualiser les résultats. Commençons par préparer les données.\n",
    "\n",
    "**Consigne** : Définir une fonction `agregate_result` qui prend en paramètre:\n",
    "* *results*: le dictionnaire de résultat, au format décrit précédemment\n",
    "* *network_type*: chaîne de caractère identifiant le type de réseau\n",
    "* *metric_name*: le nom de la métrique d'intérêt\n",
    "\n",
    "La fonction renverra deux matrices de tailles (nombre de comparaisons, nombre d'époque) : une pour le dataset d'entraînement et une pour le dataset de validation. On concatène donc les différentes courbes d'apprentissage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "def agregate_result(results: list, normalized: bool, metric_name: str = 'accuracy') -> pd.DataFrame:\n",
    "    train_curves = []\n",
    "    val_curves = []\n",
    "\n",
    "    for res in results:\n",
    "        if res.get(\"normalization\") == normalized:\n",
    "            hist_obj = res.get(\"history\")\n",
    "            train_curves.append(hist_obj.history[metric_name])\n",
    "            val_curves.append(hist_obj.history[f\"val_{metric_name}\"])\n",
    "\n",
    "    return np.array(train_curves).flatten(), np.array(val_curves).flatten()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Consigne** : Visualiser les courbes d'apprentissage en faisant apparaître des intervals de confiance. On prendra exemple sur la fonction `show_results` du TP précédent. Commenter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 189244 characters]",
      "text/plain": [
       "<Figure size 1400x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axs = plt.subplots(1, 2, figsize=(14, 5))\n",
    "metrics = [\"loss\", \"accuracy\"]\n",
    "colors = sns.color_palette(\"husl\", 2)\n",
    "\n",
    "for idx, metric in enumerate(metrics):\n",
    "    ax = axs[idx]\n",
    "    for normalized in [True, False]:\n",
    "        train, val = agregate_result(training_curves, normalized=normalized, metric_name=metric)\n",
    "        train_runs = train.reshape(-1, epochs)\n",
    "        val_runs = val.reshape(-1, epochs)\n",
    "\n",
    "        mean_train = np.mean(train_runs, axis=0)\n",
    "        std_train = np.std(train_runs, axis=0)\n",
    "        mean_val = np.mean(val_runs, axis=0)\n",
    "        std_val = np.std(val_runs, axis=0)\n",
    "\n",
    "        color = colors[0] if normalized else colors[1]\n",
    "        label_prefix = \"With BN\" if normalized else \"Without BN\"\n",
    "\n",
    "        ax.plot(mean_train, label=label_prefix, color=color, linestyle=\"-\")\n",
    "        ax.fill_between(range(epochs), mean_train - std_train, mean_train + std_train, color=color, alpha=0.2)\n",
    "\n",
    "        ax.plot(mean_val, color=color, linestyle=\"--\")\n",
    "        ax.fill_between(range(epochs), mean_val - std_val, mean_val + std_val, color=color, alpha=0.2)\n",
    "\n",
    "    ax.set_title(f\"Training and Validation {metric.capitalize()}\")\n",
    "    ax.set_xlabel(\"Epochs\")\n",
    "    ax.set_ylabel(metric.capitalize())\n",
    "    ax.legend()\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pour continuer\n",
    "\n",
    "Choisir une ou plusieurs pistes de recherche parmi les suivantes. Il est possible de choisir une autre direction, mais elle doit être validé auparavant.\n",
    "\n",
    "1. Nous avons utilisé la couche [`MaxPool2D`](https://keras.io/api/layers/pooling_layers/max_pooling2d/), mais on peut se poser la question de l'utilisation de la couche [`AveragePooling2D`](https://keras.io/api/layers/pooling_layers/average_pooling2d/) voire l'absence de couche de pooling.\n",
    "2. Nous avons vu en cours qu'une agencement particulier de couches permet d'avoir les meilleurs performance pour la compétition ImageNet: les ResNet. Comment écrire un réseau résiduel à la main ?\n",
    "3. Dans un [billet de blog](https://www.rpisoni.dev/posts/cossim-convolution/) est proposée une alternative à la couche convolutionnelle traditionnelle. On se propose de l'implémenter et d'explorer ses capacités."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "studies",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
