diff --git a/M2/Reinforcement Learning/Lab 3 - Solving Maze Game with Dynamic Programming.ipynb b/M2/Reinforcement Learning/Lab 3 - Solving Maze Game with Dynamic Programming.ipynb new file mode 100644 index 0000000..623abe8 --- /dev/null +++ b/M2/Reinforcement Learning/Lab 3 - Solving Maze Game with Dynamic Programming.ipynb @@ -0,0 +1,2152 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "44b75d44", + "metadata": {}, + "source": [ + "# Lab 3 - Maze Game as a Markov Decision Process Part 2\n", + "\n", + "## **1. Objectives**\n", + "\n", + "Last week in Lab 2, we \n", + "\n", + "- Modeled a simple **maze game** as a **Markov Decision Process (MDP)** by defining:\n", + " - **States**\n", + " - **Actions**\n", + " - **Transition probabilities**\n", + " - **Rewards**\n", + "\n", + "- Implemented **policy evaluation** to compute the value function of a given policy.\n", + "\n", + "We consider a **discounted MDP** with discount factor $\\gamma \\in (0,1)$.\n", + "\n", + "\n", + "This week, we will use **dynamic programming** to find **an optimal policy**.\n", + "\n", + "**Important: Lab 3 starts with Question 12. Questions 1–11 are already included in Lab 2.**\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "100d1e0d", + "metadata": {}, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "\n", + "np.set_printoptions(precision=3, suppress=True)\n", + "# (not mandatory) This line is for limiting floats to 3 decimal places,\n", + "# avoiding scientific notation (like 1.23e-04) for small numbers.\n", + "\n", + "# For reproducibility\n", + "rng = np.random.default_rng(seed=42) # This line creates a random number generator.\n" + ] + }, + { + "cell_type": "markdown", + "id": "1018deab", + "metadata": {}, + "source": [ + "## 2. Maze definition and MDP formulation\n", + "\n", + "We consider a small 2D maze on a grid. The agent is a **robot** that moves on the grid.\n", + "\n", + "- `S` : start state\n", + "- `G` : goal state, with positive reward\n", + "- `#` : wall (not accessible)\n", + "- `.` : empty cell\n", + "- `X` : \"trap\" (negative reward)\n", + "\n", + "At each step, the robot can choose among 4 actions:\n", + "\n", + "$$\n", + "\\mathcal{A} = \\{\\text{Up} \\uparrow, \\quad \\text{Right} \\rightarrow, \\quad \\text{Down} \\downarrow, \\quad \\text{Left}\\leftarrow\\}.\n", + "$$\n", + "\n", + "The movement is deterministic, but here we set a small probability of “error” to make the example more realistic.\n", + "- With probability $1 - p_{\\text{error}}$, it moves in the chosen direction.\n", + "- With probability $p_{\\text{error}}$, it moves in a random *other* direction.\n", + "- If the movement would hit a wall or go outside the grid, the agent stays in place.\n", + "\n", + "We will represent the MDP with:\n", + "\n", + "- A list of **states $\\mathcal{S} = \\{0, \\dots, n_{S - 1}\\}$, each corresponding to a grid cell.**\n", + "- For each action $a$, a transition matrix $P[a]$ of size $(n_S, n_S)$, where\n", + " $$\n", + " P[a][s, s'] = \\mathbb{P}(S_{t+1} = s' \\mid S_t = s, A_t = a).\n", + " $$\n", + "- A reward vector $R$ of length $n_S$, where $R[s]$ is the immediate reward obtained when **leaving** state $s$.\n", + "\n", + "We will use a discount factor $\\gamma = 0.95$.\n" + ] + }, + { + "cell_type": "markdown", + "id": "ca4fa301-c14f-44ec-b04f-b01ca42d979a", + "metadata": {}, + "source": [ + "### 2.1 Define the maze \n", + "\n", + "Let us now define the maze as follows." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "f91cda05", + "metadata": {}, + "outputs": [], + "source": [ + "maze_str = [\n", + " \"#######\",\n", + " \"S...#.#\",\n", + " \"#.#...#\",\n", + " \"#.#..##\",\n", + " \"#..#..G\",\n", + " \"#..X..#\",\n", + " \"#######\",\n", + "]\n" + ] + }, + { + "cell_type": "markdown", + "id": "99820cf4-292d-49ba-b662-f9f05f901f62", + "metadata": {}, + "source": [ + "**Exercise 1.** Compute the dimensions of the maze (complete the “TO DO” parts):\n", + "- How many rows does the maze have?\n", + "- How many columns does the maze have?" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "24d7b74c-66c7-4615-b5e6-c2973a975fc9", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "7\n", + "7\n" + ] + } + ], + "source": [ + "# Solution 1.\n", + "\n", + "n_rows = len(maze_str)\n", + "print(n_rows)\n", + "n_cols = len(maze_str[0])\n", + "print(n_cols)\n" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "26c821d3-2362-4b60-8c77-3d09296d130d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Maze:\n", + "#######\n", + "S...#.#\n", + "#.#...#\n", + "#.#..##\n", + "#..#..G\n", + "#..X..#\n", + "#######\n" + ] + } + ], + "source": [ + "print(\"Maze:\")\n", + "for row in maze_str:\n", + " print(row)" + ] + }, + { + "cell_type": "markdown", + "id": "adc49d58-2730-41d8-96fb-ca7c9cb4fcdf", + "metadata": {}, + "source": [ + "### 2.2 Map each walkable cell (not a wall '#') to a state index\n", + "\n", + "Now we convert the maze grid into state indices for the MDP.\n", + "\n", + "\n", + "The cells where the robot is allowed to stand are \n", + "\n", + "- . : empty space\n", + "\n", + "- S : start\n", + "\n", + "- G : goal\n", + "\n", + "- X : trap\n", + "\n", + "Everything else (i.e., #) is a wall and cannot be a state in the MDP.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "7116044b-c134-43de-9f30-01ab62325300", + "metadata": {}, + "outputs": [], + "source": [ + "FREE = {\n", + " \".\",\n", + " \"S\",\n", + " \"G\",\n", + " \"X\",\n", + "} # The vector Free represents cells that the agent is allowed to move into." + ] + }, + { + "cell_type": "markdown", + "id": "1c9ad05e-9c6c-4e00-918c-44b858f45298", + "metadata": {}, + "source": [ + "**Dictionaries to convert between grid and state index**\n", + "\n", + "We now want to identify all **valid states** of the maze (all non-wall cells). \n", + "To do this, we need two mappings:\n", + "\n", + "1. `state_to_pos[s] = (i, j)`: Given a state index $s$, return its grid coordinates (row, column).\n", + "2. `pos_to_state[(i, j)] = s`: Given coordinates (i, j), return the corresponding state index $s$.\n", + "\n", + "These two dictionaries allow easy conversion between **MDP state indices** and the **physical maze positions**. " + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "a1258de4", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Number of states (non-wall cells): 22\n", + "Start state: 0 at (1, 0)\n", + "Goal states: [16] at (4, 6)\n", + "Trap states: [19] at (5, 3)\n" + ] + } + ], + "source": [ + "state_to_pos = {} # s -> (i,j)\n", + "pos_to_state = {} # (i,j) -> s\n", + "\n", + "start_state = None # will store the state index of start state\n", + "goal_states = [] # will store the state index of goal state\n", + "trap_states = [] # will store the state index of trap state\n", + "\n", + "s = 0\n", + "for i in range(n_rows): # i = row index\n", + " for j in range(n_cols): # j = column index\n", + " cell = maze_str[i][j] # cell = the character at that position (S, ., #, etc.)\n", + "\n", + " if cell in FREE:\n", + " # FREE contains: free cells \".\", start cell \"S\", goal cell \"G\" and trap cell \"X\"\n", + " # Walls # are ignored, they are not MDP states.\n", + " state_to_pos[s] = (i, j)\n", + " pos_to_state[(i, j)] = s\n", + "\n", + " if cell == \"S\":\n", + " start_state = s\n", + " elif cell == \"G\":\n", + " goal_states.append(s)\n", + " elif cell == \"X\":\n", + " trap_states.append(s)\n", + "\n", + " s += 1\n", + "\n", + "n_states = s\n", + "\n", + "print(\"Number of states (non-wall cells):\", n_states)\n", + "print(\"Start state:\", start_state, \"at\", state_to_pos[start_state])\n", + "print(\"Goal states:\", goal_states, \"at\", state_to_pos[goal_states[0]])\n", + "print(\"Trap states:\", trap_states, \"at\", state_to_pos[trap_states[0]])\n" + ] + }, + { + "cell_type": "markdown", + "id": "721b968c-a355-46eb-aae4-5950441ba604", + "metadata": {}, + "source": [ + "*Hint.* If you don’t know what a dictionary is in Python, try the following code to help you understand." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "68744dd6-7278-4c20-8b82-34212685352f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "value2\n" + ] + } + ], + "source": [ + "my_dict = {\"key1\": \"value1\", \"key2\": \"value2\"}\n", + "print(my_dict[\"key2\"])" + ] + }, + { + "cell_type": "markdown", + "id": "0c76f4e1-b0ba-49c5-b9d5-cfb523024ba9", + "metadata": {}, + "source": [ + "**Exercise 2.** Read the program above and answer the following questions:\n", + "1. What is the purpose of state_to_pos and pos_to_state?\n", + "2. Why do we only assign states to cells in FREE?\n", + "3. What would happen if the maze had multiple goal cells?\n", + "4. What is the total number of states (n_states) in this maze? Does this match the number of non-wall cells you can count visually?" + ] + }, + { + "cell_type": "markdown", + "id": "d45828e3-43be-4318-a14c-1242d3a0dcbc", + "metadata": {}, + "source": [ + "**Solution 2.**\n", + "1. `state_to_pos` maps: \n", + "$$\n", + "\\text{state index} \\quad s\\quad \\rightarrow \\quad \\text{grid position} \\quad (i, j)\n", + "$$\n", + "\n", + "`pos_to_state` maps: \n", + "$$\n", + " \\text{grid position} \\quad (i, j) \\quad\\rightarrow \\quad \\text{state index} \\quad s\n", + "$$\n", + "\n", + "We need both because:\n", + "\n", + "- `state_to_pos` lets us visualize, display, or plot the value function on the grid. \n", + "- `pos_to_state` lets us convert a grid position into the correct MDP state index, useful when building transition probabilities.\n", + "\n", + "2. We only assign states to cells in `FREE = {'.', 'S', 'G', 'X'}` because only these cells are **walkable**. Wall cells (`'#'`) **cannot be entered** by the agent, so they are **not included as MDP states**.\n", + "\n", + "3. If the maze had multiple `'G'` cells (several goal locations), we store them in a **list**, for example, goal_states = [5, 12, 23].\n", + "\n", + "4. 22 states. (Row 1: 5 free cells; Row 2: 4 free cells; Row 3: 3 free cells; Row 4: 5 free cells; Row 5: 5 free cells)\n" + ] + }, + { + "cell_type": "markdown", + "id": "6d0fa298-7b7c-44fc-bbed-15ea002037c2", + "metadata": {}, + "source": [ + "-----\n", + "\n", + "The following function `plot_maze_with_states` creates a figure showing:\n", + "- the maze walls and free cells\n", + "- the state index for each non-wall cell\n", + "- special labels and colors for S (start state), G (goal state), and X (trap state). " + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "fc61ceef-217c-47f4-8eba-0353369210db", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGbCAYAAAAr/4yjAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAK99JREFUeJzt3Qd4VFX6x/E3CS1CEiSAofcS+spKEZCmgiBZKQqKiIuCrIDArmBBKSu661IUQYoNpFhoYhCpSlEEFekIiHThTwslEaRm/s97xjlMMAlRk9wp38/zXObOzZA5d+7k/O4pdybE5XK5BAAAEQl1ugAAAN9BKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUowHEhISEydOjQDD+2d+/eEoyaNGlilqz8nfv27TOv8ZQpUzL1eeA/CAUfoX+E+seoy5dffvmbn+unkZQoUcL8/O6775ZA9tVXX5mQOH36tF8+9/jx46lU4bdyOF0ApJQnTx557733pGHDhim2r1y5Un766SfJnTu3BJpffvlFcuTIkaJiHjZsmDz88MOSP3/+bC1LZjy3hkLBggXN78hMS5YskaxWqlQpczxy5syZ5c8F30RLwce0atVKZs2aJZcvX06xXYOidu3aEhMTI4EYhN6hgNTlypXLLFlJW6J6PMLCwrL0eeC7CAUfc//990tCQoIsXbrUbrt48aLMnj1bHnjggVT/z8iRI+XWW2+V6OhoCQ8PN+Ghj/emZ62e7qlrF+/+/AsXLsiQIUOkfPnyplWiXVYDBw4029Pz2muvmYrEu9tl1KhR5vf/85//tNuuXLkiERER8tRTT9lt3mXQ2wEDBpj1MmXK2DJqX7e3efPmSbVq1UwZq1atKosWLZKMGDt2rHn8DTfcIDfeeKP89a9/NYGbkeeePHmyNGvWTAoXLmyet0qVKjJhwoQUv7906dKybds207Lz/H/vPnt9ffr162deV/0d+jq//PLLkpyc/Lv7/1esWGF+/8yZM+XFF1+U4sWLmwq9efPm8uOPP/7m/7/xxhtSrlw58x6pU6eOfPHFF795TFpjCjt27JD77rtPChUqZP5/pUqVZNCgQSkec+jQIenWrZvcdNNN9ri88847v+sYwHmcnvkYrVTq168v77//vtx1111m28KFC+XMmTPSqVMnU/lea8yYMRIXFyedO3c2AfLBBx/IvffeK5988om0bt3aPOaxxx6T22+/PcX/04p0xowZppJTWjHp79ExjR49ekhsbKxs2bJFXnnlFfnhhx9MRZyWRo0amf+v/9cz5qGVTmhoaIrKZ8OGDfLzzz/LbbfdlurvadeunXku3X99Xu2GUVoZeehzzJ07Vx5//HETMPqatG/fXg4cOGCCMS1vvvmmPPHEE9KhQwfp27evnD9/XjZv3ixff/21CdzrPbcGgFZm+hppy2b+/PmmDLrfvXr1Mo959dVXpU+fPpIvXz5baWolqc6dOyeNGzc2lacej5IlS5ruqmeeeUb+7//+z/zfP+K///2veZ2ffPJJ8z753//+Z94Lul8eb7/9tnlOPXnQUNqzZ4/ZjwIFCpiASo++Rnp8tUtJ3xf6Ht29e7fZfw0jdfToUalXr56dCKCvmb5vH3nkEUlMTDTPmZFjAB+g36cA502ePFm/18L17bffusaNG+eKiIhwnTt3zvzs3nvvdTVt2tSslypVytW6desU/9fzOI+LFy+6qlWr5mrWrFmaz7dr1y5XVFSU64477nBdvnzZbJs2bZorNDTU9cUXX6R47MSJE03ZVq9enebvu3LliisyMtI1cOBAcz85OdkVHR1tyh4WFuZKSkoy20ePHm2e49SpU/b/6u8eMmSIvT9ixAizbe/evb95Ht2eK1cu148//mi3bdq0yWwfO3asKz1/+9vfXFWrVk33Mek997Wvs2rRooWrbNmyKbbpczRu3Pg3j33hhRdcefPmdf3www8ptj/99NPmNTpw4EC6ZdPf6f17ly9fbsoaGxvrunDhgt0+ZswYs33Lli32/VC4cGFXrVq1UjzujTfeMI/z/p2637pN348et912m3k/7t+/P0V59Bh7PPLII64iRYq4Tpw4keIxnTp1Mu8zz2uXkWMAZ9F95IO0ma6DfXqmn5SUZG7TO4vS5rzHqVOnzNmintmtX78+1cefPXtW2rZta5ruelbs6T/WsQxtHVSuXFlOnDhhF+0yUcuXL0+zDHqmqmehq1atMve3b99uusGefvppM3NqzZo1Zru2GrTb588MIGuLR7tBPGrUqCGRkZHm7Dc9+pw6WP/tt9/+oef1fp31NdbXRs/89Xn1/vXo66vHRV9379dX90e71Tyv3e/197//PcVYgz6H8rwe69atk2PHjknPnj1TPE67FKOiotL93cePHzfl0m4hbdl401aB0uM7Z84cadOmjVn33rcWLVqY18bzXvyzxwBZj+4jH6RNb60otJ9Vuxy0wtDmdlo0NIYPHy4bN25M0ffv+aO9Vvfu3U3zX7suvLtbdu3aZSpz764ab1qxpEcrI+2X10DTyr9IkSJy8803S82aNc39O+64w3T9aOj9GddWTkorWg3E9Og4xrJly0x/uvbl33nnnSZsGzRokKHnXb16tRlv0YDT4+JNK77rVbD6+mpXyR99fTP6euhroTyvx/79+81thQoVUjxOu4PKli2b7u/2BIsGeXrBoWMlOmahS3r79mePAbIeoeCj9A9FK+8jR46YsYW0zqy1stW+Ye2j16mQWhHrH7sOiqY2eKfjD9o6mD59utSqVSvFz7RvvHr16jJ69OhUn+t6fc86jfbSpUum0tRyec5Y9Vbv62ClViCe7X9UWjNjrvfNstoK2rlzpwlRHU/Rs1t9zQYPHmymoaZHQ1QHcLUVpa+PvhZ61v3pp5+a8YeMDBTrYzQYdeA+NRUrVpTsfD0yi2ffH3zwQenatWuqj9HW3J89BsgehIKP0u4dHRhcu3atfPjhh2k+Tv+odMbJ4sWLU1zDoKFwLa2YdTBSB/10IPJa2iWzadMmU/ml1cpIj579aUWpz6OLZyaPBpYOMH722Wf2fnr+yHNnVN68eaVjx45m0UF5HVzWwVId7NXXMa3n1kFVbYXFx8enODNPrUstrd+hr68Osl874J8d1x54WiqerkClAb53717TkkuLpyWxdevWNB+jLR8d8NcWbUb27XrHAM5iTMFH6ewVne2i3THaV5veWaJWQvoH6T2t8NqZQjq7Rbtt9Gx+xIgRqf4u/bnOjNEK/FraJaRjEenRP+hbbrnFtER0JpB3S0H/v84S0opRWzPXqzRUZl/RrGMc3jTAdFqpnlFrBZnec3vOxr3PvrXLKLXw1d+RWtn19dVWlAb4tfTx116bkll0yqdW3BMnTjSVsIdOO73ea6z/T0Ncp5bqMfXmeS30tdHZX3qCklp4aOvw9xwDOIuWgg9LqynuTaecandGy5YtTZeT9t2+/vrrpr9W+689dBqg/nFq14VOWb22aa9Lly5dzJx3HZDUM2Dt59Ww0W4f3a6VmVYw6dEA0CmS2r+uXVFKp7zqvHbtNsjIVb56nYXSKZ06DVe7wzQYPRX2H6X913rxn+6XThPV8ZNx48aZ11DPdNN7bv2/WoHpurbg9Ixfw1P3TQP32vJroOs4jx4HfYyeoWvLSVsaOmVXXwd9nAatTvvV60o0zD3TYDOT7oOWRcut5dAzdG0haKBdb0xBaZjryYSOD+mUVL2GQ8u6YMECM46l9Jjre6Zu3bqm21Mr+pMnT5oBZh1D0PWMHgM4zOHZT0hlSmp6UpuS+vbbb7sqVKjgyp07t6ty5crmd+kUT+/Dq9MO9X5qi/d0UJ2++PLLL5tpg/r7brzxRlft2rVdw4YNc505c+a6+7FgwQLzO++6664U2x999FGzXct6rWvL4Jm+WaxYMTN91XuKqK736tUr1dela9eu6ZZt0qRJZnqlTpXVfStXrpxrwIABv9mvtJ47Pj7eVaNGDVeePHlcpUuXNq/TO++885sprEeOHDHHSKdxXjvlU6fmPvPMM67y5cubqbUFCxZ03Xrrra6RI0ea1/6PTEmdNWtWiselNq1UjR8/3lWmTBmz73/9619dq1at+s3vTOv/bt261dW2bVtX/vz5zf5XqlTJ9fzzz6d4zNGjR82xKVGihCtnzpyumJgYV/Pmzc3U1997DOCcEP3H6WACAPgGxhQAABahAACwCAUAgEUoAAAsQgEA8PuuU9DL2A8fPmzmEWfl1aYAgKyhE031AzaLFi1qPsDyT4WCBsL1PvcGAOD7Dh48aL6Q6U+FgudKQ/1cE742EQD8j36Mil5dfr0rxzNUw3u6jDQQ+EJvAPBf1xsCYKAZAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAK4f8Dh9++KFERkb+nv8C4BpxcXESaOLj4yXQxAXgccoIWgoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAAH/sA/GywsUrF2XkVyNl+ubpsv/MfgkLCZPCeQtL9Zuqy9DGQ6VmTE2niwgAQcPxlsKAJQNk0OeDZPuJ7VIsopiUzl9ajp09JvN2zJNdJ3c5XTwACCqOtxQ+3PahuR1822AZ1nSYWXe5XPLVwa9MiwEAEEShkOxKNrdL9iyRW4rdIrcUvUVuyneTNCjZwOmiAUDQcbz76PFbHje3a39aK23ebyMxo2Kk8rjK8sLKF+T85fNOFw8AgorjoTC0yVCZe99caVOxjUTmdn+r286EnTJ4xWDp+UlPp4sHAEHF8VBQbWPbSvz98XLqqVPyzaPfSPXC1c12HWwGAARRKDz3+XOy8chGd2FCQs24QsXoiuZ+VJ4oh0sHAMHF8YHmt9a/JS9+8aIUvKGglIwqaaaj/pT4k/nZA9UecLp4ABBUHG8pDG82XP5W6W8SkStCdpzYYUKhUnQlGdJ4iLzQ7AXxV6tWrZJWrVpJoUKFJCQkxCwTJ04UfzVq1Chp0qSJFClSRHLnzi2lSpWSrl27yp49e8Sfvfrqq1KzZk3Jnz+/2a/ixYvLvffeK5s3b3a6aEjFfffdZ/+eOnXq5HRxApLjLYVHb37ULIFm/fr1snTpUilbtqycOHFC/N3YsWPlwIEDUqlSJQkPD5e9e/fK1KlTZcmSJbJz506JjHRPEvA3K1eulOPHj5vjdP78ebMvs2fPls8//9zsb968eZ0uIn41efJkmTVrltPFCHiOtxQCVZcuXSQxMVEWL14sgaB79+6yb98+2b59u2kd9OvXz2w/cuSIfPbZZ+Kv3n//fTl8+LAJ8e+//16effZZs/3kyZOyY8cOp4uHX+3evVueeOIJqV+/vmnNIesQClkkOjranFEHikGDBknJkiXt/UaNGtl17XbxV3ny5JGPPvpI6tWrJ1WqVJGXXnrJbNduv4oV3RMe4KzLly9L586dJTQ0VGbMmCFhYWFOFymgOd59BP9z5coVeeONN8y6drs0b95c/NnRo0fl66+/tvfLlCkj8+fPl4iICEfLBbdhw4aZ4zN9+nRzbJC1aCngdzl79qy0bdvWdIvFxMSYytOfWwqqZ8+ekpycLPv375eOHTua8RK9TUpKcrpoQW/dunXyn//8Rx588EHTWkDWIxSQYTp+0LhxYxME2rWyevVq0+USCHQ2i3aPecYUtm3bZsYb4KytW7ealqkO/ufLl88sOgFAzZkzx9w/c+aM08UMKIQCMkQrSe13/+6778x4wpo1a0zXkT9LSEiQadOmycWLF+22Tz/9NEWrCL5BZ4bp8dBFP0XZM9bgfR8BFgofbP1Abp50s4S/GC4FXi4gHWZ2kN0nd4u/mjt3rpQvX97M7fcYPHiw2eaPzeB27dqZ7hWl3Sp6DYaGhC5vvfWW+CPdj4ceeshco1C9enXTUnjmmWfMz3Q8QfcZznr44YdNpe+96DUySrv49L4ePwTYQPPb69+WR+e7r1Uok7+MJPySIHO2z5EvDnwhm3pukph8MeJvdDqqTqPzpvPhdfHHKXUXLlyw6xs3uj+WxKNly5bij7Qy0QugvvnmG3OsLl26JCVKlDBdZNqN5Kl8gGAS4spA20sruKioKNN3l9kXKenXcRYbXUxOnDsh7WPby+z7ZsvhpMPm47OTLiZJnzp95LW7XsvU5wScFBcXJ4EmPj5eAk1cgB0nPelZtGjRdetxx7uPvj30rQkEpaGgikYUlXrF65n1RT8ucrR8ABBMHA+Fg4kH7br312/qt6+pA2fcMw0AAEEQCmlhRgEABGEolIgsYdf1E1KvXdeP0wYABEko6JfqRIdHm3WdcaR0oFm/s1m1LO+fM1sAwB85Hgq5wnLJS81fsqFQdkxZiX091sw80i/eebrh004XEQCChuOhoHrU7iHT206XWjG1TCshREKkXWw7+arbV2YmEgAgiC5eU51rdDYLACDIWwoAAN9AKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAIA/9oF4HTt2lJw5c0qg4MvGfV8gHqNA3CcEDloKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahkAWGDh0qISEhqS6XL192uni4xvHjx6VPnz5SqlQpyZUrlxQsWFCaN28ue/bsEX+zb9++NN97uuh709+cPXtWBg4cKBUqVJAbbrhBoqKipEaNGjJixAhxuVxOFy/g5HC6AIFMK5dy5cql2KZ/mPAdJ06ckLp168revXtNIFSsWNFUNGvWrJHDhw9L2bJlxZ/kzp3b7I+306dPy86dO816kSJFxN/06tVL3n33XbNetWpVOXPmjGzZssUERZ48eUygI/MQClmodevWMmXKFKeLgXQ899xzJhC0slm6dKmtNC9evOiXZ6Fa/rVr16bY1rt3bxMKN954o3Tu3Fn8zZdffmluW7ZsKQsXLpRffvlFChQoIOfPn5f9+/c7XbyAQ/dRFpozZ46Eh4ebP9S7775bNmzY4HSR4EUr/ZkzZ5r1EiVKyB133CF58+aVmjVrmmOnZ93+LiEhQSZPnmzW//GPf0i+fPnE3zRq1MjcLlq0SKpVq2ZacxoIuv1f//qX08ULOIRCFgkLC5OYmBgpXbq0HDlyRBYsWCD169cnGHxsLOHUqVO2wtFuFj2b3rx5szzwwAMye/Zs8Xfjx4+Xc+fOmYDz126WiRMnykMPPWTWt23bJj/99JPp6tNxBT1eyFyEQhbQCuXYsWOya9cu2b59u6lw1IULF+T11193unj4lfegf2xsrBlY1kXX1bhx48Sfeb/fHnzwQXOS4o9eeeUVmTZtmjRo0MD8XWkwREREmH17+umnnS5ewCEUsoA2b7XP06NFixYSHR1t1g8cOOBgyeCtUKFC5oxTaZeRruui656ZPP5s6tSpcvToUTO5wV+7WbSV8/zzz5uuvvbt25tjVqVKFRMQatmyZU4XMeAQClng5ZdfTlH56wCm9u0q7U6Cb8iZM6fcdtttZl27jC5dumQWXVc6BdJfaSU6atQoO+HB0/rxx1DwtOi+++47c6vjCdpaUDoGhMxFKGSBCRMmmMpf573rWY22FDxv4H79+jldPHgZPny4aR18//33UqZMGbPouo4JPfvss+Kv5s+fb6ehDhgwQPx5WrcnuGfMmGGCWv+2du/ebbZ17drV4RIGHkIhC2hlohc/6Vmn9lFrOOhUQD3T0ZCA79A5/Z9//rk0adLEDDrrWejtt98uq1evlqZNm4q/GjlypLmtU6eOrVT91bx588w1Cdotq9eO6HRhPW7Tp0+Xxx9/3OniBZwQVwYmYycmJpqrCHWesDa5A0V8fLzTRch0cXFxEkgC8RjBP8QF2N+SnqTqpBe9+C8yMjLNx9FSAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALByXF0FgD8m0L7kPpjRUgAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoZJGff/5ZhgwZIpUrV5bw8HApWrSo/OMf/5BTp045XbSgtWrVKmnVqpUUKlRIQkJCzDJx4sQUj7l06ZIMGzZMypYtK7ly5ZLixYtL//79zfH0x/2ZNGmSNGzYUPLmzWsfs2PHDsfKDN9HKGSRNm3ayL///W/58ccfpWLFiqZS0T/YO++8Uy5fvux08YLS+vXrZenSpVKgQIE0H9OtWzcZOnSo7N+/3wTDsWPH5NVXX5W7775bkpOTxd/2Z+HChbJhwwYTHEBGEApZ4Pvvv5cVK1aY9TFjxsimTZvku+++M/fXrVsnM2fOdLiEwalLly6SmJgoixcvTrOSnT59uj1uekY9Z84cc3/lypUyb9488af9UePHjzeP0aADMoJQyALeZ5ShoaEpbtWyZcscKVewi46ONl156Z1Ve7Rv397ctm7dWvLkyWPWFy1aJP60P0q7LcPCwrKtTPB/hEIWiI2NlWrVqpn1Pn36SK1ateTmm2+2Pz906JCDpUNaDh48aNcLFy5sw7xgwYJm/cCBA46VDcguhEIW0DMzPevs3LmzqVD27NkjjRo1knLlypmf58yZ0+ki4ndwuVxOFwHINjmy76mCi85a8fRPq/Pnz0tMTIxZr1SpkoMlQ1pKlChh13WAuUiRIqYrMCEhwWwrWbKkg6UDsgcthSyig5ZJSUlm/cqVKzJgwAA5c+aMud+xY0eHS4fUtGzZ0q57BpgXLFhgAv3anwOBilDIIu+8847pl65evbppIYwbN85s79evn9SpU8fp4gWluXPnSvny5aVJkyZ22+DBg8027eqrXbu23H///WZ73759zdiQZ8BZu//uuece8af9UU899ZS5r7ceLVq0MNtee+01R8oN30b3URbRin/58uVmPEH7pLXC0YvXHnnkEaeLFrR0aubu3btTbDt+/LhZtLtPvfvuu1KhQgWZOnWqeazO7+/QoYMMHz48xQwyf9mfo0eP/uYxngHzkydPZmNp4S9CXBkYRdM3X1RUlGk+B9IgaXx8vASauLg4CSSBeIwCUaC97wKRXq2v06q1GzsyMjLNx/nWqQ8AwFGEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsHJcXUUgCLQvug/EL4QPtGMUqPsUaBITEyUqKuq6j6OlAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAIBsdv68yCuviNx6q0j+/CK5c4uULCly++0io0c7WzY+OhsAslFCgkjz5iKbNrnv33CDSMWKIklJIitXinz2mcg//+lc+WgpAEA26t37aiD07esOiS1bRPbtEzlxQmTyZGfLR0sBALLJ6dMis2a512vWdHcVhXqdmut34Dz8sDiKlgIAZJMffhC5csW93qjR1UC45x6RkJCry5QpzpWRUAAAB4R61b6VKrlbDr6AUACAbFKpkkhYmHv9q6+ubn/5ZZEPPhCfQCgAQDaJihK57z73+rp1IkOGXO1O8hWEQiZYtWqVtGrVSgoVKiQhISFmmThxov15UlKS9OvXT2rXri0FCxaU8PBwqVixojz//PPmZ/64T6pbt25SoUIFyZcvn+TNm1fKlSsnTzzxhJw8edKxcgeTjBwjD32f6fG53uP8YZ+aNGlif+a9NGzYUPzB2LEiNWq41//9b5ECBUT+8hfdL/EJhEImWL9+vSxdulQK6NFNRUJCgowZM0a2bdsmxYsXN5Xorl27ZPjw4dKxY0fxx31SH3/8sVy5ckUqV65swm7Pnj0yduxYeeCBB7K1rMEqI8fIo3fv3ub4BNI+lS1bVurWrWuXqlWrij+IjhZZu9bdZVS7tkhyssiOHSLh4SItWohoBurAs1MIhUzQpUsXSUxMlMWLF6f68zx58siIESPk+PHjsnHjRjl48KDUq1fP/GzhwoVy6tQp8bd9UocOHTIVzbp162T//v32TG316tXZWNLglZFjpGbOnClTp06V+zz9FgGwT0pb2mvXrrXLpEmTxF+Eh4sMHOjuQtLOgl9+Edm7V2TRIpHHHnNf5ewUQiETREdHmy6htMTExMiTTz4pERERNiRuueUWsx4aGio5cuTwu33y7If+YepZWunSpeXLL7802/2lGe/vMnKM9ATkscceM12X2jINhH3y6N+/v+TOndu0GHr06CFHjx7N8vIFA0LBAceOHZM5c+aY9U6dOtmw8EfaDfbNN9+YloK6/fbbzZkpnJecnGzOvC9duiTvvfee5MyZUwKFBkexYsXM2MPevXvlzTfflPr168vZs2edLprfIxSy2e7du82Z9OHDh6VBgwY+O+CXUR988IFcvHhRNmzYINWqVZNly5ZJr169nC4WRMw41sqVK82tTmwIFK+88orpct26datpCT3zzDNmu4bDRx995HTx/B6hkI3WrFljxhL07LpNmzayZMkSv24leOgZaK1ataR79+7m/rRp0+QHvXQTjtr06wfs9O3b10xu8B6I1dlwt+pHdPqhv/zlL6bbSOmsI++JDQcOHHCwZIGBUMgms2fPlmbNmsmJEyekT58+Mm/ePLlBPx7RT3377beyYsUKe19bC9pK8KAZ7zv0WOhy7tw5u+3ChQsp7vtT1+vo0aNTTOX+8MMP7bqObeHPIRQywdy5c6V8+fJm/rTH4MGDzbbOnTubriKd+XH+/HnJlSuX6YPXszRtNeii0/D8bZ90em3Tpk3N1EFtJRQpUkTmz59vHqf3a/rKNfsB7HrHaMqUKeJyueyi3SseEyZMMDPh/G2fNMj+9a9/mfddbGyslCxZ0g6g6/127dqJr1m1SqRVK5FCha5+tlFqvcabN4t06OB+XK5cIsWKXb3QLTv53rQXP6RT6HSswJtOP9VFr0vQs2j9o1S6/vXXX//m//vbPun4QcuWLU0Xxffffy9hYWHmj7J169by7LPPmllVcPYYBeI+6cDyoEGDTNerPu6XX34x18ncc889MnDgQDMjztesXy+ydKleV+H+aOzU6MS9O+90T02NjBTRnr6ff9ZrgbK7tCIhLk9tdZ0DFRUVZSqBQJrBEB8f73QRcB1xcXESaHjfBZeEBPcX6eiM2TJl3NsmTBDp2dO9rjVwlSruC9g6dxZ58033dQxKe8kya9jRU4+fOXNGIjV50sDpHABk8RXM4elceqHdRhoInoDQD83Tz0hq1sz9UdvZjVAAAAft3Hl1/b333K0KtXy5+/OQ9BvZshOhAAAOunz56vojj7hbDToHQD9iW8cVsvsLdwgFAHBQsWJX13/99Bsz9qCzkBQtBQAIInXquGccKf2APKWfGnP8uHu9QoXsLQ+hAABZaO5ckfLlU35fwuDB7m0620gHoYcOdW9/6y293sL91Zz65TsxMSI9emRveQkFAMhCiYn6mWfus38PbQXotkOH3Pf793cHQrVq7o/Q1mmoXbq4Ww6ebqTswsVrAJCFHn7YvVyPDjLr4jRaCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFh+IB5/Gl9zDKXFxcRJILl26lKHH0VIAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSggOC1fLhIaKhISIvLf/17dfuWKSL167u2lSokkJjpZSiDbEQoITk2bivTt614fMkRk82b3ugbE11+7Q+Hdd0UiIx0tJpDdCAUEr//8RyQ2VuTiRZEuXUS++UZk2DD3z/r1E2nSxOkSAtmOUEDwypNHZNo0kRw53C2Fxo31m0hEqlQReeklp0sHOIJQQHCrXVvkuefc6+fPi4SFuYNCAwMIQoQCsGtXyoHmffucLA3gKEIBwW3OHJEZM9zrOttIPfaYyNGjjhYLcAqhgOClFX/Pnu71Vq1E1qwRiY4WOXFCpHt3p0sHOIJQyASrVq2SVq1aSaFChSQkJMQsEydOtD+fMmWK3Z7asmLFCvG3fVK7d++WBx98UEqUKCG5c+eWggULSuPGjeXjjz8Wv/Doo+4AKFBA5K23RIoUEZkwwf2z+fNF3n5bfNmoUaOkSZMmUqRIEfP6lypVSrp27Sp79uyxj7l06ZIMGzZMypYtK7ly5ZLixYtL//795eeffxZ/3adJkyZJw4YNJW/evPa9uWPHDkfLHUgIhUywfv16Wbp0qRTQyiUVWrHWrVs3xaJveo+YmBjxt31yuVxyxx13yIwZM+T48eNStWpVuXLligmTtm3byqZNm8SnaQh88ol7ffx4dyCoe+8V6dzZvd6/v0+PL4wdO9a83vnz55dixYrJgQMHZOrUqdKgQQNJ/PWiu27dusnQoUNl//79JhiOHTsmr776qtx9992SnJws/rhPCxculA0bNpi/K2Q+QiETdOnSxbxhFy9enOrPW7duLWvXrk2x6Fm10oq1cuXK4m/7dOjQIdm7d69Z1zNRDZG5c+fawDh48KD4fCvB5XIvHTum/Nn06e7tWgmVLi2+qnv37rJv3z7Zvn27OZPup9dWiMiRI0fks88+M8dkuu6LiIwZM8acTc/RMRQRWblypcybN0/8bZ/U+PHjzXtTww6Zj1DIBNHR0RIeHp7hxy9atEi2bNli1gcMGCD+uE/a0ilfvrxZHzJkiNx8883Srl07yZEjhzk7veuuu7KxtMFp0KBBUrJkSXu/UaNGdl27XvSM2qN9+/b2BCXPr9Nt9X3ob/ukihYtKmE6dRhZglBwwIgRI8xtzZo1TUvBH+kf5fLly6V27dpy4cIF05w/ffq03HjjjSYg+KPNXtp198Ybb5h17SZq3rx5itZa4cKFzW1oaKhtpWrXjL/tE7IeoZDNtPL8/PPPzfqTTz4p/kr7o3v27Cnfffed9O3b1wxczpo1y4wv9O7d2ye7JgLV2bNnzTiOdvXp+NT8+fPtWXVqtHsv0PYJmYdQyGYjR440tzpjp1OnTuKvtH93wYIFZl1nh+hMkA4dOkjkrx8gt2zZModLGBy0r11nfGmlWbFiRVm9erVU0Y/p+PU95qEDzJ4wT0hIMOve3TT+sk/IeoRCNtLm+syZM826nl1r/7u/OnPmjF1ft26duf3hhx8kKSnJrGtIIGtt27ZN6tWrZ1pr2ve+Zs0a083i0bJlS7vuGWDWID+vH+dxzc/9ZZ+Q9QiFTKCzbnTQVedXewwePNhs6+yZ3ihipgJevnxZoqKipEePHuLP+9S0aVMzfqC0G6l69epmLEG7JnLmzCn333+/+JxVq9wXqelURv1obF2uufbCTEF9+GH31c06IFupksj//qen2OJrdGBfp5oqDWO9rkQrVF3eeustM97jOQ56EhIbG2sHnLXCveeee8Tf9kk99dRT5n2otx4tWrQw21577TXHyh4o/PdU1Yfo9Di9kMub9q3rohcLec6sPW9qDYSIiAjx533S2UnarH/xxRfliy++kF27dpmQ0Gb/c889J7Vq1RKfs369yNKlOmrpvmjtWsePi9Sp477Nl09Epwpv3aq1kMjhw5rq4kt0gN9j48aNKX7maQW8++67UqFCBTPXX4+nzu3Xbr7hw4ebQWdfk5F9Onr06G/em55B85MnT2ZLOQNZiCsDo05aQejZrR4UPQsMFPHx8U4XAdlJ+9JvuMH98RZlyri36RXMno+60IvYevVyr2sYVK0q8uabmuLuT0/V6zK8+ukR2OLi4iSQ6NXtOg1ZT1A9Y3+p8b1TBSCr6OcapXc9iXcXkecs2nOrn56qX+EJBDhCAfDQ8QbtNlJ164poF5inFaEOHXKsaEB2IRQADx1rWLLE/f3N2kLQcQQddNYBaRVAXadAWggFwFv9+iJ6ceHp0zq5Xz9Rzv05SEpnIgEBjlAAvH35pXv8QJ06pZedu9f1oyH4mAUEAUIBwUM/xVU/xM/r2gsZPNi9zXM9iY4haADUqCGi04m/+so980ivZ9CZS0CAIxQQPPSjsHV++68XRxl6TYJu8wwi33mniE7X27lTRK841/vanfTrRV9AoOPiNQQPHTTWJT2jR7sXIEjRUgAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsIL6A/EC7Yu5AeDPoqUAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAAKwckgEul8vcXr58OSMPBwD4GE/97anP/1QoJCUlmdtly5ZlRtkAAA7R+jwqKirNn4e4rhcbIpKcnCyHDx+WiIgICQkJyewyAgCymFb1GghFixaV0NDQPxcKAIDgwEAzAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAPH4f8EOdr53tgDzAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "def plot_maze_with_states() -> None:\n", + " \"\"\"Plot the maze with state indices.\"\"\"\n", + " grid = np.ones(\n", + " (n_rows, n_cols),\n", + " ) # Start with a matrix of ones. Here 1 means “free cell”\n", + " for i in range(n_rows):\n", + " for j in range(n_cols):\n", + " if maze_str[i][j] == \"#\":\n", + " grid[i, j] = 0 # We replace walls (#) with 0\n", + "\n", + " _fig, ax = plt.subplots()\n", + " ax.imshow(grid, cmap=\"gray\", alpha=0.7)\n", + "\n", + " # Plot state indices\n", + " for (\n", + " s,\n", + " (i, j),\n", + " ) in state_to_pos.items():\n", + " cell = maze_str[i][j]\n", + "\n", + " if cell == \"S\":\n", + " label = f\"S\\n{s}\"\n", + " color = \"green\"\n", + " elif cell == \"G\":\n", + " label = f\"G\\n{s}\"\n", + " color = \"blue\"\n", + " elif cell == \"X\":\n", + " label = f\"X\\n{s}\"\n", + " color = \"red\"\n", + " else:\n", + " label = str(s)\n", + " color = \"black\"\n", + "\n", + " ax.text(\n", + " j,\n", + " i,\n", + " label, # Attention : matplotlib, text(x, y, ...) expects (column, row)\n", + " ha=\"center\",\n", + " va=\"center\",\n", + " fontsize=10,\n", + " fontweight=\"bold\",\n", + " color=color,\n", + " )\n", + "\n", + " ax.set_xticks([]) # remove numeric axes, we don't need.\n", + " ax.set_yticks([])\n", + " ax.set_title(\"Maze with state indices\")\n", + "\n", + " plt.show()\n", + "\n", + "\n", + "plot_maze_with_states()\n" + ] + }, + { + "cell_type": "markdown", + "id": "db078d86", + "metadata": {}, + "source": [ + "### 2.4 Actions and deterministic movement" + ] + }, + { + "cell_type": "markdown", + "id": "96e7f1f2-9d73-410b-853d-e39f40dfb5da", + "metadata": {}, + "source": [ + "We first define integer codes for each action. \n", + "\n", + "**Exercise 3.** How many possible actions can the agent take in the maze?" + ] + }, + { + "cell_type": "markdown", + "id": "22259ab4-527e-4d7c-bb30-98fb240da6d5", + "metadata": {}, + "source": [ + "We have four possible actions in the maze. \n", + "\n", + "In this following cell, each action is mapped to an integer (0,1,2,3). This makes it easy to store and use actions inside arrays and matrices\n", + "\n", + "Here we use Unicode arrow character:\n", + "\n", + "- \"\\u2191\" : ↑ (up arrow)\n", + "\n", + "- \"\\u2192\" : → (right arrow)\n", + "\n", + "- \"\\u2193\" : ↓ (down arrow)\n", + "\n", + "- \"\\u2190\" : ← (left arrow)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "f7f0b8e4-1f48-4d03-9e5f-a47e59c3e827", + "metadata": {}, + "outputs": [], + "source": [ + "A_UP, A_RIGHT, A_DOWN, A_LEFT = 0, 1, 2, 3\n", + "ACTIONS = [A_UP, A_RIGHT, A_DOWN, A_LEFT]\n", + "action_names = {A_UP: \"\\u2191\", A_RIGHT: \"\\u2192\", A_DOWN: \"\\u2193\", A_LEFT: \"\\u2190\"}" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "3773781c-a0cd-48db-967b-d4b432d17046", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "↑\n" + ] + } + ], + "source": [ + "print(action_names[0])" + ] + }, + { + "cell_type": "markdown", + "id": "4b957f5a-ee39-4437-abc1-4809105ad83c", + "metadata": {}, + "source": [ + "**Exercise 4.** Now we define a **deterministic movement function** `move_deterministic(i, j, a)`. \n", + "\n", + "This function simulates the robot trying to move from (i, j) in direction a.\n", + "\n", + "But if the movement hits a wall or boundary, the agent stays in place.\n", + "\n", + "**Complete the `# !!TO DO HERE !!` part in the program below.**" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "4b06da5e-bc63-48e5-a336-37bce952443d", + "metadata": {}, + "outputs": [], + "source": [ + "def move_deterministic(i: int, j: int, a: int) -> tuple[int, int]:\n", + " \"\"\"Deterministic movement on the grid. If the movement hits a wall or boundary, the agent stays in place.\n", + "\n", + " Args:\n", + " i (int): current row index\n", + " j (int): current column index\n", + " a (int): action to take (A_UP, A_DOWN, A_LEFT, A_RIGHT)\n", + "\n", + " Returns:\n", + " (tuple[int, int]): new (row, column) position after taking action a\n", + "\n", + " \"\"\"\n", + " candidate_i, candidate_j = (\n", + " i,\n", + " j,\n", + " ) # It means “Unless the action succeeds, the robot stays in place.”\n", + "\n", + " # Now each action changes the coordinates of the robot:\n", + " if a == A_UP:\n", + " candidate_i, candidate_j = (\n", + " i - 1,\n", + " j,\n", + " ) # if the action is UP, then row becomes row -1\n", + " elif a == A_DOWN:\n", + " candidate_i, candidate_j = (\n", + " i + 1,\n", + " j,\n", + " ) # if the action is DOWN, then row becomes row +1\n", + " elif a == A_LEFT:\n", + " candidate_i, candidate_j = (\n", + " i,\n", + " j - 1,\n", + " ) # if the action is LEFT, then column becomes column -1\n", + " elif a == A_RIGHT:\n", + " candidate_i, candidate_j = (\n", + " i,\n", + " j + 1,\n", + " ) # if the action is RIGHT, then column becomes column +1\n", + "\n", + " # Check boundaries\n", + " if not (0 <= candidate_i < n_rows and 0 <= candidate_j < n_cols):\n", + " # If the robot tries to move outside the maze\n", + " # It will not move and it stays at (i, j).\n", + " return i, j\n", + "\n", + " # Check wall\n", + " if maze_str[candidate_i][candidate_j] == \"#\":\n", + " # If the next cell is a wall, the robot stays in place.\n", + " return i, j\n", + "\n", + " return candidate_i, candidate_j # Otherwise, return the new position\n" + ] + }, + { + "cell_type": "markdown", + "id": "c9e620e6", + "metadata": {}, + "source": [ + "### 2.5 Transition probabilities and reward function" + ] + }, + { + "cell_type": "markdown", + "id": "80bd2bca-7717-4b5f-bffa-76fe86a51d35", + "metadata": {}, + "source": [ + "Recall that we set the discount factor $\\gamma \\in(0,1)$, that is, the future rewards are multiplied by $\\gamma$, so immediate rewards matter a little bit more than future ones. \n", + "\n", + "\n", + "Moreover, we consider a probability error $p_{\\text{error}}$, which means, with probability $p_{\\text{error}}$, the robot **does not** execute the intended action but one of the 3 other directions (chosen uniformly). With probability $1-p_{\\text{error}}$, the robot executes the action that we asked." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "610253e7-f3f7-4a30-be3e-2ec5a1e2ed04", + "metadata": {}, + "outputs": [], + "source": [ + "gamma = 0.95\n", + "p_error = 0.1 # probability of the error to a random other direction\n" + ] + }, + { + "cell_type": "markdown", + "id": "0d1ceff8-86e0-4c45-83d3-af9fae974608", + "metadata": {}, + "source": [ + "Now we initialize the state–transition probability : the probability of reaching next state $s'$ after taking action $a$ in state $s$. \n", + "$$\n", + " p(s' \\mid s, a)\n", + " = \\mathbb{P} \\big[S_t=s'\\,|\\, S_{t-1}=s, \\,A_{t-1}=a\\big]\n", + "$$\n", + "\n", + "We store these transition probabilities in the 3D array `P` (`P[a][s, s_next]`), which has shape `(n_actions, n_states, n_states)`:\n", + "\n", + "`P[a, s, s_next] = P(S_{t+1} = s_next | S_t = s, A_t = a)`.\n", + "\n", + "We also initialize the reward vector `R`, which has length `n_states`, where `R[s]` is the reward received when the agent is in state `s`.\n", + "\n", + "In this maze game, we assume that the reward depends only on the current state, which is natural: in navigation tasks, being in a particular location is what matters, not the direction you used to reach it." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "7a51f242-fe4e-4e74-8a1f-a8df32b194b8", + "metadata": {}, + "outputs": [], + "source": [ + "# Initialize transition matrices and reward vector\n", + "P = np.zeros((len(ACTIONS), n_states, n_states))\n", + "R = np.zeros(n_states)" + ] + }, + { + "cell_type": "markdown", + "id": "c08f4af5-a2a7-4baa-b5da-c7ce636d8a4a", + "metadata": {}, + "source": [ + "Now we assign the reward to each state. \n", + "\n", + "For each state index s:\n", + "\n", + "1. If s is a goal, then the reward = +1.0\n", + "2. If s is a trap, then the reward = −1.0\n", + "3. Otherwise for the normal cell, the reward = −0.01 every time you leave this cell.\n", + "\n", + "Recall that rewards are received at the moment the agent executes an action. Here when the agent moves out of the cell, we set reward −0.01. " + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "49d54d1f-dc29-45b6-ad31-ad0e848f920d", + "metadata": {}, + "outputs": [], + "source": [ + "# Set rewards for each state\n", + "step_penalty = -0.01\n", + "goal_reward = 1.0\n", + "trap_reward = -1.0" + ] + }, + { + "cell_type": "markdown", + "id": "dd571ec8-c36a-4e20-bec6-9e6458dc622b", + "metadata": {}, + "source": [ + "**Exercise 5.** Why do we set the step penalty to -0.01 in this MDP?" + ] + }, + { + "cell_type": "markdown", + "id": "00c51189-3ff0-4a5e-ad52-92747b971e16", + "metadata": {}, + "source": [ + "**Solution 5** We assign a small negative reward for every step, which encourages the agent to reach the goal quickly.\n" + ] + }, + { + "cell_type": "markdown", + "id": "07bfb065-b1af-4df1-885e-780fe250f2fb", + "metadata": {}, + "source": [ + "**Exercise 6.** We now define the reward vector. Recall that we have already initialized\n", + "`R = np.zeros(n_states)`.\n", + "If a state belongs to `goal_states`, we assign the `goal_reward`.\n", + "If it belongs to `trap_states`, we assign the `trap_reward`.\n", + "Otherwise, we assign the `step_penalty`. \n", + "\n", + "**Complete the `# TO DO` part in the program below.** " + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "c70885b4-a301-42f2-ab70-2901d941cde7", + "metadata": {}, + "outputs": [], + "source": [ + "for s in range(n_states):\n", + " if s in goal_states:\n", + " R[s] = goal_reward\n", + " elif s in trap_states:\n", + " R[s] = trap_reward\n", + " else:\n", + " R[s] = step_penalty" + ] + }, + { + "cell_type": "markdown", + "id": "b90fb80c-9452-48a2-889f-286703c2ae93", + "metadata": {}, + "source": [ + "Now we define terminal states and a helper function. Here terminal_states is a set containing all absorbing states, which means, reaching them ends the episode conceptually. \n", + "\n", + "Moreover, `is_terminal(s)` is a small helper to check if a state is terminal." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "eca4c571-39c7-468b-af86-0bab9489415e", + "metadata": {}, + "outputs": [], + "source": [ + "terminal_states = set(goal_states + trap_states)\n", + "\n", + "\n", + "def is_terminal(s: int) -> bool:\n", + " \"\"\"Check if a state is terminal.\"\"\"\n", + " return s in terminal_states\n" + ] + }, + { + "cell_type": "markdown", + "id": "3a9a1d54-8339-402b-84e9-105961ed78d7", + "metadata": {}, + "source": [ + "Now we need to fill the transition matrices `P[a][s, s_next]`. \n" + ] + }, + { + "cell_type": "markdown", + "id": "d9cfd15c-12cc-48bb-bd88-07f3ae3db31c", + "metadata": {}, + "source": [ + "**Exercise 7.** **Complete the `# TO DO` part in the program below** to fill the transition matrices `P[a][s, s_next]`. (There are only 2 # TO DO here)" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "2d03276b-e206-4d1f-9024-f6948ca61523", + "metadata": {}, + "outputs": [], + "source": [ + "for s in range(n_states): # We loop over all states s.\n", + " i, j = state_to_pos[\n", + " s\n", + " ] # We recover the states to their coordinates (i, j) in the maze.\n", + "\n", + " # First, in a goal or trap state,\n", + " # No matter which action you “choose”, you stay in the same state with probability 1.\n", + " # This makes the terminal states as the absorbing states.\n", + " if is_terminal(s):\n", + " # Terminal states: stay forever\n", + " for a in ACTIONS:\n", + " P[a, s, s] = goal_reward\n", + " continue\n", + "\n", + " # If the state is non-terminal, we define the stochastic movement.\n", + " # For a given state s and intended action a,\n", + " # With probability 1 - p_error, the robot will move in direction a;\n", + " # With probability p_error, the robot will move in one of the other 3 directions, each with probability p_error / 3.\n", + " for a in ACTIONS:\n", + " # main action (intended action)\n", + " main_i, main_j = move_deterministic(i, j, a)\n", + " s_main = pos_to_state[\n", + " (main_i, main_j)\n", + " ] # s_main is the state index of that next cell.\n", + " P[a, s, s_main] += (\n", + " 1 - p_error\n", + " ) # We add probability 1 - p_error to P[a, s, s_main].\n", + "\n", + " # error actions\n", + " other_actions = [\n", + " a2 for a2 in ACTIONS if a2 != a\n", + " ] # other_actions = the 3 actions different from a.\n", + " for a2 in other_actions: # for each of the error action,\n", + " error_i, error_j = move_deterministic(i, j, a2)\n", + " s_error = pos_to_state[(error_i, error_j)] # get its state index s_error\n", + " P[a, s, s_error] += p_error / len(\n", + " other_actions,\n", + " ) # add p_error / 3 to P[a, s, s_error]\n", + "# So for each (s,a), probabilities over all s_next sum to 1.\n" + ] + }, + { + "cell_type": "markdown", + "id": "7841b264-af00-4322-b728-adcffac0ef89", + "metadata": {}, + "source": [ + "Now we check if the transition matrices `P[a][s, s_next]` are computed correctly.\n", + "For each action `a`, we sum the transition probabilities over all possible next states `s_next` and verify that these sums are equal to 1.\n", + "\n", + "This is because the matrix `P[a, s, s_next]` stores the transition probability\n", + "\n", + "$\\mathbb{P} \\big[S_t=s_{\\text{next}}\\,|\\, S_{t-1}=s, \\,A_{t-1}=a\\big]$. \n", + "\n", + "Therefore, for each action $a$, and for each state $s$, the sum over $s_{\\text{next}}$ of $\\mathbb{P} \\big[S_t=s_{\\text{next}}\\,|\\, S_{t-1}=s, \\,A_{t-1}=a\\big]$ should be 1. " + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "341fe630-8f87-4773-84ad-92d3516e53e2", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Action ↑: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]\n", + "Action →: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]\n", + "Action ↓: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]\n", + "Action ←: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]\n" + ] + } + ], + "source": [ + "for a in ACTIONS:\n", + " # For each action a:\n", + " # P[a] is a matrix of shape (n_states, n_states).\n", + " # P[a].sum(axis=1) sums over next states s_next, giving for each state s:\n", + " # We print these row sums.\n", + " # If everything is correct, they should be very close to 1.\n", + "\n", + " probs = P[a].sum(axis=1)\n", + " print(f\"Action {action_names[a]}:\", probs)\n" + ] + }, + { + "cell_type": "markdown", + "id": "46d23991", + "metadata": {}, + "source": [ + "## 3. Policy evaluation\n", + "\n", + "### 3.1 Bellman expectation equation" + ] + }, + { + "cell_type": "markdown", + "id": "305b047c-e83b-4f42-b64e-e2050d5deeff", + "metadata": {}, + "source": [ + "Recall that the value function under a policy $\\pi$ is defined as:\n", + "$$\n", + "V^{\\pi}(s)=\\mathbb{E}\\Big[\\:G_t \\:\\Big|\\: S_t=s\\:\\Big]\n", + "$$\n", + "where the return $G_t$ is\n", + "$$\n", + "G_t=R_t +\\gamma R_{t+1}+\\gamma^2 R_{t+2}+... . \n", + "$$\n", + "This means *The value of a state is the expected discounted sum of all future rewards\n", + "when following policy $\\pi$.*\n", + "\n", + "We know that $G_t=R_t+\\gamma G_{t+1}$, and plugging this equation into the definition of $V^{\\pi}(s)$, we get \n", + "$$\n", + "V^{\\pi}(s)=\\mathbb{E}\\Big[\\:R_t \\:\\Big|\\: S_t=s\\:\\Big]+\\gamma\\mathbb{E}\\Big[\\:G_{t+1} \\:\\Big|\\: S_t=s\\:\\Big]. \n", + "$$\n", + "This step shows simply ``The total future reward = immediate reward + discounted reward from next state.''" + ] + }, + { + "cell_type": "markdown", + "id": "88ea8d56-3b62-4690-9ff7-469e43726fbc", + "metadata": {}, + "source": [ + "For the expected immediate reward part $\\mathbb{E}[R_t| S_t=s]$, as we are in a maze problem, the reward depends only on the current state, not the time step, i.e., $\\mathbb{E}[R_t| S_t=s]=R(s)$. Hence we get \n", + "$$\n", + "V^{\\pi}(s)=R(s)+\\gamma\\mathbb{E}\\Big[\\:G_{t+1} \\:\\Big|\\: S_t=s\\:\\Big]. \n", + "$$\n", + "\n", + "Moreover, in this maze problem, we consider a deterministic policy $A_t=\\pi(s)$ (the action depends only on the state). Therefore, \n", + "$$\n", + "V^{\\pi}(s)=\\mathbb{E}\\Big[\\:R_t \\:\\Big|\\: S_t=s\\:\\Big]+\\gamma\\mathbb{E}\\Big[\\:G_{t+1} \\:\\Big|\\: S_t=s, A_t=\\pi(s)\\:\\Big]. \n", + "$$\n", + "\n", + "Now **given the state $S_t=s$ and $A_t=a$**, the next state is random (because of the error probability) and we know the transition probability \n", + "$$\n", + "\\mathbb{P}\\big(\\:S_{t+1}=s' \\:|\\:S_t=s, \\, A_t=a\\big)=P\\big(s'\\:\\big|\\:s, a\\big). \n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c25e255d-8f58-4eaf-9485-cee6ab3bea6c", + "metadata": {}, + "source": [ + "Therefore,\n", + "$$\n", + "\\mathbb{E}\\big[\\,G_{t+1}\\,|\\,S_t=s,A_t=a\\,\\big] =\\sum_{s'}\\mathbb{E}\\big[\\,G_{t+1}\\,|\\,S_{t+1}=s'\\,\\big]\\times \\mathbb{P}\\big[S_{t+1}=s'\\,\\big|\\,S_t=s, A_t=a\\, \\big]\n", + "$$\n", + "$$\n", + "\\hspace{-1.2cm}=\\sum_{s'}V^{\\pi}(s')P\\big(s'\\:\\big|\\:s, a\\big),\n", + "$$\n", + "where here we use the Markov property. (**Question: Can you show the detailed computations here?**)" + ] + }, + { + "cell_type": "markdown", + "id": "9a2b6cff-e848-44a2-b504-973067b367b3", + "metadata": {}, + "source": [ + "In conclusion, we have (the Bellman expectation equation)\n", + "$$\n", + "V^{\\pi}(s)=R(s)+\\gamma \\sum_{s'}P\\big(\\,s'\\,\\big|\\,s, \\pi(s)\\,\\big)V^{\\pi}(s').\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "15049fdb-f3af-4f78-b556-817284260ed0", + "metadata": {}, + "source": [ + "### 3.2 Define a function which computes the value function $V^{\\pi}(s)$ for a given deterministic policy. \n", + "\n", + "\n", + "**Exercise $8^*$.** Now we define `policy_evaluation(...)`, which computes the value function $V^{\\pi}(s)$ for a given deterministic policy. \n", + "\n", + "The input of this function `policy_evaluation(...)` are:\n", + "1. policy: array of size `n_states`, each entry is an action 0,1,2,3, which correspond to UP, RIGHT, DOWN, LEFT.\n", + "2. `P`: the transition probabilities `P[a, s, s']`.\n", + "3. `R`: the reward vector `R[s]`.\n", + "4. gamma: the discount factor $\\gamma\\in(0,1)$.\n", + "5. theta: convergence threshold.\n", + "6. max_iter: which is used to avoid infinite loops.\n", + "\n", + "How can we apply the Bellman expectation equation\n", + "$$\n", + "V^{\\pi}(s)=R(s)+\\gamma \\sum_{s'}P\\big(\\,s'\\,\\big|\\,s, \\pi(s)\\,\\big)V^{\\pi}(s').\n", + "$$\n", + "here ?\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "id": "20ef113f-0872-46e1-95ab-3cf5016f5a14", + "metadata": {}, + "source": [ + "We start with an initial guess of $V^{\\pi}$(e.g., all values = 0) and repeatedly apply the Bellman equation to update each state:\n", + "$$\n", + "V_{k+1}^\\pi(s) \\leftarrow R(s)+\\gamma \\sum_{s'}P\\big(\\,s'\\,\\big|\\,s, \\pi(s)\\,\\big)V^{\\pi}_k(s').\n", + "$$\n", + "until values converge.\n", + "\n", + "**Complete the `# TO DO HERE` part in the program below** " + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "3a05f8bc-2b8f-4a4c-9931-6d28c3b0db35", + "metadata": {}, + "outputs": [], + "source": [ + "def policy_evaluation( # noqa: PLR0913\n", + " policy: np.ndarray,\n", + " P: np.ndarray,\n", + " R: np.ndarray,\n", + " gamma: float,\n", + " theta: float = 1e-6,\n", + " max_iter: int = 10_000,\n", + ") -> np.ndarray:\n", + " \"\"\"Evaluate a deterministic policy for the given MDP.\n", + "\n", + " Args:\n", + " policy: array of shape (n_states,), with values in {0,1,2,3}\n", + " P: array of shape (n_actions, n_states, n_states)\n", + " R: array of shape (n_states,)\n", + " gamma: discount factor\n", + " theta: convergence threshold\n", + " max_iter: maximum number of iterations\n", + "\n", + " \"\"\"\n", + " n_states = len(R) # get the number of states\n", + " V = np.zeros(n_states) # initialize the value function\n", + "\n", + " for _it in range(max_iter): # Main iterative loop\n", + " V_new = np.zeros_like(\n", + " V,\n", + " ) # Create a new value vector and we will compute an updated value for each state.\n", + "\n", + " # Now we update each state using the Bellman expectation equation\n", + " for s in range(n_states):\n", + " a = policy[s] # Extract the action chosen by the policy in state\n", + " V_new[s] = R[s] + gamma * np.sum(P[a, s, :] * V)\n", + "\n", + " delta = np.max(\n", + " np.abs(V_new - V),\n", + " ) # This measures how much the value function changed in this iteration:\n", + " # If delta is small, the values start to converge; otherwise, we need to keep iterating.\n", + " V = V_new # Update V, i.e. Set the new values for the next iteration.\n", + "\n", + " if delta < theta: # Check convergence: When changes are tiny, we stop.\n", + " break\n", + "\n", + " return V # Return the final value function, this is our estimate for V^{pi}(s), s in the state set.\n" + ] + }, + { + "cell_type": "markdown", + "id": "09ef3439", + "metadata": {}, + "source": [ + "### 3.3 Evaluating a random policy" + ] + }, + { + "cell_type": "markdown", + "id": "eecbca15-f89f-47bf-a13d-7d7c051699b8", + "metadata": {}, + "source": [ + "Now we use the policy evaluation function `policy_evaluation` to evaluate a random policy. \n", + "\n", + "We first generate a `random_policy`, which is an array like [2, 0, 1, 3, 0, 2, ...] and has the size `n_states`. (Recall that the policy is a mapping from states to actions)." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "b4a44e38", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[0 3 2 1 1 3 0 2 0 0 2 3 2 3 2 3 2 0 3 1 2 1]\n" + ] + } + ], + "source": [ + "# Random policy: for each state, pick a random action\n", + "random_policy = rng.integers(low=0, high=len(ACTIONS), size=n_states)\n", + "\n", + "print(random_policy)" + ] + }, + { + "cell_type": "markdown", + "id": "3fe07992-ce82-4124-aebc-a6384d417f64", + "metadata": {}, + "source": [ + "Now we call the function `policy_evaluation(...)` to compute $V^{\\pi_{\\text{random}}}(s)$." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "c5f559b2-452a-477c-a1fa-258b40805670", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Value function under random policy:\n", + "[ -0.2 -0.2 -0.201 -0.204 -0.205 -0.202 -0.214 -0.429 -0.212\n", + " -0.207 -0.276 -0.459 -0.352 -0.366 -5.827 -4.605 20. -0.366\n", + " -0.999 -20. -6.4 -3.163]\n" + ] + } + ], + "source": [ + "V_random = policy_evaluation(random_policy, P, R, gamma)\n", + "print(\"Value function under random policy:\")\n", + "print(V_random)" + ] + }, + { + "cell_type": "markdown", + "id": "f46c70ba-2932-49af-b568-b5477260bc94", + "metadata": {}, + "source": [ + "Here in this value vector of the policy, \n", + "- If it is a negative values, then the agent tends to move around aimlessly, fall in traps, or take too long.\n", + "- It it is a higher values, then the agent is closer to the goal or more likely to reach it" + ] + }, + { + "cell_type": "markdown", + "id": "1efcb076-467c-42d8-94e8-87453f688bbd", + "metadata": {}, + "source": [ + "Now we define a function `plot_values`, which displays the value function $V(s)$ and displays it on the maze grid. It helps students visually understand:\n", + "- which states are good (high value, near the goal),\n", + "- which states are bad (low value, near traps),\n", + "- how a policy affects the long-term expected reward." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "4c428327", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdcAAAGbCAYAAACWHtrWAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAARjhJREFUeJzt3Qd4FGX+B/DvptcNoSQhEHqHUAUUUWnSBEQ9FRtgvb/lUBEBGyJFBBThlAPvTkVQBPQUEAREBFHpJfROIAFSSCC9kez8n/dddskmm7rvbrLL9+MzLjsz+868O5v5zdtmdJqmaSAiIiJl3NQlRURERAKDKxERkWIMrkRERIoxuBIRESnG4EpERKQYgysREZFiDK5ERESKMbgSEREpxuBKRESkGIOrg507dw46nQ6LFi2qku2vX78eHTt2hI+Pj9yPlJQUVEdi3yZPnlzVu1GtiN+M+F7Eb8hViWMu8lhYo0aNMHr06CrbJ6LKYHAtxbBhw+Dn54f09PQS13nsscfg5eWF5ORkVHdiHx966CH4+vpi/vz5WLJkCfz9/atsf37++WcGUCJySR5VvQPVmQicP/30E3788UeMHDmy2PKsrCysWrUKAwcORK1atVDd7d69W14oTJ06Ff369avq3ZHBVQR5awE2OzsbHh78eRJw4sQJuLmxHEDOhb/YMkqugYGBWLp0qdXlIrBmZmbKIOwMEhMT5WuNGjVQ3Ylq6+ocXMXzLsQFANmft7c3PD09q3o3iCqEwbUUovr0/vvvx6ZNm8yBqTARdEXwFUH4ypUrGDduHCIjIxEQEAC9Xo9BgwbhwIEDZW6nV69ecipKtDOJ9qbCDAYD5s6di7Zt28oAFBoair///e+4evVqmdsYNWqU/HfXrl1lu5apHaukNq2i+7Vlyxb5uRUrVmD69OmoX7++3Ie+ffvi9OnTxT6/c+dODB48GMHBwbL6uX379pg3b545b6LUKog0TVNpba779++X36n4bsV3LLa7Y8cOq+2Sf/31F8aOHYs6derIbd933324fPmyxbqpqak4fvy4fC2L+I6GDBmCDRs24JZbbpG/jc8++0wu+/LLL9GnTx+EhITIQNCmTRssWLCgxDT+/PNPdOvWTX53TZo0weLFi4ute+TIEZmm2I74nqdNmyaPvTX/+te/5O9BbDs8PBwvvvhisbZ0cRzbtWuHgwcP4q677pLNHc2aNcP3338vl//+++/o3r273F7Lli3x66+/lvmdmH4Py5cvx5tvvomwsDD5XYu/h9jY2GLrf/fdd+jSpYvcRu3atfH444/j4sWLZW7H2u9T5O/VV1+Vy0S+xXckapeSkpKQkZEh9+Pll18ultaFCxfg7u6OGTNmlLldIpuIR85RyX755RfxSD7tk08+sZifnJyseXp6aiNHjpTvd+/erTVt2lSbOHGi9tlnn2lTpkzR6tWrpwUFBWkXL140fy46Olqm9+WXX5rn3XXXXXIqatSoUVrDhg0t5j3zzDOah4eH9uyzz2oLFy7UJkyYoPn7+2tdu3bV8vLySs3Hc889J7ct9m3JkiXatm3b5DKxDbGtooru1+bNm+XnO3XqpHXp0kX7+OOPtcmTJ2t+fn5at27dim3Py8tLpv3uu+9qCxYs0MaMGaP169dPLhfbvvvuu2V6Yl9Mk4mYLz5ncvjwYZnPunXralOnTtU++OADrXHjxpq3t7e2Y8cO83riezXtY58+feRxe+211zR3d3ftoYcesthH07qFj0VJRD6aNWumBQcHy2MsvnvxfQjiux89erT8PsT2+vfvL9P99NNPi6XRsmVLLTQ0VHvzzTfl8s6dO2s6nU7mzyQuLk6rU6eO3Jb4fmfPnq01b95ca9++vUxX/IZMxHck5onvVWz7pZdeknkt+nsQxzE8PFyLiIjQXn/9dblumzZt5LrLli3TwsLC5Lbmzp1r/t2mpaWV+p2Yfg+RkZFy3+bMmSO/Gx8fH61FixZaVlZWse9a7Jf4nsR6vr6+WqNGjbSrV68Wy0/R763w7zM9PV1r166d3HfxdyB+W+I3IdLev3+/XOexxx6T33N+fr5FWrNmzZLf9/nz58s85kS2YHAtg/jjFCf02267zWK+OLmKk8CGDRvk+5ycHK2goMBiHXESFCd/EcxUBNc//vhDfvabb76xWG/9+vVW5xdlOsGJC4HCKhpcW7dureXm5prnz5s3T84/dOiQ+TsTgU+kW/jEKRgMBvO/X3zxxWIn0pKC6/Dhw2WwPnPmjHnepUuXtMDAQO3OO+8slkcRbApv69VXX5Un45SUlEoHV7Gu+K6LKhxETAYMGKA1adLEahpbt241z0tMTJS/EXEBYPLKK6/I9Xbu3Gmxngh4hYOrmCe+ExHMC//2RNAW633xxRfmeeI4inlLly41zzt+/Lic5+bmZnGBIn7T5fleTL8HEYwLB+IVK1bI+eJ3IYggHxISIgNidna2eb01a9bI9SZNmlSh4CrWF+v88MMPxfbJdMxNeVi3bp3FcnERYO1vjUg1VguXQVQhjRgxAtu3b7cYAiGqhEWVrKiaFETVlKnTRUFBgeyZK6ouRRXbvn37lOyLqFYLCgrC3XffLau/TJOoahPb2rx5MxzhySeflD2kTe644w75evbsWXP1bXR0NF555ZVi7btFh1mUh/g+f/nlFwwfPlxWo5rUrVsXjz76qKxmTUtLs/jMc889Z7EtsY8infPnz5vniapGEcfLO8yjcePGGDBgQLH5oprTRFQxi2Miql7F91G0yllUGZu+L0FUW4vfiOm7M3X0uvXWW2XVceH1irbti6rbvLw8+T0X7vDz7LPPyqrztWvXWqwvfiPit2witiuOT+vWrWWVsInp34X3qTSiOlY0j5j87W9/k8dG5EPYs2ePbFZ54YUXZFW4yT333INWrVoV28+y/O9//0OHDh1kVX9RpmMuOuyJKvJvvvnGvOzw4cOyWlxURxPZG4NrOZhOaqaOTaLd5o8//pAnKhF8BdEe9vHHH6N58+Yy0Io2JXFCFH/M5WnTK49Tp07JtETbnki78CTamay1C9tDgwYNLN6LNlXB1O575swZ+Sra+FQQbaWiZ7YIBkWJwCC++6JtfGXtY2WI4GqNaN8VJ3PRzieClTgeog1SKHrsi+6Xad8K75e4ABC/o6KK5t90oVB0vrjwERchhS8kBNEuWfTiRlysRUREFJtXke+q6L6KbYj2XNPFaEn7KYjgWnQ/yyJ+X2X9tsTFhvi7XblypfztCCLQiuD+4IMPVmh7RJVRfbtjViOiZChOAt9++608aYpXUeIpXJJ4//338c477+Cpp56SQ11q1qwp/8BFqaKkjiiFT0bGmlBLoqRVmEhHBNbCV+OFiZN6ZZRUmhTbN108FGZtnmAtD1XFHvtYuIRa+EQvai/E72POnDkyUIngJkpt4mKr6LGvyu+upG07w/GsDFGinj17tgywjzzyiLw4Fh3KTBcPRPbE4FpOIpCK4ClKouKPVFyti163JqLXZe/evfH5558X69UoSrGlESUXa1VwRa/omzZtKqsCb7/9dqsn+soS27d2pyax/cLVsOUl9tNUDVfaeNryVhGLiwbRu1WMdyxK9PYVFzFFS1+OIsZB5+bmYvXq1RalUluq6Bs2bChrKYoqmn+xnml+4eMkqopFtbyjxjIX3VcRlEXvcdE7vOh+ih7QhYl5puUV+X2J31ZZROm2U6dO8mJUlNpjYmLwySefVGhbRJXFauFyMpVSJ02ahKioqGLtX+Lqv+iVvmgjLc9QA3GyEEGi8FARMYRHVDcWJu6uJEqTomRcVH5+fqVvZSi2L4a0iJOyyZo1a6wOpyiPzp07yypUMWSo6D4V/o5Md4cqa7/Fd9u/f385rrhwu3dCQoK80OnZs6dsY6yoigzFKW3fiuZLpCeG51SWGL4kjseuXbvM88Rvo2iNhQieopT8z3/+02L74gJP7INo03QEMZSo8F3MxIVmXFycHDYliKFLosZl4cKF8kLEZN26dTh27FiF9/OBBx6Qfx/i5i5FFf0bfOKJJ2R7vfgtihu9mPaJyN5Yci0nESx69OghT/BC0eAqqpumTJkiO/uI9Q4dOiRPhuUp+YmqZFGlKDrLPP3007LtVJyIxNjFwh11RCcZMaZVjNETAV4EHDG4XpQcRCAXY0hFZ5KKeuaZZ+QJUdxpSgRwUdX59ddfm0ugFSVKkmKc59ChQ+V9jMV3Ijq4iEAmxm+KsaKm6nZhzJgxMu+mzmPWiHGeGzdulIFUdIwRN5gQ40zFyXrWrFmV2k9xchb7JgJhZe9dK46BCHAir+LYiLbv//znPzKYiABTGePHj5e3phTHQ4zVFBch//73v2UJT9ScFC7Rv/HGG3jvvffkumJ8qSgJinGvolbFUR13RBOIOC7iuxQXPCKQiTZX0bFKEL/RmTNnyuXiNyyqaMV64vcqxqmK8aoV8frrr8vfq2g7FX874nckxpmL2gPxdyM6O5mIDm/i+xTH+vnnn+fNKMhxlPc/dmHz58+X3fuLjuk0DcURwynEsB0xfu/222/Xtm/fXmw4i7WhOMLXX38th26IoRUdO3aUQwmsjXMV/v3vf8txpmI7YiiKGGc4fvx4OTSlMkNxhI8++kgOqRDDQsS+79mzp8ShON99953FZ0vK059//inHsop9FGNUxTCIwuOFxZCdf/zjH3JMpxh7WPjnWHQojrBv3z45xCUgIECOre3du7d5rG5ZeTTtu2lsamWG4txzzz1Wl61evVrmTYzvFOM2Z86cKYfBFB2TWlIa1oZiHTx4UM4TaYrjIsZxfv7558XSNA29adWqlRx3LcZ2Pv/888WGQIm02rZtW+58ie2IoVKlMX2n3377rfbGG2/I4TbiNynSszaOdPny5XL8sfiN1axZU45FvXDhgsU65RmKYxpnLsb0iu9G/M3Ur19frpOUlFRsu4MHD5ZpFv2tENmTTvzPgbGciFyEuEOT6Gcgak0qU2PiKGLIjqhJsnYXMSJ7YZsrEbksUTUvxtGKtlciR2KbKxG5HNFbWnQI/O9//yvbWUV7OJEjseRKRC5HPIhAlFZFkP3qq6/kQwWIHIltrkRERIqx5EpERFQVba7iFm6XLl2SN+euzI3XiYioaolKSnGzD/FAg8IPelApJyfH4mY0thDjxws/6MElg6sIrFV1ezkiIlJH3HlN3A7SHoG1ceN6iI+/oiQ90U4u2sydNcCWK7iaHiclDkplbjNHRERVS9ztTRSSCj8eUCVRYhWB9dz5FdDr/WxKKy0tC40aPiTTdOngaqoKFoGVwZWIyHnZu2lPH+ADfYCNDxYp40lizoDjXImISB0RGA02BkcXCK7sLUxERKQYS65ERKQOS64SgysREakj7kuk2XhvIhe4txGrhYmIiBRjyZWIiNQxaAqqhZ2/5MrgSkRE6rDNVWK1MBERkWIsuRIRkTosuUoMrkREpA6Dq8TgSkRE6mgKgqtIw8mxzZWIiEgxllyJiEgZnWaQk61pODsGVyIiUodtrhKrhYmIiBRjyZWIiBTfoUmzPQ0nx+BKRETqsFpYYrUwERGRYiy5EhGROiy5SgyuRESk+HmuBtvTcHKsFiYiIlKMJVciIlKH1cISgysREanDoTgSgysREanDkqvENlciIiLFWHIlIiJ1+Mg5icGViIiU0RkMcrI1DWfHamEiIiLFWHIlIiLFN5HQbE/DyTG4EhGROuwtLLFamIiISDGWXImISB2WXCUGVyIiUod3aJIYXImISB2WXCW2uRIRESnGkisRESmuFjbYnoaTY3AlIiJ1OM5VYrUwERGRYiy5EhGROuzQJDG4EhGROqJK18BqYVYLExGRU9u6dSuGDh2K8PBw6HQ6rFy50mL56NGj5fzC08CBA+26Tyy5EhGRU1cLZ2ZmokOHDnjqqadw//33W11HBNMvv/zS/N7b2xv2xOBKREROHVwHDRokp9KIYBoWFgZHYbUwERFVS2lpaRZTbm5updPasmULQkJC0LJlSzz//PNITk6GPTG4EhGR+nsLG2ycAERERCAoKMg8zZgxo1K7JKqEFy9ejE2bNmHmzJn4/fffZUm3oKAA9sJqYSIiUkczGCdb0wAQGxsLvV5vczvpiBEjzP+OjIxE+/bt0bRpU1ma7du3L+yBJVciIqqWJVe9Xm8xqeqE1KRJE9SuXRunT5+GvTC4EhHRTeXChQuyzbVu3bp22warhYmIyKl7C2dkZFiUQqOjoxEVFYWaNWvK6b333sMDDzwgewufOXMG48ePR7NmzTBgwADYC4MrERE59cPS9+zZg969e5vfjx07Vr6OGjUKCxYswMGDB/HVV18hJSVF3miif//+mDp1ql3HulYouK7vPQV+7vYdeEvk6obsmg5Xs6bbW3A1rnicXFWvXr2glXLLxA0bNsDRWHIlIiJ1+DxXicGViIiculq4OmJvYSIiIsVYciUiIoUU3ERCpOHkGFyJiEgdVgtLrBYmIiJSjCVXIiJShyVXicGViIic+g5N1RGDKxERqcOSq8Q2VyIiIsVYciUiInVYcpUYXImISB22uUqsFiYiIlKMJVciIlJHPJ1Gs7Fa19bPVwMMrkREpA7bXCVWCxMRESnGkisREanDkqvE4EpEROqIJ+IYbOzta/NTdaoeq4WJiIgUY8mViIjUYbWwxOBKRETqiBpdg63BFU7PYcE1uH0DRE4YBv+IWsiIScahmauQcijW6roht7dE05F3ILBpGLT8AlzZfw5HPl6LnMQ08zqhd7VGm38MhE+IHqnHL+HA9B+ReT4JjuRqeXK1/LhqnlwRj5MLYcnVcW2unnpfdJ0zEudW7MCGvtNw/rsd6DZnJDwCfKyu7xHgjTOL/8CmobPw2/APcS0zF53fH2Fe7t+gNjpNeQhH5v6MDf2mI2nPWXT98HHo3B3XhOxqeXK1/LhqnlwRjxO5Iof82sJ6tUHO5TTErNoDw7UC+ZqbnC7nW3Npw0Ek/nUCBdl5KMi5huhlfyG4bYT5j6PeoI5I3nsWiX+egCEvH6c+3wyv4ADU7NjQEdlxyTy5Wn5cNU+uiMfJtWgGTcnk7BwSXPXNwpB2Ms5innivbx5Wrs/X6twY6ecuQyswWE1PzM+ITpTzHcXV8uRq+bG2D66QJ1fE4+Sitz/UbJycnEOCq7ufF/LTcyzmXUvPgYefd5mf1beoi5Z/74ejH681z/Pw85KfL5qeu3/Z6anianlytfy4ap5cEY8TuSK7dGiqN6ADIt+4V/47Oz4FSbvOyHYViw0H+CAvJbPUdAKbhqLbvFE4PPsnmYZJflZesfYYzwBvFGTmwl5cLU+ulh9XzZMr4nFycezQZL/genHDATmZRAzrgsYjehS74oxe+lepfzi3fvoUjs3fgIvrb6QlpJ2OR1CLuub3oq0loHEI0s4kwF5cLU+ulh9XzZMr4nFycQyujqsWjt9yFD4hQfKPSOfhLl99agcifssRq+sHNAmRfzgnFm7EhTX7ii2/uC4KtW5pgpAeLeDm6Y7mT/VCXmqW7JLvKK6WJ1fLj6vmyRXxOJEr0mla2S3HaWlpCAoKwvLOr8HPvXLtFsEdGiJyvHEcW2ZsEg59sBpXD8XIZT6hQei1/GVseXgechJS0eGd+1H/nk6yJ2BhpuWC6EnY+qUB8o8y9cQlHJj2g+PHULpYnlwtP9U1T0N2TYerWdPtLZs+z+Nkf6bzeGpqKvR6vd3Svzr7Seh9vWxLKzsPwa9/abd9dangSkSuedJWEVyrI1c7Tg4LrjNHqwmuExY5dXDlqGoiIiLFeG9hIiJSRlSGajZ2SCpHhWq1x+BKRETqsLewxOBKRETqMLhKbHMlIiKntnXrVgwdOhTh4eHQ6XRYuXJlsWrmSZMmoW7duvD19UW/fv1w6tQpu+4TgysREakvuRpsnCogMzMTHTp0wPz5860unzVrFv75z39i4cKF2LlzJ/z9/TFgwADk5FjeJlMlVgsTEZE6Km68r1Xs84MGDZKT9aQ0zJ07F2+//Tbuvdd4283FixcjNDRUlnBHjLjxuEKVWHIlIqJqKS0tzWLKza34/aGjo6MRHx8vq4JNxHjc7t27Y/v27bAXBlciIlJGM6iZhIiICBkITdOMGTMqvD8isAqipFqYeG9aZg+sFiYiomrZWzg2NtbiDk3e3s5zh0CWXImIqFrS6/UWU2WCa1hYmHxNSLB8KpJ4b1pmDwyuRETk1L2FS9O4cWMZRDdt2mSeJ9pvRa/h2267DfbCamEiIlKmcJtpZVX08xkZGTh9+rRFJ6aoqCjUrFkTDRo0wCuvvIJp06ahefPmMti+8847ckzs8OHDYS8MrkRE5NT27NmD3r17m9+PHTtWvo4aNQqLFi3C+PHj5VjY5557DikpKejZsyfWr18PHx8fu+0TgysREamjKajWreA41169epV6s39x16YpU6bIyVEYXImISB1RpWtQkIaTY3AlIiJlxOPmNFsfOccb9xMREVFRLLkSEZE6rBaWGFyJiEgdUaOrKUjDybFamIiIqCpLrgM3T7K4z6OzW9PtLbiaIbumw5W44jFa7YJ54lU6mbBDkxGrhYmISB22uUq84CQiIlKMJVciInLqewtXRwyuRESkDquFJVYLExERKcaSKxERKcNqYSMGVyIiUkeMojEoSMPJMbgSEZEy4slvmmOfOFctsc2ViIhIMZZciYhIGba5GjG4EhGROhyKI7FamIiISDGWXImISBlWCxsxuBIRkTLsLWzEamEiIiLFWHIlIiJ1DDrjZGsaTo7BlYiIlGGbqxGrhYmIiBRjyZWIiJTRNJ2cbE3D2TG4EhGRMqwWNmJwJSIitUNxDLan4ewYXG0Q3L4BIicMg39ELWTEJOPQzFVIORRrdd2Q21ui6cg7ENg0DFp+Aa7sP4cjH69FTmKaeZ3Qu1qjzT8GwidEj9Tjl3Bg+o/IPJ/kwBy5HpXHKLBJCFq/Mhg1WoXDq4Y/1veZivyMHAfnCKhZKE+ZMck4OHMVrpaQp8IaDu+KDm8Ox+E5a3F22TY5L+S2FmjzjwHwCQmSZzTxuzs892ekn0mAI7nicaKbGzs0VZKn3hdd54zEuRU7sKHvNJz/bge6zRkJjwAfq+t7BHjjzOI/sGnoLPw2/ENcy8xF5/dHmJf7N6iNTlMewpG5P2NDv+lI2nMWXT98HDp3HqLqcowM+QbE/XoIUVP+h6rMk8hD9IodWN93GqK/24HupeTJxLt2IJo+3hNpp+It5qeejMP2fyzC+n7TsGHgDCT8dQLdZj0GR3LF43QzM7W5ajZOzo5n7koK69UGOZfTELNqDwzXCuRrbnK6nG/NpQ0HkfjXCRRk56Eg5xqil/2F4LYR5uBZb1BHJO89i8Q/T8CQl49Tn2+GV3AAanZs6OCcuQ7VxygzJgmxq/c6vFRXWF0recpJTpfzS9N+/DCc/GIz8tKyLOaL70NMJprBAN+6NRx6UeeKx+mmZtBBs3HiONebmL5ZGNJOxlnME+/1zcPK9flanRsj/dxlaAUGq+mJ+RnRiXJ+8t5oxXt/c1B9jJw1T3X7tIWHvzcu/ByFBkO7FFvuGxqEXkv/AQ8/b0AHnPzyd4fm2RWPExGDayW5+3khP92yHedaeo7xBFUGfYu6aPn3ftj7xrfmeR5+XvLzRdNz9y87PXLMMaouebpWgTx5BvqgzZiB2PGPRSWmmZ2QinV9p8m0I+7pjJyEVDiSKx6nmxnvLWzE4FpO9QZ0QOQb98p/Z8enIGnXGdlWVJhoI8pLySw1ncCmoeg2bxQOz/5JpmGSn5VXrI3JM8AbBZm5SvPhyux9jKoqTx2u5ymrhDx5Bvggt4Q8tRkzCDGr9yIzNrnMbRVk5eHc9zsx8Jc3sXXUv5B16SrswRWPE93Aca5GDK7ldHHDATmZRAzrgsYjehS7io5e+lepJ4NbP30Kx+ZvwMX1N9IS0k7HI6hFXfN70X4U0DgEaWw3qjbHqDrkqcGwLmhiJU9nSshTna5NZZWw6TMiENdoXU+25e+ZaKW0pwPcvD1ku6u9gqsrHieiotihqZLitxyVwxfEiUHn4S5ffWoHIn7LEavrBzQJkSeDEws34sKafcWWX1wXhVq3NEFIjxZw83RH86d6IS81Sw4zoOpxjAQ3Lw+4eRqvSd283OV7R4q7nqcG1/PUoIw8/fH0Qmx57BP8/vincko5dhGnv/4TB2esksvD746Ef/2agE4nS4uRY4egIPuaHJLjKK54nG5mtnZm0kydmipg8uTJ0Ol0FlOrVq1QlfiLq6RradnY/doSRI4fhnbjhiIzNgm7xy4xt4f5iE4iy1/GlofnyTaspo/1hFewH9q8OlhOJqbloodj1Lvfoe3Ye+SJJvXEJZk+O2lUn2MkSnN9V71unt9//ZvyddO9s5Edl+KwPO16bYns/Rs5bigyYpOws1CeROek3stfxuaH58m21NzkDIvPi57oYsynuHAT/OoGo/WL/eEdHCB73149egHbX/oS+Q5sjnDF43Qzq6o217Zt2+LXX381v/fwqNrwptO0srORlpaGoKAgpKamQq/Xw1Ws6fYWXM2QXdPhSlzxGLni5ZIrVoG52t+Svc/jpvRPDXscgZ5eNqWVfi0PzVd/Xe59FSXXlStXIioqCtWFK/5NEBGRC9xEIi0tzWLKzS25RuXUqVMIDw9HkyZN8NhjjyEmJgZVicGViIiUMRh0SiYhIiJCloZN04wZM6xus3v37li0aBHWr1+PBQsWIDo6GnfccQfS02/cIMXR2OZKRETVss01NjbWolrY29v62OdBgwaZ/92+fXsZbBs2bIgVK1bg6aefRlVgcCUiompJr9dXqn24Ro0aaNGiBU6fPo2qwmphIiJyqRv3Z2Rk4MyZM6hb98a9AxyNwZWIiJw6uI4bNw6///47zp07h23btuG+++6Du7s7HnnkEVQVVgsTEZFTu3DhggykycnJqFOnDnr27IkdO3bIf1cVBlciIlLGoOnkZGsaFbFs2TJUNwyuRESkTGVuX1iUrZ+vDtjmSkREpBhLrkREpAyf52rE4EpERMoYoKDNVTz70MmxWpiIiEgxllyJiEgZFTeB0Gz8fHXA4EpERMqIwGhgcGVwJSIidVhyNWKbKxERkWIsuRIRkTKG65MtbP18dcDgSkREyrBa2IjVwkRERIqx5EpENhuya3pV7wJVEwat4jfet5aGs2NwJSIiZVgtbMRqYSIiIsVYciUiIsXVwrA5DWfH4EpERMqwWtiI1cJERESKseRKRERqHzkHPnKOwZWIiJThw9KNGFyJiEgZMcbVYPM4V+cvubLNlYiISDGWXImISBlNQZurxjZXIiKiG9jmasRqYSIiIsVYciUiImXYocmIwZWIiJQR7aUa21xZLUxERKQaS65ERKQMb9xvxOBKRETKsM3ViNXCREREirHkSkREyrBDkxGDKxERKcM2VyMGVyIiUoYlVyO2uRIRESnGkqsNgts3QOSEYfCPqIWMmGQcmrkKKYdira4bcntLNB15BwKbhkHLL8CV/edw5OO1yElMk8sDm4Sg9SuDUaNVOLxq+GN9n6nIz8hxcI5cj8pjJDR7shcaDL8FnoG+yLp4Bcc+3YCknacdmCOgZqE8ZcYk4+DMVbhaQp4Kazi8Kzq8ORyH56zF2WXbzPN9QvRo9+pg1OnWTL6/euQCdoxZZNc8kOtitbARS66V5Kn3Rdc5I3FuxQ5s6DsN57/bgW5zRsIjwMfq+h4B3jiz+A9sGjoLvw3/ENcyc9H5/RHm5YZ8A+J+PYSoKf9zYC5cm+pjFHpXazR5rCd2j12CDX2m4uzSv3DLrMfkdhyZJ5GH6BU7sL7vNER/twPdS8mTiXftQDR9vCfSTsVbzHf38USPfz2N1FPx+GXoLKzv/z6OL9ho51zQzTAUx2DjVFHz589Ho0aN4OPjg+7du2PXrl2oSgyulRTWqw1yLqchZtUeGK4VyNfc5HQ535pLGw4i8a8TKMjOQ0HONUQv+wvBbSOgczcegsyYJMSu3ov0MwkOzonrUn2M/OrVROrRC+ZjdHFdFNw83OR8R6lrJU85yelyfmnajx+Gk19sRl5alsX8iCGdkZeahVNfbEFBVh60AgNSjl20cy6I1Fq+fDnGjh2Ld999F/v27UOHDh0wYMAAJCYmoqowuFaSvlkY0k7GWcwT7/XNw8r1+VqdGyP93GV5MiPnOEZxGw/Bu1Yg9C3qAm461B/SGdmJaQ69IKpMnur2aQsPf29c+DnKah5zElPRfe4oDNz4Fu786gWE9Ghhl32nm4OmaBLS0tIsptzcXFgzZ84cPPvss3jyySfRpk0bLFy4EH5+fvjiiy9QVRhcK8ndzwv56ZZtotfSc+Dh513mZ8XJueXf++Hox2vtuIek+hjlXsmQJds7vnoBg/98D23H3oOD7/8IQ14+HJknkYfy5skz0AdtxgzEwQ9WWV3upfdF3V5tcf7HXdgwcIYs3d7ywSPwr++40ji54MPSNdsmU2/hiIgIBAUFmacZM2YU215eXh727t2Lfv36mee5ubnJ99u3b0dVYYemcqo3oAMi37hX/js7PgVJu84Ua2sT7V55KZmlphPYNBTd5o3C4dk/yTTIeY5R82f6IOT2Ftj8t4+RdekqanVqhC4fPIIdL36JtFNxdstTh+t5yiohT54BPsgtIU9txgxCzOq9yIxNtro8PzsPVw7FIP73Y/K9eE09fgl1ujdH5oWdyvNDVBGxsbHQ6/Xm997exS8ik5KSUFBQgNDQUIv54v3x48dRVRhcy+nihgNyMokY1gWNR/QoVtqJXvpXqSftWz99Csfmb8DF9TfSIuc4RkEt6+LSpsOyl7CQvC9adhCq3a2p3YJr0Tw1GNYFTazk6UwJearTtamsEjZ9RgTiGq3roWbHhtgz8VuknYxH7a5N7LLvdHMSjSgGBWkIIrAWDq7OhNXClRS/5Sh8QoLkCVzn4S5ffWoHIn7LEavrBzQJkSftEws34sKafVbXcfPygJun8XrHzctdvqfqc4zEcJe6fdrBN6yGeZhPjTb1i7WB2lPc9Tw1uJ6nBmXk6Y+nF2LLY5/g98c/lZPorHT66z9xcIaxmjj25/0IahmO0J4tAZ1Ovor3iTtOOSxP5Fo0Ua2r2T6VV+3ateHu7o6EBMu+D+J9WFj5+lfYA8/elXQtLRu7X1uCyPHD0G7cUGTGJskhGqb2MJ/QIPRa/jK2PDwPOQmpaPpYT3gF+6HNq4PlZGJa7lu3Bvquet08v//6N+XrpntnIzsupQpy6PxUH6Mzi7fKKtke/3kWngG+spfu8QW/IGn3GYfmaddrS2Tv38hxQ5ERm4SdhfLkGxqE3stfxuaH5yE7IRW5yRkWnxftw2L8tOghLIhSuCjBtn15ELpMexiZF65g94Sl5tI5UXXn5eWFLl26YNOmTRg+fLicZzAY5PuXXnqpyvZLp2lamcN1RS8t0ZicmprqtEV0a9Z0ewuuZsiu6XAlrniMXLF/+DAX+925Inufx03pL+08Dn7uZXcaLE1WQS4e3fdhufdVDMUZNWoUPvvsM3Tr1g1z587FihUrZJtr0bZYR2HJlYiInPoOTQ8//DAuX76MSZMmIT4+Hh07dsT69eurLLAKDK5EROT0N+5/6aWXqrQauCh2aCIiIlKMJVciIlKGN+43YnAlIiJl+DxXI1YLExERKcaSKxERKcNqYSMGVyIiUobB1YjVwkRERIqx5EpERMqwQ5MRgysRESkjbqhrsLFat+yb8lZ/rBYmIiJSjCVXIiKqls9zdWYMrkREpExFn8dqja2frw4YXImISBmWXI3Y5kpERKQYS65ERKQMbyJhxOBKRETKiLioKUjD2bFamIiISDGWXImISHG1sM7mNJzdTR1cXaFHWlGru70FVzJs13S4mjebToGrOdzS9fI0/uidcCX5BZkO2Q6rhY1YLUxERKTYTV1yJSIitdhb2IjBlYiIlOFNJIxYLUxERKQYS65ERKSMeFycxkfOMbgSEZE64kHnBj4sncGViIjUYcnViG2uREREirHkSkREyrC3sBGDKxERKcNxrkasFiYiIlKMJVciIlKG9xY2YnAlIiJlWC1sxGphIiIixVhyJSIiZTjO1YjBlYiIlOFQHCNWCxMR0U2jUaNG0Ol0FtMHH3ygfDssuRIR0U3VoWnKlCl49tlnze8DAwOVb4PBlYiIquVQnLS0NIv53t7ecrKVCKZhYWGwJ1YLExGR8pKrwcZJiIiIQFBQkHmaMWOGkn0U1cC1atVCp06dMHv2bOTn50M1llyJiKhaio2NhV6vN79XUWodM2YMOnfujJo1a2Lbtm144403EBcXhzlz5kAlBlcb1GzfAJEThsE/ohYyY5JxcOYqXD0Ua3XdoJbh6PDmcPiFB0PnpkN6dCKOzv8FV/afk8trdW6M2xc+g/ysXPNnYtfsx6EPf3LK/LSfeC/qD+xw4wNuOnj4eOH3J+Yj9cQlR2XJ5dSoF4TxW19Gbmaeed7ZHeew5LllJX7mloc64c7neiCwTgDSEtLx2ydbceCnw3JZeNsw3Pf+EATXNx7HxNOXsWHWJpzbHYOq0PGhzhg8dQg2vr8Bu7/aWeJ6gaGB6PfmADTu0US+v3TgIpY98415eaeHO6PH/90B3xq+iNl1Hmvf/gmZlzPssMc6uOmaQ6cLBuAJIA8GLQaaFn99uTvcdC2g09WSfWAN2kVo2vlS0itr/Yqm53jiWayaoue5isBaOLiWZOLEiZg5c2ap6xw7dgytWrXC2LFjzfPat28PLy8v/P3vf5elYhXB24TBtZI89b7oNmckjn6yHhd+3o/6gzuh+5yR+PW+j5CfkVNs/az4q9g9YSmy41Pk+7q92uDWOSOxfuD7MOQaqySupWdjXd9pcIX8HPxglZxMmj56Oxre15WBVZGZt3+MnPQbF2IlqdsmDMPeG4xFT34jg3DTHo0x8r+PIO5YPBJPJ+HqxVR88/x3SLmUKtdv278VRv33EUzv9hHyr/8uHSUgJAC3Pn0bEk8klLqep68nHls8EodWHsTaN1fjWs41hLWpa17e8NZG6D2uH5Y9/Q0un0pE/3cG4t4P78PSUUvssNciCOShwHAAgPg70cPdLRIGLRcarsrAC50nCgw7ZPB1d+sAA3KgadbzWNb6FU2vKmgKOiRpFVz/tddew+jRo0tdp0kT44VYUd27d5fVwufOnUPLli2hCttcK0kEk5zLaYhZtQeGawXyNSc5Xc635lpqtjkQQaeDZtDg4e8Nn1rqe6lVx/w0GHYLYn7aa88skBU169dAysUUGViFM9uikRqXipBmdeT77JRsc2DV6QCDwQDvAG9ZynW0AZMG489//SH3qTTt7+uArKvZ+GvBH8jLzINWoCHu0I2Ltvb3d8Th1Qdx6eBFXMu+hi0f/YYGXRuiRv0adthrUXoU363pAjQNGlKg0wXJ06tOFwKDIRqAuFDJliVNN92NCwFLZa1f0fRuHnXq1JGl0tImUUK1JioqCm5ubggJCVG6Tyy5VpK+WRjSTsZZzBPv9c1L74E2aNPbcPf1gpuHO2LX7kPWpavmZWJ+/7UTZKBK3h+No59skAHPWfNjEhwZAf8GtRC7Zp/y/b5Zvbzuebh5uOHCgYtYP/NXXD6bbHW9k3+cQa8Xe6LZ7U1wZttZNOvZFL6BPji3x7La95394+Hl5wV3Dzfs++EArl64fuHkIK0GtJZB/fCqg+jwQMdS123QrSHSE9Lw8H8eRXj7eki5cBVb523Bma2n5fKQliHYs2S3ef3M5ExkJmWgTssQpNg9X27QQQ+DlgjADzqdKL8Uqo7WMgBdgxI+W9b6FU2valTnoTjbt2/Hzp070bt3b9ljWLx/9dVX8fjjjyM4WFTtq8PgWknufl64lm5ZXSree/iVXmcvqn3dvD0Q3rst3LxFG41RxvnL+P3xT5F+7jK8g/3R9pXB6PbRE9g66l8OuReY6vwU1uDeW5Dw5wnkXslUus83o6yrWfjXff/FpaPx8PL1RO+X7sSTix/HvIELkJtxox3WRJTcolYewhP/flheAGkFBvxv4mpkJFkei6mdZsHD2wPtBraWr47ko/dBn/H98O1T35Rv/SBfNOzeCD/84zt89/wyNLurOe7/54P477CFuBpzVV4k5Bb5Leek5cDLX117WkncdC2hIQsaLoueCdC0AotKTk2WOEv6ft3LWL+s5dVDdX4qjre3N5YtW4bJkycjNzcXjRs3lsG1cDusKtXrqFRj9QZ0QIc37pX/zopPQdKuM7KdsjDPAB/kppQdQESb5IX1B9Br2RhknLuMKwfOIzc5Q06CeD3w/koM/u0dBDSohYzzSU6XHxNRqq3XNxJ731muPA83gw7D2mH4tCHy36J6d96ghbhw0FgFKtpc183YiI73RqJB5wic2nqm2Oe7PNgRPZ+5DQse+AIJJxIQ2jIUI/87AjlpuTix5ZTFuqKNNWrVIby87v9w+UwSzu+13pnNVm2HtsOg94x5Sr2UgotRF3Dg+yhcPX+lXJ/Py8rDxf0XcHLTCflevMYfiUPjnk1xdekeudw70DKQegf6IC+z7DZqWxg7Nvleb38VRCB0u94uawwXOnnKLaktu6z1K5oeFSV6Ce/YIdqr7Y/BtZwubjggJ5MGw7qgyYgeFuvoW9TFmaV/lTtNUZIQPXMLByMzO5dWHZWfev3b41pmLhK2nVS05zeXA6sPy6k0Wim/lfA2YTj5+2nEHzd2eBGvp/84ixZ3NS0WXE3cPd1Rq1FNuwXXIz8dlpPJC5vGyCrhrqO6y/fi33XbhSOiSwP8MOa7Yp9PPJ6ARrc2LjH9xBOJCG11oznDr6YfAuoE4PIJUVVrz8Cqvx5YRRAUsq4HQf8bVbk60ZZd0gVrWetXNL2qUZ2rhR2JHZoqKW7LUfiEBMmgpPNwl68+tQMRv+WI1fVDe7aEvlkodO5ucPf2RPPRd8EnRI9k01CcLo3lsBbBM8hXDmVJP5uIjNhkp8yPiUhHtMW6xF9LNVC/Qz3UaVpbDpvx8vPEgPF95fk2Zt8Fq+vH7L+A5nc2RUhzYwcm8dr8jqayWllo2bs5wlqGwM1dB08fD9z1fE/ow/QOHYrz1cOfyyrdz+/9TE5xh+Ow4/NtWPfuGqvri17CoW3C0KyX6DkL+Sren/3DWHI/+EMU2g6LRN3IcHj4eKDX2D6I2X3ebu2txsAadD2wFi5FGqBpiXBzExcC7gB84aarB4Nm2beh/OtXNL2qoSn6z9mx5FpJ19Kyseu1JWg/fhgixw1FRmwSdo5dYm639A0NQu/lL2Pzw/OQnZAKryA/tH15EHzq6FGQl4/00wnY+epiZF28Yh432nny3+Cp90N+Zi6S9p7FzrGLHRaUVOdHCGhcB8Ft62PfOysckoebQc0GNXD3q71lb17Rnhp74CK+HP01cjOMVZ5BdfV4ZcMLmDvgX0iNS5Ol3hrhQRj57xHwr+WPrJQs7P0+Cnu/i5Lr+9f0w+A374Y+VC+rhRNOJmLxM9/iSkzxjmn2klmk/Vf8nkR+sq8aew2LEqzovPRhZ+PN1VNir+KHl79Hvwl3Y/icB3A15oos4Yr5wvkd57Blzm944NOHZHuuCKyrxv1op733hptbPWiaAe5ut5nniqExBu0kDNopuKHF9WWmcak3hs24uUVC01KhacaLmbLWL2s5VR86rbQ6pUL3dxS3nkpNTS3XgF5nsbrbW1W9C1SGYbumw9W82XQKXE2AC16mjz96J1xJWlomatUcYrfzuClOvNzwDXi7+diUVq4hB/POz3DqmOOCfxJERFRVqnNvYUdicCUiImXYocmIHZqIiIgUY8mViIiUEb14NFvvLewCJVcGVyIiUsZwfbKFrZ+vDlgtTEREpBhLrkREpAw7NBkxuBIRkToK2lzhAsGV1cJERESKseRKRETKsEOTEYMrEREpw6E4RqwWJiIiUowlVyIiUobVwkYMrkREpIx40JpmY72urZ+vDhhciYhIGY5zNWKbKxERkWIsuRIRkTJ8nqsRgysRESnDamEjVgsTEREpxpIrEREpw5KrEYMrEREpbnPVbE7D2TG4upg8g2vV9Ot0rvcTHddoElxNkKcrnA4tebj3givxcE+r6l24qbjemYuIiKoMq4WNGFyJiEgZ3rjfiMGViIiUEe2tBpvbXJ0/urpWAx0REVE1wJIrEREpw2phIwZXIiJSho+cM2K1MBERkWIsuRIRkTJ8nqsRgysRESnDca5GrBYmIqKbxvTp09GjRw/4+fmhRo0aVteJiYnBPffcI9cJCQnB66+/jvz8/ApthyVXIiJSxqBgnKvBjuNc8/Ly8OCDD+K2227D559/Xmx5QUGBDKxhYWHYtm0b4uLiMHLkSHh6euL9998v93YYXImISO2N+zXb07CX9957T74uWrTI6vJffvkFR48exa+//orQ0FB07NgRU6dOxYQJEzB58mR4eXmVazusFiYiomopLS3NYsrNzbX7Nrdv347IyEgZWE0GDBggt3/kyJFyp8PgSkREyquFDTZOQkREBIKCgszTjBkz7L7/8fHxFoFVML0Xy8qL1cJERKT2Dk2wPQ0hNjYWer3ePN/b29vq+hMnTsTMmTNLTfPYsWNo1aoVHIXBlYiIqmWHJr1ebxFcS/Laa69h9OjRpa7TpEmTcm1bdGTatWuXxbyEhATzsvJicCUiIqdWp04dOakgehGL4TqJiYlyGI6wceNGGeTbtGlT7nQYXImISBmDpqDkasc7NIkxrFeuXJGvYthNVFSUnN+sWTMEBASgf//+Mog+8cQTmDVrlmxnffvtt/Hiiy+WWC1tDYMrEREpI57FqlXj57lOmjQJX331lfl9p06d5OvmzZvRq1cvuLu7Y82aNXj++edlKdbf3x+jRo3ClClTKrQdBlciIrppLFq0qMQxriYNGzbEzz//bNN2GFyJiEgZTcEj4zQ4PwZXG9Rs3wCRE4bBP6IWMmOScXDmKlw9FGt13aCW4ejw5nD4hQdD56ZDenQijs7/BVf2n5PL20+8F/UHdrjxATcdPHy88PsT85F64pLd8+JTKwCd37oXwa3D4VtHj42PzkfqydLHdIXf1RqRLw+Ab0ggUo7HYe/UlUg/n1Tu5aoNHjwYEya8jsjIdrh27Rq2bv0Dr7wyFhcvXjSvc++9wzB79kzUq1cP+/btxzPPPIcTJ06UmGZZ61c0PRV8Ar0x5K270e7uVnD3dMfl6GQseHgRruWUfu/TgeP6oO+LPbHoueU4svHGPt76SGf0efEO+Af74syO8/hu4k9Iv5wBR6ndOgy9Jw+Fvn4wdDodrpy9jO0f/4q4vTFW16/ZrA5uf70/6rQJh2+wH/5z2wfIS7e8uUBwk9roOWEAwjpGwJBvwNlNx7F50moH5ejmVt1vf+govIlEJXnqfdFtzkhEr9iB9X2nIfq7Heg+ZyQ8Anysrp8VfxW7JyzF+runY13faTjz9Z+4dc5IuHkbr28OfrAKP/eaYp5OLPwVGecvOySwmh7xFL/tFLaNW1qu9QMa1ka3aX/DwTk/Y3WfGUjcfRY95jwGnbtbuZbbQ1CQHjNnzkZERCM0btxM3lFlxYpl5uUtWrTAN98swauvjkPNmnXw22+bsWrVD7KNxZqy1q9oeirodMBTnz8CwzUDZvb5FJM6zMT3b6xBQX7pZYW6rUPRpm9zpCakW8xvelsjDJ7YD0te+h6Tb/kI6UkZeHTufXCk9EupWP/KCnx++yz8t8dMRC3ahiH/ehTu1/82ihLB8vSGo9j09kqry/3qBGD4F6PkOl/c+SG+7PURDn1rObSCyN4YXCupbq82yLmchphVe2C4ViBfc5LT5XxrrqVmIzs+xfhGp4Nm0ODh7w2fWoFW128w7BbE/LQXjpJ7JRNnv9+Fq0dulPJK03BQB1zeE424P0/CkJePY//dAu9gf9Tu2LBcy+3h22+XyXaSzMxMZGVlYe7cf6J7927mYPf4449h8+YtWLt2rbyN2tSp02RX+zvuuMNqemWtX9H0VGjZqzlqhAdh5eR1yE7NkYPtLx2NlwGnJKKm5MEZQ7By8noUXCuwWNb1wY7Yt/IgYqMu4lr2Nayb/RuadG+ImhHWnxZiD7mp2UiPS72+s4BWoMHL3xt+tQOsrp9yLhnHftiPK6cSrS7vOPI2XNgZLdcpyM2Xf59Jx8p/Zx1S8zxXzcbJ2bFauJL0zcKQdjLOYp54r29e+iDjQZvehruvF9w83BG7dh+yLl0ttk5wZAT8G9RC7Jp9qK6CmocipVD+tQID0qIT5fzLe6PLXO4Id911p7wri+huL7RvH4moqAPm5eIRUkePHpPzt2zZUuzzZa1f0fRUaNq9IZLPX8GIOfehxR1NZPXtls+2Ye8PB0v8zJ1P34q444k4u/N8sWV1W4Xgr692m99nJGXKNMX8K7HXLwYd5JltE+DpJ/423HB8VRTSL1Zu++G3NETS8Xjcv+RJBDeujStnLmPbhxuRcKh8F45kG1YLGzG4VpK7nxeupedYzBPvPfxKHwclqoRFVXB477Zw8/a0uk6De29Bwp8nZGmyuvLwLSH//t7lWm5vxidZvIcHHxxhnifGsKWkWJ6wxfvAQOu1B2WtX9H0VPCt4YNmPRrjx3fXYfm4lYhoXw9PL3oUVy6kIHpX8TZKUQLtMbIr5g75t9X0vP28kJ1meZzEe28HHafCRJWwqApuenfrEquEy8MnyBfNB7fDT//3DRIPXUTbB7vgnvmP4JshnyK3SF6J7IXBtZzqDeiADm/cK/+dFZ+CpF1nZLtrYZ4BPshNKTsgGnLzcWH9AfRaNgYZ5y7jyoEbJQpRqq3XNxJ731kOe4oY2B5d3hwm/50Zl4qND39Soc/nZ+fJ/BYm3udn5pZruQqPPvoIPvtsgfz3+fPn0a6dsUNYu3btsG7dGrz00hj52CiTjIwMefPvwsT79HTLdsjyrl/R9Cqj073t8MD0IfLfVy+m4NSfZ5FyKRXbFhtLm+f2xsrOSW36tLAaXP/2/hBs+GizrEK2JjcrT3aQKswn0Ae5Co9TUS3uiUSvd415Sr+Ugm+HG4+hIKpxT645hEdWPo+Us0mI22+9g2BprmXlIf5ALOKvf/bQt7vR6enbEdahPs7/cVphTsgallyNGFzL6eKGA3IyaTCsC5qM6GGxjr5FXZxZ+le50xRVw6KnceHgWq9/e1zLzEXCtpOwp9j1B+VUWamnEhDU4kYVuOiopG9cB6mnE8q1XIWlS7+VU2EisP766wZMnPgmvvnGsnPWwYOH0LHjjR7ZHh4eaNOmNQ4dOmw1/bLWr2h6lbF/1WE5mdzytw6IHNi63J9v3rMJwtuEYdg7A+R73yAfjPhoOHat2I+fpv0iq4vFchP/Wn7QhwTI+fZycu0hOZX1txHUsFalgmvSiXjZzkxVw/RcG1vY+vnqgB2aKiluy1H4hATJIKvzcJevPrUDEb/F+vP+Qnu2hL5ZqAwy7t6eaD76LviE6JF8fSiOiUhHtMXC4PgrNzcvDznJf3u6G/8tuqdacX7dAYR0bYKw25vLdVs/fRdyU7KQtP98uZbbg7hlmQisb789CYsW3bgDi8nXX3+DPn16Y9CgQfKBx2+99SaSkpKwdetWq+mVtX5F01Ph8Ibj8PD2wK2PdpEBJKJjPbTt1xJHfrU+/GfabR/j43s+M09pCelYPW0Dfv2ncR93fxeFzsMjEdEhHJ4+Hhj0eh/ZNuvI9taGdzVHrRYh0LmL4Wce6PJsT/iH6nFpb8m/FXcvd7hf/62KV/He5Oj3+9C4dyuERtaT31Hbh7rA3dMDcVEVD9RUtY+cc2YsuVbStbRs7HptCdqPH4bIcUOREZuEnWOXmNsZfUOD0Hv5y9j88DxkJ6TCK8gPbV8eBJ86ehTk5SP9dAJ2vroYWRevmNMMaFwHwW3rY987K6okT/dve9f8775f/Z98/f3vn+Py3nOyl2/Pfz6BlXdOk/Mzzidh1zvfo8Nr98AvRI+rJy5h29hvZMel8iy3h3Hjxsqbd3/88UdyMmnTJlI+uurkyZN4/PGRmDdvDurXry/HpQ4bdp+5w1PPnj1ldXJgoLGnbFnrl7XcHnLSc/HF09/ivvcGYehbdyMlPg0/vvszzu0xBo7GXRvg6S8fxdvtPpDvU+Mtq6gNBg1ZV7PN7axntp/Dulm/YeSCh+AX5IMzO89j6Ss/wpF8a/jh9nH9ERCqR35uPpJPJWDtC0uRFmvs7Fe3cwMMXfgY/t3N+CzPwPAgjPzlFfPnn/p9nHxd3H+uHNYjSrtb31+H/rMfgE+wH5JPJWLti0uLjYUlsiedVo4+z2K8oGhLSk1NLdfjf5zF6m5vwdXkGVyrMuLBvfZ/OLKjjWs0Ca6mkb/zlzSKevHwjYtNV2Dv87gp/e765+Ghs61DXL6Wi51pC5w65rDkSkREyhiu/2cLWz9fHbhWMYeIiKgaYMmViIiU0XQaNJ2tvYWdv5mBwZWIiJTRFPT21VwguLJamIiISDGWXImISBnRGUnHDk0MrkREpA7v0GTEamEiIiLFWHIlIiJlDDoDdDb2Fma1MBERUSFsczVicCUiImUYXI3Y5kpERKQYS65ERKQMewsbMbgSEZEyBhRAhwKb03B2rBYmIiJSjCVXIiJSRtwXWLO5Wtj57y3M4EpERMpwnKsRq4WJiIgUY8mViIgUd2hyszkNZ8fgSkRECtk+FEek4exYLUxERKTYTV1yHbZrelXvApVBw9Sq3gUiqgCDJqp03RSk4dxu6uBKRERq8Q5NRgyuRESkjIYCaDaWXEUazo5trkREdNOYPn06evToAT8/P9SoUcPqOjqdrti0bNmyCm2HJVciIlLGeAMIg4I07CMvLw8PPvggbrvtNnz++eclrvfll19i4MCB5vclBeKSMLgSEdFNc/vD9957T74uWrSo1PVEMA0LC6v0dlgtTERE1VJaWprFlJub67Btv/jii6hduza6deuGL774AppWsYDPkisRESmjaaJDk87mNISIiAiL+e+++y4mT54Me5syZQr69Okj22V/+eUXvPDCC8jIyMCYMWPKnQaDKxERVcs219jYWOj1evN8b29vq+tPnDgRM2fOLDXNY8eOoVWrVuXa/jvvvGP+d6dOnZCZmYnZs2czuBIRkfPT6/UWwbUkr732GkaPHl3qOk2aNKn0fnTv3h1Tp06V1dIlBfiiGFyJiEjxOFedzWlURJ06deRkL1FRUQgODi53YBUYXImISBlNU3CHJs1+Q3FiYmJw5coV+VpQUCADp9CsWTMEBATgp59+QkJCAm699Vb4+Phg48aNeP/99zFu3LgKbYfBlYiIbhqTJk3CV199ZdGmKmzevBm9evWCp6cn5s+fj1dffVX2EBZBd86cOXj22WcrtB2dVo7+xaILdFBQEFJTU8tV/01ERNWLvc/jpvRr67vDTWdbuc2g5SMpbadTxxyWXImIqFoOxXFmDK5ERHTT3KHJUXiHJiIiIsVYciUiIsW9hXU2p+HsGFyJiEgh0eZqexrOjtXCREREirHkSkREyhirdHUK0nBuDK5ERKQMg6sRq4WJiIgUY8mViIiUEY+L09l8437nL7kyuBIRkTKsFjZitTAREZFiLLkSEZEyKu4LrPHewkREREXvC2xQkIZzY3AlIiJlVLSXamxzJSIioqJYciUiImVYcjVicCUiImVUjFHVXGCcK6uFiYiIFGPJlYiIlGG1sBGDKxERKcPgasRqYSIiIsVYciUiIoVUlDqdv+TK4EpERMqwWtiI1cJERESKseRKRETKcJyrEYMrEREpo2kKbtwv03BuDK5ERKSQeFyczuayq7NjmysREZFiLLkSEZEyxp6+OhvTcP6SK4MrEREpZHtwZbUwERERFcOSKxERqaOgWhisFiYiIrpBU1Clq7FamIiIiIpiyZWIiBRihyaBwZWIiBTSFHT2ZbUwERERVabkahrQm5aWVp7ViYiomjGdv+1/gwbRHcn5S54OCa7p6enyNSIiwt77Q0REdiTO50FBQcrT9fLyQlhYGOLj45WkJ9ISaTornVaOyxiDwYBLly4hMDAQOp2td94gIiJHE6d6EVjDw8Ph5mafFsGcnBzk5eUpSUsEVh8fH7h0cCUiIqLyY4cmIiIixRhciYiIFGNwJSIiUozBlYiISDEGVyIiIsUYXImIiBRjcCUiIoJa/w/0ndzpKjipNgAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "def plot_values(V: np.ndarray, title: str = \"Value function\") -> None:\n", + " \"\"\"Plot the value function V on the maze as a heatmap.\"\"\"\n", + " grid_values = np.full(\n", + " (n_rows, n_cols),\n", + " np.nan,\n", + " ) # Initializes a grid the same size as the maze. Every cell starts as NaN.\n", + " for (\n", + " s,\n", + " (i, j),\n", + " ) in (\n", + " state_to_pos.items()\n", + " ): # recall that state_to_pos maps each state index to its maze coordinates (i,j).\n", + " grid_values[i, j] = V[\n", + " s\n", + " ] # For each reachable cell, we write the value V[s] in the grid.\n", + " # Walls # never get values, and they stay as NaN.\n", + "\n", + " _fig, ax = plt.subplots()\n", + " im = ax.imshow(grid_values, cmap=\"magma\")\n", + " plt.colorbar(im, ax=ax)\n", + "\n", + " # For each state:\n", + " # Place the text label at (column j, row i).\n", + " # Display value to two decimals.\n", + " # Use white text so it's visible on the heatmap.\n", + " # Center the text inside each cell.\n", + "\n", + " for s, (i, j) in state_to_pos.items():\n", + " ax.text(\n", + " j,\n", + " i,\n", + " f\"{V[s]:.2f}\",\n", + " ha=\"center\",\n", + " va=\"center\",\n", + " color=\"white\",\n", + " fontsize=9,\n", + " )\n", + "\n", + " # Remove axis ticks and set title\n", + " ax.set_xticks([])\n", + " ax.set_yticks([])\n", + " ax.set_title(title)\n", + " plt.show()\n", + "\n", + "\n", + "plot_values(V_random, title=\"Value function: random policy\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "8275a1eb-b58e-4e05-ae5d-5635ff9a1556", + "metadata": {}, + "source": [ + "The next function `plot_policy` visualizes a policy on the maze.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "c1ab67f0-bd5e-4ffe-b655-aec030401b78", + "metadata": {}, + "outputs": [], + "source": [ + "def plot_policy(policy: np.ndarray, title: str = \"Policy\") -> None:\n", + " \"\"\"Plot the given policy on the maze.\"\"\"\n", + " _fig, ax = plt.subplots()\n", + " # draw walls as dark cells\n", + " wall_grid = np.zeros((n_rows, n_cols))\n", + " for i in range(n_rows):\n", + " for j in range(n_cols):\n", + " if maze_str[i][j] == \"#\":\n", + " wall_grid[i, j] = 1\n", + " ax.imshow(wall_grid, cmap=\"Greys\", alpha=0.5)\n", + "\n", + " for s, (i, j) in state_to_pos.items():\n", + " cell = maze_str[i][j]\n", + " if cell == \"#\":\n", + " continue\n", + "\n", + " if s in goal_states:\n", + " ax.text(\n", + " j,\n", + " i,\n", + " \"G\",\n", + " ha=\"center\",\n", + " va=\"center\",\n", + " fontsize=14,\n", + " fontweight=\"bold\",\n", + " color=\"blue\",\n", + " )\n", + " elif s in trap_states:\n", + " ax.text(\n", + " j,\n", + " i,\n", + " \"X\",\n", + " ha=\"center\",\n", + " va=\"center\",\n", + " fontsize=14,\n", + " fontweight=\"bold\",\n", + " color=\"red\",\n", + " )\n", + " elif s == start_state:\n", + " ax.text(\n", + " j,\n", + " i,\n", + " \"S\",\n", + " ha=\"center\",\n", + " va=\"center\",\n", + " fontsize=14,\n", + " fontweight=\"bold\",\n", + " color=\"green\",\n", + " )\n", + " else:\n", + " a = policy[s]\n", + " ax.text(\n", + " j,\n", + " i,\n", + " action_names[a],\n", + " ha=\"center\",\n", + " va=\"center\",\n", + " fontsize=14,\n", + " color=\"black\",\n", + " )\n", + "\n", + " ax.set_xticks(np.arange(-0.5, n_cols, 1))\n", + " ax.set_yticks(np.arange(-0.5, n_rows, 1))\n", + " ax.set_xticklabels([])\n", + " ax.set_yticklabels([])\n", + " ax.grid(visible=True)\n", + " ax.set_title(title)\n", + " plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "id": "48037254-dccc-4f9c-a4d7-349adba5c74f", + "metadata": {}, + "source": [ + "Now let’s visualize the `random_policy`. Does it seem like a good policy?" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "d452681c-c89c-41cc-95dc-df75993b0391", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAGgCAYAAAC0SSBAAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAH6JJREFUeJzt3Q1wlNWh//Ff1pCQxLw0AQUvEVulcm16UyJSShlem5bSoS1cKCUoOmqrcOWC5Ho7KUVIteoES6UjU65ixb6kV+xM6aU4XAIBW4tN9Z8/oyGASlHBvzLgSyIGQ0L2P+fETUKCxw0meXZPvp+Z4+4+2eA5eZ7d356XZ5+EcDgcFgAAHyH0UT8AAMAgKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIiiADiZNmmRLxCuvvKKEhARt3Lgx0HoBQSIoEPfMm7h5M4+UgQMH6rOf/axuu+02HTt2LOjqAXEvMegKAD3lxz/+sT796U/rgw8+0NNPP61f/OIXevLJJ1VTU6PU1NTz+jeHDx+uU6dOacCAAT1eXyBeEBTwxte//nWNHj3a3r/55puVk5OjNWvW6I9//KPmzZt3Xv9mpIcC9GcMPcFbU6ZMsbeHDx9Wc3Oz7rrrLl1++eVKTk7WZZddph/+8IdqbGx0/hsfNUdx4MABfec739HgwYOVkpKiK6+8UsuXL7c/27Vrl/2dP/zhD13+vfLycvuzZ555pkfbCvQmggLeOnTokL01PQvTw7jzzjtVUFCgn/3sZ5o4caLuvfdeffe73+32v/v888/ri1/8oiorK/W9731Pa9eu1be//W1t2bLF/txMhufm5uq3v/1tl98120xYfelLX+qBFgJ9g6EneKOurk4nTpywcxR//etf7ZyF+bQ/cuRI3XrrrTYsHn74YfvcRYsW6aKLLtL9999vewCTJ0+O+v+zePFimcu4VFdX69JLL23bft9999lb02O49tpr7bCXqVNmZqbdfvz4cW3fvr2t5wHEC3oU8MZXvvIVOxRkPs2bnsKFF15oh3/27Nljf75s2bKznl9cXGxvt27dGvX/w7zZ//nPf9aNN954VkhEAiJiwYIFdljr97//fdu2xx9/3A6BmRAB4gk9Cnhj3bp1dllsYmKiLr74YjtvEAqFbFiY2yuuuOKs5w8ZMkRZWVl69dVXo/5//OMf/7C3eXl5zueZXsw111xjh5puuukmu83cHzt2bJd6ALGOoIA3xowZ07bq6Vw6fuLvC6ZXsWTJEh09etT2Lv72t7/pwQcf7NM6AD2BoSd4z5wL0dLSopdeeums7eZkvHfffdf+PFqf+cxn7K05N+PjmOGvCy64QL/73e9sb8KcizF37tzzaAEQLIIC3ps+fbq9feCBB87abiabjW984xtR/1tmDmTChAn65S9/qddee+2sn5kJ7o4GDRpkz+34zW9+Y4Ni2rRpdhsQbxh6gvfy8/N1/fXX66GHHrI9CLM09u9//7see+wxu6y1OyuejJ///OcaP368XWr7/e9/354Nbs63MJPie/fu7TL8NHv2bHvfnMcBxCOCAv3Chg0b7LCROXHOTG6bieySkhKtXLnyvILHzDesWLHCfk2IWY5rhq/MCXidzZgxQ5/61Kfs0Nc3v/nNHmoN0LcSwp37ywB6jFkOe8kll9jAeOSRR4KuDnBemKMAetHmzZvtuRdmCAqIV/QogF5QVVVlv+rDzEuYCWxzFjcQr+hRAL3AzF0sXLjQfk3Ir371q6CrA3wi9CgAAE70KAAATgQFAKBnzqMw31XT8SIvZl3422+/bb/rv6+/QwcA8MmYWYf33nvPLt82X5rZI0FhLvJSWlr6CasGAIglR44c0bBhw3pmMrtzj8JckMV8H/8dd9zhzYXnTapeddVVqq2ttT2meOdbewzaFB9oU+xramrS6tWr7dfaRC6u9Yl7FOY6w6Z0ZkIiKSlJvhwIqamptj0+HAi+tcegTfGBNsWPaKYOmMwGADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAKTHaJzY2NtoSUV9fb29DoZAtPoi0g/bELtoUH2hT7OtOOxLC4XA4mieuWrVKpaWlXbaXl5crNTW1ezUEAASqoaFBRUVFqqurU0ZGRs8Exbl6FLm5uXrjjTeUk5MjHzQ1NamiokKFhYUaMGCA4p1v7fG9TTU1NWppaZEvn1bz8vLYTzHs9OnTuvvuu6MKiqiHnpKTk23pzBwEvhwIvrbJt/b42ibz5uPDG1BH7KfY1Z02+DHYBgDoNQQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEA6JkLF/W1/675bz2691HtfXOv3j71tlIHpCo7JVvDM4cr/+J8TR8xXV+74mtBVxMfWrJkiR5//HG9+eabQVcFQH8IigV/WKBfP//rs7bVN9bb8sq7r+ipV5/Sq3WvEhQxxFxO8dixY0FXA0B/CIptL287KySuHnq1vnb513Rh0oU63nBc1W9U65mjzwRaRwDoT2IuKLYf2t52/4rsK1R1c5UuCF1w1nNMz+KFYy8o3q5Pe8cdd+imm27SVVddFXR1cB4Xol+2bJndh8OHDw+6OjgH9lE/msxubmluu//uB+/aoabOMpIz9OVLv6x4cebMGV133XVas2aNHn300aCrg/NQXV2tDRs2aMKECTp06FDQ1cE5sI/6UVAUDC1ou3+i4YQ+++BndfVDV+vWP92qh//Pw3r57ZcVT5qbmzVv3jyVl5dr0aJFKisrC7pKOA9jx47Vli1bdPz4cftGdPDgwaCrhE7YR/1o6Onaf7lW655dp+f+33P2cUu4xc5LmBIx/tLxevDrDyp/SL5i3Zw5c7R582ZlZmYqISFBixcvjnoV0YgRI3q9fpBWrFihd955J6rn5uXl6dlnn9XEiRP11FNP6corr+z1+oF9FLSYC4rEUKIqF1Tq3qfv1S//7y917P2uK2mefu1pFf66UPsW7dPgtMGK5XmJ3bt3t60KWrduXdS/O3v2bIKij5jhwNdff71bv2NWeNXW1vIm1EfYR8GKuaEnIz05XfdMvUdvFL+hmoU1euSbj+j6/OuVnpTe9hyzAqrzEtpYEwqFtHPnTmVnZysrK0tVVVUKh8NRlUmTJgVd/X7j6NGjUe2TkydPtu2X0tJSzZw5M+iq9xvso2DFZFBEmKGaz130Od046kZt/PZGPb/weYUS2qv80lsvKdYVFBSosrJSiYmJKiws1J49e+TT5OH69eu7bN+/f7/Wrl0rn7z33nuaNm2a7SHed999uvPOO4OuEjphH/WjoafH9j6mD5o/0LzPz7OrmzpKG5Bmg8LMWxhZA7MUD/Lz87Vr1y5NnTrV9jDGjRsnHyxfvlzbtm1TQ0ND27Z9+/ZpypQpOnXqlGbNmqXc3Fz54OWXX9YLL7xgV67dfvvtQVcH58A+6kdBcfjdwyp9qlRL/3epnbT+wsVfsF/d8dapt/T72t+ftXx22hXTFC/MBJt5Ex00aJB8sWnTJk2fPl3FxcUaPLh1rmjy5Ml2Pfv27du9CQlj1KhR9o3Ip/3nG/ZRPwqKCNOr2PGPHbacy/cKvqeJl01UPPHtAE5PT7c9ihkzZtgeU2Q58I4dOzR69Gj5xrf95yP2UT8JiqVjl+rzF31elYcr9dwbz+nNk2/q+PvHdSZ8RoNTB+vqS662E9uz/nlW0FWFGQ5MS9PWrVv1rW99y85ZVFRU2E92APwRc0Fh5h3+9ap/tQXxISUlxQ41AfBTTK96AgAEj6AAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAA9MylUBsbG22JqK+vt7dNTU22+CDSDtoTu3xuUyjkz+e2SFvYT7GrO+1ICIfD4WieuGrVKpWWlnbZXl5ertTU1O7VEAAQqIaGBhUVFamurk4ZGRk906MoKSnRsmXLzupR5Obmqra2VklJSfIlYfPy8lRYWKgBAwbIh09AFRUVqqmpUUtLi3zg2z7quJ9oU2xr8uz1dPr06aifG3VQJCcn29KZ+YP58EfryBzYvhzcBvsoPtCm+NDiyeupO23wY7ANANBrCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAnTQ3N2v+/PkaOXKkXnzxRfliyZIlGjJkiHzh636KRQQF0OnykHPmzLHXgj948KAmTZqkAwcOyAfm2sjHjh2TD3zeT7GIoAA+1NjYqFmzZmnz5s1tF5s/efKkfRPat29f0NXDh9hPfY+g6MPr0xYXF6u2tjboquAjzJ07V1u3blVJSYlmzpxpt23fvl2nTp3S5MmTdfTo0aCrCPZTIBKD+d/2L2fOnNGCBQtsNzkUCmn16tVBVwnnsHTpUl1zzTVavny5brjhBrtt7Nixqqio0JYtWzRs2LCgqwj2UyAIij6YcCsqKtITTzyhRYsWqaysLOgq4SOYoQtTOhszZowtiA3sp75HUPQyM+FmxlIzMzOVkJCgxYsXR71CZcSIEb1ePwD4OARFL89L7N69u23Fybp166L+3dmzZxMUAGICk9m9yMxH7Ny5U9nZ2crKylJVVZXC4XBU5VxdawAIAkHRywoKClRZWanExEQVFhZqz549QVcJ/Uh1dbXWr1/fZfv+/fu1du3aQOqE+MPQUx/Iz8/Xrl27NHXqVNvDGDduXNBVQj9hVgZt27ZNDQ0NbdvMuQZTpkyxy0nN+Qi5ubmB1hGxj6DoI3l5efYFOmjQoKCrgn5k06ZNmj59uj2HZ/DgwXabOdfAnNlszj0gJBANhp76ECGBvpaenm57FCYcjh8/3rZke8eOHfbcAyAaBAXgubS0NHsms5kjy8nJscOfo0ePDrpaiCMEBXAOGzdutKvPfJGSkmKHmk6cOKFRo0bJF77tp1hFUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgFOiotTY2GhLRH19vb0NhUK2+CDSjqamJvkg0g5f9o+P+6hjW3xsU1lZmVpaWuTLsZeXl+fN66k77UgIR3nB2VWrVqm0tLTL9vLycqWmpnavhgCAQDU0NKioqEh1dXXKyMjomR5FSUmJli1bdlaPIjc3V7W1tUpKSpJPnxgKCws1YMAA+fCprqKiQjU1Nd59qvNlH3XcTz62ycdjr8aTNp0+fTrq50YdFMnJybZ0Zv5gPvzROjIvVl9esAb7KD742CYfj70WT9rUnTb4MdgGAOg1BAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIij6WHNzs+bPn6+RI0fqxRdfDLo6+AhLlizRkCFDgq4GEBMIij6+9OCcOXPsdcYPHjyoSZMm6cCBA0FXC+dgriN87NixoKsBxASCoo80NjZq1qxZ2rx5c9uFzE+ePGnDYt++fUFXDwA+EkHRR+bOnautW7eqpKREM2fOtNu2b9+uU6dOafLkyTp69GjQVYSHzHWRi4uLVVtbG3RVEMcSg65Af7F06VJdc801Wr58uW644Qa7bezYsaqoqNCWLVs0bNiwoKsIz5w5c0YLFiywQ52hUEirV68OukqIUwRFHzFDTKZ0NmbMGFuAnl40UVRUpCeeeEKLFi1SWVlZ0FVCHCMoAA+ZRRNmPiwzM1MJCQlavHhx1Ku9RowY0ev1Q3whKAAP5yV2797dtnpr3bp1Uf/u7NmzCQp0wWQ24BkzH7Fz505lZ2crKytLVVVVCofDUZVzDY8CBAXwoerqaq1fv77L9v3792vt2rWKJwUFBaqsrFRiYqIKCwu1Z8+eoKuEOMbQE/AhsyJt27ZtamhoaNtmznGZMmWKXcZszoPJzc1VvMjPz9euXbs0depU28MYN25c0FVCnCIogA9t2rRJ06dPt+cdDB482G4z57iYM+rNOS/xFBIReXl5NuwGDRoUdFUQxxh6Aj6Unp5uexQmHI4fP962zHTHjh32nJd4RUjgkyIogA7S0tLsGfRmXD8nJ8cO2YwePTroagGBIigCsHHjRrvCBLEpJSXFDjWdOHFCo0aNCro6QOAICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcEpUlBobG22JqK+vt7ehUMgWH0Ta0dTUJB9E2lFcXKwBAwbIlzZVVFSorKxMLS0t8uW4y8vL8+a48/3YK/akTW+99ZbuueeeqJ6bEI7y4s2rVq1SaWlpl+3l5eVKTU3tfi0BAIFpaGhQUVGR6urqlJGR0TNBca4eRW5urn70ox8pKSlJPn2yKyws9OITQ+QTkC/t6dimmpoa73oUPu4n2hTbPYqhQ4dGFRRRDz0lJyfb0pl5sfrygo0wB4EPB4Kv7TE47uIDbYpd3WmDH5MLAIBeQ1AAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAFYsmSJhgwZEnQ10M9w3MWut96Sysqkr35VuuQSaeBAcw0gaehQacIE6Y47pL/8RYruMnM9L+oLF6HnmCtKHTt2LOhqoJ/huItNDz0kLVsmvf9+15+9+WZrMSFx//3SG29IQWQ9QQEAAVm9WvrP/2x/nJAgTZ4sjR0rXXih9Pbb0t690tNPSx98EFw9CQoACMD+/VJJSfvjnBzpf/5HGjeu63NPnpR+/WspJUWBYI4C581cs7q4uFi1tbVBVwX9iC/H3c9/Lp050/54/fpzh4RhehcLF0qZmQoEPQqclzNnzmjBggUqLy9XKBTSatOHBnqZT8fdzp3t9z/1KWnWLMUsehTotubmZs2bN8++WBctWqQys1wD6GW+HXevv95+f8QIKdTh3fjAgdb5is7lhhsCqSo9CnTfnDlztHnzZmVmZiohIUGLFy+OennmCPOKAM6Dz8ddQoJiGkGBbo8P7969u2255bp166L+3dmzZ8f8Cxaxycfj7p/+SXrppdb75tacIxEJjIsual0RZaxcKTU0BFdPg6GnPlJdXa31Zraqk/3792vt2rWKF2ZceOfOncrOzlZWVpaqqqoUDoejKpMmTQq6+v0Ox13sHndTp7bfN8tgzYqniOxs6T/+o7UEtdKpI4KijyxfvlwLFy7UmjVr2rbt27fPHsQrVqzQkSNHFC8KCgpUWVmpxMREFRYWas+ePUFXCR+B4y523XabdMEF7Y9vvbX1nIlYRFD0kU2bNmn8+PF2Wd+TTz5pt02ePFmNjY3avn27cnNzFU/y8/O1a9cuDRw40H7SQ2ziuItdn/ucdNdd7Y/NGdijR0szZkirVkk/+Yl0881Sfb0CxxxFH0lPT9e2bds0Y8YMe6BHVnHs2LFDo83REYfy8vLsp9NBgwYFXRV8BI672FZSIqWltZ6d3djYel7Fn/7UWs7FnJQXBHoUfSgtLU1bt2613eacnBz7iSheX6wRPrxYfcdxF9v+/d+lw4dbexHjx0uDB0uJia1zE5deKhUWtv6sulr66U+DqSM9ij6WkpJiu/xAX+K4i21Dh7aubjIlFtGjAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICANAz18xubGy0JaK+vt7ehkIhW3wQaUdTU5N8EGmHL+3p2Jbi4mINGDBAvrSpoqLCy/3kY5vKysrU0tKieHf69Omon5sQDofD0Txx1apVKi0t7bK9vLxcqamp3ashACBQDQ0NKioqUl1dnTIyMnqmR1FSUqJly5ad1aPIzc1VbW2tkpKS5EuPIi8vT4WFhV58Wo18UvWlPQZtig8+t6mmpqbf9SiiDork5GRbOjN/MB/+aB2ZA9uXg9vH9hi0KT742KYWT97zutMGPyYXAAC9hqAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKPpYc3Oz5s+fr5EjR+rFF18MujowDh2SLrxQSkhoLV/9qtT5wo/mcWFh+3PS0qSXXlI8WbJkiYYMGRJ0NRCHCIo+vqLUnDlz7OVjDx48qEmTJunAgQNBVwuXXy799KftjysqpHXrzn7Ogw9KO3a0P77/fmnECMUTc8nLY8eOBV0NxCGCoo80NjZq1qxZ2rx5c9v1aU+ePGnDYt++fUFXD7fcIk2f3v74Bz+QIj0+c2seR0ybJi1c2Pd1BAJCUPSRuXPnauvWrfba4zNnzrTbtm/frlOnTmny5Mk6evRo0FXEhg1STk7r/YYGacECk/DSdddJp061bs/Olh55JNBqAn2NoOgjS5cu1d1336177rmnbdvYsWPtxdpvueUWDRs2TPF4zd3i4mLV1tbKC0OHSr/4RfvjqippzBjp739v32Z+fsklgVQPHz+0e9ttt+nVV18NuireISj6iBliWr58eZftY8aM0V133aV4c+bMGV133XVas2aNHn30UXljzhxp/vz2x88/336/qEj6zncCqRY+XnV1tTZs2KAJEybokFmggB5DUOC8Vm7NmzfPTsovWrRIZWVl8oqZuDa9i44uvrjrBDdiiumhb9myRcePH7dhYRaMoGck9tC/g37ErNwyk/KZmZlKSEjQ4sWLo16eOSIeVgqZ+aK33z57m3n8yivSF74QVK36tRUrVuidd96J6rl5eXl69tlnNXHiRD311FO68sore71+viMo0O15id27d7ctt1zXjU/Zs2fPjv2gaGpqnbw2k9jn2v7cc1JyclC167fM8Obrr7/erd8xS4HN/BlB8ckx9IRuCYVC2rlzp7Kzs5WVlaWqqiqFw+GoipmniXkrV0p797Y//rd/a79fUyP96EeKt3H79evXd9m+f/9+rV27VvHCrAqM5hiLLDk3SktL21YY4pMhKNBtBQUFqqysVGJiogoLC7Vnzx55wbSj43zLjTe2zlfcdFP7tjVrpL/8RfHCLKBYuHChXXQQYc7bMW+mZjjnyJEj8sV7772nadOm2R7vfffdpzvvvDPoKnmDoMB5yc/P165duzRw4EDbw4h7778vXX+9Wc7V+viyy6QHHmi9b24/85nW+y0trc87eVLxYNOmTRo/frxdxvzkk0/abea8HXMCqDmPJzc3V754+eWX9cILL9hQ/EHHEyTxiTFHgfNmJg3Np9NBgwYp7hUXm3ea1vuhkPTYY1J6eutj8z1Qv/qVNHFia5AcPizdfrv08MOKdenp6dq2bZtmzJhhgz2yam3Hjh0aPXq0fDJq1CgbFl4cjzGGHgU+ES9elNu2Sf/1X+2PTQhMmHD2c7785bO/xsOcxb11q+JBWlqa/VYAM0yYk5Nje4C+hYRXx2MMokcRgI0bN9qCGGG+u6nzt8Wey09+0lriUEpKih1qAs4HPQoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAOiZS6E2NjbaElFfX29vQ6GQLT6ItKOpqUk+iLTDl/YYtCk++NymkGfvd9FICIejuViwtGrVKpWWlnbZXl5ertTU1O7VEAAQqIaGBhUVFamurk4ZGRk906MoKSnRsmXLzupR5Obmqra2VklJSfIlYfPy8lRTU6OWlhbFO9/aY9Cm+ECbYt/p06ejfm7UQZGcnGxLZ+YP5sMfzec2+dYegzbFB9oUu7rTBj8G2wAAvYagAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwClRUWpsbLQloq6uzt42NTXJF6FQSA0NDTp9+rRaWloU73xrj0Gb4gNtin2R9+5wOPzxTw5HaeXKleZfo1AoFIr8KYcOHfrY9/8E85/z6VG8++67Gj58uF577TVlZmbKB/X19crNzdWRI0eUkZGheOdbewzaFB9oU+wzo0KXXnqp3nnnHWVlZfXM0FNycrItnZmQ8OGP1pFpj09t8q09Bm2KD7QpPobUPvY5fVITAEDcIigAAL0TFGYYauXKleccjopXvrXJt/YYtCk+0Ca/2hP1ZDYAoH9i6AkA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAufx/wkBHS6w2LYIAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plot_policy(random_policy, title=\"Policy\")" + ] + }, + { + "cell_type": "markdown", + "id": "cbad5bf1-0150-4c3f-8cce-c82e0f1d1695", + "metadata": {}, + "source": [ + "**Exercise 9.** Define your own policy and evaluate it using the functions `policy_evaluation(...)` and `plot_values(...)`. **Can you identify an optimal policy visually?** Plot your own policy using `plot_policy`. \n" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "929707e6-3022-4d86-96cc-12f251f890a9", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdcAAAGbCAYAAACWHtrWAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAATRhJREFUeJzt3Ql8VNX5N/Dfmcm+EwgkgbALCLKJiigquCHiglvdRau2r0utW9X2X/e2VG2t1lq1dcHWvYtaN1xZVDYFQVkFZAlbWJOQPZl73s9zbiaZSSbJhJwsE35fP+Ms986Ze2fCPPd5zjl3lNZag4iIiKzx2GuKiIiIBIMrERGRZQyuREREljG4EhERWcbgSkREZBmDKxERkWUMrkRERJYxuBIREVnG4EpERGQZg2sE27hxI5RSmDFjRru8/syZMzFq1CjExcWZ7cjPz0dHJNt23333tfdmdFoTJkwwl47yd0nUETC4tpGzzjoLCQkJ2L9/f4PrXHrppYiJicGePXvQ0ck2/uhHP0J8fDyefPJJ/POf/0RiYmK7bc/777/PAEpEHUZUe2/AwUIC5zvvvIM333wTV1xxRb3lJSUlePvtt3Haaaeha9eu6Oi++uorc6Dw4IMP4uSTT27vzTHBVYJ8qABbWlqKqCj+qbeVPn36mPc8Ojq6vTeFqN0wc23DzDU5ORmvvPJKyOUSWIuLi00QjgQ7d+4012lpaejopGzN4Np2pCQs77nX623vTSFqNwyubUTKp+eeey4+/fTTmsAUSIKuBF8Jwnv37sXtt9+O4cOHIykpCSkpKZg8eTKWLVvW7P4vvyuvvBJ9+/YNesxxHDz22GMYNmyY+TLs0aMHfvrTn2Lfvn1Nvsa0adPM7SOPPNJ8mUr7Ql7Df7ux7Zo9e7Z53htvvIHf/va36NWrl9mGk046CevWrav3/IULF+L0009Hly5dTPl5xIgRePzxx2v2TbJWIW36L431uX7zzTfmPZX3Vt5jed0FCxYErSN9hvLcL7/8ErfeeisyMjLMa59zzjnYtWtX0LoFBQVYvXq1uW6KvEdnnHGGeQ+OOOII87chn7XcF//973/NfXk/xowZY7bV74UXXjDbFPiY3+9+9zsT0LZu3drga8v7IM+XbZWyvuy/VEp+/vOfo6ysLGjdqqoqU5kYMGAAYmNjzXb/6le/Qnl5eaP711Cfq/815X2UfR48eDD+7//+zyybNWuWeY5UdkL925Bl8+fPb/R1iToSBtc2JFmpfGFJQAkkwfTDDz80X9rypfPDDz/grbfeMl/Ajz76KH7xi1/gu+++wwknnIBt27ZZ2x4JpNL2scceawLVVVddhZdffhmTJk1CZWVlg8+TL8Sf/OQn5vYDDzxg+lulrQPx+9//3nyhysHEL3/5SxPg6mbvH3/8MY4//nisXLnSBIE//vGPmDhxIt59992a/TjllFPMbdkW/6UhK1aswHHHHWcOVu644w7cfffd2LBhgwn+EsTr+tnPfmbWvffee3HdddeZ8v6NN94YtI7sw6GHHhoyOIQiBxCXXHIJzjzzTEyfPt0c0Mhtef9vueUWXHbZZbj//vuxfv16E5DkQEicf/755m9E1qtLHpN96NmzZ5OvL21KMJXXloOWP//5zzWfqd8111yDe+65B4cffjj+9Kc/mb8/Wf+iiy5Cc3377bcYO3YsPvvsM1x77bXm723q1KnmvRSy3Tk5OQ3ulwT4cePGNft1idqN/J4rtY2qqiqdlZWlx40bF/T4008/Lb+pqz/88ENzv6ysTPt8vqB1NmzYoGNjY/UDDzwQ9Jg874UXXqh57IQTTjCXuqZNm6b79OlTc//zzz83z3355ZeD1ps5c2bIx+uS15T1vvrqq6DH5TXkteqqu12zZs0yzz/00EN1eXl5zeOPP/64efy7776rec/69etn2t23b19Qm47j1Ny+4YYbzPNCkcfvvffemvtTp07VMTExev369TWPbdu2TScnJ+vjjz++3j6efPLJQa91yy23aK/Xq/Pz8+utG/hZNET2RdadN29ezWPy2ctj8fHxetOmTTWPP/PMM+Zxeb/8Lr74Yp2dnR30N7JkyZKwXl/eB1nvrLPOCnr8+uuvN48vW7bM3F+6dKm5f8011wStd/vtt5vHP/vsswY/21B/l/K+yvsbuG8i8H395S9/af7GA9/XnTt36qioqKDPjygSMHNtQ1Kyk6N+KW9J6Syw7CUlWSlNCinBeTzuR+Pz+czIXCldShltyZIlVrblX//6F1JTU03Gt3v37pqLlCHltaRM1xYkW5YR0n6SUQrJ3oWUPyWrvPnmm+v17waWfsMl7+dHH31ksqb+/fvXPJ6VlWUyyS+++AKFhYVBz5GMLvC1ZBulnU2bNtU8JqVpieOhSuKhDB06NCgTk6xOnHjiiejdu3e9x/3vh5ABcVLBCPyMJLuTjPa8884L6/VvuOGGetm5f2BY4LWUwwPddttt5vq9995DuKSEPnfuXPz4xz8O2jcR+L7KfknJ+d///nfNY6+//rqp9kgmTxRJGFzbmL/k6R/YtGXLFnz++ecm6PoHgEgJUMpwhxxyiAm03bp1M/1UUloLp08vHGvXrjVtde/e3bQdeCkqKgrZL9wa6n7ZSp+q8Pf7SllUHHbYYVZeT77oZWS2HKjUJWVdee9zc3ObtY0Hom6bcqAjpDQa6vHA15IDIjkY8JdQZZtfffVVnH322abfPhzytxVIyq5yQOc/6JMDB7k/cODAoPUyMzPNQU7ggUVT/AcGTX2GQ4YMMX34gaVhuX300UfX2w6ijo5DKNuYZIbyJSJfhjI4RK4l4wnsZ5SBKdIPKEf6MqAkPT3dfNFJ9ubve2uIZAJuJTSYZFqBpB0JrKH6uIQE2QPRUDYprx9q9GhDI0pD7UN7aY1tbKjNcF5L1pEs++9//zv++te/mgFXksm2JLtr6HM7kOpAS0j2Kv3qctApWaz0wf/lL39p020gsoHBtR1IIJXgKZmoZLCSRcgRu5+UxWTAznPPPRf0PDkDkmSxjZGsKrCE6Fc305BM5ZNPPjGDmaScaIu8fqgzNcnrB5ZhwyXbKZYvX97ofNpwg4AcNMjJPNasWVNvmYxmlYOYutljRyRBSAZ2yYCgDz74wOyXDERrTuWiX79+QQOs5IDLP6Jc5qrKfVlPMnq/vLw88/nK8nD5P3f5DJsiFRwpRctBp3+u7IUXXhj2axF1FCwLtwN/liojMZcuXVpvdKxkJnWzIukjbWyKRWAwkiAROFVERrpKdlN3tKhkk5IZ1yV9XAd6KkN5fck2Kioqah6TUb11S63hkpGqEgRkylDdbQp8j/xnh2pqu+W9PfXUU8284sB+bwkacqAzfvx4Mz2luZozFccGmYokl2effRb/+c9/TFBqzlxe/9QlvyeeeMJcy/QkISOIhbzvgWT0upgyZUrYryWBX0Z7P//889i8eXPQsrp/53LwKNvw0ksvmaqKnFSlqQNKoo6ImWs7kGBxzDHHmC94UTe4yhQcmeIig31kPZmGI1804WR+UkqWL0DJYq6++mrTd/r000+buayBA3VkWoVMYZGpFRLgJeBIliCZigRymSoh0z6aS6ZvSOYtX4oSwKXPVL4o/Rloc0km+dRTT5lpKnIeY3lPpL9RAplMqZEpTP5yu7jpppvMvvsHj4Xym9/8xkzvkUB6/fXXm6D0zDPPmDLkww8/fEDbKVNwZNtkHmq4g5psZK8yhUk0tyQsg8RkTrV8TjLATj4jKTWPHDnSLJdrmcv8t7/9zRywyN/LokWL8OKLL5rBYFJZaQ6Z6iPvtxwsyQAx+TcgBzcyMEr+/urul/9vL9TBH1FEaO/hygerJ5980kxXOOqoo+otk6k4t912m5m2I1Mzjj32WD1//vywpjyIl156Sffv399MNxk1apSZ5lF3Ko7f3/72Nz1mzBjzOjJVYvjw4fqOO+4wU1MOZCqO+OMf/6h79uxpplXItn/99dcNTsX517/+FfTchvbpiy++0KeccorZxsTERD1ixAj9xBNP1CyXKTs/+9nPdEZGhlZKBU3LqTsVxz91ZdKkSTopKUknJCToiRMnBk2NaWwf/dseOD2muVNxpkyZUu9xeb5MKQr1fjzyyCP11t++fbuZEjRo0CAdLv9UnJUrV+rzzz/fvJ9dunTRN954oy4tLQ1at7KyUt9///1mKlR0dLTOyckx02Xk7zNQuH+Xy5cv1+ecc45OS0vTcXFxevDgwfruu++ut40yNUu2KTU1td42EUUKJf9r7wBPRM0nU6cki5fuBenDD4ecoUlOTiHdBh213CrdEtnZ2aZaUXfcAVGkYJ8rUYSS0wtKv/nll1+OzkTOTibBP9QPXBBFCva5EkUYOYWgnApSzsks/Z91zxkdqeTUkzKCXvpZR48ebfp5iSIVgytRhJHBbvPmzTPTqPyjfDsDGbgmA6tk4Bp/aJ0iHftciYiILGOfKxERUXuUheVMLXJ6NTlvaVufDo2IiFpOipT79+83I7H9Pwxim/yMYeAJZFpCftBDftO4UwdXCayRcEo4IiJqnJwtrVevXq0SWPv164kdO/ZaaU9+JEJOdhKpATas4Or/pQ35UA7k1HBERNS+5AxtkiSF+8tJzSUZqwTWjZveQEpKQovaKiwsQd8+PzJtdurg6i8FS2BlcCUiilyt3bWXkhSHlKQW/hhIE7/+FQk4FYeIiOyRwOi0MDh2guDK0cJERESWMXMlIiJ7mLkaDK5ERGSPnJdIt/DcRJ3g3EYsCxMREVnGzJWIiOxxtIWycORnrgyuRERkD/tcDZaFiYiILGPmSkRE9jBzNRhciYjIHgZXg8GViIjs0RaCq7QR4djnSkREZBkzVyIiskZpx1xa2kakY3AlIiJ72OdqsCxMRERkGTNXIiKyfIYm3fI2IhyDKxER2cOysMGyMBERkWXMXImIyB5mrgaDKxERWf49V6flbUQ4loWJiIgsY+ZKRET2sCxsMLgSEZE9nIpjMLgSEZE9zFwN9rkSERFZxsyViIjs4U/OGQyuRERkjXIcc2lpG5GOZWEiIiLLmLkSEZHlk0jolrcR4RhciYjIHo4WNlgWJiIisoyZKxER2cPM1WBwJSIie3iGJoPBlYiI7GHmarDPlYiIyDJmrkREZLks7LS8jQjH4EpERPZwnqvBsjAREZFlzFyJiMgeDmgyGFyJiMgeKek6LAuzLExERBFt7ty5OPPMM5GdnQ2lFN56662g5VdeeaV5PPBy2mmnteo2MXMlIqKILgsXFxdj5MiR+PGPf4xzzz035DoSTF944YWa+7GxsWhNDK5ERBTRwXXy5Mnm0hgJppmZmWgrLAsTEVGHVFhYGHQpLy8/4LZmz56N7t27Y/DgwbjuuuuwZ88etCYGVyIisn9uYaeFFwA5OTlITU2tuUyfPv2ANklKwv/4xz/w6aef4qGHHsKcOXNMpuvz+dBaWBYmIiJ7tONeWtoGgNzcXKSkpLS4n/Siiy6quT18+HCMGDECAwYMMNnsSSedhNbAzJWIiDpk5pqSkhJ0sTUIqX///ujWrRvWrVuH1sLgSkREB5UtW7aYPtesrKxWew2WhYmIKKJHCxcVFQVloRs2bMDSpUuRnp5uLvfffz/OO+88M1p4/fr1uOOOOzBw4EBMmjQJrYXBlYiIIvrH0r/++mtMnDix5v6tt95qrqdNm4annnoK3377LV588UXk5+ebE02ceuqpePDBB1t1rmuzgqvz4m1w4mPQacRGo7NRCZ3o8xHxrTvRuz2o0x9EZ6PfvxudTWf8nDqrCRMmQDdyysQPP/wQbY2ZKxER2cPfczUYXImIKKLLwh0RRwsTERFZxsyViIgssnASCWkjwjG4EhGRPSwLGywLExERWcbMlYiI7GHmajC4EhFRRJ+hqSNicCUiInuYuRrscyUiIrKMmSsREdnDzNVgcCUiInvY52qwLExERGQZM1ciIrJHfp1Gt7Cs29LndwAMrkREZA/7XA2WhYmIiCxj5kpERPYwczUYXImIyB75RRynhaN9W/yrOu2PZWEiIiLLmLkSEZE9LAsbDK5ERGSPVHSdlgZXRDz7wXXoCVCHjAXSs4HcldCf/K1mkRpzBtBnBJCWCaycA73gP023N/JUqCHjgbgkoCQfevY/gF0bgYy+UGOmAN16S8vA7k1ue/k7rO8SBo+HGnAUkJYNbFsFPfs59/G4JKgjpgLdBwLRcUDRbuhlHwBbVoRux+OFOuk6IK0H4IkGSgugV80G1s6vXT7+cqBrb6ikdDjyOrnfoVUMOAbocwSQkgnkrQbm/6N22djLgK59gagYoKIE2LgIWP1Zw23FpQBjzge69QcqioFVn7rP8et+CHDY6UBSN6A0H/j2HSDve7v702cs0Gs0kNQD2LUWWPJK7bLRFwFdegPeaKCyFMhdDKyf03Bb3QYAg08FEruazwirPgB2r6tdnpgBDD8bSM0CygqBVR8CO1fb3Z/OyubnJOsOOQ1IygB8FcDWpcCaT6TDLrzl1DqYubZScJUAuHQmVPYQIDEtaJEu3AUsegtqyLFhNaWOOAvIHAj9wROAPDcpHfBVuQtjE6C/XwB89jxQVQE1ejLUaTdAv36P/QnIJYXQ330MlTkoeJ+iYqH3bgWWvGPWQa+hUMddAf3+o0BBXv12tAP91X/cZdJhn9oD6pQboeX+zh/cVXZuAFbPBSTItqbSQmD1p+6BQXxq8LJVnwBFuwDHB8SnAeOvBor3AbnfhG7rqEuA4j3Au/e7wXr8NeZAA7t/ABLTgXFXAAtfAXasBjKHAEdfAXzyKFC8197+lO0H1s0BuvYH4ursz7pZQPFud39k2ZFXuEF+27L67cR3AQ6/GFj6BrBzrXtgIPc//wtQug9QHuCIS4Ft3wKLZrgHFKN+BHzxV6DE4v50VrY+JzmgHnMpsOFLYP7f3b/hsT8GSuTv9OswlhNF2oCmjcuATd8C5UX1l61dCGxZCVSUNd1ObAJw2InQc19yA6so2usGBSHt/LAYqCg1/xj1t5+YbM8EYNtyv3UzyPLi4MeL9gArZwElBe7RsGSsBTuBbn1DtyNBP3977Ug4cwyggeRu7n35Ulk9xw20rX2Gkm3LgW0rgPKS+ssKd7jb4t9I2RbJOkOR4Cn7u/wDwFcJ7Mt1g3DfI9zlPQYD+7YCO1a5bcn13lyg9xi7+5O3EshbBVSG2J/9ecH7IxfJSkPJOAQo3A7slMxau9cFW4Beo9zl6X2BmARg3WzAqXKX790I9KxeTm3zOUXHQcnnsEUO+LQbhHevB5J7hLecWo12tJVLpOu4fa4Z/dwvrwFHuJmu/KP7YTH04ncD/gEGyBoILYGiaB/ajZSuU3sA+7Y1upqaeC2QNQjKG+1mvptbqfTbEqOmmrKxioqBlgxzUwNH+1IalQOewIOp/G3AgHHubaXcSyC5n5qJNjXsDFOOVN4YaMletiwJvV7dbXUfBJKrt1e+nPfvDJ4qIAcj/NJu28+pshRaysY5Y4D1c93MVMr5K94Jbzm1Hp7+sIMH17gEqJh4IDUD+l8PmExWTboOqCwHls4MXjexC9SxF0Mv/G/7zY+S/tLjrgA2LXUzs0boWX83X+I6oz/QY6DbH9TRLH0LWPo2dFpPIHuo2wcWSlQsUFmnEiHryuMiby0wfAqQPQzYvgrIOhTo2sctGbelFe8CK96DloOB7kPqb7OfZDdDJgE9DgV2rgG6D3b77iQ7FbJfVY3sL7XN5yS2LweGTwUGToDyeKE3LnD7ccNdTnRQznOVICqBaPF7QFW56fPTy2dB9R4evF5CGtTpN0GvnAt8Xz0wqD0C6/FXAVWV0AteD//IbOd6qPgkYNiJ6JikjL3Fff+HnxF6FVkmg7kCRce7jwvpu134MnDoycAZ9wB9jwS2LAtdjm51GijY5m7boZNCryJ9ft+8AQycCJx0J9DrcPdLWgZ2CXlu3UAq++/fX2qbzymxGzDmEnew2YcPQH/6kDtwSQaihbOcWn9Ak9PCS4TruJnrnq1NryOBdcrPodd9BSz7EO0XWK8013r2s6FL1k09PzmjY49flEE8DfW5FmwH4lOA2MTaPum0LKAgYNT29pXuxW/ijcCmxWg3Hi+Q0EBfnpCRv4Gjf4/5CbBlaW2/4MAJ7nvir5KkyP423hVAlj8nKcPLSO0d1SPzpVtC+lcHHAes+ajp5dR6OFq4lTJX+dLxRrnX0n8lt+UfSb1lAbdDKdoDvXWVGQVshuYnpEINmwC9+Vt3udyf8nN3UNM371vfjXr75JH9kG1V1be95nETWKVfUqbNNBVYu/Q0fa1mf6TNnkOBfmOgtwV8kUu70n7g64bsB7S0T+b999+WL7M0IPswwBvj7mt6H2DgeCBvTeh2pD9290ZgWPXn1CUHyBkNbPyqdp20Xu7rSMY35GR3QFBDfbg29kfeL//+yKjTzKG1+5OWA/Q9OnhqTV2p2dV/nzFuII1OALZWj5SW8rCUgQee4H5WMgBKBjnJNA9qu8+pYCsQl+yW72V9+ZuSQWUyGC2c5UStTGnddM9xYWEhUlNTse/P1yAlPqbxBg8/HerwKUGP6e3fQ7/3ONTxl0MNOjp42fcLoOf+033ueb+GXvohsP6r2nmk4y8Bsge7fS/rFkF//Y6bMYw+HZ4xU6Cry8c17c18EshbH97ex0aHtZoacRrUyNOCX2fHOjOn1TPpZ9BVFUEd8Hr5x8BymU8HqDPvhJbbGxYD6TlQYy8AUrq7pa+ivdDffwmsnVf7Wufc4456DuB8+Qrww6LwtjWh8c+nxqGnQA09JXifdq0Hvn4dOPJid0qNfPnJ0f+mJcCaWbXzA0+5FVg9q3ZqTtA81xJ3Kk/gPFeZmpMu85G12wcr81xl/mg44sPsyzxkItQhweV1vWcDsOw/wKjzqwccKaB8P7B1mTvIxb8/x/3MnU8p02vEUdOA1F61fbCr3nffBz8pLx4m81yzq+e5zmzWPFd1+oPobPT7d7f95yR9sodMBBLS3cGP8lmtfL92JHJTyw+yz8n/PV5QUICUlJRWa3/fI1c1GSeabKu0Al1+8UKrbWtEBteIEmZwjSRhB9dIEW5wjSCd7Uu7WcE1gnS2z6nNgutDV9oJrnfOiOjg2nEHNBEREUWojjugiYiIIo4UQ3ULBySFUVDt8BhciYjIHo4WNhhciYjIHgZXg32uREQU0ebOnYszzzwT2dnZUErhrbfeqldmvueee5CVlYX4+HicfPLJWLu2dc/WxeBKREQRfYam4uJijBw5Ek8++WTI5Q8//DD+/Oc/4+mnn8bChQuRmJiISZMmoawsjB+ROUAsCxMRUUSfuH/y5MnmEropjcceewy//vWvcfbZZ5vH/vGPf6BHjx4mw73ooovQGpi5EhFRh1RYWBh0KS9v/jm8N2zYgB07dphSsJ/Mxx07dizmz2+989EzuBIRkTVyAj1t4SJycnJMIPRfpk+f3uztkcAqJFMNJPf9y1oDy8JERNQhRwvn5uYGnaEpNjZyztjGzJWIiDqklJSUoMuBBNfMzExznZeXF/S43Pcvaw0MrkRE1Gl/z7Vfv34miH766ac1j0n/rYwaHjduHFoLy8JERGRNYJ/pgWru84uKirBu3bqgQUxLly5Feno6evfujZtvvhm/+c1vcMghh5hge/fdd5s5sVOnTkVrYXAlIqKI9vXXX2PixIk192+99VZzPW3aNMyYMQN33HGHmQv7k5/8BPn5+Rg/fjxmzpyJuLi4VtsmBlciIrJHWyjrNnOe64QJExo92b+ctemBBx4wl7bC4EpERPZISdex0EaEY3AlIiJr5OfmdEt/co4n7iciIqK6mLkSEZE9LAsbDK5ERGSPVHS1hTYiHMvCRERE7Zm5eqb9EZ6A8zxGOufZG9DZqMsfQ2ei5/wOnY1e9Cg6ncT49t4C6iA4oMnFsjAREdnDPleDZWEiIiLLmLkSEVFEn1u4I2JwJSIie1gWNlgWJiIisoyZKxERWcOysIvBlYiI7JFZNI6FNiIcgysREVkjv/ym2/YX5zok9rkSERFZxsyViIisYZ+ri8GViIjs4VQcg2VhIiIiy5i5EhGRNSwLuxhciYjIGo4WdrEsTEREZBkzVyIissdR7qWlbUQ4BlciIrKGfa4uloWJiIgsY+ZKRETWaK3MpaVtRDoGVyIisoZlYReDKxER2Z2K47S8jUjH4BqOoSdAHTIWSM8GcldCf/K3mkVqzBlAnxFAWiawcg70gv803d7IU6GGjAfikoCSfOjZ/wB2bXSXpfWAOu5SoGsOULwPeuGbwObvWnHnOonsMUDmCCAxA9i7HlgR4nOITgSO/AlQXggsfi50O6k5wPALgx/zRANbvwbWf1z7WM44IHs0EJ0AlO8HVr8D7N9md5+6jwIyhgLx3YCCjcDa/9UuG3IBkJQV/C327QtAZXHD7WUcBmQeAcQkA1UlwKbZQP762vem36lAci+gqhTYthDY9V1kf04xScCg04G03kBlKbDpS2DHUvv7RBQCg2s4JAAunQmVPQRITAtapAt3AYveghpybFhNqSPOAjIHQn/wBCDPTUoHfFXVCz1Qp1wHrP8K+v0/A9lDoE78MfSb0911qWEVRe6XZ5e+QGxy6HUOORUoygOi4xtupyAX+OIPwV/0R98I7FpZ+1i/E4DU3sCyV4GyfUBsCqB9sK6yCNi6EEjt4waKunI/B/K+Ca+tjOFA5uHA+veAkl1AVALgja5dPmAKUJ4PfPOUG8wHn+vu2/4tiNjP6dCpQOk+YN7jbjAfcRFQuhco2Gxxh6gu9rm6OFo4HBuXAZu+BcqL6i9buxDYshKoKGu6ndgE4LAToee+VBssi/YCpYXu7axDgLhE6G8+cANu7nJg+1qogUdZ3qFOaPcaYM/3boYSStdDgKh4IK+Z2VjmcPcLunCrez8qDug1Fljznht8hGRYFY1kjAdq3zo3s5RMskUU0PMYN1OVwCokcy0vcG/HpgLJUpX5AnCqgOIdwJ7VQMYwROznFJcGpPYCNswGnEq3qpC3ws2aqXU5CrqFF85zpebJ6Od+eQ04ws10HR/ww2Loxe+6t6XsvG97cKlv7xb3cTpw3lhgwMnAd68BKb2a99zMkcD2gOwwpaf7GXYfCmSNdjPWnauAjXPafhRG9tFAz3FucN+xGNizKvR6cV2gYhKhE7sD/U42FRLkbwQ2zwGcCiAhwy0nS8D1kyDcfSQi9nOSfZUsObBMLtlw9uH2tpeoEQyubSkuASomHkjNgP7XAyaTVZOuAyrLgaUzgahYoKIkuDxSXgoVHddum9wp9D8R2PGtm9k050tb+vUkA8pbXvtYVBxUVBx0fDqw6Gm3dHnYjwBfBbD5S7SZLV8ApXvcQJ+SAww4w83QJNutS7JtkdIbWPFybRm4zwRgw0duX2VVefBzqsoAbwwi9nOSbQ+1T1FtvE8HIZ5b2MWycFuSICp/OIvfc//hy4Cl5bOgeg93l8tjEnwDqJg4oDKMkjM1/MUr5cHc+c1/rmRDe9YClQEHPL5K93rjXDeYSda49Su3nNmWira7AV2y5YJNwK5vgfRBodeV7RTbF7kBRi5yO61/7XLJGgPJgZ60H7GfU0Xofapqw306yPtcdQsvkY6Za1vaU90f1JC924BRk92ynb/E2LUXsDu3TTavU0rr62Y1425y7yuvO5DnmJuBr//ecF+pZD4ZQ4AV/w1+vDgPHVJjh/ql+6Alw22IlIBjEt2+Tn//bkJ3oHQ3Ivdz2gnEJrmjuf1BN6kHUMyBgdQ2mLmGQ4KdN8q9Vsq97fGGWBZwO5SiPdBbV0GNnux+cSSkQg2bAL35W3f59rVAeQnU6NMATxTQaxiQNQh63aK229eIpdwvZPPeB9zestAt3379nHuRjLNkj3u7Tgk+SPdh7qCbfT8EP15WAL1vA9BnvPsZySjenke4g3Raa5/kOnCfJCNL7ee+vjwuZeHuI4C9a0M3o6uA3auArCPd58pFbu+rnoYjA5tkwE+v6n1KzAS6DgF2LY/gzykfKNgC9Jvg7lNylrvujmWtsE8UqKWDmbR/UFMz3HfffVBKBV2GDBmC9sTMNQwS7NThU2rvX/U49Pbvod973MxJVYOOrl0mwfL7BdBz/+neP+/X0Es/NNNrhJ41A2r8JVCX/t4t90rgXFY9L0870B8/DXXcJVAjTgGK8836nIYThj7jofoeV3v/+Duh8zcBy14OLm9KSVSqAhX7ax874lpg8zxg54rgUqP0/4Wy6m1g0GRg3M/dtqWvL3eB/X3qeTSUDFjyO/Ln0IW5wLp3zTLEn+4+LqVpGZy0LyC4DjoH2L/VLf+KzbOAPicBI692B2FJMNo8u3b99e8D/U4BRl/nvkcyzcf2NJw2/5zeAgZPcbNf+bf2wyxOw+nEfa7Dhg3DJ598UnM/Kqp9w5vSuundKCwsRGpqKgoKCpCSkoLOwnn2BnQ2nmueRGei5/wOnU58JxygVtr5xgWoE36FzqS1v8f97a896zIkR7ds4Nj+ygoc8r+Xwt5WyVzfeustLF3acU4SwrIwERF1yAFNhYWFQZfy8jojwAOsXbsW2dnZ6N+/Py699FJs3ty+VQoGVyIissZxlJWLyMnJMdmw/zJ9+vSQrzl27FjMmDEDM2fOxFNPPYUNGzbguOOOw/79Ad0KbYx9rkRE1CH7XHNzc4PKwrGxdaZXVZs8eXLN7REjRphg26dPH7zxxhu4+uqr0R4YXImIqENKSUk5oP7htLQ0DBo0COvWhTipShthWZiIiDrVSSSKioqwfv16ZGVlob0wuBIRUUQH19tvvx1z5szBxo0bMW/ePJxzzjnwer24+OKL0V5YFiYiooi2ZcsWE0j37NmDjIwMjB8/HgsWLDC32wuDKxERWeNoZS4tbaM5XnvtNXQ0DK5ERGTNgZy+sK6WPr8jYJ8rERGRZcxciYjIGv6eq4vBlYiIrHFgoc/V/BJUZGNZmIiIyDJmrkREZI2Nk0DoFj6/I2BwJSIiayQwOgyuDK5ERGQPM1cX+1yJiIgsY+ZKRETWONWXlmjp8zsCBlciIrKGZWEXy8JERESWHdSZqy6tbO9NoKYkxqOz0YkJ6Gw8R93a3ptAHYSjm3/i/VBtRLqDOrgSEZFdLAu7WBYmIiKyjJkrERFZLgujxW1EOgZXIiKyhmVhF8vCREREljFzJSIiuz85B/7kHIMrERFZwx9LdzG4EhGRNTLH1WnxPNfIz1zZ50pERGQZM1ciIrJGW+hz1exzJSIiqsU+VxfLwkRERJYxcyUiIms4oMnF4EpERNZIf6lmnyvLwkRERLYxcyUiImt44n4XgysREVnDPlcXy8JERESWMXMlIiJrOKDJxeBKRETWsM/VxeBKRETWMHN1sc+ViIjIMmauYVAjJkINGQd06wlsWgHnvb/WLoyOg5p4KVS/EUBVJfS3s6C/eq/hxjJ6w3P8RW5bpUXQi96BXr3AXZbWHZ5jzgMy+wNR0cCebXDm/QfYvr71dzLSZYwEug0D4rsCBRuB9e/ULht8PpCYBWin9rHlM4DK4obb63YYkDkGiE4GqkqA3NlA/g+A8gKDzgHiugIeL1BRDOQtAXZ/Z3+fugyDSh0MxKYDxZuht3wUvDxtCFT6SCA6Eagqg877EijaFLIpNeASICre5ASGdqC/n1G7QmJPqO5jgehUoKoYOm8+UJxrf5+o02NZ2MXgGgZdnA/99ftQOYdCJXUJWqZOuAgqLhHOC3cBCcnwTL0V2L+nNmAGiomH56yboBf+D/q/nwPd+8Jz9s3QBbuB7euA2AToTcuhP/snUF4MNfRYeM68Cc4//g8oK2q7HY5EEii3LwRSegPRSfWXb/kC2PlNeG11Gw70GA2sfx8o3QVEJQCeaHeZBOjNs4DSvW6gikt3g3fZXqBoq919qiqB3r0EKrGnG0ADpR0KlT4ceusnQPkewBsPeBr/56y3fgoUbay/IDoZqtckt62izUBSb6hep0D/8C+gcr/dfaJOr72m4jz55JN45JFHsGPHDowcORJPPPEEjjrqKLQXloXDsf4b4IelJtMMEhUDNehIOAveBipKgfyd0Ms+gxo6PnQ7WQMAXxX08rnuzz7kbYBevwRqWPX6eRuhV3zuBlKtoVd84X6ZS5ZLjctfB+SvB6pKW9iQAnqOczNVCaxCMteKgurlGijdU5sB+sWmwrr9G9xg6Curt40q4wjovHluYBW+0gMPhEk5QNluN7AKuS7dCZU6qGXbT9RGXn/9ddx666249957sWTJEhNcJ02ahJ07d6K9MHNtibQeUN5oYFdt+UzvzoU6YnLo9VWIozGloLr2qvtV7eoqGUscsHe7vW0+WGWNBbKPBioK3TLunlWh14vrAhWdCJ3QA+hzMqA8bpk5dy7gVNSuN/BskyUrTxR0yS43sLeVmFSoqATouG5Qmce7f1dFudA75wNOZYNPU1nHATjevAd69+KAsm+oLEEBsV1bbReo85LvMm2hDVFYWBj0eGxsrLnU9eijj+Laa6/FVVddZe4//fTTeO+99/D888/jrrvuQntg5toSMbHQFWXBfXnlJUBMXOj1d/wARMeaPlzTX5c1AGrA6NDrSwl50rXQi98HSoL/wKiZtnwJLH8eWPaMWx7OmQikDQi9blT1ZyHl5VWvACtfcrPSnBOC11v3NrDkL9Cr3wD2rQWcKrQZr7uNUi7WG/8LveE/QEwyVI9jGnyK3vYZ9LpXode9BL1vOVSvU4G4DHdh8Rb3dlJfN6jKdUImIAeORAfyY+m6ZRf/aOGcnBykpqbWXKZPn17v9SoqKrB48WKcfPLJNY95PB5zf/78+WgvzFxboqIciI5xsxt/gI1NACTghlJWDOfdv8Bz7PlQR50J7NsOvXIelAxgqhtYpS92+zrohQEDc+jAFAdk/oWb3MFH6YNDZ5u+6sxv+yIzSKjmdv/TgXpjhbTbz5o+yB38JOu1hersVO9eWlMyltuq50kNP6d0R+3twnVAcl+o5H7QZbtMyVv6W6XUjOwTgJI8oHC9+3dN1I5yc3ORkpJScz9U1rp79274fD706NEj6HG5v3r1arQXBteWyM8DHB/QrRewy+2vUnJ7TyMDW7avh/Pvh2ruqtOuhd76fZ3A+nPovdugZ73Uqpt/0JL+7oaU7YVubhYqI4hjgwe6taqK/OZvY1PvQdEm6ICRxqrvVOiCgL9LojBJmuFYaENIYA0MrpGEh6bhkCN4b5TUGtyymbntBaoqoNd+Dc/RZ5ugiNTuUCNOhF75RcNtdctxR3V6o81AJtVzMPSyT9xl0XHuaOL8POhP/9Fmu9c5KDfIyZ+0qr5tPrdYILVv9UhaBSTnABnD3VJuKNrn9sdmHuk+Vy5y25/lxme4JWPzWgpI7QekDwEKN7bePpkMMmD/ZBsL1kJ1HQl4YszF3N7fwDZEJQHxWdXteIDk/iZz1YHrx3VzX0NGRXc73C095zO4UvNpKevqll/C1a1bN3i9XuTl5QU9LvczMzPRXpi5hkEdOQWesWfW3Pde/1foLWvgvPlH6NmvAideBs9VD7nBVua5BkzDMcFy21rorz9w2xp5otvPKl90O34wbaDYHYkqjysZUdytF1T/0TVtSAarv2+jkmOkyh4LlT2u9v6Ym6D35wLr3wOyjgb6p7uPlxe6g5MCg+shU4H9W4EdX7n3ZaRw7xOB4T92A5nMb82d4y6Tz63nsWbgkxl1IQOkZNneNdZ3SXU73C3V+u8PuQa6eBv05nfMSGGVOR5q4CXuNu7f5A5o8q/b/wLo3d+4JWBPFFTmMWYglOm+qC4Do6x2JKXKGAvEd3dL3cVboTe9A+g27EcmOkAxMTEYM2YMPv30U0ydOtU85jiOuX/jjTeivSitG6uR1Y7Yks7kgoKCiE3RQ/E98RN0Nt6f/Q2dif76T+hsdGICOhvPoT9t702gdv4e97f/yuG3I0EqPi1Q4ivHJUv+EPa2ylScadOm4ZlnnjFzWx977DG88cYbps+1bl9sW2HmSkREEX2GpgsvvBC7du3CPffcY04iMWrUKMycObPdAqtgcCUioog/cf+NN97YrmXgujigiYiIyDJmrkREZA1P3O9icCUiImv4e64uloWJiIgsY+ZKRETWsCzsYnAlIiJrGFxdLAsTERFZxsyViIis4YAmF4MrERFZIyfUdVpY1m36pLwdH8vCREREljFzJSKiDvl7rpGMwZWIiKxp7u+xhtLS53cEDK5ERGQNM1cX+1yJiIgsY+ZKRETW8CQSLgZXIiKyRuKittBGpGNZmIiIyDJmrkREZLksrFrcRqQ7qIOrLvOhs6l88Ep0JtF3z0Bnoze/hM7GV/A/dDY6KQWdSZWvuE1eh2VhF8vCRERElh3UmSsREdnF0cIuBlciIrKGJ5FwsSxMRERkGTNXIiKyRn4uTvMn5xhciYjIHvmhc4c/ls7gSkRE9jBzdbHPlYiIyDJmrkREZA1HC7sYXImIyBrOc3WxLExERGQZM1ciIrKG5xZ2MbgSEZE1LAu7WBYmIiKyjJkrERFZw3muLgZXIiKyhlNxXCwLExHRQaNv375QSgVdfv/731t/HWauRER0UA1oeuCBB3DttdfW3E9OTrb+GgyuRETUIafiFBYWBj0eGxtrLi0lwTQzMxOtiWVhIiKynrk6LbyInJwcpKam1lymT59uZRulDNy1a1eMHj0ajzzyCKqqqmAbM1ciIuqQcnNzkZKSUnPfRtZ600034fDDD0d6ejrmzZuHX/7yl9i+fTseffRR2MTgGgY1+kR4DjsW6NYTesNyOG/9pXZhTBw8p1wONWAkUFUB55vPoOe/G7qhhGR4Jl4ElTMIiIkH8nfB+fIt6PXLgl9v7OnwjDwBiE8GivbB9/6zwPYNrbuTXTLgPe1yqJ4DgMoKOIs+gjP/gwZXV6OOh/eYyUByOlCyH74PX4b+/hvAGwXvJbdDZWQDUdHA/nz4FsyE/mZO625/Z5Q0CCpxABCTBpRug95d/R56E6CyzgxeV3mr15ndcHuJA6FShprnwymD3vc1ULoleJ3oVKjM04NfzyIV0xcqujfgTQaqdsIp+ap2oScJnvjhgDcV0A501Q7o0hUAfKEba3L9KKj4EVDRPdzlFRugy7+3vUfwqEOgVBd58wBUwNGbofWO6uVeeNQgKNXVjIF19FZovamR9ppav7nttT35LVZt6fdcJbAGBteG3HXXXXjooYcaXWfVqlUYMmQIbr311prHRowYgZiYGPz0pz81WbGN4O3H4BqOonw489+F6nOoG0wCeE66BIhPhO+ZXwAJKfD+6DY4hXugV8yv3050LPTOzXDm/Nu0qQaMgOeMn8L30oPAnu1ue8edC9VrEHxv/BHI3wmkdAV89ksWQZRC1IU3w1mzBL7XHzeBNurSX0Dv3we9fEH91UefAO/Rk1D136eAHZuBxBSzb4bjg+/Dl4Bd28wXGrplI+ryO+HbvR061/YXWyfnK4Uu/A4qLssNiDWPl0BveT1gRQ9Uz3OhSzY2EVgPhd79OVC5D/DEAar+P3+VfjRQvgutRTvlJsCpqAwo2YYAnoQx0FV7oYsXACoanoSxQOwg6PJVIdtqan0VPxxKxcAp/BjwxMKTOA5wSqAr6xxQtIgEgQr4HDlALpNwAK9nOBxdDo19JvDKtvkc+XcUDa9nJByUQeu80PvUxPrNba89aAsDknQz17/ttttw5ZVXNrpO//79Qz4+duxYUxbeuHEjBg8eDFsYXMOg1y4x16p7TnBwjYqBGnIUfK9MB8pLzcVZ8ik8w4+DL1RwLdgN/dWHte1KxrpvB1TWAGgJrnGJUEecCt+Me93AKgr3tP4Ods0CumbCmfOWCY7YswPO0rnwjJ4AX93gqhS8E86F7+2/uYFVFBcGz/7euaX+P5P07gCDa/OU5rrXMenBwbWuhF7ul3xJ9edRj4JKGwm9Z54bWIUjgaCO5CFAZYEJ3oiWTKwVVLkHkSbbRHBwhScBuvJb929GV5hMVHm7NPxF2+j6XqjobDjFX8iLAk4VdPkGqJjeloOrZI+BBzWF0MiHUqnQugBKdYfP+cbdBlSZTNOjsuALGQw9Tazf1PKDV0ZGhrkciKVLl8Lj8aB79+5Wt4nBtSXSM6Gk9Lmz+ktQyO2jp4T3/IRkID0Lepf7fJXVH/BVQh061i0L+6qgV38F54s33aDXWpQKvq6+rXrIl3YdXbOgkqR02BfeKVcBHg/0uu/g+/hVoKL2C9t74c1Q/YeZ90fnbYZe7R6gkH0qcSBQsqHhqfdRKVDeeOiYdKj0sW4gLtsGvW8JoCvddbyJUMlDoHe8b67bgy5fDxWdA+0rMNmZisqCrtx0YOt7EqGkVO6rPfDTTiGU95BW3gsPFFLgaDk4ToBSMma0KGCjiwDVu4HnNrV+c9trHx15Ks78+fOxcOFCTJw40YwYlvu33HILLrvsMnTpYveAksG1JaTMKwFFyp/VdHmJ6YdtksdrSsJ6zVdAXvUXQnwiVGwCVJfu8D37K5PJes+7CaqyrOF+XBv27ADyd8Mz4Rw4s980WaZn5HFAbHy9VVV8onvdbyiqnr3P3Paeex28p14C37vP16zne/0xN0DnDILqM9j0R1Mr8CYCcZnQOxo5ePHGmCtl1nP70VW38VBdxkDvdSsTEnR1/jLAab/PSVfthCd+FFTK6SaI6Mrt0BWbD2x9FQWtJbsL+JY2BxKt+5XnUYOhUQINKa1L9uoL2gZtMs6GtsHbxPpNLe8YOvKv4sTGxuK1117Dfffdh/LycvTr188E18B+WFs61qcSaSrLgegYQI4mqwOskoAUkME1GFjPvt4dAPXhi7WPV5SbK+fLt922K8vhLP7EZLG+1gyujg9VbzwO7ymXIOrmPwGF++As+wKewyfUW1XXbOO7QGlRzW0JsPVX1tCb10ANPQqecZPhfPFO6+3DQUolDQAq9gGV+Q2v5Lh99rpgBeCU19yWAGsk9HOrFib7bS/Rpk9Ul62GrthYPRhpOFT84dCli5u/vgms3uo+UR3wddd64xfcgU3x1f2vQgKhJ2gbVKPb0NT6zW2P6pJRwgsW1B9H0hoYXFti7w7A5wOkL9affXbvDeza2nhgPes6wOuF8+Zfgsq9/vJwu9i1Db5X/lBz13PSBSYw1rNnO3Rl87Ib5fUC6T1sbCXVldgfulBGyDaiqhC6OsCGIhktYrpB9Ty/+oEot+rQ8zzorf9BmzB9yl4zotdVaYKmJ/Fo6NIDWN8pdsvknhTAKXB3S/p5fftbMbCmVAdW/7/pkuogmFhbylVJMkihgVaaWr+57bWPjlwWbks8iUQ4JDP1RpnAaI7w/berKkxZ1zN+qju1Jq07PKNPgvPd3MYDa0ysG1jrjgIu2A1n4wp4xp1pBkshMQ2ew0+Cs25p6+9j915uFu7xQg0ZY8rCvs//V3+9qkro7+bDc8wUIC4BiE0wt/UaGWQBoEdvqH7D3Gk4ygM1cCTUYePg/LC89feh01EBmYr/dsA/WRlFLCNuixsZJSyklFiywZ2Go6TSEu3erh4wpfctht7+julvlQuK1gJlee7tNtknBfiKTLYpU3Xcx71QMX0A6U8Npcn1fdCV2+CJk/5j+fcqXS79oCs2tVJgTa0OrIH/ph1ovRMeT7/qLDoeHtUTjq4e1FVPU+s3t732oS39F+mYuYbBM+4MeI49u/b+rc9Ab14N3+uPwPnkZXhOvQLe6/7gzg+Vea4BI4U9590MveV76IXvA9kD4DlktMn8vDc+XrOOs+A9d7ncfu/v8Jw6Dd4b/mRGH+uVC6AXzWz9fZTS7ZgTTVCUAUi+N/5cM+rXe/Gt0Ju/d0vB8rX10cvwTr4CUT/7gxtsv1/qDmgyxyEeeE48D0pGIMvIYTlg+PjVkFN6qHEqdThU6oja+70vgZagt/PjgIFMm2oHJQU+N2MidPlOoDqrlTmtqstRUD2nusG2dIsJqu7CCsAXUI2Q9mQdX6n9fYodBE9c7XQHb+oZ0FW74RTPg1OyCJ64Q6HiDnX/dnx74ZR8EzA4eCy0by90+VoTPJtaX5d+B8SPgCflVLM/Zp6r1ZHCIhYeT09o7cDrGVf72joPjv4ejl4LDwZVL/PPS60d2evxDDejirV2+4qbWr+p5dRxKK2b/uU8Ob+jnHqqoKAgrAm9kaLqkavR2eiyVhxV3A6i756BzsbZ/BI6G53aeb4X/HRS59qnwsJidE0/o9W+x/1x4ud9fonYOnOYm6vcKcPjm6ZHdMxh5kpERAfFaOG2xOBKRETWcECTiwOaiIiILGPmSkRE1sgoHt3Scwt3gsyVwZWIiKxxGj4RZ9ha+vyOgGVhIiIiy5i5EhGRNRzQ5GJwJSIieyz0uaITBFeWhYmIiCxj5kpERNZwQJOLwZWIiKzhVBwXy8JERESWMXMlIiJrWBZ2MbgSEZE18kNruoV13ZY+vyNgcCUiIms4z9XFPlciIiLLmLkSEZE1/D1XF4MrERFZw7Kwi2VhIiIiy5i5EhGRNcxcXQyuRERkuc9Vt7iNSHdQB1dd5kNn49tXhc4kRnW+P1Fn4cPobLSn8/UwRaWehc4kylvY3ptwUOl831xERNRuWBZ2MbgSEZE1PHG/i8GViIiskf5Wp8V9rpEfXTtfRwkREVE7Y+ZKRETWsCzsYnAlIiJr+JNzLpaFiYiILGPmSkRE1vD3XF0MrkREZA3nubpYFiYiooPGb3/7WxxzzDFISEhAWlpayHU2b96MKVOmmHW6d++OX/ziF6iqat7Z75i5EhGRNY6Fea5OK85zraiowAUXXIBx48bhueeeq7fc5/OZwJqZmYl58+Zh+/btuOKKKxAdHY3f/e53Yb8OgysREdk9cb9ueRut5f777zfXM2bMCLn8o48+wsqVK/HJJ5+gR48eGDVqFB588EHceeeduO+++xATExPW67AsTEREHVJhYWHQpby8vNVfc/78+Rg+fLgJrH6TJk0yr79ixYqw22FwJSIi62Vhp4UXkZOTg9TU1JrL9OnTW337d+zYERRYhf++LAsXy8JERGT3DE1oeRsiNzcXKSkpNY/HxsaGXP+uu+7CQw891Gibq1atwpAhQ9BWGFyJiKhDDmhKSUkJCq4Nue2223DllVc2uk7//v3Dem0ZyLRo0aKgx/Ly8mqWhYvBlYiIIlpGRoa52CCjiGW6zs6dO800HPHxxx+bID906NCw22FwJSIiaxxtIXNtxTM0yRzWvXv3mmuZdrN06VLz+MCBA5GUlIRTTz3VBNHLL78cDz/8sOln/fWvf40bbrihwbJ0KAyuRERkjfwWq+7Av+d6zz334MUXX6y5P3r0aHM9a9YsTJgwAV6vF++++y6uu+46k8UmJiZi2rRpeOCBB5r1OgyuRER00JgxY0aDc1z9+vTpg/fff79Fr8PgSkRE1mgLPxmnEfkYXA9Elwx4T7scqucAoLICzqKP4Mz/oMHV1ajj4T1mMpCcDpTsh+/Dl6G//wbwRsF7ye1QGdlAVDSwPx++BTOhv5mDNpfaBdHnToOn/2Dzl+2sW4nK/8wAivfXW9U7/hR4jzwOKisHzqplqHzhsfrrjJ0A78QpUKldTBuVb/4TzoolrboLp59+Ou688xcYPvwwVFZWYu7cz3Hzzbdi69atNeucffZZeOSRh9CzZ08sWfINrrnmJ1izZk2DbTa1fnPba5buo4CMoUB8N6BgI7D2f7XLhlwAJGUBOuBr7NsXgMri0G01tn5UPNB7ApDSC/DGAGUFwNZ5QP4PsE0lDoJK7AdEpwFl2+Ds+dxd4E2Ap8eUOit7q9eZ23B7CQOgkg81z4dTBid/MVBW/XnHZMCTOhqITgF0FXTxBujCZdb3iSLr9IdthcG1uZRC1IU3w1mzBL7XHzeBNurSX0Dv3we9fEH91UefAO/Rk1D136eAHZuBxBQgurpT3PHB9+FLwK5t7pdet2xEXX4nfLu3Q+d+36a7JYFVlD94s9nH6EuvR/Q5V6DypSfrrasL9qHq47fhGTQMKjW93nLv0RPhPeE0VP7zL9BbNwFJKVAx4Q8EOFCpqSl46KFHMGfOHPOTVU888TjeeOM1HHvscWb5oEGD8PLL/8SFF15iTm32q1/9Em+//V8MGzbCDGyoq6n1m9tes1UWAVsXAql9gJik+stzPwfyvgm/vYbWl4BastNdLq+Z1h8YMAVY8TJQthc2aV8JdOEKqLhMKG987QJfCZxt/wpY0wNP1lTokk0NtqUSB0AlDYGz90ugch/giXMDsrsUnq7HQxetgt61yg3eGScBvmLo4nVW94koFJ6hqbm6ZgFdM+HMecsER+zZAWfpXHhGT6i/rlLwTjjXZKomsIriQiB/l3tbRsTt3BKQTVQfraW7w7/bkuraHb5lC4GKcqC8DL6lC6CyeoVc1/nuazjLFwPFRSEaUog67TxUvflPN7CKokLovdX73IpeffU1009SXFyMkpISPPbYnzF27FFmgIK47LJLMWvWbLz33nvmNGoPPvgbM9T+uOPc4FtXU+s3t71m27cOyF8PVJWiVZUXADsWu4FVSMYqQVUyXdvKtrgXp/HT2Kn4XuZvSZfmNrQGVMoIN1OVwCqcMhM83cXRUN5Yk62af1cSVMt2uBkztcnvueoWXiIdM9fmUir4uvq26hEiEHXNgkpKhcrsC++UqwCPB3rdd/B9/CpQUVazmvfCm6H6D4OKiobO2wy9unXLp6H45nwA78ixcFbKsHQF7+hxcFY2IyuqprpnQaWkQfXqh9gfXQ14vPCtXoaqt18Byls5SNRxwgnHm7Oy+LPIESOGY+nS2rKg/ITUypWrzOOzZ8+u9/ym1m9ue9ZlHw30HAeUF7rBcc8qO+tLmTi+K1C6G+1FslJdsrHh3ruoZJP5qph0qC5HmTxBl22DLlhiSsDQFXCK17vt7F8BeBNNtuzkf9XWu3LQYVnYxeDaXHt2APm74ZlwDpzZb5os0zPyOCA2oMRVTcUnutf9hqLq2fvMbe+518F76iXwvft8zXq+1x9zA3TOIKg+g4GqCrQ1Z8P3ppwb+5tnzH29aR0qP3mn+Q0luOVLKRmX/+luczv68hsRNfVSVL3+LNqK+0sW9+OCCy6qeUzmsOXn5wetJ/eTk5NDttHU+s1tz6otXwClewCnCkjJAQacATiVbrbbkvWVBxg4Bdi7Bih2z0rT5qT/NLYHdEEjB3cet5tBxWbC2fmh+1D6sUDaGOh9C819XbIZni5HQaUcBqU8cIrWAGXb22Yf6KDHsnBzOT5UvfE4VI8+iLr5T4ia+v/gLPsCKKlfItVSYpWnfPkuUFpkLnJbDRpVv10phWxeAySmwjNuMtqUUoj5f3eZAFv+y2vMRW7H/L87m99WuZuR+z59xy0bFxeZ296h7lwymy655GLs359vLsuX12aQhx12GD744F3ceONNpi/Ur6ioyJz8O5Dc37+//qCtcNZvbntWFW0HfBVul0LBJmDXt0D6oJatbwLrmW4A3vAx2otkm6bUWxl84BJEV5orR7JSKTE75ea2iuvpLo9Khqfb8XAKlsDZ+jp82/4LFZUKlRri3x512BP3RzJmrgdi1zb4XvlDzV3PSRe4gbGuPduhK5uXhSrpH0wP/kWGVpeQCJWegarPPzKjn0XVFx8h7sQzgMSk0H2rDdC7mr/PB+qVV141l0ASWD/55EPcddev8PLLrwQt+/bb7zBq1Mia+1FRURg69FB8993ykO03tX5z22tVze2jqru+CaxnuAOC1r4dPKq4jamE/m4ptzGV+6Gl/NsQ6Vv1lQD+PlunDLrkB6jkodAF7hl5qHX4w2NLtPT5HQEz1wPRvRcQHWP6E9WQMaYs7Ps8YJqEX1Ul9Hfz4TlmChAnpa4Ec1uvqS539egN1W+YOw1HeaAGjoQ6bBycH9r4y7m4CM6uHYgaf7K7LVHRiDr2FOh9e0IHVo/HXU+uVfXt6kFDqKyEb/GX8Epgjk8w+y23fa08DUfIKcsksP761/dgxozaM7D4vfTSyzjxxImYPHmy+cHj//u/X2H37t2YOzf0VI+m1m9ue82nqke/qtrb8n57Y4HUfoBHjo2VW+btPgLYuzZ0M02t7w+snujqwGphpHNj+2S+dlTA7YCvodgsU/JtbJSwywddvBGe5KFm8JJc5LYu3eIurtgLeOKBuOqxEJ5YqIR+0BXVg5+o1TBzdSkdxrAs+ZFYKXcVFBSE9QsFkaLywcZ/RaEhngnnwjPmRBNUZACS88nr0FvcvivvxbdCb/7eLQWL6Bh4J18BNfhwN9h+v7RmQJPK6gvP6VdAyQhk+RgKdsP5+jM4Sw58MIxvXyNH841QPbIRdfZl8OT0M1+2ztaNqPrfK2bEb9T5V5l1qv79grmOmnSuuQRy1q1CxV9/696JiXXnzA4fI6N8TGCtevvlmpJxc8T/6bWw133++WcxbdoVZqRwoKFDh5ufrhJTp56Nhx/+PXr16mXmpV599bU181LHjx9vysnJybUjShtbP5zloTgLHw5vh3qOg5IBSAF0YS6w7l1g0FQgvnoalBmgtATYHZDtDToH2L8V2L7IHaDU2PrJvaAO/RG0lIMDM9Zti9znh8HJDu/XQlTKcHhShgfvU3kenF2fusul31T7oPfVn9bm6TYBunwn9P6V1Y15odKOdEcWy3NKt9YOaBJxPd3Xikpyl5ftcJc3MVLZz9vrEnQmrf097m9/XMr1iFItm3pXpcsxv/CvER1zGFw7mQMNrh1Vc4JrpAg7uEaQcINrJGFwPbD2x6ZcZyW4Lix8KqJjDvtciYjIGqf6v5Zo6fM7Ava5EhERWcbMlYiIrNFKQ6uWjhaO/AFNDK5ERGSNtjDaV3eC4MqyMBERkWXMXImIyBoZjKQ4oInBlYiI7OEZmlwsCxMREVnGzJWIiKxxlAPVwtHCLAsTEREFYJ+ri8GViIisYXB1sc+ViIjIMmauRERkDUcLuxhciYjIGgc+KPha3EakY1mYiIjIMmauRERkjZwXWLe4LBz55xZmcCUiIms4z9XFsjAREZFlzFyJiMjygCZPi9uIdAyuRERkUcun4kgbkY5lYSIiIssO6sw1+u4Z6Gyi0bnoR19q702gMHjbewOow3C0lHQ9FtqIbAd1cCUiIrt4hiYXgysREVmj4YNuYeYqbUQ69rkSEdFB47e//S2OOeYYJCQkIC0tLeQ6Sql6l9dee61Zr8PMlYiIrHFPAOFYaKN1VFRU4IILLsC4cePw3HPPNbjeCy+8gNNOO63mfkOBuCEMrkREdNCc/vD+++831zNmND6gVYJpZmbmAb8Oy8JERNQhFRYWBl3Ky8vb7LVvuOEGdOvWDUcddRSef/55aN28gM/MlYiIrNFaBjSpFrchcnJygh6/9957cd9996G1PfDAAzjxxBNNv+xHH32E66+/HkVFRbjpppvCboPBlYiIOmSfa25uLlJSUmoej42NDbn+XXfdhYceeqjRNletWoUhQ4aE9fp33313ze3Ro0ejuLgYjzzyCIMrERFFvpSUlKDg2pDbbrsNV155ZaPr9O/f/4C3Y+zYsXjwwQdNWbqhAF8XgysREVme56pa3EZzZGRkmEtrWbp0Kbp06RJ2YBUMrkREZI3WFs7QpFtvKs7mzZuxd+9ec+3z+UzgFAMHDkRSUhLeeecd5OXl4eijj0ZcXBw+/vhj/O53v8Ptt9/erNdhcCUiooPGPffcgxdffDGoT1XMmjULEyZMQHR0NJ588knccsstZoSwBN1HH30U1157bbNeR+kwxhfLEOjU1FQUFBSEVf8mIqKOpbW/x/3td0sZC49qWd7m6CrsLlwY0TGHmSsREXXIqTiRjMGViIgOmjM0tRWeoYmIiMgyZq5ERGR5tLBqcRuRjsGViIgskj7XlrcR6VgWJiIisoyZKxERWeOWdJWFNiIbgysREVnD4OpiWZiIiMgyZq5ERGSN/FycavGJ+yM/c2VwJSIia1gWdrEsTEREZBkzVyIissbGeYE1zy1MRERU97zAjoU2IhuDKxERWWOjv1Szz5WIiIjqYuZKRETWMHN1MbgSEZE1Nuao6k4wz5VlYSIiIsuYuRIRkTUsC7sYXImIyBoGVxfLwkRERJYxcyUiIotsZJ2Rn7kyuBIRkTUsC7tYFiYiIrKMmSsREVnDea4uBlciIrJGawsn7jdtRDYGVyIiskh+Lk61OHeNdOxzJSIisoyZKxERWeOO9FUtbCPyM1cGVyIisqjlwZVlYSIiIqqHmSsREdljoSwMloWJiIhqaQslXc2yMBEREdXFzJWIiCzigCbB4EpERBZpC4N9WRYmIiKiA8lc/RN6CwsLw1mdiIg6GP/3d+ufoEGGI0V+5tkmwXX//v3mOicnp7W3h4iIWpF8n6emplpvNyYmBpmZmdixY4eV9qQtaTNSKR3GYYzjONi2bRuSk5OhVEvPvEFERG1NvuolsGZnZ8PjaZ0ewbKyMlRUVFhpSwJrXFwcOnVwJSIiovBxQBMREZFlDK5ERESWMbgSERFZxuBKRERkGYMrERGRZQyuREREljG4EhERwa7/D2LZ+Kya8v0xAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAGgCAYAAAC0SSBAAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAITBJREFUeJzt3Q10VNW99/FfYl4gkcDlRcFLxFtLoTUWQUqptkC0EYr1JQilhIqutndRKcpLrrUpBZKWUhp6WaULKq14wWub3mrbhxZZRcJrtbZceGgrIRAkj6Kg0hQhIQ4mgcyz9o4TQoKbiYY5c06+n7UOmTk5IfufmTm/OXvvMychHA6HBQDAe0h8r28AAGAQFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERRAB9m+fbsSEhLs14j7779f11xzjaftAj4oggKeW7t2rd3BmuX5559v833zKTOZmZn2+5///Oc9aSPQmREUiBtdunRRSUlJm/U7duzQkSNHlJqaKr957LHHVFFR4XUzgA+EoEDcGD9+vJ5++mmdOXPmvPUmPG688Ub17dtXfpOcnOzLgANaIigQN6ZMmaLjx4+rtLS0eV19fb1+/etfKy8vr013lOn7v+uuu9r8P++88466d++u6dOnO3+f6cqaOXOmfvGLX2jQoEH2iMYE0h//+Mc22/71r3/V5z73OWVkZOjyyy/Xrbfeqr/85S8XrelCYxSNjY1avny5rr/+evs7+/Tpo3Hjxmn37t32+6NHj9aQIUMu+P+Zdo4dO/aivxfoSAQF4obZoX7qU5/SL3/5y+Z1f/jDH1RdXa0vfvGLbXbyX/rSl+z333rrrfO+t379etXU1NjvX4zp1po9e7bd9jvf+Y4NKrPTLisra95m3759+sxnPqO///3v+sY3vqH58+fr5Zdf1pgxY7Rz58521/mVr3zF/k4z7vKDH/xA3/zmN21gRILn3nvv1YsvvnheG4xdu3bp4MGDUdUFdChzPQrAS2vWrDHXRAnv2rUrvGLFinC3bt3CoVDIfm/SpEnh7Oxse3vAgAHh22+/vfnnKioq7M89+uij5/1/d955Z/iaa64JNzY2On+v+Vmz7N69u3nd4cOHw126dAnn5uY2r7v77rvDKSkp4crKyuZ1r7/+um3nqFGjmtdt27bN/n/ma8R9991n2x2xdetWu81DDz3Upj2R9p48edK24ZFHHjnv++Zn0tPTw7W1tc66gI7GEQXiyhe+8AWdPn1azzzzjE6dOmW/tu52ivjIRz6iT37yk7brKMIcXZijjKlTp9qjjosxRzCmuyni6quvtt1Zzz77rM6ePWuXTZs26e6779aHPvSh5u369etn22VmaZmjl2j95je/se1auHBhm+9F2mu6zUwbzJFV5Lpiph2/+tWvbDvS09Oj/n1ARyAoEFdMf/1nP/tZO4D929/+1u4gJ06c+J7bT5s2TX/60590+PBhe98Mhjc0NNjum2gMHDjwggEUCoVUVVVlF3PbjA209tGPftSON7z22mtR11dZWamrrrpKPXv2dG5n6nr11Vf13HPP2fubN2/WsWPHoq4L6EgEBeKOeadujgpWrVplB5B79OjxntuasQszsyhyVPHzn/9cw4cPv+CO3U/MgPWVV15p6zHMVzPry4QoEGsEBeJObm6uEhMT7eDue3U7RZh35rfffrsNCnNUYY4u2vOu+6WXXmqzzgwYp6Wl2aMbs5jbFzoX4sCBA7adZlA6Wtdee61ef/31NgPwrV122WW2djPj68SJE1q3bp2dFWbWA7FGUCDumOmnjz76qAoLC3XHHXdcdHsTDOXl5Xr44YftjrT1DCmXP//5z9qzZ0/zfdON9Lvf/U633Xab/b/MYm6bda+88krzdqYbyHSPffrTn7ZTZqN1zz332HGHoqKiNt+LjEe0rMuEhJnmW1tby2wneCbJu18NvLf77rsv6m3NEUWvXr3s+ITpqrriiiui/tmsrCzbzfPQQw/ZE+N+8pOf2PUtd+SLFi2y53aYUJgxY4aSkpL005/+VHV1dSouLm5XXdnZ2TYAfvzjH9ujGTMV14xzmLEI8z1zXkfE0KFDbftMXWY8ZNiwYe36XUBH4YgCvpeSkqLJkyfb2+0d7DUnt/3oRz/Sk08+qQULFtiuLDM+8vGPf7x5m+uuu87uyM1O+/vf/74NkQEDBmjbtm121lV7rVmzRkuXLrXnYpijoMWLF9uZXjfddNMFB7XfT11AR0owc2Q79H8EPDBnzhw9/vjjevPNN+2YQjTMdNSvf/3rWrFiheKVOYPb1Ga6vczUXcALHFHA98xHdphZQab/P9qQ8APzHs6EnznqISTgJcYo4Fv/+Mc/7PkFZmaQ+eiNWbNmKQjefvtt/f73v7ddW3v37rUD6YCXCAr4lpnpZM7ANoPXZnD4hhtuUBCYk/zM1Fhz/si3vvUt3XnnnV43CZ0cYxQAACfGKAAATgQFAKBjxijMyUVmiTAnCZmPITAnOkXzKZ0AgPhhRh3MJzSbD6k0H0XTIUEROdEIABAc5mNr+vfv3zGD2a2PKMxVx8zcbnNmqfn0ziAwqfqxj33MzqYxR0x+F7R6DGryB2qKf+bj+M0nBJw8edJeA6VDjijM5+Bc6CLxJiTMRygE5YlgTtgy9QThiRC0egxq8gdq8o9ohg4YzAYAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMApKdoN6+rq7BJRU1NjvyYmJtolCCJ1UE/8oiZ/oKb41546EsLhcDiaDQsLC1VUVNRmfUlJidLS0trXQgCAp0KhkPLy8lRdXa2MjIyOCYoLHVFkZmbqjTfeUK9evRQEDQ0NKi0tVU5OjpKTk+V3Qasn6DWVlZWpsbFRQXm3mpWVxeMUx+rr67Vo0aKogiLqrqfU1FS7tGaeBEF5IgS1pqDVE9SazM4nCDuglnic4ld7aghGZxsA4JIhKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIA/g+J/yv5HY38+Vlf+8EolfzdZ3Zd0178t/zeNWTtGs/4wS88eetbrJiLAZs2apb59+3rdDFwEj1NsRH2Fu1ia9n+m6ckXnzxvXU1djV1eOfmKdhzeocPVhzX2w2M9ayOCzVwe8tixY143AxfB49RJg2LjoY3nhcSN/W7U2GvH6vKUy1UVqtKeN/boz0f+7GkbAaAzibug2FS5qfn2h3t+WDu/ulOXJV523jbmyGLvsb0KAnOB87lz5+rhhx/WgAEDFARBrAnozOJujOJM45nm2yffOWm7mlrLSM3QzVffrCDYs2ePVq9erVGjRqmyslJBEMSagM4s7oJiWL9hzbf/GfqnPrLiI7rxZzfqa898TY/938d06K1DCpKRI0dq/fr1qqqqsjvWiooK+V0QawI6s7jrevrSx7+klbtWavfru+39xnCjHZcwS8Snr/60VnxuhYb0HaJ4Nn/+fJ04cSKqbbOysrRr1y6NHj1aO3bs0KBBgxSPglgTAJ8FRVJikrZO26rvP/99/ddf/0vH3m47o+H5V59XzpM52jdjn/qk91G8WrNmjY4ePdqunzEzOMrLy+N2pxrEmgD4rOvJ6JbaTYtvXaw38t9Q2QNlevzOx3XfkPvULaVb8zZmBlTrKbTx5siRIwqHwxddamtrNWbMGPszRUVFys3NVbwKYk0AfBgUEQkJCbruiuv05aFf1tq71+rFB15UYsK5Jr90/CX53alTpzRu3Dht375dS5Ys0YIFC+R3fq3JDMKvWrWqzfr9+/dr+fLlnrQJbfE4xV7cdT098bcn9M6ZdzTl+il2dlNL6cnpNijMuIXRo0sP+d2hQ4e0d+9eLVu2THPmzFEQ+LWmefPmaePGjQqFQs3r9u3bp1tuuUWnT5/WhAkTlJmZ6WkbwePkhbgLipdPvqyiHUWa/exsO2h9w5U3qGfXnjp++rh+Xf7r86bPjvvwOPnd0KFD7Y61d+/eCgq/1vTUU09p/Pjxys/PV58+TWNf2dnZ9ryQTZs2sfOJEzxOsRd3QRFhjio2/7/NdrmQfx/27xp9zWgFgd92qEGtqVu3bvad6h133KFt27bZdWfOnNHmzZs1fPhwr5uHd/E4xV7cBcXskbN1/RXXa+vLW7X7jd16s/ZNVb1dpbPhs+qT1kc3XnWjHdie8NEJXjcVAZSenq4NGzborrvusn3hpaWl9ggJ8YXHqZMHhRl3uOdj99gF8ELXrl1tFwbiG49T7MT1rCcAgPcICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAEDHXAq1rq7OLhE1NTX2a0NDg12CIFIH9cSvINeUmBic922RWnic4ld76kgIh8PhaDYsLCxUUVFRm/UlJSVKS0trXwsBAJ4KhULKy8tTdXW1MjIyOuaIoqCgQHPnzj3viCIzM1Pl5eVKSUlRUBI2KytLOTk5Sk5OVhDeAZWWlqqsrEyNjY0KgqA9Ri0fJ2qKbw0Bez3V19dHvW3UQZGammqX1swfLAh/tJbMEzsoT26Dx8gfqMkfGgPyempPDcHobAMAXDIEBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKGLszJkzmjp1qgYPHqyDBw963Ry8h1mzZqlv374KkiDWhNggKGJ86cFJkybZ64xXVFRozJgxOnDggNfNwgWY6wgfO3ZMQRLEmhAbBEWM1NXVacKECVq3bl3zhcxra2ttWOzbt8/r5gHAeyIoYmTy5MnasGGDCgoKlJuba9dt2rRJp0+fVnZ2to4cOeJ1EwHfXOs5Pz9f5eXlXjel00jyugGdxezZs/WJT3xC8+bN0/3332/XjRw5UqWlpVq/fr369+/vdROBuHf27FlNmzbNdt8mJiZq6dKlXjepUyAoYsR0MZmltREjRtgFwMUnguTl5enpp5/WjBkzVFxc7HWTOg2CAoAvmIkgZoyve/fuSkhI0IMPPhj1bK+BAwde8vYFGUEBwBfjEtu3b2+evbVy5cqof3bixIkExQfEYDaAuGfGI7Zs2aKePXuqR48e2rlzp8LhcFTLhbp80T4EBfCuPXv2aNWqVW3W79+/X8uXL5cfBammYcOGaevWrUpKSlJOTo5eeOEFr5vUadD1BLzLzEjbuHGjQqFQ8zpzjsstt9xipzGb82AyMzPlJ0GraciQIdq2bZtuvfVWe4Rx0003ed2kToGgAN711FNPafz48XaOfp8+few6c46LOaPenPPipx1qkGvKysqyYde7d2+vm9Jp0PUEvKtbt2723bfZkVZVVTVPydy8ebM958WPgliTQUjEFkEBtJCenm7PoDd94L169bLdG8OHD5efBbEmxBZdTx5Yu3atXRCfunbtartlgiSINSF2OKIAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAICOuRRqXV2dXSJqamrs18TERLsEQaSOhoYGBUGkjqA8PkF8jFrWEsSaiouL1djYqKA897KysgLzempPHQnhcDgczYaFhYUqKipqs76kpERpaWntayEAwFOhUEh5eXmqrq5WRkZGxxxRFBQUaO7cuecdUWRmZqq8vFwpKSkK0juGnJwcJScnKwjv6kpLS1VWVha4d3VBeYxaPk5BrCmIz72ygNRUX18f9bZRB0VqaqpdWjN/sCD80VoyL9agvGANHiN/CGJNQXzuNQakpvbUEIzONgDAJUNQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCIsbOnDmjqVOnavDgwTp48KDXzUEnMmvWLPXt29frZsCHCIoYX3pw0qRJ9jrjFRUVGjNmjA4cOOB1s9BJmGsjHzt2zOtmwIcIihipq6vThAkTtG7duuYLmdfW1tqw2Ldvn9fNA4D3RFDEyOTJk7VhwwYVFBQoNzfXrtu0aZNOnz6t7OxsHTlyxOsm4iJHgzNnztThw4e9bgoQcwRFjMyePVuLFi3S4sWLm9eNHDlSpaWlmj59uvr37+9p++C2Z88erV69WqNGjVJlZaXXzQFiKim2v67zMl1MZmltxIgRdkF8M6G+fv163XXXXTYstm7dqkGDBnndLCAmCAp0evPnz9eJEyei2jYrK0u7du3S6NGjtWPHDsICnQJBgU5vzZo1Onr0aLt+xsweKi8vJyjQKTBGgU7PTCQIh8MXXSKz1IyioqLmSQlA0BEUQBROnTqlcePGafv27VqyZIkWLFggvwzCr1q1qs36/fv3a/ny5Z60Cf5D1xMQhUOHDmnv3r1atmyZ5syZI7+YN2+eNm7cqFAo1LzOnLdzyy232KnZ5tyezMxMT9uI+EdQAFEYOnSoDYvevXvLT5566imNHz9e+fn56tOnj11nztsx54WY83gICUSDricgSn4LCaNbt272iMKEQ1VVVfPnjW3evNlO+QWiQVAAAZeenm4/FSAnJ0e9evXSli1bNHz4cK+bBR8hKDywdu1aO4sGiJWuXbvarqZ//vOfthsNaA+CAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnJIUpbq6OrtE1NTU2K+JiYl2CYJIHQ0NDQqCSB35+flKTk5WUGoqLS1VcXGxGhsbFZTnXVZWVmCed0F/7uUHpKbjx49r8eLFUW2bEI7y4s2FhYUqKipqs76kpERpaWntbyUAwDOhUEh5eXmqrq5WRkZGxwTFhY4oMjMz9e1vf1spKSkK0ju7nJycQLxjiLwDCko9LWsqKysL3BFFEB8naorvI4p+/fpFFRRRdz2lpqbapTXzYg3KCzbCPAmC8EQIaj0Gzzt/oKb41Z4agjG4AAC4ZAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAngiLGzpw5o6lTp2rw4ME6ePCg180BEAeOH5eKi6XbbpOuukrq0sVcA0jq108aNUp6+GHpueek6C4z1/GivnARPrj6+npNnjxZ69ats/fHjBmjrVu32tAA0Dn97GfS3LnS22+3/d6bbzYtJiR++EPpjTekvn1j30aCIkbMZWTvuecebdiwwV520FxKtra21obFli1bdN1113ndRAAxtnSp9I1vnLufkCBlZ0sjR0qXXy699Zb0t79Jzz8vvfOOd+0kKGLEHEmYkCgoKNDrr7+uJ554Qps2bdLYsWOVnZ2tPXv2qH///l43E0CM7N8vFRScu9+rl/T730s33dR229pa6cknpa5d5QnGKGJk9uzZWrRokRYvXty8buTIkfZi7dOnT/dlSJhrVufn56u8vNzrpuB9dIPOnDlThw8fVlD4raYf/1g6e/bc/VWrLhwShjm6eOABqXt3eYKgiBHTxTRv3rw260eMGKHvfve78puzZ8/q3nvv1bJly7RmzRqvm4N2Mkewq1ev1qhRo1RZWakg8FtNW7acu/0v/yJNmKC4RVDgfc3cmjJlikpKSjRjxgwVm+ka8BVzNLt+/XpVVVXZHWtFRYX8zm81HT167vbAgVJii73xgQNN4xWtl/vv96SpjFGg/SZNmmRnbnXv3l0JCQl68MEHo/q5WbNmaaB5ReCSmj9/vk6cOBHVtllZWdq1a5dGjx6tHTt2aNCgQYpHQaypJRMC8YygQLvHJbZv325vV1dXa+XKlVH/7MSJEwmKGDBdgUdbvl2NwrFjx+xYU7zuVINY07/+q/TSS023zVdzjkQkMK64omlGlLFwoRQKeddOg64ntEtiYqKdztuzZ0/16NFDO3fuVDgcjmox4zS49I4cORLV4xGZnm0UFRUpNzdX8SqINd1667nbZhqsmfEU0bOn9B//0bR4NdOpJYIC7TZs2DB7omBSUpJycnL0wgsveN0ktNOpU6c0btw4e3S4ZMkSLViwQH7nt5pmzpQuu+zc/a99remciXhEUOB9GTJkiLZt26YuXbrYIwz4y6FDh7R37147a+2RRx5REPitpuuuk1pOeDRnYA8fLt1xh1RYKH3ve9JXvyrV1MhzjFHgfTODhvv27VPv3r29bgraaejQoXbHGqTHzo81FRRI6elNZ2fX1TWdV/HMM03LhZiT8rxAUOAD8dOLEsF/7PxY00MPmZmETZ/5tHmzZGb1mgleyclSnz6SGYu/+WbpzjtNGHrTRoLCA2vXrrULABjmU2LN7CazxCPGKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAB0zDWz6+rq7BJRU1NjvyYmJtolCCJ1NDQ0KAgidQSlnpa15OfnK9lcfT4gNZWWlgbycQpiTcXFxWpsbJTf1dfXR71tQjgcDkezYWFhoYqKitqsLykpUVpaWvtaCADwVCgUUl5enqqrq5WRkdExRxQFBQWaO3fueUcUmZmZKi8vV0pKioJyRJGVlaWcnJxAvFuNvFMNSj0GNflDkGsqKyvrdEcUUQdFamqqXVozf7Ag/NFaMk/soDy5g1iPQU3+EMSaGgOyz2tPDcEYXAAAXDIEBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0ERY2fOnNHUqVM1ePBgHTx40OvmwKislC6/XEpIaFpuu01qfeFHcz8n59w26enSSy951WLwWoopgiLGV5SaNGmSvXxsRUWFxowZowMHDnjdLFx7rfSf/3nufmmptHLl+dusWCFt3nzu/g9/KA0cGLs24jy8lmKLoIiRuro6TZgwQevWrWu+Pm1tba19gu/bt8/r5mH6dGn8+HP3H3lEirxLNV/N/Yhx46QHHoh9G2HxWoo9giJGJk+erA0bNthrj+fm5tp1mzZt0unTp5Wdna0jR4543USsXi316tV0OxSSpk0zeyXp3nul06eb1vfsKT3+uKfN7Ox4LcUeQREjs2fP1qJFi7R48eLmdSNHjrQXa58+fbr69+8vP15zNz8/X+Xl5QqEfv2kRx89d3/nTmnECOl///fcOvP9q65SELpuZs6cqcOHD8tvgvhaindJXjegszCHxWZpbcSIEXbxm7Nnz2ratGm2jzgxMVFLly5VIEyaJE2dKv3iF033X3zx3Pfy8qQvfEFBsGfPHq1evVrr16/X1q1bda0Zp/GJoL2W/IAjCryv2SZTpkyxITFjxgwVFxcrUMzAtTm6aOnKK9sOcPuYeQduQqKqqkqjRo2yA8LAe+GIAu1mZpuYgcTu3bsrISFBDz74YFQ/N2vWLA30w0wh08f91lvnrzP3X3lFuuEGxbv58+frxIkTUW2blZWlXbt2afTo0dqxY4cGDRp0ydsH/yEo0O5xie3bt9vb1dXVWtmOd9kTJ06M/6BoaGgavDaD2Bdav3u3lJqqeLZmzRodPXq0XT9z7NgxO9ZEUOBC6HpCu5jxiC1btqhnz57q0aOHdu7cqXA4HNVyoX7luLNwofS3v527//Wvn7tdViZ9+9uKd2bWTzSPR2RKqVFUVNQ8gwhojaBAuw0bNswOgCYlJSknJ0cvvPCCAsHU0XK85ctfbhqv+MpXzq1btkx67jn53alTpzRu3Dh7dLhkyRItWLDA6yYhjhEUeF+GDBmibdu2qUuXLvYIw/feflu67z4znavp/jXXSD/6UdNt8/VDH2q63djYtF1trfzs0KFD2rt3r5YtW6ZHWp5MCFwAYxR438xAqDkTtnfv3vK9/Hyz92y6nZgoPfGE1K1b033zOVD//d/S6NFNQfLyy9KcOdJjj8mvhg4dasMiEI8dLjmOKPCBBGJHs3Gj9NOfnrtvQmDUqPO3ufnm8z/Gw5zFvWGD/CwQjx1igqDwwNq1a+1gIuKE+ewm83hEFvOBfxfyve+dv93tt8e6pWiF11JsEBQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOCUpCjV1dXZJaKmpsZ+TUxMtEsQROpoaGhQEETqCEo9BjX5Q5BrSgzY/i4aCeEoLzhbWFiooqKiNutLSkqUlpbWvhYCADwVCoWUl5en6upqZWRkdMwRRUFBgebOnXveEUVmZqbKy8uVkpKioCRsVlaWysrK1NjYKL8LWj0GNfkDNcW/+vr6qLeNOihSU1Pt0pr5gwXhjxbkmoJWj0FN/kBN8as9NQSjsw0AcMkQFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADglKUp1dXV2iaiurrZfGxoaFBSJiYkKhUKqr69XY2Oj/C5o9RjU5A/UFP8i++5wOHzxjcNRWrhwofnfWFhYWFgUnKWysvKi+/8E88/7OaI4efKkBgwYoFdffVXdu3dXENTU1CgzM1OvvfaaMjIy5HdBq8egJn+gpvhneoWuvvpqnThxQj169OiYrqfU1FS7tGZCIgh/tJZMPUGqKWj1GNTkD9Tkjy61i24Tk5YAAHyLoAAAXJqgMN1QCxcuvGB3lF8Fraag1WNQkz9QU7DqiXowGwDQOdH1BABwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIBc/j+c6Peoa2v/OgAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "my_policy = np.array(\n", + " [\n", + " A_RIGHT,\n", + " A_RIGHT,\n", + " A_RIGHT,\n", + " A_DOWN,\n", + " A_DOWN, # First row\n", + " A_UP,\n", + " A_DOWN,\n", + " A_DOWN,\n", + " A_LEFT, # Second row\n", + " A_UP,\n", + " A_RIGHT,\n", + " A_DOWN, # Third row\n", + " A_UP,\n", + " A_LEFT,\n", + " A_RIGHT,\n", + " A_RIGHT,\n", + " A_RIGHT, # Fourth row\n", + " A_UP,\n", + " A_LEFT,\n", + " A_DOWN,\n", + " A_RIGHT,\n", + " A_UP, # Fifth row\n", + " ],\n", + ")\n", + "\n", + "V_my_policy = policy_evaluation(policy=my_policy, P=P, R=R, gamma=gamma)\n", + "\n", + "plot_values(V=V_my_policy, title=\"Value function: my policy\")\n", + "plot_policy(policy=my_policy, title=\"My policy\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "9cd7b3d0-e5ea-48c1-a68a-be3b3e782e9f", + "metadata": {}, + "source": [ + "-----------------------------------\n", + "\n", + "## 4. Dynamic programming : Policy improvement and Policy iteration" + ] + }, + { + "cell_type": "markdown", + "id": "bb35acc8-6469-499b-b565-3f2d590b13bc", + "metadata": {}, + "source": [ + "**Exercise 12.** \n", + "\n", + "Write a `policy_improvement` function whose inputs are the state-value function `V`, the transition probability matrix `P`, the reward vector `R`, and the discount factor $\\gamma$. \n", + "The function should return a **greedy policy** that, for each state, selects the action that maximizes the expected return according to the input `V`.\n", + "\n", + "\n", + "*Question: Why don’t we input the old policy in this policy improvement step?*\n", + "\n", + "\n", + "*Remark.* In this maze game, we consider a deterministic policy $\\pi:s\\in\\mathcal{S}\\mapsto a\\in\\mathcal{A}$ that assigns one single action to each state.\n" + ] + }, + { + "cell_type": "markdown", + "id": "94f00eeb-63d4-43a1-813f-37dda7276693", + "metadata": {}, + "source": [ + "------------------\n", + "\n", + "*Hint.* 1. This exercise can be completed in two steps. \n", + "\n", + "In the first step, compute the action-value function $q^{\\pi}(s,a)$ from the state-value function $s' \\mapsto v^{\\pi}(s') $, for a fixed state $ s $. \n", + "Which formula should be used to express $ q^{\\pi}(s,a) $ in terms of $ v^{\\pi} $?\n", + "\n", + "In the second step, perform the greedy policy improvement step by computing a new policy $ \\pi' $ such that\n", + "$$\n", + "\\pi'(s) = \\arg\\max_{a} q^{\\pi}(s,a).\n", + "$$\n", + "\n", + "Attention, for terminal states, action choice is irrelevant, we can set 0 to terminal states. \n", + "\n", + "2. Bellman action-value equation for the maze: \n", + "\n", + "In this maze environment, the **immediate reward depends only on the current state (for non-terminal state)**:\n", + "\n", + "$$\n", + "r(s,a,s') = R(s).\n", + "$$\n", + "\n", + "This means:\n", + "\n", + "- The reward does **not** depend on the action taken.\n", + "- The reward does **not** depend on the next state.\n", + "- All actions taken from the same state yield the same immediate reward.\n", + "\n", + "The general Bellman equation for the action-value function is:\n", + "\n", + "$$\n", + "Q(s,a)\n", + "=\n", + "\\sum_{s', r} P(s',r\\mid s,a)\n", + "\\left(\n", + "r(s,a,s') + \\gamma V(s')\n", + "\\right).\n", + "$$\n", + "\n", + "Since the reward satisfies \n", + "$$\n", + "r(s,a,s') = R(s),\n", + "$$\n", + "we can simplify the expression:\n", + "\n", + "$$\n", + "\\begin{aligned}\n", + "Q(s,a)\n", + "&= \\sum_{s'} P(s' \\mid s,a)\n", + "\\left(\n", + "R(s) + \\gamma V(s')\n", + "\\right) \\\\\n", + "&= R(s) + \\gamma \\sum_{s'} P(s' \\mid s,a) V(s').\n", + "\\end{aligned}\n", + "$$\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "998e3005-9a8b-4759-a008-aeccedd25924", + "metadata": {}, + "outputs": [], + "source": [ + "def policy_improvement(\n", + " V: np.ndarray, P: np.ndarray, R: np.ndarray, gamma: float,\n", + ") -> np.ndarray:\n", + " \"\"\"Given a value function V, compute the improved policy.\n", + "\n", + " Args:\n", + " V: array of shape (n_states,)\n", + " P: array of shape (n_actions, n_states, n_states)\n", + " R: array of shape (n_states,)\n", + " gamma: discount factor\n", + "\n", + " Returns:\n", + " policy: array of shape (n_states,), with values in {0,1,2,3}\n", + "\n", + " \"\"\"\n", + " n_actions = P.shape[0]\n", + " policy = np.zeros(n_states, dtype=int)\n", + " for s in range(n_states): # We decide the best action separately for each state.\n", + " # For terminal states, action choice is irrelevant; keep 0\n", + " if is_terminal(s):\n", + " continue\n", + "\n", + " Q_values = np.zeros(n_actions)\n", + " for a in range(n_actions):\n", + " Q_values[a] = R[s] + gamma * np.dot(P[a, s, :], V)\n", + " policy[s] = np.argmax(Q_values)\n", + "\n", + " return policy" + ] + }, + { + "cell_type": "markdown", + "id": "28800a10-f76b-4f27-a697-1238678f6bb3", + "metadata": {}, + "source": [ + "**Exercise 13.** \n", + "\n", + "Write a `policy_iteration` function whose inputs are the initial policy `initial_policy`, the transition probability matrix `P`, the reward vector `R`, the discount factor $\\gamma$ `gamma`, the tolerance parameter `theta` used in policy evaluation (the evaluation stops when the value function changes by less than `theta`), and `max_iter`, which serves as a safety limit to prevent the loop from running indefinitely. \n", + "\n", + "The function should return two outputs: \n", + "- `policy`, the final (optimal) policy, represented as an array of action indices; \n", + "- `V`, the value function corresponding to this policy.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "id": "22663608-c3a4-47d0-b1c9-b7d1bb67fa64", + "metadata": {}, + "source": [ + "--------------------------\n", + "\n", + "*Hint.* The `policy_iteration` algorithm consists of two main steps. \n", + "First, the **policy evaluation** step, where you will use the function implemented in **Exercise 8**. \n", + "Second, the **policy improvement** step, where you will use the function implemented in **Exercise 12**.\n", + "\n", + "--------------------------" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8b6c2216", + "metadata": {}, + "outputs": [], + "source": [ + "def policy_iteration( # noqa: PLR0913\n", + " initial_policy: np.ndarray,\n", + " P: np.ndarray,\n", + " R: np.ndarray,\n", + " gamma: float,\n", + " theta: float = 1e-6,\n", + " max_iter: int = 1000,\n", + ") -> tuple[np.ndarray, np.ndarray]:\n", + " \"\"\"Policy Iteration.\n", + "\n", + " Goal:\n", + " Learn an optimal policy by alternating:\n", + " 1) Policy Evaluation\n", + " 2) Policy Improvement\n", + "\n", + " Inputs:\n", + " initial_policy : array of shape (num_states,)\n", + " Initial deterministic policy.\n", + " P : transition probabilities\n", + " R : reward function\n", + " gamma : discount factor\n", + " theta : stopping threshold for policy evaluation\n", + " max_iter : maximum number of policy iteration steps\n", + "\n", + " Returns:\n", + " policy : optimal policy\n", + " V : value function of the optimal policy\n", + "\n", + " \"\"\"\n", + " policy = initial_policy\n", + "\n", + " for _it in range(max_iter):\n", + " # Compute the value function V^pi for the current policy.\n", + " V = policy_evaluation(policy, P, R, gamma, theta)\n", + "\n", + " # Improve the policy by acting greedily with respect to V.\n", + " new_policy = policy_improvement(V, P, R, gamma)\n", + "\n", + " # Check whether the policy has stopped changing.\n", + " if np.array_equal(new_policy, policy):\n", + " break\n", + " policy = new_policy\n", + " return policy, V\n" + ] + }, + { + "cell_type": "markdown", + "id": "7ed35eca-81fd-45ac-bc8c-550117124e21", + "metadata": {}, + "source": [ + "**Exercise 14.** \n", + "\n", + "Starting from a random policy (see Section 3.3), compute an optimal policy for the Maze game. \n", + "Then, plot the value function of this optimal policy and visualize the policy itself by displaying arrows on the maze.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "e12d62ee-3324-4e1b-b5e2-6be96404ac2c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Optimal policy found by Policy Iteration:\n", + "[1 1 1 2 2 0 2 2 3 0 1 2 0 3 1 1 0 0 0 0 1 0]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAGgCAYAAAC0SSBAAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAALB1JREFUeJzt3Qt4FNX5x/E3EYgECMhFQYl4Q1SiiCB/FMtNIxSvoBQNitcWRZBLan0iBRJFRFQUlYqKBS+NrVqLpVQkXNXiQ7FUJYSLUEVBxIiQEINJIPN/3oMTNpvksAlhd2fy/TzPEHZ2dvecuf1mzpzZjXEcxxEAAKoQW9UTAAAoggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4IiwNy5cyUmJka+/PLLOvXZp5xyitx6661lj5cvX27Kon/DafXq1XLxxRdLo0aNzOd/8sknEo3S09NN+Q6nd+/ekpSUJF5S2Xqo9dABdn6eT1EdFOvWrZObbrpJTjrpJImLi5MTTzxRhg4dasYfiSlTpsi8efNqrZw4ciUlJTJ48GD54Ycf5Mknn5RXX31V2rVrF+lieYLu2N0hNjbWbCeXX3552IO+tg9errzyyrLHhYWFJqAjXaecnBxTjkgc0EVSPYlSb7/9ttx4443SvHlzueOOO+TUU081C+ell16St956S/785z/LwIEDaxwU119/vVx77bXlxt98881yww03mFCqy3r27Cn79u2TBg0ahO0zt2zZIlu3bpUXX3xR7rzzzrB9rl8kJyfLsGHDRL+67YsvvpA//OEP0rdvX1mwYIH88pe/PKL3XrRokUSaBkVGRob5fySP2nNyckw5tAwaZtE2n+pUUOhOQ3fap512mrz//vvSqlWrsudGjx4tv/jFL8zzn332mZmmthxzzDFmqOv0qPTYY48N62d+99135m+zZs3C+rl+ceaZZ5qzb5ceRJ133nny1FNPHXFQhPOAIdx+/PFH09RZGxr4eD5FZdPTY489Zo4gXnjhhXIhoVq2bCnPP/+8WcDTpk2r0G68YcMG+dWvfiUJCQnSokULEyw//fRT2XQ6jb725ZdfLjtdd9vnK2ufdU+B9ZS3a9eu0rBhQzn33HPLToH1zEcf6461S5cu8t///rdceTXM9P010HSa1q1by+233y67du2q0bzR92rcuLH873//k379+pmVXJsaHnzwQXM0GUjrmZqaKomJieYsqUOHDvL4449XmC5YVdcoVq1aJQMGDJDjjjvOfK7uiGbMmGGemzNnjnlNcP3dMzgN4O3bt1dZp169epn/a/OTvk/gUePSpUvNwYF+pgbJNddcI+vXr6/wHsFHeFVdT9DHI0eONM2Peg1B503Hjh1l4cKFFV7/4YcfyoUXXmiW3emnn27Wver6z3/+Y6696LqjZ8azZs0qe66goMDUS9fTYNu2bTPz7ZFHHqn2Z+o6qduKnl1UZz6G2vau25TOWw0onTdt2rSRQYMGmYM8Xb90Wej7B9PXNW3aVIYPHx5yXXR7dPcDejTvbrf6+S7d7rWVQFsgtDy6rf79738v9z7u9r1ixQoZMWKEHH/88dK2bVvznJ7N6jjdRnQ56b5D18XAfcHcuXPNONWnT5+ycrjbSWXzSQ+AtEXkhBNOMOXq1KmT2fcE10/fR7dN3efpeqbrpK53et0uKjhR6MQTT3ROOeUU6zT6fNu2bcseT5o0Sfd+zrnnnutcddVVzrPPPuvcdNNNZtzNN99cNt2rr77qxMXFOb/4xS/M/3VYuXKleW7OnDlm+i+++KJs+nbt2jkdOnRw2rRp46SnpztPPvmkc9JJJzmNGzd2XnvtNefkk092pk6daoamTZs6Z5xxhnPgwIGy1z/++OPmsx588EHnhRdecEaPHu00bNjQ6datm1NaWlo2XWWfXZlbbrnFOfbYY5327dubemk9r7zySvPaCRMmlE2n7923b18nJibGufPOO810Ol90ujFjxpR7T62jvq9r2bJlZjr961q0aJHToEEDM63O6+eee8659957ncsuu8w8n5+fb+qVmppaocznnHOOKUtVdP4/8MAD5jP1PXWZ6OeprKwsp169es6ZZ57pTJs2zcnIyHBatmzpHHfcceXmlZZfyxbMXS8C6eNOnTqZZfrQQw85Tz31lHPaaac58fHxzvfff1823WeffWbqpMv4kUceMdOecMIJznnnnVfhPSvTq1cvsy4ff/zxzsiRI52nn37aueSSS8xrX3rppbLphg4dat53//795V6v9dXlt3XrVuvn6Pvdc8895cb98MMPzjHHHON07969WvOxsvVQ66GDS8t56aWXmuluuOEGs27p/NFlPG/ePDPN+PHjnfr16zu7du0qV6433njDvO7999+31kmX5RVXXGH+X1BQYNY3fd3AgQPLtttPP/3UPJ+dnW22PV3PHn30UVOenj17mnn39ttvV6ibTqf1eeaZZ8x2q958802zTkycONFsp7o+6rzRcvz4449mmi1btpj1U99Dn3fL8e2331Y6nwoLC52zzz7bzIexY8ea5a/7An29rnMundc6rnPnzmb/oXXQZaTLR/dxxcXFTqRFXVDs2bPHzLRrrrnGOt3VV19tptMdVOAOQccHGjFihBnvrlSqUaNG5XaMrqqCQse5YaLee+89M053IoEb8fPPP19hB6srS7DXX3+9wsZSnaDQ6UaNGlUuFHSj0h15bm6uGacbrE43efLkcq+//vrrzQa0efPmkINCdwynnnqqmW737t3l3i8w7G688UazYwwMyjVr1pj30vrZuJ+pG2yg888/3+xoA3c4uixjY2OdYcOG1TgodF4FzgN9Tx2vOw/Xtddea0I5cBnn5OSYHXCoQaHTPfHEE2XjioqKyurk7gDc9endd98t93oNpMAdT1X0tXfccYdZ9t99952zatWqsh25+9mhzsdQguKPf/yjmWb69OkVyuKuDxs3bjTT6A4+kG6fepAXuN4cLiiU1k3fT5dnMK2rHiD+9NNP5cpx8cUXmwOq4LppWAeHcmXb6UcffWSmf+WVV8rG6foZvI1XNZ80DHRaPaB06TK/6KKLzIGmu+9yg6JFixYm4F3vvPOOGT9//nwn0qKu6Wnv3r3mb5MmTazTuc/n5+eXG3/PPfeUezxq1Cjz95///GeNy3TOOefIRRddVPb4//7v/8xfvVh48sknVxivzUIuPY0NPO3+/vvvpXv37ubxmjVralwmbToJbkopLi6WxYsXl9VXmy3uvffecq/Tpijdt7z77rshf5Y2J2kTxpgxYypcQwhs1tGLqd98840sW7asbNyf/vQnMw+uu+66atdxx44dpousNitpk4JLm7z04u2RLNPLLrvMnOIHvqc2V7rL7sCBA/Lee++ZDg+By/jss882TX6hqlevXrlmFm3H1sfaJKFNUm5ZtPlQ55UrOzvbNFsGXnew0U4e2jyjzSm6Hv7rX/+ScePGmWVW2/Pxr3/9q2nWcretytYHbZLScgTWSXu06XqnPRdD6V4cCn1PbVLT5mbdd+j2pYM27epy+vzzzys0ef7617+ucC0ycDvVHnj6+jPOOMOs72tquJ3qfNWmZu2U46pfv77ZJrXJUZvAAg0ZMsQ067q0mTB4fxIpURcUbgC4gVHdQGnfvn25x7oz0IuzR9KdLXBHobSNVWnbf2Xjd+/eXW5F1vZnbaPUlVE3Zm2nVnl5eTUqj9Yn+CK+bpjKrae2uerOJ3j+6I7OfT5U2u6sDndPgO50tK3a3TmUlpbK66+/btqqDxf8lXHLqO3GwbQeukPQ6zC1sUyVbqTussvNzTU9v4LXp6rKUxVdBsEXS4OXlS5P3XnqNRO9Nqd0Hmqbttsmfjg6j7OyssyBgl5L0nnzxBNPmPeu7fmo64O+l4agjR44aGC5n//mm2+anbB2RKktmzdvNgc+EyZMMNtW4DBp0qRyHSVc7vYXSJf1xIkTy67naRDqe+zZs6fG26nWW9cfXQahbIPB66QbGoH7k0iJul5PurPVnY0eTdno83p/hR4F2tTGkUtVPaGqGh94sViPdFauXCn33XefnH/++eZCtO5A+/fvb/76ic6PlJQU08VVu2fqTkLPMEI9Kj4SVS1nPTOo6bILJ92paicODQs9As3MzDSdKNyDj8PRi7J6ZhJNtKv52LFjTeg98MAD8tprr5mLzNUJ2sNxt6Hf/va3VZ7p6ZlBVWcPLj070g4ZegamrQc633Wd0jqUhmk7jbZ1MqqDQukGojsb7XFyySWXVHj+gw8+MEdjlfWc0FPNwCMGPeLQBR3YI6a2TnsPR48ElixZYnpq6NFKYBmPhNZHT0fdI1O1adMm89etp96spkeXeuYVeDSvvUPc50PlNtFoc8jhdka6w9Mj2fnz55tmBj0qq05TTSC3jBs3bqzwnNZDj/rco3U9+tKjv2DVOXMKpOXWHUply6qy8lRFgzK4C2bwsnLP1jp37mx2qrrT/+qrr+SZZ56R2lCd+Rjq+qBnLXp2oE0pVdFmriuuuMLUSc+Y9MBBu+vWRFXbrHtmreU4kqDUe7NuueUWs+4GNhUHr1Mx1dh36HzXA1rdXgPPKmqyDUZa1DU9KT361o1UgyC4G6k25dx1110SHx9vpgs2c+bMco/djS2wL7luFJXtVI7WEULwEUFNN5ZAzz77bNn/9f31sW4sl156qRmn3Vj1aDpwOqV3PevKXp2+9RdccIEJXy138HwLrpu2e+swe/Zs05atR2SHa6Koip5Z6lmYdicM/FwNLL25SesYuPPSJoLAM1Ftm//b3/5W42WnAadH+LrTdml3Ur12Ear9+/eX61Kr15H0sQaRdqcOpE0yWi+dz9o980jvf6jJfAyFXm/S5qrgdauy9UHrpDep6baq81TXh5rQ7V0Fr396TUa7pOo81eUdTJsQQ6FlCy677juCz0gb/Ryooew/dL5+++238pe//KXc+qDvqy0LbpdwL4jKMwpt19OVWo9CtD948J3ZupJq23fgxUiXXnS9+uqrTdPORx99ZE53tTlE+y+7dAPVo+3p06ebNmR9b/dCdG3SZjG9y1nv99CjL20q0w0zsG97TWjbtfb51yMgLbceuesduHp67/Y3v+qqq0xf7/Hjx5v5pvXXz37nnXfM6XVl864qejT03HPPmffUHc5tt91mdj56ZKRfpxK849SzCm0KUEfa7KTNMbrD1OYAXQ+0LVk3NG0aCOxHrzug+++/39xophcLta1fy6xnXTW9GKlngjqf9aKi9rF3N3K95+JwTaMuXb8effRRswy0LLrT0AvL2l8++Ghc19Pf/e53Jtzuvvtu69H60ZqPodDl+8orr5iL5f/+97/N/NGzJt2mdD4F3j+hZxQaenp9Qj9fd+w1oQeO2qlE55/ORz1b0bMwHfTgUFsedF+hF6r1LGPnzp1m+9d7UT799NOQWjH0a2N0fujn6Gu1Plr2QLr+a6joMtUDE72eoZ1aKqvXb37zGxNg2olAOy7oGaSeubhnVjW5bhcxThTTfuza5VL7u2tf5NatW5vHa9eurbIbpHZf1C6gTZo0Mf2gtf/6vn37yk27YcMG089au7fqa9yuoVV1jw3spmfru+52c3vsscfKxm3bts30/W7WrJnp6z148GDnm2++qdDVrzrdY7V7r/bpvvzyy03ff+2Dr+8V2C1V7d271/Tf1i6rOv+0q6CWLbhrYij3UagPP/zQSU5ONvNWy6DdNwO7k7p27NhhupBqn/1QVdU9Vi1evNjp0aOHWV4JCQnmfhBdzsH03oukpCTT9VXvfdFuiVV1jw1edpXNB7VixQqnS5cu5j31XotZs2ZV+p6V0a6SHTt2dD7++GPTJVK72upnaD//qgwYMKBCd+zDqao+NZmPoXSPdbuT6r0S2m3a3TZ1u9P1MpjbRT0zMzPkOlW23ek8cZdF8Pajn6vdfLUcWh6910nvL3rrrbcq1G316tUVPk+7fd92223m3gXtutqvXz+zn6hsnXjxxRfNuuB2k3a3k8rm086dO8veV8ut3XiDu4pXtt9wVdUlONyiOiiqw9143fsI/MoNimimy0Bv7tKbDFE9eu/G6aef7viJ3uCpBxfujWvwnqi8RgFv06860Lbd2uwGWRdoG7s2IfppvukFYW3+1esa7nUGeE9UXqOAN+mNT3rh8uGHHzY3qlX23UuoSK9Zabu1dgDQ6xLV+R6kaKX3Lmgbv7bJa4eUyr7LCt5BUKDW6BcT6j0jPXr0qLWunXWB3qGrHQT0hivtxKF383qdHjBoZxS9yPv000+bi8Dwrhhtf4p0IQAA0YtrFAAAK4ICAFA71yiKiorM4NLb0vUuab0hJVxfiQEAqB161UG/4kdvCg3+4sIaB4X+ypb7m7UAAH/4+uuvy37p74gvZgefUejt69pLQ7/DpTa/aiCSNFX19n3tseGHb3b1W30UdfIG6hT99GuF9Ktd9HurDvctxSGfUeh3mugQTEPCLz8qriuC3hSk9fHDiuC3+ijq5A3UyTtCuXTAxWwAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAABWBAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAABWBAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACs6oU6YVFRkRlc+fn55m9sbKwZ/MCtB/WJXtTJG6hT9KtOPWIcx3FCmTA9PV0yMjIqjM/MzJT4+PjqlRAAEFGFhYWSkpIieXl5kpCQUDtBUdkZRWJiouzYsUNatGghflBSUiJZWVmSnJws9evXF6/zW338Xqfs7GwpLS0VvxytJiUlsZyiWHFxsUyePDmkoAi56SkuLs4MwXQl8MuK4Nc6+a0+fq2T7nz8sAMKxHKKXtWpgz8a2wAARw1BAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAABWBAUAwIqgAABYERQAACuCAgBgRVAAALwZFH/O/rP0e62fnPD4CVL/ofrSdGpTOXXGqdJ7bm8Z/e5oeW/ze5EuInxs9OjR0rp160gXA4fBcgqPkH/hLpyG/W2YvPrZq+XG5Rflm+HLPV/Kiq0rZGveVul3Rr+IlRH+pj8PuXPnzkgXA4fBcqqjQbFw88JyIdGlTRfpd3o/adygseQW5sqaHWvko20fRbSMAFCXRF1QLNqyqOz/ZzQ/Q1bduUqOiT2m3DR6ZrF251rxA/2B83Hjxsl9990n7dq1Ez/wY52AuizqrlHsL91f9v89P+0xTU3BEuISpMfJPcQP1qxZI7Nnz5aePXvKli1bxA/8WCegLou6oLigzQVl//++8Hs589kzpcsLXeSuf9wlL/7nRdn8w2bxk+7du8v8+fMlNzfX7Fg3btwoXufHOgF1WdQ1Pd103k0yc/VM+fibj83jUqfUXJfQwXXJyZfIs798Vjq17iTRbMKECbJ79+6Qpk1KSpLVq1dLr169ZMWKFdKhQweJRn6sEwCPBUW92HqydNhSeeTDR+SP//2j7PyxYo+GD7/6UJJfTZZ1I9ZJq0atJFrNmTNHtm/fXq3XaA+OnJycqN2p+rFOADzW9KSaxDWRKZdOkR2pOyT77mx56eqX5JZOt0iTBk3KptEeUMFdaKPNtm3bxHGcww4FBQXSu3dv85qMjAwZOHCgRCs/1gmAB4PCFRMTIx2P7yi3d75d5l47Vz67+zOJjTlU5M93fS5et3fvXunfv78sX75cpk6dKhMnThSv82qd9CL8rFmzKoxfv369zJgxIyJlQkUsp/CLuqanlz95WX7a/5PceO6NpndToEb1G5mg0OsWqtmxzcTrNm/eLGvXrpXp06fL2LFjxQ+8Wqfx48fLwoULpbCwsGzcunXrpG/fvrJv3z4ZNGiQJCYmRrSMYDlFQtQFxRd7vpCMFRky5r0x5qL1+SecL80bNpdd+3bJWzlvles+2/+M/uJ1nTt3NjvWli1bil94tU5vvPGGDBgwQFJTU6VVq4PXvvr06WPuC1m0aBE7nyjBcgq/qAsKl55VLP7fYjNU5tcX/Fp6ndJL/MBrO1S/1qlJkybmSPWqq66SZcuWmXH79++XxYsXS9euXSNdPPyM5RR+URcUY7qPkXOPP1eWfrFUPt7xsXxb8K3k/pgrB5wD0iq+lXQ5sYu5sD3o7EGRLip8qFGjRrJgwQK55pprTFt4VlaWOUNCdGE51fGg0OsO151znRmASGjYsKFpwkB0YzmFT1T3egIARB5BAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAABWBAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAFUEBAKidn0ItKioygys/P9/8LSkpMYMfuPWgPtHLz3WKjfXPcZtbF5ZT9KpOPWIcx3FCmTA9PV0yMjIqjM/MzJT4+PjqlRAAEFGFhYWSkpIieXl5kpCQUDtnFGlpaTJu3LhyZxSJiYmSk5MjDRo0EL8kbFJSkiQnJ0v9+vXFD0dAWVlZkp2dLaWlpeIHfltGgcuJOkW3Ep9tT8XFxSFPG3JQxMXFmSGYzjA/zLRAumL7ZeVWLCNvoE7eUOqT7ak6dfBHYxsA4KghKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQRFm+/fvl6FDh8pZZ50lmzZtinRxUIXRo0dL69atxU/8WCeEB0ER5p8eHDx4sPmd8Y0bN0rv3r1lw4YNkS4WKqG/I7xz507xEz/WCeFBUIRJUVGRDBo0SObNm1f2Q+YFBQUmLNatWxfp4gFAlQiKMBkyZIgsWLBA0tLSZODAgWbcokWLZN++fdKnTx/Ztm1bpIsIeOa3nlNTUyUnJyfSRakz6kW6AHXFmDFj5MILL5Tx48fLrbfeasZ1795dsrKyZP78+dK2bdtIFxGIegcOHJBhw4aZ5tvY2Fh57LHHIl2kOoGgCBNtYtIhWLdu3cwA4PAdQVJSUuTNN9+UESNGyLRp0yJdpDqDoADgCdoRRK/xNW3aVGJiYmTUqFEh9/Zq3779US+fnxEUADxxXWL58uVlvbdmzpwZ8muvv/56guIIcTEbQNTT6xFLliyR5s2bS7NmzWTVqlXiOE5IQ2VNvqgeggL42Zo1a2TWrFkVxq9fv15mzJghXuSnOl1wwQWydOlSqVevniQnJ8vKlSsjXaQ6g6Yn4GfaI23hwoVSWFhYNk7vcenbt6/pxqz3wSQmJoqX+K1OnTp1kmXLlsmll15qzjAuvvjiSBepTiAogJ+98cYbMmDAANNHv1WrVmac3uOid9TrPS9e2qH6uU5JSUkm7Fq2bBnpotQZND0BP2vSpIk5+tYdaW5ublmXzMWLF5t7XrzIj3VShER4ERRAgEaNGpk76LUNvEWLFqZ5o2vXruJlfqwTwoumpwiYO3euGRCdGjZsaJpl/MSPdUL4cEYBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAAC181OoRUVFZnDl5+ebv7GxsWbwA7ceJSUl4gduPfyyfPy4jALr4sc6TZs2TUpLS8Uv615SUpJvtqfq1CPGcRwnlAnT09MlIyOjwvjMzEyJj4+vXgkBABFVWFgoKSkpkpeXJwkJCbVzRpGWlibjxo0rd0aRmJgoOTk50qBBA/HTEUNycrLUr19f/HBUl5WVJdnZ2b47qvPLMgpcTn6skx/XvWyf1Km4uDjkaUMOiri4ODME0xnmh5kWSDdWv2ywimXkDX6skx/XvVKf1Kk6dfBHYxsA4KghKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQRFm+/fvl6FDh8pZZ50lmzZtinRxUIeMHj1aWrduHeliwIMIijD/9ODgwYPN74xv3LhRevfuLRs2bIh0sVBH6G8j79y5M9LFgAcRFGFSVFQkgwYNknnz5pX9kHlBQYEJi3Xr1kW6eABQJYIiTIYMGSILFiyQtLQ0GThwoBm3aNEi2bdvn/Tp00e2bdsW6SLiMGeDI0eOlK1bt0a6KEDYERRhMmbMGJk8ebJMmTKlbFz37t0lKytLhg8fLm3bto1o+WC3Zs0amT17tvTs2VO2bNkS6eIAYVUvvB9Xd2kTkw7BunXrZgZENw31+fPnyzXXXGPCYunSpdKhQ4dIFwsIC4ICdd6ECRNk9+7dIU2blJQkq1evll69esmKFSsIC9QJBAXqvDlz5sj27dur9RrtPZSTk0NQoE7gGgXqPO1I4DjOYQe3l5rKyMgo65QA+B1BAYRg79690r9/f1m+fLlMnTpVJk6cKF65CD9r1qwK49evXy8zZsyISJngPTQ9ASHYvHmzrF27VqZPny5jx44Vrxg/frwsXLhQCgsLy8bpfTt9+/Y1XbP13p7ExMSIlhHRj6AAQtC5c2cTFi1bthQveeONN2TAgAGSmpoqrVq1MuP0vh29L0Tv4yEkEAqanoAQeS0kVJMmTcwZhYZDbm5u2feNLV682HT5BUJBUAA+16hRI/OtAMnJydKiRQtZsmSJdO3aNdLFgocQFBEwd+5c04sGCJeGDRuapqbvv//eNKMB1UFQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAABWBAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAVT0JUVFRkRlc+fn55m9sbKwZ/MCtR0lJifiBW4/U1FSpX7+++KVOWVlZMm3aNCktLRW/rHdJSUm+We/8vu6l+qROu3btkilTpoQ0bYwT4o83p6enS0ZGRoXxmZmZEh8fX/1SAgAiprCwUFJSUiQvL08SEhJqJygqO6NITEyU3//+99KgQQPx05FdcnKyL44Y3CMgv9QnsE7Z2dm+O6Pw43KiTtF9RtGmTZuQgiLkpqe4uDgzBNON1S8brEtXAj+sCH6tj2K98wbqFL2qUwd/XFwAABw1BAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBEWb79++XoUOHyllnnSWbNm2KdHEARIFdu0SmTRO5/HKRE08UOfZY/Q0gkTZtRHr2FLnvPpEPPhAJ7Wfmal/IP1yEI1dcXCxDhgyRefPmmce9e/eWpUuXmtAAUDe98ILIuHEiP/5Y8blvvz04aEg8/rjIjh0irVuHv4wERZjoz8hed911smDBAvOzg/pTsgUFBSYslixZIh07dox0EQGE2WOPifzud4cex8SI9Okj0r27SOPGIj/8IPLJJyIffijy00+RKydBESZ6JqEhkZaWJt988428/PLLsmjRIunXr5/06dNH1qxZI23bto10MQGEyfr1Imlphx63aCHy97+LXHxxxWkLCkRefVWkYUOJCK5RhMmYMWNk8uTJMmXKlLJx3bt3Nz/WPnz4cE+GhP5mdWpqquTk5ES6KKhBM+jIkSNl69at4hdeq9PTT4scOHDo8axZlYeE0rOLu+8WadpUIoKgCBNtYho/fnyF8d26dZOHHnpIvObAgQNy8803y/Tp02XOnDmRLg6qSc9gZ8+eLT179pQtW7aIH3itTkuWHPr/cceJDBokUYugQI16bt14442SmZkpI0aMkGnaXQOeomez8+fPl9zcXLNj3bhxo3id1+q0ffuh/7dvLxIbsDfesOHg9Yrg4dZbI1JUrlGg+gYPHmx6bjVt2lRiYmJk1KhRIb1u9OjR0l63CBxVEyZMkN27d4c0bVJSkqxevVp69eolK1askA4dOkg08mOdAmkIRDOCAtW+LrF8+XLz/7y8PJk5c2bIr73++usJijDQpsDtgYerIdi5c6e51hStO1U/1umkk0Q+//zg//Wv3iPhBsbxxx/sEaUmTRIpLIxcORVNT6iW2NhY0523efPm0qxZM1m1apU4jhPSoNdpcPRt27YtpOXhds9WGRkZMnDgQIlWfqzTpZce+r92g9UeT67mzUV++9uDQ6R6OgUiKFBtF1xwgblRsF69epKcnCwrV66MdJFQTXv37pX+/fubs8OpU6fKxIkTxeu8VqeRI0WOOebQ47vuOnjPRDQiKFAjnTp1kmXLlsmxxx5rzjDgLZs3b5a1a9eaXmv333+/+IHX6tSxo0hgh0e9A7trV5GrrhJJTxd5+GGRO+8Uyc+XiOMaBWpMLxquW7dOWrZsGemioJo6d+5sdqx+WnZerFNamkijRgfvzi4qOnhfxT/+cXCojN6UFwkEBY6IlzZK+H/ZebFO996rPQkPfufT4sUi2qtXO3jVry/SqpWIXovv0UPk6qs1DCNTRoIiAubOnWsGAFD6LbHau0mHaMQ1CgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAAC185vZRUVFZnDl5+ebv7GxsWbwA7ceJSUl4gduPfxSn8C6pKamSn399Xmf1CkrK8uXy8mPdZo2bZqUlpaK1xUXF4c8bYzjOE4oE6anp0tGRkaF8ZmZmRIfH1+9EgIAIqqwsFBSUlIkLy9PEhISaueMIi0tTcaNG1fujCIxMVFycnKkQYMG4pcziqSkJElOTvbF0ap7pOqX+ijq5A1+rlN2dnadO6MIOSji4uLMEExnmB9mWiBdsf2ycvuxPoo6eYMf61Tqk31edergj4sLAICjhqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAABWBAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAFUEBALAiKMJs//79MnToUDnrrLNk06ZN4geer9OWLSKNG4vExBwcLr9cJPiHH/VxcvKhaRo1Evn880iVGH5Y7zyEoAjzL0oNHjzY/Hzsxo0bpXfv3rJhwwbxMl/U6fTTRZ544tDjrCyRmTPLT/PssyKLFx96/PjjIu3bh6+M8N965yEERZgUFRXJoEGDZN68eWW/T1tQUGBW8HXr1okX+apOw4eLDBhw6PH994u4R6n6Vx+7+vcXufvu8JcR/lvvPIKgCJMhQ4bIggULzG+PDxw40IxbtGiR7Nu3T/r06SPbtm0Tr/FdnWbPFmnR4uD/CwtFhg3TvZLIzTeL7Nt3cHzz5iIvvRTRYtZ1vlvvPICgCJMxY8bI5MmTZcqUKWXjunfvbn6sffjw4dK2bVvxGt/VqU0bkeeeO/R41SqRbt1E/v3vQ+P0+RNPFD803YwcOVK2bt0qXuO79c4D6kW6AHWFnhbrEKxbt25m8CI/1kkGDxYZOlTkT386+Pizzw49l5Ii8qtfiR+sWbNGZs+eLfPnz5elS5fK6XqdxiN8ud5FOc4ogGB64VrPLgKdcELFC9wepkfgGhK5ubnSs2dPc0EYqApnFEAwbeP+4Yfy4/Txl1+KnH++RLsJEybI7t27Q5o2KSlJVq9eLb169ZIVK1ZIhw4djnr54D0EBRCopOTgxWu9iF3Z+I8/FomLk2g2Z84c2b59e7Ves3PnTsnJySEoUCmanoBAkyaJfPLJocf33HPo/9nZIr//vUQ77fXjOM5hB7dLqcrIyCjrQQQEIygA18qVItOmHXp8++0Hr1fcccehcdOni3zwgXjd3r17pX///rJ8+XKZOnWqTJw4MdJFQhQjKAD1448it9wicuDAwcennCLy1FMH/69/Tzvt4P9LSw9OV1AgXrZ582ZZu3atTJ8+Xe4PvJkQqARBAajUVN17Hvx/bKzIyy+LNGly8LF+D9Qrr4gcc8zBx198ITJ2rHhZ586dTViM9Xg9EB4EBbBwocjzzx96rDvPnj3LT9OjR/mv8dC7uBcsEC9r2bJlpIsAjyAoImDu3LnmYqKfeLpO+t1NWnZ30C/8q8zDD5ef7oorwl1S+Gm98xCCAgBgRVAAAKwICgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArOpJiIqKiszgys/PN39jY2PN4AduPUpKSsQP3Hr4pT6KOnmDn+sU67P9XShinBB/cDY9PV0yMjIqjM/MzJT4+PjqlRAAEFGFhYWSkpIieXl5kpCQUDtnFGlpaTJu3LhyZxSJiYmSk5MjDRo0EL8kbFJSkmRnZ0tpaal4nd/qo6iTN1Cn6FdcXBzytCEHRVxcnBmC6Qzzw0zzc538Vh9FnbyBOkWv6tTBH41tAICjhqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAABWBAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAABWBAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAqp6EqKioyAyuvLw887ekpET8IjY2VgoLC6W4uFhKS0vF6/xWH0WdvIE6RT933+04zuEndkI0adIkfTcGBgYGBvHPsGXLlsPu/2P0n5qcUezZs0fatWsnX331lTRt2lT8ID8/XxITE+Xrr7+WhIQE8Tq/1UdRJ2+gTtFPW4VOPvlk2b17tzRr1qx2mp7i4uLMEExDwg8zLZDWx0918lt9FHXyBurkjSa1w04TlpIAADyLoAAAHJ2g0GaoSZMmVdoc5VV+q5Pf6qOokzdQJ3/VJ+SL2QCAuommJwCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAADE5v8BAnWrNS8uKpYAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdcAAAGbCAYAAACWHtrWAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAUO5JREFUeJzt3Qd8VFX6N/DfmUmvEAgkQOgCgjRRAUUFVBDRFV0rFnQt+7q66qprW3Utu8ta1rquursq9rJ/29pQcSkWioKg9CIllNCTkJ7MnPfznJtJZpJJMmFOyoTfl8+Qmbl3zr137sx97nnOOXeU1lqDiIiIrHHZK4qIiIgEgysREZFlDK5ERESWMbgSERFZxuBKRERkGYMrERGRZQyuREREljG4EhERWcbgSkREZBmDaxCbNm2CUgozZsxokeXPnDkTw4YNQ1xcnFmP3NxctEaybvfeey9as++++w7HHnssEhMTzfouXbq00WX07NkTp59+OtqSyy67zGzXobTsYN9r+fzKc0S2RXxw/cUvfoGEhAQcOHCgznkuuugixMTEYO/evWjtZB3PO+88xMfH4+mnn8Yrr7xiAkNL+eSTT1p9AK1LeXk5zj33XOzbtw+PPfaYeS979OgRdN6VK1ea7ZQDcFuxfft2s00Hc0JBRGHSEe7NN9+UayPrl156Kej0wsJCnZiYqM8444yQy9y4caMp88UXX9TN7dNPPzXL/uKLL3RrcO2115r1Caa4uFiXl5fr1mrVqlVm3f/1r381OO9//vMfM+/s2bNrTevRo4eePHmyjjTfffddnZ/jsrIyXVJS0iLrNW3aNPOeNrdg32v5/MrnmMi2NlFzTU5Oxuuvvx50+gcffIDCwkJTe40Eu3btMn/btWuH1k7S1lFRUWitIum9bG7R0dGIjY3FoU4+v/I5JrIt4oOrpE/PPvtsfPnll1UHU38SdCX4ShCW9OAtt9yCwYMHIykpCSkpKZg0aRKWLVvW4HLGjh1rbqG0H3m9Xjz++OMYNGiQ+eJ27twZv/71r7F///4GlzFt2jRz/+ijjzZtQVK+kGX47te3XnPmzDGve/vtt/HnP/8Z3bp1M+tw0kknYf369bVev3DhQpx22mlo3769ST8PGTIETzzxRNW2SWpaSJm+W31trj/88IN5T+W9lfdYlrtgwYKAeaTNS177zTff4KabbkJ6erpZ9llnnYXdu3cjFP/73/9w/PHHm9dJ8DzzzDOxatWqqumy7ieeeKK5L6lhWV6w/edbH5lHjBs3rmo75b309/XXX+OYY44x72fv3r3x8ssv1ypL2sdvvPFGZGVlmeDVt29fPPjgg+YzEYp//OMf5nMjr+3SpQuuvfbaWm3ush1HHHEEFi9ebNqT5TvQq1cvPPvss1XzyLrLZ0hcfvnlVdvka2+s+bn1tUc+8sgjZp/L9klzy4QJE5CdnS2pCzzwwAPm8yTLk/dbvk81T2QnT55s1lvWv0+fPuY1Ho8HB8PX1v35559X9UEYOHAg3n333Vrz/vzzz2YfpqWlmfUeNWoUPv744waXUVeb66uvvmr2tZQl340TTjjBrIeQ72jHjh1Ns0NN8n7179//oLaX2hjdBnz++ecm3fPUU08FPL93714dHR2tL7300qo0WZ8+ffTtt9+un3vuOX3//ffrrl276tTUVL1t27Z600cnnniiuYWS4rryyit1VFSUvuqqq/Szzz6rb7vtNpOaPvroo006rr7tuPrqq82yZd1eeeUV/e2335ppsgxZVk0110vSmvL64cOH6xEjRujHHntM33vvvTohIUEfc8wxtZYXExNjyv7jH/+on3nmGX399dfrk08+2UyXZZ9yyimmPFkX381HnpfX+SxfvtxsZ2Zmpn7ggQf0X//6V92rVy8dGxurFyxYUDWfvK++dRw/frzZbzfffLN2u936vPPO0w2RlLm8v/369dMPPfSQvu+++3THjh11+/btzb7zrfudd95pliPbJOst2xvMhg0bzDwyr7zGt505OTlV733//v11586dzfS///3v+sgjj9RKKbPN/k0QQ4YM0R06dDDzyb6Xz57Md8MNNzS4XfJeyjrI+y/vyXXXXWfek5qfG9nfXbp00Z06dTLzPPnkk3rMmDHmtc8//7yZR9ZdPkPynHymfNsk2xrsc+v7zA8bNkwPHDhQP/roo/quu+4yn49Ro0aZ7Tn22GPNsuS9km26/PLLA9Z/ypQpZv89/PDD5rN07rnnmjJvueWWg0oLyzyyj9u1a2e+s7JOgwcP1i6XK2BfyrbKvklOTtZ/+MMfzHxDhw4187377rv1fq9977k/+b7Ic7K9si1PPPGEnjp1qvke+z5/Mv3DDz8MeN2OHTvM/pL3nahNBNeKigpzQB89enTA83Jwky/BZ599Zh5LG5PH4wmYR75wcvD3/0KEE1y/+uor89rXXnstYL6ZM2cGfb4mX+CREwF/jQ2uhx9+uC4tLa16Xg4Q8vxPP/1U9Z5J4JNy9+/fH1Cm1+sNqc21ZnCVg6scjH0HcLF9+3Zz0DvhhBNqbaMEEf9l/e53vzMHp9zc3HrfIwkAEljk5Mln2bJl5mDqO5Hyfy+kPTXcNleZNm/evKrndu3aZT43clLgIycUcnKxdu3agNdLYJDt2rJlS53Ll/LkvZswYULAZ1QCuSz7hRdeqHpO9rc897e//a3qOdnXvvfFF4jra3OtK7imp6cHvP933HGHeV6ClX/7+oUXXmjW17/dtqioqNZyfv3rX5sTO//5GhNcZdnvvPNO1XN5eXnmuy4nZj433nijmU++ez4HDhwwn++ePXtWvZ+hBNd169aZz9FZZ51V61jh+6zK8926ddPnn39+wHQJ6nLS8fPPPze4bdT2RXxaWLjdblxwwQWYP39+QG9PSQlLSlZSk0JSVS6Xs8mSqpKeuZK6lDTOkiVLrKzLf/7zH6SmpuKUU07Bnj17qm4jRowwy5o9ezaag6QCpYe0j6RQfekzX/p248aNJoVZs03yYIYmyPspabMpU6aYlKJPZmYmpk6dalKq+fn5Aa+5+uqrA5Yl6yjlbN68uc7l7Nixw/R+lbSmpAB9JJ0t77n0bm4Kko70vYdCUtnyufG9n759L/NIGtF/35988slmu+bNm1dn+bNmzUJZWZnZH77PqLjqqqtMir1milPaCqWpwUf2tTyWphFJFx8sSa3K59dn5MiR5u/FF18c0L4uz8v6btu2reo5SRf7SO992XZ5P4qKirB69eqDWh9JMUtzgY+8F5deeqn5/Obk5JjnZJ9LCnfMmDFV88l3TT5fcjyQnuChev/9900K/5577gnYD8L3WZXnpQ/Hf//734BRCq+99ppJ00uKnqhNBFfh67Dk69i0detWfPXVVyboSvAV8qWRIRmHHXaYCbTSbiIHyR9//BF5eXlW1mPdunWmrE6dOpmy/W8FBQVB24WbQvfu3QMeywFf+Np9N2zYYP5K250N0lYqB9Fg7U2HH364ee+l7a4x6xiML/DWtRw5oEsHNttqrqtvff3XVfa9jFGuud8luIr69n1d2yVBU05Wap5wSNCpOUSrX79+5m84w4lqbqcv0EobcrDn/bd/xYoVJhDKNAmCsu0SlMXBfr+kzbrmyV7N7ZT3pq7Pg296qOR7IcFTTqbqIwG+uLgY7733nnm8Zs0ac1JzySWXhLwsattab1fPRpKa4YABA/DGG2/gzjvvNH8lc+nfS/gvf/kL7r77bvzqV78yHS2k5iNfJKktNNThRL7gTiY0UM3OGlKOBFY5iw1GDjgHo67apCzfd/LgL9hzItg2tJRIWMfGrKvse6k933rrrUHn9QWF1qyu7Wxo+6XTlXQgk6B6//33m85M0gFJMkK33XZbyB26IoUEXznmSMcnCbTyV06EZIw6UZsKrkICqQRPqYlKDVZqqL4ek+L//u//TG/Q559/PuB1cmCQWmx9pJbinwL0qXlWLAcVSfEdd9xxAWmycMnyg12pSZbvn4YNlaynWL58eVXNKphQU8Ry0iA9K+UMviZJCcpJTM3az8HwXQSiruXIfjyYi27YuEqPvKeSnajv/Qxlu/z3p6ReJX1fs0y5QITU0P23de3ateavrxdwc155SHonSzOL9OSVnrU+su7hkB7uEsD9t6Xmdsp7V9fnwTe9MftQTgQklSw9lOsjQVV6u0tThRxvpKe0L/tC1GbSwsJXS5X2EmmXqzm2Vc6+a9aKpJ3Mv92ovi+dfFn9h4rIEB4ZTuJPzlylNik145oqKioO+lKGsnwZ0iIHW5+PPvqoVqo1VEceeaRpG5IhQzXXyf898h28G1pveW9lGIIMx/BPS+7cudMceKQ9TGo14ZI2XDnovfTSSwHrJCcJ0uYrw4oORqjbWR/Z99Lu/9lnn9WaJuXK/q+LBE+p+Tz55JMB77+cCEpKVQ7c/qSs5557ruqxfC7ksZzkSI3K1jaFylez9V93WScZWhQOOYnwpV6FtNvLECj5DGRkZJjnZJ8vWrTIvPc+cuLxz3/+0wTghlK8/qTPgJwISu27Zm275rHjwgsvNEH/hhtuMCfevhQ4UZuruUqwkA4FcoAXNYOrjJmTL4109pH5fvrpJ5O+DaXmJ6nkRx99FBMnTsQVV1xh2s9kXKGMSfTvqCOpMelYMn36dBPgJeDIgH1pj5NALmNIzznnnEZv25VXXmlq3qeeeqo5iEvbkKSifDXQxpIDyDPPPIMzzjjDHKjkPZHAJScQ0nbmCxC+A/X1119vtt3XeSyYP/3pT/jiiy9MIP3Nb35jOsDIAb+0tBQPPfQQbHn44YfNWNrRo0ebfSFtX0899ZRp6zvYSzXKeyDbJmNSJZhJm/z48eNNij9Uv//9700nF/mcSYcree/kIC+fM9l3ctJRV4ZEguIdd9yB++67z+xjGZcttTEJTpJ9qXngljZXWVcpU9LNb731lvm8SUCRz5uQz4Z0VpPPqYz1lmArHZGaosONfJ+k1iZjQOWzIkFHLjcZbopftk32sVwjWjonvvDCC+aE7cUXX6ya5/bbbzfNQPKZkGVLc4+cfEmt+Z133qnVMamhNt4//OEP5uRYOmPJGHr5LMjy5T2X77X/PpN9Jd9reZ9rngDRIU63MU8//bTpWl9zTKeQ4QAydEK68sfHx+vjjjtOz58/v9Zwlrouf/jqq6/q3r17myEIMuxBhvjUNazgn//8pxlnKsuRoSgyPu/WW281Q1MOZiiOkKEXMi5XhoDIun///fd1DsWpOfykrm36+uuvzVhWWUcZRiLjNP3HC8uQnd/+9rdmiIYMM/D/yNQciiOWLFmiJ06cqJOSkswQjHHjxlWN1W1oG33rHmw4TE2zZs0y74G8vykpKebylitXrgxaXihDcYRcJlH2rwyb8V+Pui5/GGx4lgwBkeErffv2NZ8TGX8r4yUfeeSResc4+w+9GTBggBmfLWM3r7nmmlpDpWSZgwYNMvtfhp/FxcWZdZTX1vTBBx+YcasyLth//9c1FEfGdYbyHgbbh998840ZEyv7RMbhyuddviM192ljhuLI+y5lyOdSPvfy3gTbnzL865xzzjFjYuX9kO//Rx99FDBPqONchQx9kuE+skwZPy3vebBLkr799ttVY4mJ/Cn5r6UDPBGFTq7QJL2iJRXelklKV3qzS/NHayVZMkklyzAr/6FaRG2qzZWIqDn961//Ms1K/mNsidpcmysRUXN48803zagEubiH9KPgb8JSTQyuRESNJD2F5SpQ0tlKOu8R1cQ2VyIiIsvY5kpERNQSaWEZTC2DuWWsHNsWiIgijyQp5YcGZLxuY8b+NkZJSUnAhW7CIRdVieQfsg8puEpgtXHpOiIiallyVTf50fumCKy9enVFTs4+K+XJFbjkQiCRGmBDCq5SY/XtFBuXsCMiouYlV5KTSpLveG6b1FglsG7a/DZSUhLCKis/vwg9e5xnymzTwdWXCpbAyuBKRBS5mrppLyUpDilJYf5oSRv4FSUOxSEiInskMHrDDI5tILiytzAREZFlrLkSEZE9rLkaDK5ERGSPXJdIh3ltojZwbSOmhYmIiCxjzZWIiOzxagtp4civuTK4EhGRPWxzNZgWJiIisow1VyIisoc1V4PBlYiI7GFwNRhciYjIHm0huEoZEY5trkRERJax5kpERNYo7TW3cMuIdAyuRERkD9tcDaaFiYiILGPNlYiILF+hSYdfRoRjcCUiInuYFjaYFiYiIrKMNVciIrKHNVeDwZWIiCz/nqs3/DIiHNPCRERElrHmSkRE9jAtbDC4EhGRPRyKYzC4EhGRPay5GmxzJSIisow1VyIisoc/OWcwuBIRkTXK6zW3cMuIdEwLExERWcaaKxERWb6IhA6/jAjH4EpERPawt7DBtDAREZFlrLkSEZE9rLkaDK5ERGQPr9BkMLgSEZE9rLkabHMlIiKyjDVXIiKynBb2hl9GhGNwJSIiezjO1WBamIiIyDLWXImIyB52aDIYXImIyB5J6XqZFmZamIiIItq8efNwxhlnoEuXLlBK4f333w+Yftlll5nn/W+nnnpqk64Ta65ERBTRaeHCwkIMHToUv/rVr3D22WcHnUeC6Ysvvlj1ODY2Fk2JwZWIiCI6uE6aNMnc6iPBNCMjA82FaWEiImqV8vPzA26lpaUHXdacOXPQqVMn9O/fH9dccw327t2LpsTgSkRE9q8t7A3zBiArKwupqalVt+nTpx/UKklK+OWXX8aXX36JBx98EHPnzjU1XY/Hg6bCtDAREdmjvc4t3DIAZGdnIyUlJex20gsuuKDq/uDBgzFkyBD06dPH1GZPOukkNAXWXImIqFXWXFNSUgJutjoh9e7dGx07dsT69evRVBhciYjokLJ161bT5pqZmdlky2BamIiIIrq3cEFBQUAtdOPGjVi6dCnS0tLM7b777sMvf/lL01t4w4YNuPXWW9G3b19MnDgRTYXBlYiIIvrH0r///nuMGzeu6vFNN91k/k6bNg3PPPMMfvzxR7z00kvIzc01F5qYMGECHnjggSYd69qo4Op96WZ442PQZsRGo61RCW1o/4j4ph3o3RLUaQ+grdGf3I22pi3up7Zq7Nix0PVcMvGzzz5Dc2PNlYiI7OHvuRoMrkREFNFp4daIvYWJiIgsY82ViIgssnARCSkjwjG4EhGRPUwLG0wLExERWcaaKxER2cOaq8HgSkREEX2FptaIwZWIiOxhzdVgmysREZFlrLkSEZE9rLkaDK5ERGQP21wNpoWJiIgsY82ViIjskV+n0WGmdcN9fSvA4EpERPawzdVgWpiIiMgy1lyJiMge1lwNBlciIrJHfhHHG2Zv37B/VaflMS1MRERkGWuuRERkD9PCBoMrERHZIxldb7jBFRHPfnAdeCLUYSOBtC5A9kroWf+smqRGnA70GAK0ywBWzoVe8E7D5Q2dADVgDBCXBBTlQs95Gdi9CUjvCTViMtCxu5QM7NnslJebY32T0H8MVJ9jgHZdgO2roOc87zwflwR11BSgU18gOg4o2AO97FNg64rg5bjcUCddA7TrDLiigeI86FVzgHXzq6ePuQTo0B0qKQ1eWU72T2gSfY4FehwFpGQAO1cD81+unjbyYqBDTyAqBigrAjYtAlb/r+6y4lKAEecAHXsDZYXAqi+d1/h0Ogw44jQgqSNQnAv8+CGwc63d7ekxEug2HEjqDOxeByx5vXra8AuA9t0BdzRQXgxkLwY2zK27rI59gP4TgMQOZh9h1afAnvXV0xPTgcFnAqmZQEk+sOozYNdqu9vTVtncTzLvgFOBpHTAUwZsWwqsmSUNdqFNp6bBmmsTBVcJgEtnQnUZACS2C5ik83cDi96HGnBcSEWpo34BZPSF/vQpQF6blAZ4KpyJsQnQaxcA/3sBqCiDGj4J6tRrod+6x/4A5KJ86J++gMroF7hNUbHQ+7YBSz4086DbQKjjL4X+5FEgb2ftcrQX+rt3nGnSYJ/aGeqU66Dl8a6fnVl2bQRWzwMkyDal4nxg9ZfOiUF8auC0VbOAgt2A1wPEtwPGXAEU7geyfwhe1jFTgcK9wEf3OcF6zJXmRAN7fgYS04DRlwILXwdyVgMZA4BRlwKzHgUK99nbnpIDwPq5QIfeQFyN7Vk/Gyjc42yPTDv6UifIb19Wu5z49sCRFwJL3wZ2rXNODOTxV38HivcDygUcdRGw/Udg0QznhGLYecDX/wCKLG5PW2VrP8kJ9YiLgI3fAPP/5XyGR/4KKJLP6fchTCeKtA5Nm5YBm38ESgtqT1u3ENi6Eigrabic2ATgiPHQ8151Aqso2OcEBSHl/LwYKCs2X0b94yxT2zMB2LbsH50aZGlh4PMFe4GVs4GiPOdsWGqsebuAjj2DlyNBP3dHdU84cw6ggeSOzmM5qKye6wTapr5CyfblwPYVQGlR7Wn5Oc66+FZS1kVqncFI8JTtXf4p4CkH9mc7QbjnUc70zv2B/duAnFVOWfJ3XzbQfYTd7dm5Eti5CigPsj0HdgZuj9ykVhpM+mFA/g5gl9SstfM3byvQbZgzPa0nEJMArJ8DeCuc6fs2AV0rp1Pz7KfoOCjZD1vlhE87QXjPBiC5c2jTqclor7Zyi3Stt801vZdz8OpzlFPTlS/dz4uhF3/k9wX0k9kXWgJFwX60GEldp3YG9m+vdzY17iogsx+UO9qp+W5potRvOIZNMWljFRUDLTXMzXWc7UtqVE54/E+mcrcDfUY795Vybv7kcWoGmtWg0006UrljoKX2snVJ8PlqrqvzJJBcub5ycD6wK3CogJyM8KDdvPupvBha0sZZI4AN85yaqaTzV3wY2nRqOrz8YSsPrnEJUDHxQGo69H/uNzVZNfEaoLwUWDozcN7E9lDHXQi98N2WGx8l7aXHXwpsXurUzOqhZ//LHMR1em+gc1+nPai1Wfo+sPQD6HZdgS4DnTawYKJigfIamQiZV54XO9cBgycDXQYBO1YBmYcDHXo4KePmtOIjYMXH0HIy0GlA7XX2kdrNgIlA58OBXWuATv2dtjupnQrZrop6tpeaZz+JHcuBwVOAvmOhXG7oTQucdtxQpxMdkuNcJYhKIFr8MVBRatr89PLZUN0HB86X0A7qtOuhV84D1lZ2DGqJwHrC5UBFOfSCt0I/M9u1ASo+CRg0Hq2TpLG3Ou//4NODzyLTpDOXv+h453khbbcLXwMOPxk4/R6g59HA1mXB09FNTgN52511O3xi8Fmkze+Ht4G+44CTbgO6HekcpKVjl5DX1gyksv2+7aXm2U+JHYERU53OZp/dD/3lg07HJemIFsp0avoOTd4wbxGu9dZc925reB4JrJNvgF7/HbDsM7RcYL3M/NVz/h08Zd3Q65PTW3f/RenEU1eba94OID4FiE2sbpNulwnk+fXa3rHSufmMuw7YvBgtxuUGEupoyxPS89e/9++xVwNbl1a3C/Yd67wnvixJimxv/U0BZHk/SRpeemrnVPbMl2YJaV/tczyw5vOGp1PTYW/hJqq5ykHHHeX8lfYruS9fklrT/O4HU7AXetsq0wvYdM1PSIUaNBZ6y4/OdHk8+QanU9MPn1jfjFrb5JLtkHVVlffd5nkTWKVdUobNNBRY23c1ba1me6TMrgOBXiOgt/sdyKVcKd9/uUHbAS1tk3n/ffflYNYO6HIE4I5xtjWtB9B3DLBzTfBypD12zyZgUOV+ap8FZA0HNn1XPU+7bs5ypMY34GSnQ1Bdbbg2tkfeL9/2SK/TjIHV29MuC+g5KnBoTU2pXSo/nzFOII1OALZV9pSW9LCkgfue6Owr6QAlnZxkmAc1337K2wbEJTvpe5lfPlPSqUw6o4UynaiJKa0bbjnOz89Hamoq9j95JVLiY+ov8MjToI6cHPCc3rEW+uMnoE64BKrfqMBpaxdAz3vFee0v74Je+hmw4bvqcaRjpgJd+jttL+sXQX//oVNjGH4aXCMmQ1emj6vKm/k0sHNDaFsfGx3SbGrIqVBDTw1cTs56M6bVNfG30BVlAQ3wevkXwHIZTweoM26DlvsbFwNpWVAjzwVSOjmpr4J90Gu/AdZ9W72ss+5xej378X7zOvDzotDWNaH+/VPl8FOgBp4SuE27NwDfvwUcfaEzpEYOfnL2v3kJsGZ29fjAU24CVs+uHpoTMM61yBnK4z/OVYbmpMl4ZO20wco4Vxk/Gor4ENsyDxsHdVhgel3v3QgsewcYdk5lhyMFlB4Ati1zOrn4tuf43zrjKWV4jThmGpDarboNdtUnzvvgI+nFI2Sca5fKca4zGzXOVZ32ANoa/cndzb+fpE32sHFAQprT+VH21cpPqnsiNzT9ENtPvuN4Xl4eUlJSmqz8/Q9f3mCcaLCs4jK0//2LTbauERlcI0qIwTWShBxcI0WowTWCtLWDdqOCawRpa/up2YLrg5fZCa63zYjo4Np6OzQRERFFqNbboYmIiCKOJEN1mB2SQkiotnoMrkREZA97CxsMrkREZA+Dq8E2VyIiimjz5s3DGWecgS5dukAphffff79Wmvmee+5BZmYm4uPjcfLJJ2Pduqa9WheDKxERRfQVmgoLCzF06FA8/fTTQac/9NBDePLJJ/Hss89i4cKFSExMxMSJE1FSEsKPyBwkpoWJiCiiL9w/adIkcwtelMbjjz+Ou+66C2eeeaZ57uWXX0bnzp1NDfeCCy5AU2DNlYiIWqX8/PyAW2lp46/hvXHjRuTk5JhUsI+Mxx05ciTmz2+669EzuBIRkTVyAT1t4SaysrJMIPTdpk+f3uj1kcAqpKbqTx77pjUFpoWJiKhV9hbOzs4OuEJTbGzkXLGNNVciImqVUlJSAm4HE1wzMjLM3507dwY8L49905oCgysREbXZ33Pt1auXCaJffvll1XPSfiu9hkePHo2mwrQwERFZ499merAa+/qCggKsX78+oBPT0qVLkZaWhu7du+PGG2/En/70Jxx22GEm2N59991mTOyUKVPQVBhciYgoon3//fcYN25c1eObbrrJ/J02bRpmzJiBW2+91YyFvfrqq5Gbm4sxY8Zg5syZiIuLa7J1YnAlIiJ7tIW0biPHuY4dO7bei/3LVZvuv/9+c2suDK5ERGSPpHS9FsqIcAyuRERkjfzcnA73J+d44X4iIiKqiTVXIiKyh2lhg8GViIjskYyutlBGhGNamIiIqCVrrq5pf4PL7zqPkc7772vR1qhLHkdbouf+BW2NXvQo2pzE+JZeA2ol2KHJwbQwERHZwzZXg2lhIiIiy1hzJSKiiL62cGvE4EpERPYwLWwwLUxERGQZa65ERGQN08IOBlciIrJHRtF4LZQR4RhciYjIGvnlN928vzjXKrHNlYiIyDLWXImIyBq2uToYXImIyB4OxTGYFiYiIrKMNVciIrKGaWEHgysREVnD3sIOpoWJiIgsY82ViIjs8SrnFm4ZEY7BlYiIrGGbq4NpYSIiIstYcyUiImu0VuYWbhmRjsGViIisYVrYweBKRER2h+J4wy8j0jG4hmLgiVCHjQTSugDZK6Fn/bNqkhpxOtBjCNAuA1g5F3rBOw2XN3QC1IAxQFwSUJQLPedlYPcmZ1q7zlDHXwR0yAIK90MvfA/Y8lMTblwb0WUEkDEESEwH9m0AVgTZD9GJwNFXA6X5wOLng5eTmgUMPj/wOVc0sO17YMMX1c9ljQa6DAeiE4DSA8DqD4ED2+1uU6dhQPpAIL4jkLcJWPff6mkDzgWSMgOPYj++CJQX1l1e+hFAxlFATDJQUQRsngPkbqh+b3pNAJK7ARXFwPaFwO6fIns/xSQB/U4D2nUHyouBzd8AOUvtbxNREAyuoZAAuHQmVJcBQGK7gEk6fzew6H2oAceFVJQ66hdARl/oT58C5LVJaYCnonKiC+qUa4AN30F/8iTQZQDU+F9BvzfdmZfqVlbgHDzb9wRik4PPc9gEoGAnEB1fdzl52cDXjwQe6EddB+xeWf1crxOB1O7AsjeAkv1AbAqgPbCuvADYthBI7eEEipqyvwJ2/hBaWemDgYwjgQ0fA0W7gagEwB1dPb3PZKA0F/jhGSeY9z/b2bYDWxGx++nwKUDxfuDbJ5xgPuQCoHgfkLfF4gZRTWxzdbC3cCg2LQM2/wiUFtSetm4hsHUlUFbScDmxCcAR46HnvVodLAv2AcX5zv3Mw4C4ROgfPnUCbvZyYMc6qL7HWN6gNmjPGmDvWqeGEkyHw4CoeGBnI2tjGYOdA3T+NudxVBzQbSSw5mMn+AipYZXVU2M8WPvXOzVLqUmGRQFdj3VqqhJYhdRcS/Oc+7GpQLJkZb4GvBVAYQ6wdzWQPggRu5/i2gGp3YCNcwBvuZNV2LnCqTVT0/Iq6DBvHOdKjZPeyzl49TnKqel6PcDPi6EXf+Tcl7Tz/h2Bqb59W53n6eC5Y4E+JwM/vQmkdGvcazOGAjv8aocpXZ192GkgkDncqbHuWgVsmtv8vTC6jAK6jnaCe85iYO+q4PPFtYeKSYRO7AT0OtlkSJC7CdgyF/CWAQnpTjpZAq6PBOFOQxGx+0m2VWrJ/mlyqQ13OdLe+hLVg8G1OcUlQMXEA6np0P+539Rk1cRrgPJSYOlMICoWKCsKTI+UFkNFx7XYKrcJvccDOT86NZvGHLSlXU9qQDuXVz8XFQcVFQcdnwYsetZJXR5xHuApA7Z8g2az9WugeK8T6FOygD6nOzU0qe3WJLVtkdIdWPFadRq4x1hg4+dOW2VFaeBrKkoAdwwidj/Jugfbpqhm3qZDEK8t7GBauDlJEJUPzuKPnS++dFhaPhuq+2BnujwnwdePiokDykNIOVPdB15JD2bPb/xrpTa0dx1Q7nfC4yl3/m6a5wQzqTVu+85JZzangh1OQJfact5mYPePQFq/4PPKeoodi5wAIze536539XSpNfqTEz0pP2L3U1nwbapoxm06xNtcdZi3SMeaa3PaW9keVJd924Fhk5y0nS/F2KEbsCe7WVavTWrX06nVjL7eeazcTkeeY28Evv9X3W2lUvNJHwCseDfw+cKdaJXqO9Uv3g8tNdy6SAo4JtFp6/S17yZ0Aor3IHL30y4gNsnpze0LukmdgUJ2DKTmwZprKCTYuaOcv0o5913uINP87gdTsBd62yqo4ZOcA0dCKtSgsdBbfnSm71gHlBZBDT8VcEUB3QYBmf2g1y9qvm2NWMo5IJv33u/+1oVO+vb7552b1DiL9jr3a6TgA3Qa5HS62f9z4PMledD7NwI9xjj7SHrxdj3K6aTTVNskf/23SWpkqb2c5cvzkhbuNATYty54MboC2LMKyDzaea3c5P7+ymE40rFJOvx0q9ymxAygwwBg9/II3k+5QN5WoNdYZ5uSM515c5Y1wTaRv3A7M2lfp6ZGuPfee6GUCrgNGDAALYk11xBIsFNHTq5+fPkT0DvWQn/8hBmTqvqNqp4mwXLtAuh5rziPf3kX9NLPzPAaoWfPgBozFeqivzrpXgmcyyrH5Wkv9BfPQh0/FWrIKUBhrpmfw3BC0GMMVM/jqx+fcBt07mZg2WuB6U1JiUpWoOxA9XNHXQVs+RbYtSIw1Sjtf8Gs+gDoNwkYfYNTtrT1ZS+wv01dR0FJhyWfo2+Azs8G1n9kpiH+NOd5SU1L56T9fsG131nAgW1O+ldsmQ30OAkYeoXTCUuC0ZY51fNv+ATodQow/BrnPZJhPraH4TT7fnof6D/Zqf3Kd+3n2RyG04bbXAcNGoRZs2ZVPY6KatnwprRueDPy8/ORmpqKvLw8pKSkoK3w/vtatDWuK59GW6Ln/gVtTnwb7KBW3Pb6BagT70Rb0tTHcV/5635xMZKjw+s4dqC8DIf999WQ11Vqru+//z6WLm09FwlhWpiIiFplh6b8/PyAW2lpjR7gftatW4cuXbqgd+/euOiii7BlS8tmKRhciYjIGq9XWbmJrKwsUxv23aZPnx50mSNHjsSMGTMwc+ZMPPPMM9i4cSOOP/54HDjg16zQzNjmSkRErbLNNTs7OyAtHBtbY3hVpUmTJlXdHzJkiAm2PXr0wNtvv40rrrgCLYHBlYiIWqWUlJSDah9u164d+vXrh/Xrg1xUpZkwLUxERG3qIhIFBQXYsGEDMjMz0VIYXImIKKKD6y233IK5c+di06ZN+Pbbb3HWWWfB7XbjwgsvREthWpiIiCLa1q1bTSDdu3cv0tPTMWbMGCxYsMDcbykMrkREZI1XK3MLt4zGePPNN9HaMLgSEZE1B3P5wprCfX1rwDZXIiIiy1hzJSIia/h7rg4GVyIissYLC22u5pegIhvTwkRERJax5kpERNbYuAiEDvP1rQGDKxERWSOB0cvgyuBKRET2sObqYJsrERGRZay5EhGRNd7KWzjCfX1rwOBKRETWMC3sYFqYiIjIskO65qqLy1t6FaghifFoa3RiAtoa1zE3tfQqUCvh1Y2/8H6wMiLdIR1ciYjILqaFHUwLExERWcaaKxERWU4LI+wyIh2DKxERWcO0sINpYSIiIstYcyUiIrs/OQf+5ByDKxERWcMfS3cwuBIRkTUyxtUb9jjXyK+5ss2ViIjIMtZciYjIGm2hzVWzzZWIiKga21wdTAsTERFZxporERFZww5NDgZXIiKyRtpLNdtcmRYmIiKyjTVXIiKyhhfudzC4EhGRNWxzdTAtTEREZBlrrkREZA07NDkYXImIyBq2uToYXImIyBrWXB1scyUiIrKMNdcQqCHjoAaMBjp2BTavgPfjf1RPjI6DGncRVK8hQEU59I+zob/7uO7C0rvDdcIFTlnFBdCLPoRevcCZ1q4TXMf+EsjoDURFA3u3w/vtO8CODU2/kZEufSjQcRAQ3wHI2wRs+LB6Wv9zgMRMQHurn1s+AygvrLu8jkcAGSOA6GSgogjIngPk/gwoN9DvLCCuA+ByA2WFwM4lwJ6f7G9T+0FQqf2B2DSgcAv01s8Dp7cbAJU2FIhOBCpKoHd+AxRsDlqU6jMViIo3dQJDe6HXzqieIbErVKeRQHQqUFEIvXM+UJhtf5uozWNa2MHgGgJdmAv9/SdQWYdDJbUPmKZOvAAqLhHeF28HEpLhmnITcGBvdcD0FxMP1y+uh174X+h3vwI69YTrzBuh8/YAO9YDsQnQm5dD/+8VoLQQauBxcJ1xPbwv/wEoKWi+DY5EEih3LARSugPRSbWnb/0a2PVDaGV1HAx0Hg5s+AQo3g1EJQCuaGeaBOgts4HifU6giktzgnfJPqBgm91tqiiC3rMEKrGrE0D9tTscKm0w9LZZQOlewB0PuOr/OuttXwIFm2pPiE6G6jbRKatgC5DUHarbKdA//wcoP2B3m6jNa6mhOE8//TQefvhh5OTkYOjQoXjqqadwzDHHoKUwLRyKDT8APy81Nc0AUTFQ/Y6Gd8EHQFkxkLsLetn/oAaOCV5OZh/AUwG9fJ7zsw87N0JvWAI1qHL+nZugV3zlBFKtoVd87RzMpZZL9ctdD+RuACqKwyxIAV1HOzVVCaxCaq5leZXTNVC8t7oG6BObCusObHSCoaek1jqq9KOgd37rBFbhKT74QJiUBZTscQKrkL/Fu6BS+4W3/kTN5K233sJNN92EP/7xj1iyZIkJrhMnTsSuXbvQUlhzDUe7zlDuaGB3dfpM78mGOmpS8PlVkLMxpaA6dKt5qHZ0kBpLHLBvh711PlRljgS6jALK8p007t5VweeLaw8VnQid0BnocTKgXE6aOXse4C2rnq/vmaaWrFxR0EW7ncDeXGJSoaISoOM6QmWc4HyuCrKhd80HvOV1vkxlHg/gBPMe6D2L/dK+wWoJCojt0GSbQG2XHMu0hTJEfn5+wPOxsbHmVtOjjz6Kq666Cpdffrl5/Oyzz+Ljjz/GCy+8gNtvvx0tgTXXcMTEQpeVBLbllRYBMXHB58/5GYiONW24pr0usw9Un+HB55cU8sSroBd/AhQFfsCokbZ+Ayx/AVj2nJMezhoHtOsTfN6oyn0h6eVVrwMrX3VqpVknBs63/gNgyd+hV78N7F8HeCvQbNzOOkq6WG96F3rjO0BMMlTnY+t8id7+P+j1b0CvfxV6/3KobhOAuHRnYuFW535STyeoyt+EDEBOHIkO5sfSdXg3X2/hrKwspKamVt2mT59ea3llZWVYvHgxTj755KrnXC6XeTx//ny0FNZcw1FWCkTHOLUbX4CNTQAk4AZTUgjvR3+H67hzoI45A9i/A3rlt1DSgalmYJW22B3roRf6dcyhg1PoV/PP3+x0PkrrH7y26ams+e1YZDoJVd3vfRpQq6+QdtpZ0/o5nZ9kvuZQWTvVe5ZWpYzlvup6Ut2vKc6pvp+/HkjuCZXcC7pkt0l5S3urpJrR5USgaCeQv8H5XBO1oOzsbKSkpFQ9DlZr3bNnDzweDzp37hzwvDxevXo1WgqDazhydwJeD9CxG7Dbaa9Scn9vPR1bdmyA9/8erHqoTr0KetvaGoH1Buh926Fnv9qkq3/IkvbuupTsg25sLVR6EMcGdnRrUmW5jV/Hht6Dgs3Qfj2NVc8p0Hl+n0uiEEk1w2uhDCGB1T+4RhKemoZCzuDdUZJrcNJm5r4bqCiDXvc9XKPONEERqZ2ghoyHXvl13WV1zHJ6dbqjTUcm1bU/9LJZzrToOKc3ce5O6C9fbrbNaxuUE+TkI60q75v9Fguk9qzsSauA5CwgfbCTyg1Ge5z22IyjndfKTe77arnx6U7K2CxLAam9gLQBQP6mptsmU4P02z5Zx7x1UB2GAq4YczP3D9SxDlFJQHxmZTkuILm3qblq//njOjrLkF7RHY90Us+5DK7UeFrSujr8W6g6duwIt9uNnTt3BjwvjzMyMtBSWHMNgTp6Mlwjz6h67P7NP6C3roH3vb9Bz3kDGH8xXJc/6ARbGefqNwzHBMvt66C//9Qpa+h4p51VDnQ5P5syUOj0RJXnlfQo7tgNqvfwqjKkBqvXNlPKMVJ1GQnVZXT14xHXQx/IBjZ8DGSOAnqnOc+X5judk/yD62FTgAPbgJzvnMfSU7j7eGDwr5xAJuNbs+c602S/dT3OdHwyvS6kg5RM27fG+iapjkc6qVrf4wFXQhduh97yoekprDLGQPWd6qzjgc1OhybfvL3Phd7zg5MCdkVBZRxrOkKZ5ovKNDBKqntSqvSRQHwnJ9VduA1684eAbsZ2ZKKDFBMTgxEjRuDLL7/ElClTzHNer9c8vu6669BSlNb15ciqe2xJY3JeXl7EVtGD8Tx1Ndoa92//ibZEf/8Y2hqdmIC2xnX4r1t6FaiFj+O+8l8/8hYkSMYnDEWeUkxd8kjI6ypDcaZNm4bnnnvOjG19/PHH8fbbb5s215ptsc2FNVciIoroKzSdf/752L17N+655x5zEYlhw4Zh5syZLRZYBYMrERFF/IX7r7vuuhZNA9fEDk1ERESWseZKRETW8ML9DgZXIiKyhr/n6mBamIiIyDLWXImIyBqmhR0MrkREZA2Dq4NpYSIiIstYcyUiImvYocnB4EpERNbIBXW9YaZ1G74ob+vHtDAREZFlrLkSEVGr/D3XSMbgSkRE1jT291iDCff1rQGDKxERWcOaq4NtrkRERJax5kpERNbwIhIOBlciIrJG4qK2UEakY1qYiIjIMtZciYjIclpYhV1GpDukg6su8aCtKX/gMrQl0XfPQFujt7yKtsaT91+0NTopBW1JhaewWZbDtLCDaWEiIiLLDumaKxER2cXewg4GVyIisoYXkXAwLUxERGQZa65ERGSN/Fyc5k/OMbgSEZE98kPnXv5YOoMrERHZw5qrg22uRERElrHmSkRE1rC3sIPBlYiIrOE4VwfTwkRERJax5kpERNbw2sIOBlciIrKGaWEH08JERESWseZKRETWcJyrg8GViIis4VAcB9PCRER0yOjZsyeUUgG3v/71r9aXw5orEREdUh2a7r//flx11VVVj5OTk60vg8GViIha5VCc/Pz8gOdjY2PNLVwSTDMyMtCUmBYmIiLrNVdvmDeRlZWF1NTUqtv06dOtrKOkgTt06IDhw4fj4YcfRkVFBWxjzZWIiFql7OxspKSkVD22UWu9/vrrceSRRyItLQ3ffvst7rjjDuzYsQOPPvoobGJwDYEaPh6uI44DOnaF3rgc3vf/Xj0xJg6uUy6B6jMUqCiD94f/Qc//KHhBCclwjbsAKqsfEBMP5O6G95v3oTcsC1zeyNPgGnoiEJ8MFOyH55N/Azs2Nu1Gtk+H+9RLoLr2AcrL4F30ObzzP61zdjXsBLiPnQQkpwFFB+D57DXotT8A7ii4p94Cld4FiIoGDuTCs2Am9A9zm3b926KkflCJfYCYdkDxdug9le+hOwEq84zAeZW7cp45dZeX2BcqZaB5Pbwl0Pu/B4q3Bs4TnQqVcVrg8ixSMT2horsD7mSgYhe8Rd9VT3QlwRU/GHCnAtoLXZEDXbwCgCd4YQ3OHwUVPwQqurMzvWwjdOla21sElzoMSrWXNw9AGbx6C7TOqZzuhkv1g1IdTB9Yr94GrTfXU15D8ze2vOYnv8WqLf2eqwRW/+Bal9tvvx0PPvhgvfOsWrUKAwYMwE033VT13JAhQxATE4Nf//rXplZsI3j7MLiGoiAX3vkfQfU43AkmflwnTQXiE+F57vdAQgrc590Mb/5e6BXza5cTHQu9awu8c//PlKn6DIHr9F/D8+oDwN4dTnnHnw3VrR88b/8NyN0FpHQAPPZTFgGUQtT5N8K7Zgk8bz1hAm3URb+HPrAfevmC2rMPPxHuURNR8e4zQM4WIDHFbJvh9cDz2avA7u3mgIaOXRB1yW3w7NkBnW37wNbGeYqh83+Cist0AmLV80XQW9/ym9EF1fVs6KJNDQTWw6H3fAWU7wdccYCq/fVXaaOA0t1oKtpbagKcikqHknXw40oYAV2xD7pwAaCi4UoYCcT2gy5dFbSshuZX8YOhVAy8+V8Arli4EkcD3iLo8honFGGRIFAGj1dOkEskHMDtGgyvLoXGfhN4Zd08XvkeRcPtGgovSqD1zuDb1MD8jS2vJWgLHZJ0I+e/+eabcdlll9U7T+/evYM+P3LkSJMW3rRpE/r37w9bGFxDoNctMX9Vp6zA4BoVAzXgGHhenw6UFpubd8mXcA0+Hp5gwTVvD/R3n1WXKzXW/TlQmX2gJbjGJUIdNQGeGX90AqvI39v0G9ghE+iQAe/c901wxN4ceJfOg2v4WHhqBlel4B57Njwf/NMJrKIwP3D0966ttb8maZ0ABtfGKc52/sakBQbXmhK6OQf5osr9UYuCajcUeu+3TmAVXgkENSQPAMrzTPBGtNTEmkCFcxJpapsIDK5wJUCX/+h8ZnSZqYkqd/u6D7T1zu+Giu4Cb+HXslDAWwFduhEqprvl4Cq1R/+Tmnxo5EKpVGidB6U6weP9wVkHVJiapktlwhM0GLoamL+h6Yeu9PR0czsYS5cuhcvlQqdOnayuE4NrONIyoCT1uavyICjk/qjJob0+IRlIy4Te7bxeZfYGPOVQh4900sKeCujV38H79XtO0GsqSgX+rbyvOstBu4YOmVBJkjrsCffkywGXC3r9T/B88QZQVn3Adp9/I1TvQeb90Tu3QK92TlDIPpXYFyjaWPfQ+6gUKHc8dEwaVNpIJxCXbIfevwTQ5c487kSo5AHQOZ+Yvy1Bl26Ais6C9uSZ2pmKyoQu33xw87sSoSRV7qk+8dPefCj3YU28FS4opMCr5eQ4AUpJn9ECv5UuAFT3Ol7b0PyNLa9ltOahOPPnz8fChQsxbtw402NYHv/ud7/DxRdfjPbt7Z5QMriGQ9K8ElAk/VlJlxaZdtgGudwmJazXfAfsrDwgxCdCxSZAte8Ez7/vNDVZ9y+vhyovqbsd14a9OUDuHrjGngXvnPdMLdM19HggNr7WrCo+0fnbayAq/n2vue8++xq4J0yF56MXqubzvPW4E6Cz+kH16G/ao6kJuBOBuAzonHpOXtwx5o8y8znt6KrjGKj2I6D3OZkJCbo6dxngbbn9pCt2wRU/DCrlNBNEdPkO6LItBze/ioLWUrvzO0qbE4mmPeS5VH9oFEFDUutSe/UErIM2Nc661sHdwPwNTW8dWvOv4sTGxuLNN9/Evffei9LSUvTq1csEV/92WFta116JNOWlQHQMIGeTlQFWSUDyq8HVGVjP/I3TAeqzl6qfLys1f7zffOCUXV4K7+JZphbracrg6vWg4u0n4D5lKqJufAzI3w/vsq/hOnJsrVl11Tp+BBQXVN2XAFt7Zg29ZQ3UwGPgGj0J3q8/bLptOESppD5A2X6gPLfumbxOm73OWwF4S6vuS4A1Eno5WQtT+20p0aZNVJeshi7bVNkZaTBU/JHQxYsbP78JrO7KNlHtd7hruv4LTsem+Mr2VyGB0BWwDqredWho/saWRzVJL+EFC2r3I2kKDK7h2JcDeDyAtMX6ap+dugO7t9UfWH9xDeB2w/ve3wPSvb70cIvYvR2e1x+peug66VwTGGvZuwO6vHG1G+V2A2mdbawl1ZTYGzpfesjWoyIfujLABiM1WsR0hOp6TuUTUU7Woesvobe9g2Zh2pTdpkevo9wETVfiKOjig5jfW+ikyV0pgDfP2Sxp5/UcaMLAmlIZWH3f6aLKIJhYncpVSdJJoY5SGpq/seW1jNacFm5OvIhEKKRm6o4ygdGc4fvuV5SZtK5rzBRnaE27TnANPwnen+bVH1hjYp3AWrMXcN4eeDetgGv0GaazFBLbwXXkSfCuX9r029ipm1MLd7mhBowwaWHPV/+tPV9FOfRP8+E6djIQlwDEJpj7eo10sgDQuTtUr0HOMBzlguo7FOqI0fD+vLzpt6HNUX41Fd99v6+s9CKWHreF9fQSFpJKLNroDMNRkmmJdu5XdpjS+xdD7/jQtLfKDQXrgJKdzv1m2SYFeApMbVOG6jjPu6FiegDSnhpMg/N7oMu3wxUn7cfyfZUml17QZZubKLCmVgZW/++0F1rvgsvVq7IWHQ+X6gqvruzUVUtD8ze2vJahLf2LdKy5hsA1+nS4jjuz+vFNz0FvWQ3PWw/DO+s1uCZcCvc1jzjjQ2Wcq19PYdcvb4TeuhZ64SdAlz5wHTbc1Pzc1z1RNY93wcfOdLn/8b/gmjAN7msfM72P9coF0ItmNv02Sup2xHgTFKUDkuftJ6t6/bovvAl6y1onFSyHrc9fg3vSpYj67SNOsF271OnQZM5DXHCN/yWU9ECWnsNywvDFG0GH9FD9VOpgqNQh1Y+7T4WWoLfrC7+OTJurOyX5vzZ9HHTpLqCyVitjWlX7Y6C6TnGCbfFWE1SdiWWAxy8bIeXJPJ5i+9sU2w+uuOrhDu7U06Er9sBb+C28RYvgijscKu5w57Pj2Qdv0Q9+nYNHQnv2QZeuM8Gzofl18U9A/BC4UiaY7THjXK32FBaxcLm6Qmsv3K7R1cvWO+HVa+HV6+BCv8ppvnGp1T17Xa7Bplex1k5bcUPzNzSdWg+ldcO/nCfXd5RLT+Xl5YU0oDdSVDx8BdoaXdKEvYpbQPTdM9DWeLe8irZGp7ad44KPTmpb25SfX4gOaac32XHcFydu6HEHYmuMYW6sUm8Jntg8PaJjDmuuRER0SPQWbk4MrkREZA07NDnYoYmIiMgy1lyJiMga6cWjw722cBuouTK4EhGRNd66L8QZsnBf3xowLUxERGQZa65ERGQNOzQ5GFyJiMgeC22uaAPBlWlhIiIiy1hzJSIia9ihycHgSkRE1nAojoNpYSIiIstYcyUiImuYFnYwuBIRkTXyQ2s6zLxuuK9vDRhciYjIGo5zdbDNlYiIyDLWXImIyBr+nquDwZWIiKxhWtjBtDAREZFlrLkSEZE1rLk6GFyJiMhym6sOu4xId0gHV13iQVvj2V+BtiRGtb2PqHfhQ2hrtKvttTBFpf4CbUmUO7+lV+GQ0vaOXERE1GKYFnYwuBIRkTW8cL+DwZWIiKyR9lZv2G2ukR9d215DCRERUQtjzZWIiKxhWtjB4EpERNbwJ+ccTAsTERFZxporERFZw99zdTC4EhGRNRzn6mBamIiIDhl//vOfceyxxyIhIQHt2rULOs+WLVswefJkM0+nTp3w+9//HhUVjbv6HWuuRERkjdfCOFdvE45zLSsrw7nnnovRo0fj+eefrzXd4/GYwJqRkYFvv/0WO3bswKWXXoro6Gj85S9/CXk5DK5ERGT3wv06/DKayn333Wf+zpgxI+j0zz//HCtXrsSsWbPQuXNnDBs2DA888ABuu+023HvvvYiJiQlpOUwLExFRq5Sfnx9wKy0tbfJlzp8/H4MHDzaB1WfixIlm+StWrAi5HAZXIiKynhb2hnkTWVlZSE1NrbpNnz69ydc/JycnILAK32OZFiqmhYmIyO4VmhB+GSI7OxspKSlVz8fGxgad//bbb8eDDz5Yb5mrVq3CgAED0FwYXImIqFV2aEpJSQkIrnW5+eabcdlll9U7T+/evUNatnRkWrRoUcBzO3furJoWKgZXIiKKaOnp6eZmg/QiluE6u3btMsNwxBdffGGC/MCBA0Muh8GViIis8WoLNdcmvEKTjGHdt2+f+SvDbpYuXWqe79u3L5KSkjBhwgQTRC+55BI89NBDpp31rrvuwrXXXltnWjoYBlciIrJGfotVt+Lfc73nnnvw0ksvVT0ePny4+Tt79myMHTsWbrcbH330Ea655hpTi01MTMS0adNw//33N2o5DK5ERHTImDFjRp1jXH169OiBTz75JKzlMLgSEZE12sJPxmlEPgbXg9E+He5TL4Hq2gcoL4N30efwzv+0ztnVsBPgPnYSkJwGFB2A57PXoNf+ALij4J56C1R6FyAqGjiQC8+CmdA/zEWzS22P6LOnwdW7v/lke9evRPk7M4DCA7VmdY85Be6jj4fKzIJ31TKUv/h47XlGjoV73GSo1PamjPL3XoF3xZIm3YTTTjsNt932ewwefATKy8sxb95XuPHGm7Bt27aqec488xd4+OEH0bVrVyxZ8gOuvPJqrFmzps4yG5q/seU1SqdhQPpAIL4jkLcJWPff6mkDzgWSMgHtdxj78UWgvDB4WfXNHxUPdB8LpHQD3DFASR6w7Vsg92fYphL7QSX2AqLbASXb4d37lTPBnQBX58k1ZnZXzjOv7vIS+kAlH25eD28JvLmLgZLK/R2TDlfqcCA6BdAV0IUbofOXWd8miqzLHzYXBtfGUgpR598I75ol8Lz1hAm0URf9HvrAfujlC2rPPvxEuEdNRMW7zwA5W4DEFCC6slHc64Hns1eB3dudg17HLoi65DZ49uyAzl7brJslgVWUPnCj2cboi36D6LMuRfmrT9eaV+ftR8UXH8DVbxBUalqt6e5R4+A+8VSUv/J36G2bgaQUqJjQOwIcrNTUFDz44MOYO3eu+cmqp556Am+//SaOO+54M71fv3547bVXcP75U82lze688w588MG7GDRoiOnYUFND8ze2vEYrLwC2LQRSewAxSbWnZ38F7Pwh9PLqml8CatEuZ7oss11voM9kYMVrQMk+2KQ9RdD5K6DiMqDc8dUTPEXwbv+P35wuuDKnQBdtrrMsldgHKmkAvPu+Acr3A644JyA7U+HqcAJ0wSro3auc4J1+EuAphC5cb3WbiILhFZoaq0Mm0CED3rnvm+CIvTnwLp0H1/CxtedVCu6xZ5uaqgmsojAfyN3t3Jcecbu2+tUmKs/W0pzu381JdegEz7KFQFkpUFoCz9IFUJndgs7r/el7eJcvBgoLghSkEHXqL1Hx3itOYBUF+dD7Kre5Cb3xxpumnaSwsBBFRUV4/PEnMXLkMaaDgrj44oswe/YcfPzxx+Yyag888CfT1f74453gW1ND8ze2vEbbvx7I3QBUFKNJleYBOYudwCqkxipBVWq6tpVsdW7e+i9jp+K7mc+SLs6uaw6olCFOTVUCq/CWmODpTI6Gcsea2qr5XklQLclxaszULL/nqsO8RTrWXBtLqcC/lfdV5yCBqEMmVFIqVEZPuCdfDrhc0Ot/gueLN4CykqrZ3OffCNV7EFRUNPTOLdCrmzZ9Goxn7qdwDx0J70rplq7gHj4a3pWNqBVVUp0yoVLaQXXrhdjzrgBcbnhWL0PFB68DpU0cJGo48cQTzFVZfLXIIUMGY+nS6rSg/ITUypWrzPNz5syp9fqG5m9sedZ1GQV0HQ2U5jvBce8qO/NLmji+A1C8By1FaqW6aFPdrXdRyabmq2LSoNofY+oJumQ7dN4SkwKGLoO3cINTzoEVgDvR1Ja9ud8196YccpgWdjC4NtbeHCB3D1xjz4J3znumlukaejwQ65fiqqTiE52/vQai4t/3mvvus6+Be8JUeD56oWo+z1uPOwE6qx9Uj/5ARRmam3fjWpPOjf3Tc+ax3rwe5bM+bHxBCU76UlLGpY/dbe5HX3IdoqZchIq3/o3m4vySxX0499wLqp6TMWy5ubkB88nj5OTkoGU0NH9jy7Nq69dA8V7AWwGkZAF9Tge85U5tN5z5lQvoOxnYtwYodK5K0+yk/TS2M3RePSd3LqeZQcVmwLvrM+eptOOAdiOg9y80j3XRFrjaHwOVcgSUcsFbsAYo2dE820CHPKaFG8vrQcXbT0B17oGoGx9D1JT/B++yr4Gi2ilSLSlWeck3HwHFBeYm91W/YbXLlVTIljVAYipcoyehWSmFmP93uwmwpXdcaW5yP+b/3db4skqdGrnnyw+dtHFhgbnvHuiMJbNp6tQLceBArrktX15dgzziiCPw6acf4brrrjdtoT4FBQXm4t/+5PGBA7U7bYUyf2PLs6pgB+Apc5oU8jYDu38E0vqFN78JrGc4AXjjF2gpUts0qd7ywBOXALrc/PFKrVRSzN5Sc1/FdXWmRyXD1fEEePOWwLvtLXi2vwsVlQqVGuS7R632wv2RjDXXg7F7OzyvP1L10HXSuU5grGnvDujyxtVClbQPpgX+IkOTS0iESktHxVefm97PouLrzxE3/nQgMSl422od9O7Gb/PBev31N8zNnwTWWbM+w+2334nXXns9YNqPP/6EYcOGVj2OiorCwIGH46eflgctv6H5G1tek2psG1XN+U1gPd3pELTug8Bexc1MJfR2Urn1KT8ALenfukjbqqcI8LXZekugi36GSh4InedckYeahi88hiPc17cGrLkejE7dgOgY056oBowwaWHPV37DJHwqyqF/mg/XsZOBOEl1JZj7ek1luqtzd6heg5xhOMoF1Xco1BGj4f25mQ/OhQXw7s5B1JiTnXWJikbUcadA798bPLC6XM588ldV3q/sNITycngWfwO3BOb4BLPdct/TxMNwhFyyTALrXXfdgxkzqq/A4vPqq69h/PhxmDRpkvnB4z/84U7s2bMH8+YFH+rR0PyNLa/xVGXvV1V9X95vdyyQ2gtwybmxctK8nYYA+9YFL6ah+X2B1RVdGVgt9HSub5vMYUf53fc7DMVmmpRvfb2EHR7owk1wJQ80nZfkJvd18VZnctk+wBUPxFX2hXDFQiX0gi6r7PxETYY1V4fSIXTLkh+JlXRXXl5eSL9QECnKH6j/VxTq4hp7NlwjxpugIh2QvLPegt7qtF25L7wJestaJxUsomPgnnQpVP8jnWC7dmlVhyaV2ROu0y6Fkh7Ishvy9sD7/f/gXXLwnWE8++s5m6+H6twFUWdeDFdWL3Ow9W7bhIr/vm56/Eadc7mZp+L/XjR/oyaebW7+vOtXoewff3YexMQ6Y2YHj5BePiawVnzwWlXKuDHiH3sz5HlfeOHfmDbtUtNT2N/AgYPNT1eJKVPOxEMP/RXdunUz41KvuOKqqnGpY8aMMenk5OTqHqX1zR/K9GC8Cx8KbYO6joaSDkh+dH42sP4joN8UIL5yGJTpoLQE2ONX2+t3FnBgG7BjkdNBqb75k7tBHX4etKSD/Wus2xc5rw+Bt0tovxaiUgbDlTI4cJtKd8K7+0tnurSbag/0/trD2lwdx0KX7oI+sLKyMDdUu6OdnsXymuJt1R2aRFxXZ1lRSc70khxnegM9lX3c3aaiLWnq47iv/NEpv0GUCm/oXYUuxfz8f0R0zGFwbWMONri2Vo0JrpEi5OAaQUINrpGEwfXgyh+Zco2V4Low/5mIjjlscyUiImu8lf/CEe7rWwO2uRIREVnGmisREVmjlYZW4fYWjvwOTQyuRERkjbbQ21e3geDKtDAREZFlrLkSEZE10hlJsUMTgysREdnDKzQ5mBYmIiKyjDVXIiKyxqu8UGH2FmZamIiIyA/bXB0MrkREZA2Dq4NtrkRERJax5kpERNawt7CDwZWIiKzxwgMFT9hlRDqmhYmIiCxjzZWIiKyR6wLrsNPCkX9tYQZXIiKyhuNcHUwLExERWcaaKxERWe7Q5Aq7jEjH4EpERBaFPxRHyoh0TAsTERFZdkjXXKPvnoG2Jhpti3701ZZeBQqBu6VXgFoNr5aUrstCGZHtkA6uRERkF6/Q5GBwJSIiazQ80GHWXKWMSMc2VyIiOmT8+c9/xrHHHouEhAS0a9cu6DxKqVq3N998s1HLYc2ViIiscS4A4bVQRtMoKyvDueeei9GjR+P555+vc74XX3wRp556atXjugJxXRhciYjokLn84X333Wf+zphRf4dWCaYZGRkHvRymhYmIqFXKz88PuJWWljbbsq+99lp07NgRxxxzDF544QVo3biAz5orERFZo7V0aFJhlyGysrICnv/jH/+Ie++9F03t/vvvx/jx40277Oeff47f/OY3KCgowPXXXx9yGQyuRETUKttcs7OzkZKSUvV8bGxs0Plvv/12PPjgg/WWuWrVKgwYMCCk5d99991V94cPH47CwkI8/PDDDK5ERBT5UlJSAoJrXW6++WZcdtll9c7Tu3fvg16PkSNH4oEHHjBp6boCfE0MrkREZHmcqwq7jMZIT083t6aydOlStG/fPuTAKhhciYjIGq0tXKFJN91QnC1btmDfvn3mr8fjMYFT9O3bF0lJSfjwww+xc+dOjBo1CnFxcfjiiy/wl7/8BbfcckujlsPgSkREh4x77rkHL730UkCbqpg9ezbGjh2L6OhoPP300/jd735neghL0H300Udx1VVXNWo5SofQv1i6QKempiIvLy+k/DcREbUuTX0c95XfMWUkXCq8eptXV2BP/sKIjjmsuRIRUascihPJGFyJiOiQuUJTc+EVmoiIiCxjzZWIiCz3FlZhlxHpGFyJiMgiaXMNv4xIx7QwERGRZay5EhGRNU5KV1koI7IxuBIRkTUMrg6mhYmIiCxjzZWIiKyRn4tTYV+4P/JrrgyuRERkDdPCDqaFiYiILGPNlYiIrLFxXWDNawsTERHVvC6w10IZkY3BlYiIrLHRXqrZ5kpEREQ1seZKRETWsObqYHAlIiJrbIxR1W1gnCvTwkRERJax5kpERNYwLexgcCUiImsYXB1MCxMREVnGmisREVlko9YZ+TVXBlciIrKGaWEH08JERESWseZKRETWcJyrg8GViIis0drChftNGZGNwZWIiCySn4tTYdddIx3bXImIiCxjzZWIiKxxevqqMMuI/JorgysREVkUfnBlWpiIiIhqYc2ViIjssZAWBtPCRERE1bSFlK5mWpiIiIhqYs2ViIgsYocmweBKREQWaQudfZkWJiIiooOpufoG9Obn54cyOxERtTK+43fTX6BBuiNFfs2zWYLrgQMHzN+srKymXh8iImpCcjxPTU21Xm5MTAwyMjKQk5NjpTwpS8qMVEqHcBrj9Xqxfft2JCcnQ6lwr7xBRETNTQ71Eli7dOkCl6tpWgRLSkpQVlZmpSwJrHFxcWjTwZWIiIhCxw5NREREljG4EhERWcbgSkREZBmDKxERkWUMrkRERJYxuBIREVnG4EpERAS7/j9gxq6z4ncPUgAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "opt_policy, V = policy_iteration(random_policy, P, R, gamma)\n", + "print(\"Optimal policy found by Policy Iteration:\")\n", + "print(opt_policy)\n", + "plot_policy(opt_policy, title=\"Optimal policy found by Policy Iteration\")\n", + "plot_values(V, title=\"Value function of the optimal policy\")" + ] + }, + { + "cell_type": "markdown", + "id": "7d52252c-7aee-4be9-9896-939348add5de", + "metadata": {}, + "source": [ + "## 5. Dynamic programming : Value iteration" + ] + }, + { + "cell_type": "markdown", + "id": "00c8cbd1-e0ea-4919-b61f-ff0bc3c95880", + "metadata": {}, + "source": [ + "**Exercise 15.** Write a `value_iteration` function whose inputs are the transition probability matrix `P`, the reward vector `R`, the discount factor $\\gamma$ `gamma`, the parameter `theta`, which is a stopping tolerance (stop when the value function changes by less than theta), and `max_iter`, which serves as a safety limit to prevent the loop from running indefinitely. \n", + "\n", + "The outputs of value_iteration are `V`, which is an approximation of the optimal value function, and `policy`, which is a greedy policy derived from the final `V`.\n", + "\n", + "*Question:* Do `value_iteration` and `policy_iteration` find the same optimal policy?" + ] + }, + { + "cell_type": "markdown", + "id": "fd52fa5c-dbaa-4281-a1a6-dd7180123756", + "metadata": {}, + "source": [ + "*Hint.* Value iteration repeatedly applies the Bellman optimality operator. In the maze case, it is \n", + "$$\n", + "(\\mathcal{T}^* V)(s)=\\max_a \\Big\\{ R(s) + \\gamma \\sum_{s'}P(s'|s,a)V(s')\\Big\\}\n", + "$$" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "293ba7fc-f9dc-41b0-ad78-677af1ac7e0f", + "metadata": {}, + "outputs": [], + "source": [ + "def value_iteration(\n", + " P: np.ndarray,\n", + " R: np.ndarray,\n", + " gamma: float,\n", + " theta: float = 1e-6,\n", + " max_iter: int = 10_000,\n", + ") -> tuple[np.ndarray, np.ndarray]:\n", + " \"\"\"Value Iteration (student version).\n", + "\n", + " Goal:\n", + " Approximate the optimal value function V*\n", + " and derive an optimal policy.\n", + "\n", + " Inputs:\n", + " P : array of shape (n_actions, n_states, n_states)\n", + " Transition probabilities.\n", + " R : array of shape (n_states,)\n", + " Reward for each state.\n", + " gamma : float\n", + " Discount factor.\n", + " theta : float\n", + " Stopping tolerance for convergence.\n", + " max_iter : int\n", + " Maximum number of iterations.\n", + "\n", + " Returns:\n", + " V : array of shape (n_states,)\n", + " Approximation of the optimal value function V*.\n", + " policy : array of shape (n_states,)\n", + " Greedy policy derived from V.\n", + "\n", + " \"\"\"\n", + " n_states = len(R)\n", + " n_actions = len(P)\n", + " V = np.zeros(n_states)\n", + "\n", + " # Main value iteration loop\n", + " for _it in range(max_iter):\n", + " V_new = np.zeros_like(V)\n", + "\n", + " # Loop over all states\n", + " for s in range(n_states):\n", + " if is_terminal(s):\n", + " V_new[s] = R[s] / (1 - gamma)\n", + " continue\n", + "\n", + " Q_values = np.zeros(n_actions)\n", + " for a in range(n_actions):\n", + " Q_values[a] = R[s] + gamma * np.sum(P[a, s, :] * V)\n", + " V_new[s] = np.max(Q_values)\n", + "\n", + " delta = np.max(np.abs(V_new - V))\n", + " if delta < theta:\n", + " break\n", + "\n", + " V = V_new\n", + " policy = policy_improvement(V, P, R, gamma)\n", + "\n", + " return V, policy\n" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "9b6ff9d3-ccc9-4f35-a6c3-545aeed552f7", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdcAAAGkCAYAAABjMm+uAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAUu5JREFUeJzt3Ql8FOX9P/DP7OYOOQgEEiDcAoLcKqCggMohHqi1XvWq1f4869FWbL1ppYq1WmutbVW03v3bar1PBA8OAblBDjnCkXAmIXey8/xf32eyyW6ySTbZZ5Ns+Lx9jdndmZ2d2V3mu9/v8zwzllJKgYiIiIxxmVsVERERCQZXIiIiwxhciYiIDGNwJSIiMozBlYiIyDAGVyIiIsMYXImIiAxjcCUiIjKMwZWIiMiwoza4zps3D5ZlYfv27UfNa99///36dVtDZWUlfv3rXyMrKwsulwszZ85EW9Sa3wtRWFiILl264OWXX26X34PevXvjqquuQiSR90res3AaO3as/vdB7UebCa7r1q3DT37yE3Tv3h2xsbHo1q0bLrvsMv14KB566CG89dZbxraTmue5557D3Llz8aMf/QgvvPACbrvttlbdnrb6vXjiiSeQlJSEiy++GEeD9evX68DVWj9mvN5///2wB9CG3HnnnXjqqaeQk5PTattAhqk24M0331QxMTEqIyND/fa3v1X//Oc/1d13360yMzP14//5z3+ave7ExER15ZVX1nm8srJSlZSUKNu2VUt7/vnn5XzOatu2bS36uvfdd59+3dZw0UUXqe7du6u2oi1+L8rLy1V6erp66KGH2u33oLS0VO+n17///W+9LfPnz1et6cYbb6z3PZHvQ0VFRVhf3+Px6OPfPffcE9bXoZYThVa2detWXH755ejbty8WLlyI9PT06nm/+MUvMGHCBD1/9erVehlT3G63nqhl7Nu3D6mpqWjrWvN78e6772L//v348Y9/jPZKqlItoaioCImJiUbWFRcXh3CTphKp6rz44ot44IEHWq1sTwapVvbzn/9c/2JcuHBhwPkLFizQ82W52r+8N2zYoC688EKVlJSk0tLS1C233KJ/ZXrJMrUnb7YSKHvs1auXmjFjhv4VPXr0aBUXF6eOO+646l/VkmHL/djYWDVq1Ci1YsUKv21dtWqVXn+fPn30Ml27dlVXX321OnDgQJMz17lz5+pltm/fXmferFmzVHR0tDp06JC+L+/dj370I5WVlaUz/R49eqhbb71VFRcXN5ixyOvLfdme2uRxWd7Xrl279P506dJFv87gwYPVs88+W+8++L5G7UneU5kCZS2BtkveV8k2ZRvOPfdcfbtz587qjjvu0Nlm7Szg8ccfr/6sZLmpU6eqb7/9tlnfC/HUU0/p/ZX9lorKDTfcoA4fPuy3zKmnnqqGDBmi1q1bpyZOnKji4+NVt27d1MMPP6yCccUVV6jevXu3y++B77+x2u91oO+G1/vvv6/Gjx+vEhISVIcOHdSZZ56p1q5d67dO73djy5Ytavr06Xo5+Y4E+57I8wNtR0PvgfzbnzZtmj72yGtPnjxZLVq0yG8Z7/599dVX6rbbbtPfQ9mPmTNnqn379tV5b95++229fO3jCkWmVm9zfeedd3QnB8lQAznllFP0/Pfee6/OPPmFX1paijlz5uDMM8/En//8Z1x33XXV8//1r3/pX8qybrkt089//vMGt2fLli249NJLcfbZZ+v1Hj58WN+WDibSTijtwvLLUjJueX3btquf+8knn+CHH37A1VdfjSeffFK3m7322mt625p6ZT9Zt/x6feONN+rMk8emTJmCjh076vv//ve/UVxcjOuvv16/7tSpU/XfK664Aqbk5ubqTheffvopbrrpJt022L9/f1xzzTV4/PHH632eVCLkfR80aBB69OhR/Tkce+yxTd4Gj8ej961Tp0549NFHceqpp+KPf/wj/v73v/stJ9t066236s5TDz/8MGbNmqWzj8WLFzfreyFtcTfeeKPuByCvd8EFF+CZZ57Rn0FFRYXfsvJ9mTZtGoYPH66Xlf2W9rQPPvig0f375ptvMGrUqHb5Pajv3/Ytt9yib//mN7+p892Q2zNmzECHDh3053jPPffoNtrx48fXaaOVDnOyv9IZTL4b8hkF+57IZ3/GGWdUv6Z3qo/0A5HvzqpVq3QnJNmubdu2YeLEiViyZEmd5W+++Wa97H333ae3Q4558t7VNnr0aP3366+/btL7SG1Ua0b2vLw8/UvN+yuzPuecc45erqCgwO+XtzzuS7IJeVwyyMba1urLXOWxb775pvqxjz76SD8mWciOHTuqH3/mmWfq/MqunSGIV199tU5mHmyb67hx43QG7Wvp0qX6uS+++GKDrztnzhxlWZbfNoeSsVxzzTU6Y6udhV988cUqJSUl4DYEyup8NTVzlccefPBBv2VHjhzp9x59/vnnejmpYtTm244a7PdCMgzJeKZMmaIzYq+//OUvernnnnvObx9rfzZlZWW6Le2CCy5o8P2RNj35vCQTb8/fA9/MtaE21yNHjqjU1FR17bXX+j2ek5OjX8f3ce93QzL52oJ9Txpqc639HkjmKd+JrVu3Vj+2Z88encWecsopdb5Lp59+ut93T7JYt9utj3+1yXqvv/76gNtBkaVVM9cjR47ov9I7siHe+QUFBX6PSzZR+xeit+dfcw0ePBjjxo2rvj9mzBj9d/LkyejZs2edxyVT9YqPj6++LRn1gQMH9K98sWLFiiZvy0UXXYTly5frLNnr9ddf11nXueeeG/B1pa1JXvekk07S2fJ3332HUMl63nzzTZ3By21Zv3eSTCA/P79Z+9cc//d//+d3XzII389AtlMyPckSamtOO5ZkaOXl5ToTlnYxr2uvvRbJycl1KiqSZUl1wysmJgYnnnii3zYGcujQIf3eerPQo/17IFWgvLw8XHLJJX6vI+3h8m9v/vz5dZ4jWWFtpt8TqZ58/PHHeiiZbx+QzMxMXfH66quv6hynpJrm+92T76ysZ8eOHXXWL5+/bCNFvlYNrt6g6Q2yTQ3CxxxzjN/9fv366QNgKN36fQOoSElJ0X+lxBjocSkD+h4gpRNW165d9T9qKYn26dNHz5MDT1NdeOGFen/kQCrkgCBlrunTp+sDu9fOnTv12MG0tDR9cJfXlZJpc1+3NulkIwc6Kb/Kun0nKYF7OyyFm5R2fTu8eQ9Gvp+BBCAp38p7YYL3ADhw4EC/xyVoysG19gFSSt+1g3jtbWxIoOaDo/F7sHnz5uoftbVfS4Jb7deJiorS731tpt8TeQ+kzFz7+yCknC3NRNnZ2Q0eU7w/oAJ9J+SzZWem9qFVewtLgJJffNITuCEyX8a/+h5IAjHxpayvp2h9j/seDKV9TNrNfvWrX2HEiBH6H7P8Y5M2ON+22WBJkJBfudK2Jm1S0mYoBwtpf/KSX8DSXiSBXdr2pI1Peknu3r1bH1Qaet363i9Zpy/vOiQju/LKKwM+Z9iwYU3ev2Bf3ysSencH8z0JRA7+8n4EOuC29+9BIN7XkrbPjIyMOvMlmPqSLN63shDqe9Ja3wn58dK5c+cW2CoKt1YfinPWWWfhH//4hy6nSEeF2r788kudiQbqcCK/br2ZobczkvyDkQ5QXi31K1AOip999pnu7HTvvff6bWMopCR4ww034Pvvv9eZS0JCgi7Lea1ZswabNm3SJ2bw7aQhZbXGeH9Byz9oX7WzMfm1L1UDOVidfvrpIe1Pc16/KaR68dFHH+kDakPZa7Dfi169eum/8v77lgGlVCydWEy9HxIsZNtlnUfT96C+z0HeCyEdlJr7Wk15T4L9Psh7IO+9fA61bdy4UQf42lWuYEnQl+9Vczr7UdvT6r2FJcuTEqoEz4MHD/rNkwOktLHJl1mWq03OaOJLegEKKZd5yS/V2geNcP46rf1rtKk9KGuTXo+y7ldffVWXAuXHiO/4vUCvK7elF2djpBIgv5JlfLGvv/71r3735TVkO6S9be3atQFLZc0hgUvW3djrN4Vsp+y//Mipzfc9CvZ7IQd2KQFLT3Tf5z/77LO6rCi9WU2Rtv5ly5YdVd8D7z7U/iykDVe2S86kVbtHdrCv1ZT3pL7tCLRO6aH99ttv+zU/SS/qV155RScIjVXY6iPt6kLahCnytXrmKu2m8stSTnU4dOhQ3aVfslH54soBTBr35YDi/SXrS37ln3POObrsumjRIrz00ku6U4EMg/Dt3i6dUh577DFdXpN1ezsjmST/oGRowSOPPKIPBlLGlrah+jKRYMkv90mTJuntl7ZnyWB8SalL3ptf/vKX+pevbIcc/IJt4/vZz36GP/zhD/rv8ccfrw+w8mu/NllGOpHIeyedeaTjl/z4kQ4s8v7K7eY0C0h7ovwoksxB9kNOpBBKu528V3LSEQmGUjXwluSlAiLzvEMggv1eSKZy11136WAt65Lvm2QtEnhOOOEEv85LoZLOSVIGlfd/wIABR8X3QJpPJGBJiVt+rEh5V9pZZX+ffvpp/VnK8CQZ1iafhZTDpRPZySefjL/85S8Nrrsp74l3GIwMDZLALttU3ykof/e73+nsVwKpVBOk6iBDs8rKyvS//+aSdUr77MiRI5u9DmpDVBuxevVqdckll+hu/jIwXoYvyP01a9bUWdY7lGD9+vV6gLh0ge/YsaO66aab/E4iITZu3Ki7x8tQmmBPIlGbLCdd9X15hy/IIH/fwfXnnXeeHkIgwwXkBBfSRb92V/6mnv7wH//4h15e9rP2/gl5H6S7vwyel4HqMkxBhiPVHl4R6LR3MlRBhlfI9sr6f/zjH+vhJ4EGzufm5ur3QQbkez+j0047Tf39739vdB8CDcUR+/fv18NUZHC9fIZyshA5SUB9J5GoLdA+yUkl5HMZNGiQHtogpxSUkwssX768Wd8L79AbWZ/st5wcRIZL1HcSidpk3fLdaowM25HPb/bs2e32e1B7KI53v/r27auHp9QeliO35QQgsl1yUpd+/fqpq666Si1btqzR70ZT3hP5ztx88836uyLDdII5iYRsl6xXvruTJk3yG8Ln+13ynrzEd59q76cM85Jjn5z2ldoHS/6HCCOD+iWTkNIQG/+pPZk9ezaef/55nXVHQgcuMkMuIiFVN+ntLp08KfK1epsrEdWQs4DJZefkzF509JCyuDRZMLC2H63e5kpENWT4VkuMGaa2RfqMUPvCzJWIiMiwiAyu0uYqTcVsbyUiojlz5uje+zIOW3qay+kpa49FllPSyilz5cIfUiGSYWUyhCpcIjK4EhEReS1YsEAHTjl7mQxpkuGQMh5Zzift259Brkgk48Rl+T179uD8889HuERkb2EiIqL6yEgSyWAliMr5B2QMtYyTlhN9yEXpvWfUkrNhSXu39wIrLd6hSQbhS5SXlJsnlSYiijySR8kJSOSkKbXPw2yKlF7lFI4mBLqIgZxkRKbGeC/K4D0Fqpz9SrJZ31NpyklG5KQdrRpcJbA293yZRETUdshVewJdQchEYO3Tpztycpp+trZApF1UhqX5kktJSp+bxpJBuUSknMXruOOO04/l5OTo05impqb6LStXMJN54RBUcPVe6k0+lOaeN5OIiFqPXGdWkqTGrp/dXJKxSmDdvu01JCcnhLSugoJi9O5zcZ2YE0zWKm2vcu5ruRhMawoquHpTc9lJBlciosgV7qY9CazJyYmG1tW0mCMn4pDzk8u5sX2zc7lsoQR/uTCDb/YqvYUDXdLQBPYWJiIic+Q6ubaBqYntsxJY//vf/+Lzzz/3uxSp98IM0dHR+rKgXjJURy4EIVejCgeeoYmIiMxpRnCso4nPl1Kw9ASWSwFK2dvbjipX3pJLmspfueLa7bffrjs5STZ8880368Aajs5MgsGViIgi2tNPP63/Tpw40e9xuQjGVVddpW//6U9/0r2k5eQRcnlAubRgKNeObgyDKxERmSOnTlAhnj6hic8P5nQNcXFxeOqpp/TUEhhciYjIHFsZKAtH/rmN2KGJiIjIMGauREQU0R2a2iIGVyIiMofBVWNwJSIicxhcNba5EhERGcbMlYiIzFEGMldZR4RjcCUiImMsZesp1HVEOpaFiYiIDGPmSkRE5rBDk8bgSkREhs/QpEJfR4RjWZiIiMgwZq5ERGQOy8IagysREZnD4KqxLExERGQYM1ciIjJ8PVc79HVEOAZXIiIyh2VhjcGViIjM4VAcjW2uREREhjFzJSIic1gW1hhciYjIHF4VR2NZmIiIyDBmrkREZIxl23oKdR2RjsGViIgMj3NVoa8jwrEsTEREZBgzVyIiMoe9hTUGVyIiMofBVWNZmIiIyDBmrkREZA5Pf6gxuBIRkTksC2sMrkREZDhztUNfR4RjmysREZFhzFyJiMgcnkRCY3AlIiJz2OaqsSxMRERkGIMrERGZIyVdO8SpiWXhhQsX4uyzz0a3bt1gWRbeeustv/lXXXWVftx3mjZtGsKJZWEiIorosnBRURGGDx+On/70pzj//PMDLiPB9Pnnn6++Hxsbi3BicCUioog2ffp0PTVEgmlGRkaLbRPLwkREZD5ztUOcABQUFPhNZWVlzd6sL774Al26dMHAgQNx/fXX4+DBgwgnBlciIjIn1PZWu+b0iVlZWUhJSame5syZ06xNkpLwiy++iM8++wwPP/wwFixYoDNdj8eDcGFZmIiI2qTs7GwkJyeH3E568cUXV98eOnQohg0bhn79+uls9rTTTkM4MHMlIiJzlG1mAnRg9Z1MdULq27cvOnfujC1btiBcmLkSEdFRdVWcXbt26TbXzMzMsL0GgysREUX0UJzCwkK/LHTbtm1YuXIl0tLS9PTAAw/gggsu0L2Ft27dil//+tfo378/pk6dinBhcCUiooi2bNkyTJo0qfr+7bffrv9eeeWVePrpp7F69Wq88MILyMvL0yeamDJlCmbPnh3Wsa5NCq72C3fAjo9BuxEbjfbGSmhHn4+ID+9A79ZgnTkb7Y16/x60N+3xc2qvZeGJEydCNXBWp48++ggtjZkrERGZw+u5auwtTEREZBgzVyIiOqp6C7cEBlciIjKoZpxqSOuIcCwLExERGcbMlYiIzGFZWGNwJSIicxhcNZaFiYiIDGPmSkREEX36w7aIwZWIiMxhWVhjcCUiInMYXDW2uRIRERnGzJWIiMxhm6vG4EpERObI1WlUiGXdUJ/fBrAsTEREZBgzVyIiMocdmjQGVyIiMofBVWNZmIiIyDBmrkREZI5cbs4OsbdvyJesa30MrkREZA7LwhrLwkRERG0+cx18KqxjxgBp3YDs9VCf/r16ljX6LKDXMCA1A1i/AGrxm42vb/gUWIPGA3EdgOI8qC9eBPZvB9J7wxo9A+jcU9YMHNjhrC8vx/guYeB4WP1OBFK7AXs2QH3xrPN4XAdYx88EuvQHouOAwgNQqz4Adq0LvB6XG9Zp1wOpXQFXNFCSD7XhC2Dzopr54y8HOvWE1SENtrxO9hqERb+TgF7HA8kZQO5GYNGLNfPG/ATo1BuIigHKi4HtS4GNn9e/rrhkYPSPgM59gfIiYMNnznO8uhwDHHcm0KEzUJIHrH4HyN1kdn96jQF6jAQ6dAX2bwZWvFIzb+TFQMeegDsaqCgBspcDWxfUv67O/YCBU4DETvozwoYPgANbauYnpgNDzwVSMoHSAmDDR8C+jWb3p70y+TnJsoOmAR3SAU85sHsl8P2nUlMMbj6Fh1R07VAzV0Q888FVAuDKD2F1GwQkpvrNUgX7gaVvwRp0clCrso4/B8joD/XBk4A8t0Ma4Kl0ZsYmQG1aDHz+HFBZDmvkdFjTboR6/V7zA5CLC6DWfAIrY4D/PkXFQh3aDax4Ry+DHoNhTbgC6v3HgPzcuutRNtS3bzrzpE0hpSusM26Ckvv7fnAW2bcN2LgQkCAbTiUFwMbPnB8G8Sn+8zZ8ChTuB2wPEJ8KjL8GKDoMZH8XeF0nXgoUHQTefcAJ1uN/pn9o4MAPQGIaMO4KYMkrQM5GIGMQMPYK4NPHgKJD5van9AiwZQHQqS8QV2t/tswHig44+yPzTrjCCfJ7VtVdT3xHYNQlwMo3gH2bnR8Gcv/LvwAlhwHLBRx/GbBnNbB0nvODYsSPga/+ChQb3J/2ytTnJD+oR18GbPsaWPQP5zs85qdAsXxPlwUxn8KGZeEwlYW3rwJ2rAbKCuvO27wE2LUeKC9tfD2xCcBxk6EWvuQEVlF4yAkKQtbzw3KgvET/Y1SrP9XZng7ApmWvdjLIsiL/xwsPAuvnA8X5zq9hyVjz9wGdewdejwT9vL01jfX6+6OApM7OfTmobFzgBNpwn6Fkz1pgzzqgrLjuvIIcZ1u8GynbIllnIBI8ZX/XfgB4KoDD2U4Q7n28M7/rQODwbiBng7Mu+XsoG+g52uz+5K4HcjcAFQH250iu//7IJFlpIOnHAAV7gX2SWSvnb/4uoMcIZ35abyAmAdjyBWBXOvMPbQe6V82nlvmcouNgyeewS37wKScIH9gKJHUNbj6FjbKVkSnStd0OTel9nINXv+OdTFf+0f2wHGr5uz7/AH1k9oeSQFF4GK1GStcpXYHDexpczJp0LZA5AJY72sl8d4ap9BuKETN12diKioGSDHNHPb/2pTQqP3h8f0zl7QH6jXNuW5Yz+ZL7KRloUUPO0uVIyx0DJdnLrhWBl6u9rc6DQFLV9srB+cg+/96M8mOEB+2W/ZwqSqCkbJw1Gti60MlMpZy/7p3g5hMdtcE1LgFWTDyQkg717wd1JmtNvR6oKANWfui/bGJHWCdfArXkP63XhVvaSydcAexY6WRmDVDz/6EP4iq9L9C1v9Me1NasfAtY+TZUaneg22CnDSyQqFigolYlQpaVx0XuZmDoDKDbEGDvBiDzWKBTL6dk3JLWvQusew9Kfgx0GVR3m70kuxk0Feh6LLDve6DLQKftTrJTIftV2cD+Ust8TmLvWmDoTKD/RFguN9T2xU47brDzKTx4buE23ltYgqi8x8vfAyrLdJufWjsfVs+h/sslpMI68xao9QuBTVUdg1ojsJ5yNVBZAbX49eC/PPu2worvAAyZjLZJyti7nPd/6FmBF5F50pnLV3S887iQttslLwPHng6cdS/Q+wRg16rA5eiwU0D+Hmfbjp0aeBFp8/vuDaD/JOC0O4Eeo5yDtHTsEvLc2oFU9t+7v9Qyn1NiZ2D0pU5ns48ehPrsYafjknREC2Y+hb/N1Q5xinBtN3M9uLvxZSSwzvgF1JZvgVUfofUC61X6r/rin4FL1o09Pym9bfdflE489bW55u8F4pOB2MSaNunUTCDfp9f23vXO5DXpJmDHcrQalxtIqKctT0jPX9/evyddB+xaWdMu2H+i8554qyTJsr8NNwWQ4c9JyvDSUzunqme+NEtI+2q/CcD3Hzc+nyjiMlc56LijnL/SfiW35R9JnXk+twMpPAi1e4PuBay75iekwBoyEWrname+3J/xC6dT03fvG9+NOvvkkv2QbbWqbrv14zqwSrukDJtpLLB27K7bWvX+yDq7Dwb6jIba43Mgl/XK+n1fN2A7oKF90u+/97YczFKBbscB7hhnX9N6Af3HA7nfB16PtMce2A4MqfqcOmYBWSOB7d/WLJPaw3kdyfgGne50CKqvDdfE/sj75d0f6XWaMbhmf1KzgN5j/YfW1JbSrer7GeME0ugEYHdVT2kpD0sZuP+pzmclHaCkk5MM86CW+5zydwNxSU75XpaX75R0KpPOaMHMp/Bh5qpZSjVe3C4oKEBKSgoO//lnSI6XL3/9rFFnwho1w+8xtXcT1HtPwDrlclgDxvrP27QYauG/nOdecDfUyo+Ard/WjCMdfynQbaDT9rJlKdSyd5yMYeSZcI2eAVVVPq5e34dPAblbEZTY6KAWs4ZNgzV8mv/r5GzRY1pdU2+Gqiz3ayNQaz8B1sp4OsA6+04oub1tOZCWBWvMhUByF6f0VXgIatPXwOZval7rvHudXs8+7K9fAX5YGty2JjT8+VQ79gxYg8/w36f9W4FlrwMnXOIMqZGDn/z637EC+H5+zfjAM24HNs6vGZrjN8612BnK4zvOVYbmpMl4ZOW0wco4Vxk/Goz4INsyj5kE6xj/8ro6uA1Y9SYw4kdVHY4soOwIsHuV08nFuz8TbnbGU8rwGnHilUBKj5o22A3vO++Dl5QXj5Nxrt2qxrl+2KRxrtaZs9HeqPfvafnPSdpkj5kEJKQ5nR/ls1r/fk1P5MbmH2Wfk/c4np+fj+Tk5LCt//DcqxuNE42uq6QcHX/1fNi2NSKDa0QJMrhGkqCDa6QINrhGkPZ20G5ScI0g7e1zYnBtWW23zZWIiCIPTyKhMbgSEZExUgxVIQbHIAqqbV7bHYpDREQUoZi5EhGROSwLa8xciYgooofiLFy4EGeffTa6desGy7Lw1ltv1Skz33vvvcjMzER8fDxOP/10bN4c3rN1MbgSEVFEB9eioiIMHz4cTz31VMD5jzzyCP785z/jb3/7G5YsWYLExERMnToVpaVBXESmmVgWJiKiiDZ9+nQ9BSJZ6+OPP467774b5557rn7sxRdfRNeuXXWGe/HFF4dlm5i5EhGR+RP3qxCnqrGzvlNZWdPP4b1t2zbk5OToUrCXjMcdM2YMFi0K3/noGVyJiMgYOYGeMjCJrKwsHQi905w5c5q8PRJYhWSqvuS+d144sCxMRERtUnZ2tt8ZmmJjI+eMbQyuRETUJofiJCcnh3z6w4yMDP03NzdX9xb2kvsjRoxAuLAsTERE7faqOH369NEB9rPPPqt+TNpvpdfwuHHjEC7MXImIKKIVFhZiy5Ytfp2YVq5cibS0NPTs2RO33norfve73+GYY47Rwfaee+7RY2JnzpwZtm1icCUiImN8OyQ1V1Ofv2zZMkyaNKn6/u23367/XnnllZg3bx5+/etf67Gw1113HfLy8jB+/Hh8+OGHiIuLQ7gwuBIRkTnKQFm3iSfunzhxYoMn+5ezNj344IN6ailscyUiIjKMmSsREZkjJV3bwDoiHIMrEREZI9dyVaFez7UdXBWHwZWIiMxh5qqxzZWIiMgwZq5ERGSOVHSVgXUcTcHVdeUf4QrxVFRtif3PG9HeWJc/jvZELXgI7Y1a+hjancT41t4CaiPY5upgWZiIiMgwloWJiMgcdmjSGFyJiCiiT3/YFrEsTEREZBgzVyIiModlYY3BlYiIjGFZ2MGyMBERkWHMXImIyBwZomobWEeEY3AlIiJj5LKqqmUv59omMbgSEZExbHN1sM2ViIjIMGauRERkDofiaAyuRERkDMvCDpaFiYiIDGPmSkRExrC3sIPBlYiIzLEtZwp1HRGOZWEiIiLDmLkSEZEx7NDkYHAlIiJjlLL0FOo6Ih3LwkRERIYxcyUiImNYFnYwuAZj8KmwjhkDpHUDstdDffr36lnW6LOAXsOA1Axg/QKoxW82vr7hU2ANGg/EdQCK86C+eBHYv92Zl9oV1oTLgE5ZQNFhqCX/BXauCePOtRPdRgMZw4DEdODQVmBdgM8hOhE44TqgrABY/mzg9aRkAUMv8n/MFQ3sXgZs/aTmsaxxQLeRQHQCUHYE2PgOcGSP2X3qMgJIHwzEdwbytwOb/1czb9CFQIdM/6PQ6ueBiqL615d+HJBxPBCTBFQWAzu+APK21rw3faYAST2AyhJgzxJg/5rI/pxiOgADzgRSewIVJcCOr4Gcleb3ieoOxbFDX0ekY3ANhgTAlR/C6jYISEz1m6UK9gNL34I16OSgVmUdfw6Q0R/qgycBeW6HNMBTWTXTBeuM64Gt30K9/2eg2yBYk38K9d85zrJUv/JC5+DZsTcQmxR4mWOmAIW5QHR8/evJzwa+etT/QD/2JmD/+prH+pwKpPQEVr0KlB4GYpMB5YFxFYXA7iVASi8nUNSW/SWQ+11w60ofCmSMAra+BxTvB6ISAHd0zfx+M4CyPOC7p51gPvB8Z9+O7ELEfk7HzgRKDgPfPOEE82EXAyWHgPydBneIamObq4NtrsHYvgrYsRooK6w7b/MSYNd6oLy08fXEJgDHTYZa+FJNsCw8BJQUOLczjwHiEqG++8AJuNlrgb2bYfU/0fAOtUMHvgcObnIylEA6HQNExQO5TczGMoY6B+iC3c79qDigxxjg+/ec4CMkwypvIGNsrsNbnMxSMsmQWED3k5xMVQKrkMy1LN+5HZsCJElV5ivArgSKcoCDG4H0IYjYzykuFUjpAWz7ArArnKpC7jonayZqAcxcW1J6H+fg1e94J9O1PcAPy6GWv+vclrLz4b3+NZVDu5zHqfncsUC/04E1rwHJPZr23IzhwF6f7DC5u/MZdhkMZI50MtZ9G4DtC1q+oajbWKD7OCe45ywHDm4IvFxcR1gxiVCJXYA+p+sKCfK2AzsXAHY5kJDulJMl4HpJEO4yHBH7Ocm+SpbsWyaXbLjbKHPbS4HZFhRPIsHg2qLiEmDFxAMp6VD/flBnstbU64GKMmDlh0BULFBe7F8eKSuBFR3XapvcLvSdDOSsdjKbphy0pV1PMqDctTWPRcXBioqDik8Dlv7NKV0e92PAUw7s/BotZtdXQMlBJ9AnZwH9znIyNMl2a5NsWyT3BNa9XFMG7jUR2Pax01ZZWeb/nMpSwB2DiP2cZNsD7VNUC+/TUYinP3SwLNySJIjKF2f5e84/fOmwtHY+rJ5DnfnymARfH1ZMHFARRMmZ6j/wSnkwe1HTnyvZ0MHNQIXPDx5PhfN3+0InmEnWuPtbp5zZkgr3OgFdsuX8HcD+1UDagMDLynaKvUudACOT3E7tWzNfskZf8kNP1h+xn1N54H2qbMF9oqMaM9eWdLCqPag+h/YAI6Y7ZTtvibFTD+BAdotsXruU2tvJasbd4ty33E5HnpNuBZb9o/62Usl80gcB6/7j/3hRLtqkhn7qlxyGkgy3PlICjkl02jq97bsJXYCSA4jcz2kfENvB6c3tDbodugJF7BgYbuzQ5GDmGgwJdu4o569lObdd7gDzfG4HUngQavcGWCOnOweOhBRYQyZC7VztzN+7GSgrhjVyGuCKAnoMATIHQG1Z2nL7GrEs54Cs33uf27uWOOXbZc86k2ScxQed27VK8H66DHE63Rz+wf/x0nyow9uAXuOdz0h68XY/3umkE659kr+++yQZWUof5/XlcSkLdxkGHNoceDWqEjiwAcg8wXmuTHL7cNUwHOnYJB1+elTtU2IG0GkQsH9tBH9OeUD+LqDPRGefkjKdZXNWhWGfyJe0tyoDU1Pcf//9sCzLbxo0aBBaEzPXIEiws0bNqLl/9RNQezdBvfeEHpNqDRhbM0+C5abFUAv/5dy/4G6olR/p4TVCzZ8Ha/ylsC77g1PulcC5qmpcnrKhPvkbrAmXwhp2BlCUp5fnMJwg9BoPq/eEmvun3AmVtwNY9bJ/eVNKolIVKD9S89jx1wI7vwH2rfMvNUr7XyAb3gYGTAfG/cJZt7T1ZS82v0/dx8KSDkteJ/wCqiAb2PKunof4M53HpTQtnZMO+wTXAecBR3Y75V+xcz7Q6zRg+DVOJywJRju/qFl+6/tAnzOAkdc775EM8zE9DKfFP6e3gIEznOxX/q39MJ/DcNqxIUOG4NNPP62+HxXVuuHNUqrxpuOCggKkpKQgPz8fycnJaC/sf96I9sb1s6fQnqgFD6HdiW+HHdRK2l+/AOvU36A9Cfdx3Lv+zef8BEnRoXUcO1JRjmP+91LQ2yqZ61tvvYWVK9vOSUJYFiYiIuNtrirEyRuwfaeyslo9wH1s3rwZ3bp1Q9++fXHZZZdh587WrVIwuBIRkTG2bRmZRFZWls6GvdOcOXMCvuaYMWMwb948fPjhh3j66aexbds2TJgwAUeO+DQrtDC2uRIRUZuUnZ3tVxaOja01vKrK9OnTq28PGzZMB9tevXrhjTfewDXXXIPWwOBKRERt8iQSycnJzWofTk1NxYABA7BlS4CTqrQQloWJiKhNtrk2V2FhIbZu3YrMzEy0FgZXIiKKaL/85S+xYMECbN++Hd988w3OO+88uN1uXHLJJa22TSwLExFRRJ+hadeuXTqQHjx4EOnp6Rg/fjwWL16sb7cWBlciIjLGVpaeQl1HU7z22mtoa1gWJiIiMoyZKxERGdOccwPXFvL1YNsABlciIjKG13N1sCxMRERkGDNXIiIyxoaBDk36MouRjcGViIiM4cXSHQyuRERkjARGm8GVba5ERESmMXMlIiJjWBZ2MLgSEZExdtUUilCf3xawLExERGQYM1ciIjKGZWHHUR1cVUlFa28CNSYxHu2NSkxAe+M68fbW3gRqI2zV9BPvB1pHpGNZmIiIyLCjOnMlIiKzWBZ2MLgSEZHhsjBCXkekY1mYiIjIMGauRERkDMvCDgZXIiIye1Uc8Ko4DK5ERGQML5buYJsrERGRYcxciYjIGDmBhB3ySSRYFiYiIqqmDLS5qnbQ5sqyMBERkWHMXImIyBh2aHIwuBIRkTFsc3WwLExERGQYM1ciIjJGOiMpdmhicCUiInN44n4Hy8JERESGMXMlIiJj2KHJweBKRETGsM3VweBKRETGsM3VwTZXIiIiw5i5EhGRMSwLOxhcg2ANmwRr0Digc3dgxzrY7/21ZmZ0HKxJl8HqMwyorIBaPR/q2/fqX1l6T7hOudhZV0kh1NJ3oDYudualdoHrpAuAjL5AVDRwcA/sb94E9m4N/05GuvThQOchQHwnIH87sPWdmnkDfwQkZgLKrnls7Tygoqj+9XU+DsgYDUQnAZXFQPYXQN4PgOUGBpwHxHUCXG6gvAjIXQEcWGN+nzoOgZUyEIhNA4p2Qu362H9+6iBYacOB6ESgshQq92ugcEfAVVn9LgWi4vVhS1M21KZ5NQskdofVZQwQnQJUFkHlLgKKss3vE7V7LAs7GFyDoIryoJa9DyvrWFgdOvrNs069GFZcIuznZwEJSXDNvB04crAmYPqKiYfrnFuglvwP6j9fAl16w3XurVD5B4C9W4DYBKgda6E+/xdQVgRr8MlwnX0L7Bd/C5QWttwORyIJlHuXAMk9gegOdefv+grY911w6+o8FOg6Etj6PlCyH4hKAFzRzjwJ0DvnAyWHnEAVl+YE79JDQOFus/tUWQx1YAWsxO5OAPWVeiystKFQuz8Fyg4C7njA1fA/Z7X7M6Bwe90Z0Umwekx11lW4E+jQE1aPM6B++DdQccTsPhGFyVNPPYW5c+ciJycHw4cPx5NPPokTTzwRrYVtrsHY+h3ww0qdafqJioE14ATYi98GykuAvH1Qqz6HNXh84PVk9gM8lVBrFzpnps7dBrV1BawhVcvnboda96UTSJWCWveVczCXLJcalrcFyNsKVJaEuCIL6D7OyVQlsArJXMvzq+YroORgTQboFZsC445sc4Khp7TONlrpx0PlfuMEVuEpaX4g7JAFlB5wAquQvyX7YKUMCG376ageimOHODXF66+/jttvvx333XcfVqxYoYPr1KlTsW/fPrQWZq6hSO0Kyx0N7K8pn6kD2bCOnx54eSvAF8ayYHXqUftQ7egkGUsccGivuW0+WmWOAbqNBcoLnDLuwQ2Bl4vrCCs6ESqhK9DrdMByOWXm7IWAXV6zXP9zdZZsuaKgivc7gb2lxKTAikqAiusMK+MU53tVmA21bxFgV9T7NCtzAoBT9HugDiz3KfsGOpBZQGynsO0CtV9yLFMG1tEUjz32GK699lpcffXV+v7f/vY3vPfee3juuecwa9YstAYG11DExEKVl/q35ZUVAzFxgZfP+QGIjtVtuDp77dobVr+RQPGRwCXkqddCLX8fKC4I3z4cDXZ9DZQeBOxKICkL6DsD8JQHDohRVZ+dlJc3vOLcluWzTgV2fFKz3Ja3dQBSHboBST2cdbcUt7ONUi5W2//j3O5+GqyuJ0HtXRDwKWrP5052ChtI6gurxxSoHf8DSvcDRbuALmOBDr2dNtsOvYCEDKB4T8vtE1EABQX+x77Y2Fg9+SovL8fy5ctx1113VT/mcrlw+umnY9GiRWgtLAuHorwMiI5xshuv2ARAAm4gpUWw3/0LrAEnwvXTuXCddD7U+m/043UCq7TF7t0CtcSnYw41T9FeJ5jKj6CCHU7no7SBgZf1VGV+e5fqTkJ6ktupfQMsrJx21ugEp/NTS6nKTtWBlU7J2FPq3JagWJ+SHEBVVr0HW3QQtZL6OPPK83V7q5U+GtaAK2ClDgIKtgKeshbaIWpPpKevHWJJ2NtbOCsrCykpKdXTnDlz6rzegQMH4PF40LVrV7/H5b60v7YWZq6hyMsFbA/QuQew32mvsuT2wQY6tuzdCvv/PVx915p2LdTuTbUC6y+gDu2Bmv9SWDf/qNXQlZhLD0E1NQuVHsSx/h3dwqo8r+nb2Nh7ULgDyqensdV7JlS+z/eSKEhSx7MNrENkZ2cjOTm5+vHaWWtbxsw1GJKZuqOk1uC0RenbbqCyHGrzMrjGnquDIlK6wBo2GWr9V/Wvq3OW06vTHa07MlndB0Kt+tSZFx3n9CbOy4X67MUW2732wXKCnHylrarb+nOLBVJ6V/WktZyycPpQ4PDmwKtRHqc9NuME57kyyW1vCTk+3SkZ69eygJQ+QJpketvDt0+6MuKzf7KN+ZthdRoOuGL0pG8fqWcbojoA8ZlV63HpsjCSekP5Lh/X2XkN6RXdeZRTes5jcKXWlZyc7DcFCq6dO3eG2+1Gbm6u3+NyPyMjA62FmWsQrBNmwDXm7Or77hv+CrXre9j//SPUF68Ck38C19UPO8FWxrn6DMPRwXLPZqhlHzjrGj7ZaWeVA13OD3odKHJ6osrjlvQo7twDVt+R1euQDFZtWtqi+xxxuo2B1W1czf3Rt0AdyQa2vgdkjgX6pjmPlxU4nZN8g+sxM4Eju4Gcb5370lO452Rg6E+dQCbjW7Or2jLlc+t+su74pHtdSAcpmXfoe+O7ZHUepXsFV98f9DOooj1QO9/RPYWtjPGw+l/qbOORHU6HJu+yfS+EOvCdUwJ2RcHKOEl3hNJl4aoyMEprelJa6WOA+C5OqbtoN9SOd5wyMlETKSnrqhBPItGE58fExGD06NH47LPPMHPmTP2Ybdv6/k033YTWYinVUI2splFZ6t35+fl+KXqk8zx5Hdob981/R3uilv0J7Y1KTEB74zr25629CdTKx3Hv+l8Z9UskSMUnBMWeMly64tGgt1WG4lx55ZV45pln9NjWxx9/HG+88QY2btxYpy22pTBzJSKiiD5D00UXXYT9+/fj3nvv1Z2YRowYgQ8//LDVAqtgcCUiooh30003tWoZuDYGVyIiMoYn7ncwuBIRkTE8cb+DQ3GIiIgMY+ZKRETGsCzsYHAlIiJjWBZ2sCxMRERkGDNXIiIyhpmrg8GViIiMYZurg2VhIiIiw5i5EhGRMXK2ejvEsm7jZ7xv+xhciYioTV7PNZIxuBIRUcRecq6tYpsrERGRYcxciYjIGJaFHQyuRERkDMe5OlgWJiIiMoyZKxERGSNJpzKwjkjH4EpERIbLwlbI64h0R3VwVaUetDcVs69CexJ9zzy0N2rnS2hvPPn/Q3ujOiSjPan0FLX2JhxVjurgSkREZrEs7GBwJSIiY9hb2MHewkRERIYxcyUiImN4EgkHgysRERkjV7RRvCoOgysREZkjFzq3ebF0trkSERGZxsyViIiMYVnYweBKRETGsEOTg2VhIiIiw5i5EhGRMTyJhIPBlYiIjOHpDx0sCxMRERnGzJWIiIxhWdjB4EpERMZwKI6DZWEiIiLDGFyJiMj4OFc7xClcevfuDcuy/KY//OEPxl+HZWEiIjqq2lwffPBBXHvttdX3k5KSjL8GgysRER1VQ3GSkpKQkZER1tdgWZiIiNqkgoICv6msrMzIeqUM3KlTJ4wcORJz585FZWUlTGPmSkREbbIsnJWV5ff4fffdh/vvvz+kdd9yyy0YNWoU0tLS8M033+Cuu+7C3r178dhjj8EkBtcgWCMnw3XcyUDn7lDb1sJ+6y81M2Pi4Drjclj9hgOV5bC/+xxq0buBV5SQBNeki2FlDQBi4oG8/bC/fgtq6yr/1xtzJlzDTwXik4DCw/C8/09g77bw7mTHdLinXQ6rez+gohz20o9hL/qg3sWtEafAfdJ0ICkNKD4Cz0cvQ236DnBHwX3pL2GldwOiooEjefAs/hDquwXh3f72qMMAWIn9gJhUoGQP1IGq99CdACvzbP9lLXfVMl/Uv77E/rCSB+vnwy6FOrwMKNnlv0x0CqyMM/1fzyArpjes6J6AOwmo3Ae7+Nuama4OcMUPBdwpgLKhKnOgStYB8AReWaPLR8GKHwYruqszv3wbVNkm03sEl3UMLKujvHkAymGrnVAqp2q+Gy5rACyrk+6mY6vdUGpHA+trbPmmrq/lybVYlaHruWZnZyM5Obn68djY2IDLz5o1Cw8//HCD69ywYQMGDRqE22+/vfqxYcOGISYmBj//+c8xZ86cetffHAyuwSjMg73oXVi9jnWCiQ/XaZcC8YnwPPMrICEZ7h/fAbvgINS6RXXXEx0LtW8n7AX/T6/T6jcMrrN+Ds9Ls4GDe531TTgfVo8B8LzxRyBvH5DcCfCYL1n4sSxEXXQr7O9XwPP6EzrQRl32K6gjh6HWLq67+MhT4R47FZX/eRrI2QkkJut902wPPB+9BOzfow9o6NwNUZffCc+BvVDZpg9s7ZynBKpgDay4TCcgVj9eDLXrdZ8FXbC6nw9VvL2RwHos1IEvgYrDgCsOsOr+87fSxgJl+xEuyi7TAc6KSocl2+DDlTAaqvIQVNFiwIqGK2EMEDsAqmxDwHU1trwVPxSWFQO74BPAFQtX4jjALoaqqPWDIiQSBMrhseUHcimAZLhdQ2GrMigc1oFXts1jy7+jaLhdw2GjFErlBt6nRpZv6voiXXJysl9wrc8dd9yBq666qsFl+vbtG/DxMWPG6LLw9u3bMXDgQJjC4BoEtXmF/mt1yfIPrlExsAadCM8rc4CyEj3ZKz6Da+gEeAIF1/wDUN9+VLNeyVgP58DK7AclwTUuEdbxU+CZd58TWEXBwfDvYKdMoFMG7AVv6eCIgzmwVy6Ea+REeGoHV8uCe+L58Lz9dyewiqIC/9Hf+3wPXlX1nbQuAINr05RkO39j0vyDa20JPZyDfHHV51GHBSt1ONTBb5zAKmwJBLUkDQIq8nXwRrRkYmFQ6fyI1Nkm/IMrXAlQFaud74wq15mo5e5Yf+eWBpd3w4ruBrvoK3lRwK6EKtsGK6an4eAq2aPvj5oCKOTBslKgVD4sqws89nfONqBSZ5ouKxOegMHQ1cjyjc1vG+T9t0M9iUQTl09PT9dTc6xcuRIulwtdunSBSQyuoUjLgCWlz31VB0Eht8fOCO75CUlAWibUfuf5VmZfwFMB69gxTlnYUwm18VvYX/3XCXrhYln+f6tuW13loF1Lp0xYHaR02BvuGVcDLhfUljXwfPIqUF5zwHZfdCusvkP0+6Nyd0JtdH6gkHlWYn+geFv9owOjkmG546Fi0mCljXECcekeqMMrAFXhLONOhJU0CCrnff23NaiyrbCis6A8+To7s6IyoSp2NG95VyIsKZV7an74KbsAlvuYMO+FCxaSYSv5cZwAy5I+o4U+G10IWD3reW5jyzd1fa2jLQ/FWbRoEZYsWYJJkybpHsNy/7bbbsNPfvITdOxo9gclg2sopMwrAUXKn1VUWbFuh22Uy61Lwur7b4HcqgNCfCKs2ARYHbvA88/f6EzWfcEtsCpK62/HNeFgDpB3AK6J58H+4r86y3QNnwDExtdZ1IpPdP72GYzKfzodC9znXw/3lEvhefe56uU8rz/uBOisAbB6DdTt0RQG7kQgLgMqp4EfL+4Y/cfSyznt6Fbn8bA6joY65FQmJOiqvFWA3Xqfk6rcB1f8CFjJZ+ogoir2QpXvbN7yVhSUkuzO5yitf0iE95DnsgZCoRgKUlqX7NXjtw1KZ5z1bYO7keUbm0+NkTbV1157TXeKkp7Hffr00cHVtx3WFH4qoagoA6JjAPk1WRVgLQlIPhlcvYH13BucDlAfvVDzeLnTzdz++m1n3RVlsJd/qrNYTziDq+1B5RtPwH3GpYi69U9AwWHYq76Ca9TEOouq6m18FygprL4tAbbuwgpq5/ewBp8I17jpsL96J3z7cJSyOvQDyg8DFXn1L2Q7bfYqfx1gl1XflgCrJfRxqhY6+20t0bpNVJVuhCrfXtUZaSis+FFQJcubvrwOrO6qNlHlc7gLX/8Fp2NTfFX7q5BA6PLbBqvBbWhs+aaur3W05XGuo0aNwuLFdfuRhAODaygO5QAeDyBtsd7ss0tPYP/uhgPrOdcDbjfs//7Fr9zrLQ+3iv174Hnl0eq7rtMu1IGxjoN7oSqalt1YbjeQ1tXEVlJtiX2hCqSHbAMqC6CqAmwgktEipjOs7j+qeiDKqTp0vwBq95toEbpN2a179DoqdNB0JY6FKmnG8naRUyZ3JQN2vrNb0s7rORLGwJpcFVi9/6aLq8JEYk0p1+ognRTqWUtjyzd1fa2jLZeFWxJPIhEMyUzdUTow6l/43tuV5bqs6xo/0xlak9oFrpGnwV6zsOHAGhPrBNbavYDzD8Devg6ucWfrzlJITIVr1Gmwt6wM/z526eFk4S43rEGjdVnY8+X/6i5XWQG1ZhFcJ80A4hKA2AR9W30vnSwAdO0Jq88QZxiO5YLVfzis48bB/mFt+Peh3bF8MhXvbZ9/stKLWHrcFjXQS1hIKbF4mzMMx5JKS7Rzu6rDlDq8HGrvO7q9VSYUbgZKc53bLbJPFuAp1NmmDNVxHnfDiukFSHtqII0u74Gq2ANXnLQfy79XaXLpA1W+I0yBNaUqsPr+m7ah1D64XH2qsuh4uKzusFVVp646Glu+qeuj1sTMNQiucWfBdfK5NfdvfwZq50Z4Xp8L+9OX4ZpyBdzXP+qMD5Vxrj49hV0X3Aq1axPUkveBbv3gOmakzvzcNz1RvYy9+D1nvtx+7x9wTbkS7hv/pHsfq/WLoZZ+GP59lNLt6Mk6KEoHJM8bf67u9eu+5HaonZucUrActj5+Ge7pVyDq5kedYLtppdOhSf8OccE1+QJY0gNZeg7LD4ZPXg04pIcaZqUMhZUyrOZ+z0uhJOjt+8SnI9OOmk5Jvs9NnwRVtg+oymplTKvV8URY3Wc6wbZklw6qzsxywONTjZD1yTKeEvP7FDsArria4Q7ulLOgKg/ALvoGdvFSuOKOhRV3rPPd8RyCXfydT+fgMVCeQ1Blm3XwbGx5VbIGiB8GV/IUvT96nKvRnsIiFi5Xdyhlw+0aV/PaKhe22gRbbYYLA6rmecel1vTsdbmG6l7FSjltxY0t39j8tkBV/ReKUJ/fFlhKNX7lPDntVEpKCvLz84MacxQpKudeg/ZGlYaxV3EriL5nHtobe+dLaG9USvs5LnipDu1rnwoKitAp7aywHce9ceIXve5CbK0xzE1VZpfiiR1zIjrmMHMlIqKjokNTS2KbKxERkWHMXImIyBj2FnYwuBIRkTHSi0eFevrDdhBcWRYmIiIyjJkrEREZY9d/luughfr8toDBlYiIjGGbq4NlYSIiIsOYuRIRkTkGOjShHWSuDK5ERGQM21wdLAsTEREZxsyViIiM4ThXB4MrEREZw7Kwg8GViIiMkQutqRBTz1Cf3xawzZWIiMgwZq5ERGQMTyLhYHAlIiJjeD1XB8vCREREhjFzJSIiY1gWdjC4EhGRMQyuDpaFiYiIDDuqM1dV6kF74zlcifYkxmp/X1F7ySNob5Sr/f1Oj0o5B+1JlLugBTs0qZDXEena35GLiIhaDcvCjvb3c5OIiKiVMXMlIiJjeOJ+B4MrEREZI+2tdshtrpEfXRlciYjIGGauDra5EhERGcbMlYiIjOH1XB0MrkREZAyv5+pgWZiIiMgwBlciIjJ+Egk7xClcfv/73+Okk05CQkICUlNTAy6zc+dOzJgxQy/TpUsX/OpXv0JlZdPOfseyMBERGWMbGIpjh3EoTnl5OS688EKMGzcOzz77bJ35Ho9HB9aMjAx888032Lt3L6644gpER0fjoYceCvp1mLkSEdFR44EHHsBtt92GoUOHBpz/8ccfY/369XjppZcwYsQITJ8+HbNnz8ZTTz2lA3OwGFyJiMjsiftViFPVugoKCvymsrKysG//okWLdODt2rVr9WNTp07Vr79u3bqg18PgSkRExsvCdoiTyMrKQkpKSvU0Z86csG9/Tk6OX2AV3vsyL1gMrkRE1CZlZ2cjPz+/errrrrsCLjdr1ixYltXgtHHjxhbddnZoIiIiY3zLus3lHeaanJysp8bccccduOqqqxpcpm/fvkG9tnRkWrp0qd9jubm51fOCxeBKREQR3Vs4PT1dTyZIL2IZrrNv3z49DEd88sknOsgPHjw46PUwuBIRkTG2MhBcw3iGJhnDeujQIf1Xht2sXLlSP96/f3906NABU6ZM0UH08ssvxyOPPKLbWe+++27ceOONiI2NDfp1GFyJiOioce+99+KFF16ovj9y5Ej9d/78+Zg4cSLcbjfeffddXH/99TqLTUxMxJVXXokHH3ywSa/D4EpERMbItVhVG76e67x58/TUkF69euH9998P6XUYXImIyBhl4Ko2CpGPwbU5OqbDPe1yWN37ARXlsJd+DHvRB/Uubo04Be6TpgNJaUDxEXg+ehlq03eAOwruS38JK70bEBUNHMmDZ/GHUN8tQItL6Yjo86+Eq+9A/c22t6xHxZvzgKIjdRZ1jz8D7hMmwMrMgr1hFSqef7zuMmMmwj1pBqyUjnodFf/9F+x1K8K6C2eeeSbuvPNXGDr0OFRUVGDhwi9x6623Y/fu3dXLnHvuOZg792F0794dK1Z8h5/97Dp8//339a6zseWbur4m6TICSB8MxHcG8rcDm/9XM2/QhUCHTED5HMZWPw9UFAVeV0PLR8UDPScCyT0AdwxQmg/s/gbI+wGmWYkDYCX2AaJTgdI9sA9+6cxwJ8DVdUathd1Vyyysf30J/WAlHaufD7sUdt5yoLTq845JhytlJBCdDKhKqKJtUAWrjO8TUSAMrk1lWYi66FbY36+A5/UndKCNuuxXUEcOQ61dXHfxkafCPXYqKv/zNJCzE0hMBqKrGsVtDzwfvQTs3+Mc9Dp3Q9Tld8JzYC9U9qYW3S0JrKJs9q16H6MvuwHR512BipeeqrOsyj+Myk/ehmvAEFgpaXXmu8dOgvvUaaj411+gdu8AOiTDigm+I0BzpaQk4+GH52LBggX6klVPPvkE3njjNZx88gQ9f8CAAXj55X/hoosuxaefforf/OYuvP32fzBkyDDdsaG2xpZv6vqarKIQ2L0ESOkFxHSoOz/7SyD3u+DXV9/yElCL9znz5TVT+wL9ZgDrXgZKD8Ek5SmGKlgHKy4Dlju+ZoanGPaef/ss6YIrcyZU8Y5612Ul9oPVYRDsQ18DFYcBV5wTkJ25cHU6BapwA9T+DU7wTj8N8BRBFW0xuk8UWecWbik8iURTdcoEOmXAXvCWDo44mAN75UK4Rk6su6xlwT3xfJ2p6sAqigqAvP3ObekRt2+XTzZR9YVKc7p/tySrUxd4Vi0BysuAslJ4Vi6Gldkj4LL2mmWw1y4HigoDrMhC1LQLUPnffzmBVRQWQB2q2ucwevXV13Q7SVFREYqLi/H443/GmDEn6g4K4ic/uQzz53+B9957T59Gbfbs3+mu9hMmOMG3tsaWb+r6muzwFiBvK1BZgrAqywdyljuBVUjGKkFVMl3TSnc5k93waeys+B76u6RKsutbAlbyMCdTlcAq7FIdPJ3Z0bDcsTpb1f+uJKiW5jgZM7XI9VxViFOkY+baVJbl/7fqttU1QCDqlAmrQwqsjN5wz7gacLmgtqyB55NXgfLS6sXcF90Kq+8QWFHRULk7oTaGt3waiGfBB3APHwN7vXRLt+AeOQ72+iZkRVWsLpmwklNh9eiD2B9fA7jc8Gxchcq3XwHKwhwkajn11FOwYcOG6ixy2LChWLmypiwol5Bav36DfvyLL76o8/zGlm/q+ozrNhboPg4oK3CC48ENZpaXMnF8J6DkAFqLZKWqeHv9rXdRSTrztWLSYHU8UecJqnQPVP4KXQKGKoddtNVZz5F1gDtRZ8t23rctvSt0lGJwbaqDOUDeAbgmngf7i//qLNM1fAIQ61PiqmLFJzp/+wxG5T/v17fd518P95RL4Xn3uerlPK8/7gTorAGweg0EKoO/8oIp9rZNupwb+7tn9H21YwsqPn2n6StKcMqXUjIu+9M9+nb05TchauZlqHz9n2gpcjWL2bMfwIUXXlz9mIxhy8vL81tO7iclJQVcR2PLN3V9Ru36Cig5CNiVQHIW0O8swK5wst1QlrdcQP8ZwKHvgSLnrDQtTtpPY7tC5Tfw487lNDNYsRmw933kPJR2MpA6GurwEn1fFe+Eq+OJsJKPg2W5YBd+D5TubZl9OIqxLOxgWbipbA8q33gCVtdeiLr1T4ia+X+wV30FFNctkSopscpTvn4XKCnUk9y2Boyou14phez8HkhMgWvcdLQoy0LM/83SAbbsrp/pSW7H/N+dTV9XmZORez57xykbFxXq2+7Bzlgyky699BIcOZKnp7VrazLI4447Dh988C5uuukW3RbqVVhYqE/+7UvuHzlSt9NWMMs3dX1GFe4FPOVOk0L+DmD/aiBtQGjL68B6thOAt32C1iLZpi71Vvj/cPGjKvQfW7JSKTHbZfq2FdfdmR+VBFfnU2Dnr4C9+3V49vwHVlQKrJQA//aozZ64P5Ixc22O/XvgeeXR6ruu0y50AmNtB/dCVTQtC7WkfTDN/4oMYZeQCCstHZVffqx7P4vKrz5G3OSzgMQOgdtW66H2N32fm+uVV17Vky8JrJ9++hFmzfoNXn75Fb95q1evwYgRw6vvR0VFYfDgY7FmzdqA629s+aauL6ya2kZVe3kdWM9yOgRtftu/V3ELsxL6OqXchlQcgZLyb32kbdVTDHjbbO1SqOIfYCUNhsp3zshDFE7MXJujSw8gOka3J1qDRuuysOdLn2ESXpUVUGsWwXXSDCBOSl0J+rb6vqrc1bUnrD5DnGE4lgtW/+GwjhsH+4cWPjgXFcLen4Oo8ac72xIVjaiTz4A6fDBwYHW5nOXkr1V1u6rTECoq4Fn+NdwSmOMT9H7LbU+Yh+EIOWWZBNa7774X8+bVnIHF66WXXsbkyZP0xY9jYmLw29/+BgcOHMDChYGHejS2fFPX13RWVe9Xq+a2vN/uWCClD+CS38aWU+btMgw4tDnwahpb3htYXdFVgdVAT+eG9kkfdiyf2z6HodhMXfJtqJewwwNVtB2upMG685JMcluV7HJmlx8CXPFAXFVfCFcsrIQ+UOVVnZ8obMzkrTYinaWC6JYlF4mVcpdc8ieYKxREiorZDV9FoT6uiefDNXqyDirSAcn+9HWoXU7blfuS26F2bnJKwSI6Bu7pV8AaOMoJtptWVndosjJ7w3XmFbCkB7J8DPkHYC/7HPaK5neG8Rxu4Nd8A6yu3RB17k/gyuqjD7b27u2o/N8rusdv1I+u1stU/r/n9d+oqefryZe9ZQPK//p7505MrDNmduho6eWjA2vl2y9Xl4ybIv5PrwW97HPP/RNXXnmF7insa/DgofrSVWLmzHPxyCN/QI8ePfS41GuuubZ6XOr48eN1OTkpqaZHaUPLBzM/EHvJI8HtUPdxsKQDkg9VkA1seRcYMBOIrxoGpTsorQAO+GR7A84DjuwG9i51Oig1tHxSD1jH/hhKysG+Geuepc7zg2B3C+5qIVbyULiSh/rvU1ku7P2fOfOl3VR5oA7XHdbm6jwRqmwf1JH1VStzw0o9welZLM8p2V3ToUnEdXdeK6qDM780x5nfSE9lL3ePS9GehPs47l3/uOQbEGWFNvSuUpVhUcFfIzrmMLi2M80Nrm1VU4JrpAg6uEaQYINrJGFwbd76xyRfbyS4Lil4OqJjDsvCREREhrFDExERGWNX/ReKUJ/fFjC4EhGRMcpSUJbdZq+K01JYFiYiIjKMmSsRERmjDJwEQrWDzJXBlYiIjJH2UottriwLExERmcbMlYiIjDFxhiXVDjJXBlciIjLGtmxYIfYWZlmYiIiI6mDmSkRExrBDk4PBlYiIjGFwdTC4EhGRMezQ5GCbKxERkWHMXImIyBgbHljwhLyOSMfgSkRExsipC1XIZeHIP/0hy8JERESGMXMlIiJjeBIJB4MrEREZbnN1hbyOSMeyMBERkWHMXImIyKDQx7nKOiLdUR1co++Zh/YmGu2Leuyl1t4ECoK7tTeA2gxbSUnXZWAdkY1lYSIiIsOO6syViIjM4ukPHcxciYjIGAWPkSlcfv/73+Okk05CQkICUlNTAy5jWVad6bXXXmvS6zBzJSIiY5wxqraBdYRHeXk5LrzwQowbNw7PPvtsvcs9//zzmDZtWvX9+gJxfRhciYjoqPHAAw/ov/PmNdyhVYJpRkZGs1+HZWEiIjJ+bmEV0uScW7igoMBvKisra7H9uPHGG9G5c2eceOKJeO6556BU0853zMyViIiMUUraTK2Q1yGysrL8Hr/vvvtw//33I9wefPBBTJ48WbfLfvzxx7jhhhtQWFiIW265Jeh1MLgSEVGblJ2djeTk5Or7sbGxAZebNWsWHn744QbXtWHDBgwaNCio173nnnuqb48cORJFRUWYO3cugysREUV+h6bk5GS/4FqfO+64A1dddVWDy/Tt27fZ2zNmzBjMnj1bl6XrC/C1MbgSEZExzlAaK+R1NEV6erqewmXlypXo2LFj0IFVMLgSEdFRY+fOnTh06JD+6/F4dOAU/fv3R4cOHfDOO+8gNzcXY8eORVxcHD755BM89NBD+OUvf9mk12FwJSIiY5QycIYmFb5xrvfeey9eeOEFvzZVMX/+fEycOBHR0dF46qmncNttt+kewhJ0H3vsMVx77bVNeh1LBdG/WLpAp6SkID8/P6j6NxERtS3hPo571985eQxcVmh5m60qcaBgSUTHHI5zJSIiMoxlYSIiapPjXCMZgysRERk/Q1MovGdoimQMrkREZLhDkxXyOiId21yJiIgMY+ZKREQGSZtr6OuIdAyuRERkjFPStQysI7KxLExERGQYM1ciIjKGmauDwZWIiIyRK9pYIZ+4P/KDK8vCREREhjFzJSIiY1gWdjC4EhGRMSZOXajawekPWRYmIiIyjJkrEREZ45wX2DawjsjG4EpERMaYaC9VbHMlIiKqweDqYJsrERGRYcxciYjIGBMngFDt4CQSDK5ERGQMy8IOloWJiIgMY+ZKRETGMHN1MLgSEZFBJgJj5AdXloWJiIgMY+ZKRETGsCzsYHAlIiJjOBTHwbIwERGRYcxciYjIGKUMnLhfryOyMbgSEZFBci1WK+TCcKRjcCUiImOczkhWiOuI/ODKNlciIiLDmLkSEZFBoWeuLAsTERH5MlAWBsvCREREVBszVyIiMkYZKOmqdlAWZuZKREQGSVnYxGTe9u3bcc0116BPnz6Ij49Hv379cN9996G8vNxvudWrV2PChAmIi4tDVlYWHnnkkSa/FjNXIiI6KmzcuBG2beOZZ55B//79sXbtWlx77bUoKirCo48+qpcpKCjAlClTcPrpp+Nvf/sb1qxZg5/+9KdITU3FddddF/RrWSqIAUXyYikpKcjPz0dycnJoe0dERC0u3Mdx7/olZ7NCHeeqy8KVLRJz5s6di6effho//PCDvi+3f/vb3yInJwcxMTH6sVmzZuGtt97Swdlo5uqNv/LmERFR5PEev8N/ggYJjZHTZioBPC0trfr+okWLcMopp1QHVjF16lQ8/PDDOHz4MDp27GguuB45ckT/ldozERFFLjmeOxmmWRKMMjIydMZngqyrtLTU77HY2Fg9mbJlyxY8+eST1SVhIdsvbbK+unbtWj3PaHDt1q0bsrOzkZSUBMsKdXAwERG1NMlYJbDK8TwcpPPPtm3b6nQOai7pROQNal7S+ej++++vs6yUbSWzbMiGDRswaNCg6vu7d+/GtGnTcOGFF+p2V9OCanMlIiJqSWVlZXoKJnPdv38/Dh482OD6+vbtW13q3bNnDyZOnIixY8di3rx5cLlqBs5cccUVuoQubaxe8+fPx+TJk3Ho0CGzmSsREVFLim1CCTg9PV1PwZCMddKkSRg9ejSef/55v8Aqxo0bpzs0VVRUIDo6Wj/2ySefYODAgUEHVsHMlYiIjgq7d+/WGWuvXr3wwgsvwO12+7Xxejs4SSCV4Th33nmnHq4jQ3H+9Kc/mR+KQ0REFOnmzZuHq6++OuA831AoJ5G48cYb8e2336Jz5864+eabdaBtCgZXIiIiw3j6QyIiIsMYXImIiAxjcCUiIjKMwZWIiMgwBlciIiLDGFyJiIgMY3AlIiIyjMGViIjIMAZXIiIiwxhciYiIDGNwJSIiMozBlYiICGb9f0VZb1ZQ+9FHAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAGgCAYAAAC0SSBAAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAKWRJREFUeJzt3Qt4FNXdx/F/AiEQICAXBSXiDaEaRS6l1As3jVC8IAilBkVrfYsXkEu0NiKQKFIaLJZXUbT4AtXGFqvSIi0SrmrpQ7FoJURAqKIgYlRICMFcyLzP/8QNm01ysoGwuzP5fp5nSHZ2yJ4zMzu/mXPO7EY5juMIAAA1iK7pCQAAFEEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBEYEWL14sUVFR8sknnzSo1z7nnHPkjjvuqHi8fv16Uxb9GWoZGRnSrVs3KSsrO2WvoXVLS0uThrSNT5SuJy3zqbRy5Upp0aKF5ObmntLXcSOCIgjbtm2TW2+9Vc466yyJjY2VM888U8aMGWPmn4xZs2bJsmXL6q2cqB/5+fny61//Wh566CGJjm4Yb5FnnnnGBEg4FRYWmkAIx4mBGjJkiFxwwQXyq1/9KiyvH9H0s55Qs1dffdVp0qSJ06FDB2fq1KnOwoULnUceecTp2LGjmf/aa6+d8N9u3ry5c/vtt1eZX1pa6hw9etQpKytzQm3RokX62V/Oxx9/HPLX7ty5c6X1cezYMbMe9GcoPfnkk058fLx57VNJ1/OMGTOcUKtu/7r44oud/v37O+GUm5tb4zopKSk55dtDPfPMM05cXJyTn59/yl/LTRrG6dIJ2r17t9x2221y3nnnyQcffCAzZ86Un/3sZ/LYY4+Zxzpfn//vf/9br6/bqFEjadq06Sm/1I50ejav6yHUZ/WLFi2SG2+80by2F4Vq/yotLZXi4uJ6+VuNGzcOyfa4+eabpaioSF555ZVT/lpuQlBYzJkzx1wOP//889K+fftKz7Vr106ee+45OXLkiGnPDmxL3b59u/z4xz+W+Ph4adu2rUycOFG+/fbbiuV0Gf2/S5YsMb/r5Gufr64NWdvvr7/+enNZ3rt3b2nWrJlccsklFZfpr732mnmsb6ZevXrJe++9V6m8Gmz69zXcdJkOHTrInXfeKV9//fUJrRv9W9qeqyE5ePBgad68uWmSe/TRR/UqtdKyWs+UlBRJSEgwTXddu3aVJ554ospygWrqo9i0aZMMHTpUTjvtNPO6l156qcybN6/iIK//J7D+vqY+PUju27evxtf8+OOPzbq65pprKuaVlJRImzZt5Kc//Wm1zVS6Ph944AHzWA+M06dPN9ugVatWpnxXXXWVrFu3ToJZp7qdg22ff+mll8zr6L6g5fvJT34in332Wa2vE7h/6WtqM+qGDRsq9sUBAwZULH/o0CGZNGlSxfbT5hltmvPvv9G/pf9Pt+tvf/tbOf/8882yOTk5Qa0T/f++91h6enpFOXx9ONWtAw0iPWnzvZbW4+GHHzYHen++984777wjffr0MdtL3we///3vq6yb008/3exPf/nLX2pdjw1KuC9pItmZZ57pnHPOOdZl9PlOnTpVPNbLZl2tl1xyiXPDDTc4Tz/9tHPrrbeaebfddlvFci+++KITGxvrXHXVVeZ3nTZu3Fhj8482y3Tt2tU0eaWlpZnmkbPOOstp0aKF89JLLzlnn322M3v2bDO1atXKueCCCyo12TzxxBPmtR599FHn+eefdyZOnOg0a9bM6dOnT6UmiGCbnrSJqGnTpk6XLl1MvbSe119/vfm/06ZNq1hO//agQYOcqKgo56677jLL6XrR5SZNmmRtelq3bp1ZTn/6rFq1yjT56bK6rp999lnn/vvvd6655hrzvDYZaL1SUlKqlPmiiy4yZbHRdamv+cEHH1Saf+eddzqtW7d2ioqKKs1fsmSJWX7z5s0VzSe6jaZMmWLKlpGRYbZbTEyM895771X6v4HNLFp3rVcg3z7lb+bMmWadjh492jSXpKenO+3atTP748GDB611DNzGr7/+utmHu3XrVrEv6npWR44ccS699FKnbdu2zsMPP+wsWLDAGTt2rHlt3Yd89G/p39R1fN5555n9UPfRPXv2BLVOCgoKzHP6N4YPH15Rjv/85z81rgNdXzpv5MiRzvz580259PFNN91UaTnfe+eMM84wddB9sGfPnqYO2dnZVdaP7qe6LnEcQVGDQ4cOmZ1u2LBh1uVuvPFGs5yvTdO3Q+t8f/fee6+Z79vxbX0UNQWFzvOFiXrzzTfNPD0w6hvS57nnnqtygC0sLKzyOi+//LJZ7q233rK+dnV8b9IJEyZUCoXrrrvOHMj14KCWLVtmltMDmz99c+sbddeuXUEHhbatn3vuuWa5wIOhf9jdcsstJuT9g3LLli3mb2n9bLT/SZc7fPhwpfm+db18+fJK84cOHWoOjD5axsAw0bLqQUrDpj6C4pNPPnEaNWrkPP7445WW27p1q9O4ceMq8wNVt41r6qN47LHHzH66c+fOSvN/+ctfmjJ8+umnlYJC+3a+/PLLSssGu05sfRSB6+D99983j/Wg7u+BBx4w89euXVvlveO/n2sZ9UStuhOKWbNmmeUPHDhQ5bmGiqanGhw+fNj8bNmypXU53/PaBOHvvvvuq/R4woQJ5uff/va3Ey7TRRddJD/84Q8rHv/gBz8wPwcNGiRnn312lfn+fSfaPOGjTWBfffWV9O3b1zzesmXLCZdp/PjxFb9r04A+1qaG1atXV9RXm3vuv//+Sv9Pm6L0WPn3v/896NfS5iRtGtJmkNatW1d6zr9ZYuzYsfL5559Xatr4wx/+YNaBtkHbaFOctodrs5o/Xcfa3PinP/2pYt7BgwclKytLRo8eXTFP69qkSRPzuzbNfPPNN6aJRJsLT2Y9+9NmRv3b2rSp29E3aXNily5dgmrmCpa21WszkTbz+b+WNs0dO3ZM3nrrrUrL6/oNbKY9FevE9z6aMmVKlf1KrVixosp7R+vho2XUJtDq+he1rkrriXKNv/uJGgLAFxh1DRR9w/rTdlTtlD2Zsev+YaC0vVdp23F18/VA5qNvTm37/eMf/yhffvllpeXz8vJOqDxaH23r9XfhhRean7567tmzx/RdBK6f733vexXP12VwgUpMTLQul5SUJB07djThcPXVV5uD08svvyzDhg2rNfhrouGhB8HMzEzTBq5t4nrA1v4L/6BQ2u/0m9/8xvRT6fM+5557rtSHjz76yIRs4D7mExMTUy+v43st7bMJPPj7BO5LNdWxvteJ7je6/2l/iT8NSz2JCNyvAt87vkDwf4/4+PrOGvpgEn8ERQ30YKsHG32T2Ojzen+Fdlrb1MdOp2dmdZnv31msZ58bN26UBx98UC677DJzxqwHUB07fipvKgsHXR/Jycnyu9/9ztwf8I9//MNcYei9MLXRgQd6tqsnAIGhop3FOoBBr4JuuukmWbp0qbkpr3v37pU6mLVTWp/Xda2do1oeHZvvC7q67iN65u5Pt5cuq+WobtsHXg2dDH0tDd5f/OIX1T7vOzGo7sq1PtZJbYJ9XwXzHvHxhYdeQaIcQWGhIyX0YKOjJa688soqz7/99tvmzHncuHHVnon5ny3t2rXLvOn8R7WE6oxFd/w1a9aYKwodfeJfxpOh9dFLd/+Dxc6dO81PXz07d+5smqECD7x6Zul7Plh6Vaays7MrjUqqjjY/6Rns8uXLzQFVz4h1dFZt9MCvtIlLR7/469evnzl50OYn3R/Wrl0rU6dOrbTMn//8Z3OVpVcb/tt3xowZtb62nuHqCKNAgWfHuh70AKf7V+CB+kTVtC/qaxUUFNS6vm2CXSd1eT/ofqP7n+7DvqtTdeDAAbMO67JfBdJtryFR01VUQ0QfhYWe/egZkgZB4DBSbcq5++67JS4uziwXaP78+ZUeP/XUU+bnj370o4p5OkywugNDffOdTQWePekwxpP19NNPV/yuf18fa9OHNvkoHcaqZ8T+y6knn3zSHBj810dtevbsaQ6OWu7A9RZYNz3I67Rw4UJ59dVXzdWANh/VxtcH9O6771Z5Tps6Ro4cacLnxRdfNFcegc1O1a1rHc77z3/+s9bX1oOyNgP6X8Xu379fXn/99UrLjRgxwryOBn9gvfXxiQx5rmlf1CtRLfubb75Z5TldXtdBbYJdJ/pe8v3d2uh+Vd0+PHfuXPPzuuuukxP173//u1JfILiisNI2YG1b1Y/r0HsU9GY7PVDpVcQLL7xgOru07dt3pht4VqI3bWnTjr4h9PJbm0P8myl0XLmebevOre34+rd9HdH1SZvF9GxY7/fQ9mFtKlu1apUp48nQ8ej6+Ti33367KbeeuWsnoo5l952N3XDDDTJw4EBz5q3rTeuvr63j1LVTurp1VxM9UD/77LPmb2rzmd7XoGf4enWi9wEEHsz0qsJ3f0MwzU5Kz3y1D0S3i95nEkiDQUNfz4Z1n/A/m/VdheqZ8/Dhw83BStfxggULTGeqnpnbaJjpx4bo/9XOf72HR+urVw3+nb66zvTmz9TUVLNOtUlHr9b0tTRUfv7zn1fUO1i6L+pr6d/Vdn9tHtIOfD0J+utf/2rqpc1HupzeF7N161ZzpaCvX1sTTbDrRE/KdJ5esWmd9d4Q3RbV9UnpfqT7nd7jpMHSv39/+de//mXer7o+dJ87EdrnokEdOBilwQv3sCs30DH1OuRSx4Lr2G/9OA99rMMRaxrGl5OTY4aAtmzZ0jnttNOc8ePHV/kIgu3btzv9+vUzw1v1//iGhtY0PFaHngbS5e67775K83xDFefMmVMxb+/evWZ8ut4LoPdZjBo1yvn888+rDEesy/BYHTa5e/du59prrzUfe6DDHfVvBX7khg41nTx5shmyqutP773QsgV+REkw91God955x0lKSjLrVsug4/yfeuqpKmXcv3+/GcJ54YUXOnUxd+5cc39KdUOKtcwJCQnVDvn1Pa/DK7UuOvyyR48ezhtvvFHt0NfqhoLq/QuJiYlmiLGO/df7Oqq7h8D38TJXXnmlWQc66X0Qui/s2LHDWr/qtvEXX3xh9i9dp/qc/1BZ3X6pqanm3hwtl95jcPnll5t7c4qLi2vc505knejw7169epnX8V8/1a0D/VgPvX9Eh0zrfqXbRcv57bffVlqupveO1jFwSLDey8FHeFRFUNQz3w7tu4/Aq3xBEcl0G+h9BXqTYV3voWnTpo35XC80LJdddlmVG0HBfRTwMP2oCu0f0c/jquuINx3lox/h4rURYaiZNqNq57g26aEy+ijgOToaST9j6PHHHzft1dV9flJttK9AJzQc2p9YWz9SQ0VQwHP0gwn1npErrriiYrQZgBMXpe1PJ/H/AQAeRx8FAMCKoAAA1E8fhX4Qmv8Xgvg+BVI/G4cPzwIAd9FeB/1oHb3Zt7ZvkQw6KPQDvPQjAwAA3qHfitipU6f66cwOvKLQz6TRj+7VW/zr82ONw0lTVT9CQIdWemH8vNfqo6iTO1CnyKcf56P3CulHoPi+muCkryj08/d1CqQh4ftSEi/sCPrBZFofL+wIXquPok7uQJ3cI5iuAzqzAQBWBAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAABWBAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsGoc7IJFRUVm8snPzzc/o6OjzeQFvnpQn8hFndyBOkW+utQjynEcJ5gF09LSJD09vcr8zMxMiYuLq1sJAQBhVVhYKMnJyZKXlyfx8fH1ExTVXVEkJCTI/v37pW3btuIFJSUlkpWVJUlJSRITEyNu57X6eL1O2dnZUlZWJl45W01MTGQ7RbDi4mKZOXNmUEERdNNTbGysmQLpTuCVHcGrdfJafbxaJz34eOEA5I/tFLnqUgdvNLYBAE4ZggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAABWBAUAwIqgAAC4Myj+mP1HGfzSYDnjiTMk5rEYaTW7lZw771wZsHiATPz7RHlz15vhLiI8bOLEidKhQ4dwFwO1YDuFRtDfcBdKY18fKy9+8GKleflF+Wb65NAnsmHPBtmTt0cGXzA4bGWEt+nXQx44cCDcxUAt2E4NNChW7lpZKSR6dewlg88fLC2atJDcwlzZsn+L/HPvP8NaRgBoSCIuKFbtXlXx+wVtLpBNd22SRtGNKi2jVxZbD2wVL9AvOJ8yZYo8+OCD0rlzZ/ECL9YJaMgiro+itKy04vdD3x4yTU2B4mPj5YqzrxAv2LJliyxcuFD69esnu3fvFi/wYp2AhizigqJnx54Vv39V+JVc+PSF0uv5XnL3G3fL7/79O9n1zS7xkr59+8ry5cslNzfXHFh37NghbufFOgENWcQ1Pd166a0yf/N8effzd83jMqfM9Evo5HPl2VfK0z96Wrp36C6RbNq0aXLw4MGglk1MTJTNmzdL//79ZcOGDdK1a1eJRF6sEwCXBUXj6Mayduxa+dU7v5L/e+//5MCRqiMa3vn0HUl6MUm23btN2jdvL5Fq0aJFsm/fvjr9Hx3BkZOTE7EHVS/WCYDLmp5Uy9iWMuvqWbI/Zb9k35MtL9z4gtze/XZp2aRlxTI6AipwCG2k2bt3rziOU+tUUFAgAwYMMP8nPT1dhg8fLpHKi3UC4MKg8ImKipKLT79Y7uxxpyy+abF8cM8HEh11vMgfff2RuN3hw4dlyJAhsn79epk9e7ZMnz5d3M6tddJO+AULFlSZ/+GHH8q8efPCUiZUxXYKvYhrelry/hL5tvRbueWSW8zoJn/NY5qboNB+C9W6aWtxu127dsnWrVtl7ty5MnnyZPECt9Zp6tSpsnLlSiksLKyYt23bNhk0aJAcPXpURowYIQkJCWEtI9hO4RBxQfHxoY8lfUO6THpzkum0vuyMy6RNszby9dGv5c85f640fHbIBUPE7Xr06GEOrO3atROvcGudli5dKkOHDpWUlBRp376872vgwIHmvpBVq1Zx8IkQbKfQi7ig8NGritX/XW2m6vxPz/+R/uf0Fy9w2wHVq3Vq2bKlOVO94YYbZN26dWZeaWmprF69Wnr37h3u4uE7bKfQi7igmNR3klxy+iWy9uO18u7+d+WLgi8k90iuHHOOSfu49tLrzF6mY3vE90aEu6jwoObNm8uKFStk2LBhpi08KyvLXCEhsrCdGnhQaL/DzRfdbCYgHJo1a2aaMBDZ2E6hE9GjngAA4UdQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAABWBAUAwIqgAABYERQAACuCAgBgRVAAAOrnq1CLiorM5JOfn29+lpSUmMkLfPWgPpHLy3WKjvbOeZuvLmynyFWXekQ5juMEs2BaWpqkp6dXmZ+ZmSlxcXF1KyEAIKwKCwslOTlZ8vLyJD4+vn6uKFJTU2XKlCmVrigSEhIkJydHmjRpIl5J2MTERElKSpKYmBjxwhlQVlaWZGdnS1lZmXiB17aR/3aiTpGtxGPvp+Li4qCXDTooYmNjzRRIV5gXVpo/3bG9snMrtpE7UCd3KPPI+6kudfBGYxsA4JQhKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQRFipaWlMmbMGOnWrZvs3Lkz3MVBDSZOnCgdOnQQL/FinRAaBEWIv3pw1KhR5nvGd+zYIQMGDJDt27eHu1iohn6P8IEDB8RLvFgnhAZBESJFRUUyYsQIWbZsWcUXmRcUFJiw2LZtW7iLBwA1IihCZPTo0bJixQpJTU2V4cOHm3mrVq2So0ePysCBA2Xv3r3hLiLgmu96TklJkZycnHAXpcFoHO4CNBSTJk2S73//+zJ16lS54447zLy+fftKVlaWLF++XDp16hTuIgIR79ixYzJ27FjTfBsdHS1z5swJd5EaBIIiRLSJSadAffr0MROA2geCJCcnyyuvvCL33nuvZGRkhLtIDQZBAcAVdCCI9vG1atVKoqKiZMKECUGP9urSpcspL5+XERQAXNEvsX79+orRW/Pnzw/6/44cOZKgOEl0ZgOIeNofsWbNGmnTpo20bt1aNm3aJI7jBDVV1+SLuiEogO9s2bJFFixYUGX+hx9+KPPmzRM38lKdevbsKWvXrpXGjRtLUlKSbNy4MdxFajBoegK+oyPSVq5cKYWFhRXz9B6XQYMGmWHMeh9MQkKCuInX6tS9e3dZt26dXH311eYK4/LLLw93kRoEggL4ztKlS2Xo0KFmjH779u3NPL3HRe+o13te3HRA9XKdEhMTTdi1a9cu3EVpMGh6Ar7TsmVLc/atB9Lc3NyKIZmrV68297y4kRfrpAiJ0CIoAD/Nmzc3d9BrG3jbtm1N80bv3r3FzbxYJ4QWTU9hsHjxYjMhMjVr1sw0y3iJF+uE0OGKAgBgRVAAAKwICgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAA6uerUIuKiszkk5+fb35GR0ebyQt89SgpKREv8NXDK9vHi9vIvy5erFNGRoaUlZWJV/a9xMREz7yf6lKPKMdxnGAWTEtLk/T09CrzMzMzJS4urm4lBACEVWFhoSQnJ0teXp7Ex8fXzxVFamqqTJkypdIVRUJCguTk5EiTJk3ES2cMSUlJEhMTI144q8vKypLs7GzPndV5ZRv5bycv1smL+162R+pUXFwc9LJBB0VsbKyZAukK88JK86dvVq+8YRXbyB28WCcv7ntlHqlTXergjcY2AMApQ1AAAKwICgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4IixEpLS2XMmDHSrVs32blzZ7iLgwZk4sSJ0qFDh3AXAy5EUIT4qwdHjRplvmd8x44dMmDAANm+fXu4i4UGQr8b+cCBA+EuBlyIoAiRoqIiGTFihCxbtqzii8wLCgpMWGzbti3cxQOAGhEUITJ69GhZsWKFpKamyvDhw828VatWydGjR2XgwIGyd+/ecBcRtVwNjh8/Xvbs2RPuogAhR1CEyKRJk2TmzJkya9asinl9+/aVrKwsGTdunHTq1Cms5YPdli1bZOHChdKvXz/ZvXt3uIsDhFTj0L5cw6VNTDoF6tOnj5kQ2TTUly9fLsOGDTNhsXbtWunatWu4iwWEBEGBBm/atGly8ODBoJZNTEyUzZs3S//+/WXDhg2EBRoEggIN3qJFi2Tfvn11+j86eignJ4egQINAHwUaPB1I4DhOrZNvlJpKT0+vGJQAeB1BAQTh8OHDMmTIEFm/fr3Mnj1bpk+fLm7phF+wYEGV+R9++KHMmzcvLGWC+9D0BARh165dsnXrVpk7d65MnjxZ3GLq1KmycuVKKSwsrJin9+0MGjTIDM3We3sSEhLCWkZEPoICCEKPHj1MWLRr107cZOnSpTJ06FBJSUmR9u3bm3l6347eF6L38RASCAZNT0CQ3BYSqmXLluaKQsMhNze34vPGVq9ebYb8AsEgKACPa968uflUgKSkJGnbtq2sWbNGevfuHe5iwUUIijBYvHixGUUDhEqzZs1MU9NXX31lmtGAuiAoAABWBAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAqrEEqaioyEw++fn55md0dLSZvMBXj5KSEvECXz1SUlIkJiZGvFKnrKwsycjIkLKyMvHKfpeYmOiZ/c7r+16KR+r09ddfy6xZs4JaNsoJ8sub09LSJD09vcr8zMxMiYuLq3spAQBhU1hYKMnJyZKXlyfx8fH1ExTVXVEkJCTII488Ik2aNBEvndklJSV54ozBdwbklfr41yk7O9tzVxRe3E7UKbKvKDp27BhUUATd9BQbG2umQPpm9cob1kd3Ai/sCF6tj2K/cwfqFLnqUgdvdC4AAE4ZggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAABWBAUAwIqgCLHS0lIZM2aMdOvWTXbu3Bnu4gCIAF9/LZKRIXLttSJnninStKl+B5BIx44i/fqJPPigyNtviwT3NXP1L+gvLsLJKy4ultGjR8uyZcvM4wEDBsjatWtNaABomJ5/XmTKFJEjR6o+98UX5ZOGxBNPiOzfL9KhQ+jLSFCEiH6N7M033ywrVqwwXzuoXyVbUFBgwmLNmjVy8cUXh7uIAEJszhyRX/zi+OOoKJGBA0X69hVp0ULkm29E3n9f5J13RL79NnzlJChCRK8kNCRSU1Pl888/lyVLlsiqVatk8ODBMnDgQNmyZYt06tQp3MUEECIffiiSmnr8cdu2In/9q8jll1ddtqBA5MUXRZo1k7CgjyJEJk2aJDNnzpRZs2ZVzOvbt6/5svZx48a5MiT0O6tTUlIkJycn3EXBCTSDjh8/Xvbs2SNe4bY6/e//ihw7dvzxggXVh4TSq4t77hFp1UrCgqAIEW1imjp1apX5ffr0kccee0zc5tixY3LbbbfJ3LlzZdGiReEuDupIr2AXLlwo/fr1k927d4sXuK1Oa9Yc//2000RGjJCIRVDghEZu3XLLLZKZmSn33nuvZOhwDbiKXs0uX75ccnNzzYF1x44d4nZuq9O+fcd/79JFJNrvaLx9e3l/ReB0xx1hKSp9FKi7UaNGmZFbrVq1kqioKJkwYUJQ/2/ixInSRd8ROKWmTZsmBw8eDGrZxMRE2bx5s/Tv3182bNggXbt2lUjkxTr50xCIZAQF6twvsX79evN7Xl6ezJ8/P+j/O3LkSIIiBLQpcJ//6WoQDhw4YPqaIvWg6sU6nXWWyEcflf+uP/UeCV9gnH56+YgoNWOGSGFh+MqpaHpCnURHR5vhvG3atJHWrVvLpk2bxHGcoCbtp8Gpt3fv3qC2h294tkpPT5fhw4dLpPJina6++vjvOgxWRzz5tGkj8sAD5VO4Rjr5IyhQZz179jQ3CjZu3FiSkpJk48aN4S4S6ujw4cMyZMgQc3U4e/ZsmT59urid2+o0frxIo0bHH999d/k9E5GIoMAJ6d69u6xbt06aNm1qrjDgLrt27ZKtW7eaUWsPPfSQeIHb6nTxxSL+Ax71DuzevUVuuEEkLU3k8cdF7rpLJD9fwo4+Cpww7TTctm2btGvXLtxFQR316NHDHFi9tO3cWKfUVJHmzcvvzi4qKr+v4o03yqfq6E154UBQ4KS46U0J7287N9bp/vt1JGH5Zz6tXi2io3p1gFdMjEj79iLaF3/FFSI33qhhGJ4yEhRhsHjxYjMBgNJPidXRTTpFIvooAABWBAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAANTPd2YXFRWZySc/P9/8jI6ONpMX+OpRUlIiXuCrh1fq41+XlJQUidFvn/dInbKysjy5nbxYp4yMDCkrKxO3Ky4uDnrZKMdxnGAWTEtLk/T09CrzMzMzJS4urm4lBACEVWFhoSQnJ0teXp7Ex8fXzxVFamqqTJkypdIVRUJCguTk5EiTJk3EK1cUiYmJkpSU5ImzVd+Zqlfqo6iTO3i5TtnZ2Q3uiiLooIiNjTVTIF1hXlhp/nTH9srO7cX6KOrkDl6sU5lHjnl1qYM3OhcAAKcMQQEAsCIoAABWBAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQhFhpaamMGTNGunXrJjt37hQvcH2ddu8WadFCJCqqfLr2WpHAL37Ux0lJx5dp3lzko4/CVWJ4Yb9zEYIixN8oNWrUKPP1sTt27JABAwbI9u3bxc08Uafzzxf5zW+OP87KEpk/v/IyTz8tsnr18cdPPCHSpUvoygjv7XcuQlCESFFRkYwYMUKWLVtW8f20BQUFZgfftm2buJGn6jRunMjQoccfP/SQiO8sVX/qY58hQ0TuuSf0ZYT39juXIChCZPTo0bJixQrz3ePDhw8381atWiVHjx6VgQMHyt69e8VtPFenhQtF2rYt/72wUGTsWD0qidx2m8jRo+Xz27QReeGFsBazofPcfucCBEWITJo0SWbOnCmzZs2qmNe3b1/zZe3jxo2TTp06idt4rk4dO4o8++zxx5s2ifTpI/Kvfx2fp8+feaZ4oelm/PjxsmfPHnEbz+13LtA43AVoKPSyWKdAffr0MZMbebFOMmqUyJgxIn/4Q/njDz44/lxyssiPfyxesGXLFlm4cKEsX75c1q5dK+drP41LeHK/i3BcUQCBtONary78nXFG1Q5uF9MzcA2J3Nxc6devn+kQBmrCFQUQSNu4v/mm8jx9/MknIpddJpFu2rRpcvDgwaCWTUxMlM2bN0v//v1lw4YN0rVr11NePrgPQQH4Kykp77zWTuzq5r/7rkhsrESyRYsWyb59++r0fw4cOCA5OTkEBapF0xPgb8YMkfffP/74vvuO/56dLfLIIxLpdNSP4zi1Tr4hpSo9Pb1iBBEQiKAAfDZuFMnIOP74zjvL+yt+9rPj8+bOFXn7bXG7w4cPy5AhQ2T9+vUye/ZsmT59eriLhAhGUADqyBGR228XOXas/PE554j89rflv+vP884r/72srHy5ggJxs127dsnWrVtl7ty58pD/zYRANQgKQKWk6NGz/PfoaJElS0Ratix/rJ8D9fvfizRqVP74449FJk8WN+vRo4cJi8kurwdCg6AAVq4Uee6544/14NmvX+Vlrrii8sd46F3cK1aIm7Vr1y7cRYBLEBRhsHjxYtOZ6CWurpN+dpOW3TfpB/5V5/HHKy933XWhLim8tN+5CEEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAABWjSVIRUVFZvLJz883P6Ojo83kBb56lJSUiBf46uGV+ijq5A5erlO0x453wYhygvzC2bS0NElPT68yPzMzU+Li4upWQgBAWBUWFkpycrLk5eVJfHx8/VxRpKamypQpUypdUSQkJEhOTo40adJEvJKwiYmJkp2dLWVlZeJ2XquPok7uQJ0iX3FxcdDLBh0UsbGxZgqkK8wLK83LdfJafRR1cgfqFLnqUgdvNLYBAE4ZggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAABWBAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAAArggIAYEVQAACsCAoAgBVBAQCwIigAAFYEBQDAiqAAAFgRFAAAK4ICAGBFUAAArAgKAIAVQQEAsCIoAABWBAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAFUEBALAiKAAAVgQFAMCKoAAAWBEUAACrxhKkoqIiM/nk5eWZnyUlJeIV0dHRUlhYKMXFxVJWViZu57X6KOrkDtQp8vmO3Y7j1L6wE6QZM2boX2NiYmJiEu9Mu3fvrvX4H6X/nMgVxaFDh6Rz587y6aefSqtWrcQL8vPzJSEhQT777DOJj48Xt/NafRR1cgfqFPm0Vejss8+WgwcPSuvWreun6Sk2NtZMgTQkvLDS/Gl9vFQnr9VHUSd3oE7uaFKrdZmQlAQA4FoEBQDg1ASFNkPNmDGj2uYot/JanbxWH0Wd3IE6eas+QXdmAwAaJpqeAABWBAUAwIqgAABYERQAACuCAgBgRVAAAKwICgCAFUEBABCb/wc8f34TfrWgtgAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n" + ] + } + ], + "source": [ + "V_vi, policy_vi = value_iteration(P, R, gamma)\n", + "\n", + "plot_values(V_vi, title=\"Optimal value function (value iteration)\")\n", + "plot_policy(policy_vi, title=\"Optimal policy (value iteration)\")\n", + "\n", + "print(np.abs(opt_policy - policy_vi))\n" + ] + }, + { + "cell_type": "markdown", + "id": "f4db246d-07c2-4587-b185-7298fe292674", + "metadata": {}, + "source": [ + "## 6. Advanced exercises \n", + "\n" + ] + }, + { + "cell_type": "markdown", + "id": "b0c04200-39f7-41fe-a479-62c87efab8a3", + "metadata": {}, + "source": [ + "**Exercise 16 (Policy Iteration vs Value Iteration)**\n", + "\n", + "In this exercise, we compare the number of iterations required by **policy iteration** and **value iteration** to reach an optimal policy in the Maze game.\n", + "\n", + "1. Modify the definition of `policy_iteration` and `value_iteration` so that they can record:\n", + " - the **number of iterations** until convergence,\n", + " - and optionally the runtime.\n", + "2. Run both algorithms starting from:\n", + " - the same random initialization (a random policy for policy iteration, and $V_0 \\equiv 0$ for value iteration),\n", + " - and repeat the experiment over several random seeds in order to compute the **average number of iterations** and **average runtime**.\n", + "3. Report and interpret the results.\n", + "\n", + "*Question:* What do you observe?\n", + "\n", + "-------------\n", + "\n", + "*Hint.* the word “iteration” means something different for **policy iteration** and **value iteration**:\n", + "\n", + "- Policy iteration: one “iteration” = one outer loop step = policy evaluation + policy improvement.\n", + "- Value iteration: one “iteration” = one Bellman optimality sweep over all states.\n", + "\n", + "-------------\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cce78e3c-ca82-4002-9a8f-08af9457147c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Same policy? True\n", + "Same policy? True\n", + "Same policy? True\n", + "Same policy? True\n", + "Same policy? True\n", + "Same policy? True\n", + "Same policy? True\n", + "Same policy? True\n", + "Same policy? True\n", + "Same policy? True\n", + "Policy Iteration - outer iterations: [1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000]\n", + "Value Iteration - iterations: [100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000]\n", + "Mean PI iterations: 1000.0\n", + "Mean VI iterations: 100000.0\n", + "Mean PI runtime: 0.022669649124145506\n", + "Mean VI runtime: 0.008730292320251465\n" + ] + } + ], + "source": [ + "import time\n", + "\n", + "\n", + "def policy_iteration_count( # noqa: PLR0913\n", + " P: np.ndarray,\n", + " R: np.ndarray,\n", + " gamma: float,\n", + " theta: float = 1e-6,\n", + " max_iter: int = 1_000,\n", + " seed: int = 0,\n", + ") -> tuple[np.ndarray, np.ndarray, int, float]:\n", + " \"\"\"Policy Iteration with iteration count and runtime.\"\"\"\n", + " start_time = time.time()\n", + " rng = np.random.default_rng(seed)\n", + " policy = rng.integers(low=0, high=len(ACTIONS), size=n_states)\n", + " for _it in range(max_iter):\n", + " V = policy_evaluation(policy, P, R, gamma, theta)\n", + "\n", + " new_policy = policy_improvement(V, P, R, gamma)\n", + "\n", + " if np.array_equal(new_policy, policy):\n", + " break\n", + " policy = new_policy\n", + " runtime = time.time() - start_time\n", + " return policy, V, max_iter, runtime\n", + "\n", + "\n", + "def value_iteration_count(\n", + " P: np.ndarray,\n", + " R: np.ndarray,\n", + " gamma: float,\n", + " theta: float = 1e-6,\n", + " max_iter: int = 100_000,\n", + ") -> tuple[np.ndarray, np.ndarray, int, float]:\n", + " \"\"\"Value Iteration with iteration count and runtime.\"\"\"\n", + " start_time = time.time()\n", + " n_states = len(R)\n", + " n_actions = len(P)\n", + " V = np.zeros(n_states)\n", + " for _it in range(max_iter):\n", + " V_new = np.zeros_like(V)\n", + " for s in range(n_states):\n", + " if is_terminal(s):\n", + " V_new[s] = R[s] / (1 - gamma)\n", + " continue\n", + " Q_values = np.zeros(n_actions)\n", + " for a in range(n_actions):\n", + " Q_values[a] = R[s] + gamma * np.sum(P[a, s, :] * V)\n", + " V_new[s] = np.max(Q_values)\n", + " delta = np.max(np.abs(V_new - V))\n", + " if delta < theta:\n", + " break\n", + " V = V_new\n", + " runtime = time.time() - start_time\n", + " policy = policy_improvement(V, P, R, gamma)\n", + "\n", + " return V, policy, max_iter, runtime\n", + "\n", + "\n", + "# Next, run the comparison over several seeds\n", + "\n", + "gamma = 0.9\n", + "theta = 1e-6\n", + "seeds = list(range(10))\n", + "\n", + "pi_iters = []\n", + "vi_iters = []\n", + "pi_times = []\n", + "vi_times = []\n", + "\n", + "for seed in seeds:\n", + " # Policy iteration\n", + " pi_policy, pi_V, n_pi, t_pi = policy_iteration_count( # noqa: N816\n", + " P,\n", + " R,\n", + " gamma,\n", + " theta=theta,\n", + " seed=seed,\n", + " )\n", + " # Value iteration\n", + " vi_V, vi_policy, n_vi, t_vi = value_iteration_count(P, R, gamma, theta=theta) # noqa: N816\n", + "\n", + " pi_iters.append(n_pi)\n", + " vi_iters.append(n_vi)\n", + " pi_times.append(t_pi)\n", + " vi_times.append(t_vi)\n", + "\n", + " # Optional: check both found the same final policy\n", + "\n", + " print(\"Same policy?\", np.array_equal(pi_policy, vi_policy))\n", + "\n", + "print(\"Policy Iteration - outer iterations:\", pi_iters)\n", + "print(\"Value Iteration - iterations:\", vi_iters)\n", + "\n", + "print(\"Mean PI iterations:\", np.mean(pi_iters))\n", + "print(\"Mean VI iterations:\", np.mean(vi_iters))\n", + "print(\"Mean PI runtime:\", np.mean(pi_times))\n", + "print(\"Mean VI runtime:\", np.mean(vi_times))\n" + ] + }, + { + "cell_type": "markdown", + "id": "c4e197e2-b0e4-4d8c-b5ab-8028385c4cd3", + "metadata": {}, + "source": [ + "**Exercise 17.** (Asynchronous Value Iteration)\n", + "\n", + "Implement asynchronous value iteration, where the value function is updated in place :\n", + "$$\n", + "V(s) \\leftarrow \\max_a\\left\\{R(s)+\\gamma \\sum_{s'}P(s'|s,a)V(s')\\right\\}.\n", + "$$\n", + "\n", + "Compare the number of iterations needed for convergence with the synchronous version.\n", + "\n", + "-------------------\n", + "\n", + "Hint. Synchronous value iteration uses a copy `V_new` and updates all states from the old `V`. Asynchronous value iteration updates `V[s]` immediately, so later states in the same sweep can use the newest values.\n", + "\n", + "-------------------\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "id": "0b3469a6", + "metadata": {}, + "outputs": [], + "source": [ + "def asynchronous_value_iteration(\n", + " P: np.ndarray,\n", + " R: np.ndarray,\n", + " gamma: float,\n", + " theta: float = 1e-6,\n", + " max_iter: int = 200_000,\n", + ") -> tuple[np.ndarray, np.ndarray, int]:\n", + " \"\"\"Asynchronous (in-place) value iteration: updates V[s] immediately inside the loop.\"\"\"\n", + " n_states = len(R)\n", + " n_actions = len(P)\n", + " V = np.zeros(n_states)\n", + "\n", + " for _it in range(max_iter):\n", + " delta = 0\n", + " for s in range(n_states):\n", + " if is_terminal(s):\n", + " V[s] = R[s] / (1 - gamma)\n", + " continue\n", + "\n", + " Q_values = np.zeros(n_actions)\n", + " for a in range(n_actions):\n", + " Q_values[a] = R[s] + gamma * np.sum(P[a, s, :] * V)\n", + " v_new = np.max(Q_values)\n", + "\n", + " delta = max(delta, abs(v_new - V[s]))\n", + " V[s] = v_new\n", + "\n", + " if delta < theta:\n", + " break\n", + "\n", + " pi = policy_improvement(V, P, R, gamma)\n", + "\n", + " return (\n", + " V,\n", + " pi,\n", + " _it,\n", + " ) # return final value function, greedy policy from V, and number of iteration performed\n" + ] + }, + { + "cell_type": "markdown", + "id": "48ae870c-1f18-4f83-8b6d-1ae393d08de3", + "metadata": {}, + "source": [ + "**Exercise 18.** (Bellman Optimality Operator)\n", + "\n", + "Show numerically that the Bellman optimality operator $\\mathcal{T}^*$ satisfies the contraction property, which means, for arbitrary value functions $V$ and $W$, we have \n", + "$$\n", + "\\big\\Vert \\mathcal{T}^* V - \\mathcal{T}^* W \\big\\Vert_{\\infty}\\leq \\gamma \\Vert V-W\\Vert_{\\infty}\n", + "$$\n", + "\n", + "---------------\n", + "\n", + "*Hint.* Generate random value functions $V$ and $W$, apply one Bellman optimality update to each, and compare both sides of the inequality.\n", + "\n", + "---------------" + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "id": "b64a1e78", + "metadata": {}, + "outputs": [], + "source": [ + "def T_opt(V: np.ndarray, P: np.ndarray, R: np.ndarray, gamma: float) -> np.ndarray:\n", + " \"\"\"Bellman optimality operator in the maze: (T* V)(s) = max_a [ R[s] + gamma * sum_{s'} P[a,s,s'] V[s'] ].\"\"\"\n", + " n_states = len(R)\n", + " n_actions = len(P)\n", + " V_new = np.zeros_like(V)\n", + "\n", + " for s in range(n_states):\n", + " if is_terminal(s):\n", + " V_new[s] = R[s] / (1 - gamma)\n", + " continue\n", + "\n", + " Q_values = np.zeros(n_actions)\n", + " for a in range(n_actions):\n", + " Q_values[a] = R[s] + gamma * np.sum(P[a, s, :] * V)\n", + " V_new[s] = np.max(Q_values)\n", + "\n", + " return V_new\n" + ] + }, + { + "cell_type": "markdown", + "id": "d58d20fd-509a-41e4-bde1-eca6c9da6ddc", + "metadata": {}, + "source": [ + "**Exercise 19** (Effect of the Discount Factor)\n", + "\n", + "Recall that the discount factor controls how future rewards are weighted relative to immediate rewards. Run value iteration for different values of the discount factor $\\gamma\\in\\{0.2, 0.5, 0.9, 0.99\\}$. \n", + "\n", + "For each value of $\\gamma$: \n", + "\n", + "1. Compute the optimal value function $V^*$.\n", + "2. Compute the corresponding optimal policy.\n", + "3. Plot the value function and visualize the policy on the maze.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 42, + "id": "47219cdc-b99b-4b73-a3d1-c133afb0e215", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Max difference between V_opt and V_vi: 18.749999999999982\n", + "Asynchronous VI iterations for gamma = 0.2 : 9\n", + "Max difference between V_async and V_vi: 18.749999999999982\n", + "Same policy between async VI and VI? True\n", + "Max difference between V_opt and V_vi: 17.999999999999982\n", + "Asynchronous VI iterations for gamma = 0.5 : 14\n", + "Max difference between V_async and V_vi: 17.999999999999982\n", + "Same policy between async VI and VI? True\n", + "Max difference between V_opt and V_vi: 9.99999999999998\n", + "Asynchronous VI iterations for gamma = 0.9 : 21\n", + "Max difference between V_async and V_vi: 9.99999999999998\n", + "Same policy between async VI and VI? True\n", + "Max difference between V_opt and V_vi: 79.99999999999993\n", + "Asynchronous VI iterations for gamma = 0.99 : 25\n", + "Max difference between V_async and V_vi: 79.99999999999993\n", + "Same policy between async VI and VI? True\n" + ] + } + ], + "source": [ + "for gamma in [0.2, 0.5, 0.9, 0.99]:\n", + " V_opt = T_opt(V_vi, P, R, gamma)\n", + " print(\"Max difference between V_opt and V_vi:\", np.max(np.abs(V_opt - V_vi)))\n", + "\n", + " V_async, pi_async, n_async = asynchronous_value_iteration(P, R, gamma, theta=theta)\n", + " print(\"Asynchronous VI iterations for gamma =\", gamma, \":\", n_async)\n", + "\n", + " print(\"Max difference between V_async and V_vi:\", np.max(np.abs(V_async - V_vi)))\n", + " print(\"Same policy between async VI and VI?\", np.array_equal(pi_async, policy_vi))" + ] + }, + { + "cell_type": "markdown", + "id": "31083905-cc29-431e-9f87-6595e187e5d0", + "metadata": {}, + "source": [ + "**Exercise 20** (What we will learn in the next weeks)\n", + "\n", + "Assume now that the transition matrix $P$ is unknown.\n", + "\n", + "1. Which parts of policy iteration and value iteration can no longer be applied?\n", + "2. Which quantities would need to be learned from data?\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "id": "e8a7738a-584d-43ae-ba2f-2598608b38fa", + "metadata": {}, + "source": [ + "**Exercise 21.** Try different configurations of the maze and compute an optimal policy." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "453188a8-bc26-463b-9784-be9c68328495", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "studies", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.9" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}