{ "cells": [ { "cell_type": "markdown", "id": "44b75d44", "metadata": {}, "source": [ "# Lab 2 - Maze Game as a Markov Decision Process Part 1\n", "\n", "## **1. Objectives**\n", "\n", "In this lab, we will:\n", "\n", "- Model a simple **maze game** as a **Markov Decision Process (MDP)** by defining:\n", " - **States**\n", " - **Actions**\n", " - **Transition probabilities**\n", " - **Rewards**\n", "\n", "- Implement **policy evaluation** to compute the value function of a given policy.\n", "\n", "This week, we **do not** improve the policy and search for an optimal one yet. \n", "We will continue working on the Maze Game **next week**, where we will use these components to compute an **optimal policy**.\n", "\n", "We consider a **discounted MDP** with discount factor $\\gamma \\in (0,1)$.\n", "\n", "\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "id": "100d1e0d", "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "\n", "np.set_printoptions(\n", " precision=3,\n", " suppress=True,\n", ")\n", "# (not mandatory) This line is for limiting floats to 3 decimal places,\n", "# avoiding scientific notation (like 1.23e-04) for small numbers.\n", "\n", "# For reproducibility\n", "rng = np.random.default_rng(seed=42) # This line creates a random number generator." ] }, { "cell_type": "markdown", "id": "1018deab", "metadata": {}, "source": [ "## 2. Maze definition and MDP formulation\n", "\n", "We consider a small 2D maze on a grid. The agent is a **robot** that moves on the grid.\n", "\n", "- `S` : start state\n", "- `G` : goal state, with positive reward\n", "- `#` : wall (not accessible)\n", "- `.` : empty cell\n", "- `X` : \"trap\" (negative reward)\n", "\n", "At each step, the robot can choose among 4 actions:\n", "\n", "$$\n", "\\mathcal{A} = \\{\\text{Up} \\uparrow, \\quad \\text{Right} \\rightarrow, \\quad \\text{Down} \\downarrow, \\quad \\text{Left}\\leftarrow\\}.\n", "$$\n", "\n", "The movement is deterministic, but here we set a small probability of “error” to make the example more realistic.\n", "- With probability $1 - p_{\\text{error}}$, it moves in the chosen direction.\n", "- With probability $p_{\\text{error}}$, it moves in a random *other* direction.\n", "- If the movement would hit a wall or go outside the grid, the agent stays in place.\n", "\n", "We will represent the MDP with:\n", "\n", "- A list of **states** $\\mathcal{S} = \\{0, \\dots, n_{S - 1}\\}$, **each corresponding to a grid cell.**\n", "- For each action $a$, a transition matrix $P[a]$ of size $(n_S, n_S)$, where\n", " $$\n", " P[a][s, s'] = \\mathbb{P}(S_{t+1} = s' \\mid S_t = s, A_t = a).\n", " $$\n", "- A reward vector $R$ of length $n_S$, where $R[s]$ is the immediate reward obtained when **leaving** state $s$.\n", "\n", "We will use a discount factor $\\gamma = 0.95$.\n" ] }, { "cell_type": "markdown", "id": "ca4fa301-c14f-44ec-b04f-b01ca42d979a", "metadata": {}, "source": [ "### 2.1 Define the maze \n", "\n", "Let us now define the maze as follows." ] }, { "cell_type": "code", "execution_count": 74, "id": "f91cda05", "metadata": {}, "outputs": [], "source": [ "maze_str = [\n", " \"#######\",\n", " \"S...#.#\",\n", " \"#.#...#\",\n", " \"#.#..##\",\n", " \"#..#..G\",\n", " \"#..X..#\",\n", " \"#######\",\n", "]" ] }, { "cell_type": "markdown", "id": "99820cf4-292d-49ba-b662-f9f05f901f62", "metadata": {}, "source": [ "**Exercise 1.** Compute the dimensions of the maze (complete the “TO DO” parts):\n", "- How many rows does the maze have?\n", "- How many columns does the maze have?" ] }, { "cell_type": "code", "execution_count": 75, "id": "564cb757-eefe-4be6-9b6f-bb77ace42a97", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "7\n", "7\n" ] } ], "source": [ "n_rows = len(maze_str)\n", "print(n_rows)\n", "n_cols = len(maze_str[0])\n", "print(n_cols)" ] }, { "cell_type": "code", "execution_count": 76, "id": "26c821d3-2362-4b60-8c77-3d09296d130d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Maze:\n", "#######\n", "S...#.#\n", "#.#...#\n", "#.#..##\n", "#..#..G\n", "#..X..#\n", "#######\n" ] } ], "source": [ "print(\"Maze:\")\n", "for row in maze_str:\n", " print(row)" ] }, { "cell_type": "markdown", "id": "adc49d58-2730-41d8-96fb-ca7c9cb4fcdf", "metadata": {}, "source": [ "### 2.2 Map each walkable cell (not a wall '#') to a state index\n", "\n", "Now we convert the maze grid into state indices for the MDP.\n", "\n", "\n", "The cells where the robot is allowed to stand are \n", "\n", "- . : empty space\n", "\n", "- S : start\n", "\n", "- G : goal\n", "\n", "- X : trap\n", "\n", "Everything else (i.e., #) is a wall and cannot be a state in the MDP.\n" ] }, { "cell_type": "code", "execution_count": 77, "id": "7116044b-c134-43de-9f30-01ab62325300", "metadata": {}, "outputs": [], "source": [ "FREE = {\n", " \".\",\n", " \"S\",\n", " \"G\",\n", " \"X\",\n", "} # The vector Free represents cells that the agent is allowed to move into." ] }, { "cell_type": "markdown", "id": "1c9ad05e-9c6c-4e00-918c-44b858f45298", "metadata": {}, "source": [ "**Dictionaries to convert between grid and state index**\n", "\n", "We now want to identify all **valid states** of the maze (all non-wall cells). \n", "To do this, we need two mappings:\n", "\n", "1. `state_to_pos[s] = (i, j)`: Given a state index $s$, return its grid coordinates (row, column).\n", "2. `pos_to_state[(i, j)] = s`: Given coordinates (i, j), return the corresponding state index $s$.\n", "\n", "These two dictionaries allow easy conversion between **MDP state indices** and the **physical maze positions**. " ] }, { "cell_type": "code", "execution_count": null, "id": "a1258de4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of states (non-wall cells): 22\n", "Start state: 0 at (1, 0)\n", "Goal states: [16] at (4, 6)\n", "Trap states: [19] at (5, 3)\n" ] } ], "source": [ "state_to_pos = {} # s -> (i,j)\n", "pos_to_state = {} # (i,j) -> s\n", "\n", "start_state = None # will store the state index of start state\n", "goal_states = [] # will store the state index of goal state\n", "trap_states = [] # will store the state index of trap state\n", "\n", "s = 0\n", "for i in range(n_rows): # i = row index\n", " for j in range(n_cols): # j = column index\n", " cell = maze_str[i][j] # cell = the character at that position (S, ., #, etc.)\n", "\n", " if cell in FREE:\n", " # FREE contains: free cells \".\", start cell \"S\", goal cell \"G\" and trap cell \"X\"\n", " # Walls # are ignored, they are not MDP states.\n", " state_to_pos[s] = (i, j)\n", " pos_to_state[(i, j)] = s\n", "\n", " if cell == \"S\":\n", " start_state = s\n", " elif cell == \"G\":\n", " goal_states.append(s)\n", " elif cell == \"X\":\n", " trap_states.append(s)\n", "\n", " s += 1\n", "\n", "n_states = s\n", "\n", "print(\"Number of states (non-wall cells):\", n_states)\n", "print(\"Start state:\", start_state, \"at\", state_to_pos[start_state])\n", "print(\"Goal states:\", goal_states, \"at\", state_to_pos[goal_states[0]])\n", "print(\"Trap states:\", trap_states, \"at\", state_to_pos[trap_states[0]])" ] }, { "cell_type": "markdown", "id": "721b968c-a355-46eb-aae4-5950441ba604", "metadata": {}, "source": [ "*Hint.* If you don’t know what a dictionary is in Python, try the following code to help you understand." ] }, { "cell_type": "code", "execution_count": 79, "id": "68744dd6-7278-4c20-8b82-34212685352f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "value2\n" ] } ], "source": [ "my_dict = {\"key1\": \"value1\", \"key2\": \"value2\"}\n", "print(my_dict[\"key2\"])" ] }, { "cell_type": "markdown", "id": "0c76f4e1-b0ba-49c5-b9d5-cfb523024ba9", "metadata": {}, "source": [ "**Exercise 2.** Read the program above and answer the following questions:\n", "1. What is the purpose of state_to_pos and pos_to_state?\n", "2. Why do we only assign states to cells in FREE?\n", "3. What would happen if the maze had multiple goal cells?\n", "4. What is the total number of states (n_states) in this maze? Does this match the number of non-wall cells you can count visually?" ] }, { "cell_type": "markdown", "id": "4c26a18f-2d03-401c-8eae-f9a17ac55f6d", "metadata": {}, "source": [ "1. What is the purpose of `state_to_pos` and `pos_to_state`? These dictionaries establish a bijective mapping between the mathematical representation of the state and its spatial representation:\n", "\n", " `state_to_pos`: Maps the scalar state index `s` (an integer used for matrix/vector operations in RL algorithms like Q-learning) to the grid coordinates (i,j).\n", "\n", " `pos_to_state`: Maps the grid coordinates (`i,j`) (used to calculate movement and dynamics within the 2D grid) back to the unique state index s.\n", "\n", "2. Why do we only assign states to cells in FREE? In a Markov Decision Process (MDP), walls (#) are obstructions, not valid states.\n", "\n", " The agent can never \"be\" in a wall, so assigning a state index to a wall would needlessly increase the dimensionality of the state space (∣S∣).\n", "\n", " Excluding walls ensures the transition matrices and value vectors remain compact and contain only reachable positions.\n", "\n", "3. What would happen if the maze had multiple goal cells?\n", "\n", " In the code: The logic is robust. Since goal_states is initialized as a list (`[]`), the code would simply append the state index `s` of every `G` cell found during the iteration. The list would contain multiple integers representing all terminal states.\n", "\n", " Caveat: While the logic holds, the final print statement in the provided script (`state_to_pos[goal_states[0]]`) would only display the coordinates of the first goal found, ignoring the others in the console output.\n", "\n", "4. What is the total number of states (`n_states`) in this maze? Does this match the number of non-wall cells you can count visually?\n", "\n", " `n_states` represents the total count of walkable cells (Start, Goal, Trap, and empty space).\n", "\n", " Yes, this value matches exactly the number of non-wall cells visible in the maze, as the counter s is incremented precisely when a cell is found in the FREE set." ] }, { "cell_type": "markdown", "id": "6d0fa298-7b7c-44fc-bbed-15ea002037c2", "metadata": {}, "source": [ "-----\n", "\n", "The following function `plot_maze_with_states` creates a figure showing:\n", "- the maze walls and free cells\n", "- the state index for each non-wall cell\n", "- special labels and colors for S (start state), G (goal state), and X (trap state). " ] }, { "cell_type": "code", "execution_count": null, "id": "fc61ceef-217c-47f4-8eba-0353369210db", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGbCAYAAAAr/4yjAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAK99JREFUeJzt3Qd4VFX6x/E3CS1CEiSAofcS+spKEZCmgiBZKQqKiIuCrIDArmBBKSu661IUQYoNpFhoYhCpSlEEFekIiHThTwslEaRm/s97xjlMMAlRk9wp38/zXObOzZA5d+7k/O4pdybE5XK5BAAAEQl1ugAAAN9BKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUowHEhISEydOjQDD+2d+/eEoyaNGlilqz8nfv27TOv8ZQpUzL1eeA/CAUfoX+E+seoy5dffvmbn+unkZQoUcL8/O6775ZA9tVXX5mQOH36tF8+9/jx46lU4bdyOF0ApJQnTx557733pGHDhim2r1y5Un766SfJnTu3BJpffvlFcuTIkaJiHjZsmDz88MOSP3/+bC1LZjy3hkLBggXN78hMS5YskaxWqlQpczxy5syZ5c8F30RLwce0atVKZs2aJZcvX06xXYOidu3aEhMTI4EYhN6hgNTlypXLLFlJW6J6PMLCwrL0eeC7CAUfc//990tCQoIsXbrUbrt48aLMnj1bHnjggVT/z8iRI+XWW2+V6OhoCQ8PN+Ghj/emZ62e7qlrF+/+/AsXLsiQIUOkfPnyplWiXVYDBw4029Pz2muvmYrEu9tl1KhR5vf/85//tNuuXLkiERER8tRTT9lt3mXQ2wEDBpj1MmXK2DJqX7e3efPmSbVq1UwZq1atKosWLZKMGDt2rHn8DTfcIDfeeKP89a9/NYGbkeeePHmyNGvWTAoXLmyet0qVKjJhwoQUv7906dKybds207Lz/H/vPnt9ffr162deV/0d+jq//PLLkpyc/Lv7/1esWGF+/8yZM+XFF1+U4sWLmwq9efPm8uOPP/7m/7/xxhtSrlw58x6pU6eOfPHFF795TFpjCjt27JD77rtPChUqZP5/pUqVZNCgQSkec+jQIenWrZvcdNNN9ri88847v+sYwHmcnvkYrVTq168v77//vtx1111m28KFC+XMmTPSqVMnU/lea8yYMRIXFyedO3c2AfLBBx/IvffeK5988om0bt3aPOaxxx6T22+/PcX/04p0xowZppJTWjHp79ExjR49ekhsbKxs2bJFXnnlFfnhhx9MRZyWRo0amf+v/9cz5qGVTmhoaIrKZ8OGDfLzzz/LbbfdlurvadeunXku3X99Xu2GUVoZeehzzJ07Vx5//HETMPqatG/fXg4cOGCCMS1vvvmmPPHEE9KhQwfp27evnD9/XjZv3ixff/21CdzrPbcGgFZm+hppy2b+/PmmDLrfvXr1Mo959dVXpU+fPpIvXz5baWolqc6dOyeNGzc2lacej5IlS5ruqmeeeUb+7//+z/zfP+K///2veZ2ffPJJ8z753//+Z94Lul8eb7/9tnlOPXnQUNqzZ4/ZjwIFCpiASo++Rnp8tUtJ3xf6Ht29e7fZfw0jdfToUalXr56dCKCvmb5vH3nkEUlMTDTPmZFjAB+g36cA502ePFm/18L17bffusaNG+eKiIhwnTt3zvzs3nvvdTVt2tSslypVytW6desU/9fzOI+LFy+6qlWr5mrWrFmaz7dr1y5XVFSU64477nBdvnzZbJs2bZorNDTU9cUXX6R47MSJE03ZVq9enebvu3LliisyMtI1cOBAcz85OdkVHR1tyh4WFuZKSkoy20ePHm2e49SpU/b/6u8eMmSIvT9ixAizbe/evb95Ht2eK1cu148//mi3bdq0yWwfO3asKz1/+9vfXFWrVk33Mek997Wvs2rRooWrbNmyKbbpczRu3Pg3j33hhRdcefPmdf3www8ptj/99NPmNTpw4EC6ZdPf6f17ly9fbsoaGxvrunDhgt0+ZswYs33Lli32/VC4cGFXrVq1UjzujTfeMI/z/p2637pN348et912m3k/7t+/P0V59Bh7PPLII64iRYq4Tpw4keIxnTp1Mu8zz2uXkWMAZ9F95IO0ma6DfXqmn5SUZG7TO4vS5rzHqVOnzNmintmtX78+1cefPXtW2rZta5ruelbs6T/WsQxtHVSuXFlOnDhhF+0yUcuXL0+zDHqmqmehq1atMve3b99uusGefvppM3NqzZo1Zru2GrTb588MIGuLR7tBPGrUqCGRkZHm7Dc9+pw6WP/tt9/+oef1fp31NdbXRs/89Xn1/vXo66vHRV9379dX90e71Tyv3e/197//PcVYgz6H8rwe69atk2PHjknPnj1TPE67FKOiotL93cePHzfl0m4hbdl401aB0uM7Z84cadOmjVn33rcWLVqY18bzXvyzxwBZj+4jH6RNb60otJ9Vuxy0wtDmdlo0NIYPHy4bN25M0ffv+aO9Vvfu3U3zX7suvLtbdu3aZSpz764ab1qxpEcrI+2X10DTyr9IkSJy8803S82aNc39O+64w3T9aOj9GddWTkorWg3E9Og4xrJly0x/uvbl33nnnSZsGzRokKHnXb16tRlv0YDT4+JNK77rVbD6+mpXyR99fTP6euhroTyvx/79+81thQoVUjxOu4PKli2b7u/2BIsGeXrBoWMlOmahS3r79mePAbIeoeCj9A9FK+8jR46YsYW0zqy1stW+Ye2j16mQWhHrH7sOiqY2eKfjD9o6mD59utSqVSvFz7RvvHr16jJ69OhUn+t6fc86jfbSpUum0tRyec5Y9Vbv62ClViCe7X9UWjNjrvfNstoK2rlzpwlRHU/Rs1t9zQYPHmymoaZHQ1QHcLUVpa+PvhZ61v3pp5+a8YeMDBTrYzQYdeA+NRUrVpTsfD0yi2ffH3zwQenatWuqj9HW3J89BsgehIKP0u4dHRhcu3atfPjhh2k+Tv+odMbJ4sWLU1zDoKFwLa2YdTBSB/10IPJa2iWzadMmU/ml1cpIj579aUWpz6OLZyaPBpYOMH722Wf2fnr+yHNnVN68eaVjx45m0UF5HVzWwVId7NXXMa3n1kFVbYXFx8enODNPrUstrd+hr68Osl874J8d1x54WiqerkClAb53717TkkuLpyWxdevWNB+jLR8d8NcWbUb27XrHAM5iTMFH6ewVne2i3THaV5veWaJWQvoH6T2t8NqZQjq7Rbtt9Gx+xIgRqf4u/bnOjNEK/FraJaRjEenRP+hbbrnFtER0JpB3S0H/v84S0opRWzPXqzRUZl/RrGMc3jTAdFqpnlFrBZnec3vOxr3PvrXLKLXw1d+RWtn19dVWlAb4tfTx116bkll0yqdW3BMnTjSVsIdOO73ea6z/T0Ncp5bqMfXmeS30tdHZX3qCklp4aOvw9xwDOIuWgg9LqynuTaecandGy5YtTZeT9t2+/vrrpr9W+689dBqg/nFq14VOWb22aa9Lly5dzJx3HZDUM2Dt59Ww0W4f3a6VmVYw6dEA0CmS2r+uXVFKp7zqvHbtNsjIVb56nYXSKZ06DVe7wzQYPRX2H6X913rxn+6XThPV8ZNx48aZ11DPdNN7bv2/WoHpurbg9Ixfw1P3TQP32vJroOs4jx4HfYyeoWvLSVsaOmVXXwd9nAatTvvV60o0zD3TYDOT7oOWRcut5dAzdG0haKBdb0xBaZjryYSOD+mUVL2GQ8u6YMECM46l9Jjre6Zu3bqm21Mr+pMnT5oBZh1D0PWMHgM4zOHZT0hlSmp6UpuS+vbbb7sqVKjgyp07t6ty5crmd+kUT+/Dq9MO9X5qi/d0UJ2++PLLL5tpg/r7brzxRlft2rVdw4YNc505c+a6+7FgwQLzO++6664U2x999FGzXct6rWvL4Jm+WaxYMTN91XuKqK736tUr1dela9eu6ZZt0qRJZnqlTpXVfStXrpxrwIABv9mvtJ47Pj7eVaNGDVeePHlcpUuXNq/TO++885sprEeOHDHHSKdxXjvlU6fmPvPMM67y5cubqbUFCxZ03Xrrra6RI0ea1/6PTEmdNWtWiselNq1UjR8/3lWmTBmz73/9619dq1at+s3vTOv/bt261dW2bVtX/vz5zf5XqlTJ9fzzz6d4zNGjR82xKVGihCtnzpyumJgYV/Pmzc3U1997DOCcEP3H6WACAPgGxhQAABahAACwCAUAgEUoAAAsQgEA8PuuU9DL2A8fPmzmEWfl1aYAgKyhE031AzaLFi1qPsDyT4WCBsL1PvcGAOD7Dh48aL6Q6U+FgudKQ/1cE742EQD8j36Mil5dfr0rxzNUw3u6jDQQ+EJvAPBf1xsCYKAZAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAK4f8Dh9++KFERkb+nv8C4BpxcXESaOLj4yXQxAXgccoIWgoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAAH/sA/GywsUrF2XkVyNl+ubpsv/MfgkLCZPCeQtL9Zuqy9DGQ6VmTE2niwgAQcPxlsKAJQNk0OeDZPuJ7VIsopiUzl9ajp09JvN2zJNdJ3c5XTwACCqOtxQ+3PahuR1822AZ1nSYWXe5XPLVwa9MiwEAEEShkOxKNrdL9iyRW4rdIrcUvUVuyneTNCjZwOmiAUDQcbz76PFbHje3a39aK23ebyMxo2Kk8rjK8sLKF+T85fNOFw8AgorjoTC0yVCZe99caVOxjUTmdn+r286EnTJ4xWDp+UlPp4sHAEHF8VBQbWPbSvz98XLqqVPyzaPfSPXC1c12HWwGAARRKDz3+XOy8chGd2FCQs24QsXoiuZ+VJ4oh0sHAMHF8YHmt9a/JS9+8aIUvKGglIwqaaaj/pT4k/nZA9UecLp4ABBUHG8pDG82XP5W6W8SkStCdpzYYUKhUnQlGdJ4iLzQ7AXxV6tWrZJWrVpJoUKFJCQkxCwTJ04UfzVq1Chp0qSJFClSRHLnzi2lSpWSrl27yp49e8Sfvfrqq1KzZk3Jnz+/2a/ixYvLvffeK5s3b3a6aEjFfffdZ/+eOnXq5HRxApLjLYVHb37ULIFm/fr1snTpUilbtqycOHFC/N3YsWPlwIEDUqlSJQkPD5e9e/fK1KlTZcmSJbJz506JjHRPEvA3K1eulOPHj5vjdP78ebMvs2fPls8//9zsb968eZ0uIn41efJkmTVrltPFCHiOtxQCVZcuXSQxMVEWL14sgaB79+6yb98+2b59u2kd9OvXz2w/cuSIfPbZZ+Kv3n//fTl8+LAJ8e+//16effZZs/3kyZOyY8cOp4uHX+3evVueeOIJqV+/vmnNIesQClkkOjranFEHikGDBknJkiXt/UaNGtl17XbxV3ny5JGPPvpI6tWrJ1WqVJGXXnrJbNduv4oV3RMe4KzLly9L586dJTQ0VGbMmCFhYWFOFymgOd59BP9z5coVeeONN8y6drs0b95c/NnRo0fl66+/tvfLlCkj8+fPl4iICEfLBbdhw4aZ4zN9+nRzbJC1aCngdzl79qy0bdvWdIvFxMSYytOfWwqqZ8+ekpycLPv375eOHTua8RK9TUpKcrpoQW/dunXyn//8Rx588EHTWkDWIxSQYTp+0LhxYxME2rWyevVq0+USCHQ2i3aPecYUtm3bZsYb4KytW7ealqkO/ufLl88sOgFAzZkzx9w/c+aM08UMKIQCMkQrSe13/+6778x4wpo1a0zXkT9LSEiQadOmycWLF+22Tz/9NEWrCL5BZ4bp8dBFP0XZM9bgfR8BFgofbP1Abp50s4S/GC4FXi4gHWZ2kN0nd4u/mjt3rpQvX97M7fcYPHiw2eaPzeB27dqZ7hWl3Sp6DYaGhC5vvfWW+CPdj4ceeshco1C9enXTUnjmmWfMz3Q8QfcZznr44YdNpe+96DUySrv49L4ePwTYQPPb69+WR+e7r1Uok7+MJPySIHO2z5EvDnwhm3pukph8MeJvdDqqTqPzpvPhdfHHKXUXLlyw6xs3uj+WxKNly5bij7Qy0QugvvnmG3OsLl26JCVKlDBdZNqN5Kl8gGAS4spA20sruKioKNN3l9kXKenXcRYbXUxOnDsh7WPby+z7ZsvhpMPm47OTLiZJnzp95LW7XsvU5wScFBcXJ4EmPj5eAk1cgB0nPelZtGjRdetxx7uPvj30rQkEpaGgikYUlXrF65n1RT8ucrR8ABBMHA+Fg4kH7br312/qt6+pA2fcMw0AAEEQCmlhRgEABGEolIgsYdf1E1KvXdeP0wYABEko6JfqRIdHm3WdcaR0oFm/s1m1LO+fM1sAwB85Hgq5wnLJS81fsqFQdkxZiX091sw80i/eebrh004XEQCChuOhoHrU7iHT206XWjG1TCshREKkXWw7+arbV2YmEgAgiC5eU51rdDYLACDIWwoAAN9AKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAIA/9oF4HTt2lJw5c0qg4MvGfV8gHqNA3CcEDloKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahkAWGDh0qISEhqS6XL192uni4xvHjx6VPnz5SqlQpyZUrlxQsWFCaN28ue/bsEX+zb9++NN97uuh709+cPXtWBg4cKBUqVJAbbrhBoqKipEaNGjJixAhxuVxOFy/g5HC6AIFMK5dy5cql2KZ/mPAdJ06ckLp168revXtNIFSsWNFUNGvWrJHDhw9L2bJlxZ/kzp3b7I+306dPy86dO816kSJFxN/06tVL3n33XbNetWpVOXPmjGzZssUERZ48eUygI/MQClmodevWMmXKFKeLgXQ899xzJhC0slm6dKmtNC9evOiXZ6Fa/rVr16bY1rt3bxMKN954o3Tu3Fn8zZdffmluW7ZsKQsXLpRffvlFChQoIOfPn5f9+/c7XbyAQ/dRFpozZ46Eh4ebP9S7775bNmzY4HSR4EUr/ZkzZ5r1EiVKyB133CF58+aVmjVrmmOnZ93+LiEhQSZPnmzW//GPf0i+fPnE3zRq1MjcLlq0SKpVq2ZacxoIuv1f//qX08ULOIRCFgkLC5OYmBgpXbq0HDlyRBYsWCD169cnGHxsLOHUqVO2wtFuFj2b3rx5szzwwAMye/Zs8Xfjx4+Xc+fOmYDz126WiRMnykMPPWTWt23bJj/99JPp6tNxBT1eyFyEQhbQCuXYsWOya9cu2b59u6lw1IULF+T11193unj4lfegf2xsrBlY1kXX1bhx48Sfeb/fHnzwQXOS4o9eeeUVmTZtmjRo0MD8XWkwREREmH17+umnnS5ewCEUsoA2b7XP06NFixYSHR1t1g8cOOBgyeCtUKFC5oxTaZeRruui656ZPP5s6tSpcvToUTO5wV+7WbSV8/zzz5uuvvbt25tjVqVKFRMQatmyZU4XMeAQClng5ZdfTlH56wCm9u0q7U6Cb8iZM6fcdtttZl27jC5dumQWXVc6BdJfaSU6atQoO+HB0/rxx1DwtOi+++47c6vjCdpaUDoGhMxFKGSBCRMmmMpf573rWY22FDxv4H79+jldPHgZPny4aR18//33UqZMGbPouo4JPfvss+Kv5s+fb6ehDhgwQPx5WrcnuGfMmGGCWv+2du/ebbZ17drV4RIGHkIhC2hlohc/6Vmn9lFrOOhUQD3T0ZCA79A5/Z9//rk0adLEDDrrWejtt98uq1evlqZNm4q/GjlypLmtU6eOrVT91bx588w1Cdotq9eO6HRhPW7Tp0+Xxx9/3OniBZwQVwYmYycmJpqrCHWesDa5A0V8fLzTRch0cXFxEkgC8RjBP8QF2N+SnqTqpBe9+C8yMjLNx9FSAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALByXF0FgD8m0L7kPpjRUgAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoZJGff/5ZhgwZIpUrV5bw8HApWrSo/OMf/5BTp045XbSgtWrVKmnVqpUUKlRIQkJCzDJx4sQUj7l06ZIMGzZMypYtK7ly5ZLixYtL//79zfH0x/2ZNGmSNGzYUPLmzWsfs2PHDsfKDN9HKGSRNm3ayL///W/58ccfpWLFiqZS0T/YO++8Uy5fvux08YLS+vXrZenSpVKgQIE0H9OtWzcZOnSo7N+/3wTDsWPH5NVXX5W7775bkpOTxd/2Z+HChbJhwwYTHEBGEApZ4Pvvv5cVK1aY9TFjxsimTZvku+++M/fXrVsnM2fOdLiEwalLly6SmJgoixcvTrOSnT59uj1uekY9Z84cc3/lypUyb9488af9UePHjzeP0aADMoJQyALeZ5ShoaEpbtWyZcscKVewi46ONl156Z1Ve7Rv397ctm7dWvLkyWPWFy1aJP60P0q7LcPCwrKtTPB/hEIWiI2NlWrVqpn1Pn36SK1ateTmm2+2Pz906JCDpUNaDh48aNcLFy5sw7xgwYJm/cCBA46VDcguhEIW0DMzPevs3LmzqVD27NkjjRo1knLlypmf58yZ0+ki4ndwuVxOFwHINjmy76mCi85a8fRPq/Pnz0tMTIxZr1SpkoMlQ1pKlChh13WAuUiRIqYrMCEhwWwrWbKkg6UDsgcthSyig5ZJSUlm/cqVKzJgwAA5c+aMud+xY0eHS4fUtGzZ0q57BpgXLFhgAv3anwOBilDIIu+8847pl65evbppIYwbN85s79evn9SpU8fp4gWluXPnSvny5aVJkyZ22+DBg8027eqrXbu23H///WZ73759zdiQZ8BZu//uuece8af9UU899ZS5r7ceLVq0MNtee+01R8oN30b3URbRin/58uVmPEH7pLXC0YvXHnnkEaeLFrR0aubu3btTbDt+/LhZtLtPvfvuu1KhQgWZOnWqeazO7+/QoYMMHz48xQwyf9mfo0eP/uYxngHzkydPZmNp4S9CXBkYRdM3X1RUlGk+B9IgaXx8vASauLg4CSSBeIwCUaC97wKRXq2v06q1GzsyMjLNx/nWqQ8AwFGEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsHJcXUUgCLQvug/EL4QPtGMUqPsUaBITEyUqKuq6j6OlAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAIBsdv68yCuviNx6q0j+/CK5c4uULCly++0io0c7WzY+OhsAslFCgkjz5iKbNrnv33CDSMWKIklJIitXinz2mcg//+lc+WgpAEA26t37aiD07esOiS1bRPbtEzlxQmTyZGfLR0sBALLJ6dMis2a512vWdHcVhXqdmut34Dz8sDiKlgIAZJMffhC5csW93qjR1UC45x6RkJCry5QpzpWRUAAAB4R61b6VKrlbDr6AUACAbFKpkkhYmHv9q6+ubn/5ZZEPPhCfQCgAQDaJihK57z73+rp1IkOGXO1O8hWEQiZYtWqVtGrVSgoVKiQhISFmmThxov15UlKS9OvXT2rXri0FCxaU8PBwqVixojz//PPmZ/64T6pbt25SoUIFyZcvn+TNm1fKlSsnTzzxhJw8edKxcgeTjBwjD32f6fG53uP8YZ+aNGlif+a9NGzYUPzB2LEiNWq41//9b5ECBUT+8hfdL/EJhEImWL9+vSxdulQK6NFNRUJCgowZM0a2bdsmxYsXN5Xorl27ZPjw4dKxY0fxx31SH3/8sVy5ckUqV65swm7Pnj0yduxYeeCBB7K1rMEqI8fIo3fv3ub4BNI+lS1bVurWrWuXqlWrij+IjhZZu9bdZVS7tkhyssiOHSLh4SItWohoBurAs1MIhUzQpUsXSUxMlMWLF6f68zx58siIESPk+PHjsnHjRjl48KDUq1fP/GzhwoVy6tQp8bd9UocOHTIVzbp162T//v32TG316tXZWNLglZFjpGbOnClTp06V+zz9FgGwT0pb2mvXrrXLpEmTxF+Eh4sMHOjuQtLOgl9+Edm7V2TRIpHHHnNf5ewUQiETREdHmy6htMTExMiTTz4pERERNiRuueUWsx4aGio5cuTwu33y7If+YepZWunSpeXLL7802/2lGe/vMnKM9ATkscceM12X2jINhH3y6N+/v+TOndu0GHr06CFHjx7N8vIFA0LBAceOHZM5c+aY9U6dOtmw8EfaDfbNN9+YloK6/fbbzZkpnJecnGzOvC9duiTvvfee5MyZUwKFBkexYsXM2MPevXvlzTfflPr168vZs2edLprfIxSy2e7du82Z9OHDh6VBgwY+O+CXUR988IFcvHhRNmzYINWqVZNly5ZJr169nC4WRMw41sqVK82tTmwIFK+88orpct26datpCT3zzDNmu4bDRx995HTx/B6hkI3WrFljxhL07LpNmzayZMkSv24leOgZaK1ataR79+7m/rRp0+QHvXQTjtr06wfs9O3b10xu8B6I1dlwt+pHdPqhv/zlL6bbSOmsI++JDQcOHHCwZIGBUMgms2fPlmbNmsmJEyekT58+Mm/ePLlBPx7RT3377beyYsUKe19bC9pK8KAZ7zv0WOhy7tw5u+3ChQsp7vtT1+vo0aNTTOX+8MMP7bqObeHPIRQywdy5c6V8+fJm/rTH4MGDzbbOnTubriKd+XH+/HnJlSuX6YPXszRtNeii0/D8bZ90em3Tpk3N1EFtJRQpUkTmz59vHqf3a/rKNfsB7HrHaMqUKeJyueyi3SseEyZMMDPh/G2fNMj+9a9/mfddbGyslCxZ0g6g6/127dqJr1m1SqRVK5FCha5+tlFqvcabN4t06OB+XK5cIsWKXb3QLTv53rQXP6RT6HSswJtOP9VFr0vQs2j9o1S6/vXXX//m//vbPun4QcuWLU0Xxffffy9hYWHmj7J169by7LPPmllVcPYYBeI+6cDyoEGDTNerPu6XX34x18ncc889MnDgQDMjztesXy+ydKleV+H+aOzU6MS9O+90T02NjBTRnr6ff9ZrgbK7tCIhLk9tdZ0DFRUVZSqBQJrBEB8f73QRcB1xcXESaHjfBZeEBPcX6eiM2TJl3NsmTBDp2dO9rjVwlSruC9g6dxZ58033dQxKe8kya9jRU4+fOXNGIjV50sDpHABk8RXM4elceqHdRhoInoDQD83Tz0hq1sz9UdvZjVAAAAft3Hl1/b333K0KtXy5+/OQ9BvZshOhAAAOunz56vojj7hbDToHQD9iW8cVsvsLdwgFAHBQsWJX13/99Bsz9qCzkBQtBQAIInXquGccKf2APKWfGnP8uHu9QoXsLQ+hAABZaO5ckfLlU35fwuDB7m0620gHoYcOdW9/6y293sL91Zz65TsxMSI9emRveQkFAMhCiYn6mWfus38PbQXotkOH3Pf793cHQrVq7o/Q1mmoXbq4Ww6ebqTswsVrAJCFHn7YvVyPDjLr4jRaCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFh+IB5/Gl9zDKXFxcRJILl26lKHH0VIAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSggOC1fLhIaKhISIvLf/17dfuWKSL167u2lSokkJjpZSiDbEQoITk2bivTt614fMkRk82b3ugbE11+7Q+Hdd0UiIx0tJpDdCAUEr//8RyQ2VuTiRZEuXUS++UZk2DD3z/r1E2nSxOkSAtmOUEDwypNHZNo0kRw53C2Fxo31m0hEqlQReeklp0sHOIJQQHCrXVvkuefc6+fPi4SFuYNCAwMIQoQCsGtXyoHmffucLA3gKEIBwW3OHJEZM9zrOttIPfaYyNGjjhYLcAqhgOClFX/Pnu71Vq1E1qwRiY4WOXFCpHt3p0sHOIJQyASrVq2SVq1aSaFChSQkJMQsEydOtD+fMmWK3Z7asmLFCvG3fVK7d++WBx98UEqUKCG5c+eWggULSuPGjeXjjz8Wv/Doo+4AKFBA5K23RIoUEZkwwf2z+fNF3n5bfNmoUaOkSZMmUqRIEfP6lypVSrp27Sp79uyxj7l06ZIMGzZMypYtK7ly5ZLixYtL//795eeffxZ/3adJkyZJw4YNJW/evPa9uWPHDkfLHUgIhUywfv16Wbp0qRTQyiUVWrHWrVs3xaJveo+YmBjxt31yuVxyxx13yIwZM+T48eNStWpVuXLligmTtm3byqZNm8SnaQh88ol7ffx4dyCoe+8V6dzZvd6/v0+PL4wdO9a83vnz55dixYrJgQMHZOrUqdKgQQNJ/PWiu27dusnQoUNl//79JhiOHTsmr776qtx9992SnJws/rhPCxculA0bNpi/K2Q+QiETdOnSxbxhFy9enOrPW7duLWvXrk2x6Fm10oq1cuXK4m/7dOjQIdm7d69Z1zNRDZG5c+fawDh48KD4fCvB5XIvHTum/Nn06e7tWgmVLi2+qnv37rJv3z7Zvn27OZPup9dWiMiRI0fks88+M8dkuu6LiIwZM8acTc/RMRQRWblypcybN0/8bZ/U+PHjzXtTww6Zj1DIBNHR0RIeHp7hxy9atEi2bNli1gcMGCD+uE/a0ilfvrxZHzJkiNx8883Srl07yZEjhzk7veuuu7KxtMFp0KBBUrJkSXu/UaNGdl27XvSM2qN9+/b2BCXPr9Nt9X3ob/ukihYtKmE6dRhZglBwwIgRI8xtzZo1TUvBH+kf5fLly6V27dpy4cIF05w/ffq03HjjjSYg+KPNXtp198Ybb5h17SZq3rx5itZa4cKFzW1oaKhtpWrXjL/tE7IeoZDNtPL8/PPPzfqTTz4p/kr7o3v27Cnfffed9O3b1wxczpo1y4wv9O7d2ye7JgLV2bNnzTiOdvXp+NT8+fPtWXVqtHsv0PYJmYdQyGYjR440tzpjp1OnTuKvtH93wYIFZl1nh+hMkA4dOkjkrx8gt2zZModLGBy0r11nfGmlWbFiRVm9erVU0Y/p+PU95qEDzJ4wT0hIMOve3TT+sk/IeoRCNtLm+syZM826nl1r/7u/OnPmjF1ft26duf3hhx8kKSnJrGtIIGtt27ZN6tWrZ1pr2ve+Zs0a083i0bJlS7vuGWDWID+vH+dxzc/9ZZ+Q9QiFTKCzbnTQVedXewwePNhs6+yZ3ihipgJevnxZoqKipEePHuLP+9S0aVMzfqC0G6l69epmLEG7JnLmzCn333+/+JxVq9wXqelURv1obF2uufbCTEF9+GH31c06IFupksj//qen2OJrdGBfp5oqDWO9rkQrVF3eeustM97jOQ56EhIbG2sHnLXCveeee8Tf9kk99dRT5n2otx4tWrQw21577TXHyh4o/PdU1Yfo9Di9kMub9q3rohcLec6sPW9qDYSIiAjx533S2UnarH/xxRfliy++kF27dpmQ0Gb/c889J7Vq1RKfs369yNKlOmrpvmjtWsePi9Sp477Nl09Epwpv3aq1kMjhw5rq4kt0gN9j48aNKX7maQW8++67UqFCBTPXX4+nzu3Xbr7hw4ebQWdfk5F9Onr06G/em55B85MnT2ZLOQNZiCsDo05aQejZrR4UPQsMFPHx8U4XAdlJ+9JvuMH98RZlyri36RXMno+60IvYevVyr2sYVK0q8uabmuLuT0/V6zK8+ukR2OLi4iSQ6NXtOg1ZT1A9Y3+p8b1TBSCr6OcapXc9iXcXkecs2nOrn56qX+EJBDhCAfDQ8QbtNlJ164poF5inFaEOHXKsaEB2IRQADx1rWLLE/f3N2kLQcQQddNYBaRVAXadAWggFwFv9+iJ6ceHp0zq5Xz9Rzv05SEpnIgEBjlAAvH35pXv8QJ06pZedu9f1oyH4mAUEAUIBwUM/xVU/xM/r2gsZPNi9zXM9iY4haADUqCGi04m/+so980ivZ9CZS0CAIxQQPPSjsHV++68XRxl6TYJu8wwi33mniE7X27lTRK841/vanfTrRV9AoOPiNQQPHTTWJT2jR7sXIEjRUgAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsIL6A/EC7Yu5AeDPoqUAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAAKwckgEul8vcXr58OSMPBwD4GE/97anP/1QoJCUlmdtly5ZlRtkAAA7R+jwqKirNn4e4rhcbIpKcnCyHDx+WiIgICQkJyewyAgCymFb1GghFixaV0NDQPxcKAIDgwEAzAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAPH4f8EOdr53tgDzAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def plot_maze_with_states() -> None:\n", " \"\"\"Plot the maze with state indices.\"\"\"\n", " grid = np.ones(\n", " (n_rows, n_cols),\n", " ) # Start with a matrix of ones. Here 1 means “free cell”\n", " for i in range(n_rows):\n", " for j in range(n_cols):\n", " if maze_str[i][j] == \"#\":\n", " grid[i, j] = 0 # We replace walls (#) with 0\n", "\n", " _fig, ax = plt.subplots()\n", " ax.imshow(grid, cmap=\"gray\", alpha=0.7)\n", "\n", " # Plot state indices\n", " for (\n", " s,\n", " (i, j),\n", " ) in state_to_pos.items():\n", " cell = maze_str[i][j]\n", "\n", " if cell == \"S\":\n", " label = f\"S\\n{s}\"\n", " color = \"green\"\n", " elif cell == \"G\":\n", " label = f\"G\\n{s}\"\n", " color = \"blue\"\n", " elif cell == \"X\":\n", " label = f\"X\\n{s}\"\n", " color = \"red\"\n", " else:\n", " label = str(s)\n", " color = \"black\"\n", "\n", " ax.text(\n", " j,\n", " i,\n", " label, # Attention : matplotlib, text(x, y, ...) expects (column, row)\n", " ha=\"center\",\n", " va=\"center\",\n", " fontsize=10,\n", " fontweight=\"bold\",\n", " color=color,\n", " )\n", "\n", " ax.set_xticks([]) # remove numeric axes, we don't need.\n", " ax.set_yticks([])\n", " ax.set_title(\"Maze with state indices\")\n", "\n", " plt.show()\n", "\n", "\n", "plot_maze_with_states()" ] }, { "cell_type": "markdown", "id": "db078d86", "metadata": {}, "source": [ "### 2.4 Actions and deterministic movement" ] }, { "cell_type": "markdown", "id": "96e7f1f2-9d73-410b-853d-e39f40dfb5da", "metadata": {}, "source": [ "We first define integer codes for each action. \n", "\n", "**Exercise 3.** How many possible actions can the agent take in the maze?" ] }, { "cell_type": "markdown", "id": "22259ab4-527e-4d7c-bb30-98fb240da6d5", "metadata": {}, "source": [ "We have four possible actions in the maze. \n", "\n", "In this following cell, each action is mapped to an integer (0,1,2,3). This makes it easy to store and use actions inside arrays and matrices\n", "\n", "Here we use Unicode arrow character:\n", "\n", "- \"\\u2191\" : ↑ (up arrow)\n", "\n", "- \"\\u2192\" : → (right arrow)\n", "\n", "- \"\\u2193\" : ↓ (down arrow)\n", "\n", "- \"\\u2190\" : ← (left arrow)" ] }, { "cell_type": "code", "execution_count": 81, "id": "f7f0b8e4-1f48-4d03-9e5f-a47e59c3e827", "metadata": {}, "outputs": [], "source": [ "A_UP, A_RIGHT, A_DOWN, A_LEFT = 0, 1, 2, 3\n", "ACTIONS = [A_UP, A_RIGHT, A_DOWN, A_LEFT]\n", "action_names = {A_UP: \"\\u2191\", A_RIGHT: \"\\u2192\", A_DOWN: \"\\u2193\", A_LEFT: \"\\u2190\"}" ] }, { "cell_type": "code", "execution_count": 82, "id": "3773781c-a0cd-48db-967b-d4b432d17046", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "↑\n" ] } ], "source": [ "print(action_names[0])" ] }, { "cell_type": "markdown", "id": "4b957f5a-ee39-4437-abc1-4809105ad83c", "metadata": {}, "source": [ "**Exercise 4.** Now we define a **deterministic movement function** `move_deterministic(i, j, a)`. \n", "\n", "This function simulates the robot trying to move from (i, j) in direction a.\n", "\n", "But if the movement hits a wall or boundary, the agent stays in place." ] }, { "cell_type": "code", "execution_count": null, "id": "4b06da5e-bc63-48e5-a336-37bce952443d", "metadata": {}, "outputs": [], "source": [ "def move_deterministic(i: int, j: int, a: int) -> tuple[int, int]:\n", " \"\"\"Deterministic movement on the grid. If the movement hits a wall or boundary, the agent stays in place.\n", "\n", " Args:\n", " i (int): current row index\n", " j (int): current column index\n", " a (int): action to take (A_UP, A_DOWN, A_LEFT, A_RIGHT)\n", "\n", " Returns:\n", " (tuple[int, int]): new (row, column) position after taking action a\n", "\n", " \"\"\"\n", " candidate_i, candidate_j = (\n", " i,\n", " j,\n", " ) # It means “Unless the action succeeds, the robot stays in place.”\n", "\n", " # Now each action changes the coordinates of the robot:\n", " if a == A_UP:\n", " candidate_i, candidate_j = (\n", " i - 1,\n", " j,\n", " ) # if the action is UP, then row becomes row -1\n", " elif a == A_DOWN:\n", " candidate_i, candidate_j = (\n", " i + 1,\n", " j,\n", " ) # if the action is DOWN, then row becomes row +1\n", " elif a == A_LEFT:\n", " candidate_i, candidate_j = (\n", " i,\n", " j - 1,\n", " ) # if the action is LEFT, then column becomes column -1\n", " elif a == A_RIGHT:\n", " candidate_i, candidate_j = (\n", " i,\n", " j + 1,\n", " ) # if the action is RIGHT, then column becomes column +1\n", "\n", " # Check boundaries\n", " if not (0 <= candidate_i < n_rows and 0 <= candidate_j < n_cols):\n", " # If the robot tries to move outside the maze\n", " # It will not move and it stays at (i, j).\n", " return i, j\n", "\n", " # Check wall\n", " if maze_str[candidate_i][candidate_j] == \"#\":\n", " # If the next cell is a wall, the robot stays in place.\n", " return i, j\n", "\n", " return candidate_i, candidate_j # Otherwise, return the new position" ] }, { "cell_type": "markdown", "id": "c9e620e6", "metadata": {}, "source": [ "### 2.5 Transition probabilities and reward function" ] }, { "cell_type": "markdown", "id": "80bd2bca-7717-4b5f-bffa-76fe86a51d35", "metadata": {}, "source": [ "Recall that we set the discount factor $\\gamma \\in(0,1)$, that is, the future rewards are multiplied by $\\gamma$, so immediate rewards matter a little bit more than future ones. \n", "\n", "\n", "Moreover, we consider a probability error $p_{\\text{error}}$, which means, with probability $p_{\\text{error}}$, the robot **does not** execute the intended action but one of the 3 other directions (chosen uniformly). With probability $1-p_{\\text{error}}$, the robot executes the action that we asked." ] }, { "cell_type": "code", "execution_count": 84, "id": "610253e7-f3f7-4a30-be3e-2ec5a1e2ed04", "metadata": {}, "outputs": [], "source": [ "gamma = 0.95\n", "p_error = 0.1 # probability of the error to a random other direction" ] }, { "cell_type": "markdown", "id": "0d1ceff8-86e0-4c45-83d3-af9fae974608", "metadata": {}, "source": [ "Now we initialize the state–transition probability : the probability of reaching next state $s'$ after taking action $a$ in state $s$. \n", "$$\n", " p(s' \\mid s, a)\n", " = \\mathbb{P} \\big[S_t=s'\\,|\\, S_{t-1}=s, \\,A_{t-1}=a\\big]\n", "$$\n", "\n", "We store these transition probabilities in the 3D array `P` (`P[a][s, s_next]`), which has shape `(n_actions, n_states, n_states)`:\n", "\n", "`P[a, s, s_next] = P(S_{t+1} = s_next | S_t = s, A_t = a)`.\n", "\n", "We also initialize the reward vector `R`, which has length `n_states`, where `R[s]` is the reward received when the agent is in state `s`.\n", "\n", "In this maze game, we assume that the reward depends only on the current state, which is natural: in navigation tasks, being in a particular location is what matters, not the direction you used to reach it." ] }, { "cell_type": "code", "execution_count": 85, "id": "7a51f242-fe4e-4e74-8a1f-a8df32b194b8", "metadata": {}, "outputs": [], "source": [ "# Initialize transition matrices and reward vector\n", "P = np.zeros((len(ACTIONS), n_states, n_states))\n", "R = np.zeros(n_states)" ] }, { "cell_type": "markdown", "id": "c08f4af5-a2a7-4baa-b5da-c7ce636d8a4a", "metadata": {}, "source": [ "Now we assign the reward to each state. \n", "\n", "For each state index s:\n", "\n", "1. If s is a goal, then the reward = +1.0\n", "2. If s is a trap, then the reward = −1.0\n", "3. Otherwise for the normal cell, the reward = −0.01 every time you leave this cell.\n", "\n", "Recall that rewards are received at the moment the agent executes an action. Here when the agent moves out of the cell, we set reward −0.01. " ] }, { "cell_type": "code", "execution_count": 86, "id": "49d54d1f-dc29-45b6-ad31-ad0e848f920d", "metadata": {}, "outputs": [], "source": [ "# Set rewards for each state\n", "step_penalty = -0.01\n", "goal_reward = 1.0\n", "trap_reward = -1.0" ] }, { "cell_type": "markdown", "id": "dd571ec8-c36a-4e20-bec6-9e6458dc622b", "metadata": {}, "source": [ "**Exercise 5.** Why do we set the step penalty to -0.01 in this MDP?" ] }, { "cell_type": "markdown", "id": "1e8ea171", "metadata": {}, "source": [ "We set a small negative step penalty (`-0.01`) for two main reasons:\n", "\n", "- Incentivize Efficiency: It forces the agent to find the shortest path to the goal. By losing a small amount of reward at every step, the agent learns that the faster it reaches the goal, the higher its total cumulative return will be.\n", "\n", "- Prevent Loitering: It discourages infinite loops or wandering. Without this penalty (i.e., if step reward = 0), the agent might be indifferent between reaching the goal now or in 1000 steps, potentially leading to a policy that never terminates." ] }, { "cell_type": "markdown", "id": "07bfb065-b1af-4df1-885e-780fe250f2fb", "metadata": {}, "source": [ "**Exercise 6.** We now define the reward vector. Recall that we have already initialized\n", "`R = np.zeros(n_states)`.\n", "If a state belongs to `goal_states`, we assign the `goal_reward`.\n", "If it belongs to `trap_states`, we assign the `trap_reward`.\n", "Otherwise, we assign the `step_penalty`. " ] }, { "cell_type": "code", "execution_count": 87, "id": "b9b7495a-c233-425c-99c0-5bddaf6c3225", "metadata": {}, "outputs": [], "source": [ "for s in range(n_states):\n", " if s in goal_states:\n", " R[s] = goal_reward\n", " elif s in trap_states:\n", " R[s] = trap_reward\n", " else:\n", " R[s] = step_penalty" ] }, { "cell_type": "markdown", "id": "b90fb80c-9452-48a2-889f-286703c2ae93", "metadata": {}, "source": [ "Now we define terminal states and a helper function. Here terminal_states is a set containing all absorbing states, which means, reaching them ends the episode conceptually. \n", "\n", "Moreover, `is_terminal(s)` is a small helper to check if a state is terminal." ] }, { "cell_type": "code", "execution_count": 88, "id": "eca4c571-39c7-468b-af86-0bab9489415e", "metadata": {}, "outputs": [], "source": [ "terminal_states = set(goal_states + trap_states)\n", "\n", "\n", "def is_terminal(s: int) -> bool:\n", " \"\"\"Check if a state is terminal (goal or trap).\"\"\"\n", " return s in terminal_states" ] }, { "cell_type": "markdown", "id": "3a9a1d54-8339-402b-84e9-105961ed78d7", "metadata": {}, "source": [ "Now we need to fill the transition matrices `P[a][s, s_next]`. \n" ] }, { "cell_type": "markdown", "id": "d9cfd15c-12cc-48bb-bd88-07f3ae3db31c", "metadata": {}, "source": [ "**Exercise 7.** **Complete the `# TO DO` part in the program below** to fill the transition matrices `P[a][s, s_next]`. " ] }, { "cell_type": "code", "execution_count": 89, "id": "2d03276b-e206-4d1f-9024-f6948ca61523", "metadata": {}, "outputs": [], "source": [ "for s in range(n_states): # We loop over all states s.\n", " i, j = state_to_pos[\n", " s\n", " ] # We recover the states to their coordinates (i, j) in the maze.\n", "\n", " # First, in a goal or trap state,\n", " # No matter which action you “choose”, you stay in the same state with probability 1.\n", " # This makes the terminal states as the absorbing states.\n", " if is_terminal(s):\n", " # Terminal states: stay forever\n", " for a in ACTIONS:\n", " P[a, s, s] = goal_reward\n", " continue\n", "\n", " # If the state is non-terminal, we define the stochastic movement.\n", " # For a given state s and intended action a,\n", " # With probability 1 - p_error, the robot will move in direction a;\n", " # With probability p_error, the robot will move in one of the other 3 directions, each with probability p_error / 3.\n", " for a in ACTIONS:\n", " # main action (intended action)\n", " main_i, main_j = move_deterministic(i, j, a)\n", " s_main = pos_to_state[\n", " (main_i, main_j)\n", " ] # s_main is the state index of that next cell.\n", " P[a, s, s_main] += (\n", " 1 - p_error\n", " ) # We add probability 1 - p_error to P[a, s, s_main].\n", "\n", " # error actions\n", " other_actions = [\n", " a2 for a2 in ACTIONS if a2 != a\n", " ] # other_actions = the 3 actions different from a.\n", " for a2 in other_actions: # for each of the error action,\n", " error_i, error_j = move_deterministic(i, j, a2)\n", " s_error = pos_to_state[(error_i, error_j)] # get its state index s_error\n", " P[a, s, s_error] += p_error / len(\n", " other_actions,\n", " ) # add p_error / 3 to P[a, s, s_error]\n", "# So for each (s,a), probabilities over all s_next sum to 1." ] }, { "cell_type": "markdown", "id": "7841b264-af00-4322-b728-adcffac0ef89", "metadata": {}, "source": [ "Now we check if the transition matrices `P[a][s, s_next]` are computed correctly.\n", "For each action `a`, we sum the transition probabilities over all possible next states `s_next` and verify that these sums are equal to 1.\n", "\n", "This is because the matrix `P[a, s, s_next]` stores the transition probability\n", "\n", "$\\mathbb{P} \\big[S_t=s_{\\text{next}}\\,|\\, S_{t-1}=s, \\,A_{t-1}=a\\big]$. \n", "\n", "Therefore, for each action $a$, and for each state $s$, the sum over $s_{\\text{next}}$ of $\\mathbb{P} \\big[S_t=s_{\\text{next}}\\,|\\, S_{t-1}=s, \\,A_{t-1}=a\\big]$ should be 1. " ] }, { "cell_type": "code", "execution_count": 90, "id": "341fe630-8f87-4773-84ad-92d3516e53e2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Action ↑: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]\n", "Action →: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]\n", "Action ↓: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]\n", "Action ←: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]\n" ] } ], "source": [ "for a in ACTIONS:\n", " # For each action a:\n", " # P[a] is a matrix of shape (n_states, n_states).\n", " # P[a].sum(axis=1) sums over next states s_next, giving for each state s:\n", " # We print these row sums.\n", " # If everything is correct, they should be very close to 1.\n", "\n", " probs = P[a].sum(axis=1)\n", " print(f\"Action {action_names[a]}:\", probs)" ] }, { "cell_type": "markdown", "id": "46d23991", "metadata": {}, "source": [ "## 3. Policy evaluation\n", "\n", "### 3.1 Bellman expectation equation" ] }, { "cell_type": "markdown", "id": "305b047c-e83b-4f42-b64e-e2050d5deeff", "metadata": {}, "source": [ "Recall that the value function under a policy $\\pi$ is defined as:\n", "$$\n", "V^{\\pi}(s)=\\mathbb{E}\\Big[\\:G_t \\:\\Big|\\: S_t=s\\:\\Big]\n", "$$\n", "where the return $G_t$ is\n", "$$\n", "G_t=R_t +\\gamma R_{t+1}+\\gamma^2 R_{t+2}+... . \n", "$$\n", "This means *The value of a state is the expected discounted sum of all future rewards\n", "when following policy $\\pi$.*\n", "\n", "We know that $G_t=R_t+\\gamma G_{t+1}$, and plugging this equation into the definition of $V^{\\pi}(s)$, we get \n", "$$\n", "V^{\\pi}(s)=\\mathbb{E}\\Big[\\:R_t \\:\\Big|\\: S_t=s\\:\\Big]+\\gamma\\mathbb{E}\\Big[\\:G_{t+1} \\:\\Big|\\: S_t=s\\:\\Big]. \n", "$$\n", "This step shows simply ``The total future reward = immediate reward + discounted reward from next state.''" ] }, { "cell_type": "markdown", "id": "88ea8d56-3b62-4690-9ff7-469e43726fbc", "metadata": {}, "source": [ "For the expected immediate reward part $\\mathbb{E}[R_t| S_t=s]$, as we are in a maze problem, the reward depends only on the current state, not the time step, i.e., $\\mathbb{E}[R_t| S_t=s]=R(s)$. Hence we get \n", "$$\n", "V^{\\pi}(s)=R(s)+\\gamma\\mathbb{E}\\Big[\\:G_{t+1} \\:\\Big|\\: S_t=s\\:\\Big]. \n", "$$\n", "\n", "Moreover, in this maze problem, we consider a deterministic policy $A_t=\\pi(s)$ (the action depends only on the state). Therefore, \n", "$$\n", "V^{\\pi}(s)=\\mathbb{E}\\Big[\\:R_t \\:\\Big|\\: S_t=s\\:\\Big]+\\gamma\\mathbb{E}\\Big[\\:G_{t+1} \\:\\Big|\\: S_t=s, A_t=\\pi(s)\\:\\Big]. \n", "$$\n", "\n", "Now **given the state $S_t=s$ and $A_t=a$**, the next state is random (because of the error probability) and we know the transition probability \n", "$$\n", "\\mathbb{P}\\big(\\:S_{t+1}=s' \\:|\\:S_t=s, \\, A_t=a\\big)=P\\big(s'\\:\\big|\\:s, a\\big). \n", "$$" ] }, { "cell_type": "markdown", "id": "c25e255d-8f58-4eaf-9485-cee6ab3bea6c", "metadata": {}, "source": [ "Therefore,\n", "$$\n", "\\mathbb{E}\\big[\\,G_{t+1}\\,|\\,S_t=s,A_t=a\\,\\big] =\\sum_{s'}\\mathbb{E}\\big[\\,G_{t+1}\\,|\\,S_{t+1}=s'\\,\\big]\\times \\mathbb{P}\\big[S_{t+1}=s'\\,\\big|\\,S_t=s, A_t=a\\, \\big]\n", "$$\n", "$$\n", "\\hspace{-1.2cm}=\\sum_{s'}V^{\\pi}(s')P\\big(s'\\:\\big|\\:s, a\\big),\n", "$$\n", "where here we use the Markov property. (**Question: Can you show the detailed computations here?**)" ] }, { "cell_type": "markdown", "id": "9a2b6cff-e848-44a2-b504-973067b367b3", "metadata": {}, "source": [ "In conclusion, we have (the Bellman expectation equation)\n", "$$\n", "V^{\\pi}(s)=R(s)+\\gamma \\sum_{s'}P\\big(\\,s'\\,\\big|\\,s, \\pi(s)\\,\\big)V^{\\pi}(s').\n", "$$" ] }, { "cell_type": "markdown", "id": "15049fdb-f3af-4f78-b556-817284260ed0", "metadata": {}, "source": [ "### 3.2 Define a function which computes the value function $V^{\\pi}(s)$ for a given deterministic policy. \n", "\n", "\n", "**Exercise $8^*$.** Now we define `policy_evaluation(...)`, which computes the value function $V^{\\pi}(s)$ for a given deterministic policy. \n", "\n", "The input of this function `policy_evaluation(...)` are:\n", "1. policy: array of size `n_states`, each entry is an action 0,1,2,3, which correspond to UP, RIGHT, DOWN, LEFT.\n", "2. `P`: the transition probabilities `P[a, s, s']`.\n", "3. `R`: the reward vector `R[s]`.\n", "4. gamma: the discount factor $\\gamma\\in(0,1)$.\n", "5. theta: convergence threshold.\n", "6. max_iter: which is used to avoid infinite loops.\n", "\n", "How can we apply the Bellman expectation equation\n", "$$\n", "V^{\\pi}(s)=R(s)+\\gamma \\sum_{s'}P\\big(\\,s'\\,\\big|\\,s, \\pi(s)\\,\\big)V^{\\pi}(s').\n", "$$\n", "here ?\n", "\n" ] }, { "cell_type": "markdown", "id": "5c48f489-3508-4981-8b35-5bedc2e5838c", "metadata": {}, "source": [ "We start with an initial guess of $V^{\\pi}$(e.g., all values = 0) and repeatedly apply the Bellman equation to update each state:\n", "$$\n", "V_{k+1}^\\pi(s) \\leftarrow R(s)+\\gamma \\sum_{s'}P\\big(\\,s'\\,\\big|\\,s, \\pi(s)\\,\\big)V^{\\pi}_k(s').\n", "$$\n", "until values converge." ] }, { "cell_type": "code", "execution_count": null, "id": "2fffe0b7", "metadata": {}, "outputs": [], "source": [ "def policy_evaluation( # noqa: PLR0913\n", " policy: np.ndarray,\n", " P: np.ndarray,\n", " R: np.ndarray,\n", " gamma: float,\n", " theta: float = 1e-6,\n", " max_iter: int = 10_000,\n", ") -> np.ndarray:\n", " \"\"\"Evaluate a deterministic policy for the given MDP.\n", "\n", " Args:\n", " policy: array of shape (n_states,), with values in {0,1,2,3}\n", " P: array of shape (n_actions, n_states, n_states)\n", " R: array of shape (n_states,)\n", " gamma: discount factor\n", " theta: convergence threshold\n", " max_iter: maximum number of iterations\n", "\n", " \"\"\"\n", " n_states = len(R) # get the number of states\n", " V = np.zeros(n_states) # initialize the value function\n", "\n", " for _it in range(max_iter): # Main iterative loop\n", " V_new = np.zeros_like(\n", " V,\n", " ) # Create a new value vector and we will compute an updated value for each state.\n", "\n", " # Now we update each state using the Bellman expectation equation\n", " for s in range(n_states):\n", " a = policy[s] # Extract the action chosen by the policy in state\n", " V_new[s] = R[s] + gamma * np.sum(P[a, s, :] * V)\n", "\n", " delta = np.max(\n", " np.abs(V_new - V),\n", " ) # This measures how much the value function changed in this iteration:\n", " # If delta is small, the values start to converge; otherwise, we need to keep iterating.\n", " V = V_new # Update V, i.e. Set the new values for the next iteration.\n", "\n", " if delta < theta: # Check convergence: When changes are tiny, we stop.\n", " break\n", "\n", " return V # Return the final value function, this is our estimate for V^{pi}(s), s in the state set." ] }, { "cell_type": "markdown", "id": "09ef3439", "metadata": {}, "source": [ "### 3.3 Evaluating a random policy" ] }, { "cell_type": "markdown", "id": "eecbca15-f89f-47bf-a13d-7d7c051699b8", "metadata": {}, "source": [ "Now we use the policy evaluation function `policy_evaluation` to evaluate a random policy. \n", "\n", "We first generate a `random_policy`, which is an array like [2, 0, 1, 3, 0, 2, ...] and has the size `n_states`. (Recall that the policy is a mapping from states to actions)." ] }, { "cell_type": "code", "execution_count": 92, "id": "b4a44e38", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0 3 2 1 1 3 0 2 0 0 2 3 2 3 2 3 2 0 3 1 2 1]\n" ] } ], "source": [ "# Random policy: for each state, pick a random action\n", "random_policy = rng.integers(low=0, high=len(ACTIONS), size=n_states)\n", "\n", "print(random_policy)" ] }, { "cell_type": "markdown", "id": "3fe07992-ce82-4124-aebc-a6384d417f64", "metadata": {}, "source": [ "Now we call the function `policy_evaluation(...)` to compute $V^{\\pi_{\\text{random}}}(s)$." ] }, { "cell_type": "code", "execution_count": 93, "id": "c5f559b2-452a-477c-a1fa-258b40805670", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Value function under random policy:\n", "[ -0.2 -0.2 -0.201 -0.204 -0.205 -0.202 -0.214 -0.429 -0.212\n", " -0.207 -0.276 -0.459 -0.352 -0.366 -5.827 -4.605 20. -0.366\n", " -0.999 -20. -6.4 -3.163]\n" ] } ], "source": [ "V_random = policy_evaluation(policy=random_policy, P=P, R=R, gamma=gamma)\n", "print(\"Value function under random policy:\")\n", "print(V_random)" ] }, { "cell_type": "markdown", "id": "f46c70ba-2932-49af-b568-b5477260bc94", "metadata": {}, "source": [ "Here in this value vector of the policy, \n", "- If it is a negative values, then the agent tends to move around aimlessly, fall in traps, or take too long.\n", "- It it is a higher values, then the agent is closer to the goal or more likely to reach it" ] }, { "cell_type": "markdown", "id": "1efcb076-467c-42d8-94e8-87453f688bbd", "metadata": {}, "source": [ "Now we define a function `plot_values`, which displays the value function $V(s)$ and displays it on the maze grid. It helps students visually understand:\n", "- which states are good (high value, near the goal),\n", "- which states are bad (low value, near traps),\n", "- how a policy affects the long-term expected reward." ] }, { "cell_type": "code", "execution_count": null, "id": "4c428327", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdcAAAGbCAYAAACWHtrWAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAARjhJREFUeJzt3Qd4FGX+B/DvptcNoSQhEHqHUAUUUWnSBEQ9FRtgvb/lUBEBGyJFBBThlAPvTkVQBPQUEAREBFHpJfROIAFSSCC9kez8n/dddskmm7rvbrLL9+MzLjsz+868O5v5zdtmdJqmaSAiIiJl3NQlRURERAKDKxERkWIMrkRERIoxuBIRESnG4EpERKQYgysREZFiDK5ERESKMbgSEREpxuBKRESkGIOrg507dw46nQ6LFi2qku2vX78eHTt2hI+Pj9yPlJQUVEdi3yZPnlzVu1GtiN+M+F7Eb8hViWMu8lhYo0aNMHr06CrbJ6LKYHAtxbBhw+Dn54f09PQS13nsscfg5eWF5ORkVHdiHx966CH4+vpi/vz5WLJkCfz9/atsf37++WcGUCJySR5VvQPVmQicP/30E3788UeMHDmy2PKsrCysWrUKAwcORK1atVDd7d69W14oTJ06Ff369avq3ZHBVQR5awE2OzsbHh78eRJw4sQJuLmxHEDOhb/YMkqugYGBWLp0qdXlIrBmZmbKIOwMEhMT5WuNGjVQ3Ylq6+ocXMXzLsQFANmft7c3PD09q3o3iCqEwbUUovr0/vvvx6ZNm8yBqTARdEXwFUH4ypUrGDduHCIjIxEQEAC9Xo9BgwbhwIEDZW6nV69ecipKtDOJ9qbCDAYD5s6di7Zt28oAFBoair///e+4evVqmdsYNWqU/HfXrl1lu5apHaukNq2i+7Vlyxb5uRUrVmD69OmoX7++3Ie+ffvi9OnTxT6/c+dODB48GMHBwbL6uX379pg3b545b6LUKog0TVNpba779++X36n4bsV3LLa7Y8cOq+2Sf/31F8aOHYs6derIbd933324fPmyxbqpqak4fvy4fC2L+I6GDBmCDRs24JZbbpG/jc8++0wu+/LLL9GnTx+EhITIQNCmTRssWLCgxDT+/PNPdOvWTX53TZo0weLFi4ute+TIEZmm2I74nqdNmyaPvTX/+te/5O9BbDs8PBwvvvhisbZ0cRzbtWuHgwcP4q677pLNHc2aNcP3338vl//+++/o3r273F7Lli3x66+/lvmdmH4Py5cvx5tvvomwsDD5XYu/h9jY2GLrf/fdd+jSpYvcRu3atfH444/j4sWLZW7H2u9T5O/VV1+Vy0S+xXckapeSkpKQkZEh9+Pll18ultaFCxfg7u6OGTNmlLldIpuIR85RyX755RfxSD7tk08+sZifnJyseXp6aiNHjpTvd+/erTVt2lSbOHGi9tlnn2lTpkzR6tWrpwUFBWkXL140fy46Olqm9+WXX5rn3XXXXXIqatSoUVrDhg0t5j3zzDOah4eH9uyzz2oLFy7UJkyYoPn7+2tdu3bV8vLySs3Hc889J7ct9m3JkiXatm3b5DKxDbGtooru1+bNm+XnO3XqpHXp0kX7+OOPtcmTJ2t+fn5at27dim3Py8tLpv3uu+9qCxYs0MaMGaP169dPLhfbvvvuu2V6Yl9Mk4mYLz5ncvjwYZnPunXralOnTtU++OADrXHjxpq3t7e2Y8cO83riezXtY58+feRxe+211zR3d3ftoYcesthH07qFj0VJRD6aNWumBQcHy2MsvnvxfQjiux89erT8PsT2+vfvL9P99NNPi6XRsmVLLTQ0VHvzzTfl8s6dO2s6nU7mzyQuLk6rU6eO3Jb4fmfPnq01b95ca9++vUxX/IZMxHck5onvVWz7pZdeknkt+nsQxzE8PFyLiIjQXn/9dblumzZt5LrLli3TwsLC5Lbmzp1r/t2mpaWV+p2Yfg+RkZFy3+bMmSO/Gx8fH61FixZaVlZWse9a7Jf4nsR6vr6+WqNGjbSrV68Wy0/R763w7zM9PV1r166d3HfxdyB+W+I3IdLev3+/XOexxx6T33N+fr5FWrNmzZLf9/nz58s85kS2YHAtg/jjFCf02267zWK+OLmKk8CGDRvk+5ycHK2goMBiHXESFCd/EcxUBNc//vhDfvabb76xWG/9+vVW5xdlOsGJC4HCKhpcW7dureXm5prnz5s3T84/dOiQ+TsTgU+kW/jEKRgMBvO/X3zxxWIn0pKC6/Dhw2WwPnPmjHnepUuXtMDAQO3OO+8slkcRbApv69VXX5Un45SUlEoHV7Gu+K6LKhxETAYMGKA1adLEahpbt241z0tMTJS/EXEBYPLKK6/I9Xbu3Gmxngh4hYOrmCe+ExHMC//2RNAW633xxRfmeeI4inlLly41zzt+/Lic5+bmZnGBIn7T5fleTL8HEYwLB+IVK1bI+eJ3IYggHxISIgNidna2eb01a9bI9SZNmlSh4CrWF+v88MMPxfbJdMxNeVi3bp3FcnERYO1vjUg1VguXQVQhjRgxAtu3b7cYAiGqhEWVrKiaFETVlKnTRUFBgeyZK6ouRRXbvn37lOyLqFYLCgrC3XffLau/TJOoahPb2rx5MxzhySeflD2kTe644w75evbsWXP1bXR0NF555ZVi7btFh1mUh/g+f/nlFwwfPlxWo5rUrVsXjz76qKxmTUtLs/jMc889Z7EtsY8infPnz5vniapGEcfLO8yjcePGGDBgQLH5oprTRFQxi2Miql7F91G0yllUGZu+L0FUW4vfiOm7M3X0uvXWW2XVceH1irbti6rbvLw8+T0X7vDz7LPPyqrztWvXWqwvfiPit2witiuOT+vWrWWVsInp34X3qTSiOlY0j5j87W9/k8dG5EPYs2ePbFZ54YUXZFW4yT333INWrVoV28+y/O9//0OHDh1kVX9RpmMuOuyJKvJvvvnGvOzw4cOyWlxURxPZG4NrOZhOaqaOTaLd5o8//pAnKhF8BdEe9vHHH6N58+Yy0Io2JXFCFH/M5WnTK49Tp07JtETbnki78CTamay1C9tDgwYNLN6LNlXB1O575swZ+Sra+FQQbaWiZ7YIBkWJwCC++6JtfGXtY2WI4GqNaN8VJ3PRzieClTgeog1SKHrsi+6Xad8K75e4ABC/o6KK5t90oVB0vrjwERchhS8kBNEuWfTiRlysRUREFJtXke+q6L6KbYj2XNPFaEn7KYjgWnQ/yyJ+X2X9tsTFhvi7XblypfztCCLQiuD+4IMPVmh7RJVRfbtjViOiZChOAt9++608aYpXUeIpXJJ4//338c477+Cpp56SQ11q1qwp/8BFqaKkjiiFT0bGmlBLoqRVmEhHBNbCV+OFiZN6ZZRUmhTbN108FGZtnmAtD1XFHvtYuIRa+EQvai/E72POnDkyUIngJkpt4mKr6LGvyu+upG07w/GsDFGinj17tgywjzzyiLw4Fh3KTBcPRPbE4FpOIpCK4ClKouKPVFyti163JqLXZe/evfH5558X69UoSrGlESUXa1VwRa/omzZtKqsCb7/9dqsn+soS27d2pyax/cLVsOUl9tNUDVfaeNryVhGLiwbRu1WMdyxK9PYVFzFFS1+OIsZB5+bmYvXq1RalUluq6Bs2bChrKYoqmn+xnml+4eMkqopFtbyjxjIX3VcRlEXvcdE7vOh+ih7QhYl5puUV+X2J31ZZROm2U6dO8mJUlNpjYmLwySefVGhbRJXFauFyMpVSJ02ahKioqGLtX+Lqv+iVvmgjLc9QA3GyEEGi8FARMYRHVDcWJu6uJEqTomRcVH5+fqVvZSi2L4a0iJOyyZo1a6wOpyiPzp07yypUMWSo6D4V/o5Md4cqa7/Fd9u/f385rrhwu3dCQoK80OnZs6dsY6yoigzFKW3fiuZLpCeG51SWGL4kjseuXbvM88Rvo2iNhQieopT8z3/+02L74gJP7INo03QEMZSo8F3MxIVmXFycHDYliKFLosZl4cKF8kLEZN26dTh27FiF9/OBBx6Qfx/i5i5FFf0bfOKJJ2R7vfgtihu9mPaJyN5Yci0nESx69OghT/BC0eAqqpumTJkiO/uI9Q4dOiRPhuUp+YmqZFGlKDrLPP3007LtVJyIxNjFwh11RCcZMaZVjNETAV4EHDG4XpQcRCAXY0hFZ5KKeuaZZ+QJUdxpSgRwUdX59ddfm0ugFSVKkmKc59ChQ+V9jMV3Ijq4iEAmxm+KsaKm6nZhzJgxMu+mzmPWiHGeGzdulIFUdIwRN5gQ40zFyXrWrFmV2k9xchb7JgJhZe9dK46BCHAir+LYiLbv//znPzKYiABTGePHj5e3phTHQ4zVFBch//73v2UJT9ScFC7Rv/HGG3jvvffkumJ8qSgJinGvolbFUR13RBOIOC7iuxQXPCKQiTZX0bFKEL/RmTNnyuXiNyyqaMV64vcqxqmK8aoV8frrr8vfq2g7FX874nckxpmL2gPxdyM6O5mIDm/i+xTH+vnnn+fNKMhxlPc/dmHz58+X3fuLjuk0DcURwynEsB0xfu/222/Xtm/fXmw4i7WhOMLXX38th26IoRUdO3aUQwmsjXMV/v3vf8txpmI7YiiKGGc4fvx4OTSlMkNxhI8++kgOqRDDQsS+79mzp8ShON99953FZ0vK059//inHsop9FGNUxTCIwuOFxZCdf/zjH3JMpxh7WPjnWHQojrBv3z45xCUgIECOre3du7d5rG5ZeTTtu2lsamWG4txzzz1Wl61evVrmTYzvFOM2Z86cKYfBFB2TWlIa1oZiHTx4UM4TaYrjIsZxfv7558XSNA29adWqlRx3LcZ2Pv/888WGQIm02rZtW+58ie2IoVKlMX2n3377rfbGG2/I4TbiNynSszaOdPny5XL8sfiN1axZU45FvXDhgsU65RmKYxpnLsb0iu9G/M3Ur19frpOUlFRsu4MHD5ZpFv2tENmTTvzPgbGciFyEuEOT6Gcgak0qU2PiKGLIjqhJsnYXMSJ7YZsrEbksUTUvxtGKtlciR2KbKxG5HNFbWnQI/O9//yvbWUV7OJEjseRKRC5HPIhAlFZFkP3qq6/kQwWIHIltrkRERIqx5EpERFQVba7iFm6XLl2SN+euzI3XiYioaolKSnGzD/FAg8IPelApJyfH4mY0thDjxws/6MElg6sIrFV1ezkiIlJH3HlN3A7SHoG1ceN6iI+/oiQ90U4u2sydNcCWK7iaHiclDkplbjNHRERVS9ztTRSSCj8eUCVRYhWB9dz5FdDr/WxKKy0tC40aPiTTdOngaqoKFoGVwZWIyHnZu2lPH+ADfYCNDxYp40lizoDjXImISB0RGA02BkcXCK7sLUxERKQYS65ERKQOS64SgysREakj7kuk2XhvIhe4txGrhYmIiBRjyZWIiNQxaAqqhZ2/5MrgSkRE6rDNVWK1MBERkWIsuRIRkTosuUoMrkREpA6Dq8TgSkRE6mgKgqtIw8mxzZWIiEgxllyJiEgZnWaQk61pODsGVyIiUodtrhKrhYmIiBRjyZWIiBTfoUmzPQ0nx+BKRETqsFpYYrUwERGRYiy5EhGROiy5SgyuRESk+HmuBtvTcHKsFiYiIlKMJVciIlKH1cISgysREanDoTgSgysREanDkqvENlciIiLFWHIlIiJ1+Mg5icGViIiU0RkMcrI1DWfHamEiIiLFWHIlIiLFN5HQbE/DyTG4EhGROuwtLLFamIiISDGWXImISB2WXCUGVyIiUod3aJIYXImISB2WXCW2uRIRESnGkisRESmuFjbYnoaTY3AlIiJ1OM5VYrUwERGRYiy5EhGROuzQJDG4EhGROqJK18BqYVYLExGRU9u6dSuGDh2K8PBw6HQ6rFy50mL56NGj5fzC08CBA+26Tyy5EhGRU1cLZ2ZmokOHDnjqqadw//33W11HBNMvv/zS/N7b2xv2xOBKREROHVwHDRokp9KIYBoWFgZHYbUwERFVS2lpaRZTbm5updPasmULQkJC0LJlSzz//PNITk6GPTG4EhGR+nsLG2ycAERERCAoKMg8zZgxo1K7JKqEFy9ejE2bNmHmzJn4/fffZUm3oKAA9sJqYSIiUkczGCdb0wAQGxsLvV5vczvpiBEjzP+OjIxE+/bt0bRpU1ma7du3L+yBJVciIqqWJVe9Xm8xqeqE1KRJE9SuXRunT5+GvTC4EhHRTeXChQuyzbVu3bp22warhYmIyKl7C2dkZFiUQqOjoxEVFYWaNWvK6b333sMDDzwgewufOXMG48ePR7NmzTBgwADYC4MrERE59cPS9+zZg969e5vfjx07Vr6OGjUKCxYswMGDB/HVV18hJSVF3miif//+mDp1ql3HulYouK7vPQV+7vYdeEvk6obsmg5Xs6bbW3A1rnicXFWvXr2glXLLxA0bNsDRWHIlIiJ1+DxXicGViIiculq4OmJvYSIiIsVYciUiIoUU3ERCpOHkGFyJiEgdVgtLrBYmIiJSjCVXIiJShyVXicGViIic+g5N1RGDKxERqcOSq8Q2VyIiIsVYciUiInVYcpUYXImISB22uUqsFiYiIlKMJVciIlJHPJ1Gs7Fa19bPVwMMrkREpA7bXCVWCxMRESnGkisREanDkqvE4EpEROqIJ+IYbOzta/NTdaoeq4WJiIgUY8mViIjUYbWwxOBKRETqiBpdg63BFU7PYcE1uH0DRE4YBv+IWsiIScahmauQcijW6roht7dE05F3ILBpGLT8AlzZfw5HPl6LnMQ08zqhd7VGm38MhE+IHqnHL+HA9B+ReT4JjuRqeXK1/LhqnlwRj5MLYcnVcW2unnpfdJ0zEudW7MCGvtNw/rsd6DZnJDwCfKyu7xHgjTOL/8CmobPw2/APcS0zF53fH2Fe7t+gNjpNeQhH5v6MDf2mI2nPWXT98HHo3B3XhOxqeXK1/LhqnlwRjxO5Iof82sJ6tUHO5TTErNoDw7UC+ZqbnC7nW3Npw0Ek/nUCBdl5KMi5huhlfyG4bYT5j6PeoI5I3nsWiX+egCEvH6c+3wyv4ADU7NjQEdlxyTy5Wn5cNU+uiMfJtWgGTcnk7BwSXPXNwpB2Ms5innivbx5Wrs/X6twY6ecuQyswWE1PzM+ITpTzHcXV8uRq+bG2D66QJ1fE4+Sitz/UbJycnEOCq7ufF/LTcyzmXUvPgYefd5mf1beoi5Z/74ejH681z/Pw85KfL5qeu3/Z6anianlytfy4ap5cEY8TuSK7dGiqN6ADIt+4V/47Oz4FSbvOyHYViw0H+CAvJbPUdAKbhqLbvFE4PPsnmYZJflZesfYYzwBvFGTmwl5cLU+ulh9XzZMr4nFycezQZL/genHDATmZRAzrgsYjehS74oxe+lepfzi3fvoUjs3fgIvrb6QlpJ2OR1CLuub3oq0loHEI0s4kwF5cLU+ulh9XzZMr4nFycQyujqsWjt9yFD4hQfKPSOfhLl99agcifssRq+sHNAmRfzgnFm7EhTX7ii2/uC4KtW5pgpAeLeDm6Y7mT/VCXmqW7JLvKK6WJ1fLj6vmyRXxOJEr0mla2S3HaWlpCAoKwvLOr8HPvXLtFsEdGiJyvHEcW2ZsEg59sBpXD8XIZT6hQei1/GVseXgechJS0eGd+1H/nk6yJ2BhpuWC6EnY+qUB8o8y9cQlHJj2g+PHULpYnlwtP9U1T0N2TYerWdPtLZs+z+Nkf6bzeGpqKvR6vd3Svzr7Seh9vWxLKzsPwa9/abd9dangSkSuedJWEVyrI1c7Tg4LrjNHqwmuExY5dXDlqGoiIiLFeG9hIiJSRlSGajZ2SCpHhWq1x+BKRETqsLewxOBKRETqMLhKbHMlIiKntnXrVgwdOhTh4eHQ6XRYuXJlsWrmSZMmoW7duvD19UW/fv1w6tQpu+4TgysREakvuRpsnCogMzMTHTp0wPz5860unzVrFv75z39i4cKF2LlzJ/z9/TFgwADk5FjeJlMlVgsTEZE6Km68r1Xs84MGDZKT9aQ0zJ07F2+//Tbuvdd4283FixcjNDRUlnBHjLjxuEKVWHIlIqJqKS0tzWLKza34/aGjo6MRHx8vq4JNxHjc7t27Y/v27bAXBlciIlJGM6iZhIiICBkITdOMGTMqvD8isAqipFqYeG9aZg+sFiYiomrZWzg2NtbiDk3e3s5zh0CWXImIqFrS6/UWU2WCa1hYmHxNSLB8KpJ4b1pmDwyuRETk1L2FS9O4cWMZRDdt2mSeJ9pvRa/h2267DfbCamEiIlKmcJtpZVX08xkZGTh9+rRFJ6aoqCjUrFkTDRo0wCuvvIJp06ahefPmMti+8847ckzs8OHDYS8MrkRE5NT27NmD3r17m9+PHTtWvo4aNQqLFi3C+PHj5VjY5557DikpKejZsyfWr18PHx8fu+0TgysREamjKajWreA41169epV6s39x16YpU6bIyVEYXImISB1RpWtQkIaTY3AlIiJlxOPmNFsfOccb9xMREVFRLLkSEZE6rBaWGFyJiEgdUaOrKUjDybFamIiIqCpLrgM3T7K4z6OzW9PtLbiaIbumw5W44jFa7YJ54lU6mbBDkxGrhYmISB22uUq84CQiIlKMJVciInLqewtXRwyuRESkDquFJVYLExERKcaSKxERKcNqYSMGVyIiUkeMojEoSMPJMbgSEZEy4slvmmOfOFctsc2ViIhIMZZciYhIGba5GjG4EhGROhyKI7FamIiISDGWXImISBlWCxsxuBIRkTLsLWzEamEiIiLFWHIlIiJ1DDrjZGsaTo7BlYiIlGGbqxGrhYmIiBRjyZWIiJTRNJ2cbE3D2TG4EhGRMqwWNmJwJSIitUNxDLan4ewYXG0Q3L4BIicMg39ELWTEJOPQzFVIORRrdd2Q21ui6cg7ENg0DFp+Aa7sP4cjH69FTmKaeZ3Qu1qjzT8GwidEj9Tjl3Bg+o/IPJ/kwBy5HpXHKLBJCFq/Mhg1WoXDq4Y/1veZivyMHAfnCKhZKE+ZMck4OHMVrpaQp8IaDu+KDm8Ox+E5a3F22TY5L+S2FmjzjwHwCQmSZzTxuzs892ekn0mAI7nicaKbGzs0VZKn3hdd54zEuRU7sKHvNJz/bge6zRkJjwAfq+t7BHjjzOI/sGnoLPw2/ENcy8xF5/dHmJf7N6iNTlMewpG5P2NDv+lI2nMWXT98HDp3HqLqcowM+QbE/XoIUVP+h6rMk8hD9IodWN93GqK/24HupeTJxLt2IJo+3hNpp+It5qeejMP2fyzC+n7TsGHgDCT8dQLdZj0GR3LF43QzM7W5ajZOzo5n7koK69UGOZfTELNqDwzXCuRrbnK6nG/NpQ0HkfjXCRRk56Eg5xqil/2F4LYR5uBZb1BHJO89i8Q/T8CQl49Tn2+GV3AAanZs6OCcuQ7VxygzJgmxq/c6vFRXWF0recpJTpfzS9N+/DCc/GIz8tKyLOaL70NMJprBAN+6NRx6UeeKx+mmZtBBs3HiONebmL5ZGNJOxlnME+/1zcPK9flanRsj/dxlaAUGq+mJ+RnRiXJ+8t5oxXt/c1B9jJw1T3X7tIWHvzcu/ByFBkO7FFvuGxqEXkv/AQ8/b0AHnPzyd4fm2RWPExGDayW5+3khP92yHedaeo7xBFUGfYu6aPn3ftj7xrfmeR5+XvLzRdNz9y87PXLMMaouebpWgTx5BvqgzZiB2PGPRSWmmZ2QinV9p8m0I+7pjJyEVDiSKx6nmxnvLWzE4FpO9QZ0QOQb98p/Z8enIGnXGdlWVJhoI8pLySw1ncCmoeg2bxQOz/5JpmGSn5VXrI3JM8AbBZm5SvPhyux9jKoqTx2u5ymrhDx5Bvggt4Q8tRkzCDGr9yIzNrnMbRVk5eHc9zsx8Jc3sXXUv5B16SrswRWPE93Aca5GDK7ldHHDATmZRAzrgsYjehS7io5e+lepJ4NbP30Kx+ZvwMX1N9IS0k7HI6hFXfN70X4U0DgEaWw3qjbHqDrkqcGwLmhiJU9nSshTna5NZZWw6TMiENdoXU+25e+ZaKW0pwPcvD1ku6u9gqsrHieiotihqZLitxyVwxfEiUHn4S5ffWoHIn7LEavrBzQJkSeDEws34sKafcWWX1wXhVq3NEFIjxZw83RH86d6IS81Sw4zoOpxjAQ3Lw+4eRqvSd283OV7R4q7nqcG1/PUoIw8/fH0Qmx57BP8/vincko5dhGnv/4TB2esksvD746Ef/2agE4nS4uRY4egIPuaHJLjKK54nG5mtnZm0kydmipg8uTJ0Ol0FlOrVq1QlfiLq6RradnY/doSRI4fhnbjhiIzNgm7xy4xt4f5iE4iy1/GlofnyTaspo/1hFewH9q8OlhOJqbloodj1Lvfoe3Ye+SJJvXEJZk+O2lUn2MkSnN9V71unt9//ZvyddO9s5Edl+KwPO16bYns/Rs5bigyYpOws1CeROek3stfxuaH58m21NzkDIvPi57oYsynuHAT/OoGo/WL/eEdHCB73149egHbX/oS+Q5sjnDF43Qzq6o217Zt2+LXX381v/fwqNrwptO0srORlpaGoKAgpKamQq/Xw1Ws6fYWXM2QXdPhSlzxGLni5ZIrVoG52t+Svc/jpvRPDXscgZ5eNqWVfi0PzVd/Xe59FSXXlStXIioqCtWFK/5NEBGRC9xEIi0tzWLKzS25RuXUqVMIDw9HkyZN8NhjjyEmJgZVicGViIiUMRh0SiYhIiJCloZN04wZM6xus3v37li0aBHWr1+PBQsWIDo6GnfccQfS02/cIMXR2OZKRETVss01NjbWolrY29v62OdBgwaZ/92+fXsZbBs2bIgVK1bg6aefRlVgcCUiompJr9dXqn24Ro0aaNGiBU6fPo2qwmphIiJyqRv3Z2Rk4MyZM6hb98a9AxyNwZWIiJw6uI4bNw6///47zp07h23btuG+++6Du7s7HnnkEVQVVgsTEZFTu3DhggykycnJqFOnDnr27IkdO3bIf1cVBlciIlLGoOnkZGsaFbFs2TJUNwyuRESkTGVuX1iUrZ+vDtjmSkREpBhLrkREpAyf52rE4EpERMoYoKDNVTz70MmxWpiIiEgxllyJiEgZFTeB0Gz8fHXA4EpERMqIwGhgcGVwJSIidVhyNWKbKxERkWIsuRIRkTKG65MtbP18dcDgSkREyrBa2IjVwkRERIqx5EpENhuya3pV7wJVEwat4jfet5aGs2NwJSIiZVgtbMRqYSIiIsVYciUiIsXVwrA5DWfH4EpERMqwWtiI1cJERESKseRKRERqHzkHPnKOwZWIiJThw9KNGFyJiEgZMcbVYPM4V+cvubLNlYiISDGWXImISBlNQZurxjZXIiKiG9jmasRqYSIiIsVYciUiImXYocmIwZWIiJQR7aUa21xZLUxERKQaS65ERKQMb9xvxOBKRETKsM3ViNXCREREirHkSkREyrBDkxGDKxERKcM2VyMGVyIiUoYlVyO2uRIRESnGkqsNgts3QOSEYfCPqIWMmGQcmrkKKYdira4bcntLNB15BwKbhkHLL8CV/edw5OO1yElMk8sDm4Sg9SuDUaNVOLxq+GN9n6nIz8hxcI5cj8pjJDR7shcaDL8FnoG+yLp4Bcc+3YCknacdmCOgZqE8ZcYk4+DMVbhaQp4Kazi8Kzq8ORyH56zF2WXbzPN9QvRo9+pg1OnWTL6/euQCdoxZZNc8kOtitbARS66V5Kn3Rdc5I3FuxQ5s6DsN57/bgW5zRsIjwMfq+h4B3jiz+A9sGjoLvw3/ENcyc9H5/RHm5YZ8A+J+PYSoKf9zYC5cm+pjFHpXazR5rCd2j12CDX2m4uzSv3DLrMfkdhyZJ5GH6BU7sL7vNER/twPdS8mTiXftQDR9vCfSTsVbzHf38USPfz2N1FPx+GXoLKzv/z6OL9ho51zQzTAUx2DjVFHz589Ho0aN4OPjg+7du2PXrl2oSgyulRTWqw1yLqchZtUeGK4VyNfc5HQ535pLGw4i8a8TKMjOQ0HONUQv+wvBbSOgczcegsyYJMSu3ov0MwkOzonrUn2M/OrVROrRC+ZjdHFdFNw83OR8R6lrJU85yelyfmnajx+Gk19sRl5alsX8iCGdkZeahVNfbEFBVh60AgNSjl20cy6I1Fq+fDnGjh2Ld999F/v27UOHDh0wYMAAJCYmoqowuFaSvlkY0k7GWcwT7/XNw8r1+VqdGyP93GV5MiPnOEZxGw/Bu1Yg9C3qAm461B/SGdmJaQ69IKpMnur2aQsPf29c+DnKah5zElPRfe4oDNz4Fu786gWE9Ghhl32nm4OmaBLS0tIsptzcXFgzZ84cPPvss3jyySfRpk0bLFy4EH5+fvjiiy9QVRhcK8ndzwv56ZZtotfSc+Dh513mZ8XJueXf++Hox2vtuIek+hjlXsmQJds7vnoBg/98D23H3oOD7/8IQ14+HJknkYfy5skz0AdtxgzEwQ9WWV3upfdF3V5tcf7HXdgwcIYs3d7ywSPwr++40ji54MPSNdsmU2/hiIgIBAUFmacZM2YU215eXh727t2Lfv36mee5ubnJ99u3b0dVYYemcqo3oAMi37hX/js7PgVJu84Ua2sT7V55KZmlphPYNBTd5o3C4dk/yTTIeY5R82f6IOT2Ftj8t4+RdekqanVqhC4fPIIdL36JtFNxdstTh+t5yiohT54BPsgtIU9txgxCzOq9yIxNtro8PzsPVw7FIP73Y/K9eE09fgl1ujdH5oWdyvNDVBGxsbHQ6/Xm997exS8ik5KSUFBQgNDQUIv54v3x48dRVRhcy+nihgNyMokY1gWNR/QoVtqJXvpXqSftWz99Csfmb8DF9TfSIuc4RkEt6+LSpsOyl7CQvC9adhCq3a2p3YJr0Tw1GNYFTazk6UwJearTtamsEjZ9RgTiGq3roWbHhtgz8VuknYxH7a5N7LLvdHMSjSgGBWkIIrAWDq7OhNXClRS/5Sh8QoLkCVzn4S5ffWoHIn7LEavrBzQJkSftEws34sKafVbXcfPygJun8XrHzctdvqfqc4zEcJe6fdrBN6yGeZhPjTb1i7WB2lPc9Tw1uJ6nBmXk6Y+nF2LLY5/g98c/lZPorHT66z9xcIaxmjj25/0IahmO0J4tAZ1Ovor3iTtOOSxP5Fo0Ua2r2T6VV+3ateHu7o6EBMu+D+J9WFj5+lfYA8/elXQtLRu7X1uCyPHD0G7cUGTGJskhGqb2MJ/QIPRa/jK2PDwPOQmpaPpYT3gF+6HNq4PlZGJa7lu3Bvquet08v//6N+XrpntnIzsupQpy6PxUH6Mzi7fKKtke/3kWngG+spfu8QW/IGn3GYfmaddrS2Tv38hxQ5ERm4SdhfLkGxqE3stfxuaH5yE7IRW5yRkWnxftw2L8tOghLIhSuCjBtn15ELpMexiZF65g94Sl5tI5UXXn5eWFLl26YNOmTRg+fLicZzAY5PuXXnqpyvZLp2lamcN1RS8t0ZicmprqtEV0a9Z0ewuuZsiu6XAlrniMXLF/+DAX+925Inufx03pL+08Dn7uZXcaLE1WQS4e3fdhufdVDMUZNWoUPvvsM3Tr1g1z587FihUrZJtr0bZYR2HJlYiInPoOTQ8//DAuX76MSZMmIT4+Hh07dsT69eurLLAKDK5EROT0N+5/6aWXqrQauCh2aCIiIlKMJVciIlKGN+43YnAlIiJl+DxXI1YLExERKcaSKxERKcNqYSMGVyIiUobB1YjVwkRERIqx5EpERMqwQ5MRgysRESkjbqhrsLFat+yb8lZ/rBYmIiJSjCVXIiKqls9zdWYMrkREpExFn8dqja2frw4YXImISBmWXI3Y5kpERKQYS65ERKQMbyJhxOBKRETKiLioKUjD2bFamIiISDGWXImISHG1sM7mNJzdTR1cXaFHWlGru70FVzJs13S4mjebToGrOdzS9fI0/uidcCX5BZkO2Q6rhY1YLUxERKTYTV1yJSIitdhb2IjBlYiIlOFNJIxYLUxERKQYS65ERKSMeFycxkfOMbgSEZE64kHnBj4sncGViIjUYcnViG2uREREirHkSkREyrC3sBGDKxERKcNxrkasFiYiIlKMJVciIlKG9xY2YnAlIiJlWC1sxGphIiIixVhyJSIiZTjO1YjBlYiIlOFQHCNWCxMR0U2jUaNG0Ol0FtMHH3ygfDssuRIR0U3VoWnKlCl49tlnze8DAwOVb4PBlYiIquVQnLS0NIv53t7ecrKVCKZhYWGwJ1YLExGR8pKrwcZJiIiIQFBQkHmaMWOGkn0U1cC1atVCp06dMHv2bOTn50M1llyJiKhaio2NhV6vN79XUWodM2YMOnfujJo1a2Lbtm144403EBcXhzlz5kAlBlcb1GzfAJEThsE/ohYyY5JxcOYqXD0Ua3XdoJbh6PDmcPiFB0PnpkN6dCKOzv8FV/afk8trdW6M2xc+g/ysXPNnYtfsx6EPf3LK/LSfeC/qD+xw4wNuOnj4eOH3J+Yj9cQlR2XJ5dSoF4TxW19Gbmaeed7ZHeew5LllJX7mloc64c7neiCwTgDSEtLx2ydbceCnw3JZeNsw3Pf+EATXNx7HxNOXsWHWJpzbHYOq0PGhzhg8dQg2vr8Bu7/aWeJ6gaGB6PfmADTu0US+v3TgIpY98415eaeHO6PH/90B3xq+iNl1Hmvf/gmZlzPssMc6uOmaQ6cLBuAJIA8GLQaaFn99uTvcdC2g09WSfWAN2kVo2vlS0itr/Yqm53jiWayaoue5isBaOLiWZOLEiZg5c2ap6xw7dgytWrXC2LFjzfPat28PLy8v/P3vf5elYhXB24TBtZI89b7oNmckjn6yHhd+3o/6gzuh+5yR+PW+j5CfkVNs/az4q9g9YSmy41Pk+7q92uDWOSOxfuD7MOQaqySupWdjXd9pcIX8HPxglZxMmj56Oxre15WBVZGZt3+MnPQbF2IlqdsmDMPeG4xFT34jg3DTHo0x8r+PIO5YPBJPJ+HqxVR88/x3SLmUKtdv278VRv33EUzv9hHyr/8uHSUgJAC3Pn0bEk8klLqep68nHls8EodWHsTaN1fjWs41hLWpa17e8NZG6D2uH5Y9/Q0un0pE/3cG4t4P78PSUUvssNciCOShwHAAgPg70cPdLRIGLRcarsrAC50nCgw7ZPB1d+sAA3KgadbzWNb6FU2vKmgKOiRpFVz/tddew+jRo0tdp0kT44VYUd27d5fVwufOnUPLli2hCttcK0kEk5zLaYhZtQeGawXyNSc5Xc635lpqtjkQQaeDZtDg4e8Nn1rqe6lVx/w0GHYLYn7aa88skBU169dAysUUGViFM9uikRqXipBmdeT77JRsc2DV6QCDwQDvAG9ZynW0AZMG489//SH3qTTt7+uArKvZ+GvBH8jLzINWoCHu0I2Ltvb3d8Th1Qdx6eBFXMu+hi0f/YYGXRuiRv0adthrUXoU363pAjQNGlKg0wXJ06tOFwKDIRqAuFDJliVNN92NCwFLZa1f0fRuHnXq1JGl0tImUUK1JioqCm5ubggJCVG6Tyy5VpK+WRjSTsZZzBPv9c1L74E2aNPbcPf1gpuHO2LX7kPWpavmZWJ+/7UTZKBK3h+No59skAHPWfNjEhwZAf8GtRC7Zp/y/b5Zvbzuebh5uOHCgYtYP/NXXD6bbHW9k3+cQa8Xe6LZ7U1wZttZNOvZFL6BPji3x7La95394+Hl5wV3Dzfs++EArl64fuHkIK0GtJZB/fCqg+jwQMdS123QrSHSE9Lw8H8eRXj7eki5cBVb523Bma2n5fKQliHYs2S3ef3M5ExkJmWgTssQpNg9X27QQQ+DlgjADzqdKL8Uqo7WMgBdgxI+W9b6FU2valTnoTjbt2/Hzp070bt3b9ljWLx/9dVX8fjjjyM4WFTtq8PgWknufl64lm5ZXSree/iVXmcvqn3dvD0Q3rst3LxFG41RxvnL+P3xT5F+7jK8g/3R9pXB6PbRE9g66l8OuReY6vwU1uDeW5Dw5wnkXslUus83o6yrWfjXff/FpaPx8PL1RO+X7sSTix/HvIELkJtxox3WRJTcolYewhP/flheAGkFBvxv4mpkJFkei6mdZsHD2wPtBraWr47ko/dBn/H98O1T35Rv/SBfNOzeCD/84zt89/wyNLurOe7/54P477CFuBpzVV4k5Bb5Leek5cDLX117WkncdC2hIQsaLoueCdC0AotKTk2WOEv6ft3LWL+s5dVDdX4qjre3N5YtW4bJkycjNzcXjRs3lsG1cDusKtXrqFRj9QZ0QIc37pX/zopPQdKuM7KdsjDPAB/kppQdQESb5IX1B9Br2RhknLuMKwfOIzc5Q06CeD3w/koM/u0dBDSohYzzSU6XHxNRqq3XNxJ731muPA83gw7D2mH4tCHy36J6d96ghbhw0FgFKtpc183YiI73RqJB5wic2nqm2Oe7PNgRPZ+5DQse+AIJJxIQ2jIUI/87AjlpuTix5ZTFuqKNNWrVIby87v9w+UwSzu+13pnNVm2HtsOg94x5Sr2UgotRF3Dg+yhcPX+lXJ/Py8rDxf0XcHLTCflevMYfiUPjnk1xdekeudw70DKQegf6IC+z7DZqWxg7Nvleb38VRCB0u94uawwXOnnKLaktu6z1K5oeFSV6Ce/YIdqr7Y/BtZwubjggJ5MGw7qgyYgeFuvoW9TFmaV/lTtNUZIQPXMLByMzO5dWHZWfev3b41pmLhK2nVS05zeXA6sPy6k0Wim/lfA2YTj5+2nEHzd2eBGvp/84ixZ3NS0WXE3cPd1Rq1FNuwXXIz8dlpPJC5vGyCrhrqO6y/fi33XbhSOiSwP8MOa7Yp9PPJ6ARrc2LjH9xBOJCG11oznDr6YfAuoE4PIJUVVrz8Cqvx5YRRAUsq4HQf8bVbk60ZZd0gVrWetXNL2qUZ2rhR2JHZoqKW7LUfiEBMmgpPNwl68+tQMRv+WI1fVDe7aEvlkodO5ucPf2RPPRd8EnRI9k01CcLo3lsBbBM8hXDmVJP5uIjNhkp8yPiUhHtMW6xF9LNVC/Qz3UaVpbDpvx8vPEgPF95fk2Zt8Fq+vH7L+A5nc2RUhzYwcm8dr8jqayWllo2bs5wlqGwM1dB08fD9z1fE/ow/QOHYrz1cOfyyrdz+/9TE5xh+Ow4/NtWPfuGqvri17CoW3C0KyX6DkL+Sren/3DWHI/+EMU2g6LRN3IcHj4eKDX2D6I2X3ebu2txsAadD2wFi5FGqBpiXBzExcC7gB84aarB4Nm2beh/OtXNL2qoSn6z9mx5FpJ19Kyseu1JWg/fhgixw1FRmwSdo5dYm639A0NQu/lL2Pzw/OQnZAKryA/tH15EHzq6FGQl4/00wnY+epiZF28Yh432nny3+Cp90N+Zi6S9p7FzrGLHRaUVOdHCGhcB8Ft62PfOysckoebQc0GNXD3q71lb17Rnhp74CK+HP01cjOMVZ5BdfV4ZcMLmDvgX0iNS5Ol3hrhQRj57xHwr+WPrJQs7P0+Cnu/i5Lr+9f0w+A374Y+VC+rhRNOJmLxM9/iSkzxjmn2klmk/Vf8nkR+sq8aew2LEqzovPRhZ+PN1VNir+KHl79Hvwl3Y/icB3A15oos4Yr5wvkd57Blzm944NOHZHuuCKyrxv1op733hptbPWiaAe5ut5nniqExBu0kDNopuKHF9WWmcak3hs24uUVC01KhacaLmbLWL2s5VR86rbQ6pUL3dxS3nkpNTS3XgF5nsbrbW1W9C1SGYbumw9W82XQKXE2AC16mjz96J1xJWlomatUcYrfzuClOvNzwDXi7+diUVq4hB/POz3DqmOOCfxJERFRVqnNvYUdicCUiImXYocmIHZqIiIgUY8mViIiUEb14NFvvLewCJVcGVyIiUsZwfbKFrZ+vDlgtTEREpBhLrkREpAw7NBkxuBIRkToK2lzhAsGV1cJERESKseRKRETKsEOTEYMrEREpw6E4RqwWJiIiUowlVyIiUobVwkYMrkREpIx40JpmY72urZ+vDhhciYhIGY5zNWKbKxERkWIsuRIRkTJ8nqsRgysRESnDamEjVgsTEREpxpIrEREpw5KrEYMrEREpbnPVbE7D2TG4upg8g2vV9Ot0rvcTHddoElxNkKcrnA4tebj3givxcE+r6l24qbjemYuIiKoMq4WNGFyJiEgZ3rjfiMGViIiUEe2tBpvbXJ0/urpWAx0REVE1wJIrEREpw2phIwZXIiJSho+cM2K1MBERkWIsuRIRkTJ8nqsRgysRESnDca5GrBYmIqKbxvTp09GjRw/4+fmhRo0aVteJiYnBPffcI9cJCQnB66+/jvz8/ApthyVXIiJSxqBgnKvBjuNc8/Ly8OCDD+K2227D559/Xmx5QUGBDKxhYWHYtm0b4uLiMHLkSHh6euL9998v93YYXImISO2N+zXb07CX9957T74uWrTI6vJffvkFR48exa+//orQ0FB07NgRU6dOxYQJEzB58mR4eXmVazusFiYiomopLS3NYsrNzbX7Nrdv347IyEgZWE0GDBggt3/kyJFyp8PgSkREyquFDTZOQkREBIKCgszTjBkz7L7/8fHxFoFVML0Xy8qL1cJERKT2Dk2wPQ0hNjYWer3ePN/b29vq+hMnTsTMmTNLTfPYsWNo1aoVHIXBlYiIqmWHJr1ebxFcS/Laa69h9OjRpa7TpEmTcm1bdGTatWuXxbyEhATzsvJicCUiIqdWp04dOakgehGL4TqJiYlyGI6wceNGGeTbtGlT7nQYXImISBmDpqDkasc7NIkxrFeuXJGvYthNVFSUnN+sWTMEBASgf//+Mog+8cQTmDVrlmxnffvtt/Hiiy+WWC1tDYMrEREpI57FqlXj57lOmjQJX331lfl9p06d5OvmzZvRq1cvuLu7Y82aNXj++edlKdbf3x+jRo3ClClTKrQdBlciIrppLFq0qMQxriYNGzbEzz//bNN2GFyJiEgZTcEj4zQ4PwZXG9Rs3wCRE4bBP6IWMmOScXDmKlw9FGt13aCW4ejw5nD4hQdD56ZDenQijs7/BVf2n5PL20+8F/UHdrjxATcdPHy88PsT85F64pLd8+JTKwCd37oXwa3D4VtHj42PzkfqydLHdIXf1RqRLw+Ab0ggUo7HYe/UlUg/n1Tu5aoNHjwYEya8jsjIdrh27Rq2bv0Dr7wyFhcvXjSvc++9wzB79kzUq1cP+/btxzPPPIcTJ06UmGZZ61c0PRV8Ar0x5K270e7uVnD3dMfl6GQseHgRruWUfu/TgeP6oO+LPbHoueU4svHGPt76SGf0efEO+Af74syO8/hu4k9Iv5wBR6ndOgy9Jw+Fvn4wdDodrpy9jO0f/4q4vTFW16/ZrA5uf70/6rQJh2+wH/5z2wfIS7e8uUBwk9roOWEAwjpGwJBvwNlNx7F50moH5ejmVt1vf+govIlEJXnqfdFtzkhEr9iB9X2nIfq7Heg+ZyQ8Anysrp8VfxW7JyzF+runY13faTjz9Z+4dc5IuHkbr28OfrAKP/eaYp5OLPwVGecvOySwmh7xFL/tFLaNW1qu9QMa1ka3aX/DwTk/Y3WfGUjcfRY95jwGnbtbuZbbQ1CQHjNnzkZERCM0btxM3lFlxYpl5uUtWrTAN98swauvjkPNmnXw22+bsWrVD7KNxZqy1q9oeirodMBTnz8CwzUDZvb5FJM6zMT3b6xBQX7pZYW6rUPRpm9zpCakW8xvelsjDJ7YD0te+h6Tb/kI6UkZeHTufXCk9EupWP/KCnx++yz8t8dMRC3ahiH/ehTu1/82ihLB8vSGo9j09kqry/3qBGD4F6PkOl/c+SG+7PURDn1rObSCyN4YXCupbq82yLmchphVe2C4ViBfc5LT5XxrrqVmIzs+xfhGp4Nm0ODh7w2fWoFW128w7BbE/LQXjpJ7JRNnv9+Fq0dulPJK03BQB1zeE424P0/CkJePY//dAu9gf9Tu2LBcy+3h22+XyXaSzMxMZGVlYe7cf6J7927mYPf4449h8+YtWLt2rbyN2tSp02RX+zvuuMNqemWtX9H0VGjZqzlqhAdh5eR1yE7NkYPtLx2NlwGnJKKm5MEZQ7By8noUXCuwWNb1wY7Yt/IgYqMu4lr2Nayb/RuadG+ImhHWnxZiD7mp2UiPS72+s4BWoMHL3xt+tQOsrp9yLhnHftiPK6cSrS7vOPI2XNgZLdcpyM2Xf59Jx8p/Zx1S8zxXzcbJ2bFauJL0zcKQdjLOYp54r29e+iDjQZvehruvF9w83BG7dh+yLl0ttk5wZAT8G9RC7Jp9qK6CmocipVD+tQID0qIT5fzLe6PLXO4Id911p7wri+huL7RvH4moqAPm5eIRUkePHpPzt2zZUuzzZa1f0fRUaNq9IZLPX8GIOfehxR1NZPXtls+2Ye8PB0v8zJ1P34q444k4u/N8sWV1W4Xgr692m99nJGXKNMX8K7HXLwYd5JltE+DpJ/423HB8VRTSL1Zu++G3NETS8Xjcv+RJBDeujStnLmPbhxuRcKh8F45kG1YLGzG4VpK7nxeupedYzBPvPfxKHwclqoRFVXB477Zw8/a0uk6De29Bwp8nZGmyuvLwLSH//t7lWm5vxidZvIcHHxxhnifGsKWkWJ6wxfvAQOu1B2WtX9H0VPCt4YNmPRrjx3fXYfm4lYhoXw9PL3oUVy6kIHpX8TZKUQLtMbIr5g75t9X0vP28kJ1meZzEe28HHafCRJWwqApuenfrEquEy8MnyBfNB7fDT//3DRIPXUTbB7vgnvmP4JshnyK3SF6J7IXBtZzqDeiADm/cK/+dFZ+CpF1nZLtrYZ4BPshNKTsgGnLzcWH9AfRaNgYZ5y7jyoEbJQpRqq3XNxJ731kOe4oY2B5d3hwm/50Zl4qND39Soc/nZ+fJ/BYm3udn5pZruQqPPvoIPvtsgfz3+fPn0a6dsUNYu3btsG7dGrz00hj52CiTjIwMefPvwsT79HTLdsjyrl/R9Cqj073t8MD0IfLfVy+m4NSfZ5FyKRXbFhtLm+f2xsrOSW36tLAaXP/2/hBs+GizrEK2JjcrT3aQKswn0Ae5Co9TUS3uiUSvd415Sr+Ugm+HG4+hIKpxT645hEdWPo+Us0mI22+9g2BprmXlIf5ALOKvf/bQt7vR6enbEdahPs7/cVphTsgallyNGFzL6eKGA3IyaTCsC5qM6GGxjr5FXZxZ+le50xRVw6KnceHgWq9/e1zLzEXCtpOwp9j1B+VUWamnEhDU4kYVuOiopG9cB6mnE8q1XIWlS7+VU2EisP766wZMnPgmvvnGsnPWwYOH0LHjjR7ZHh4eaNOmNQ4dOmw1/bLWr2h6lbF/1WE5mdzytw6IHNi63J9v3rMJwtuEYdg7A+R73yAfjPhoOHat2I+fpv0iq4vFchP/Wn7QhwTI+fZycu0hOZX1txHUsFalgmvSiXjZzkxVw/RcG1vY+vnqgB2aKiluy1H4hATJIKvzcJevPrUDEb/F+vP+Qnu2hL5ZqAwy7t6eaD76LviE6JF8fSiOiUhHtMXC4PgrNzcvDznJf3u6G/8tuqdacX7dAYR0bYKw25vLdVs/fRdyU7KQtP98uZbbg7hlmQisb789CYsW3bgDi8nXX3+DPn16Y9CgQfKBx2+99SaSkpKwdetWq+mVtX5F01Ph8Ibj8PD2wK2PdpEBJKJjPbTt1xJHfrU+/GfabR/j43s+M09pCelYPW0Dfv2ncR93fxeFzsMjEdEhHJ4+Hhj0eh/ZNuvI9taGdzVHrRYh0LmL4Wce6PJsT/iH6nFpb8m/FXcvd7hf/62KV/He5Oj3+9C4dyuERtaT31Hbh7rA3dMDcVEVD9RUtY+cc2YsuVbStbRs7HptCdqPH4bIcUOREZuEnWOXmNsZfUOD0Hv5y9j88DxkJ6TCK8gPbV8eBJ86ehTk5SP9dAJ2vroYWRevmNMMaFwHwW3rY987K6okT/dve9f8775f/Z98/f3vn+Py3nOyl2/Pfz6BlXdOk/Mzzidh1zvfo8Nr98AvRI+rJy5h29hvZMel8iy3h3Hjxsqbd3/88UdyMmnTJlI+uurkyZN4/PGRmDdvDurXry/HpQ4bdp+5w1PPnj1ldXJgoLGnbFnrl7XcHnLSc/HF09/ivvcGYehbdyMlPg0/vvszzu0xBo7GXRvg6S8fxdvtPpDvU+Mtq6gNBg1ZV7PN7axntp/Dulm/YeSCh+AX5IMzO89j6Ss/wpF8a/jh9nH9ERCqR35uPpJPJWDtC0uRFmvs7Fe3cwMMXfgY/t3N+CzPwPAgjPzlFfPnn/p9nHxd3H+uHNYjSrtb31+H/rMfgE+wH5JPJWLti0uLjYUlsiedVo4+z2K8oGhLSk1NLdfjf5zF6m5vwdXkGVyrMuLBvfZ/OLKjjWs0Ca6mkb/zlzSKevHwjYtNV2Dv87gp/e765+Ghs61DXL6Wi51pC5w65rDkSkREyhiu/2cLWz9fHbhWMYeIiKgaYMmViIiU0XQaNJ2tvYWdv5mBwZWIiJTRFPT21VwguLJamIiISDGWXImISBnRGUnHDk0MrkREpA7v0GTEamEiIiLFWHIlIiJlDDoDdDb2Fma1MBERUSFsczVicCUiImUYXI3Y5kpERKQYS65ERKQMewsbMbgSEZEyBhRAhwKb03B2rBYmIiJSjCVXIiJSRtwXWLO5Wtj57y3M4EpERMpwnKsRq4WJiIgUY8mViIgUd2hyszkNZ8fgSkRECtk+FEek4exYLUxERKTYTV1yHbZrelXvApVBw9Sq3gUiqgCDJqp03RSk4dxu6uBKRERq8Q5NRgyuRESkjIYCaDaWXEUazo5trkREdNOYPn06evToAT8/P9SoUcPqOjqdrti0bNmyCm2HJVciIlLGeAMIg4I07CMvLw8PPvggbrvtNnz++eclrvfll19i4MCB5vclBeKSMLgSEdFNc/vD9957T74uWrSo1PVEMA0LC6v0dlgtTERE1VJaWprFlJub67Btv/jii6hduza6deuGL774AppWsYDPkisRESmjaaJDk87mNISIiAiL+e+++y4mT54Me5syZQr69Okj22V/+eUXvPDCC8jIyMCYMWPKnQaDKxERVcs219jYWOj1evN8b29vq+tPnDgRM2fOLDXNY8eOoVWrVuXa/jvvvGP+d6dOnZCZmYnZs2czuBIRkfPT6/UWwbUkr732GkaPHl3qOk2aNKn0fnTv3h1Tp06V1dIlBfiiGFyJiEjxOFedzWlURJ06deRkL1FRUQgODi53YBUYXImISBlNU3CHJs1+Q3FiYmJw5coV+VpQUCADp9CsWTMEBATgp59+QkJCAm699Vb4+Phg48aNeP/99zFu3LgKbYfBlYiIbhqTJk3CV199ZdGmKmzevBm9evWCp6cn5s+fj1dffVX2EBZBd86cOXj22WcrtB2dVo7+xaILdFBQEFJTU8tV/01ERNWLvc/jpvRr67vDTWdbuc2g5SMpbadTxxyWXImIqFoOxXFmDK5ERHTT3KHJUXiHJiIiIsVYciUiIsW9hXU2p+HsGFyJiEgh0eZqexrOjtXCREREirHkSkREyhirdHUK0nBuDK5ERKQMg6sRq4WJiIgUY8mViIiUEY+L09l8437nL7kyuBIRkTKsFjZitTAREZFiLLkSEZEyKu4LrPHewkREREXvC2xQkIZzY3AlIiJlVLSXamxzJSIioqJYciUiImVYcjVicCUiImVUjFHVXGCcK6uFiYiIFGPJlYiIlGG1sBGDKxERKcPgasRqYSIiIsVYciUiIoVUlDqdv+TK4EpERMqwWtiI1cJERESKseRKRETKcJyrEYMrEREpo2kKbtwv03BuDK5ERKSQeFyczuayq7NjmysREZFiLLkSEZEyxp6+OhvTcP6SK4MrEREpZHtwZbUwERERFcOSKxERqaOgWhisFiYiIrpBU1Clq7FamIiIiIpiyZWIiBRihyaBwZWIiBTSFHT2ZbUwERERVabkahrQm5aWVp7ViYiomjGdv+1/gwbRHcn5S54OCa7p6enyNSIiwt77Q0REdiTO50FBQcrT9fLyQlhYGOLj45WkJ9ISaTornVaOyxiDwYBLly4hMDAQOp2td94gIiJHE6d6EVjDw8Ph5mafFsGcnBzk5eUpSUsEVh8fH7h0cCUiIqLyY4cmIiIixRhciYiIFGNwJSIiUozBlYiISDEGVyIiIsUYXImIiBRjcCUiIoJa/w/0ndzpKjipNgAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def plot_values(V: np.ndarray, title: str = \"Value function\") -> None:\n", " \"\"\"Plot the value function V on the maze as a heatmap.\"\"\"\n", " grid_values = np.full(\n", " (n_rows, n_cols),\n", " np.nan,\n", " ) # Initializes a grid the same size as the maze. Every cell starts as NaN.\n", " for (\n", " s,\n", " (i, j),\n", " ) in (\n", " state_to_pos.items()\n", " ): # recall that state_to_pos maps each state index to its maze coordinates (i,j).\n", " grid_values[i, j] = V[\n", " s\n", " ] # For each reachable cell, we write the value V[s] in the grid.\n", " # Walls # never get values, and they stay as NaN.\n", "\n", " _fig, ax = plt.subplots()\n", " im = ax.imshow(grid_values, cmap=\"magma\")\n", " plt.colorbar(im, ax=ax)\n", "\n", " # For each state:\n", " # Place the text label at (column j, row i).\n", " # Display value to two decimals.\n", " # Use white text so it's visible on the heatmap.\n", " # Center the text inside each cell.\n", "\n", " for s, (i, j) in state_to_pos.items():\n", " ax.text(\n", " j,\n", " i,\n", " f\"{V[s]:.2f}\",\n", " ha=\"center\",\n", " va=\"center\",\n", " color=\"white\",\n", " fontsize=9,\n", " )\n", "\n", " # Remove axis ticks and set title\n", " ax.set_xticks([])\n", " ax.set_yticks([])\n", " ax.set_title(title)\n", " plt.show()\n", "\n", "\n", "plot_values(V_random, title=\"Value function: random policy\")" ] }, { "cell_type": "markdown", "id": "8275a1eb-b58e-4e05-ae5d-5635ff9a1556", "metadata": {}, "source": [ "The next function `plot_policy` visualizes a policy on the maze.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "c1ab67f0-bd5e-4ffe-b655-aec030401b78", "metadata": {}, "outputs": [], "source": [ "def plot_policy(policy: np.ndarray, title: str =\"Policy\") -> None:\n", " \"\"\"Plot the given policy on the maze.\"\"\"\n", " _fig, ax = plt.subplots()\n", " # draw walls as dark cells\n", " wall_grid = np.zeros((n_rows, n_cols))\n", " for i in range(n_rows):\n", " for j in range(n_cols):\n", " if maze_str[i][j] == \"#\":\n", " wall_grid[i, j] = 1\n", " ax.imshow(wall_grid, cmap=\"Greys\", alpha=0.5)\n", "\n", " for s, (i, j) in state_to_pos.items():\n", " cell = maze_str[i][j]\n", " if cell == \"#\":\n", " continue\n", "\n", " if s in goal_states:\n", " ax.text(\n", " j,\n", " i,\n", " \"G\",\n", " ha=\"center\",\n", " va=\"center\",\n", " fontsize=14,\n", " fontweight=\"bold\",\n", " color=\"blue\",\n", " )\n", " elif s in trap_states:\n", " ax.text(\n", " j,\n", " i,\n", " \"X\",\n", " ha=\"center\",\n", " va=\"center\",\n", " fontsize=14,\n", " fontweight=\"bold\",\n", " color=\"red\",\n", " )\n", " elif s == start_state:\n", " ax.text(\n", " j,\n", " i,\n", " \"S\",\n", " ha=\"center\",\n", " va=\"center\",\n", " fontsize=14,\n", " fontweight=\"bold\",\n", " color=\"green\",\n", " )\n", " else:\n", " a = policy[s]\n", " ax.text(\n", " j,\n", " i,\n", " action_names[a],\n", " ha=\"center\",\n", " va=\"center\",\n", " fontsize=14,\n", " color=\"black\",\n", " )\n", "\n", " ax.set_xticks(np.arange(-0.5, n_cols, 1))\n", " ax.set_yticks(np.arange(-0.5, n_rows, 1))\n", " ax.set_xticklabels([])\n", " ax.set_yticklabels([])\n", " ax.grid(visible=True)\n", " ax.set_title(title)\n", " plt.show()" ] }, { "cell_type": "markdown", "id": "48037254-dccc-4f9c-a4d7-349adba5c74f", "metadata": {}, "source": [ "Now let’s visualize the `random_policy`. Does it seem like a good policy?" ] }, { "cell_type": "code", "execution_count": 96, "id": "d452681c-c89c-41cc-95dc-df75993b0391", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAGgCAYAAAC0SSBAAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAH6JJREFUeJzt3Q1wlNWh//Ff1pCQxLw0AQUvEVulcm16UyJSShlem5bSoS1cKCUoOmqrcOWC5Ho7KUVIteoES6UjU65ixb6kV+xM6aU4XAIBW4tN9Z8/oyGASlHBvzLgSyIGQ0L2P+fETUKCxw0meXZPvp+Z4+4+2eA5eZ7d356XZ5+EcDgcFgAAHyH0UT8AAMAgKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIiiADiZNmmRLxCuvvKKEhARt3Lgx0HoBQSIoEPfMm7h5M4+UgQMH6rOf/axuu+02HTt2LOjqAXEvMegKAD3lxz/+sT796U/rgw8+0NNPP61f/OIXevLJJ1VTU6PU1NTz+jeHDx+uU6dOacCAAT1eXyBeEBTwxte//nWNHj3a3r/55puVk5OjNWvW6I9//KPmzZt3Xv9mpIcC9GcMPcFbU6ZMsbeHDx9Wc3Oz7rrrLl1++eVKTk7WZZddph/+8IdqbGx0/hsfNUdx4MABfec739HgwYOVkpKiK6+8UsuXL7c/27Vrl/2dP/zhD13+vfLycvuzZ555pkfbCvQmggLeOnTokL01PQvTw7jzzjtVUFCgn/3sZ5o4caLuvfdeffe73+32v/v888/ri1/8oiorK/W9731Pa9eu1be//W1t2bLF/txMhufm5uq3v/1tl98120xYfelLX+qBFgJ9g6EneKOurk4nTpywcxR//etf7ZyF+bQ/cuRI3XrrrTYsHn74YfvcRYsW6aKLLtL9999vewCTJ0+O+v+zePFimcu4VFdX69JLL23bft9999lb02O49tpr7bCXqVNmZqbdfvz4cW3fvr2t5wHEC3oU8MZXvvIVOxRkPs2bnsKFF15oh3/27Nljf75s2bKznl9cXGxvt27dGvX/w7zZ//nPf9aNN954VkhEAiJiwYIFdljr97//fdu2xx9/3A6BmRAB4gk9Cnhj3bp1dllsYmKiLr74YjtvEAqFbFiY2yuuuOKs5w8ZMkRZWVl69dVXo/5//OMf/7C3eXl5zueZXsw111xjh5puuukmu83cHzt2bJd6ALGOoIA3xowZ07bq6Vw6fuLvC6ZXsWTJEh09etT2Lv72t7/pwQcf7NM6AD2BoSd4z5wL0dLSopdeeums7eZkvHfffdf+PFqf+cxn7K05N+PjmOGvCy64QL/73e9sb8KcizF37tzzaAEQLIIC3ps+fbq9feCBB87abiabjW984xtR/1tmDmTChAn65S9/qddee+2sn5kJ7o4GDRpkz+34zW9+Y4Ni2rRpdhsQbxh6gvfy8/N1/fXX66GHHrI9CLM09u9//7see+wxu6y1OyuejJ///OcaP368XWr7/e9/354Nbs63MJPie/fu7TL8NHv2bHvfnMcBxCOCAv3Chg0b7LCROXHOTG6bieySkhKtXLnyvILHzDesWLHCfk2IWY5rhq/MCXidzZgxQ5/61Kfs0Nc3v/nNHmoN0LcSwp37ywB6jFkOe8kll9jAeOSRR4KuDnBemKMAetHmzZvtuRdmCAqIV/QogF5QVVVlv+rDzEuYCWxzFjcQr+hRAL3AzF0sXLjQfk3Ir371q6CrA3wi9CgAAE70KAAATgQFAKBnzqMw31XT8SIvZl3422+/bb/rv6+/QwcA8MmYWYf33nvPLt82X5rZI0FhLvJSWlr6CasGAIglR44c0bBhw3pmMrtzj8JckMV8H/8dd9zhzYXnTapeddVVqq2ttT2meOdbewzaFB9oU+xramrS6tWr7dfaRC6u9Yl7FOY6w6Z0ZkIiKSlJvhwIqamptj0+HAi+tcegTfGBNsWPaKYOmMwGADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAKTHaJzY2NtoSUV9fb29DoZAtPoi0g/bELtoUH2hT7OtOOxLC4XA4mieuWrVKpaWlXbaXl5crNTW1ezUEAASqoaFBRUVFqqurU0ZGRs8Exbl6FLm5uXrjjTeUk5MjHzQ1NamiokKFhYUaMGCA4p1v7fG9TTU1NWppaZEvn1bz8vLYTzHs9OnTuvvuu6MKiqiHnpKTk23pzBwEvhwIvrbJt/b42ibz5uPDG1BH7KfY1Z02+DHYBgDoNQQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEA6JkLF/W1/675bz2691HtfXOv3j71tlIHpCo7JVvDM4cr/+J8TR8xXV+74mtBVxMfWrJkiR5//HG9+eabQVcFQH8IigV/WKBfP//rs7bVN9bb8sq7r+ipV5/Sq3WvEhQxxFxO8dixY0FXA0B/CIptL287KySuHnq1vnb513Rh0oU63nBc1W9U65mjzwRaRwDoT2IuKLYf2t52/4rsK1R1c5UuCF1w1nNMz+KFYy8o3q5Pe8cdd+imm27SVVddFXR1cB4Xol+2bJndh8OHDw+6OjgH9lE/msxubmluu//uB+/aoabOMpIz9OVLv6x4cebMGV133XVas2aNHn300aCrg/NQXV2tDRs2aMKECTp06FDQ1cE5sI/6UVAUDC1ou3+i4YQ+++BndfVDV+vWP92qh//Pw3r57ZcVT5qbmzVv3jyVl5dr0aJFKisrC7pKOA9jx47Vli1bdPz4cftGdPDgwaCrhE7YR/1o6Onaf7lW655dp+f+33P2cUu4xc5LmBIx/tLxevDrDyp/SL5i3Zw5c7R582ZlZmYqISFBixcvjnoV0YgRI3q9fpBWrFihd955J6rn5uXl6dlnn9XEiRP11FNP6corr+z1+oF9FLSYC4rEUKIqF1Tq3qfv1S//7y917P2uK2mefu1pFf66UPsW7dPgtMGK5XmJ3bt3t60KWrduXdS/O3v2bIKij5jhwNdff71bv2NWeNXW1vIm1EfYR8GKuaEnIz05XfdMvUdvFL+hmoU1euSbj+j6/OuVnpTe9hyzAqrzEtpYEwqFtHPnTmVnZysrK0tVVVUKh8NRlUmTJgVd/X7j6NGjUe2TkydPtu2X0tJSzZw5M+iq9xvso2DFZFBEmKGaz130Od046kZt/PZGPb/weYUS2qv80lsvKdYVFBSosrJSiYmJKiws1J49e+TT5OH69eu7bN+/f7/Wrl0rn7z33nuaNm2a7SHed999uvPOO4OuEjphH/WjoafH9j6mD5o/0LzPz7OrmzpKG5Bmg8LMWxhZA7MUD/Lz87Vr1y5NnTrV9jDGjRsnHyxfvlzbtm1TQ0ND27Z9+/ZpypQpOnXqlGbNmqXc3Fz54OWXX9YLL7xgV67dfvvtQVcH58A+6kdBcfjdwyp9qlRL/3epnbT+wsVfsF/d8dapt/T72t+ftXx22hXTFC/MBJt5Ex00aJB8sWnTJk2fPl3FxcUaPLh1rmjy5Ml2Pfv27du9CQlj1KhR9o3Ip/3nG/ZRPwqKCNOr2PGPHbacy/cKvqeJl01UPPHtAE5PT7c9ihkzZtgeU2Q58I4dOzR69Gj5xrf95yP2UT8JiqVjl+rzF31elYcr9dwbz+nNk2/q+PvHdSZ8RoNTB+vqS662E9uz/nlW0FWFGQ5MS9PWrVv1rW99y85ZVFRU2E92APwRc0Fh5h3+9ap/tQXxISUlxQ41AfBTTK96AgAEj6AAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAA9MylUBsbG22JqK+vt7dNTU22+CDSDtoTu3xuUyjkz+e2SFvYT7GrO+1ICIfD4WieuGrVKpWWlnbZXl5ertTU1O7VEAAQqIaGBhUVFamurk4ZGRk906MoKSnRsmXLzupR5Obmqra2VklJSfIlYfPy8lRYWKgBAwbIh09AFRUVqqmpUUtLi3zg2z7quJ9oU2xr8uz1dPr06aifG3VQJCcn29KZ+YP58EfryBzYvhzcBvsoPtCm+NDiyeupO23wY7ANANBrCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAnTQ3N2v+/PkaOXKkXnzxRfliyZIlGjJkiHzh636KRQQF0OnykHPmzLHXgj948KAmTZqkAwcOyAfm2sjHjh2TD3zeT7GIoAA+1NjYqFmzZmnz5s1tF5s/efKkfRPat29f0NXDh9hPfY+g6MPr0xYXF6u2tjboquAjzJ07V1u3blVJSYlmzpxpt23fvl2nTp3S5MmTdfTo0aCrCPZTIBKD+d/2L2fOnNGCBQtsNzkUCmn16tVBVwnnsHTpUl1zzTVavny5brjhBrtt7Nixqqio0JYtWzRs2LCgqwj2UyAIij6YcCsqKtITTzyhRYsWqaysLOgq4SOYoQtTOhszZowtiA3sp75HUPQyM+FmxlIzMzOVkJCgxYsXR71CZcSIEb1ePwD4OARFL89L7N69u23Fybp166L+3dmzZxMUAGICk9m9yMxH7Ny5U9nZ2crKylJVVZXC4XBU5VxdawAIAkHRywoKClRZWanExEQVFhZqz549QVcJ/Uh1dbXWr1/fZfv+/fu1du3aQOqE+MPQUx/Iz8/Xrl27NHXqVNvDGDduXNBVQj9hVgZt27ZNDQ0NbdvMuQZTpkyxy0nN+Qi5ubmB1hGxj6DoI3l5efYFOmjQoKCrgn5k06ZNmj59uj2HZ/DgwXabOdfAnNlszj0gJBANhp76ECGBvpaenm57FCYcjh8/3rZke8eOHfbcAyAaBAXgubS0NHsms5kjy8nJscOfo0ePDrpaiCMEBXAOGzdutKvPfJGSkmKHmk6cOKFRo0bJF77tp1hFUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgFOiotTY2GhLRH19vb0NhUK2+CDSjqamJvkg0g5f9o+P+6hjW3xsU1lZmVpaWuTLsZeXl+fN66k77UgIR3nB2VWrVqm0tLTL9vLycqWmpnavhgCAQDU0NKioqEh1dXXKyMjomR5FSUmJli1bdlaPIjc3V7W1tUpKSpJPnxgKCws1YMAA+fCprqKiQjU1Nd59qvNlH3XcTz62ycdjr8aTNp0+fTrq50YdFMnJybZ0Zv5gPvzROjIvVl9esAb7KD742CYfj70WT9rUnTb4MdgGAOg1BAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIij6WHNzs+bPn6+RI0fqxRdfDLo6+AhLlizRkCFDgq4GEBMIij6+9OCcOXPsdcYPHjyoSZMm6cCBA0FXC+dgriN87NixoKsBxASCoo80NjZq1qxZ2rx5c9uFzE+ePGnDYt++fUFXDwA+EkHRR+bOnautW7eqpKREM2fOtNu2b9+uU6dOafLkyTp69GjQVYSHzHWRi4uLVVtbG3RVEMcSg65Af7F06VJdc801Wr58uW644Qa7bezYsaqoqNCWLVs0bNiwoKsIz5w5c0YLFiywQ52hUEirV68OukqIUwRFHzFDTKZ0NmbMGFuAnl40UVRUpCeeeEKLFi1SWVlZ0FVCHCMoAA+ZRRNmPiwzM1MJCQlavHhx1Ku9RowY0ev1Q3whKAAP5yV2797dtnpr3bp1Uf/u7NmzCQp0wWQ24BkzH7Fz505lZ2crKytLVVVVCofDUZVzDY8CBAXwoerqaq1fv77L9v3792vt2rWKJwUFBaqsrFRiYqIKCwu1Z8+eoKuEOMbQE/AhsyJt27ZtamhoaNtmznGZMmWKXcZszoPJzc1VvMjPz9euXbs0depU28MYN25c0FVCnCIogA9t2rRJ06dPt+cdDB482G4z57iYM+rNOS/xFBIReXl5NuwGDRoUdFUQxxh6Aj6Unp5uexQmHI4fP962zHTHjh32nJd4RUjgkyIogA7S0tLsGfRmXD8nJ8cO2YwePTroagGBIigCsHHjRrvCBLEpJSXFDjWdOHFCo0aNCro6QOAICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcEpUlBobG22JqK+vt7ehUMgWH0Ta0dTUJB9E2lFcXKwBAwbIlzZVVFSorKxMLS0t8uW4y8vL8+a48/3YK/akTW+99ZbuueeeqJ6bEI7y4s2rVq1SaWlpl+3l5eVKTU3tfi0BAIFpaGhQUVGR6urqlJGR0TNBca4eRW5urn70ox8pKSlJPn2yKyws9OITQ+QTkC/t6dimmpoa73oUPu4n2hTbPYqhQ4dGFRRRDz0lJyfb0pl5sfrygo0wB4EPB4Kv7TE47uIDbYpd3WmDH5MLAIBeQ1AAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAFYsmSJhgwZEnQ10M9w3MWut96Sysqkr35VuuQSaeBAcw0gaehQacIE6Y47pL/8RYruMnM9L+oLF6HnmCtKHTt2LOhqoJ/huItNDz0kLVsmvf9+15+9+WZrMSFx//3SG29IQWQ9QQEAAVm9WvrP/2x/nJAgTZ4sjR0rXXih9Pbb0t690tNPSx98EFw9CQoACMD+/VJJSfvjnBzpf/5HGjeu63NPnpR+/WspJUWBYI4C581cs7q4uFi1tbVBVwX9iC/H3c9/Lp050/54/fpzh4RhehcLF0qZmQoEPQqclzNnzmjBggUqLy9XKBTSatOHBnqZT8fdzp3t9z/1KWnWLMUsehTotubmZs2bN8++WBctWqQys1wD6GW+HXevv95+f8QIKdTh3fjAgdb5is7lhhsCqSo9CnTfnDlztHnzZmVmZiohIUGLFy+OennmCPOKAM6Dz8ddQoJiGkGBbo8P7969u2255bp166L+3dmzZ8f8Cxaxycfj7p/+SXrppdb75tacIxEJjIsual0RZaxcKTU0BFdPg6GnPlJdXa31Zraqk/3792vt2rWKF2ZceOfOncrOzlZWVpaqqqoUDoejKpMmTQq6+v0Ox13sHndTp7bfN8tgzYqniOxs6T/+o7UEtdKpI4KijyxfvlwLFy7UmjVr2rbt27fPHsQrVqzQkSNHFC8KCgpUWVmpxMREFRYWas+ePUFXCR+B4y523XabdMEF7Y9vvbX1nIlYRFD0kU2bNmn8+PF2Wd+TTz5pt02ePFmNjY3avn27cnNzFU/y8/O1a9cuDRw40H7SQ2ziuItdn/ucdNdd7Y/NGdijR0szZkirVkk/+Yl0881Sfb0CxxxFH0lPT9e2bds0Y8YMe6BHVnHs2LFDo83REYfy8vLsp9NBgwYFXRV8BI672FZSIqWltZ6d3djYel7Fn/7UWs7FnJQXBHoUfSgtLU1bt2613eacnBz7iSheX6wRPrxYfcdxF9v+/d+lw4dbexHjx0uDB0uJia1zE5deKhUWtv6sulr66U+DqSM9ij6WkpJiu/xAX+K4i21Dh7aubjIlFtGjAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICANAz18xubGy0JaK+vt7ehkIhW3wQaUdTU5N8EGmHL+3p2Jbi4mINGDBAvrSpoqLCy/3kY5vKysrU0tKieHf69Omon5sQDofD0Txx1apVKi0t7bK9vLxcqamp3ashACBQDQ0NKioqUl1dnTIyMnqmR1FSUqJly5ad1aPIzc1VbW2tkpKS5EuPIi8vT4WFhV58Wo18UvWlPQZtig8+t6mmpqbf9SiiDork5GRbOjN/MB/+aB2ZA9uXg9vH9hi0KT742KYWT97zutMGPyYXAAC9hqAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKPpYc3Oz5s+fr5EjR+rFF18MujowDh2SLrxQSkhoLV/9qtT5wo/mcWFh+3PS0qSXXlI8WbJkiYYMGRJ0NRCHCIo+vqLUnDlz7OVjDx48qEmTJunAgQNBVwuXXy799KftjysqpHXrzn7Ogw9KO3a0P77/fmnECMUTc8nLY8eOBV0NxCGCoo80NjZq1qxZ2rx5c9v1aU+ePGnDYt++fUFXD7fcIk2f3v74Bz+QIj0+c2seR0ybJi1c2Pd1BAJCUPSRuXPnauvWrfba4zNnzrTbtm/frlOnTmny5Mk6evRo0FXEhg1STk7r/YYGacECk/DSdddJp061bs/Olh55JNBqAn2NoOgjS5cu1d1336177rmnbdvYsWPtxdpvueUWDRs2TPF4zd3i4mLV1tbKC0OHSr/4RfvjqippzBjp739v32Z+fsklgVQPHz+0e9ttt+nVV18NuireISj6iBliWr58eZftY8aM0V133aV4c+bMGV133XVas2aNHn30UXljzhxp/vz2x88/336/qEj6zncCqRY+XnV1tTZs2KAJEybokFmggB5DUOC8Vm7NmzfPTsovWrRIZWVl8oqZuDa9i44uvrjrBDdiiumhb9myRcePH7dhYRaMoGck9tC/g37ErNwyk/KZmZlKSEjQ4sWLo16eOSIeVgqZ+aK33z57m3n8yivSF74QVK36tRUrVuidd96J6rl5eXl69tlnNXHiRD311FO68sore71+viMo0O15id27d7ctt1zXjU/Zs2fPjv2gaGpqnbw2k9jn2v7cc1JyclC167fM8Obrr7/erd8xS4HN/BlB8ckx9IRuCYVC2rlzp7Kzs5WVlaWqqiqFw+GoipmniXkrV0p797Y//rd/a79fUyP96EeKt3H79evXd9m+f/9+rV27VvHCrAqM5hiLLDk3SktL21YY4pMhKNBtBQUFqqysVGJiogoLC7Vnzx55wbSj43zLjTe2zlfcdFP7tjVrpL/8RfHCLKBYuHChXXQQYc7bMW+mZjjnyJEj8sV7772nadOm2R7vfffdpzvvvDPoKnmDoMB5yc/P165duzRw4EDbw4h7778vXX+9Wc7V+viyy6QHHmi9b24/85nW+y0trc87eVLxYNOmTRo/frxdxvzkk0/abea8HXMCqDmPJzc3V754+eWX9cILL9hQ/EHHEyTxiTFHgfNmJg3Np9NBgwYp7hUXm3ea1vuhkPTYY1J6eutj8z1Qv/qVNHFia5AcPizdfrv08MOKdenp6dq2bZtmzJhhgz2yam3Hjh0aPXq0fDJq1CgbFl4cjzGGHgU+ES9elNu2Sf/1X+2PTQhMmHD2c7785bO/xsOcxb11q+JBWlqa/VYAM0yYk5Nje4C+hYRXx2MMokcRgI0bN9qCGGG+u6nzt8Wey09+0lriUEpKih1qAs4HPQoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAOiZS6E2NjbaElFfX29vQ6GQLT6ItKOpqUk+iLTDl/YYtCk++NymkGfvd9FICIejuViwtGrVKpWWlnbZXl5ertTU1O7VEAAQqIaGBhUVFamurk4ZGRk906MoKSnRsmXLzupR5Obmqra2VklJSfIlYfPy8lRTU6OWlhbFO9/aY9Cm+ECbYt/p06ejfm7UQZGcnGxLZ+YP5sMfzec2+dYegzbFB9oUu7rTBj8G2wAAvYagAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwClRUWpsbLQloq6uzt42NTXJF6FQSA0NDTp9+rRaWloU73xrj0Gb4gNtin2R9+5wOPzxTw5HaeXKleZfo1AoFIr8KYcOHfrY9/8E85/z6VG8++67Gj58uF577TVlZmbKB/X19crNzdWRI0eUkZGheOdbewzaFB9oU+wzo0KXXnqp3nnnHWVlZfXM0FNycrItnZmQ8OGP1pFpj09t8q09Bm2KD7QpPobUPvY5fVITAEDcIigAAL0TFGYYauXKleccjopXvrXJt/YYtCk+0Ca/2hP1ZDYAoH9i6AkA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAufx/wkBHS6w2LYIAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plot_policy(policy=random_policy, title=\"Policy\")" ] }, { "cell_type": "markdown", "id": "cbad5bf1-0150-4c3f-8cce-c82e0f1d1695", "metadata": {}, "source": [ "**Exercise 9.** Define your own policy and evaluate it using the functions `policy_evaluation(...)` and `plot_values(...)`. **Can you identify an optimal policy visually?** Plot your own policy using `plot_policy`. \n" ] }, { "cell_type": "code", "execution_count": null, "id": "929707e6-3022-4d86-96cc-12f251f890a9", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdcAAAGbCAYAAACWHtrWAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAATRhJREFUeJzt3Ql8VNX5N/Dfmcm+EwgkgbALCLKJiigquCHiglvdRau2r0utW9X2X/e2VG2t1lq1dcHWvYtaN1xZVDYFQVkFZAlbWJOQPZl73s9zbiaZSSbJhJwsE35fP+Ms986Ze2fCPPd5zjl3lNZag4iIiKzx2GuKiIiIBIMrERGRZQyuREREljG4EhERWcbgSkREZBmDKxERkWUMrkRERJYxuBIREVnG4EpERGQZg2sE27hxI5RSmDFjRru8/syZMzFq1CjExcWZ7cjPz0dHJNt23333tfdmdFoTJkwwl47yd0nUETC4tpGzzjoLCQkJ2L9/f4PrXHrppYiJicGePXvQ0ck2/uhHP0J8fDyefPJJ/POf/0RiYmK7bc/777/PAEpEHUZUe2/AwUIC5zvvvIM333wTV1xxRb3lJSUlePvtt3Haaaeha9eu6Oi++uorc6Dw4IMP4uSTT27vzTHBVYJ8qABbWlqKqCj+qbeVPn36mPc8Ojq6vTeFqN0wc23DzDU5ORmvvPJKyOUSWIuLi00QjgQ7d+4012lpaejopGzN4Np2pCQs77nX623vTSFqNwyubUTKp+eeey4+/fTTmsAUSIKuBF8Jwnv37sXtt9+O4cOHIykpCSkpKZg8eTKWLVvW7P4vvyuvvBJ9+/YNesxxHDz22GMYNmyY+TLs0aMHfvrTn2Lfvn1Nvsa0adPM7SOPPNJ8mUr7Ql7Df7ux7Zo9e7Z53htvvIHf/va36NWrl9mGk046CevWrav3/IULF+L0009Hly5dTPl5xIgRePzxx2v2TbJWIW36L431uX7zzTfmPZX3Vt5jed0FCxYErSN9hvLcL7/8ErfeeisyMjLMa59zzjnYtWtX0LoFBQVYvXq1uW6KvEdnnHGGeQ+OOOII87chn7XcF//973/NfXk/xowZY7bV74UXXjDbFPiY3+9+9zsT0LZu3drga8v7IM+XbZWyvuy/VEp+/vOfo6ysLGjdqqoqU5kYMGAAYmNjzXb/6le/Qnl5eaP711Cfq/815X2UfR48eDD+7//+zyybNWuWeY5UdkL925Bl8+fPb/R1iToSBtc2JFmpfGFJQAkkwfTDDz80X9rypfPDDz/grbfeMl/Ajz76KH7xi1/gu+++wwknnIBt27ZZ2x4JpNL2scceawLVVVddhZdffhmTJk1CZWVlg8+TL8Sf/OQn5vYDDzxg+lulrQPx+9//3nyhysHEL3/5SxPg6mbvH3/8MY4//nisXLnSBIE//vGPmDhxIt59992a/TjllFPMbdkW/6UhK1aswHHHHWcOVu644w7cfffd2LBhgwn+EsTr+tnPfmbWvffee3HdddeZ8v6NN94YtI7sw6GHHhoyOIQiBxCXXHIJzjzzTEyfPt0c0Mhtef9vueUWXHbZZbj//vuxfv16E5DkQEicf/755m9E1qtLHpN96NmzZ5OvL21KMJXXloOWP//5zzWfqd8111yDe+65B4cffjj+9Kc/mb8/Wf+iiy5Cc3377bcYO3YsPvvsM1x77bXm723q1KnmvRSy3Tk5OQ3ulwT4cePGNft1idqN/J4rtY2qqiqdlZWlx40bF/T4008/Lb+pqz/88ENzv6ysTPt8vqB1NmzYoGNjY/UDDzwQ9Jg874UXXqh57IQTTjCXuqZNm6b79OlTc//zzz83z3355ZeD1ps5c2bIx+uS15T1vvrqq6DH5TXkteqqu12zZs0yzz/00EN1eXl5zeOPP/64efy7776rec/69etn2t23b19Qm47j1Ny+4YYbzPNCkcfvvffemvtTp07VMTExev369TWPbdu2TScnJ+vjjz++3j6efPLJQa91yy23aK/Xq/Pz8+utG/hZNET2RdadN29ezWPy2ctj8fHxetOmTTWPP/PMM+Zxeb/8Lr74Yp2dnR30N7JkyZKwXl/eB1nvrLPOCnr8+uuvN48vW7bM3F+6dKm5f8011wStd/vtt5vHP/vsswY/21B/l/K+yvsbuG8i8H395S9/af7GA9/XnTt36qioqKDPjygSMHNtQ1Kyk6N+KW9J6Syw7CUlWSlNCinBeTzuR+Pz+czIXCldShltyZIlVrblX//6F1JTU03Gt3v37pqLlCHltaRM1xYkW5YR0n6SUQrJ3oWUPyWrvPnmm+v17waWfsMl7+dHH31ksqb+/fvXPJ6VlWUyyS+++AKFhYVBz5GMLvC1ZBulnU2bNtU8JqVpieOhSuKhDB06NCgTk6xOnHjiiejdu3e9x/3vh5ABcVLBCPyMJLuTjPa8884L6/VvuOGGetm5f2BY4LWUwwPddttt5vq9995DuKSEPnfuXPz4xz8O2jcR+L7KfknJ+d///nfNY6+//rqp9kgmTxRJGFzbmL/k6R/YtGXLFnz++ecm6PoHgEgJUMpwhxxyiAm03bp1M/1UUloLp08vHGvXrjVtde/e3bQdeCkqKgrZL9wa6n7ZSp+q8Pf7SllUHHbYYVZeT77oZWS2HKjUJWVdee9zc3ObtY0Hom6bcqAjpDQa6vHA15IDIjkY8JdQZZtfffVVnH322abfPhzytxVIyq5yQOc/6JMDB7k/cODAoPUyMzPNQU7ggUVT/AcGTX2GQ4YMMX34gaVhuX300UfX2w6ijo5DKNuYZIbyJSJfhjI4RK4l4wnsZ5SBKdIPKEf6MqAkPT3dfNFJ9ubve2uIZAJuJTSYZFqBpB0JrKH6uIQE2QPRUDYprx9q9GhDI0pD7UN7aY1tbKjNcF5L1pEs++9//zv++te/mgFXksm2JLtr6HM7kOpAS0j2Kv3qctApWaz0wf/lL39p020gsoHBtR1IIJXgKZmoZLCSRcgRu5+UxWTAznPPPRf0PDkDkmSxjZGsKrCE6Fc305BM5ZNPPjGDmaScaIu8fqgzNcnrB5ZhwyXbKZYvX97ofNpwg4AcNMjJPNasWVNvmYxmlYOYutljRyRBSAZ2yYCgDz74wOyXDERrTuWiX79+QQOs5IDLP6Jc5qrKfVlPMnq/vLw88/nK8nD5P3f5DJsiFRwpRctBp3+u7IUXXhj2axF1FCwLtwN/liojMZcuXVpvdKxkJnWzIukjbWyKRWAwkiAROFVERrpKdlN3tKhkk5IZ1yV9XAd6KkN5fck2Kioqah6TUb11S63hkpGqEgRkylDdbQp8j/xnh2pqu+W9PfXUU8284sB+bwkacqAzfvx4Mz2luZozFccGmYokl2effRb/+c9/TFBqzlxe/9QlvyeeeMJcy/QkISOIhbzvgWT0upgyZUrYryWBX0Z7P//889i8eXPQsrp/53LwKNvw0ksvmaqKnFSlqQNKoo6ImWs7kGBxzDHHmC94UTe4yhQcmeIig31kPZmGI1804WR+UkqWL0DJYq6++mrTd/r000+buayBA3VkWoVMYZGpFRLgJeBIliCZigRymSoh0z6aS6ZvSOYtX4oSwKXPVL4o/Rloc0km+dRTT5lpKnIeY3lPpL9RAplMqZEpTP5yu7jpppvMvvsHj4Xym9/8xkzvkUB6/fXXm6D0zDPPmDLkww8/fEDbKVNwZNtkHmq4g5psZK8yhUk0tyQsg8RkTrV8TjLATj4jKTWPHDnSLJdrmcv8t7/9zRywyN/LokWL8OKLL5rBYFJZaQ6Z6iPvtxwsyQAx+TcgBzcyMEr+/urul/9vL9TBH1FEaO/hygerJ5980kxXOOqoo+otk6k4t912m5m2I1Mzjj32WD1//vywpjyIl156Sffv399MNxk1apSZ5lF3Ko7f3/72Nz1mzBjzOjJVYvjw4fqOO+4wU1MOZCqO+OMf/6h79uxpplXItn/99dcNTsX517/+FfTchvbpiy++0KeccorZxsTERD1ixAj9xBNP1CyXKTs/+9nPdEZGhlZKBU3LqTsVxz91ZdKkSTopKUknJCToiRMnBk2NaWwf/dseOD2muVNxpkyZUu9xeb5MKQr1fjzyyCP11t++fbuZEjRo0CAdLv9UnJUrV+rzzz/fvJ9dunTRN954oy4tLQ1at7KyUt9///1mKlR0dLTOyckx02Xk7zNQuH+Xy5cv1+ecc45OS0vTcXFxevDgwfruu++ut40yNUu2KTU1td42EUUKJf9r7wBPRM0nU6cki5fuBenDD4ecoUlOTiHdBh213CrdEtnZ2aZaUXfcAVGkYJ8rUYSS0wtKv/nll1+OzkTOTibBP9QPXBBFCva5EkUYOYWgnApSzsks/Z91zxkdqeTUkzKCXvpZR48ebfp5iSIVgytRhJHBbvPmzTPTqPyjfDsDGbgmA6tk4Bp/aJ0iHftciYiILGOfKxERUXuUheVMLXJ6NTlvaVufDo2IiFpOipT79+83I7H9Pwxim/yMYeAJZFpCftBDftO4UwdXCayRcEo4IiJqnJwtrVevXq0SWPv164kdO/ZaaU9+JEJOdhKpATas4Or/pQ35UA7k1HBERNS+5AxtkiSF+8tJzSUZqwTWjZveQEpKQovaKiwsQd8+PzJtdurg6i8FS2BlcCUiilyt3bWXkhSHlKQW/hhIE7/+FQk4FYeIiOyRwOi0MDh2guDK0cJERESWMXMlIiJ7mLkaDK5ERGSPnJdIt/DcRJ3g3EYsCxMREVnGzJWIiOxxtIWycORnrgyuRERkD/tcDZaFiYiILGPmSkRE9jBzNRhciYjIHgZXg8GViIjs0RaCq7QR4djnSkREZBkzVyIiskZpx1xa2kakY3AlIiJ72OdqsCxMRERkGTNXIiKyfIYm3fI2IhyDKxER2cOysMGyMBERkWXMXImIyB5mrgaDKxERWf49V6flbUQ4loWJiIgsY+ZKRET2sCxsMLgSEZE9nIpjMLgSEZE9zFwN9rkSERFZxsyViIjs4U/OGQyuRERkjXIcc2lpG5GOZWEiIiLLmLkSEZHlk0jolrcR4RhciYjIHo4WNlgWJiIisoyZKxER2cPM1WBwJSIie3iGJoPBlYiI7GHmarDPlYiIyDJmrkREZLks7LS8jQjH4EpERPZwnqvBsjAREZFlzFyJiMgeDmgyGFyJiMgeKek6LAuzLExERBFt7ty5OPPMM5GdnQ2lFN56662g5VdeeaV5PPBy2mmnteo2MXMlIqKILgsXFxdj5MiR+PGPf4xzzz035DoSTF944YWa+7GxsWhNDK5ERBTRwXXy5Mnm0hgJppmZmWgrLAsTEVGHVFhYGHQpLy8/4LZmz56N7t27Y/DgwbjuuuuwZ88etCYGVyIisn9uYaeFFwA5OTlITU2tuUyfPv2ANklKwv/4xz/w6aef4qGHHsKcOXNMpuvz+dBaWBYmIiJ7tONeWtoGgNzcXKSkpLS4n/Siiy6quT18+HCMGDECAwYMMNnsSSedhNbAzJWIiDpk5pqSkhJ0sTUIqX///ujWrRvWrVuH1sLgSkREB5UtW7aYPtesrKxWew2WhYmIKKJHCxcVFQVloRs2bMDSpUuRnp5uLvfffz/OO+88M1p4/fr1uOOOOzBw4EBMmjQJrYXBlYiIIvrH0r/++mtMnDix5v6tt95qrqdNm4annnoK3377LV588UXk5+ebE02ceuqpePDBB1t1rmuzgqvz4m1w4mPQacRGo7NRCZ3o8xHxrTvRuz2o0x9EZ6PfvxudTWf8nDqrCRMmQDdyysQPP/wQbY2ZKxER2cPfczUYXImIKKLLwh0RRwsTERFZxsyViIgssnASCWkjwjG4EhGRPSwLGywLExERWcbMlYiI7GHmajC4EhFRRJ+hqSNicCUiInuYuRrscyUiIrKMmSsREdnDzNVgcCUiInvY52qwLExERGQZM1ciIrJHfp1Gt7Cs29LndwAMrkREZA/7XA2WhYmIiCxj5kpERPYwczUYXImIyB75RRynhaN9W/yrOu2PZWEiIiLLmLkSEZE9LAsbDK5ERGSPVHSdlgZXRDz7wXXoCVCHjAXSs4HcldCf/K1mkRpzBtBnBJCWCaycA73gP023N/JUqCHjgbgkoCQfevY/gF0bgYy+UGOmAN16S8vA7k1ue/k7rO8SBo+HGnAUkJYNbFsFPfs59/G4JKgjpgLdBwLRcUDRbuhlHwBbVoRux+OFOuk6IK0H4IkGSgugV80G1s6vXT7+cqBrb6ikdDjyOrnfoVUMOAbocwSQkgnkrQbm/6N22djLgK59gagYoKIE2LgIWP1Zw23FpQBjzge69QcqioFVn7rP8et+CHDY6UBSN6A0H/j2HSDve7v702cs0Gs0kNQD2LUWWPJK7bLRFwFdegPeaKCyFMhdDKyf03Bb3QYAg08FEruazwirPgB2r6tdnpgBDD8bSM0CygqBVR8CO1fb3Z/OyubnJOsOOQ1IygB8FcDWpcCaT6TDLrzl1DqYubZScJUAuHQmVPYQIDEtaJEu3AUsegtqyLFhNaWOOAvIHAj9wROAPDcpHfBVuQtjE6C/XwB89jxQVQE1ejLUaTdAv36P/QnIJYXQ330MlTkoeJ+iYqH3bgWWvGPWQa+hUMddAf3+o0BBXv12tAP91X/cZdJhn9oD6pQboeX+zh/cVXZuAFbPBSTItqbSQmD1p+6BQXxq8LJVnwBFuwDHB8SnAeOvBor3AbnfhG7rqEuA4j3Au/e7wXr8NeZAA7t/ABLTgXFXAAtfAXasBjKHAEdfAXzyKFC8197+lO0H1s0BuvYH4ursz7pZQPFud39k2ZFXuEF+27L67cR3AQ6/GFj6BrBzrXtgIPc//wtQug9QHuCIS4Ft3wKLZrgHFKN+BHzxV6DE4v50VrY+JzmgHnMpsOFLYP7f3b/hsT8GSuTv9OswlhNF2oCmjcuATd8C5UX1l61dCGxZCVSUNd1ObAJw2InQc19yA6so2usGBSHt/LAYqCg1/xj1t5+YbM8EYNtyv3UzyPLi4MeL9gArZwElBe7RsGSsBTuBbn1DtyNBP3977Ug4cwyggeRu7n35Ulk9xw20rX2Gkm3LgW0rgPKS+ssKd7jb4t9I2RbJOkOR4Cn7u/wDwFcJ7Mt1g3DfI9zlPQYD+7YCO1a5bcn13lyg9xi7+5O3EshbBVSG2J/9ecH7IxfJSkPJOAQo3A7slMxau9cFW4Beo9zl6X2BmARg3WzAqXKX790I9KxeTm3zOUXHQcnnsEUO+LQbhHevB5J7hLecWo12tJVLpOu4fa4Z/dwvrwFHuJmu/KP7YTH04ncD/gEGyBoILYGiaB/ajZSuU3sA+7Y1upqaeC2QNQjKG+1mvptbqfTbEqOmmrKxioqBlgxzUwNH+1IalQOewIOp/G3AgHHubaXcSyC5n5qJNjXsDFOOVN4YaMletiwJvV7dbXUfBJKrt1e+nPfvDJ4qIAcj/NJu28+pshRaysY5Y4D1c93MVMr5K94Jbzm1Hp7+sIMH17gEqJh4IDUD+l8PmExWTboOqCwHls4MXjexC9SxF0Mv/G/7zY+S/tLjrgA2LXUzs0boWX83X+I6oz/QY6DbH9TRLH0LWPo2dFpPIHuo2wcWSlQsUFmnEiHryuMiby0wfAqQPQzYvgrIOhTo2sctGbelFe8CK96DloOB7kPqb7OfZDdDJgE9DgV2rgG6D3b77iQ7FbJfVY3sL7XN5yS2LweGTwUGToDyeKE3LnD7ccNdTnRQznOVICqBaPF7QFW56fPTy2dB9R4evF5CGtTpN0GvnAt8Xz0wqD0C6/FXAVWV0AteD//IbOd6qPgkYNiJ6JikjL3Fff+HnxF6FVkmg7kCRce7jwvpu134MnDoycAZ9wB9jwS2LAtdjm51GijY5m7boZNCryJ9ft+8AQycCJx0J9DrcPdLWgZ2CXlu3UAq++/fX2qbzymxGzDmEnew2YcPQH/6kDtwSQaihbOcWn9Ak9PCS4TruJnrnq1NryOBdcrPodd9BSz7EO0XWK8013r2s6FL1k09PzmjY49flEE8DfW5FmwH4lOA2MTaPum0LKAgYNT29pXuxW/ijcCmxWg3Hi+Q0EBfnpCRv4Gjf4/5CbBlaW2/4MAJ7nvir5KkyP423hVAlj8nKcPLSO0d1SPzpVtC+lcHHAes+ajp5dR6OFq4lTJX+dLxRrnX0n8lt+UfSb1lAbdDKdoDvXWVGQVshuYnpEINmwC9+Vt3udyf8nN3UNM371vfjXr75JH9kG1V1be95nETWKVfUqbNNBVYu/Q0fa1mf6TNnkOBfmOgtwV8kUu70n7g64bsB7S0T+b999+WL7M0IPswwBvj7mt6H2DgeCBvTeh2pD9290ZgWPXn1CUHyBkNbPyqdp20Xu7rSMY35GR3QFBDfbg29kfeL//+yKjTzKG1+5OWA/Q9OnhqTV2p2dV/nzFuII1OALZWj5SW8rCUgQee4H5WMgBKBjnJNA9qu8+pYCsQl+yW72V9+ZuSQWUyGC2c5UStTGnddM9xYWEhUlNTse/P1yAlPqbxBg8/HerwKUGP6e3fQ7/3ONTxl0MNOjp42fcLoOf+033ueb+GXvohsP6r2nmk4y8Bsge7fS/rFkF//Y6bMYw+HZ4xU6Cry8c17c18EshbH97ex0aHtZoacRrUyNOCX2fHOjOn1TPpZ9BVFUEd8Hr5x8BymU8HqDPvhJbbGxYD6TlQYy8AUrq7pa+ivdDffwmsnVf7Wufc4456DuB8+Qrww6LwtjWh8c+nxqGnQA09JXifdq0Hvn4dOPJid0qNfPnJ0f+mJcCaWbXzA0+5FVg9q3ZqTtA81xJ3Kk/gPFeZmpMu85G12wcr81xl/mg44sPsyzxkItQhweV1vWcDsOw/wKjzqwccKaB8P7B1mTvIxb8/x/3MnU8p02vEUdOA1F61fbCr3nffBz8pLx4m81yzq+e5zmzWPFd1+oPobPT7d7f95yR9sodMBBLS3cGP8lmtfL92JHJTyw+yz8n/PV5QUICUlJRWa3/fI1c1GSeabKu0Al1+8UKrbWtEBteIEmZwjSRhB9dIEW5wjSCd7Uu7WcE1gnS2z6nNgutDV9oJrnfOiOjg2nEHNBEREUWojjugiYiIIo4UQ3ULBySFUVDt8BhciYjIHo4WNhhciYjIHgZXg32uREQU0ebOnYszzzwT2dnZUErhrbfeqldmvueee5CVlYX4+HicfPLJWLu2dc/WxeBKREQRfYam4uJijBw5Ek8++WTI5Q8//DD+/Oc/4+mnn8bChQuRmJiISZMmoawsjB+ROUAsCxMRUUSfuH/y5MnmEropjcceewy//vWvcfbZZ5vH/vGPf6BHjx4mw73ooovQGpi5EhFRh1RYWBh0KS9v/jm8N2zYgB07dphSsJ/Mxx07dizmz2+989EzuBIRkTVyAj1t4SJycnJMIPRfpk+f3uztkcAqJFMNJPf9y1oDy8JERNQhRwvn5uYGnaEpNjZyztjGzJWIiDqklJSUoMuBBNfMzExznZeXF/S43Pcvaw0MrkRE1Gl/z7Vfv34miH766ac1j0n/rYwaHjduHFoLy8JERGRNYJ/pgWru84uKirBu3bqgQUxLly5Feno6evfujZtvvhm/+c1vcMghh5hge/fdd5s5sVOnTkVrYXAlIqKI9vXXX2PixIk192+99VZzPW3aNMyYMQN33HGHmQv7k5/8BPn5+Rg/fjxmzpyJuLi4VtsmBlciIrJHWyjrNnOe64QJExo92b+ctemBBx4wl7bC4EpERPZISdex0EaEY3AlIiJr5OfmdEt/co4n7iciIqK6mLkSEZE9LAsbDK5ERGSPVHS1hTYiHMvCRERE7Zm5eqb9EZ6A8zxGOufZG9DZqMsfQ2ei5/wOnY1e9Cg6ncT49t4C6iA4oMnFsjAREdnDPleDZWEiIiLLmLkSEVFEn1u4I2JwJSIie1gWNlgWJiIisoyZKxERWcOysIvBlYiI7JFZNI6FNiIcgysREVkjv/ym2/YX5zok9rkSERFZxsyViIisYZ+ri8GViIjs4VQcg2VhIiIiy5i5EhGRNSwLuxhciYjIGo4WdrEsTEREZBkzVyIissdR7qWlbUQ4BlciIrKGfa4uloWJiIgsY+ZKRETWaK3MpaVtRDoGVyIisoZlYReDKxER2Z2K47S8jUjH4BqOoSdAHTIWSM8GcldCf/K3mkVqzBlAnxFAWiawcg70gv803d7IU6GGjAfikoCSfOjZ/wB2bXSXpfWAOu5SoGsOULwPeuGbwObvWnHnOonsMUDmCCAxA9i7HlgR4nOITgSO/AlQXggsfi50O6k5wPALgx/zRANbvwbWf1z7WM44IHs0EJ0AlO8HVr8D7N9md5+6jwIyhgLx3YCCjcDa/9UuG3IBkJQV/C327QtAZXHD7WUcBmQeAcQkA1UlwKbZQP762vem36lAci+gqhTYthDY9V1kf04xScCg04G03kBlKbDpS2DHUvv7RBQCg2s4JAAunQmVPQRITAtapAt3AYveghpybFhNqSPOAjIHQn/wBCDPTUoHfFXVCz1Qp1wHrP8K+v0/A9lDoE78MfSb0911qWEVRe6XZ5e+QGxy6HUOORUoygOi4xtupyAX+OIPwV/0R98I7FpZ+1i/E4DU3sCyV4GyfUBsCqB9sK6yCNi6EEjt4waKunI/B/K+Ca+tjOFA5uHA+veAkl1AVALgja5dPmAKUJ4PfPOUG8wHn+vu2/4tiNjP6dCpQOk+YN7jbjAfcRFQuhco2Gxxh6gu9rm6OFo4HBuXAZu+BcqL6i9buxDYshKoKGu6ndgE4LAToee+VBssi/YCpYXu7axDgLhE6G8+cANu7nJg+1qogUdZ3qFOaPcaYM/3boYSStdDgKh4IK+Z2VjmcPcLunCrez8qDug1Fljznht8hGRYFY1kjAdq3zo3s5RMskUU0PMYN1OVwCokcy0vcG/HpgLJUpX5AnCqgOIdwJ7VQMYwROznFJcGpPYCNswGnEq3qpC3ws2aqXU5CrqFF85zpebJ6Od+eQ04ws10HR/ww2Loxe+6t6XsvG97cKlv7xb3cTpw3lhgwMnAd68BKb2a99zMkcD2gOwwpaf7GXYfCmSNdjPWnauAjXPafhRG9tFAz3FucN+xGNizKvR6cV2gYhKhE7sD/U42FRLkbwQ2zwGcCiAhwy0nS8D1kyDcfSQi9nOSfZUsObBMLtlw9uH2tpeoEQyubSkuASomHkjNgP7XAyaTVZOuAyrLgaUzgahYoKIkuDxSXgoVHddum9wp9D8R2PGtm9k050tb+vUkA8pbXvtYVBxUVBx0fDqw6Gm3dHnYjwBfBbD5S7SZLV8ApXvcQJ+SAww4w83QJNutS7JtkdIbWPFybRm4zwRgw0duX2VVefBzqsoAbwwi9nOSbQ+1T1FtvE8HIZ5b2MWycFuSICp/OIvfc//hy4Cl5bOgeg93l8tjEnwDqJg4oDKMkjM1/MUr5cHc+c1/rmRDe9YClQEHPL5K93rjXDeYSda49Su3nNmWira7AV2y5YJNwK5vgfRBodeV7RTbF7kBRi5yO61/7XLJGgPJgZ60H7GfU0Xofapqw306yPtcdQsvkY6Za1vaU90f1JC924BRk92ynb/E2LUXsDu3TTavU0rr62Y1425y7yuvO5DnmJuBr//ecF+pZD4ZQ4AV/w1+vDgPHVJjh/ql+6Alw22IlIBjEt2+Tn//bkJ3oHQ3Ivdz2gnEJrmjuf1BN6kHUMyBgdQ2mLmGQ4KdN8q9Vsq97fGGWBZwO5SiPdBbV0GNnux+cSSkQg2bAL35W3f59rVAeQnU6NMATxTQaxiQNQh63aK229eIpdwvZPPeB9zestAt3379nHuRjLNkj3u7Tgk+SPdh7qCbfT8EP15WAL1vA9BnvPsZySjenke4g3Raa5/kOnCfJCNL7ee+vjwuZeHuI4C9a0M3o6uA3auArCPd58pFbu+rnoYjA5tkwE+v6n1KzAS6DgF2LY/gzykfKNgC9Jvg7lNylrvujmWtsE8UqKWDmbR/UFMz3HfffVBKBV2GDBmC9sTMNQwS7NThU2rvX/U49Pbvod973MxJVYOOrl0mwfL7BdBz/+neP+/X0Es/NNNrhJ41A2r8JVCX/t4t90rgXFY9L0870B8/DXXcJVAjTgGK8836nIYThj7jofoeV3v/+Duh8zcBy14OLm9KSVSqAhX7ax874lpg8zxg54rgUqP0/4Wy6m1g0GRg3M/dtqWvL3eB/X3qeTSUDFjyO/Ln0IW5wLp3zTLEn+4+LqVpGZy0LyC4DjoH2L/VLf+KzbOAPicBI692B2FJMNo8u3b99e8D/U4BRl/nvkcyzcf2NJw2/5zeAgZPcbNf+bf2wyxOw+nEfa7Dhg3DJ598UnM/Kqp9w5vSuundKCwsRGpqKgoKCpCSkoLOwnn2BnQ2nmueRGei5/wOnU58JxygVtr5xgWoE36FzqS1v8f97a896zIkR7ds4Nj+ygoc8r+Xwt5WyVzfeustLF3acU4SwrIwERF1yAFNhYWFQZfy8jojwAOsXbsW2dnZ6N+/Py699FJs3ty+VQoGVyIissZxlJWLyMnJMdmw/zJ9+vSQrzl27FjMmDEDM2fOxFNPPYUNGzbguOOOw/79Ad0KbYx9rkRE1CH7XHNzc4PKwrGxdaZXVZs8eXLN7REjRphg26dPH7zxxhu4+uqr0R4YXImIqENKSUk5oP7htLQ0DBo0COvWhTipShthWZiIiDrVSSSKioqwfv16ZGVlob0wuBIRUUQH19tvvx1z5szBxo0bMW/ePJxzzjnwer24+OKL0V5YFiYiooi2ZcsWE0j37NmDjIwMjB8/HgsWLDC32wuDKxERWeNoZS4tbaM5XnvtNXQ0DK5ERGTNgZy+sK6WPr8jYJ8rERGRZcxciYjIGv6eq4vBlYiIrHFgoc/V/BJUZGNZmIiIyDJmrkREZI2Nk0DoFj6/I2BwJSIiayQwOgyuDK5ERGQPM1cX+1yJiIgsY+ZKRETWONWXlmjp8zsCBlciIrKGZWEXy8JERESWHdSZqy6tbO9NoKYkxqOz0YkJ6Gw8R93a3ptAHYSjm3/i/VBtRLqDOrgSEZFdLAu7WBYmIiKyjJkrERFZLgujxW1EOgZXIiKyhmVhF8vCREREljFzJSIiuz85B/7kHIMrERFZwx9LdzG4EhGRNTLH1WnxPNfIz1zZ50pERGQZM1ciIrJGW+hz1exzJSIiqsU+VxfLwkRERJYxcyUiIms4oMnF4EpERNZIf6lmnyvLwkRERLYxcyUiImt44n4XgysREVnDPlcXy8JERESWMXMlIiJrOKDJxeBKRETWsM/VxeBKRETWMHN1sc+ViIjIMmauYVAjJkINGQd06wlsWgHnvb/WLoyOg5p4KVS/EUBVJfS3s6C/eq/hxjJ6w3P8RW5bpUXQi96BXr3AXZbWHZ5jzgMy+wNR0cCebXDm/QfYvr71dzLSZYwEug0D4rsCBRuB9e/ULht8PpCYBWin9rHlM4DK4obb63YYkDkGiE4GqkqA3NlA/g+A8gKDzgHiugIeL1BRDOQtAXZ/Z3+fugyDSh0MxKYDxZuht3wUvDxtCFT6SCA6Eagqg877EijaFLIpNeASICre5ASGdqC/n1G7QmJPqO5jgehUoKoYOm8+UJxrf5+o02NZ2MXgGgZdnA/99ftQOYdCJXUJWqZOuAgqLhHOC3cBCcnwTL0V2L+nNmAGiomH56yboBf+D/q/nwPd+8Jz9s3QBbuB7euA2AToTcuhP/snUF4MNfRYeM68Cc4//g8oK2q7HY5EEii3LwRSegPRSfWXb/kC2PlNeG11Gw70GA2sfx8o3QVEJQCeaHeZBOjNs4DSvW6gikt3g3fZXqBoq919qiqB3r0EKrGnG0ADpR0KlT4ceusnQPkewBsPeBr/56y3fgoUbay/IDoZqtckt62izUBSb6hep0D/8C+gcr/dfaJOr72m4jz55JN45JFHsGPHDowcORJPPPEEjjrqKLQXloXDsf4b4IelJtMMEhUDNehIOAveBipKgfyd0Ms+gxo6PnQ7WQMAXxX08rnuzz7kbYBevwRqWPX6eRuhV3zuBlKtoVd84X6ZS5ZLjctfB+SvB6pKW9iQAnqOczNVCaxCMteKgurlGijdU5sB+sWmwrr9G9xg6Curt40q4wjovHluYBW+0gMPhEk5QNluN7AKuS7dCZU6qGXbT9RGXn/9ddx666249957sWTJEhNcJ02ahJ07d6K9MHNtibQeUN5oYFdt+UzvzoU6YnLo9VWIozGloLr2qvtV7eoqGUscsHe7vW0+WGWNBbKPBioK3TLunlWh14vrAhWdCJ3QA+hzMqA8bpk5dy7gVNSuN/BskyUrTxR0yS43sLeVmFSoqATouG5Qmce7f1dFudA75wNOZYNPU1nHATjevAd69+KAsm+oLEEBsV1bbReo85LvMm2hDVFYWBj0eGxsrLnU9eijj+Laa6/FVVddZe4//fTTeO+99/D888/jrrvuQntg5toSMbHQFWXBfXnlJUBMXOj1d/wARMeaPlzTX5c1AGrA6NDrSwl50rXQi98HSoL/wKiZtnwJLH8eWPaMWx7OmQikDQi9blT1ZyHl5VWvACtfcrPSnBOC11v3NrDkL9Cr3wD2rQWcKrQZr7uNUi7WG/8LveE/QEwyVI9jGnyK3vYZ9LpXode9BL1vOVSvU4G4DHdh8Rb3dlJfN6jKdUImIAeORAfyY+m6ZRf/aOGcnBykpqbWXKZPn17v9SoqKrB48WKcfPLJNY95PB5zf/78+WgvzFxboqIciI5xsxt/gI1NACTghlJWDOfdv8Bz7PlQR50J7NsOvXIelAxgqhtYpS92+zrohQEDc+jAFAdk/oWb3MFH6YNDZ5u+6sxv+yIzSKjmdv/TgXpjhbTbz5o+yB38JOu1hersVO9eWlMyltuq50kNP6d0R+3twnVAcl+o5H7QZbtMyVv6W6XUjOwTgJI8oHC9+3dN1I5yc3ORkpJScz9U1rp79274fD706NEj6HG5v3r1arQXBteWyM8DHB/QrRewy+2vUnJ7TyMDW7avh/Pvh2ruqtOuhd76fZ3A+nPovdugZ73Uqpt/0JL+7oaU7YVubhYqI4hjgwe6taqK/OZvY1PvQdEm6ICRxqrvVOiCgL9LojBJmuFYaENIYA0MrpGEh6bhkCN4b5TUGtyymbntBaoqoNd+Dc/RZ5ugiNTuUCNOhF75RcNtdctxR3V6o81AJtVzMPSyT9xl0XHuaOL8POhP/9Fmu9c5KDfIyZ+0qr5tPrdYILVv9UhaBSTnABnD3VJuKNrn9sdmHuk+Vy5y25/lxme4JWPzWgpI7QekDwEKN7bePpkMMmD/ZBsL1kJ1HQl4YszF3N7fwDZEJQHxWdXteIDk/iZz1YHrx3VzX0NGRXc73C095zO4UvNpKevqll/C1a1bN3i9XuTl5QU9LvczMzPRXpi5hkEdOQWesWfW3Pde/1foLWvgvPlH6NmvAideBs9VD7nBVua5BkzDMcFy21rorz9w2xp5otvPKl90O34wbaDYHYkqjysZUdytF1T/0TVtSAarv2+jkmOkyh4LlT2u9v6Ym6D35wLr3wOyjgb6p7uPlxe6g5MCg+shU4H9W4EdX7n3ZaRw7xOB4T92A5nMb82d4y6Tz63nsWbgkxl1IQOkZNneNdZ3SXU73C3V+u8PuQa6eBv05nfMSGGVOR5q4CXuNu7f5A5o8q/b/wLo3d+4JWBPFFTmMWYglOm+qC4Do6x2JKXKGAvEd3dL3cVboTe9A+g27EcmOkAxMTEYM2YMPv30U0ydOtU85jiOuX/jjTeivSitG6uR1Y7Yks7kgoKCiE3RQ/E98RN0Nt6f/Q2dif76T+hsdGICOhvPoT9t702gdv4e97f/yuG3I0EqPi1Q4ivHJUv+EPa2ylScadOm4ZlnnjFzWx977DG88cYbps+1bl9sW2HmSkREEX2GpgsvvBC7du3CPffcY04iMWrUKMycObPdAqtgcCUioog/cf+NN97YrmXgujigiYiIyDJmrkREZA1P3O9icCUiImv4e64uloWJiIgsY+ZKRETWsCzsYnAlIiJrGFxdLAsTERFZxsyViIis4YAmF4MrERFZIyfUdVpY1m36pLwdH8vCREREljFzJSKiDvl7rpGMwZWIiKxp7u+xhtLS53cEDK5ERGQNM1cX+1yJiIgsY+ZKRETW8CQSLgZXIiKyRuKittBGpGNZmIiIyDJmrkREZLksrFrcRqQ7qIOrLvOhs6l88Ep0JtF3z0Bnoze/hM7GV/A/dDY6KQWdSZWvuE1eh2VhF8vCRERElh3UmSsREdnF0cIuBlciIrKGJ5FwsSxMRERkGTNXIiKyRn4uTvMn5xhciYjIHvmhc4c/ls7gSkRE9jBzdbHPlYiIyDJmrkREZA1HC7sYXImIyBrOc3WxLExERGQZM1ciIrKG5xZ2MbgSEZE1LAu7WBYmIiKyjJkrERFZw3muLgZXIiKyhlNxXCwLExHRQaNv375QSgVdfv/731t/HWauRER0UA1oeuCBB3DttdfW3E9OTrb+GgyuRETUIafiFBYWBj0eGxtrLi0lwTQzMxOtiWVhIiKynrk6LbyInJwcpKam1lymT59uZRulDNy1a1eMHj0ajzzyCKqqqmAbM1ciIuqQcnNzkZKSUnPfRtZ600034fDDD0d6ejrmzZuHX/7yl9i+fTseffRR2MTgGgY1+kR4DjsW6NYTesNyOG/9pXZhTBw8p1wONWAkUFUB55vPoOe/G7qhhGR4Jl4ElTMIiIkH8nfB+fIt6PXLgl9v7OnwjDwBiE8GivbB9/6zwPYNrbuTXTLgPe1yqJ4DgMoKOIs+gjP/gwZXV6OOh/eYyUByOlCyH74PX4b+/hvAGwXvJbdDZWQDUdHA/nz4FsyE/mZO625/Z5Q0CCpxABCTBpRug95d/R56E6CyzgxeV3mr15ndcHuJA6FShprnwymD3vc1ULoleJ3oVKjM04NfzyIV0xcqujfgTQaqdsIp+ap2oScJnvjhgDcV0A501Q7o0hUAfKEba3L9KKj4EVDRPdzlFRugy7+3vUfwqEOgVBd58wBUwNGbofWO6uVeeNQgKNXVjIF19FZovamR9ppav7nttT35LVZt6fdcJbAGBteG3HXXXXjooYcaXWfVqlUYMmQIbr311prHRowYgZiYGPz0pz81WbGN4O3H4BqOonw489+F6nOoG0wCeE66BIhPhO+ZXwAJKfD+6DY4hXugV8yv3050LPTOzXDm/Nu0qQaMgOeMn8L30oPAnu1ue8edC9VrEHxv/BHI3wmkdAV89ksWQZRC1IU3w1mzBL7XHzeBNurSX0Dv3we9fEH91UefAO/Rk1D136eAHZuBxBSzb4bjg+/Dl4Bd28wXGrplI+ryO+HbvR061/YXWyfnK4Uu/A4qLssNiDWPl0BveT1gRQ9Uz3OhSzY2EVgPhd79OVC5D/DEAar+P3+VfjRQvgutRTvlJsCpqAwo2YYAnoQx0FV7oYsXACoanoSxQOwg6PJVIdtqan0VPxxKxcAp/BjwxMKTOA5wSqAr6xxQtIgEgQr4HDlALpNwAK9nOBxdDo19JvDKtvkc+XcUDa9nJByUQeu80PvUxPrNba89aAsDknQz17/ttttw5ZVXNrpO//79Qz4+duxYUxbeuHEjBg8eDFsYXMOg1y4x16p7TnBwjYqBGnIUfK9MB8pLzcVZ8ik8w4+DL1RwLdgN/dWHte1KxrpvB1TWAGgJrnGJUEecCt+Me93AKgr3tP4Ods0CumbCmfOWCY7YswPO0rnwjJ4AX93gqhS8E86F7+2/uYFVFBcGz/7euaX+P5P07gCDa/OU5rrXMenBwbWuhF7ul3xJ9edRj4JKGwm9Z54bWIUjgaCO5CFAZYEJ3oiWTKwVVLkHkSbbRHBwhScBuvJb929GV5hMVHm7NPxF2+j6XqjobDjFX8iLAk4VdPkGqJjeloOrZI+BBzWF0MiHUqnQugBKdYfP+cbdBlSZTNOjsuALGQw9Tazf1PKDV0ZGhrkciKVLl8Lj8aB79+5Wt4nBtSXSM6Gk9Lmz+ktQyO2jp4T3/IRkID0Lepf7fJXVH/BVQh061i0L+6qgV38F54s33aDXWpQKvq6+rXrIl3YdXbOgkqR02BfeKVcBHg/0uu/g+/hVoKL2C9t74c1Q/YeZ90fnbYZe7R6gkH0qcSBQsqHhqfdRKVDeeOiYdKj0sW4gLtsGvW8JoCvddbyJUMlDoHe8b67bgy5fDxWdA+0rMNmZisqCrtx0YOt7EqGkVO6rPfDTTiGU95BW3gsPFFLgaDk4ToBSMma0KGCjiwDVu4HnNrV+c9trHx15Ks78+fOxcOFCTJw40YwYlvu33HILLrvsMnTpYveAksG1JaTMKwFFyp/VdHmJ6YdtksdrSsJ6zVdAXvUXQnwiVGwCVJfu8D37K5PJes+7CaqyrOF+XBv27ADyd8Mz4Rw4s980WaZn5HFAbHy9VVV8onvdbyiqnr3P3Paeex28p14C37vP16zne/0xN0DnDILqM9j0R1Mr8CYCcZnQOxo5ePHGmCtl1nP70VW38VBdxkDvdSsTEnR1/jLAab/PSVfthCd+FFTK6SaI6Mrt0BWbD2x9FQWtJbsL+JY2BxKt+5XnUYOhUQINKa1L9uoL2gZtMs6GtsHbxPpNLe8YOvKv4sTGxuK1117Dfffdh/LycvTr188E18B+WFs61qcSaSrLgegYQI4mqwOskoAUkME1GFjPvt4dAPXhi7WPV5SbK+fLt922K8vhLP7EZLG+1gyujg9VbzwO7ymXIOrmPwGF++As+wKewyfUW1XXbOO7QGlRzW0JsPVX1tCb10ANPQqecZPhfPFO6+3DQUolDQAq9gGV+Q2v5Lh99rpgBeCU19yWAGsk9HOrFib7bS/Rpk9Ul62GrthYPRhpOFT84dCli5u/vgms3uo+UR3wddd64xfcgU3x1f2vQgKhJ2gbVKPb0NT6zW2P6pJRwgsW1B9H0hoYXFti7w7A5wOkL9affXbvDeza2nhgPes6wOuF8+Zfgsq9/vJwu9i1Db5X/lBz13PSBSYw1rNnO3Rl87Ib5fUC6T1sbCXVldgfulBGyDaiqhC6OsCGIhktYrpB9Ty/+oEot+rQ8zzorf9BmzB9yl4zotdVaYKmJ/Fo6NIDWN8pdsvknhTAKXB3S/p5fftbMbCmVAdW/7/pkuogmFhbylVJMkihgVaaWr+57bWPjlwWbks8iUQ4JDP1RpnAaI7w/berKkxZ1zN+qju1Jq07PKNPgvPd3MYDa0ysG1jrjgIu2A1n4wp4xp1pBkshMQ2ew0+Cs25p6+9j915uFu7xQg0ZY8rCvs//V3+9qkro7+bDc8wUIC4BiE0wt/UaGWQBoEdvqH7D3Gk4ygM1cCTUYePg/LC89feh01EBmYr/dsA/WRlFLCNuixsZJSyklFiywZ2Go6TSEu3erh4wpfctht7+julvlQuK1gJlee7tNtknBfiKTLYpU3Xcx71QMX0A6U8Npcn1fdCV2+CJk/5j+fcqXS79oCs2tVJgTa0OrIH/ph1ovRMeT7/qLDoeHtUTjq4e1FVPU+s3t732oS39F+mYuYbBM+4MeI49u/b+rc9Ab14N3+uPwPnkZXhOvQLe6/7gzg+Vea4BI4U9590MveV76IXvA9kD4DlktMn8vDc+XrOOs+A9d7ncfu/v8Jw6Dd4b/mRGH+uVC6AXzWz9fZTS7ZgTTVCUAUi+N/5cM+rXe/Gt0Ju/d0vB8rX10cvwTr4CUT/7gxtsv1/qDmgyxyEeeE48D0pGIMvIYTlg+PjVkFN6qHEqdThU6oja+70vgZagt/PjgIFMm2oHJQU+N2MidPlOoDqrlTmtqstRUD2nusG2dIsJqu7CCsAXUI2Q9mQdX6n9fYodBE9c7XQHb+oZ0FW74RTPg1OyCJ64Q6HiDnX/dnx74ZR8EzA4eCy0by90+VoTPJtaX5d+B8SPgCflVLM/Zp6r1ZHCIhYeT09o7cDrGVf72joPjv4ejl4LDwZVL/PPS60d2evxDDejirV2+4qbWr+p5dRxKK2b/uU8Ob+jnHqqoKAgrAm9kaLqkavR2eiyVhxV3A6i756BzsbZ/BI6G53aeb4X/HRS59qnwsJidE0/o9W+x/1x4ud9fonYOnOYm6vcKcPjm6ZHdMxh5kpERAfFaOG2xOBKRETWcECTiwOaiIiILGPmSkRE1sgoHt3Scwt3gsyVwZWIiKxxGj4RZ9ha+vyOgGVhIiIiy5i5EhGRNRzQ5GJwJSIieyz0uaITBFeWhYmIiCxj5kpERNZwQJOLwZWIiKzhVBwXy8JERESWMXMlIiJrWBZ2MbgSEZE18kNruoV13ZY+vyNgcCUiIms4z9XFPlciIiLLmLkSEZE1/D1XF4MrERFZw7Kwi2VhIiIiy5i5EhGRNcxcXQyuRERkuc9Vt7iNSHdQB1dd5kNn49tXhc4kRnW+P1Fn4cPobLSn8/UwRaWehc4kylvY3ptwUOl831xERNRuWBZ2MbgSEZE1PHG/i8GViIiskf5Wp8V9rpEfXTtfRwkREVE7Y+ZKRETWsCzsYnAlIiJr+JNzLpaFiYiILGPmSkRE1vD3XF0MrkREZA3nubpYFiYiooPGb3/7WxxzzDFISEhAWlpayHU2b96MKVOmmHW6d++OX/ziF6iqat7Z75i5EhGRNY6Fea5OK85zraiowAUXXIBx48bhueeeq7fc5/OZwJqZmYl58+Zh+/btuOKKKxAdHY3f/e53Yb8OgysREdk9cb9ueRut5f777zfXM2bMCLn8o48+wsqVK/HJJ5+gR48eGDVqFB588EHceeeduO+++xATExPW67AsTEREHVJhYWHQpby8vNVfc/78+Rg+fLgJrH6TJk0yr79ixYqw22FwJSIi62Vhp4UXkZOTg9TU1JrL9OnTW337d+zYERRYhf++LAsXy8JERGT3DE1oeRsiNzcXKSkpNY/HxsaGXP+uu+7CQw891Gibq1atwpAhQ9BWGFyJiKhDDmhKSUkJCq4Nue2223DllVc2uk7//v3Dem0ZyLRo0aKgx/Ly8mqWhYvBlYiIIlpGRoa52CCjiGW6zs6dO800HPHxxx+bID906NCw22FwJSIiaxxtIXNtxTM0yRzWvXv3mmuZdrN06VLz+MCBA5GUlIRTTz3VBNHLL78cDz/8sOln/fWvf40bbrihwbJ0KAyuRERkjfwWq+7Av+d6zz334MUXX6y5P3r0aHM9a9YsTJgwAV6vF++++y6uu+46k8UmJiZi2rRpeOCBB5r1OgyuRER00JgxY0aDc1z9+vTpg/fff79Fr8PgSkRE1mgLPxmnEfkYXA9Elwx4T7scqucAoLICzqKP4Mz/oMHV1ajj4T1mMpCcDpTsh+/Dl6G//wbwRsF7ye1QGdlAVDSwPx++BTOhv5mDNpfaBdHnToOn/2Dzl+2sW4nK/8wAivfXW9U7/hR4jzwOKisHzqplqHzhsfrrjJ0A78QpUKldTBuVb/4TzoolrboLp59+Ou688xcYPvwwVFZWYu7cz3Hzzbdi69atNeucffZZeOSRh9CzZ08sWfINrrnmJ1izZk2DbTa1fnPba5buo4CMoUB8N6BgI7D2f7XLhlwAJGUBOuBr7NsXgMri0G01tn5UPNB7ApDSC/DGAGUFwNZ5QP4PsE0lDoJK7AdEpwFl2+Ds+dxd4E2Ap8eUOit7q9eZ23B7CQOgkg81z4dTBid/MVBW/XnHZMCTOhqITgF0FXTxBujCZdb3iSLr9IdthcG1uZRC1IU3w1mzBL7XHzeBNurSX0Dv3we9fEH91UefAO/Rk1D136eAHZuBxBQgurpT3PHB9+FLwK5t7pdet2xEXX4nfLu3Q+d+36a7JYFVlD94s9nH6EuvR/Q5V6DypSfrrasL9qHq47fhGTQMKjW93nLv0RPhPeE0VP7zL9BbNwFJKVAx4Q8EOFCpqSl46KFHMGfOHPOTVU888TjeeOM1HHvscWb5oEGD8PLL/8SFF15iTm32q1/9Em+//V8MGzbCDGyoq6n1m9tes1UWAVsXAql9gJik+stzPwfyvgm/vYbWl4BastNdLq+Z1h8YMAVY8TJQthc2aV8JdOEKqLhMKG987QJfCZxt/wpY0wNP1lTokk0NtqUSB0AlDYGz90ugch/giXMDsrsUnq7HQxetgt61yg3eGScBvmLo4nVW94koFJ6hqbm6ZgFdM+HMecsER+zZAWfpXHhGT6i/rlLwTjjXZKomsIriQiB/l3tbRsTt3BKQTVQfraW7w7/bkuraHb5lC4GKcqC8DL6lC6CyeoVc1/nuazjLFwPFRSEaUog67TxUvflPN7CKokLovdX73IpeffU1009SXFyMkpISPPbYnzF27FFmgIK47LJLMWvWbLz33nvmNGoPPvgbM9T+uOPc4FtXU+s3t71m27cOyF8PVJWiVZUXADsWu4FVSMYqQVUyXdvKtrgXp/HT2Kn4XuZvSZfmNrQGVMoIN1OVwCqcMhM83cXRUN5Yk62af1cSVMt2uBkztcnvueoWXiIdM9fmUir4uvq26hEiEHXNgkpKhcrsC++UqwCPB3rdd/B9/CpQUVazmvfCm6H6D4OKiobO2wy9unXLp6H45nwA78ixcFbKsHQF7+hxcFY2IyuqprpnQaWkQfXqh9gfXQ14vPCtXoaqt18Byls5SNRxwgnHm7Oy+LPIESOGY+nS2rKg/ITUypWrzOOzZ8+u9/ym1m9ue9ZlHw30HAeUF7rBcc8qO+tLmTi+K1C6G+1FslJdsrHh3ruoZJP5qph0qC5HmTxBl22DLlhiSsDQFXCK17vt7F8BeBNNtuzkf9XWu3LQYVnYxeDaXHt2APm74ZlwDpzZb5os0zPyOCA2oMRVTcUnutf9hqLq2fvMbe+518F76iXwvft8zXq+1x9zA3TOIKg+g4GqCrQ1Z8P3ppwb+5tnzH29aR0qP3mn+Q0luOVLKRmX/+luczv68hsRNfVSVL3+LNqK+0sW9+OCCy6qeUzmsOXn5wetJ/eTk5NDttHU+s1tz6otXwClewCnCkjJAQacATiVbrbbkvWVBxg4Bdi7Bih2z0rT5qT/NLYHdEEjB3cet5tBxWbC2fmh+1D6sUDaGOh9C819XbIZni5HQaUcBqU8cIrWAGXb22Yf6KDHsnBzOT5UvfE4VI8+iLr5T4ia+v/gLPsCKKlfItVSYpWnfPkuUFpkLnJbDRpVv10phWxeAySmwjNuMtqUUoj5f3eZAFv+y2vMRW7H/L87m99WuZuR+z59xy0bFxeZ296h7lwymy655GLs359vLsuX12aQhx12GD744F3ceONNpi/Ur6ioyJz8O5Dc37+//qCtcNZvbntWFW0HfBVul0LBJmDXt0D6oJatbwLrmW4A3vAx2otkm6bUWxl84BJEV5orR7JSKTE75ea2iuvpLo9Khqfb8XAKlsDZ+jp82/4LFZUKlRri3x512BP3RzJmrgdi1zb4XvlDzV3PSRe4gbGuPduhK5uXhSrpH0wP/kWGVpeQCJWegarPPzKjn0XVFx8h7sQzgMSk0H2rDdC7mr/PB+qVV141l0ASWD/55EPcddev8PLLrwQt+/bb7zBq1Mia+1FRURg69FB8993ykO03tX5z22tVze2jqru+CaxnuAOC1r4dPKq4jamE/m4ptzGV+6Gl/NsQ6Vv1lQD+PlunDLrkB6jkodAF7hl5qHX4w2NLtPT5HQEz1wPRvRcQHWP6E9WQMaYs7Ps8YJqEX1Ul9Hfz4TlmChAnpa4Ec1uvqS539egN1W+YOw1HeaAGjoQ6bBycH9r4y7m4CM6uHYgaf7K7LVHRiDr2FOh9e0IHVo/HXU+uVfXt6kFDqKyEb/GX8Epgjk8w+y23fa08DUfIKcsksP761/dgxozaM7D4vfTSyzjxxImYPHmy+cHj//u/X2H37t2YOzf0VI+m1m9ue82nqke/qtrb8n57Y4HUfoBHjo2VW+btPgLYuzZ0M02t7w+snujqwGphpHNj+2S+dlTA7YCvodgsU/JtbJSwywddvBGe5KFm8JJc5LYu3eIurtgLeOKBuOqxEJ5YqIR+0BXVg5+o1TBzdSkdxrAs+ZFYKXcVFBSE9QsFkaLywcZ/RaEhngnnwjPmRBNUZACS88nr0FvcvivvxbdCb/7eLQWL6Bh4J18BNfhwN9h+v7RmQJPK6gvP6VdAyQhk+RgKdsP5+jM4Sw58MIxvXyNH841QPbIRdfZl8OT0M1+2ztaNqPrfK2bEb9T5V5l1qv79grmOmnSuuQRy1q1CxV9/696JiXXnzA4fI6N8TGCtevvlmpJxc8T/6bWw133++WcxbdoVZqRwoKFDh5ufrhJTp56Nhx/+PXr16mXmpV599bU181LHjx9vysnJybUjShtbP5zloTgLHw5vh3qOg5IBSAF0YS6w7l1g0FQgvnoalBmgtATYHZDtDToH2L8V2L7IHaDU2PrJvaAO/RG0lIMDM9Zti9znh8HJDu/XQlTKcHhShgfvU3kenF2fusul31T7oPfVn9bm6TYBunwn9P6V1Y15odKOdEcWy3NKt9YOaBJxPd3Xikpyl5ftcJc3MVLZz9vrEnQmrf097m9/XMr1iFItm3pXpcsxv/CvER1zGFw7mQMNrh1Vc4JrpAg7uEaQcINrJGFwPbD2x6ZcZyW4Lix8KqJjDvtciYjIGqf6v5Zo6fM7Ava5EhERWcbMlYiIrNFKQ6uWjhaO/AFNDK5ERGSNtjDaV3eC4MqyMBERkWXMXImIyBoZjKQ4oInBlYiI7OEZmlwsCxMREVnGzJWIiKxxlAPVwtHCLAsTEREFYJ+ri8GViIisYXB1sc+ViIjIMmauRERkDUcLuxhciYjIGgc+KPha3EakY1mYiIjIMmauRERkjZwXWLe4LBz55xZmcCUiIms4z9XFsjAREZFlzFyJiMjygCZPi9uIdAyuRERkUcun4kgbkY5lYSIiIssO6sw1+u4Z6Gyi0bnoR19q702gMHjbewOow3C0lHQ9FtqIbAd1cCUiIrt4hiYXgysREVmj4YNuYeYqbUQ69rkSEdFB47e//S2OOeYYJCQkIC0tLeQ6Sql6l9dee61Zr8PMlYiIrHFPAOFYaKN1VFRU4IILLsC4cePw3HPPNbjeCy+8gNNOO63mfkOBuCEMrkREdNCc/vD+++831zNmND6gVYJpZmbmAb8Oy8JERNQhFRYWBl3Ky8vb7LVvuOEGdOvWDUcddRSef/55aN28gM/MlYiIrNFaBjSpFrchcnJygh6/9957cd9996G1PfDAAzjxxBNNv+xHH32E66+/HkVFRbjpppvCboPBlYiIOmSfa25uLlJSUmoej42NDbn+XXfdhYceeqjRNletWoUhQ4aE9fp33313ze3Ro0ejuLgYjzzyCIMrERFFvpSUlKDg2pDbbrsNV155ZaPr9O/f/4C3Y+zYsXjwwQdNWbqhAF8XgysREVme56pa3EZzZGRkmEtrWbp0Kbp06RJ2YBUMrkREZI3WFs7QpFtvKs7mzZuxd+9ec+3z+UzgFAMHDkRSUhLeeecd5OXl4eijj0ZcXBw+/vhj/O53v8Ptt9/erNdhcCUiooPGPffcgxdffDGoT1XMmjULEyZMQHR0NJ588knccsstZoSwBN1HH30U1157bbNeR+kwxhfLEOjU1FQUFBSEVf8mIqKOpbW/x/3td0sZC49qWd7m6CrsLlwY0TGHmSsREXXIqTiRjMGViIgOmjM0tRWeoYmIiMgyZq5ERGR5tLBqcRuRjsGViIgskj7XlrcR6VgWJiIisoyZKxERWeOWdJWFNiIbgysREVnD4OpiWZiIiMgyZq5ERGSN/FycavGJ+yM/c2VwJSIia1gWdrEsTEREZBkzVyIissbGeYE1zy1MRERU97zAjoU2IhuDKxERWWOjv1Szz5WIiIjqYuZKRETWMHN1MbgSEZE1Nuao6k4wz5VlYSIiIsuYuRIRkTUsC7sYXImIyBoGVxfLwkRERJYxcyUiIotsZJ2Rn7kyuBIRkTUsC7tYFiYiIrKMmSsREVnDea4uBlciIrJGawsn7jdtRDYGVyIiskh+Lk61OHeNdOxzJSIisoyZKxERWeOO9FUtbCPyM1cGVyIisqjlwZVlYSIiIqqHmSsREdljoSwMloWJiIhqaQslXc2yMBEREdXFzJWIiCzigCbB4EpERBZpC4N9WRYmIiKiA8lc/RN6CwsLw1mdiIg6GP/3d+ufoEGGI0V+5tkmwXX//v3mOicnp7W3h4iIWpF8n6emplpvNyYmBpmZmdixY4eV9qQtaTNSKR3GYYzjONi2bRuSk5OhVEvPvEFERG1NvuolsGZnZ8PjaZ0ewbKyMlRUVFhpSwJrXFwcOnVwJSIiovBxQBMREZFlDK5ERESWMbgSERFZxuBKRERkGYMrERGRZQyuREREljG4EhERwa7/D2LZ+Kya8v0xAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAGgCAYAAAC0SSBAAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAITBJREFUeJzt3Q10VNW99/FfYl4gkcDlRcFLxFtLoTUWQUqptkC0EYr1JQilhIqutndRKcpLrrUpBZKWUhp6WaULKq14wWub3mrbhxZZRcJrtbZceGgrIRAkj6Kg0hQhIQ4mgcyz9o4TQoKbiYY5c06+n7UOmTk5IfufmTm/OXvvMychHA6HBQDAe0h8r28AAGAQFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERRAB9m+fbsSEhLs14j7779f11xzjaftAj4oggKeW7t2rd3BmuX5559v833zKTOZmZn2+5///Oc9aSPQmREUiBtdunRRSUlJm/U7duzQkSNHlJqaKr957LHHVFFR4XUzgA+EoEDcGD9+vJ5++mmdOXPmvPUmPG688Ub17dtXfpOcnOzLgANaIigQN6ZMmaLjx4+rtLS0eV19fb1+/etfKy8vr013lOn7v+uuu9r8P++88466d++u6dOnO3+f6cqaOXOmfvGLX2jQoEH2iMYE0h//+Mc22/71r3/V5z73OWVkZOjyyy/Xrbfeqr/85S8XrelCYxSNjY1avny5rr/+evs7+/Tpo3Hjxmn37t32+6NHj9aQIUMu+P+Zdo4dO/aivxfoSAQF4obZoX7qU5/SL3/5y+Z1f/jDH1RdXa0vfvGLbXbyX/rSl+z333rrrfO+t379etXU1NjvX4zp1po9e7bd9jvf+Y4NKrPTLisra95m3759+sxnPqO///3v+sY3vqH58+fr5Zdf1pgxY7Rz58521/mVr3zF/k4z7vKDH/xA3/zmN21gRILn3nvv1YsvvnheG4xdu3bp4MGDUdUFdChzPQrAS2vWrDHXRAnv2rUrvGLFinC3bt3CoVDIfm/SpEnh7Oxse3vAgAHh22+/vfnnKioq7M89+uij5/1/d955Z/iaa64JNzY2On+v+Vmz7N69u3nd4cOHw126dAnn5uY2r7v77rvDKSkp4crKyuZ1r7/+um3nqFGjmtdt27bN/n/ma8R9991n2x2xdetWu81DDz3Upj2R9p48edK24ZFHHjnv++Zn0tPTw7W1tc66gI7GEQXiyhe+8AWdPn1azzzzjE6dOmW/tu52ivjIRz6iT37yk7brKMIcXZijjKlTp9qjjosxRzCmuyni6quvtt1Zzz77rM6ePWuXTZs26e6779aHPvSh5u369etn22VmaZmjl2j95je/se1auHBhm+9F2mu6zUwbzJFV5Lpiph2/+tWvbDvS09Oj/n1ARyAoEFdMf/1nP/tZO4D929/+1u4gJ06c+J7bT5s2TX/60590+PBhe98Mhjc0NNjum2gMHDjwggEUCoVUVVVlF3PbjA209tGPftSON7z22mtR11dZWamrrrpKPXv2dG5n6nr11Vf13HPP2fubN2/WsWPHoq4L6EgEBeKOeadujgpWrVplB5B79OjxntuasQszsyhyVPHzn/9cw4cPv+CO3U/MgPWVV15p6zHMVzPry4QoEGsEBeJObm6uEhMT7eDue3U7RZh35rfffrsNCnNUYY4u2vOu+6WXXmqzzgwYp6Wl2aMbs5jbFzoX4sCBA7adZlA6Wtdee61ef/31NgPwrV122WW2djPj68SJE1q3bp2dFWbWA7FGUCDumOmnjz76qAoLC3XHHXdcdHsTDOXl5Xr44YftjrT1DCmXP//5z9qzZ0/zfdON9Lvf/U633Xab/b/MYm6bda+88krzdqYbyHSPffrTn7ZTZqN1zz332HGHoqKiNt+LjEe0rMuEhJnmW1tby2wneCbJu18NvLf77rsv6m3NEUWvXr3s+ITpqrriiiui/tmsrCzbzfPQQw/ZE+N+8pOf2PUtd+SLFi2y53aYUJgxY4aSkpL005/+VHV1dSouLm5XXdnZ2TYAfvzjH9ujGTMV14xzmLEI8z1zXkfE0KFDbftMXWY8ZNiwYe36XUBH4YgCvpeSkqLJkyfb2+0d7DUnt/3oRz/Sk08+qQULFtiuLDM+8vGPf7x5m+uuu87uyM1O+/vf/74NkQEDBmjbtm121lV7rVmzRkuXLrXnYpijoMWLF9uZXjfddNMFB7XfT11AR0owc2Q79H8EPDBnzhw9/vjjevPNN+2YQjTMdNSvf/3rWrFiheKVOYPb1Ga6vczUXcALHFHA98xHdphZQab/P9qQ8APzHs6EnznqISTgJcYo4Fv/+Mc/7PkFZmaQ+eiNWbNmKQjefvtt/f73v7ddW3v37rUD6YCXCAr4lpnpZM7ANoPXZnD4hhtuUBCYk/zM1Fhz/si3vvUt3XnnnV43CZ0cYxQAACfGKAAATgQFAKBjxijMyUVmiTAnCZmPITAnOkXzKZ0AgPhhRh3MJzSbD6k0H0XTIUEROdEIABAc5mNr+vfv3zGD2a2PKMxVx8zcbnNmqfn0ziAwqfqxj33MzqYxR0x+F7R6DGryB2qKf+bj+M0nBJw8edJeA6VDjijM5+Bc6CLxJiTMRygE5YlgTtgy9QThiRC0egxq8gdq8o9ohg4YzAYAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMApKdoN6+rq7BJRU1NjvyYmJtolCCJ1UE/8oiZ/oKb41546EsLhcDiaDQsLC1VUVNRmfUlJidLS0trXQgCAp0KhkPLy8lRdXa2MjIyOCYoLHVFkZmbqjTfeUK9evRQEDQ0NKi0tVU5OjpKTk+V3Qasn6DWVlZWpsbFRQXm3mpWVxeMUx+rr67Vo0aKogiLqrqfU1FS7tGaeBEF5IgS1pqDVE9SazM4nCDuglnic4ld7aghGZxsA4JIhKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIA/g+J/yv5HY38+Vlf+8EolfzdZ3Zd0178t/zeNWTtGs/4wS88eetbrJiLAZs2apb59+3rdDFwEj1NsRH2Fu1ia9n+m6ckXnzxvXU1djV1eOfmKdhzeocPVhzX2w2M9ayOCzVwe8tixY143AxfB49RJg2LjoY3nhcSN/W7U2GvH6vKUy1UVqtKeN/boz0f+7GkbAaAzibug2FS5qfn2h3t+WDu/ulOXJV523jbmyGLvsb0KAnOB87lz5+rhhx/WgAEDFARBrAnozOJujOJM45nm2yffOWm7mlrLSM3QzVffrCDYs2ePVq9erVGjRqmyslJBEMSagM4s7oJiWL9hzbf/GfqnPrLiI7rxZzfqa898TY/938d06K1DCpKRI0dq/fr1qqqqsjvWiooK+V0QawI6s7jrevrSx7+klbtWavfru+39xnCjHZcwS8Snr/60VnxuhYb0HaJ4Nn/+fJ04cSKqbbOysrRr1y6NHj1aO3bs0KBBgxSPglgTAJ8FRVJikrZO26rvP/99/ddf/0vH3m47o+H5V59XzpM52jdjn/qk91G8WrNmjY4ePdqunzEzOMrLy+N2pxrEmgD4rOvJ6JbaTYtvXaw38t9Q2QNlevzOx3XfkPvULaVb8zZmBlTrKbTx5siRIwqHwxddamtrNWbMGPszRUVFys3NVbwKYk0AfBgUEQkJCbruiuv05aFf1tq71+rFB15UYsK5Jr90/CX53alTpzRu3Dht375dS5Ys0YIFC+R3fq3JDMKvWrWqzfr9+/dr+fLlnrQJbfE4xV7cdT098bcn9M6ZdzTl+il2dlNL6cnpNijMuIXRo0sP+d2hQ4e0d+9eLVu2THPmzFEQ+LWmefPmaePGjQqFQs3r9u3bp1tuuUWnT5/WhAkTlJmZ6WkbwePkhbgLipdPvqyiHUWa/exsO2h9w5U3qGfXnjp++rh+Xf7r86bPjvvwOPnd0KFD7Y61d+/eCgq/1vTUU09p/Pjxys/PV58+TWNf2dnZ9ryQTZs2sfOJEzxOsRd3QRFhjio2/7/NdrmQfx/27xp9zWgFgd92qEGtqVu3bvad6h133KFt27bZdWfOnNHmzZs1fPhwr5uHd/E4xV7cBcXskbN1/RXXa+vLW7X7jd16s/ZNVb1dpbPhs+qT1kc3XnWjHdie8NEJXjcVAZSenq4NGzborrvusn3hpaWl9ggJ8YXHqZMHhRl3uOdj99gF8ELXrl1tFwbiG49T7MT1rCcAgPcICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAEDHXAq1rq7OLhE1NTX2a0NDg12CIFIH9cSvINeUmBic922RWnic4ld76kgIh8PhaDYsLCxUUVFRm/UlJSVKS0trXwsBAJ4KhULKy8tTdXW1MjIyOuaIoqCgQHPnzj3viCIzM1Pl5eVKSUlRUBI2KytLOTk5Sk5OVhDeAZWWlqqsrEyNjY0KgqA9Ri0fJ2qKbw0Bez3V19dHvW3UQZGammqX1swfLAh/tJbMEzsoT26Dx8gfqMkfGgPyempPDcHobAMAXDIEBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKGLszJkzmjp1qgYPHqyDBw963Ry8h1mzZqlv374KkiDWhNggKGJ86cFJkybZ64xXVFRozJgxOnDggNfNwgWY6wgfO3ZMQRLEmhAbBEWM1NXVacKECVq3bl3zhcxra2ttWOzbt8/r5gHAeyIoYmTy5MnasGGDCgoKlJuba9dt2rRJp0+fVnZ2to4cOeJ1EwHfXOs5Pz9f5eXlXjel00jyugGdxezZs/WJT3xC8+bN0/3332/XjRw5UqWlpVq/fr369+/vdROBuHf27FlNmzbNdt8mJiZq6dKlXjepUyAoYsR0MZmltREjRtgFwMUnguTl5enpp5/WjBkzVFxc7HWTOg2CAoAvmIkgZoyve/fuSkhI0IMPPhj1bK+BAwde8vYFGUEBwBfjEtu3b2+evbVy5cqof3bixIkExQfEYDaAuGfGI7Zs2aKePXuqR48e2rlzp8LhcFTLhbp80T4EBfCuPXv2aNWqVW3W79+/X8uXL5cfBammYcOGaevWrUpKSlJOTo5eeOEFr5vUadD1BLzLzEjbuHGjQqFQ8zpzjsstt9xipzGb82AyMzPlJ0GraciQIdq2bZtuvfVWe4Rx0003ed2kToGgAN711FNPafz48XaOfp8+few6c46LOaPenPPipx1qkGvKysqyYde7d2+vm9Jp0PUEvKtbt2723bfZkVZVVTVPydy8ebM958WPgliTQUjEFkEBtJCenm7PoDd94L169bLdG8OHD5efBbEmxBZdTx5Yu3atXRCfunbtartlgiSINSF2OKIAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAICOuRRqXV2dXSJqamrs18TERLsEQaSOhoYGBUGkjqA8PkF8jFrWEsSaiouL1djYqKA897KysgLzempPHQnhcDgczYaFhYUqKipqs76kpERpaWntayEAwFOhUEh5eXmqrq5WRkZGxxxRFBQUaO7cuecdUWRmZqq8vFwpKSkK0juGnJwcJScnKwjv6kpLS1VWVha4d3VBeYxaPk5BrCmIz72ygNRUX18f9bZRB0VqaqpdWjN/sCD80VoyL9agvGANHiN/CGJNQXzuNQakpvbUEIzONgDAJUNQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCIsbOnDmjqVOnavDgwTp48KDXzUEnMmvWLPXt29frZsCHCIoYX3pw0qRJ9jrjFRUVGjNmjA4cOOB1s9BJmGsjHzt2zOtmwIcIihipq6vThAkTtG7duuYLmdfW1tqw2Ldvn9fNA4D3RFDEyOTJk7VhwwYVFBQoNzfXrtu0aZNOnz6t7OxsHTlyxOsm4iJHgzNnztThw4e9bgoQcwRFjMyePVuLFi3S4sWLm9eNHDlSpaWlmj59uvr37+9p++C2Z88erV69WqNGjVJlZaXXzQFiKim2v67zMl1MZmltxIgRdkF8M6G+fv163XXXXTYstm7dqkGDBnndLCAmCAp0evPnz9eJEyei2jYrK0u7du3S6NGjtWPHDsICnQJBgU5vzZo1Onr0aLt+xsweKi8vJyjQKTBGgU7PTCQIh8MXXSKz1IyioqLmSQlA0BEUQBROnTqlcePGafv27VqyZIkWLFggvwzCr1q1qs36/fv3a/ny5Z60Cf5D1xMQhUOHDmnv3r1atmyZ5syZI7+YN2+eNm7cqFAo1LzOnLdzyy232KnZ5tyezMxMT9uI+EdQAFEYOnSoDYvevXvLT5566imNHz9e+fn56tOnj11nztsx54WY83gICUSDricgSn4LCaNbt272iMKEQ1VVVfPnjW3evNlO+QWiQVAAAZeenm4/FSAnJ0e9evXSli1bNHz4cK+bBR8hKDywdu1aO4sGiJWuXbvarqZ//vOfthsNaA+CAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnJIUpbq6OrtE1NTU2K+JiYl2CYJIHQ0NDQqCSB35+flKTk5WUGoqLS1VcXGxGhsbFZTnXVZWVmCed0F/7uUHpKbjx49r8eLFUW2bEI7y4s2FhYUqKipqs76kpERpaWntbyUAwDOhUEh5eXmqrq5WRkZGxwTFhY4oMjMz9e1vf1spKSkK0ju7nJycQLxjiLwDCko9LWsqKysL3BFFEB8naorvI4p+/fpFFRRRdz2lpqbapTXzYg3KCzbCPAmC8EQIaj0Gzzt/oKb41Z4agjG4AAC4ZAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAngiLGzpw5o6lTp2rw4ME6ePCg180BEAeOH5eKi6XbbpOuukrq0sVcA0jq108aNUp6+GHpueek6C4z1/GivnARPrj6+npNnjxZ69ats/fHjBmjrVu32tAA0Dn97GfS3LnS22+3/d6bbzYtJiR++EPpjTekvn1j30aCIkbMZWTvuecebdiwwV520FxKtra21obFli1bdN1113ndRAAxtnSp9I1vnLufkCBlZ0sjR0qXXy699Zb0t79Jzz8vvfOOd+0kKGLEHEmYkCgoKNDrr7+uJ554Qps2bdLYsWOVnZ2tPXv2qH///l43E0CM7N8vFRScu9+rl/T730s33dR229pa6cknpa5d5QnGKGJk9uzZWrRokRYvXty8buTIkfZi7dOnT/dlSJhrVufn56u8vNzrpuB9dIPOnDlThw8fVlD4raYf/1g6e/bc/VWrLhwShjm6eOABqXt3eYKgiBHTxTRv3rw260eMGKHvfve78puzZ8/q3nvv1bJly7RmzRqvm4N2Mkewq1ev1qhRo1RZWakg8FtNW7acu/0v/yJNmKC4RVDgfc3cmjJlikpKSjRjxgwVm+ka8BVzNLt+/XpVVVXZHWtFRYX8zm81HT167vbAgVJii73xgQNN4xWtl/vv96SpjFGg/SZNmmRnbnXv3l0JCQl68MEHo/q5WbNmaaB5ReCSmj9/vk6cOBHVtllZWdq1a5dGjx6tHTt2aNCgQYpHQaypJRMC8YygQLvHJbZv325vV1dXa+XKlVH/7MSJEwmKGDBdgUdbvl2NwrFjx+xYU7zuVINY07/+q/TSS023zVdzjkQkMK64omlGlLFwoRQKeddOg64ntEtiYqKdztuzZ0/16NFDO3fuVDgcjmox4zS49I4cORLV4xGZnm0UFRUpNzdX8SqINd1667nbZhqsmfEU0bOn9B//0bR4NdOpJYIC7TZs2DB7omBSUpJycnL0wgsveN0ktNOpU6c0btw4e3S4ZMkSLViwQH7nt5pmzpQuu+zc/a99remciXhEUOB9GTJkiLZt26YuXbrYIwz4y6FDh7R37147a+2RRx5REPitpuuuk1pOeDRnYA8fLt1xh1RYKH3ve9JXvyrV1MhzjFHgfTODhvv27VPv3r29bgraaejQoXbHGqTHzo81FRRI6elNZ2fX1TWdV/HMM03LhZiT8rxAUOAD8dOLEsF/7PxY00MPmZmETZ/5tHmzZGb1mgleyclSnz6SGYu/+WbpzjtNGHrTRoLCA2vXrrULABjmU2LN7CazxCPGKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAB0zDWz6+rq7BJRU1NjvyYmJtolCCJ1NDQ0KAgidQSlnpa15OfnK9lcfT4gNZWWlgbycQpiTcXFxWpsbJTf1dfXR71tQjgcDkezYWFhoYqKitqsLykpUVpaWvtaCADwVCgUUl5enqqrq5WRkdExRxQFBQWaO3fueUcUmZmZKi8vV0pKioJyRJGVlaWcnJxAvFuNvFMNSj0GNflDkGsqKyvrdEcUUQdFamqqXVozf7Ag/NFaMk/soDy5g1iPQU3+EMSaGgOyz2tPDcEYXAAAXDIEBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0ERY2fOnNHUqVM1ePBgHTx40OvmwKislC6/XEpIaFpuu01qfeFHcz8n59w26enSSy951WLwWoopgiLGV5SaNGmSvXxsRUWFxowZowMHDnjdLFx7rfSf/3nufmmptHLl+dusWCFt3nzu/g9/KA0cGLs24jy8lmKLoIiRuro6TZgwQevWrWu+Pm1tba19gu/bt8/r5mH6dGn8+HP3H3lEirxLNV/N/Yhx46QHHoh9G2HxWoo9giJGJk+erA0bNthrj+fm5tp1mzZt0unTp5Wdna0jR4543USsXi316tV0OxSSpk0zeyXp3nul06eb1vfsKT3+uKfN7Ox4LcUeQREjs2fP1qJFi7R48eLmdSNHjrQXa58+fbr69+8vP15zNz8/X+Xl5QqEfv2kRx89d3/nTmnECOl///fcOvP9q65SELpuZs6cqcOHD8tvgvhaindJXjegszCHxWZpbcSIEXbxm7Nnz2ratGm2jzgxMVFLly5VIEyaJE2dKv3iF033X3zx3Pfy8qQvfEFBsGfPHq1evVrr16/X1q1bda0Zp/GJoL2W/IAjCryv2SZTpkyxITFjxgwVFxcrUMzAtTm6aOnKK9sOcPuYeQduQqKqqkqjRo2yA8LAe+GIAu1mZpuYgcTu3bsrISFBDz74YFQ/N2vWLA30w0wh08f91lvnrzP3X3lFuuEGxbv58+frxIkTUW2blZWlXbt2afTo0dqxY4cGDRp0ydsH/yEo0O5xie3bt9vb1dXVWtmOd9kTJ06M/6BoaGgavDaD2Bdav3u3lJqqeLZmzRodPXq0XT9z7NgxO9ZEUOBC6HpCu5jxiC1btqhnz57q0aOHdu7cqXA4HNVyoX7luLNwofS3v527//Wvn7tdViZ9+9uKd2bWTzSPR2RKqVFUVNQ8gwhojaBAuw0bNswOgCYlJSknJ0cvvPCCAsHU0XK85ctfbhqv+MpXzq1btkx67jn53alTpzRu3Dh7dLhkyRItWLDA6yYhjhEUeF+GDBmibdu2qUuXLvYIw/feflu67z4znavp/jXXSD/6UdNt8/VDH2q63djYtF1trfzs0KFD2rt3r5YtW6ZHWp5MCFwAYxR438xAqDkTtnfv3vK9/Hyz92y6nZgoPfGE1K1b033zOVD//d/S6NFNQfLyy9KcOdJjj8mvhg4dasMiEI8dLjmOKPCBBGJHs3Gj9NOfnrtvQmDUqPO3ufnm8z/Gw5zFvWGD/CwQjx1igqDwwNq1a+1gIuKE+ewm83hEFvOBfxfyve+dv93tt8e6pWiF11JsEBQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOCUpCjV1dXZJaKmpsZ+TUxMtEsQROpoaGhQEETqCEo9BjX5Q5BrSgzY/i4aCeEoLzhbWFiooqKiNutLSkqUlpbWvhYCADwVCoWUl5en6upqZWRkdMwRRUFBgebOnXveEUVmZqbKy8uVkpKioCRsVlaWysrK1NjYKL8LWj0GNfkDNcW/+vr6qLeNOihSU1Pt0pr5gwXhjxbkmoJWj0FN/kBN8as9NQSjsw0AcMkQFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADglKUp1dXV2iaiurrZfGxoaFBSJiYkKhUKqr69XY2Oj/C5o9RjU5A/UFP8i++5wOHzxjcNRWrhwofnfWFhYWFgUnKWysvKi+/8E88/7OaI4efKkBgwYoFdffVXdu3dXENTU1CgzM1OvvfaaMjIy5HdBq8egJn+gpvhneoWuvvpqnThxQj169OiYrqfU1FS7tGZCIgh/tJZMPUGqKWj1GNTkD9Tkjy61i24Tk5YAAHyLoAAAXJqgMN1QCxcuvGB3lF8Fraag1WNQkz9QU7DqiXowGwDQOdH1BABwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIBc/j+c6Peoa2v/OgAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "my_policy = np.ndarray(\n", " [\n", " A_RIGHT,\n", " A_RIGHT,\n", " A_RIGHT,\n", " A_DOWN,\n", " A_DOWN, # First row\n", " A_UP,\n", " A_DOWN,\n", " A_DOWN,\n", " A_LEFT, # Second row\n", " A_UP,\n", " A_RIGHT,\n", " A_DOWN, # Third row\n", " A_UP,\n", " A_LEFT,\n", " A_RIGHT,\n", " A_RIGHT,\n", " A_RIGHT, # Fourth row\n", " A_UP,\n", " A_LEFT,\n", " A_DOWN,\n", " A_RIGHT,\n", " A_UP, # Fifth row\n", " ],\n", ")\n", "\n", "V_my_policy = policy_evaluation(policy=my_policy, P=P, R=R, gamma=gamma)\n", "\n", "plot_values(V=V_my_policy, title=\"Value function: my policy\")\n", "plot_policy(policy=my_policy, title=\"My policy\")\n" ] }, { "cell_type": "markdown", "id": "e61f5ee8-f9cd-4fbc-96c0-0a8d661bd1e5", "metadata": {}, "source": [ "**Exercise 10.** (optional) How can we find an optimal policy?\n", "(We will discuss this question next week, but you can already start thinking about it!)" ] }, { "cell_type": "markdown", "id": "00ae548b", "metadata": {}, "source": [ "To find an optimal policy $π^*$ (a policy that yields the highest possible expected return from every state), we generally use one of two main dynamic programming algorithms:\n", "\n", "1. **Policy Iteration**: This method alternates between two steps until convergence:\n", "\n", "- *Policy Evaluation*: Calculate the value function Vπ(s) for the current specific policy (as we did in Exercise 8).\n", "\n", "- *Policy Improvement*: Update the policy to be greedy with respect to the current values. For every state s, we choose the action a that maximizes the expected next value:\n", " $$π_{new}​(s) = argmax​_{a} \\sum_{s\\prime} ​P({s \\prime}∣s,a)[R(s)+ \\gamma V_{\\pi}({s\\prime})]$$\n", "\n", "1. **Value Iteration**: Instead of evaluating a specific policy until convergence every time, we iteratively update the value function directly using the *Bellman Optimality Equation*:\n", " $$V_{k+1}​(s) = max_a ​(R(s)+ \\gamma \\sum_{s\\prime} ​P(s\\prime∣s,a)V_k​(s\\prime))$$\n", "\n", " Once the values converge to the optimal values $V^{*}$, we simply extract the optimal policy by acting greedily towards those values." ] } ], "metadata": { "kernelspec": { "display_name": "studies", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.12" } }, "nbformat": 4, "nbformat_minor": 5 }