mirror of
https://github.com/ArthurDanjou/ArtStudies.git
synced 2026-01-14 13:54:06 +01:00
1401 lines
133 KiB
Plaintext
1401 lines
133 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "44b75d44",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Lab 2 - Maze Game as a Markov Decision Process Part 1\n",
|
||
"\n",
|
||
"## **1. Objectives**\n",
|
||
"\n",
|
||
"In this lab, we will:\n",
|
||
"\n",
|
||
"- Model a simple **maze game** as a **Markov Decision Process (MDP)** by defining:\n",
|
||
" - **States**\n",
|
||
" - **Actions**\n",
|
||
" - **Transition probabilities**\n",
|
||
" - **Rewards**\n",
|
||
"\n",
|
||
"- Implement **policy evaluation** to compute the value function of a given policy.\n",
|
||
"\n",
|
||
"This week, we **do not** improve the policy and search for an optimal one yet. \n",
|
||
"We will continue working on the Maze Game **next week**, where we will use these components to compute an **optimal policy**.\n",
|
||
"\n",
|
||
"We consider a **discounted MDP** with discount factor $\\gamma \\in (0,1)$.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 73,
|
||
"id": "100d1e0d",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import matplotlib.pyplot as plt\n",
|
||
"import numpy as np\n",
|
||
"\n",
|
||
"np.set_printoptions(\n",
|
||
" precision=3, suppress=True\n",
|
||
") # (not mandatory) This line is for limiting floats to 3 decimal places, avoiding scientific notation (like 1.23e-04) for small numbers.\n",
|
||
"\n",
|
||
"# For reproducibility\n",
|
||
"rng = np.random.default_rng(seed=42) # This line creates a random number generator.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1018deab",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 2. Maze definition and MDP formulation\n",
|
||
"\n",
|
||
"We consider a small 2D maze on a grid. The agent is a **robot** that moves on the grid.\n",
|
||
"\n",
|
||
"- `S` : start state\n",
|
||
"- `G` : goal state, with positive reward\n",
|
||
"- `#` : wall (not accessible)\n",
|
||
"- `.` : empty cell\n",
|
||
"- `X` : \"trap\" (negative reward)\n",
|
||
"\n",
|
||
"At each step, the robot can choose among 4 actions:\n",
|
||
"\n",
|
||
"$$\n",
|
||
"\\mathcal{A} = \\{\\text{Up} \\uparrow, \\quad \\text{Right} \\rightarrow, \\quad \\text{Down} \\downarrow, \\quad \\text{Left}\\leftarrow\\}.\n",
|
||
"$$\n",
|
||
"\n",
|
||
"The movement is deterministic, but here we set a small probability of “error” to make the example more realistic.\n",
|
||
"- With probability $1 - p_{\\text{error}}$, it moves in the chosen direction.\n",
|
||
"- With probability $p_{\\text{error}}$, it moves in a random *other* direction.\n",
|
||
"- If the movement would hit a wall or go outside the grid, the agent stays in place.\n",
|
||
"\n",
|
||
"We will represent the MDP with:\n",
|
||
"\n",
|
||
"- A list of **states** $\\mathcal{S} = \\{0, \\dots, n_{S - 1}\\}$, **each corresponding to a grid cell.**\n",
|
||
"- For each action $a$, a transition matrix $P[a]$ of size $(n_S, n_S)$, where\n",
|
||
" $$\n",
|
||
" P[a][s, s'] = \\mathbb{P}(S_{t+1} = s' \\mid S_t = s, A_t = a).\n",
|
||
" $$\n",
|
||
"- A reward vector $R$ of length $n_S$, where $R[s]$ is the immediate reward obtained when **leaving** state $s$.\n",
|
||
"\n",
|
||
"We will use a discount factor $\\gamma = 0.95$.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ca4fa301-c14f-44ec-b04f-b01ca42d979a",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 2.1 Define the maze \n",
|
||
"\n",
|
||
"Let us now define the maze as follows."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 74,
|
||
"id": "f91cda05",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"maze_str = [\n",
|
||
" \"#######\",\n",
|
||
" \"S...#.#\",\n",
|
||
" \"#.#...#\",\n",
|
||
" \"#.#..##\",\n",
|
||
" \"#..#..G\",\n",
|
||
" \"#..X..#\",\n",
|
||
" \"#######\",\n",
|
||
"]\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "99820cf4-292d-49ba-b662-f9f05f901f62",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Exercise 1.** Compute the dimensions of the maze (complete the “TO DO” parts):\n",
|
||
"- How many rows does the maze have?\n",
|
||
"- How many columns does the maze have?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 75,
|
||
"id": "564cb757-eefe-4be6-9b6f-bb77ace42a97",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"7\n",
|
||
"7\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"n_rows = len(maze_str)\n",
|
||
"print(n_rows)\n",
|
||
"n_cols = len(maze_str[0])\n",
|
||
"print(n_cols)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 76,
|
||
"id": "26c821d3-2362-4b60-8c77-3d09296d130d",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Maze:\n",
|
||
"#######\n",
|
||
"S...#.#\n",
|
||
"#.#...#\n",
|
||
"#.#..##\n",
|
||
"#..#..G\n",
|
||
"#..X..#\n",
|
||
"#######\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(\"Maze:\")\n",
|
||
"for row in maze_str:\n",
|
||
" print(row)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "adc49d58-2730-41d8-96fb-ca7c9cb4fcdf",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 2.2 Map each walkable cell (not a wall '#') to a state index\n",
|
||
"\n",
|
||
"Now we convert the maze grid into state indices for the MDP.\n",
|
||
"\n",
|
||
"\n",
|
||
"The cells where the robot is allowed to stand are \n",
|
||
"\n",
|
||
"- . : empty space\n",
|
||
"\n",
|
||
"- S : start\n",
|
||
"\n",
|
||
"- G : goal\n",
|
||
"\n",
|
||
"- X : trap\n",
|
||
"\n",
|
||
"Everything else (i.e., #) is a wall and cannot be a state in the MDP.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 77,
|
||
"id": "7116044b-c134-43de-9f30-01ab62325300",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"FREE = {\n",
|
||
" \".\",\n",
|
||
" \"S\",\n",
|
||
" \"G\",\n",
|
||
" \"X\",\n",
|
||
"} # The vector Free represents cells that the agent is allowed to move into.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1c9ad05e-9c6c-4e00-918c-44b858f45298",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Dictionaries to convert between grid and state index**\n",
|
||
"\n",
|
||
"We now want to identify all **valid states** of the maze (all non-wall cells). \n",
|
||
"To do this, we need two mappings:\n",
|
||
"\n",
|
||
"1. `state_to_pos[s] = (i, j)`: Given a state index $s$, return its grid coordinates (row, column).\n",
|
||
"2. `pos_to_state[(i, j)] = s`: Given coordinates (i, j), return the corresponding state index $s$.\n",
|
||
"\n",
|
||
"These two dictionaries allow easy conversion between **MDP state indices** and the **physical maze positions**. "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 78,
|
||
"id": "a1258de4",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Number of states (non-wall cells): 22\n",
|
||
"Start state: 0 at (1, 0)\n",
|
||
"Goal states: [16] at (4, 6)\n",
|
||
"Trap states: [19] at (5, 3)\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"state_to_pos = {} # s -> (i,j)\n",
|
||
"pos_to_state = {} # (i,j) -> s\n",
|
||
"\n",
|
||
"start_state = None # will store the state index of start state\n",
|
||
"goal_states = [] # will store the state index of goal state # We use a list in case there are multiple goals\n",
|
||
"trap_states = [] # will store the state index of trap state # We use a list in case there are multiple traps\n",
|
||
"\n",
|
||
"s = 0\n",
|
||
"for i in range(n_rows): # i = row index\n",
|
||
" for j in range(n_cols): # j = column index\n",
|
||
" cell = maze_str[i][j] # cell = the character at that position (S, ., #, etc.)\n",
|
||
"\n",
|
||
" if (\n",
|
||
" cell in FREE\n",
|
||
" ): # FREE contains: free cells \".\", start cell \"S\", goal cell \"G\" and trap cell \"X\"\n",
|
||
" # Walls # are ignored, they are not MDP states.\n",
|
||
" state_to_pos[s] = (i, j)\n",
|
||
" pos_to_state[(i, j)] = s\n",
|
||
"\n",
|
||
" if cell == \"S\":\n",
|
||
" start_state = s\n",
|
||
" elif cell == \"G\":\n",
|
||
" goal_states.append(s)\n",
|
||
" elif cell == \"X\":\n",
|
||
" trap_states.append(s)\n",
|
||
"\n",
|
||
" s += 1\n",
|
||
"\n",
|
||
"n_states = s\n",
|
||
"\n",
|
||
"print(\"Number of states (non-wall cells):\", n_states)\n",
|
||
"print(\"Start state:\", start_state, \"at\", state_to_pos[start_state])\n",
|
||
"print(\"Goal states:\", goal_states, \"at\", state_to_pos[goal_states[0]])\n",
|
||
"print(\"Trap states:\", trap_states, \"at\", state_to_pos[trap_states[0]])\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "721b968c-a355-46eb-aae4-5950441ba604",
|
||
"metadata": {},
|
||
"source": [
|
||
"*Hint.* If you don’t know what a dictionary is in Python, try the following code to help you understand."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 79,
|
||
"id": "68744dd6-7278-4c20-8b82-34212685352f",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"value2\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"my_dict = {\"key1\": \"value1\", \"key2\": \"value2\"}\n",
|
||
"print(my_dict[\"key2\"])\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "0c76f4e1-b0ba-49c5-b9d5-cfb523024ba9",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Exercise 2.** Read the program above and answer the following questions:\n",
|
||
"1. What is the purpose of state_to_pos and pos_to_state?\n",
|
||
"2. Why do we only assign states to cells in FREE?\n",
|
||
"3. What would happen if the maze had multiple goal cells?\n",
|
||
"4. What is the total number of states (n_states) in this maze? Does this match the number of non-wall cells you can count visually?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "4c26a18f-2d03-401c-8eae-f9a17ac55f6d",
|
||
"metadata": {},
|
||
"source": [
|
||
"1. What is the purpose of `state_to_pos` and `pos_to_state`? These dictionaries establish a bijective mapping between the mathematical representation of the state and its spatial representation:\n",
|
||
"\n",
|
||
" `state_to_pos`: Maps the scalar state index `s` (an integer used for matrix/vector operations in RL algorithms like Q-learning) to the grid coordinates (i,j).\n",
|
||
"\n",
|
||
" `pos_to_state`: Maps the grid coordinates (`i,j`) (used to calculate movement and dynamics within the 2D grid) back to the unique state index s.\n",
|
||
"\n",
|
||
"2. Why do we only assign states to cells in FREE? In a Markov Decision Process (MDP), walls (#) are obstructions, not valid states.\n",
|
||
"\n",
|
||
" The agent can never \"be\" in a wall, so assigning a state index to a wall would needlessly increase the dimensionality of the state space (∣S∣).\n",
|
||
"\n",
|
||
" Excluding walls ensures the transition matrices and value vectors remain compact and contain only reachable positions.\n",
|
||
"\n",
|
||
"3. What would happen if the maze had multiple goal cells?\n",
|
||
"\n",
|
||
" In the code: The logic is robust. Since goal_states is initialized as a list (`[]`), the code would simply append the state index `s` of every `G` cell found during the iteration. The list would contain multiple integers representing all terminal states.\n",
|
||
"\n",
|
||
" Caveat: While the logic holds, the final print statement in the provided script (`state_to_pos[goal_states[0]]`) would only display the coordinates of the first goal found, ignoring the others in the console output.\n",
|
||
"\n",
|
||
"4. What is the total number of states (`n_states`) in this maze? Does this match the number of non-wall cells you can count visually?\n",
|
||
"\n",
|
||
" `n_states` represents the total count of walkable cells (Start, Goal, Trap, and empty space).\n",
|
||
"\n",
|
||
" Yes, this value matches exactly the number of non-wall cells visible in the maze, as the counter s is incremented precisely when a cell is found in the FREE set."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "6d0fa298-7b7c-44fc-bbed-15ea002037c2",
|
||
"metadata": {},
|
||
"source": [
|
||
"-----\n",
|
||
"\n",
|
||
"The following function `plot_maze_with_states` creates a figure showing:\n",
|
||
"- the maze walls and free cells\n",
|
||
"- the state index for each non-wall cell\n",
|
||
"- special labels and colors for S (start state), G (goal state), and X (trap state). "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 80,
|
||
"id": "fc61ceef-217c-47f4-8eba-0353369210db",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGbCAYAAAAr/4yjAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAK99JREFUeJzt3Qd4VFX6x/E3CS1CEiSAofcS+spKEZCmgiBZKQqKiIuCrIDArmBBKSu661IUQYoNpFhoYhCpSlEEFekIiHThTwslEaRm/s97xjlMMAlRk9wp38/zXObOzZA5d+7k/O4pdybE5XK5BAAAEQl1ugAAAN9BKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUowHEhISEydOjQDD+2d+/eEoyaNGlilqz8nfv27TOv8ZQpUzL1eeA/CAUfoX+E+seoy5dffvmbn+unkZQoUcL8/O6775ZA9tVXX5mQOH36tF8+9/jx46lU4bdyOF0ApJQnTx557733pGHDhim2r1y5Un766SfJnTu3BJpffvlFcuTIkaJiHjZsmDz88MOSP3/+bC1LZjy3hkLBggXN78hMS5YskaxWqlQpczxy5syZ5c8F30RLwce0atVKZs2aJZcvX06xXYOidu3aEhMTI4EYhN6hgNTlypXLLFlJW6J6PMLCwrL0eeC7CAUfc//990tCQoIsXbrUbrt48aLMnj1bHnjggVT/z8iRI+XWW2+V6OhoCQ8PN+Ghj/emZ62e7qlrF+/+/AsXLsiQIUOkfPnyplWiXVYDBw4029Pz2muvmYrEu9tl1KhR5vf/85//tNuuXLkiERER8tRTT9lt3mXQ2wEDBpj1MmXK2DJqX7e3efPmSbVq1UwZq1atKosWLZKMGDt2rHn8DTfcIDfeeKP89a9/NYGbkeeePHmyNGvWTAoXLmyet0qVKjJhwoQUv7906dKybds207Lz/H/vPnt9ffr162deV/0d+jq//PLLkpyc/Lv7/1esWGF+/8yZM+XFF1+U4sWLmwq9efPm8uOPP/7m/7/xxhtSrlw58x6pU6eOfPHFF795TFpjCjt27JD77rtPChUqZP5/pUqVZNCgQSkec+jQIenWrZvcdNNN9ri88847v+sYwHmcnvkYrVTq168v77//vtx1111m28KFC+XMmTPSqVMnU/lea8yYMRIXFyedO3c2AfLBBx/IvffeK5988om0bt3aPOaxxx6T22+/PcX/04p0xowZppJTWjHp79ExjR49ekhsbKxs2bJFXnnlFfnhhx9MRZyWRo0amf+v/9cz5qGVTmhoaIrKZ8OGDfLzzz/LbbfdlurvadeunXku3X99Xu2GUVoZeehzzJ07Vx5//HETMPqatG/fXg4cOGCCMS1vvvmmPPHEE9KhQwfp27evnD9/XjZv3ixff/21CdzrPbcGgFZm+hppy2b+/PmmDLrfvXr1Mo959dVXpU+fPpIvXz5baWolqc6dOyeNGzc2lacej5IlS5ruqmeeeUb+7//+z/zfP+K///2veZ2ffPJJ8z753//+Z94Lul8eb7/9tnlOPXnQUNqzZ4/ZjwIFCpiASo++Rnp8tUtJ3xf6Ht29e7fZfw0jdfToUalXr56dCKCvmb5vH3nkEUlMTDTPmZFjAB+g36cA502ePFm/18L17bffusaNG+eKiIhwnTt3zvzs3nvvdTVt2tSslypVytW6desU/9fzOI+LFy+6qlWr5mrWrFmaz7dr1y5XVFSU64477nBdvnzZbJs2bZorNDTU9cUXX6R47MSJE03ZVq9enebvu3LliisyMtI1cOBAcz85OdkVHR1tyh4WFuZKSkoy20ePHm2e49SpU/b/6u8eMmSIvT9ixAizbe/evb95Ht2eK1cu148//mi3bdq0yWwfO3asKz1/+9vfXFWrVk33Mek997Wvs2rRooWrbNmyKbbpczRu3Pg3j33hhRdcefPmdf3www8ptj/99NPmNTpw4EC6ZdPf6f17ly9fbsoaGxvrunDhgt0+ZswYs33Lli32/VC4cGFXrVq1UjzujTfeMI/z/p2637pN348et912m3k/7t+/P0V59Bh7PPLII64iRYq4Tpw4keIxnTp1Mu8zz2uXkWMAZ9F95IO0ma6DfXqmn5SUZG7TO4vS5rzHqVOnzNmintmtX78+1cefPXtW2rZta5ruelbs6T/WsQxtHVSuXFlOnDhhF+0yUcuXL0+zDHqmqmehq1atMve3b99uusGefvppM3NqzZo1Zru2GrTb588MIGuLR7tBPGrUqCGRkZHm7Dc9+pw6WP/tt9/+oef1fp31NdbXRs/89Xn1/vXo66vHRV9379dX90e71Tyv3e/197//PcVYgz6H8rwe69atk2PHjknPnj1TPE67FKOiotL93cePHzfl0m4hbdl401aB0uM7Z84cadOmjVn33rcWLVqY18bzXvyzxwBZj+4jH6RNb60otJ9Vuxy0wtDmdlo0NIYPHy4bN25M0ffv+aO9Vvfu3U3zX7suvLtbdu3aZSpz764ab1qxpEcrI+2X10DTyr9IkSJy8803S82aNc39O+64w3T9aOj9GddWTkorWg3E9Og4xrJly0x/uvbl33nnnSZsGzRokKHnXb16tRlv0YDT4+JNK77rVbD6+mpXyR99fTP6euhroTyvx/79+81thQoVUjxOu4PKli2b7u/2BIsGeXrBoWMlOmahS3r79mePAbIeoeCj9A9FK+8jR46YsYW0zqy1stW+Ye2j16mQWhHrH7sOiqY2eKfjD9o6mD59utSqVSvFz7RvvHr16jJ69OhUn+t6fc86jfbSpUum0tRyec5Y9Vbv62ClViCe7X9UWjNjrvfNstoK2rlzpwlRHU/Rs1t9zQYPHmymoaZHQ1QHcLUVpa+PvhZ61v3pp5+a8YeMDBTrYzQYdeA+NRUrVpTsfD0yi2ffH3zwQenatWuqj9HW3J89BsgehIKP0u4dHRhcu3atfPjhh2k+Tv+odMbJ4sWLU1zDoKFwLa2YdTBSB/10IPJa2iWzadMmU/ml1cpIj579aUWpz6OLZyaPBpYOMH722Wf2fnr+yHNnVN68eaVjx45m0UF5HVzWwVId7NXXMa3n1kFVbYXFx8enODNPrUstrd+hr68Osl874J8d1x54WiqerkClAb53717TkkuLpyWxdevWNB+jLR8d8NcWbUb27XrHAM5iTMFH6ewVne2i3THaV5veWaJWQvoH6T2t8NqZQjq7Rbtt9Gx+xIgRqf4u/bnOjNEK/FraJaRjEenRP+hbbrnFtER0JpB3S0H/v84S0opRWzPXqzRUZl/RrGMc3jTAdFqpnlFrBZnec3vOxr3PvrXLKLXw1d+RWtn19dVWlAb4tfTx116bkll0yqdW3BMnTjSVsIdOO73ea6z/T0Ncp5bqMfXmeS30tdHZX3qCklp4aOvw9xwDOIuWgg9LqynuTaecandGy5YtTZeT9t2+/vrrpr9W+689dBqg/nFq14VOWb22aa9Lly5dzJx3HZDUM2Dt59Ww0W4f3a6VmVYw6dEA0CmS2r+uXVFKp7zqvHbtNsjIVb56nYXSKZ06DVe7wzQYPRX2H6X913rxn+6XThPV8ZNx48aZ11DPdNN7bv2/WoHpurbg9Ixfw1P3TQP32vJroOs4jx4HfYyeoWvLSVsaOmVXXwd9nAatTvvV60o0zD3TYDOT7oOWRcut5dAzdG0haKBdb0xBaZjryYSOD+mUVL2GQ8u6YMECM46l9Jjre6Zu3bqm21Mr+pMnT5oBZh1D0PWMHgM4zOHZT0hlSmp6UpuS+vbbb7sqVKjgyp07t6ty5crmd+kUT+/Dq9MO9X5qi/d0UJ2++PLLL5tpg/r7brzxRlft2rVdw4YNc505c+a6+7FgwQLzO++6664U2x999FGzXct6rWvL4Jm+WaxYMTN91XuKqK736tUr1dela9eu6ZZt0qRJZnqlTpXVfStXrpxrwIABv9mvtJ47Pj7eVaNGDVeePHlcpUuXNq/TO++885sprEeOHDHHSKdxXjvlU6fmPvPMM67y5cubqbUFCxZ03Xrrra6RI0ea1/6PTEmdNWtWiselNq1UjR8/3lWmTBmz73/9619dq1at+s3vTOv/bt261dW2bVtX/vz5zf5XqlTJ9fzzz6d4zNGjR82xKVGihCtnzpyumJgYV/Pmzc3U1997DOCcEP3H6WACAPgGxhQAABahAACwCAUAgEUoAAAsQgEA8PuuU9DL2A8fPmzmEWfl1aYAgKyhE031AzaLFi1qPsDyT4WCBsL1PvcGAOD7Dh48aL6Q6U+FgudKQ/1cE742EQD8j36Mil5dfr0rxzNUw3u6jDQQ+EJvAPBf1xsCYKAZAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAK4f8Dh9++KFERkb+nv8C4BpxcXESaOLj4yXQxAXgccoIWgoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAAH/sA/GywsUrF2XkVyNl+ubpsv/MfgkLCZPCeQtL9Zuqy9DGQ6VmTE2niwgAQcPxlsKAJQNk0OeDZPuJ7VIsopiUzl9ajp09JvN2zJNdJ3c5XTwACCqOtxQ+3PahuR1822AZ1nSYWXe5XPLVwa9MiwEAEEShkOxKNrdL9iyRW4rdIrcUvUVuyneTNCjZwOmiAUDQcbz76PFbHje3a39aK23ebyMxo2Kk8rjK8sLKF+T85fNOFw8AgorjoTC0yVCZe99caVOxjUTmdn+r286EnTJ4xWDp+UlPp4sHAEHF8VBQbWPbSvz98XLqqVPyzaPfSPXC1c12HWwGAARRKDz3+XOy8chGd2FCQs24QsXoiuZ+VJ4oh0sHAMHF8YHmt9a/JS9+8aIUvKGglIwqaaaj/pT4k/nZA9UecLp4ABBUHG8pDG82XP5W6W8SkStCdpzYYUKhUnQlGdJ4iLzQ7AXxV6tWrZJWrVpJoUKFJCQkxCwTJ04UfzVq1Chp0qSJFClSRHLnzi2lSpWSrl27yp49e8Sfvfrqq1KzZk3Jnz+/2a/ixYvLvffeK5s3b3a6aEjFfffdZ/+eOnXq5HRxApLjLYVHb37ULIFm/fr1snTpUilbtqycOHFC/N3YsWPlwIEDUqlSJQkPD5e9e/fK1KlTZcmSJbJz506JjHRPEvA3K1eulOPHj5vjdP78ebMvs2fPls8//9zsb968eZ0uIn41efJkmTVrltPFCHiOtxQCVZcuXSQxMVEWL14sgaB79+6yb98+2b59u2kd9OvXz2w/cuSIfPbZZ+Kv3n//fTl8+LAJ8e+//16effZZs/3kyZOyY8cOp4uHX+3evVueeOIJqV+/vmnNIesQClkkOjranFEHikGDBknJkiXt/UaNGtl17XbxV3ny5JGPPvpI6tWrJ1WqVJGXXnrJbNduv4oV3RMe4KzLly9L586dJTQ0VGbMmCFhYWFOFymgOd59BP9z5coVeeONN8y6drs0b95c/NnRo0fl66+/tvfLlCkj8+fPl4iICEfLBbdhw4aZ4zN9+nRzbJC1aCngdzl79qy0bdvWdIvFxMSYytOfWwqqZ8+ekpycLPv375eOHTua8RK9TUpKcrpoQW/dunXyn//8Rx588EHTWkDWIxSQYTp+0LhxYxME2rWyevVq0+USCHQ2i3aPecYUtm3bZsYb4KytW7ealqkO/ufLl88sOgFAzZkzx9w/c+aM08UMKIQCMkQrSe13/+6778x4wpo1a0zXkT9LSEiQadOmycWLF+22Tz/9NEWrCL5BZ4bp8dBFP0XZM9bgfR8BFgofbP1Abp50s4S/GC4FXi4gHWZ2kN0nd4u/mjt3rpQvX97M7fcYPHiw2eaPzeB27dqZ7hWl3Sp6DYaGhC5vvfWW+CPdj4ceeshco1C9enXTUnjmmWfMz3Q8QfcZznr44YdNpe+96DUySrv49L4ePwTYQPPb69+WR+e7r1Uok7+MJPySIHO2z5EvDnwhm3pukph8MeJvdDqqTqPzpvPhdfHHKXUXLlyw6xs3uj+WxKNly5bij7Qy0QugvvnmG3OsLl26JCVKlDBdZNqN5Kl8gGAS4spA20sruKioKNN3l9kXKenXcRYbXUxOnDsh7WPby+z7ZsvhpMPm47OTLiZJnzp95LW7XsvU5wScFBcXJ4EmPj5eAk1cgB0nPelZtGjRdetxx7uPvj30rQkEpaGgikYUlXrF65n1RT8ucrR8ABBMHA+Fg4kH7br312/qt6+pA2fcMw0AAEEQCmlhRgEABGEolIgsYdf1E1KvXdeP0wYABEko6JfqRIdHm3WdcaR0oFm/s1m1LO+fM1sAwB85Hgq5wnLJS81fsqFQdkxZiX091sw80i/eebrh004XEQCChuOhoHrU7iHT206XWjG1TCshREKkXWw7+arbV2YmEgAgiC5eU51rdDYLACDIWwoAAN9AKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAIA/9oF4HTt2lJw5c0qg4MvGfV8gHqNA3CcEDloKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahkAWGDh0qISEhqS6XL192uni4xvHjx6VPnz5SqlQpyZUrlxQsWFCaN28ue/bsEX+zb9++NN97uuh709+cPXtWBg4cKBUqVJAbbrhBoqKipEaNGjJixAhxuVxOFy/g5HC6AIFMK5dy5cql2KZ/mPAdJ06ckLp168revXtNIFSsWNFUNGvWrJHDhw9L2bJlxZ/kzp3b7I+306dPy86dO816kSJFxN/06tVL3n33XbNetWpVOXPmjGzZssUERZ48eUygI/MQClmodevWMmXKFKeLgXQ899xzJhC0slm6dKmtNC9evOiXZ6Fa/rVr16bY1rt3bxMKN954o3Tu3Fn8zZdffmluW7ZsKQsXLpRffvlFChQoIOfPn5f9+/c7XbyAQ/dRFpozZ46Eh4ebP9S7775bNmzY4HSR4EUr/ZkzZ5r1EiVKyB133CF58+aVmjVrmmOnZ93+LiEhQSZPnmzW//GPf0i+fPnE3zRq1MjcLlq0SKpVq2ZacxoIuv1f//qX08ULOIRCFgkLC5OYmBgpXbq0HDlyRBYsWCD169cnGHxsLOHUqVO2wtFuFj2b3rx5szzwwAMye/Zs8Xfjx4+Xc+fOmYDz126WiRMnykMPPWTWt23bJj/99JPp6tNxBT1eyFyEQhbQCuXYsWOya9cu2b59u6lw1IULF+T11193unj4lfegf2xsrBlY1kXX1bhx48Sfeb/fHnzwQXOS4o9eeeUVmTZtmjRo0MD8XWkwREREmH17+umnnS5ewCEUsoA2b7XP06NFixYSHR1t1g8cOOBgyeCtUKFC5oxTaZeRruui656ZPP5s6tSpcvToUTO5wV+7WbSV8/zzz5uuvvbt25tjVqVKFRMQatmyZU4XMeAQClng5ZdfTlH56wCm9u0q7U6Cb8iZM6fcdtttZl27jC5dumQWXVc6BdJfaSU6atQoO+HB0/rxx1DwtOi+++47c6vjCdpaUDoGhMxFKGSBCRMmmMpf573rWY22FDxv4H79+jldPHgZPny4aR18//33UqZMGbPouo4JPfvss+Kv5s+fb6ehDhgwQPx5WrcnuGfMmGGCWv+2du/ebbZ17drV4RIGHkIhC2hlohc/6Vmn9lFrOOhUQD3T0ZCA79A5/Z9//rk0adLEDDrrWejtt98uq1evlqZNm4q/GjlypLmtU6eOrVT91bx588w1Cdotq9eO6HRhPW7Tp0+Xxx9/3OniBZwQVwYmYycmJpqrCHWesDa5A0V8fLzTRch0cXFxEkgC8RjBP8QF2N+SnqTqpBe9+C8yMjLNx9FSAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALByXF0FgD8m0L7kPpjRUgAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoZJGff/5ZhgwZIpUrV5bw8HApWrSo/OMf/5BTp045XbSgtWrVKmnVqpUUKlRIQkJCzDJx4sQUj7l06ZIMGzZMypYtK7ly5ZLixYtL//79zfH0x/2ZNGmSNGzYUPLmzWsfs2PHDsfKDN9HKGSRNm3ayL///W/58ccfpWLFiqZS0T/YO++8Uy5fvux08YLS+vXrZenSpVKgQIE0H9OtWzcZOnSo7N+/3wTDsWPH5NVXX5W7775bkpOTxd/2Z+HChbJhwwYTHEBGEApZ4Pvvv5cVK1aY9TFjxsimTZvku+++M/fXrVsnM2fOdLiEwalLly6SmJgoixcvTrOSnT59uj1uekY9Z84cc3/lypUyb9488af9UePHjzeP0aADMoJQyALeZ5ShoaEpbtWyZcscKVewi46ONl156Z1Ve7Rv397ctm7dWvLkyWPWFy1aJP60P0q7LcPCwrKtTPB/hEIWiI2NlWrVqpn1Pn36SK1ateTmm2+2Pz906JCDpUNaDh48aNcLFy5sw7xgwYJm/cCBA46VDcguhEIW0DMzPevs3LmzqVD27NkjjRo1knLlypmf58yZ0+ki4ndwuVxOFwHINjmy76mCi85a8fRPq/Pnz0tMTIxZr1SpkoMlQ1pKlChh13WAuUiRIqYrMCEhwWwrWbKkg6UDsgcthSyig5ZJSUlm/cqVKzJgwAA5c+aMud+xY0eHS4fUtGzZ0q57BpgXLFhgAv3anwOBilDIIu+8847pl65evbppIYwbN85s79evn9SpU8fp4gWluXPnSvny5aVJkyZ22+DBg8027eqrXbu23H///WZ73759zdiQZ8BZu//uuece8af9UU899ZS5r7ceLVq0MNtee+01R8oN30b3URbRin/58uVmPEH7pLXC0YvXHnnkEaeLFrR0aubu3btTbDt+/LhZtLtPvfvuu1KhQgWZOnWqeazO7+/QoYMMHz48xQwyf9mfo0eP/uYxngHzkydPZmNp4S9CXBkYRdM3X1RUlGk+B9IgaXx8vASauLg4CSSBeIwCUaC97wKRXq2v06q1GzsyMjLNx/nWqQ8AwFGEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsHJcXUUgCLQvug/EL4QPtGMUqPsUaBITEyUqKuq6j6OlAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAIBsdv68yCuviNx6q0j+/CK5c4uULCly++0io0c7WzY+OhsAslFCgkjz5iKbNrnv33CDSMWKIklJIitXinz2mcg//+lc+WgpAEA26t37aiD07esOiS1bRPbtEzlxQmTyZGfLR0sBALLJ6dMis2a512vWdHcVhXqdmut34Dz8sDiKlgIAZJMffhC5csW93qjR1UC45x6RkJCry5QpzpWRUAAAB4R61b6VKrlbDr6AUACAbFKpkkhYmHv9q6+ubn/5ZZEPPhCfQCgAQDaJihK57z73+rp1IkOGXO1O8hWEQiZYtWqVtGrVSgoVKiQhISFmmThxov15UlKS9OvXT2rXri0FCxaU8PBwqVixojz//PPmZ/64T6pbt25SoUIFyZcvn+TNm1fKlSsnTzzxhJw8edKxcgeTjBwjD32f6fG53uP8YZ+aNGlif+a9NGzYUPzB2LEiNWq41//9b5ECBUT+8hfdL/EJhEImWL9+vSxdulQK6NFNRUJCgowZM0a2bdsmxYsXN5Xorl27ZPjw4dKxY0fxx31SH3/8sVy5ckUqV65swm7Pnj0yduxYeeCBB7K1rMEqI8fIo3fv3ub4BNI+lS1bVurWrWuXqlWrij+IjhZZu9bdZVS7tkhyssiOHSLh4SItWohoBurAs1MIhUzQpUsXSUxMlMWLF6f68zx58siIESPk+PHjsnHjRjl48KDUq1fP/GzhwoVy6tQp8bd9UocOHTIVzbp162T//v32TG316tXZWNLglZFjpGbOnClTp06V+zz9FgGwT0pb2mvXrrXLpEmTxF+Eh4sMHOjuQtLOgl9+Edm7V2TRIpHHHnNf5ewUQiETREdHmy6htMTExMiTTz4pERERNiRuueUWsx4aGio5cuTwu33y7If+YepZWunSpeXLL7802/2lGe/vMnKM9ATkscceM12X2jINhH3y6N+/v+TOndu0GHr06CFHjx7N8vIFA0LBAceOHZM5c+aY9U6dOtmw8EfaDfbNN9+YloK6/fbbzZkpnJecnGzOvC9duiTvvfee5MyZUwKFBkexYsXM2MPevXvlzTfflPr168vZs2edLprfIxSy2e7du82Z9OHDh6VBgwY+O+CXUR988IFcvHhRNmzYINWqVZNly5ZJr169nC4WRMw41sqVK82tTmwIFK+88orpct26datpCT3zzDNmu4bDRx995HTx/B6hkI3WrFljxhL07LpNmzayZMkSv24leOgZaK1ataR79+7m/rRp0+QHvXQTjtr06wfs9O3b10xu8B6I1dlwt+pHdPqhv/zlL6bbSOmsI++JDQcOHHCwZIGBUMgms2fPlmbNmsmJEyekT58+Mm/ePLlBPx7RT3377beyYsUKe19bC9pK8KAZ7zv0WOhy7tw5u+3ChQsp7vtT1+vo0aNTTOX+8MMP7bqObeHPIRQywdy5c6V8+fJm/rTH4MGDzbbOnTubriKd+XH+/HnJlSuX6YPXszRtNeii0/D8bZ90em3Tpk3N1EFtJRQpUkTmz59vHqf3a/rKNfsB7HrHaMqUKeJyueyi3SseEyZMMDPh/G2fNMj+9a9/mfddbGyslCxZ0g6g6/127dqJr1m1SqRVK5FCha5+tlFqvcabN4t06OB+XK5cIsWKXb3QLTv53rQXP6RT6HSswJtOP9VFr0vQs2j9o1S6/vXXX//m//vbPun4QcuWLU0Xxffffy9hYWHmj7J169by7LPPmllVcPYYBeI+6cDyoEGDTNerPu6XX34x18ncc889MnDgQDMjztesXy+ydKleV+H+aOzU6MS9O+90T02NjBTRnr6ff9ZrgbK7tCIhLk9tdZ0DFRUVZSqBQJrBEB8f73QRcB1xcXESaHjfBZeEBPcX6eiM2TJl3NsmTBDp2dO9rjVwlSruC9g6dxZ58033dQxKe8kya9jRU4+fOXNGIjV50sDpHABk8RXM4elceqHdRhoInoDQD83Tz0hq1sz9UdvZjVAAAAft3Hl1/b333K0KtXy5+/OQ9BvZshOhAAAOunz56vojj7hbDToHQD9iW8cVsvsLdwgFAHBQsWJX13/99Bsz9qCzkBQtBQAIInXquGccKf2APKWfGnP8uHu9QoXsLQ+hAABZaO5ckfLlU35fwuDB7m0620gHoYcOdW9/6y293sL91Zz65TsxMSI9emRveQkFAMhCiYn6mWfus38PbQXotkOH3Pf793cHQrVq7o/Q1mmoXbq4Ww6ebqTswsVrAJCFHn7YvVyPDjLr4jRaCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFh+IB5/Gl9zDKXFxcRJILl26lKHH0VIAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSggOC1fLhIaKhISIvLf/17dfuWKSL167u2lSokkJjpZSiDbEQoITk2bivTt614fMkRk82b3ugbE11+7Q+Hdd0UiIx0tJpDdCAUEr//8RyQ2VuTiRZEuXUS++UZk2DD3z/r1E2nSxOkSAtmOUEDwypNHZNo0kRw53C2Fxo31m0hEqlQReeklp0sHOIJQQHCrXVvkuefc6+fPi4SFuYNCAwMIQoQCsGtXyoHmffucLA3gKEIBwW3OHJEZM9zrOttIPfaYyNGjjhYLcAqhgOClFX/Pnu71Vq1E1qwRiY4WOXFCpHt3p0sHOIJQyASrVq2SVq1aSaFChSQkJMQsEydOtD+fMmWK3Z7asmLFCvG3fVK7d++WBx98UEqUKCG5c+eWggULSuPGjeXjjz8Wv/Doo+4AKFBA5K23RIoUEZkwwf2z+fNF3n5bfNmoUaOkSZMmUqRIEfP6lypVSrp27Sp79uyxj7l06ZIMGzZMypYtK7ly5ZLixYtL//795eeffxZ/3adJkyZJw4YNJW/evPa9uWPHDkfLHUgIhUywfv16Wbp0qRTQyiUVWrHWrVs3xaJveo+YmBjxt31yuVxyxx13yIwZM+T48eNStWpVuXLligmTtm3byqZNm8SnaQh88ol7ffx4dyCoe+8V6dzZvd6/v0+PL4wdO9a83vnz55dixYrJgQMHZOrUqdKgQQNJ/PWiu27dusnQoUNl//79JhiOHTsmr776qtx9992SnJws/rhPCxculA0bNpi/K2Q+QiETdOnSxbxhFy9enOrPW7duLWvXrk2x6Fm10oq1cuXK4m/7dOjQIdm7d69Z1zNRDZG5c+fawDh48KD4fCvB5XIvHTum/Nn06e7tWgmVLi2+qnv37rJv3z7Zvn27OZPup9dWiMiRI0fks88+M8dkuu6LiIwZM8acTc/RMRQRWblypcybN0/8bZ/U+PHjzXtTww6Zj1DIBNHR0RIeHp7hxy9atEi2bNli1gcMGCD+uE/a0ilfvrxZHzJkiNx8883Srl07yZEjhzk7veuuu7KxtMFp0KBBUrJkSXu/UaNGdl27XvSM2qN9+/b2BCXPr9Nt9X3ob/ukihYtKmE6dRhZglBwwIgRI8xtzZo1TUvBH+kf5fLly6V27dpy4cIF05w/ffq03HjjjSYg+KPNXtp198Ybb5h17SZq3rx5itZa4cKFzW1oaKhtpWrXjL/tE7IeoZDNtPL8/PPPzfqTTz4p/kr7o3v27Cnfffed9O3b1wxczpo1y4wv9O7d2ye7JgLV2bNnzTiOdvXp+NT8+fPtWXVqtHsv0PYJmYdQyGYjR440tzpjp1OnTuKvtH93wYIFZl1nh+hMkA4dOkjkrx8gt2zZModLGBy0r11nfGmlWbFiRVm9erVU0Y/p+PU95qEDzJ4wT0hIMOve3TT+sk/IeoRCNtLm+syZM826nl1r/7u/OnPmjF1ft26duf3hhx8kKSnJrGtIIGtt27ZN6tWrZ1pr2ve+Zs0a083i0bJlS7vuGWDWID+vH+dxzc/9ZZ+Q9QiFTKCzbnTQVedXewwePNhs6+yZ3ihipgJevnxZoqKipEePHuLP+9S0aVMzfqC0G6l69epmLEG7JnLmzCn333+/+JxVq9wXqelURv1obF2uufbCTEF9+GH31c06IFupksj//qen2OJrdGBfp5oqDWO9rkQrVF3eeustM97jOQ56EhIbG2sHnLXCveeee8Tf9kk99dRT5n2otx4tWrQw21577TXHyh4o/PdU1Yfo9Di9kMub9q3rohcLec6sPW9qDYSIiAjx533S2UnarH/xxRfliy++kF27dpmQ0Gb/c889J7Vq1RKfs369yNKlOmrpvmjtWsePi9Sp477Nl09Epwpv3aq1kMjhw5rq4kt0gN9j48aNKX7maQW8++67UqFCBTPXX4+nzu3Xbr7hw4ebQWdfk5F9Onr06G/em55B85MnT2ZLOQNZiCsDo05aQejZrR4UPQsMFPHx8U4XAdlJ+9JvuMH98RZlyri36RXMno+60IvYevVyr2sYVK0q8uabmuLuT0/V6zK8+ukR2OLi4iSQ6NXtOg1ZT1A9Y3+p8b1TBSCr6OcapXc9iXcXkecs2nOrn56qX+EJBDhCAfDQ8QbtNlJ164poF5inFaEOHXKsaEB2IRQADx1rWLLE/f3N2kLQcQQddNYBaRVAXadAWggFwFv9+iJ6ceHp0zq5Xz9Rzv05SEpnIgEBjlAAvH35pXv8QJ06pZedu9f1oyH4mAUEAUIBwUM/xVU/xM/r2gsZPNi9zXM9iY4haADUqCGi04m/+so980ivZ9CZS0CAIxQQPPSjsHV++68XRxl6TYJu8wwi33mniE7X27lTRK841/vanfTrRV9AoOPiNQQPHTTWJT2jR7sXIEjRUgAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsIL6A/EC7Yu5AeDPoqUAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAItQAABYhAIAwCIUAAAWoQAAsAgFAIBFKAAALEIBAGARCgAAi1AAAFiEAgDAIhQAABahAACwCAUAgEUoAAAsQgEAYBEKAACLUAAAWIQCAMAiFAAAFqEAALAIBQCARSgAAKwckgEul8vcXr58OSMPBwD4GE/97anP/1QoJCUlmdtly5ZlRtkAAA7R+jwqKirNn4e4rhcbIpKcnCyHDx+WiIgICQkJyewyAgCymFb1GghFixaV0NDQPxcKAIDgwEAzAMAiFAAAFqEAALAIBQCARSgAACxCAQBgEQoAAPH4f8EOdr53tgDzAAAAAElFTkSuQmCC",
|
||
"text/plain": [
|
||
"<Figure size 640x480 with 1 Axes>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"def plot_maze_with_states():\n",
|
||
" \"\"\"Plot the maze with state indices.\"\"\"\n",
|
||
" grid = np.ones(\n",
|
||
" (n_rows, n_cols)\n",
|
||
" ) # Start with a matrix of ones. Here 1 means “free cell”\n",
|
||
" for i in range(n_rows):\n",
|
||
" for j in range(n_cols):\n",
|
||
" if maze_str[i][j] == \"#\":\n",
|
||
" grid[i, j] = 0 # We replace walls (#) with 0\n",
|
||
"\n",
|
||
" fig, ax = plt.subplots()\n",
|
||
" ax.imshow(grid, cmap=\"gray\", alpha=0.7)\n",
|
||
"\n",
|
||
" # Plot state indices\n",
|
||
" for (\n",
|
||
" s,\n",
|
||
" (i, j),\n",
|
||
" ) in state_to_pos.items(): # Calling .items() returns a list-like sequence of (key, value) pairs in the dictionary.\n",
|
||
" cell = maze_str[i][j]\n",
|
||
"\n",
|
||
" if cell == \"S\":\n",
|
||
" label = f\"S\\n{s}\"\n",
|
||
" color = \"green\"\n",
|
||
" elif cell == \"G\":\n",
|
||
" label = f\"G\\n{s}\"\n",
|
||
" color = \"blue\"\n",
|
||
" elif cell == \"X\":\n",
|
||
" label = f\"X\\n{s}\"\n",
|
||
" color = \"red\"\n",
|
||
" else:\n",
|
||
" label = str(s)\n",
|
||
" color = \"black\"\n",
|
||
"\n",
|
||
" ax.text(\n",
|
||
" j,\n",
|
||
" i,\n",
|
||
" label, # Attention : matplotlib, text(x, y, ...) expects (column, row)\n",
|
||
" ha=\"center\",\n",
|
||
" va=\"center\",\n",
|
||
" fontsize=10,\n",
|
||
" fontweight=\"bold\",\n",
|
||
" color=color,\n",
|
||
" )\n",
|
||
"\n",
|
||
" ax.set_xticks([]) # remove numeric axes, we don't need.\n",
|
||
" ax.set_yticks([])\n",
|
||
" ax.set_title(\"Maze with state indices\")\n",
|
||
"\n",
|
||
" plt.show()\n",
|
||
"\n",
|
||
"\n",
|
||
"plot_maze_with_states()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "db078d86",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 2.4 Actions and deterministic movement"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "96e7f1f2-9d73-410b-853d-e39f40dfb5da",
|
||
"metadata": {},
|
||
"source": [
|
||
"We first define integer codes for each action. \n",
|
||
"\n",
|
||
"**Exercise 3.** How many possible actions can the agent take in the maze?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "22259ab4-527e-4d7c-bb30-98fb240da6d5",
|
||
"metadata": {},
|
||
"source": [
|
||
"We have four possible actions in the maze. \n",
|
||
"\n",
|
||
"In this following cell, each action is mapped to an integer (0,1,2,3). This makes it easy to store and use actions inside arrays and matrices\n",
|
||
"\n",
|
||
"Here we use Unicode arrow character:\n",
|
||
"\n",
|
||
"- \"\\u2191\" : ↑ (up arrow)\n",
|
||
"\n",
|
||
"- \"\\u2192\" : → (right arrow)\n",
|
||
"\n",
|
||
"- \"\\u2193\" : ↓ (down arrow)\n",
|
||
"\n",
|
||
"- \"\\u2190\" : ← (left arrow)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 81,
|
||
"id": "f7f0b8e4-1f48-4d03-9e5f-a47e59c3e827",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"A_UP, A_RIGHT, A_DOWN, A_LEFT = 0, 1, 2, 3\n",
|
||
"ACTIONS = [A_UP, A_RIGHT, A_DOWN, A_LEFT]\n",
|
||
"action_names = {A_UP: \"\\u2191\", A_RIGHT: \"\\u2192\", A_DOWN: \"\\u2193\", A_LEFT: \"\\u2190\"}"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 82,
|
||
"id": "3773781c-a0cd-48db-967b-d4b432d17046",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"↑\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(action_names[0])"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "4b957f5a-ee39-4437-abc1-4809105ad83c",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Exercise 4.** Now we define a **deterministic movement function** `move_deterministic(i, j, a)`. \n",
|
||
"\n",
|
||
"This function simulates the robot trying to move from (i, j) in direction a.\n",
|
||
"\n",
|
||
"But if the movement hits a wall or boundary, the agent stays in place."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 83,
|
||
"id": "4b06da5e-bc63-48e5-a336-37bce952443d",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def move_deterministic(i: int, j: int, a: int) -> tuple[int, int]:\n",
|
||
" \"\"\"Deterministic movement on the grid. If the movement hits a wall or boundary, the agent stays in place.\n",
|
||
"\n",
|
||
" Args:\n",
|
||
" i (int): current row index\n",
|
||
" j (int): current column index\n",
|
||
" a (int): action to take (A_UP, A_DOWN, A_LEFT, A_RIGHT)\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" (tuple[int, int]): new (row, column) position after taking action a\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" candidate_i, candidate_j = (\n",
|
||
" i,\n",
|
||
" j,\n",
|
||
" ) # It means “Unless the action succeeds, the robot stays in place.”\n",
|
||
"\n",
|
||
" # Now each action changes the coordinates of the robot:\n",
|
||
" if a == A_UP:\n",
|
||
" candidate_i, candidate_j = (\n",
|
||
" i - 1,\n",
|
||
" j,\n",
|
||
" ) # if the action is UP, then row becomes row -1\n",
|
||
" elif a == A_DOWN:\n",
|
||
" candidate_i, candidate_j = (\n",
|
||
" i + 1,\n",
|
||
" j,\n",
|
||
" ) # if the action is DOWN, then row becomes row +1\n",
|
||
" elif a == A_LEFT:\n",
|
||
" candidate_i, candidate_j = (\n",
|
||
" i,\n",
|
||
" j - 1,\n",
|
||
" ) # if the action is LEFT, then column becomes column -1\n",
|
||
" elif a == A_RIGHT:\n",
|
||
" candidate_i, candidate_j = (\n",
|
||
" i,\n",
|
||
" j + 1,\n",
|
||
" ) # if the action is RIGHT, then column becomes column +1\n",
|
||
"\n",
|
||
" # Check boundaries\n",
|
||
" if not (0 <= candidate_i < n_rows and 0 <= candidate_j < n_cols):\n",
|
||
" # If the robot tries to move outside the maze\n",
|
||
" # It will not move and it stays at (i, j).\n",
|
||
" return i, j\n",
|
||
"\n",
|
||
" # Check wall\n",
|
||
" if maze_str[candidate_i][candidate_j] == \"#\":\n",
|
||
" # If the next cell is a wall, the robot stays in place.\n",
|
||
" return i, j\n",
|
||
"\n",
|
||
" return candidate_i, candidate_j # Otherwise, return the new position\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "c9e620e6",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 2.5 Transition probabilities and reward function"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "80bd2bca-7717-4b5f-bffa-76fe86a51d35",
|
||
"metadata": {},
|
||
"source": [
|
||
"Recall that we set the discount factor $\\gamma \\in(0,1)$, that is, the future rewards are multiplied by $\\gamma$, so immediate rewards matter a little bit more than future ones. \n",
|
||
"\n",
|
||
"\n",
|
||
"Moreover, we consider a probability error $p_{\\text{error}}$, which means, with probability $p_{\\text{error}}$, the robot **does not** execute the intended action but one of the 3 other directions (chosen uniformly). With probability $1-p_{\\text{error}}$, the robot executes the action that we asked."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 84,
|
||
"id": "610253e7-f3f7-4a30-be3e-2ec5a1e2ed04",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"gamma = 0.95\n",
|
||
"p_error = 0.1 # probability of the error to a random other direction\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "0d1ceff8-86e0-4c45-83d3-af9fae974608",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now we initialize the state–transition probability : the probability of reaching next state $s'$ after taking action $a$ in state $s$. \n",
|
||
"$$\n",
|
||
" p(s' \\mid s, a)\n",
|
||
" = \\mathbb{P} \\big[S_t=s'\\,|\\, S_{t-1}=s, \\,A_{t-1}=a\\big]\n",
|
||
"$$\n",
|
||
"\n",
|
||
"We store these transition probabilities in the 3D array `P` (`P[a][s, s_next]`), which has shape `(n_actions, n_states, n_states)`:\n",
|
||
"\n",
|
||
"`P[a, s, s_next] = P(S_{t+1} = s_next | S_t = s, A_t = a)`.\n",
|
||
"\n",
|
||
"We also initialize the reward vector `R`, which has length `n_states`, where `R[s]` is the reward received when the agent is in state `s`.\n",
|
||
"\n",
|
||
"In this maze game, we assume that the reward depends only on the current state, which is natural: in navigation tasks, being in a particular location is what matters, not the direction you used to reach it."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 85,
|
||
"id": "7a51f242-fe4e-4e74-8a1f-a8df32b194b8",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Initialize transition matrices and reward vector\n",
|
||
"P = np.zeros((len(ACTIONS), n_states, n_states))\n",
|
||
"R = np.zeros(n_states)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "c08f4af5-a2a7-4baa-b5da-c7ce636d8a4a",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now we assign the reward to each state. \n",
|
||
"\n",
|
||
"For each state index s:\n",
|
||
"\n",
|
||
"1. If s is a goal, then the reward = +1.0\n",
|
||
"2. If s is a trap, then the reward = −1.0\n",
|
||
"3. Otherwise for the normal cell, the reward = −0.01 every time you leave this cell.\n",
|
||
"\n",
|
||
"Recall that rewards are received at the moment the agent executes an action. Here when the agent moves out of the cell, we set reward −0.01. "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 86,
|
||
"id": "49d54d1f-dc29-45b6-ad31-ad0e848f920d",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Set rewards for each state\n",
|
||
"step_penalty = -0.01\n",
|
||
"goal_reward = 1.0\n",
|
||
"trap_reward = -1.0\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "dd571ec8-c36a-4e20-bec6-9e6458dc622b",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Exercise 5.** Why do we set the step penalty to -0.01 in this MDP?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1e8ea171",
|
||
"metadata": {},
|
||
"source": [
|
||
"We set a small negative step penalty (`-0.01`) for two main reasons:\n",
|
||
"\n",
|
||
"- Incentivize Efficiency: It forces the agent to find the shortest path to the goal. By losing a small amount of reward at every step, the agent learns that the faster it reaches the goal, the higher its total cumulative return will be.\n",
|
||
"\n",
|
||
"- Prevent Loitering: It discourages infinite loops or wandering. Without this penalty (i.e., if step reward = 0), the agent might be indifferent between reaching the goal now or in 1000 steps, potentially leading to a policy that never terminates."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "07bfb065-b1af-4df1-885e-780fe250f2fb",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Exercise 6.** We now define the reward vector. Recall that we have already initialized\n",
|
||
"`R = np.zeros(n_states)`.\n",
|
||
"If a state belongs to `goal_states`, we assign the `goal_reward`.\n",
|
||
"If it belongs to `trap_states`, we assign the `trap_reward`.\n",
|
||
"Otherwise, we assign the `step_penalty`. "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 87,
|
||
"id": "b9b7495a-c233-425c-99c0-5bddaf6c3225",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"for s in range(n_states):\n",
|
||
" if s in goal_states:\n",
|
||
" R[s] = goal_reward\n",
|
||
" elif s in trap_states:\n",
|
||
" R[s] = trap_reward\n",
|
||
" else:\n",
|
||
" R[s] = step_penalty\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b90fb80c-9452-48a2-889f-286703c2ae93",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now we define terminal states and a helper function. Here terminal_states is a set containing all absorbing states, which means, reaching them ends the episode conceptually. \n",
|
||
"\n",
|
||
"Moreover, `is_terminal(s)` is a small helper to check if a state is terminal."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 88,
|
||
"id": "eca4c571-39c7-468b-af86-0bab9489415e",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"terminal_states = set(goal_states + trap_states)\n",
|
||
"\n",
|
||
"\n",
|
||
"def is_terminal(s: int) -> bool:\n",
|
||
" \"\"\"Check if a state is terminal (goal or trap).\"\"\"\n",
|
||
" return s in terminal_states\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "3a9a1d54-8339-402b-84e9-105961ed78d7",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now we need to fill the transition matrices `P[a][s, s_next]`. \n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d9cfd15c-12cc-48bb-bd88-07f3ae3db31c",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Exercise 7.** **Complete the `# TO DO` part in the program below** to fill the transition matrices `P[a][s, s_next]`. "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 89,
|
||
"id": "2d03276b-e206-4d1f-9024-f6948ca61523",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"for s in range(n_states): # We loop over all states s.\n",
|
||
" i, j = state_to_pos[\n",
|
||
" s\n",
|
||
" ] # We recover the states to their coordinates (i, j) in the maze.\n",
|
||
"\n",
|
||
" # First, in a goal or trap state,\n",
|
||
" # No matter which action you “choose”, you stay in the same state with probability 1.\n",
|
||
" # This makes the terminal states as the absorbing states.\n",
|
||
" if is_terminal(s):\n",
|
||
" # Terminal states: stay forever\n",
|
||
" for a in ACTIONS:\n",
|
||
" P[a, s, s] = goal_reward\n",
|
||
" continue\n",
|
||
"\n",
|
||
" # If the state is non-terminal, we define the stochastic movement.\n",
|
||
" # For a given state s and intended action a,\n",
|
||
" # With probability 1 - p_error, the robot will move in direction a;\n",
|
||
" # With probability p_error, the robot will move in one of the other 3 directions, each with probability p_error / 3.\n",
|
||
" for a in ACTIONS:\n",
|
||
" # main action (intended action)\n",
|
||
" main_i, main_j = move_deterministic(i, j, a)\n",
|
||
" s_main = pos_to_state[\n",
|
||
" (main_i, main_j)\n",
|
||
" ] # s_main is the state index of that next cell.\n",
|
||
" P[a, s, s_main] += (\n",
|
||
" 1 - p_error\n",
|
||
" ) # We add probability 1 - p_error to P[a, s, s_main].\n",
|
||
"\n",
|
||
" # error actions\n",
|
||
" other_actions = [\n",
|
||
" a2 for a2 in ACTIONS if a2 != a\n",
|
||
" ] # other_actions = the 3 actions different from a.\n",
|
||
" for a2 in other_actions: # for each of the error action,\n",
|
||
" error_i, error_j = move_deterministic(i, j, a2)\n",
|
||
" s_error = pos_to_state[(error_i, error_j)] # get its state index s_error\n",
|
||
" P[a, s, s_error] += p_error / len(\n",
|
||
" other_actions\n",
|
||
" ) # add p_error / 3 to P[a, s, s_error]\n",
|
||
"# So for each (s,a), probabilities over all s_next sum to 1.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "7841b264-af00-4322-b728-adcffac0ef89",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now we check if the transition matrices `P[a][s, s_next]` are computed correctly.\n",
|
||
"For each action `a`, we sum the transition probabilities over all possible next states `s_next` and verify that these sums are equal to 1.\n",
|
||
"\n",
|
||
"This is because the matrix `P[a, s, s_next]` stores the transition probability\n",
|
||
"\n",
|
||
"$\\mathbb{P} \\big[S_t=s_{\\text{next}}\\,|\\, S_{t-1}=s, \\,A_{t-1}=a\\big]$. \n",
|
||
"\n",
|
||
"Therefore, for each action $a$, and for each state $s$, the sum over $s_{\\text{next}}$ of $\\mathbb{P} \\big[S_t=s_{\\text{next}}\\,|\\, S_{t-1}=s, \\,A_{t-1}=a\\big]$ should be 1. "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 90,
|
||
"id": "341fe630-8f87-4773-84ad-92d3516e53e2",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Action ↑: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]\n",
|
||
"Action →: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]\n",
|
||
"Action ↓: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]\n",
|
||
"Action ←: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"for a in ACTIONS:\n",
|
||
" # For each action a:\n",
|
||
" # P[a] is a matrix of shape (n_states, n_states).\n",
|
||
" # P[a].sum(axis=1) sums over next states s_next, giving for each state s:\n",
|
||
" # We print these row sums.\n",
|
||
" # If everything is correct, they should be very close to 1.\n",
|
||
"\n",
|
||
" probs = P[a].sum(axis=1)\n",
|
||
" print(f\"Action {action_names[a]}:\", probs)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "46d23991",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 3. Policy evaluation\n",
|
||
"\n",
|
||
"### 3.1 Bellman expectation equation"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "305b047c-e83b-4f42-b64e-e2050d5deeff",
|
||
"metadata": {},
|
||
"source": [
|
||
"Recall that the value function under a policy $\\pi$ is defined as:\n",
|
||
"$$\n",
|
||
"V^{\\pi}(s)=\\mathbb{E}\\Big[\\:G_t \\:\\Big|\\: S_t=s\\:\\Big]\n",
|
||
"$$\n",
|
||
"where the return $G_t$ is\n",
|
||
"$$\n",
|
||
"G_t=R_t +\\gamma R_{t+1}+\\gamma^2 R_{t+2}+... . \n",
|
||
"$$\n",
|
||
"This means *The value of a state is the expected discounted sum of all future rewards\n",
|
||
"when following policy $\\pi$.*\n",
|
||
"\n",
|
||
"We know that $G_t=R_t+\\gamma G_{t+1}$, and plugging this equation into the definition of $V^{\\pi}(s)$, we get \n",
|
||
"$$\n",
|
||
"V^{\\pi}(s)=\\mathbb{E}\\Big[\\:R_t \\:\\Big|\\: S_t=s\\:\\Big]+\\gamma\\mathbb{E}\\Big[\\:G_{t+1} \\:\\Big|\\: S_t=s\\:\\Big]. \n",
|
||
"$$\n",
|
||
"This step shows simply ``The total future reward = immediate reward + discounted reward from next state.''"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "88ea8d56-3b62-4690-9ff7-469e43726fbc",
|
||
"metadata": {},
|
||
"source": [
|
||
"For the expected immediate reward part $\\mathbb{E}[R_t| S_t=s]$, as we are in a maze problem, the reward depends only on the current state, not the time step, i.e., $\\mathbb{E}[R_t| S_t=s]=R(s)$. Hence we get \n",
|
||
"$$\n",
|
||
"V^{\\pi}(s)=R(s)+\\gamma\\mathbb{E}\\Big[\\:G_{t+1} \\:\\Big|\\: S_t=s\\:\\Big]. \n",
|
||
"$$\n",
|
||
"\n",
|
||
"Moreover, in this maze problem, we consider a deterministic policy $A_t=\\pi(s)$ (the action depends only on the state). Therefore, \n",
|
||
"$$\n",
|
||
"V^{\\pi}(s)=\\mathbb{E}\\Big[\\:R_t \\:\\Big|\\: S_t=s\\:\\Big]+\\gamma\\mathbb{E}\\Big[\\:G_{t+1} \\:\\Big|\\: S_t=s, A_t=\\pi(s)\\:\\Big]. \n",
|
||
"$$\n",
|
||
"\n",
|
||
"Now **given the state $S_t=s$ and $A_t=a$**, the next state is random (because of the error probability) and we know the transition probability \n",
|
||
"$$\n",
|
||
"\\mathbb{P}\\big(\\:S_{t+1}=s' \\:|\\:S_t=s, \\, A_t=a\\big)=P\\big(s'\\:\\big|\\:s, a\\big). \n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "c25e255d-8f58-4eaf-9485-cee6ab3bea6c",
|
||
"metadata": {},
|
||
"source": [
|
||
"Therefore,\n",
|
||
"$$\n",
|
||
"\\mathbb{E}\\big[\\,G_{t+1}\\,|\\,S_t=s,A_t=a\\,\\big] =\\sum_{s'}\\mathbb{E}\\big[\\,G_{t+1}\\,|\\,S_{t+1}=s'\\,\\big]\\times \\mathbb{P}\\big[S_{t+1}=s'\\,\\big|\\,S_t=s, A_t=a\\, \\big]\n",
|
||
"$$\n",
|
||
"$$\n",
|
||
"\\hspace{-1.2cm}=\\sum_{s'}V^{\\pi}(s')P\\big(s'\\:\\big|\\:s, a\\big),\n",
|
||
"$$\n",
|
||
"where here we use the Markov property. (**Question: Can you show the detailed computations here?**)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "9a2b6cff-e848-44a2-b504-973067b367b3",
|
||
"metadata": {},
|
||
"source": [
|
||
"In conclusion, we have (the Bellman expectation equation)\n",
|
||
"$$\n",
|
||
"V^{\\pi}(s)=R(s)+\\gamma \\sum_{s'}P\\big(\\,s'\\,\\big|\\,s, \\pi(s)\\,\\big)V^{\\pi}(s').\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "15049fdb-f3af-4f78-b556-817284260ed0",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 3.2 Define a function which computes the value function $V^{\\pi}(s)$ for a given deterministic policy. \n",
|
||
"\n",
|
||
"\n",
|
||
"**Exercise $8^*$.** Now we define `policy_evaluation(...)`, which computes the value function $V^{\\pi}(s)$ for a given deterministic policy. \n",
|
||
"\n",
|
||
"The input of this function `policy_evaluation(...)` are:\n",
|
||
"1. policy: array of size `n_states`, each entry is an action 0,1,2,3, which correspond to UP, RIGHT, DOWN, LEFT.\n",
|
||
"2. `P`: the transition probabilities `P[a, s, s']`.\n",
|
||
"3. `R`: the reward vector `R[s]`.\n",
|
||
"4. gamma: the discount factor $\\gamma\\in(0,1)$.\n",
|
||
"5. theta: convergence threshold.\n",
|
||
"6. max_iter: which is used to avoid infinite loops.\n",
|
||
"\n",
|
||
"How can we apply the Bellman expectation equation\n",
|
||
"$$\n",
|
||
"V^{\\pi}(s)=R(s)+\\gamma \\sum_{s'}P\\big(\\,s'\\,\\big|\\,s, \\pi(s)\\,\\big)V^{\\pi}(s').\n",
|
||
"$$\n",
|
||
"here ?\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "5c48f489-3508-4981-8b35-5bedc2e5838c",
|
||
"metadata": {},
|
||
"source": [
|
||
"We start with an initial guess of $V^{\\pi}$(e.g., all values = 0) and repeatedly apply the Bellman equation to update each state:\n",
|
||
"$$\n",
|
||
"V_{k+1}^\\pi(s) \\leftarrow R(s)+\\gamma \\sum_{s'}P\\big(\\,s'\\,\\big|\\,s, \\pi(s)\\,\\big)V^{\\pi}_k(s').\n",
|
||
"$$\n",
|
||
"until values converge."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 91,
|
||
"id": "2fffe0b7",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def policy_evaluation( # noqa: PLR0913\n",
|
||
" policy: np.ndarray,\n",
|
||
" P: np.ndarray,\n",
|
||
" R: np.ndarray,\n",
|
||
" gamma: float,\n",
|
||
" theta: float = 1e-6,\n",
|
||
" max_iter: int = 10_000,\n",
|
||
") -> np.ndarray:\n",
|
||
" \"\"\"Evaluate a deterministic policy for the given MDP.\n",
|
||
"\n",
|
||
" Args:\n",
|
||
" policy: array of shape (n_states,), with values in {0,1,2,3}\n",
|
||
" P: array of shape (n_actions, n_states, n_states)\n",
|
||
" R: array of shape (n_states,)\n",
|
||
" gamma: discount factor\n",
|
||
" theta: convergence threshold\n",
|
||
" max_iter: maximum number of iterations\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" n_states = len(R) # get the number of states\n",
|
||
" V = np.zeros(n_states) # initialize the value function\n",
|
||
"\n",
|
||
" for _it in range(max_iter): # Main iterative loop\n",
|
||
" V_new = np.zeros_like(\n",
|
||
" V\n",
|
||
" ) # Create a new value vector and we will compute an updated value for each state.\n",
|
||
"\n",
|
||
" # Now we update each state using the Bellman expectation equation\n",
|
||
" for s in range(n_states):\n",
|
||
" a = policy[s] # Extract the action chosen by the policy in state\n",
|
||
" V_new[s] = R[s] + gamma * np.sum(P[a, s, :] * V)\n",
|
||
"\n",
|
||
" delta = np.max(\n",
|
||
" np.abs(V_new - V)\n",
|
||
" ) # This measures how much the value function changed in this iteration:\n",
|
||
" # If delta is small, the values start to converge; otherwise, we need to keep iterating.\n",
|
||
" V = V_new # Update V, i.e. Set the new values for the next iteration.\n",
|
||
"\n",
|
||
" if delta < theta: # Check convergence: When changes are tiny, we stop.\n",
|
||
" break\n",
|
||
"\n",
|
||
" return V # Return the final value function, this is our estimate for V^{pi}(s), s in the state set.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "09ef3439",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 3.3 Evaluating a random policy"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "eecbca15-f89f-47bf-a13d-7d7c051699b8",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now we use the policy evaluation function `policy_evaluation` to evaluate a random policy. \n",
|
||
"\n",
|
||
"We first generate a `random_policy`, which is an array like [2, 0, 1, 3, 0, 2, ...] and has the size `n_states`. (Recall that the policy is a mapping from states to actions)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 92,
|
||
"id": "b4a44e38",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"[0 3 2 1 1 3 0 2 0 0 2 3 2 3 2 3 2 0 3 1 2 1]\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Random policy: for each state, pick a random action\n",
|
||
"random_policy = rng.integers(low=0, high=len(ACTIONS), size=n_states)\n",
|
||
"\n",
|
||
"print(random_policy)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "3fe07992-ce82-4124-aebc-a6384d417f64",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now we call the function `policy_evaluation(...)` to compute $V^{\\pi_{\\text{random}}}(s)$."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 93,
|
||
"id": "c5f559b2-452a-477c-a1fa-258b40805670",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Value function under random policy:\n",
|
||
"[ -0.2 -0.2 -0.201 -0.204 -0.205 -0.202 -0.214 -0.429 -0.212\n",
|
||
" -0.207 -0.276 -0.459 -0.352 -0.366 -5.827 -4.605 20. -0.366\n",
|
||
" -0.999 -20. -6.4 -3.163]\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"V_random = policy_evaluation(policy=random_policy, P=P, R=R, gamma=gamma)\n",
|
||
"print(\"Value function under random policy:\")\n",
|
||
"print(V_random)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f46c70ba-2932-49af-b568-b5477260bc94",
|
||
"metadata": {},
|
||
"source": [
|
||
"Here in this value vector of the policy, \n",
|
||
"- If it is a negative values, then the agent tends to move around aimlessly, fall in traps, or take too long.\n",
|
||
"- It it is a higher values, then the agent is closer to the goal or more likely to reach it"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1efcb076-467c-42d8-94e8-87453f688bbd",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now we define a function `plot_values`, which displays the value function $V(s)$ and displays it on the maze grid. It helps students visually understand:\n",
|
||
"- which states are good (high value, near the goal),\n",
|
||
"- which states are bad (low value, near traps),\n",
|
||
"- how a policy affects the long-term expected reward."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 94,
|
||
"id": "4c428327",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAdcAAAGbCAYAAACWHtrWAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAARjhJREFUeJzt3Qd4FGX+B/DvptcNoSQhEHqHUAUUUWnSBEQ9FRtgvb/lUBEBGyJFBBThlAPvTkVQBPQUEAREBFHpJfROIAFSSCC9kez8n/dddskmm7rvbrLL9+MzLjsz+868O5v5zdtmdJqmaSAiIiJl3NQlRURERAKDKxERkWIMrkRERIoxuBIRESnG4EpERKQYgysREZFiDK5ERESKMbgSEREpxuBKRESkGIOrg507dw46nQ6LFi2qku2vX78eHTt2hI+Pj9yPlJQUVEdi3yZPnlzVu1GtiN+M+F7Eb8hViWMu8lhYo0aNMHr06CrbJ6LKYHAtxbBhw+Dn54f09PQS13nsscfg5eWF5ORkVHdiHx966CH4+vpi/vz5WLJkCfz9/atsf37++WcGUCJySR5VvQPVmQicP/30E3788UeMHDmy2PKsrCysWrUKAwcORK1atVDd7d69W14oTJ06Ff369avq3ZHBVQR5awE2OzsbHh78eRJw4sQJuLmxHEDOhb/YMkqugYGBWLp0qdXlIrBmZmbKIOwMEhMT5WuNGjVQ3Ylq6+ocXMXzLsQFANmft7c3PD09q3o3iCqEwbUUovr0/vvvx6ZNm8yBqTARdEXwFUH4ypUrGDduHCIjIxEQEAC9Xo9BgwbhwIEDZW6nV69ecipKtDOJ9qbCDAYD5s6di7Zt28oAFBoair///e+4evVqmdsYNWqU/HfXrl1lu5apHaukNq2i+7Vlyxb5uRUrVmD69OmoX7++3Ie+ffvi9OnTxT6/c+dODB48GMHBwbL6uX379pg3b545b6LUKog0TVNpba779++X36n4bsV3LLa7Y8cOq+2Sf/31F8aOHYs6derIbd933324fPmyxbqpqak4fvy4fC2L+I6GDBmCDRs24JZbbpG/jc8++0wu+/LLL9GnTx+EhITIQNCmTRssWLCgxDT+/PNPdOvWTX53TZo0weLFi4ute+TIEZmm2I74nqdNmyaPvTX/+te/5O9BbDs8PBwvvvhisbZ0cRzbtWuHgwcP4q677pLNHc2aNcP3338vl//+++/o3r273F7Lli3x66+/lvmdmH4Py5cvx5tvvomwsDD5XYu/h9jY2GLrf/fdd+jSpYvcRu3atfH444/j4sWLZW7H2u9T5O/VV1+Vy0S+xXckapeSkpKQkZEh9+Pll18ultaFCxfg7u6OGTNmlLldIpuIR85RyX755RfxSD7tk08+sZifnJyseXp6aiNHjpTvd+/erTVt2lSbOHGi9tlnn2lTpkzR6tWrpwUFBWkXL140fy46Olqm9+WXX5rn3XXXXXIqatSoUVrDhg0t5j3zzDOah4eH9uyzz2oLFy7UJkyYoPn7+2tdu3bV8vLySs3Hc889J7ct9m3JkiXatm3b5DKxDbGtooru1+bNm+XnO3XqpHXp0kX7+OOPtcmTJ2t+fn5at27dim3Py8tLpv3uu+9qCxYs0MaMGaP169dPLhfbvvvuu2V6Yl9Mk4mYLz5ncvjwYZnPunXralOnTtU++OADrXHjxpq3t7e2Y8cO83riezXtY58+feRxe+211zR3d3ftoYcesthH07qFj0VJRD6aNWumBQcHy2MsvnvxfQjiux89erT8PsT2+vfvL9P99NNPi6XRsmVLLTQ0VHvzzTfl8s6dO2s6nU7mzyQuLk6rU6eO3Jb4fmfPnq01b95ca9++vUxX/IZMxHck5onvVWz7pZdeknkt+nsQxzE8PFyLiIjQXn/9dblumzZt5LrLli3TwsLC5Lbmzp1r/t2mpaWV+p2Yfg+RkZFy3+bMmSO/Gx8fH61FixZaVlZWse9a7Jf4nsR6vr6+WqNGjbSrV68Wy0/R763w7zM9PV1r166d3HfxdyB+W+I3IdLev3+/XOexxx6T33N+fr5FWrNmzZLf9/nz58s85kS2YHAtg/jjFCf02267zWK+OLmKk8CGDRvk+5ycHK2goMBiHXESFCd/EcxUBNc//vhDfvabb76xWG/9+vVW5xdlOsGJC4HCKhpcW7dureXm5prnz5s3T84/dOiQ+TsTgU+kW/jEKRgMBvO/X3zxxWIn0pKC6/Dhw2WwPnPmjHnepUuXtMDAQO3OO+8slkcRbApv69VXX5Un45SUlEoHV7Gu+K6LKhxETAYMGKA1adLEahpbt241z0tMTJS/EXEBYPLKK6/I9Xbu3Gmxngh4hYOrmCe+ExHMC//2RNAW633xxRfmeeI4inlLly41zzt+/Lic5+bmZnGBIn7T5fleTL8HEYwLB+IVK1bI+eJ3IYggHxISIgNidna2eb01a9bI9SZNmlSh4CrWF+v88MMPxfbJdMxNeVi3bp3FcnERYO1vjUg1VguXQVQhjRgxAtu3b7cYAiGqhEWVrKiaFETVlKnTRUFBgeyZK6ouRRXbvn37lOyLqFYLCgrC3XffLau/TJOoahPb2rx5MxzhySeflD2kTe644w75evbsWXP1bXR0NF555ZVi7btFh1mUh/g+f/nlFwwfPlxWo5rUrVsXjz76qKxmTUtLs/jMc889Z7EtsY8infPnz5vniapGEcfLO8yjcePGGDBgQLH5oprTRFQxi2Miql7F91G0yllUGZu+L0FUW4vfiOm7M3X0uvXWW2XVceH1irbti6rbvLw8+T0X7vDz7LPPyqrztWvXWqwvfiPit2witiuOT+vWrWWVsInp34X3qTSiOlY0j5j87W9/k8dG5EPYs2ePbFZ54YUXZFW4yT333INWrVoV28+y/O9//0OHDh1kVX9RpmMuOuyJKvJvvvnGvOzw4cOyWlxURxPZG4NrOZhOaqaOTaLd5o8//pAnKhF8BdEe9vHHH6N58+Yy0Io2JXFCFH/M5WnTK49Tp07JtETbnki78CTamay1C9tDgwYNLN6LNlXB1O575swZ+Sra+FQQbaWiZ7YIBkWJwCC++6JtfGXtY2WI4GqNaN8VJ3PRzieClTgeog1SKHrsi+6Xad8K75e4ABC/o6KK5t90oVB0vrjwERchhS8kBNEuWfTiRlysRUREFJtXke+q6L6KbYj2XNPFaEn7KYjgWnQ/yyJ+X2X9tsTFhvi7XblypfztCCLQiuD+4IMPVmh7RJVRfbtjViOiZChOAt9++608aYpXUeIpXJJ4//338c477+Cpp56SQ11q1qwp/8BFqaKkjiiFT0bGmlBLoqRVmEhHBNbCV+OFiZN6ZZRUmhTbN108FGZtnmAtD1XFHvtYuIRa+EQvai/E72POnDkyUIngJkpt4mKr6LGvyu+upG07w/GsDFGinj17tgywjzzyiLw4Fh3KTBcPRPbE4FpOIpCK4ClKouKPVFyti163JqLXZe/evfH5558X69UoSrGlESUXa1VwRa/omzZtKqsCb7/9dqsn+soS27d2pyax/cLVsOUl9tNUDVfaeNryVhGLiwbRu1WMdyxK9PYVFzFFS1+OIsZB5+bmYvXq1RalUluq6Bs2bChrKYoqmn+xnml+4eMkqopFtbyjxjIX3VcRlEXvcdE7vOh+ih7QhYl5puUV+X2J31ZZROm2U6dO8mJUlNpjYmLwySefVGhbRJXFauFyMpVSJ02ahKioqGLtX+Lqv+iVvmgjLc9QA3GyEEGi8FARMYRHVDcWJu6uJEqTomRcVH5+fqVvZSi2L4a0iJOyyZo1a6wOpyiPzp07yypUMWSo6D4V/o5Md4cqa7/Fd9u/f385rrhwu3dCQoK80OnZs6dsY6yoigzFKW3fiuZLpCeG51SWGL4kjseuXbvM88Rvo2iNhQieopT8z3/+02L74gJP7INo03QEMZSo8F3MxIVmXFycHDYliKFLosZl4cKF8kLEZN26dTh27FiF9/OBBx6Qfx/i5i5FFf0bfOKJJ2R7vfgtihu9mPaJyN5Yci0nESx69OghT/BC0eAqqpumTJkiO/uI9Q4dOiRPhuUp+YmqZFGlKDrLPP3007LtVJyIxNjFwh11RCcZMaZVjNETAV4EHDG4XpQcRCAXY0hFZ5KKeuaZZ+QJUdxpSgRwUdX59ddfm0ugFSVKkmKc59ChQ+V9jMV3Ijq4iEAmxm+KsaKm6nZhzJgxMu+mzmPWiHGeGzdulIFUdIwRN5gQ40zFyXrWrFmV2k9xchb7JgJhZe9dK46BCHAir+LYiLbv//znPzKYiABTGePHj5e3phTHQ4zVFBch//73v2UJT9ScFC7Rv/HGG3jvvffkumJ8qSgJinGvolbFUR13RBOIOC7iuxQXPCKQiTZX0bFKEL/RmTNnyuXiNyyqaMV64vcqxqmK8aoV8frrr8vfq2g7FX874nckxpmL2gPxdyM6O5mIDm/i+xTH+vnnn+fNKMhxlPc/dmHz58+X3fuLjuk0DcURwynEsB0xfu/222/Xtm/fXmw4i7WhOMLXX38th26IoRUdO3aUQwmsjXMV/v3vf8txpmI7YiiKGGc4fvx4OTSlMkNxhI8++kgOqRDDQsS+79mzp8ShON99953FZ0vK059//inHsop9FGNUxTCIwuOFxZCdf/zjH3JMpxh7WPjnWHQojrBv3z45xCUgIECOre3du7d5rG5ZeTTtu2lsamWG4txzzz1Wl61evVrmTYzvFOM2Z86cKYfBFB2TWlIa1oZiHTx4UM4TaYrjIsZxfv7558XSNA29adWqlRx3LcZ2Pv/888WGQIm02rZtW+58ie2IoVKlMX2n3377rfbGG2/I4TbiNynSszaOdPny5XL8sfiN1axZU45FvXDhgsU65RmKYxpnLsb0iu9G/M3Ur19frpOUlFRsu4MHD5ZpFv2tENmTTvzPgbGciFyEuEOT6Gcgak0qU2PiKGLIjqhJsnYXMSJ7YZsrEbksUTUvxtGKtlciR2KbKxG5HNFbWnQI/O9//yvbWUV7OJEjseRKRC5HPIhAlFZFkP3qq6/kQwWIHIltrkRERIqx5EpERFQVba7iFm6XLl2SN+euzI3XiYioaolKSnGzD/FAg8IPelApJyfH4mY0thDjxws/6MElg6sIrFV1ezkiIlJH3HlN3A7SHoG1ceN6iI+/oiQ90U4u2sydNcCWK7iaHiclDkplbjNHRERVS9ztTRSSCj8eUCVRYhWB9dz5FdDr/WxKKy0tC40aPiTTdOngaqoKFoGVwZWIyHnZu2lPH+ADfYCNDxYp40lizoDjXImISB0RGA02BkcXCK7sLUxERKQYS65ERKQOS64SgysREakj7kuk2XhvIhe4txGrhYmIiBRjyZWIiNQxaAqqhZ2/5MrgSkRE6rDNVWK1MBERkWIsuRIRkTosuUoMrkREpA6Dq8TgSkRE6mgKgqtIw8mxzZWIiEgxllyJiEgZnWaQk61pODsGVyIiUodtrhKrhYmIiBRjyZWIiBTfoUmzPQ0nx+BKRETqsFpYYrUwERGRYiy5EhGROiy5SgyuRESk+HmuBtvTcHKsFiYiIlKMJVciIlKH1cISgysREanDoTgSgysREanDkqvENlciIiLFWHIlIiJ1+Mg5icGViIiU0RkMcrI1DWfHamEiIiLFWHIlIiLFN5HQbE/DyTG4EhGROuwtLLFamIiISDGWXImISB2WXCUGVyIiUod3aJIYXImISB2WXCW2uRIRESnGkisRESmuFjbYnoaTY3AlIiJ1OM5VYrUwERGRYiy5EhGROuzQJDG4EhGROqJK18BqYVYLExGRU9u6dSuGDh2K8PBw6HQ6rFy50mL56NGj5fzC08CBA+26Tyy5EhGRU1cLZ2ZmokOHDnjqqadw//33W11HBNMvv/zS/N7b2xv2xOBKREROHVwHDRokp9KIYBoWFgZHYbUwERFVS2lpaRZTbm5updPasmULQkJC0LJlSzz//PNITk6GPTG4EhGR+nsLG2ycAERERCAoKMg8zZgxo1K7JKqEFy9ejE2bNmHmzJn4/fffZUm3oKAA9sJqYSIiUkczGCdb0wAQGxsLvV5vczvpiBEjzP+OjIxE+/bt0bRpU1ma7du3L+yBJVciIqqWJVe9Xm8xqeqE1KRJE9SuXRunT5+GvTC4EhHRTeXChQuyzbVu3bp22warhYmIyKl7C2dkZFiUQqOjoxEVFYWaNWvK6b333sMDDzwgewufOXMG48ePR7NmzTBgwADYC4MrERE59cPS9+zZg969e5vfjx07Vr6OGjUKCxYswMGDB/HVV18hJSVF3miif//+mDp1ql3HulYouK7vPQV+7vYdeEvk6obsmg5Xs6bbW3A1rnicXFWvXr2glXLLxA0bNsDRWHIlIiJ1+DxXicGViIiculq4OmJvYSIiIsVYciUiIoUU3ERCpOHkGFyJiEgdVgtLrBYmIiJSjCVXIiJShyVXicGViIic+g5N1RGDKxERqcOSq8Q2VyIiIsVYciUiInVYcpUYXImISB22uUqsFiYiIlKMJVciIlJHPJ1Gs7Fa19bPVwMMrkREpA7bXCVWCxMRESnGkisREanDkqvE4EpEROqIJ+IYbOzta/NTdaoeq4WJiIgUY8mViIjUYbWwxOBKRETqiBpdg63BFU7PYcE1uH0DRE4YBv+IWsiIScahmauQcijW6roht7dE05F3ILBpGLT8AlzZfw5HPl6LnMQ08zqhd7VGm38MhE+IHqnHL+HA9B+ReT4JjuRqeXK1/LhqnlwRj5MLYcnVcW2unnpfdJ0zEudW7MCGvtNw/rsd6DZnJDwCfKyu7xHgjTOL/8CmobPw2/APcS0zF53fH2Fe7t+gNjpNeQhH5v6MDf2mI2nPWXT98HHo3B3XhOxqeXK1/LhqnlwRjxO5Iof82sJ6tUHO5TTErNoDw7UC+ZqbnC7nW3Npw0Ek/nUCBdl5KMi5huhlfyG4bYT5j6PeoI5I3nsWiX+egCEvH6c+3wyv4ADU7NjQEdlxyTy5Wn5cNU+uiMfJtWgGTcnk7BwSXPXNwpB2Ms5innivbx5Wrs/X6twY6ecuQyswWE1PzM+ITpTzHcXV8uRq+bG2D66QJ1fE4+Sitz/UbJycnEOCq7ufF/LTcyzmXUvPgYefd5mf1beoi5Z/74ejH681z/Pw85KfL5qeu3/Z6anianlytfy4ap5cEY8TuSK7dGiqN6ADIt+4V/47Oz4FSbvOyHYViw0H+CAvJbPUdAKbhqLbvFE4PPsnmYZJflZesfYYzwBvFGTmwl5cLU+ulh9XzZMr4nFycezQZL/genHDATmZRAzrgsYjehS74oxe+lepfzi3fvoUjs3fgIvrb6QlpJ2OR1CLuub3oq0loHEI0s4kwF5cLU+ulh9XzZMr4nFycQyujqsWjt9yFD4hQfKPSOfhLl99agcifssRq+sHNAmRfzgnFm7EhTX7ii2/uC4KtW5pgpAeLeDm6Y7mT/VCXmqW7JLvKK6WJ1fLj6vmyRXxOJEr0mla2S3HaWlpCAoKwvLOr8HPvXLtFsEdGiJyvHEcW2ZsEg59sBpXD8XIZT6hQei1/GVseXgechJS0eGd+1H/nk6yJ2BhpuWC6EnY+qUB8o8y9cQlHJj2g+PHULpYnlwtP9U1T0N2TYerWdPtLZs+z+Nkf6bzeGpqKvR6vd3Svzr7Seh9vWxLKzsPwa9/abd9dangSkSuedJWEVyrI1c7Tg4LrjNHqwmuExY5dXDlqGoiIiLFeG9hIiJSRlSGajZ2SCpHhWq1x+BKRETqsLewxOBKRETqMLhKbHMlIiKntnXrVgwdOhTh4eHQ6XRYuXJlsWrmSZMmoW7duvD19UW/fv1w6tQpu+4TgysREakvuRpsnCogMzMTHTp0wPz5860unzVrFv75z39i4cKF2LlzJ/z9/TFgwADk5FjeJlMlVgsTEZE6Km68r1Xs84MGDZKT9aQ0zJ07F2+//Tbuvdd4283FixcjNDRUlnBHjLjxuEKVWHIlIqJqKS0tzWLKza34/aGjo6MRHx8vq4JNxHjc7t27Y/v27bAXBlciIlJGM6iZhIiICBkITdOMGTMqvD8isAqipFqYeG9aZg+sFiYiomrZWzg2NtbiDk3e3s5zh0CWXImIqFrS6/UWU2WCa1hYmHxNSLB8KpJ4b1pmDwyuRETk1L2FS9O4cWMZRDdt2mSeJ9pvRa/h2267DfbCamEiIlKmcJtpZVX08xkZGTh9+rRFJ6aoqCjUrFkTDRo0wCuvvIJp06ahefPmMti+8847ckzs8OHDYS8MrkRE5NT27NmD3r17m9+PHTtWvo4aNQqLFi3C+PHj5VjY5557DikpKejZsyfWr18PHx8fu+0TgysREamjKajWreA41169epV6s39x16YpU6bIyVEYXImISB1RpWtQkIaTY3AlIiJlxOPmNFsfOccb9xMREVFRLLkSEZE6rBaWGFyJiEgdUaOrKUjDybFamIiIqCpLrgM3T7K4z6OzW9PtLbiaIbumw5W44jFa7YJ54lU6mbBDkxGrhYmISB22uUq84CQiIlKMJVciInLqewtXRwyuRESkDquFJVYLExERKcaSKxERKcNqYSMGVyIiUkeMojEoSMPJMbgSEZEy4slvmmOfOFctsc2ViIhIMZZciYhIGba5GjG4EhGROhyKI7FamIiISDGWXImISBlWCxsxuBIRkTLsLWzEamEiIiLFWHIlIiJ1DDrjZGsaTo7BlYiIlGGbqxGrhYmIiBRjyZWIiJTRNJ2cbE3D2TG4EhGRMqwWNmJwJSIitUNxDLan4ewYXG0Q3L4BIicMg39ELWTEJOPQzFVIORRrdd2Q21ui6cg7ENg0DFp+Aa7sP4cjH69FTmKaeZ3Qu1qjzT8GwidEj9Tjl3Bg+o/IPJ/kwBy5HpXHKLBJCFq/Mhg1WoXDq4Y/1veZivyMHAfnCKhZKE+ZMck4OHMVrpaQp8IaDu+KDm8Ox+E5a3F22TY5L+S2FmjzjwHwCQmSZzTxuzs892ekn0mAI7nicaKbGzs0VZKn3hdd54zEuRU7sKHvNJz/bge6zRkJjwAfq+t7BHjjzOI/sGnoLPw2/ENcy8xF5/dHmJf7N6iNTlMewpG5P2NDv+lI2nMWXT98HDp3HqLqcowM+QbE/XoIUVP+h6rMk8hD9IodWN93GqK/24HupeTJxLt2IJo+3hNpp+It5qeejMP2fyzC+n7TsGHgDCT8dQLdZj0GR3LF43QzM7W5ajZOzo5n7koK69UGOZfTELNqDwzXCuRrbnK6nG/NpQ0HkfjXCRRk56Eg5xqil/2F4LYR5uBZb1BHJO89i8Q/T8CQl49Tn2+GV3AAanZs6OCcuQ7VxygzJgmxq/c6vFRXWF0recpJTpfzS9N+/DCc/GIz8tKyLOaL70NMJprBAN+6NRx6UeeKx+mmZtBBs3HiONebmL5ZGNJOxlnME+/1zcPK9flanRsj/dxlaAUGq+mJ+RnRiXJ+8t5oxXt/c1B9jJw1T3X7tIWHvzcu/ByFBkO7FFvuGxqEXkv/AQ8/b0AHnPzyd4fm2RWPExGDayW5+3khP92yHedaeo7xBFUGfYu6aPn3ftj7xrfmeR5+XvLzRdNz9y87PXLMMaouebpWgTx5BvqgzZiB2PGPRSWmmZ2QinV9p8m0I+7pjJyEVDiSKx6nmxnvLWzE4FpO9QZ0QOQb98p/Z8enIGnXGdlWVJhoI8pLySw1ncCmoeg2bxQOz/5JpmGSn5VXrI3JM8AbBZm5SvPhyux9jKoqTx2u5ymrhDx5Bvggt4Q8tRkzCDGr9yIzNrnMbRVk5eHc9zsx8Jc3sXXUv5B16SrswRWPE93Aca5GDK7ldHHDATmZRAzrgsYjehS7io5e+lepJ4NbP30Kx+ZvwMX1N9IS0k7HI6hFXfN70X4U0DgEaWw3qjbHqDrkqcGwLmhiJU9nSshTna5NZZWw6TMiENdoXU+25e+ZaKW0pwPcvD1ku6u9gqsrHieiotihqZLitxyVwxfEiUHn4S5ffWoHIn7LEavrBzQJkSeDEws34sKafcWWX1wXhVq3NEFIjxZw83RH86d6IS81Sw4zoOpxjAQ3Lw+4eRqvSd283OV7R4q7nqcG1/PUoIw8/fH0Qmx57BP8/vincko5dhGnv/4TB2esksvD746Ef/2agE4nS4uRY4egIPuaHJLjKK54nG5mtnZm0kydmipg8uTJ0Ol0FlOrVq1QlfiLq6RradnY/doSRI4fhnbjhiIzNgm7xy4xt4f5iE4iy1/GlofnyTaspo/1hFewH9q8OlhOJqbloodj1Lvfoe3Ye+SJJvXEJZk+O2lUn2MkSnN9V71unt9//ZvyddO9s5Edl+KwPO16bYns/Rs5bigyYpOws1CeROek3stfxuaH58m21NzkDIvPi57oYsynuHAT/OoGo/WL/eEdHCB73149egHbX/oS+Q5sjnDF43Qzq6o217Zt2+LXX381v/fwqNrwptO0srORlpaGoKAgpKamQq/Xw1Ws6fYWXM2QXdPhSlzxGLni5ZIrVoG52t+Svc/jpvRPDXscgZ5eNqWVfi0PzVd/Xe59FSXXlStXIioqCtWFK/5NEBGRC9xEIi0tzWLKzS25RuXUqVMIDw9HkyZN8NhjjyEmJgZVicGViIiUMRh0SiYhIiJCloZN04wZM6xus3v37li0aBHWr1+PBQsWIDo6GnfccQfS02/cIMXR2OZKRETVss01NjbWolrY29v62OdBgwaZ/92+fXsZbBs2bIgVK1bg6aefRlVgcCUiompJr9dXqn24Ro0aaNGiBU6fPo2qwmphIiJyqRv3Z2Rk4MyZM6hb98a9AxyNwZWIiJw6uI4bNw6///47zp07h23btuG+++6Du7s7HnnkEVQVVgsTEZFTu3DhggykycnJqFOnDnr27IkdO3bIf1cVBlciIlLGoOnkZGsaFbFs2TJUNwyuRESkTGVuX1iUrZ+vDtjmSkREpBhLrkREpAyf52rE4EpERMoYoKDNVTz70MmxWpiIiEgxllyJiEgZFTeB0Gz8fHXA4EpERMqIwGhgcGVwJSIidVhyNWKbKxERkWIsuRIRkTKG65MtbP18dcDgSkREyrBa2IjVwkRERIqx5EpENhuya3pV7wJVEwat4jfet5aGs2NwJSIiZVgtbMRqYSIiIsVYciUiIsXVwrA5DWfH4EpERMqwWtiI1cJERESKseRKRERqHzkHPnKOwZWIiJThw9KNGFyJiEgZMcbVYPM4V+cvubLNlYiISDGWXImISBlNQZurxjZXIiKiG9jmasRqYSIiIsVYciUiImXYocmIwZWIiJQR7aUa21xZLUxERKQaS65ERKQMb9xvxOBKRETKsM3ViNXCREREirHkSkREyrBDkxGDKxERKcM2VyMGVyIiUoYlVyO2uRIRESnGkqsNgts3QOSEYfCPqIWMmGQcmrkKKYdira4bcntLNB15BwKbhkHLL8CV/edw5OO1yElMk8sDm4Sg9SuDUaNVOLxq+GN9n6nIz8hxcI5cj8pjJDR7shcaDL8FnoG+yLp4Bcc+3YCknacdmCOgZqE8ZcYk4+DMVbhaQp4Kazi8Kzq8ORyH56zF2WXbzPN9QvRo9+pg1OnWTL6/euQCdoxZZNc8kOtitbARS66V5Kn3Rdc5I3FuxQ5s6DsN57/bgW5zRsIjwMfq+h4B3jiz+A9sGjoLvw3/ENcyc9H5/RHm5YZ8A+J+PYSoKf9zYC5cm+pjFHpXazR5rCd2j12CDX2m4uzSv3DLrMfkdhyZJ5GH6BU7sL7vNER/twPdS8mTiXftQDR9vCfSTsVbzHf38USPfz2N1FPx+GXoLKzv/z6OL9ho51zQzTAUx2DjVFHz589Ho0aN4OPjg+7du2PXrl2oSgyulRTWqw1yLqchZtUeGK4VyNfc5HQ535pLGw4i8a8TKMjOQ0HONUQv+wvBbSOgczcegsyYJMSu3ov0MwkOzonrUn2M/OrVROrRC+ZjdHFdFNw83OR8R6lrJU85yelyfmnajx+Gk19sRl5alsX8iCGdkZeahVNfbEFBVh60AgNSjl20cy6I1Fq+fDnGjh2Ld999F/v27UOHDh0wYMAAJCYmoqowuFaSvlkY0k7GWcwT7/XNw8r1+VqdGyP93GV5MiPnOEZxGw/Bu1Yg9C3qAm461B/SGdmJaQ69IKpMnur2aQsPf29c+DnKah5zElPRfe4oDNz4Fu786gWE9Ghhl32nm4OmaBLS0tIsptzcXFgzZ84cPPvss3jyySfRpk0bLFy4EH5+fvjiiy9QVRhcK8ndzwv56ZZtotfSc+Dh513mZ8XJueXf++Hox2vtuIek+hjlXsmQJds7vnoBg/98D23H3oOD7/8IQ14+HJknkYfy5skz0AdtxgzEwQ9WWV3upfdF3V5tcf7HXdgwcIYs3d7ywSPwr++40ji54MPSNdsmU2/hiIgIBAUFmacZM2YU215eXh727t2Lfv36mee5ubnJ99u3b0dVYYemcqo3oAMi37hX/js7PgVJu84Ua2sT7V55KZmlphPYNBTd5o3C4dk/yTTIeY5R82f6IOT2Ftj8t4+RdekqanVqhC4fPIIdL36JtFNxdstTh+t5yiohT54BPsgtIU9txgxCzOq9yIxNtro8PzsPVw7FIP73Y/K9eE09fgl1ujdH5oWdyvNDVBGxsbHQ6/Xm997exS8ik5KSUFBQgNDQUIv54v3x48dRVRhcy+nihgNyMokY1gWNR/QoVtqJXvpXqSftWz99Csfmb8DF9TfSIuc4RkEt6+LSpsOyl7CQvC9adhCq3a2p3YJr0Tw1GNYFTazk6UwJearTtamsEjZ9RgTiGq3roWbHhtgz8VuknYxH7a5N7LLvdHMSjSgGBWkIIrAWDq7OhNXClRS/5Sh8QoLkCVzn4S5ffWoHIn7LEavrBzQJkSftEws34sKafVbXcfPygJun8XrHzctdvqfqc4zEcJe6fdrBN6yGeZhPjTb1i7WB2lPc9Tw1uJ6nBmXk6Y+nF2LLY5/g98c/lZPorHT66z9xcIaxmjj25/0IahmO0J4tAZ1Ovor3iTtOOSxP5Fo0Ua2r2T6VV+3ateHu7o6EBMu+D+J9WFj5+lfYA8/elXQtLRu7X1uCyPHD0G7cUGTGJskhGqb2MJ/QIPRa/jK2PDwPOQmpaPpYT3gF+6HNq4PlZGJa7lu3Bvquet08v//6N+XrpntnIzsupQpy6PxUH6Mzi7fKKtke/3kWngG+spfu8QW/IGn3GYfmaddrS2Tv38hxQ5ERm4SdhfLkGxqE3stfxuaH5yE7IRW5yRkWnxftw2L8tOghLIhSuCjBtn15ELpMexiZF65g94Sl5tI5UXXn5eWFLl26YNOmTRg+fLicZzAY5PuXXnqpyvZLp2lamcN1RS8t0ZicmprqtEV0a9Z0ewuuZsiu6XAlrniMXLF/+DAX+925Inufx03pL+08Dn7uZXcaLE1WQS4e3fdhufdVDMUZNWoUPvvsM3Tr1g1z587FihUrZJtr0bZYR2HJlYiInPoOTQ8//DAuX76MSZMmIT4+Hh07dsT69eurLLAKDK5EROT0N+5/6aWXqrQauCh2aCIiIlKMJVciIlKGN+43YnAlIiJl+DxXI1YLExERKcaSKxERKcNqYSMGVyIiUobB1YjVwkRERIqx5EpERMqwQ5MRgysRESkjbqhrsLFat+yb8lZ/rBYmIiJSjCVXIiKqls9zdWYMrkREpExFn8dqja2frw4YXImISBmWXI3Y5kpERKQYS65ERKQMbyJhxOBKRETKiLioKUjD2bFamIiISDGWXImISHG1sM7mNJzdTR1cXaFHWlGru70FVzJs13S4mjebToGrOdzS9fI0/uidcCX5BZkO2Q6rhY1YLUxERKTYTV1yJSIitdhb2IjBlYiIlOFNJIxYLUxERKQYS65ERKSMeFycxkfOMbgSEZE64kHnBj4sncGViIjUYcnViG2uREREirHkSkREyrC3sBGDKxERKcNxrkasFiYiIlKMJVciIlKG9xY2YnAlIiJlWC1sxGphIiIixVhyJSIiZTjO1YjBlYiIlOFQHCNWCxMR0U2jUaNG0Ol0FtMHH3ygfDssuRIR0U3VoWnKlCl49tlnze8DAwOVb4PBlYiIquVQnLS0NIv53t7ecrKVCKZhYWGwJ1YLExGR8pKrwcZJiIiIQFBQkHmaMWOGkn0U1cC1atVCp06dMHv2bOTn50M1llyJiKhaio2NhV6vN79XUWodM2YMOnfujJo1a2Lbtm144403EBcXhzlz5kAlBlcb1GzfAJEThsE/ohYyY5JxcOYqXD0Ua3XdoJbh6PDmcPiFB0PnpkN6dCKOzv8FV/afk8trdW6M2xc+g/ysXPNnYtfsx6EPf3LK/LSfeC/qD+xw4wNuOnj4eOH3J+Yj9cQlR2XJ5dSoF4TxW19Gbmaeed7ZHeew5LllJX7mloc64c7neiCwTgDSEtLx2ydbceCnw3JZeNsw3Pf+EATXNx7HxNOXsWHWJpzbHYOq0PGhzhg8dQg2vr8Bu7/aWeJ6gaGB6PfmADTu0US+v3TgIpY98415eaeHO6PH/90B3xq+iNl1Hmvf/gmZlzPssMc6uOmaQ6cLBuAJIA8GLQaaFn99uTvcdC2g09WSfWAN2kVo2vlS0itr/Yqm53jiWayaoue5isBaOLiWZOLEiZg5c2ap6xw7dgytWrXC2LFjzfPat28PLy8v/P3vf5elYhXB24TBtZI89b7oNmckjn6yHhd+3o/6gzuh+5yR+PW+j5CfkVNs/az4q9g9YSmy41Pk+7q92uDWOSOxfuD7MOQaqySupWdjXd9pcIX8HPxglZxMmj56Oxre15WBVZGZt3+MnPQbF2IlqdsmDMPeG4xFT34jg3DTHo0x8r+PIO5YPBJPJ+HqxVR88/x3SLmUKtdv278VRv33EUzv9hHyr/8uHSUgJAC3Pn0bEk8klLqep68nHls8EodWHsTaN1fjWs41hLWpa17e8NZG6D2uH5Y9/Q0un0pE/3cG4t4P78PSUUvssNciCOShwHAAgPg70cPdLRIGLRcarsrAC50nCgw7ZPB1d+sAA3KgadbzWNb6FU2vKmgKOiRpFVz/tddew+jRo0tdp0kT44VYUd27d5fVwufOnUPLli2hCttcK0kEk5zLaYhZtQeGawXyNSc5Xc635lpqtjkQQaeDZtDg4e8Nn1rqe6lVx/w0GHYLYn7aa88skBU169dAysUUGViFM9uikRqXipBmdeT77JRsc2DV6QCDwQDvAG9ZynW0AZMG489//SH3qTTt7+uArKvZ+GvBH8jLzINWoCHu0I2Ltvb3d8Th1Qdx6eBFXMu+hi0f/YYGXRuiRv0adthrUXoU363pAjQNGlKg0wXJ06tOFwKDIRqAuFDJliVNN92NCwFLZa1f0fRuHnXq1JGl0tImUUK1JioqCm5ubggJCVG6Tyy5VpK+WRjSTsZZzBPv9c1L74E2aNPbcPf1gpuHO2LX7kPWpavmZWJ+/7UTZKBK3h+No59skAHPWfNjEhwZAf8GtRC7Zp/y/b5Zvbzuebh5uOHCgYtYP/NXXD6bbHW9k3+cQa8Xe6LZ7U1wZttZNOvZFL6BPji3x7La95394+Hl5wV3Dzfs++EArl64fuHkIK0GtJZB/fCqg+jwQMdS123QrSHSE9Lw8H8eRXj7eki5cBVb523Bma2n5fKQliHYs2S3ef3M5ExkJmWgTssQpNg9X27QQQ+DlgjADzqdKL8Uqo7WMgBdgxI+W9b6FU2valTnoTjbt2/Hzp070bt3b9ljWLx/9dVX8fjjjyM4WFTtq8PgWknufl64lm5ZXSree/iVXmcvqn3dvD0Q3rst3LxFG41RxvnL+P3xT5F+7jK8g/3R9pXB6PbRE9g66l8OuReY6vwU1uDeW5Dw5wnkXslUus83o6yrWfjXff/FpaPx8PL1RO+X7sSTix/HvIELkJtxox3WRJTcolYewhP/flheAGkFBvxv4mpkJFkei6mdZsHD2wPtBraWr47ko/dBn/H98O1T35Rv/SBfNOzeCD/84zt89/wyNLurOe7/54P477CFuBpzVV4k5Bb5Leek5cDLX117WkncdC2hIQsaLoueCdC0AotKTk2WOEv6ft3LWL+s5dVDdX4qjre3N5YtW4bJkycjNzcXjRs3lsG1cDusKtXrqFRj9QZ0QIc37pX/zopPQdKuM7KdsjDPAB/kppQdQESb5IX1B9Br2RhknLuMKwfOIzc5Q06CeD3w/koM/u0dBDSohYzzSU6XHxNRqq3XNxJ731muPA83gw7D2mH4tCHy36J6d96ghbhw0FgFKtpc183YiI73RqJB5wic2nqm2Oe7PNgRPZ+5DQse+AIJJxIQ2jIUI/87AjlpuTix5ZTFuqKNNWrVIby87v9w+UwSzu+13pnNVm2HtsOg94x5Sr2UgotRF3Dg+yhcPX+lXJ/Py8rDxf0XcHLTCflevMYfiUPjnk1xdekeudw70DKQegf6IC+z7DZqWxg7Nvleb38VRCB0u94uawwXOnnKLaktu6z1K5oeFSV6Ce/YIdqr7Y/BtZwubjggJ5MGw7qgyYgeFuvoW9TFmaV/lTtNUZIQPXMLByMzO5dWHZWfev3b41pmLhK2nVS05zeXA6sPy6k0Wim/lfA2YTj5+2nEHzd2eBGvp/84ixZ3NS0WXE3cPd1Rq1FNuwXXIz8dlpPJC5vGyCrhrqO6y/fi33XbhSOiSwP8MOa7Yp9PPJ6ARrc2LjH9xBOJCG11oznDr6YfAuoE4PIJUVVrz8Cqvx5YRRAUsq4HQf8bVbk60ZZd0gVrWetXNL2qUZ2rhR2JHZoqKW7LUfiEBMmgpPNwl68+tQMRv+WI1fVDe7aEvlkodO5ucPf2RPPRd8EnRI9k01CcLo3lsBbBM8hXDmVJP5uIjNhkp8yPiUhHtMW6xF9LNVC/Qz3UaVpbDpvx8vPEgPF95fk2Zt8Fq+vH7L+A5nc2RUhzYwcm8dr8jqayWllo2bs5wlqGwM1dB08fD9z1fE/ow/QOHYrz1cOfyyrdz+/9TE5xh+Ow4/NtWPfuGqvri17CoW3C0KyX6DkL+Sren/3DWHI/+EMU2g6LRN3IcHj4eKDX2D6I2X3ebu2txsAadD2wFi5FGqBpiXBzExcC7gB84aarB4Nm2beh/OtXNL2qoSn6z9mx5FpJ19Kyseu1JWg/fhgixw1FRmwSdo5dYm639A0NQu/lL2Pzw/OQnZAKryA/tH15EHzq6FGQl4/00wnY+epiZF28Yh432nny3+Cp90N+Zi6S9p7FzrGLHRaUVOdHCGhcB8Ft62PfOysckoebQc0GNXD3q71lb17Rnhp74CK+HP01cjOMVZ5BdfV4ZcMLmDvgX0iNS5Ol3hrhQRj57xHwr+WPrJQs7P0+Cnu/i5Lr+9f0w+A374Y+VC+rhRNOJmLxM9/iSkzxjmn2klmk/Vf8nkR+sq8aew2LEqzovPRhZ+PN1VNir+KHl79Hvwl3Y/icB3A15oos4Yr5wvkd57Blzm944NOHZHuuCKyrxv1op733hptbPWiaAe5ut5nniqExBu0kDNopuKHF9WWmcak3hs24uUVC01KhacaLmbLWL2s5VR86rbQ6pUL3dxS3nkpNTS3XgF5nsbrbW1W9C1SGYbumw9W82XQKXE2AC16mjz96J1xJWlomatUcYrfzuClOvNzwDXi7+diUVq4hB/POz3DqmOOCfxJERFRVqnNvYUdicCUiImXYocmIHZqIiIgUY8mViIiUEb14NFvvLewCJVcGVyIiUsZwfbKFrZ+vDlgtTEREpBhLrkREpAw7NBkxuBIRkToK2lzhAsGV1cJERESKseRKRETKsEOTEYMrEREpw6E4RqwWJiIiUowlVyIiUobVwkYMrkREpIx40JpmY72urZ+vDhhciYhIGY5zNWKbKxERkWIsuRIRkTJ8nqsRgysRESnDamEjVgsTEREpxpIrEREpw5KrEYMrEREpbnPVbE7D2TG4upg8g2vV9Ot0rvcTHddoElxNkKcrnA4tebj3givxcE+r6l24qbjemYuIiKoMq4WNGFyJiEgZ3rjfiMGViIiUEe2tBpvbXJ0/urpWAx0REVE1wJIrEREpw2phIwZXIiJSho+cM2K1MBERkWIsuRIRkTJ8nqsRgysRESnDca5GrBYmIqKbxvTp09GjRw/4+fmhRo0aVteJiYnBPffcI9cJCQnB66+/jvz8/ApthyVXIiJSxqBgnKvBjuNc8/Ly8OCDD+K2227D559/Xmx5QUGBDKxhYWHYtm0b4uLiMHLkSHh6euL9998v93YYXImISO2N+zXb07CX9957T74uWrTI6vJffvkFR48exa+//orQ0FB07NgRU6dOxYQJEzB58mR4eXmVazusFiYiomopLS3NYsrNzbX7Nrdv347IyEgZWE0GDBggt3/kyJFyp8PgSkREyquFDTZOQkREBIKCgszTjBkz7L7/8fHxFoFVML0Xy8qL1cJERKT2Dk2wPQ0hNjYWer3ePN/b29vq+hMnTsTMmTNLTfPYsWNo1aoVHIXBlYiIqmWHJr1ebxFcS/Laa69h9OjRpa7TpEmTcm1bdGTatWuXxbyEhATzsvJicCUiIqdWp04dOakgehGL4TqJiYlyGI6wceNGGeTbtGlT7nQYXImISBmDpqDkasc7NIkxrFeuXJGvYthNVFSUnN+sWTMEBASgf//+Mog+8cQTmDVrlmxnffvtt/Hiiy+WWC1tDYMrEREpI57FqlXj57lOmjQJX331lfl9p06d5OvmzZvRq1cvuLu7Y82aNXj++edlKdbf3x+jRo3ClClTKrQdBlciIrppLFq0qMQxriYNGzbEzz//bNN2GFyJiEgZTcEj4zQ4PwZXG9Rs3wCRE4bBP6IWMmOScXDmKlw9FGt13aCW4ejw5nD4hQdD56ZDenQijs7/BVf2n5PL20+8F/UHdrjxATcdPHy88PsT85F64pLd8+JTKwCd37oXwa3D4VtHj42PzkfqydLHdIXf1RqRLw+Ab0ggUo7HYe/UlUg/n1Tu5aoNHjwYEya8jsjIdrh27Rq2bv0Dr7wyFhcvXjSvc++9wzB79kzUq1cP+/btxzPPPIcTJ06UmGZZ61c0PRV8Ar0x5K270e7uVnD3dMfl6GQseHgRruWUfu/TgeP6oO+LPbHoueU4svHGPt76SGf0efEO+Af74syO8/hu4k9Iv5wBR6ndOgy9Jw+Fvn4wdDodrpy9jO0f/4q4vTFW16/ZrA5uf70/6rQJh2+wH/5z2wfIS7e8uUBwk9roOWEAwjpGwJBvwNlNx7F50moH5ejmVt1vf+govIlEJXnqfdFtzkhEr9iB9X2nIfq7Heg+ZyQ8Anysrp8VfxW7JyzF+runY13faTjz9Z+4dc5IuHkbr28OfrAKP/eaYp5OLPwVGecvOySwmh7xFL/tFLaNW1qu9QMa1ka3aX/DwTk/Y3WfGUjcfRY95jwGnbtbuZbbQ1CQHjNnzkZERCM0btxM3lFlxYpl5uUtWrTAN98swauvjkPNmnXw22+bsWrVD7KNxZqy1q9oeirodMBTnz8CwzUDZvb5FJM6zMT3b6xBQX7pZYW6rUPRpm9zpCakW8xvelsjDJ7YD0te+h6Tb/kI6UkZeHTufXCk9EupWP/KCnx++yz8t8dMRC3ahiH/ehTu1/82ihLB8vSGo9j09kqry/3qBGD4F6PkOl/c+SG+7PURDn1rObSCyN4YXCupbq82yLmchphVe2C4ViBfc5LT5XxrrqVmIzs+xfhGp4Nm0ODh7w2fWoFW128w7BbE/LQXjpJ7JRNnv9+Fq0dulPJK03BQB1zeE424P0/CkJePY//dAu9gf9Tu2LBcy+3h22+XyXaSzMxMZGVlYe7cf6J7927mYPf4449h8+YtWLt2rbyN2tSp02RX+zvuuMNqemWtX9H0VGjZqzlqhAdh5eR1yE7NkYPtLx2NlwGnJKKm5MEZQ7By8noUXCuwWNb1wY7Yt/IgYqMu4lr2Nayb/RuadG+ImhHWnxZiD7mp2UiPS72+s4BWoMHL3xt+tQOsrp9yLhnHftiPK6cSrS7vOPI2XNgZLdcpyM2Xf59Jx8p/Zx1S8zxXzcbJ2bFauJL0zcKQdjLOYp54r29e+iDjQZvehruvF9w83BG7dh+yLl0ttk5wZAT8G9RC7Jp9qK6CmocipVD+tQID0qIT5fzLe6PLXO4Id911p7wri+huL7RvH4moqAPm5eIRUkePHpPzt2zZUuzzZa1f0fRUaNq9IZLPX8GIOfehxR1NZPXtls+2Ye8PB0v8zJ1P34q444k4u/N8sWV1W4Xgr692m99nJGXKNMX8K7HXLwYd5JltE+DpJ/423HB8VRTSL1Zu++G3NETS8Xjcv+RJBDeujStnLmPbhxuRcKh8F45kG1YLGzG4VpK7nxeupedYzBPvPfxKHwclqoRFVXB477Zw8/a0uk6De29Bwp8nZGmyuvLwLSH//t7lWm5vxidZvIcHHxxhnifGsKWkWJ6wxfvAQOu1B2WtX9H0VPCt4YNmPRrjx3fXYfm4lYhoXw9PL3oUVy6kIHpX8TZKUQLtMbIr5g75t9X0vP28kJ1meZzEe28HHafCRJWwqApuenfrEquEy8MnyBfNB7fDT//3DRIPXUTbB7vgnvmP4JshnyK3SF6J7IXBtZzqDeiADm/cK/+dFZ+CpF1nZLtrYZ4BPshNKTsgGnLzcWH9AfRaNgYZ5y7jyoEbJQpRqq3XNxJ731kOe4oY2B5d3hwm/50Zl4qND39Soc/nZ+fJ/BYm3udn5pZruQqPPvoIPvtsgfz3+fPn0a6dsUNYu3btsG7dGrz00hj52CiTjIwMefPvwsT79HTLdsjyrl/R9Cqj073t8MD0IfLfVy+m4NSfZ5FyKRXbFhtLm+f2xsrOSW36tLAaXP/2/hBs+GizrEK2JjcrT3aQKswn0Ae5Co9TUS3uiUSvd415Sr+Ugm+HG4+hIKpxT645hEdWPo+Us0mI22+9g2BprmXlIf5ALOKvf/bQt7vR6enbEdahPs7/cVphTsgallyNGFzL6eKGA3IyaTCsC5qM6GGxjr5FXZxZ+le50xRVw6KnceHgWq9/e1zLzEXCtpOwp9j1B+VUWamnEhDU4kYVuOiopG9cB6mnE8q1XIWlS7+VU2EisP766wZMnPgmvvnGsnPWwYOH0LHjjR7ZHh4eaNOmNQ4dOmw1/bLWr2h6lbF/1WE5mdzytw6IHNi63J9v3rMJwtuEYdg7A+R73yAfjPhoOHat2I+fpv0iq4vFchP/Wn7QhwTI+fZycu0hOZX1txHUsFalgmvSiXjZzkxVw/RcG1vY+vnqgB2aKiluy1H4hATJIKvzcJevPrUDEb/F+vP+Qnu2hL5ZqAwy7t6eaD76LviE6JF8fSiOiUhHtMXC4PgrNzcvDznJf3u6G/8tuqdacX7dAYR0bYKw25vLdVs/fRdyU7KQtP98uZbbg7hlmQisb789CYsW3bgDi8nXX3+DPn16Y9CgQfKBx2+99SaSkpKwdetWq+mVtX5F01Ph8Ibj8PD2wK2PdpEBJKJjPbTt1xJHfrU+/GfabR/j43s+M09pCelYPW0Dfv2ncR93fxeFzsMjEdEhHJ4+Hhj0eh/ZNuvI9taGdzVHrRYh0LmL4Wce6PJsT/iH6nFpb8m/FXcvd7hf/62KV/He5Oj3+9C4dyuERtaT31Hbh7rA3dMDcVEVD9RUtY+cc2YsuVbStbRs7HptCdqPH4bIcUOREZuEnWOXmNsZfUOD0Hv5y9j88DxkJ6TCK8gPbV8eBJ86ehTk5SP9dAJ2vroYWRevmNMMaFwHwW3rY987K6okT/dve9f8775f/Z98/f3vn+Py3nOyl2/Pfz6BlXdOk/Mzzidh1zvfo8Nr98AvRI+rJy5h29hvZMel8iy3h3Hjxsqbd3/88UdyMmnTJlI+uurkyZN4/PGRmDdvDurXry/HpQ4bdp+5w1PPnj1ldXJgoLGnbFnrl7XcHnLSc/HF09/ivvcGYehbdyMlPg0/vvszzu0xBo7GXRvg6S8fxdvtPpDvU+Mtq6gNBg1ZV7PN7axntp/Dulm/YeSCh+AX5IMzO89j6Ss/wpF8a/jh9nH9ERCqR35uPpJPJWDtC0uRFmvs7Fe3cwMMXfgY/t3N+CzPwPAgjPzlFfPnn/p9nHxd3H+uHNYjSrtb31+H/rMfgE+wH5JPJWLti0uLjYUlsiedVo4+z2K8oGhLSk1NLdfjf5zF6m5vwdXkGVyrMuLBvfZ/OLKjjWs0Ca6mkb/zlzSKevHwjYtNV2Dv87gp/e765+Ghs61DXL6Wi51pC5w65rDkSkREyhiu/2cLWz9fHbhWMYeIiKgaYMmViIiU0XQaNJ2tvYWdv5mBwZWIiJTRFPT21VwguLJamIiISDGWXImISBnRGUnHDk0MrkREpA7v0GTEamEiIiLFWHIlIiJlDDoDdDb2Fma1MBERUSFsczVicCUiImUYXI3Y5kpERKQYS65ERKQMewsbMbgSEZEyBhRAhwKb03B2rBYmIiJSjCVXIiJSRtwXWLO5Wtj57y3M4EpERMpwnKsRq4WJiIgUY8mViIgUd2hyszkNZ8fgSkRECtk+FEek4exYLUxERKTYTV1yHbZrelXvApVBw9Sq3gUiqgCDJqp03RSk4dxu6uBKRERq8Q5NRgyuRESkjIYCaDaWXEUazo5trkREdNOYPn06evToAT8/P9SoUcPqOjqdrti0bNmyCm2HJVciIlLGeAMIg4I07CMvLw8PPvggbrvtNnz++eclrvfll19i4MCB5vclBeKSMLgSEdFNc/vD9957T74uWrSo1PVEMA0LC6v0dlgtTERE1VJaWprFlJub67Btv/jii6hduza6deuGL774AppWsYDPkisRESmjaaJDk87mNISIiAiL+e+++y4mT54Me5syZQr69Okj22V/+eUXvPDCC8jIyMCYMWPKnQaDKxERVcs219jYWOj1evN8b29vq+tPnDgRM2fOLDXNY8eOoVWrVuXa/jvvvGP+d6dOnZCZmYnZs2czuBIRkfPT6/UWwbUkr732GkaPHl3qOk2aNKn0fnTv3h1Tp06V1dIlBfiiGFyJiEjxOFedzWlURJ06deRkL1FRUQgODi53YBUYXImISBlNU3CHJs1+Q3FiYmJw5coV+VpQUCADp9CsWTMEBATgp59+QkJCAm699Vb4+Phg48aNeP/99zFu3LgKbYfBlYiIbhqTJk3CV199ZdGmKmzevBm9evWCp6cn5s+fj1dffVX2EBZBd86cOXj22WcrtB2dVo7+xaILdFBQEFJTU8tV/01ERNWLvc/jpvRr67vDTWdbuc2g5SMpbadTxxyWXImIqFoOxXFmDK5ERHTT3KHJUXiHJiIiIsVYciUiIsW9hXU2p+HsGFyJiEgh0eZqexrOjtXCREREirHkSkREyhirdHUK0nBuDK5ERKQMg6sRq4WJiIgUY8mViIiUEY+L09l8437nL7kyuBIRkTKsFjZitTAREZFiLLkSEZEyKu4LrPHewkREREXvC2xQkIZzY3AlIiJlVLSXamxzJSIioqJYciUiImVYcjVicCUiImVUjFHVXGCcK6uFiYiIFGPJlYiIlGG1sBGDKxERKcPgasRqYSIiIsVYciUiIoVUlDqdv+TK4EpERMqwWtiI1cJERESKseRKRETKcJyrEYMrEREpo2kKbtwv03BuDK5ERKSQeFyczuayq7NjmysREZFiLLkSEZEyxp6+OhvTcP6SK4MrEREpZHtwZbUwERERFcOSKxERqaOgWhisFiYiIrpBU1Clq7FamIiIiIpiyZWIiBRihyaBwZWIiBTSFHT2ZbUwERERVabkahrQm5aWVp7ViYiomjGdv+1/gwbRHcn5S54OCa7p6enyNSIiwt77Q0REdiTO50FBQcrT9fLyQlhYGOLj45WkJ9ISaTornVaOyxiDwYBLly4hMDAQOp2td94gIiJHE6d6EVjDw8Ph5mafFsGcnBzk5eUpSUsEVh8fH7h0cCUiIqLyY4cmIiIixRhciYiIFGNwJSIiUozBlYiISDEGVyIiIsUYXImIiBRjcCUiIoJa/w/0ndzpKjipNgAAAABJRU5ErkJggg==",
|
||
"text/plain": [
|
||
"<Figure size 640x480 with 2 Axes>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"def plot_values(V: np.ndarray, title=\"Value function\") -> None:\n",
|
||
" \"\"\"Plot the value function V on the maze as a heatmap.\"\"\"\n",
|
||
" grid_values = np.full(\n",
|
||
" (n_rows, n_cols), np.nan\n",
|
||
" ) # Initializes a grid the same size as the maze. Every cell starts as NaN.\n",
|
||
" for (\n",
|
||
" s,\n",
|
||
" (i, j),\n",
|
||
" ) in (\n",
|
||
" state_to_pos.items()\n",
|
||
" ): # recall that state_to_pos maps each state index to its maze coordinates (i,j).\n",
|
||
" grid_values[i, j] = V[\n",
|
||
" s\n",
|
||
" ] # For each reachable cell, we write the value V[s] in the grid.\n",
|
||
" # Walls # never get values, and they stay as NaN.\n",
|
||
"\n",
|
||
" fig, ax = plt.subplots()\n",
|
||
" im = ax.imshow(grid_values, cmap=\"magma\")\n",
|
||
" plt.colorbar(im, ax=ax)\n",
|
||
"\n",
|
||
" # For each state:\n",
|
||
" # Place the text label at (column j, row i).\n",
|
||
" # Display value to two decimals.\n",
|
||
" # Use white text so it’s visible on the heatmap.\n",
|
||
" # Center the text inside each cell.\n",
|
||
"\n",
|
||
" for s, (i, j) in state_to_pos.items():\n",
|
||
" ax.text(\n",
|
||
" j, i, f\"{V[s]:.2f}\", ha=\"center\", va=\"center\", color=\"white\", fontsize=9\n",
|
||
" )\n",
|
||
"\n",
|
||
" # Remove axis ticks and set title\n",
|
||
" ax.set_xticks([])\n",
|
||
" ax.set_yticks([])\n",
|
||
" ax.set_title(title)\n",
|
||
" plt.show()\n",
|
||
"\n",
|
||
"\n",
|
||
"plot_values(V_random, title=\"Value function: random policy\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "8275a1eb-b58e-4e05-ae5d-5635ff9a1556",
|
||
"metadata": {},
|
||
"source": [
|
||
"The next function `plot_policy` visualizes a policy on the maze.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 95,
|
||
"id": "c1ab67f0-bd5e-4ffe-b655-aec030401b78",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def plot_policy(policy: np.ndarray, title=\"Policy\") -> None:\n",
|
||
" \"\"\"Plot the given policy on the maze.\"\"\"\n",
|
||
" _fig, ax = plt.subplots()\n",
|
||
" # draw walls as dark cells\n",
|
||
" wall_grid = np.zeros((n_rows, n_cols))\n",
|
||
" for i in range(n_rows):\n",
|
||
" for j in range(n_cols):\n",
|
||
" if maze_str[i][j] == \"#\":\n",
|
||
" wall_grid[i, j] = 1\n",
|
||
" ax.imshow(wall_grid, cmap=\"Greys\", alpha=0.5)\n",
|
||
"\n",
|
||
" for s, (i, j) in state_to_pos.items():\n",
|
||
" cell = maze_str[i][j]\n",
|
||
" if cell == \"#\":\n",
|
||
" continue\n",
|
||
"\n",
|
||
" if s in goal_states:\n",
|
||
" ax.text(\n",
|
||
" j,\n",
|
||
" i,\n",
|
||
" \"G\",\n",
|
||
" ha=\"center\",\n",
|
||
" va=\"center\",\n",
|
||
" fontsize=14,\n",
|
||
" fontweight=\"bold\",\n",
|
||
" color=\"blue\",\n",
|
||
" )\n",
|
||
" elif s in trap_states:\n",
|
||
" ax.text(\n",
|
||
" j,\n",
|
||
" i,\n",
|
||
" \"X\",\n",
|
||
" ha=\"center\",\n",
|
||
" va=\"center\",\n",
|
||
" fontsize=14,\n",
|
||
" fontweight=\"bold\",\n",
|
||
" color=\"red\",\n",
|
||
" )\n",
|
||
" elif s == start_state:\n",
|
||
" ax.text(\n",
|
||
" j,\n",
|
||
" i,\n",
|
||
" \"S\",\n",
|
||
" ha=\"center\",\n",
|
||
" va=\"center\",\n",
|
||
" fontsize=14,\n",
|
||
" fontweight=\"bold\",\n",
|
||
" color=\"green\",\n",
|
||
" )\n",
|
||
" else:\n",
|
||
" a = policy[s]\n",
|
||
" ax.text(\n",
|
||
" j,\n",
|
||
" i,\n",
|
||
" action_names[a],\n",
|
||
" ha=\"center\",\n",
|
||
" va=\"center\",\n",
|
||
" fontsize=14,\n",
|
||
" color=\"black\",\n",
|
||
" )\n",
|
||
"\n",
|
||
" ax.set_xticks(np.arange(-0.5, n_cols, 1))\n",
|
||
" ax.set_yticks(np.arange(-0.5, n_rows, 1))\n",
|
||
" ax.set_xticklabels([])\n",
|
||
" ax.set_yticklabels([])\n",
|
||
" ax.grid(True)\n",
|
||
" ax.set_title(title)\n",
|
||
" plt.show()\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "48037254-dccc-4f9c-a4d7-349adba5c74f",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now let’s visualize the `random_policy`. Does it seem like a good policy?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 96,
|
||
"id": "d452681c-c89c-41cc-95dc-df75993b0391",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAGgCAYAAAC0SSBAAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAH6JJREFUeJzt3Q1wlNWh//Ff1pCQxLw0AQUvEVulcm16UyJSShlem5bSoS1cKCUoOmqrcOWC5Ho7KUVIteoES6UjU65ixb6kV+xM6aU4XAIBW4tN9Z8/oyGASlHBvzLgSyIGQ0L2P+fETUKCxw0meXZPvp+Z4+4+2eA5eZ7d356XZ5+EcDgcFgAAHyH0UT8AAMAgKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIiiADiZNmmRLxCuvvKKEhARt3Lgx0HoBQSIoEPfMm7h5M4+UgQMH6rOf/axuu+02HTt2LOjqAXEvMegKAD3lxz/+sT796U/rgw8+0NNPP61f/OIXevLJJ1VTU6PU1NTz+jeHDx+uU6dOacCAAT1eXyBeEBTwxte//nWNHj3a3r/55puVk5OjNWvW6I9//KPmzZt3Xv9mpIcC9GcMPcFbU6ZMsbeHDx9Wc3Oz7rrrLl1++eVKTk7WZZddph/+8IdqbGx0/hsfNUdx4MABfec739HgwYOVkpKiK6+8UsuXL7c/27Vrl/2dP/zhD13+vfLycvuzZ555pkfbCvQmggLeOnTokL01PQvTw7jzzjtVUFCgn/3sZ5o4caLuvfdeffe73+32v/v888/ri1/8oiorK/W9731Pa9eu1be//W1t2bLF/txMhufm5uq3v/1tl98120xYfelLX+qBFgJ9g6EneKOurk4nTpywcxR//etf7ZyF+bQ/cuRI3XrrrTYsHn74YfvcRYsW6aKLLtL9999vewCTJ0+O+v+zePFimcu4VFdX69JLL23bft9999lb02O49tpr7bCXqVNmZqbdfvz4cW3fvr2t5wHEC3oU8MZXvvIVOxRkPs2bnsKFF15oh3/27Nljf75s2bKznl9cXGxvt27dGvX/w7zZ//nPf9aNN954VkhEAiJiwYIFdljr97//fdu2xx9/3A6BmRAB4gk9Cnhj3bp1dllsYmKiLr74YjtvEAqFbFiY2yuuuOKs5w8ZMkRZWVl69dVXo/5//OMf/7C3eXl5zueZXsw111xjh5puuukmu83cHzt2bJd6ALGOoIA3xowZ07bq6Vw6fuLvC6ZXsWTJEh09etT2Lv72t7/pwQcf7NM6AD2BoSd4z5wL0dLSopdeeums7eZkvHfffdf+PFqf+cxn7K05N+PjmOGvCy64QL/73e9sb8KcizF37tzzaAEQLIIC3ps+fbq9feCBB87abiabjW984xtR/1tmDmTChAn65S9/qddee+2sn5kJ7o4GDRpkz+34zW9+Y4Ni2rRpdhsQbxh6gvfy8/N1/fXX66GHHrI9CLM09u9//7see+wxu6y1OyuejJ///OcaP368XWr7/e9/354Nbs63MJPie/fu7TL8NHv2bHvfnMcBxCOCAv3Chg0b7LCROXHOTG6bieySkhKtXLnyvILHzDesWLHCfk2IWY5rhq/MCXidzZgxQ5/61Kfs0Nc3v/nNHmoN0LcSwp37ywB6jFkOe8kll9jAeOSRR4KuDnBemKMAetHmzZvtuRdmCAqIV/QogF5QVVVlv+rDzEuYCWxzFjcQr+hRAL3AzF0sXLjQfk3Ir371q6CrA3wi9CgAAE70KAAATgQFAKBnzqMw31XT8SIvZl3422+/bb/rv6+/QwcA8MmYWYf33nvPLt82X5rZI0FhLvJSWlr6CasGAIglR44c0bBhw3pmMrtzj8JckMV8H/8dd9zhzYXnTapeddVVqq2ttT2meOdbewzaFB9oU+xramrS6tWr7dfaRC6u9Yl7FOY6w6Z0ZkIiKSlJvhwIqamptj0+HAi+tcegTfGBNsWPaKYOmMwGADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAKTHaJzY2NtoSUV9fb29DoZAtPoi0g/bELtoUH2hT7OtOOxLC4XA4mieuWrVKpaWlXbaXl5crNTW1ezUEAASqoaFBRUVFqqurU0ZGRs8Exbl6FLm5uXrjjTeUk5MjHzQ1NamiokKFhYUaMGCA4p1v7fG9TTU1NWppaZEvn1bz8vLYTzHs9OnTuvvuu6MKiqiHnpKTk23pzBwEvhwIvrbJt/b42ibz5uPDG1BH7KfY1Z02+DHYBgDoNQQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEA6JkLF/W1/675bz2691HtfXOv3j71tlIHpCo7JVvDM4cr/+J8TR8xXV+74mtBVxMfWrJkiR5//HG9+eabQVcFQH8IigV/WKBfP//rs7bVN9bb8sq7r+ipV5/Sq3WvEhQxxFxO8dixY0FXA0B/CIptL287KySuHnq1vnb513Rh0oU63nBc1W9U65mjzwRaRwDoT2IuKLYf2t52/4rsK1R1c5UuCF1w1nNMz+KFYy8o3q5Pe8cdd+imm27SVVddFXR1cB4Xol+2bJndh8OHDw+6OjgH9lE/msxubmluu//uB+/aoabOMpIz9OVLv6x4cebMGV133XVas2aNHn300aCrg/NQXV2tDRs2aMKECTp06FDQ1cE5sI/6UVAUDC1ou3+i4YQ+++BndfVDV+vWP92qh//Pw3r57ZcVT5qbmzVv3jyVl5dr0aJFKisrC7pKOA9jx47Vli1bdPz4cftGdPDgwaCrhE7YR/1o6Onaf7lW655dp+f+33P2cUu4xc5LmBIx/tLxevDrDyp/SL5i3Zw5c7R582ZlZmYqISFBixcvjnoV0YgRI3q9fpBWrFihd955J6rn5uXl6dlnn9XEiRP11FNP6corr+z1+oF9FLSYC4rEUKIqF1Tq3qfv1S//7y917P2uK2mefu1pFf66UPsW7dPgtMGK5XmJ3bt3t60KWrduXdS/O3v2bIKij5jhwNdff71bv2NWeNXW1vIm1EfYR8GKuaEnIz05XfdMvUdvFL+hmoU1euSbj+j6/OuVnpTe9hyzAqrzEtpYEwqFtHPnTmVnZysrK0tVVVUKh8NRlUmTJgVd/X7j6NGjUe2TkydPtu2X0tJSzZw5M+iq9xvso2DFZFBEmKGaz130Od046kZt/PZGPb/weYUS2qv80lsvKdYVFBSosrJSiYmJKiws1J49e+TT5OH69eu7bN+/f7/Wrl0rn7z33nuaNm2a7SHed999uvPOO4OuEjphH/WjoafH9j6mD5o/0LzPz7OrmzpKG5Bmg8LMWxhZA7MUD/Lz87Vr1y5NnTrV9jDGjRsnHyxfvlzbtm1TQ0ND27Z9+/ZpypQpOnXqlGbNmqXc3Fz54OWXX9YLL7xgV67dfvvtQVcH58A+6kdBcfjdwyp9qlRL/3epnbT+wsVfsF/d8dapt/T72t+ftXx22hXTFC/MBJt5Ex00aJB8sWnTJk2fPl3FxcUaPLh1rmjy5Ml2Pfv27du9CQlj1KhR9o3Ip/3nG/ZRPwqKCNOr2PGPHbacy/cKvqeJl01UPPHtAE5PT7c9ihkzZtgeU2Q58I4dOzR69Gj5xrf95yP2UT8JiqVjl+rzF31elYcr9dwbz+nNk2/q+PvHdSZ8RoNTB+vqS662E9uz/nlW0FWFGQ5MS9PWrVv1rW99y85ZVFRU2E92APwRc0Fh5h3+9ap/tQXxISUlxQ41AfBTTK96AgAEj6AAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAA9MylUBsbG22JqK+vt7dNTU22+CDSDtoTu3xuUyjkz+e2SFvYT7GrO+1ICIfD4WieuGrVKpWWlnbZXl5ertTU1O7VEAAQqIaGBhUVFamurk4ZGRk906MoKSnRsmXLzupR5Obmqra2VklJSfIlYfPy8lRYWKgBAwbIh09AFRUVqqmpUUtLi3zg2z7quJ9oU2xr8uz1dPr06aifG3VQJCcn29KZ+YP58EfryBzYvhzcBvsoPtCm+NDiyeupO23wY7ANANBrCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAnTQ3N2v+/PkaOXKkXnzxRfliyZIlGjJkiHzh636KRQQF0OnykHPmzLHXgj948KAmTZqkAwcOyAfm2sjHjh2TD3zeT7GIoAA+1NjYqFmzZmnz5s1tF5s/efKkfRPat29f0NXDh9hPfY+g6MPr0xYXF6u2tjboquAjzJ07V1u3blVJSYlmzpxpt23fvl2nTp3S5MmTdfTo0aCrCPZTIBKD+d/2L2fOnNGCBQtsNzkUCmn16tVBVwnnsHTpUl1zzTVavny5brjhBrtt7Nixqqio0JYtWzRs2LCgqwj2UyAIij6YcCsqKtITTzyhRYsWqaysLOgq4SOYoQtTOhszZowtiA3sp75HUPQyM+FmxlIzMzOVkJCgxYsXR71CZcSIEb1ePwD4OARFL89L7N69u23Fybp166L+3dmzZxMUAGICk9m9yMxH7Ny5U9nZ2crKylJVVZXC4XBU5VxdawAIAkHRywoKClRZWanExEQVFhZqz549QVcJ/Uh1dbXWr1/fZfv+/fu1du3aQOqE+MPQUx/Iz8/Xrl27NHXqVNvDGDduXNBVQj9hVgZt27ZNDQ0NbdvMuQZTpkyxy0nN+Qi5ubmB1hGxj6DoI3l5efYFOmjQoKCrgn5k06ZNmj59uj2HZ/DgwXabOdfAnNlszj0gJBANhp76ECGBvpaenm57FCYcjh8/3rZke8eOHfbcAyAaBAXgubS0NHsms5kjy8nJscOfo0ePDrpaiCMEBXAOGzdutKvPfJGSkmKHmk6cOKFRo0bJF77tp1hFUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgFOiotTY2GhLRH19vb0NhUK2+CDSjqamJvkg0g5f9o+P+6hjW3xsU1lZmVpaWuTLsZeXl+fN66k77UgIR3nB2VWrVqm0tLTL9vLycqWmpnavhgCAQDU0NKioqEh1dXXKyMjomR5FSUmJli1bdlaPIjc3V7W1tUpKSpJPnxgKCws1YMAA+fCprqKiQjU1Nd59qvNlH3XcTz62ycdjr8aTNp0+fTrq50YdFMnJybZ0Zv5gPvzROjIvVl9esAb7KD742CYfj70WT9rUnTb4MdgGAOg1BAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIij6WHNzs+bPn6+RI0fqxRdfDLo6+AhLlizRkCFDgq4GEBMIij6+9OCcOXPsdcYPHjyoSZMm6cCBA0FXC+dgriN87NixoKsBxASCoo80NjZq1qxZ2rx5c9uFzE+ePGnDYt++fUFXDwA+EkHRR+bOnautW7eqpKREM2fOtNu2b9+uU6dOafLkyTp69GjQVYSHzHWRi4uLVVtbG3RVEMcSg65Af7F06VJdc801Wr58uW644Qa7bezYsaqoqNCWLVs0bNiwoKsIz5w5c0YLFiywQ52hUEirV68OukqIUwRFHzFDTKZ0NmbMGFuAnl40UVRUpCeeeEKLFi1SWVlZ0FVCHCMoAA+ZRRNmPiwzM1MJCQlavHhx1Ku9RowY0ev1Q3whKAAP5yV2797dtnpr3bp1Uf/u7NmzCQp0wWQ24BkzH7Fz505lZ2crKytLVVVVCofDUZVzDY8CBAXwoerqaq1fv77L9v3792vt2rWKJwUFBaqsrFRiYqIKCwu1Z8+eoKuEOMbQE/AhsyJt27ZtamhoaNtmznGZMmWKXcZszoPJzc1VvMjPz9euXbs0depU28MYN25c0FVCnCIogA9t2rRJ06dPt+cdDB482G4z57iYM+rNOS/xFBIReXl5NuwGDRoUdFUQxxh6Aj6Unp5uexQmHI4fP962zHTHjh32nJd4RUjgkyIogA7S0tLsGfRmXD8nJ8cO2YwePTroagGBIigCsHHjRrvCBLEpJSXFDjWdOHFCo0aNCro6QOAICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcEpUlBobG22JqK+vt7ehUMgWH0Ta0dTUJB9E2lFcXKwBAwbIlzZVVFSorKxMLS0t8uW4y8vL8+a48/3YK/akTW+99ZbuueeeqJ6bEI7y4s2rVq1SaWlpl+3l5eVKTU3tfi0BAIFpaGhQUVGR6urqlJGR0TNBca4eRW5urn70ox8pKSlJPn2yKyws9OITQ+QTkC/t6dimmpoa73oUPu4n2hTbPYqhQ4dGFRRRDz0lJyfb0pl5sfrygo0wB4EPB4Kv7TE47uIDbYpd3WmDH5MLAIBeQ1AAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAFYsmSJhgwZEnQ10M9w3MWut96Sysqkr35VuuQSaeBAcw0gaehQacIE6Y47pL/8RYruMnM9L+oLF6HnmCtKHTt2LOhqoJ/huItNDz0kLVsmvf9+15+9+WZrMSFx//3SG29IQWQ9QQEAAVm9WvrP/2x/nJAgTZ4sjR0rXXih9Pbb0t690tNPSx98EFw9CQoACMD+/VJJSfvjnBzpf/5HGjeu63NPnpR+/WspJUWBYI4C581cs7q4uFi1tbVBVwX9iC/H3c9/Lp050/54/fpzh4RhehcLF0qZmQoEPQqclzNnzmjBggUqLy9XKBTSatOHBnqZT8fdzp3t9z/1KWnWLMUsehTotubmZs2bN8++WBctWqQys1wD6GW+HXevv95+f8QIKdTh3fjAgdb5is7lhhsCqSo9CnTfnDlztHnzZmVmZiohIUGLFy+OennmCPOKAM6Dz8ddQoJiGkGBbo8P7969u2255bp166L+3dmzZ8f8Cxaxycfj7p/+SXrppdb75tacIxEJjIsual0RZaxcKTU0BFdPg6GnPlJdXa31Zraqk/3792vt2rWKF2ZceOfOncrOzlZWVpaqqqoUDoejKpMmTQq6+v0Ox13sHndTp7bfN8tgzYqniOxs6T/+o7UEtdKpI4KijyxfvlwLFy7UmjVr2rbt27fPHsQrVqzQkSNHFC8KCgpUWVmpxMREFRYWas+ePUFXCR+B4y523XabdMEF7Y9vvbX1nIlYRFD0kU2bNmn8+PF2Wd+TTz5pt02ePFmNjY3avn27cnNzFU/y8/O1a9cuDRw40H7SQ2ziuItdn/ucdNdd7Y/NGdijR0szZkirVkk/+Yl0881Sfb0CxxxFH0lPT9e2bds0Y8YMe6BHVnHs2LFDo83REYfy8vLsp9NBgwYFXRV8BI672FZSIqWltZ6d3djYel7Fn/7UWs7FnJQXBHoUfSgtLU1bt2613eacnBz7iSheX6wRPrxYfcdxF9v+/d+lw4dbexHjx0uDB0uJia1zE5deKhUWtv6sulr66U+DqSM9ij6WkpJiu/xAX+K4i21Dh7aubjIlFtGjAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICANAz18xubGy0JaK+vt7ehkIhW3wQaUdTU5N8EGmHL+3p2Jbi4mINGDBAvrSpoqLCy/3kY5vKysrU0tKieHf69Omon5sQDofD0Txx1apVKi0t7bK9vLxcqamp3ashACBQDQ0NKioqUl1dnTIyMnqmR1FSUqJly5ad1aPIzc1VbW2tkpKS5EuPIi8vT4WFhV58Wo18UvWlPQZtig8+t6mmpqbf9SiiDork5GRbOjN/MB/+aB2ZA9uXg9vH9hi0KT742KYWT97zutMGPyYXAAC9hqAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKPpYc3Oz5s+fr5EjR+rFF18MujowDh2SLrxQSkhoLV/9qtT5wo/mcWFh+3PS0qSXXlI8WbJkiYYMGRJ0NRCHCIo+vqLUnDlz7OVjDx48qEmTJunAgQNBVwuXXy799KftjysqpHXrzn7Ogw9KO3a0P77/fmnECMUTc8nLY8eOBV0NxCGCoo80NjZq1qxZ2rx5c9v1aU+ePGnDYt++fUFXD7fcIk2f3v74Bz+QIj0+c2seR0ybJi1c2Pd1BAJCUPSRuXPnauvWrfba4zNnzrTbtm/frlOnTmny5Mk6evRo0FXEhg1STk7r/YYGacECk/DSdddJp061bs/Olh55JNBqAn2NoOgjS5cu1d1336177rmnbdvYsWPtxdpvueUWDRs2TPF4zd3i4mLV1tbKC0OHSr/4RfvjqippzBjp739v32Z+fsklgVQPHz+0e9ttt+nVV18NuireISj6iBliWr58eZftY8aM0V133aV4c+bMGV133XVas2aNHn30UXljzhxp/vz2x88/336/qEj6zncCqRY+XnV1tTZs2KAJEybokFmggB5DUOC8Vm7NmzfPTsovWrRIZWVl8oqZuDa9i44uvrjrBDdiiumhb9myRcePH7dhYRaMoGck9tC/g37ErNwyk/KZmZlKSEjQ4sWLo16eOSIeVgqZ+aK33z57m3n8yivSF74QVK36tRUrVuidd96J6rl5eXl69tlnNXHiRD311FO68sore71+viMo0O15id27d7ctt1zXjU/Zs2fPjv2gaGpqnbw2k9jn2v7cc1JyclC167fM8Obrr7/erd8xS4HN/BlB8ckx9IRuCYVC2rlzp7Kzs5WVlaWqqiqFw+GoipmniXkrV0p797Y//rd/a79fUyP96EeKt3H79evXd9m+f/9+rV27VvHCrAqM5hiLLDk3SktL21YY4pMhKNBtBQUFqqysVGJiogoLC7Vnzx55wbSj43zLjTe2zlfcdFP7tjVrpL/8RfHCLKBYuHChXXQQYc7bMW+mZjjnyJEj8sV7772nadOm2R7vfffdpzvvvDPoKnmDoMB5yc/P165duzRw4EDbw4h7778vXX+9Wc7V+viyy6QHHmi9b24/85nW+y0trc87eVLxYNOmTRo/frxdxvzkk0/abea8HXMCqDmPJzc3V754+eWX9cILL9hQ/EHHEyTxiTFHgfNmJg3Np9NBgwYp7hUXm3ea1vuhkPTYY1J6eutj8z1Qv/qVNHFia5AcPizdfrv08MOKdenp6dq2bZtmzJhhgz2yam3Hjh0aPXq0fDJq1CgbFl4cjzGGHgU+ES9elNu2Sf/1X+2PTQhMmHD2c7785bO/xsOcxb11q+JBWlqa/VYAM0yYk5Nje4C+hYRXx2MMokcRgI0bN9qCGGG+u6nzt8Wey09+0lriUEpKih1qAs4HPQoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAOiZS6E2NjbaElFfX29vQ6GQLT6ItKOpqUk+iLTDl/YYtCk++NymkGfvd9FICIejuViwtGrVKpWWlnbZXl5ertTU1O7VEAAQqIaGBhUVFamurk4ZGRk906MoKSnRsmXLzupR5Obmqra2VklJSfIlYfPy8lRTU6OWlhbFO9/aY9Cm+ECbYt/p06ejfm7UQZGcnGxLZ+YP5sMfzec2+dYegzbFB9oUu7rTBj8G2wAAvYagAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwClRUWpsbLQloq6uzt42NTXJF6FQSA0NDTp9+rRaWloU73xrj0Gb4gNtin2R9+5wOPzxTw5HaeXKleZfo1AoFIr8KYcOHfrY9/8E85/z6VG8++67Gj58uF577TVlZmbKB/X19crNzdWRI0eUkZGheOdbewzaFB9oU+wzo0KXXnqp3nnnHWVlZfXM0FNycrItnZmQ8OGP1pFpj09t8q09Bm2KD7QpPobUPvY5fVITAEDcIigAAL0TFGYYauXKleccjopXvrXJt/YYtCk+0Ca/2hP1ZDYAoH9i6AkA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAufx/wkBHS6w2LYIAAAAASUVORK5CYII=",
|
||
"text/plain": [
|
||
"<Figure size 640x480 with 1 Axes>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"plot_policy(policy=random_policy, title=\"Policy\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cbad5bf1-0150-4c3f-8cce-c82e0f1d1695",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Exercise 9.** Define your own policy and evaluate it using the functions `policy_evaluation(...)` and `plot_values(...)`. **Can you identify an optimal policy visually?** Plot your own policy using `plot_policy`. \n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 97,
|
||
"id": "929707e6-3022-4d86-96cc-12f251f890a9",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAdcAAAGbCAYAAACWHtrWAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAATRhJREFUeJzt3Ql8VNX5N/Dfmcm+EwgkgbALCLKJiigquCHiglvdRau2r0utW9X2X/e2VG2t1lq1dcHWvYtaN1xZVDYFQVkFZAlbWJOQPZl73s9zbiaZSSbJhJwsE35fP+Ms986Ze2fCPPd5zjl3lNZag4iIiKzx2GuKiIiIBIMrERGRZQyuREREljG4EhERWcbgSkREZBmDKxERkWUMrkRERJYxuBIREVnG4EpERGQZg2sE27hxI5RSmDFjRru8/syZMzFq1CjExcWZ7cjPz0dHJNt23333tfdmdFoTJkwwl47yd0nUETC4tpGzzjoLCQkJ2L9/f4PrXHrppYiJicGePXvQ0ck2/uhHP0J8fDyefPJJ/POf/0RiYmK7bc/777/PAEpEHUZUe2/AwUIC5zvvvIM333wTV1xxRb3lJSUlePvtt3Haaaeha9eu6Oi++uorc6Dw4IMP4uSTT27vzTHBVYJ8qABbWlqKqCj+qbeVPn36mPc8Ojq6vTeFqN0wc23DzDU5ORmvvPJKyOUSWIuLi00QjgQ7d+4012lpaejopGzN4Np2pCQs77nX623vTSFqNwyubUTKp+eeey4+/fTTmsAUSIKuBF8Jwnv37sXtt9+O4cOHIykpCSkpKZg8eTKWLVvW7P4vvyuvvBJ9+/YNesxxHDz22GMYNmyY+TLs0aMHfvrTn2Lfvn1Nvsa0adPM7SOPPNJ8mUr7Ql7Df7ux7Zo9e7Z53htvvIHf/va36NWrl9mGk046CevWrav3/IULF+L0009Hly5dTPl5xIgRePzxx2v2TbJWIW36L431uX7zzTfmPZX3Vt5jed0FCxYErSN9hvLcL7/8ErfeeisyMjLMa59zzjnYtWtX0LoFBQVYvXq1uW6KvEdnnHGGeQ+OOOII87chn7XcF//973/NfXk/xowZY7bV74UXXjDbFPiY3+9+9zsT0LZu3drga8v7IM+XbZWyvuy/VEp+/vOfo6ysLGjdqqoqU5kYMGAAYmNjzXb/6le/Qnl5eaP711Cfq/815X2UfR48eDD+7//+zyybNWuWeY5UdkL925Bl8+fPb/R1iToSBtc2JFmpfGFJQAkkwfTDDz80X9rypfPDDz/grbfeMl/Ajz76KH7xi1/gu+++wwknnIBt27ZZ2x4JpNL2scceawLVVVddhZdffhmTJk1CZWVlg8+TL8Sf/OQn5vYDDzxg+lulrQPx+9//3nyhysHEL3/5SxPg6mbvH3/8MY4//nisXLnSBIE//vGPmDhxIt59992a/TjllFPMbdkW/6UhK1aswHHHHWcOVu644w7cfffd2LBhgwn+EsTr+tnPfmbWvffee3HdddeZ8v6NN94YtI7sw6GHHhoyOIQiBxCXXHIJzjzzTEyfPt0c0Mhtef9vueUWXHbZZbj//vuxfv16E5DkQEicf/755m9E1qtLHpN96NmzZ5OvL21KMJXXloOWP//5zzWfqd8111yDe+65B4cffjj+9Kc/mb8/Wf+iiy5Cc3377bcYO3YsPvvsM1x77bXm723q1KnmvRSy3Tk5OQ3ulwT4cePGNft1idqN/J4rtY2qqiqdlZWlx40bF/T4008/Lb+pqz/88ENzv6ysTPt8vqB1NmzYoGNjY/UDDzwQ9Jg874UXXqh57IQTTjCXuqZNm6b79OlTc//zzz83z3355ZeD1ps5c2bIx+uS15T1vvrqq6DH5TXkteqqu12zZs0yzz/00EN1eXl5zeOPP/64efy7776rec/69etn2t23b19Qm47j1Ny+4YYbzPNCkcfvvffemvtTp07VMTExev369TWPbdu2TScnJ+vjjz++3j6efPLJQa91yy23aK/Xq/Pz8+utG/hZNET2RdadN29ezWPy2ctj8fHxetOmTTWPP/PMM+Zxeb/8Lr74Yp2dnR30N7JkyZKwXl/eB1nvrLPOCnr8+uuvN48vW7bM3F+6dKm5f8011wStd/vtt5vHP/vsswY/21B/l/K+yvsbuG8i8H395S9/af7GA9/XnTt36qioqKDPjygSMHNtQ1Kyk6N+KW9J6Syw7CUlWSlNCinBeTzuR+Pz+czIXCldShltyZIlVrblX//6F1JTU03Gt3v37pqLlCHltaRM1xYkW5YR0n6SUQrJ3oWUPyWrvPnmm+v17waWfsMl7+dHH31ksqb+/fvXPJ6VlWUyyS+++AKFhYVBz5GMLvC1ZBulnU2bNtU8JqVpieOhSuKhDB06NCgTk6xOnHjiiejdu3e9x/3vh5ABcVLBCPyMJLuTjPa8884L6/VvuOGGetm5f2BY4LWUwwPddttt5vq9995DuKSEPnfuXPz4xz8O2jcR+L7KfknJ+d///nfNY6+//rqp9kgmTxRJGFzbmL/k6R/YtGXLFnz++ecm6PoHgEgJUMpwhxxyiAm03bp1M/1UUloLp08vHGvXrjVtde/e3bQdeCkqKgrZL9wa6n7ZSp+q8Pf7SllUHHbYYVZeT77oZWS2HKjUJWVdee9zc3ObtY0Hom6bcqAjpDQa6vHA15IDIjkY8JdQZZtfffVVnH322abfPhzytxVIyq5yQOc/6JMDB7k/cODAoPUyMzPNQU7ggUVT/AcGTX2GQ4YMMX34gaVhuX300UfX2w6ijo5DKNuYZIbyJSJfhjI4RK4l4wnsZ5SBKdIPKEf6MqAkPT3dfNFJ9ubve2uIZAJuJTSYZFqBpB0JrKH6uIQE2QPRUDYprx9q9GhDI0pD7UN7aY1tbKjNcF5L1pEs++9//zv++te/mgFXksm2JLtr6HM7kOpAS0j2Kv3qctApWaz0wf/lL39p020gsoHBtR1IIJXgKZmoZLCSRcgRu5+UxWTAznPPPRf0PDkDkmSxjZGsKrCE6Fc305BM5ZNPPjGDmaScaIu8fqgzNcnrB5ZhwyXbKZYvX97ofNpwg4AcNMjJPNasWVNvmYxmlYOYutljRyRBSAZ2yYCgDz74wOyXDERrTuWiX79+QQOs5IDLP6Jc5qrKfVlPMnq/vLw88/nK8nD5P3f5DJsiFRwpRctBp3+u7IUXXhj2axF1FCwLtwN/liojMZcuXVpvdKxkJnWzIukjbWyKRWAwkiAROFVERrpKdlN3tKhkk5IZ1yV9XAd6KkN5fck2Kioqah6TUb11S63hkpGqEgRkylDdbQp8j/xnh2pqu+W9PfXUU8284sB+bwkacqAzfvx4Mz2luZozFccGmYokl2effRb/+c9/TFBqzlxe/9QlvyeeeMJcy/QkISOIhbzvgWT0upgyZUrYryWBX0Z7P//889i8eXPQsrp/53LwKNvw0ksvmaqKnFSlqQNKoo6ImWs7kGBxzDHHmC94UTe4yhQcmeIig31kPZmGI1804WR+UkqWL0DJYq6++mrTd/r000+buayBA3VkWoVMYZGpFRLgJeBIliCZigRymSoh0z6aS6ZvSOYtX4oSwKXPVL4o/Rloc0km+dRTT5lpKnIeY3lPpL9RAplMqZEpTP5yu7jpppvMvvsHj4Xym9/8xkzvkUB6/fXXm6D0zDPPmDLkww8/fEDbKVNwZNtkHmq4g5psZK8yhUk0tyQsg8RkTrV8TjLATj4jKTWPHDnSLJdrmcv8t7/9zRywyN/LokWL8OKLL5rBYFJZaQ6Z6iPvtxwsyQAx+TcgBzcyMEr+/urul/9vL9TBH1FEaO/hygerJ5980kxXOOqoo+otk6k4t912m5m2I1Mzjj32WD1//vywpjyIl156Sffv399MNxk1apSZ5lF3Ko7f3/72Nz1mzBjzOjJVYvjw4fqOO+4wU1MOZCqO+OMf/6h79uxpplXItn/99dcNTsX517/+FfTchvbpiy++0KeccorZxsTERD1ixAj9xBNP1CyXKTs/+9nPdEZGhlZKBU3LqTsVxz91ZdKkSTopKUknJCToiRMnBk2NaWwf/dseOD2muVNxpkyZUu9xeb5MKQr1fjzyyCP11t++fbuZEjRo0CAdLv9UnJUrV+rzzz/fvJ9dunTRN954oy4tLQ1at7KyUt9///1mKlR0dLTOyckx02Xk7zNQuH+Xy5cv1+ecc45OS0vTcXFxevDgwfruu++ut40yNUu2KTU1td42EUUKJf9r7wBPRM0nU6cki5fuBenDD4ecoUlOTiHdBh213CrdEtnZ2aZaUXfcAVGkYJ8rUYSS0wtKv/nll1+OzkTOTibBP9QPXBBFCva5EkUYOYWgnApSzsks/Z91zxkdqeTUkzKCXvpZR48ebfp5iSIVgytRhJHBbvPmzTPTqPyjfDsDGbgmA6tk4Bp/aJ0iHftciYiILGOfKxERUXuUheVMLXJ6NTlvaVufDo2IiFpOipT79+83I7H9Pwxim/yMYeAJZFpCftBDftO4UwdXCayRcEo4IiJqnJwtrVevXq0SWPv164kdO/ZaaU9+JEJOdhKpATas4Or/pQ35UA7k1HBERNS+5AxtkiSF+8tJzSUZqwTWjZveQEpKQovaKiwsQd8+PzJtdurg6i8FS2BlcCUiilyt3bWXkhSHlKQW/hhIE7/+FQk4FYeIiOyRwOi0MDh2guDK0cJERESWMXMlIiJ7mLkaDK5ERGSPnJdIt/DcRJ3g3EYsCxMREVnGzJWIiOxxtIWycORnrgyuRERkD/tcDZaFiYiILGPmSkRE9jBzNRhciYjIHgZXg8GViIjs0RaCq7QR4djnSkREZBkzVyIiskZpx1xa2kakY3AlIiJ72OdqsCxMRERkGTNXIiKyfIYm3fI2IhyDKxER2cOysMGyMBERkWXMXImIyB5mrgaDKxERWf49V6flbUQ4loWJiIgsY+ZKRET2sCxsMLgSEZE9nIpjMLgSEZE9zFwN9rkSERFZxsyViIjs4U/OGQyuRERkjXIcc2lpG5GOZWEiIiLLmLkSEZHlk0jolrcR4RhciYjIHo4WNlgWJiIisoyZKxER2cPM1WBwJSIie3iGJoPBlYiI7GHmarDPlYiIyDJmrkREZLks7LS8jQjH4EpERPZwnqvBsjAREZFlzFyJiMgeDmgyGFyJiMgeKek6LAuzLExERBFt7ty5OPPMM5GdnQ2lFN56662g5VdeeaV5PPBy2mmnteo2MXMlIqKILgsXFxdj5MiR+PGPf4xzzz035DoSTF944YWa+7GxsWhNDK5ERBTRwXXy5Mnm0hgJppmZmWgrLAsTEVGHVFhYGHQpLy8/4LZmz56N7t27Y/DgwbjuuuuwZ88etCYGVyIisn9uYaeFFwA5OTlITU2tuUyfPv2ANklKwv/4xz/w6aef4qGHHsKcOXNMpuvz+dBaWBYmIiJ7tONeWtoGgNzcXKSkpLS4n/Siiy6quT18+HCMGDECAwYMMNnsSSedhNbAzJWIiDpk5pqSkhJ0sTUIqX///ujWrRvWrVuH1sLgSkREB5UtW7aYPtesrKxWew2WhYmIKKJHCxcVFQVloRs2bMDSpUuRnp5uLvfffz/OO+88M1p4/fr1uOOOOzBw4EBMmjQJrYXBlYiIIvrH0r/++mtMnDix5v6tt95qrqdNm4annnoK3377LV588UXk5+ebE02ceuqpePDBB1t1rmuzgqvz4m1w4mPQacRGo7NRCZ3o8xHxrTvRuz2o0x9EZ6PfvxudTWf8nDqrCRMmQDdyysQPP/wQbY2ZKxER2cPfczUYXImIKKLLwh0RRwsTERFZxsyViIgssnASCWkjwjG4EhGRPSwLGywLExERWcbMlYiI7GHmajC4EhFRRJ+hqSNicCUiInuYuRrscyUiIrKMmSsREdnDzNVgcCUiInvY52qwLExERGQZM1ciIrJHfp1Gt7Cs29LndwAMrkREZA/7XA2WhYmIiCxj5kpERPYwczUYXImIyB75RRynhaN9W/yrOu2PZWEiIiLLmLkSEZE9LAsbDK5ERGSPVHSdlgZXRDz7wXXoCVCHjAXSs4HcldCf/K1mkRpzBtBnBJCWCaycA73gP023N/JUqCHjgbgkoCQfevY/gF0bgYy+UGOmAN16S8vA7k1ue/k7rO8SBo+HGnAUkJYNbFsFPfs59/G4JKgjpgLdBwLRcUDRbuhlHwBbVoRux+OFOuk6IK0H4IkGSgugV80G1s6vXT7+cqBrb6ikdDjyOrnfoVUMOAbocwSQkgnkrQbm/6N22djLgK59gagYoKIE2LgIWP1Zw23FpQBjzge69QcqioFVn7rP8et+CHDY6UBSN6A0H/j2HSDve7v702cs0Gs0kNQD2LUWWPJK7bLRFwFdegPeaKCyFMhdDKyf03Bb3QYAg08FEruazwirPgB2r6tdnpgBDD8bSM0CygqBVR8CO1fb3Z/OyubnJOsOOQ1IygB8FcDWpcCaT6TDLrzl1DqYubZScJUAuHQmVPYQIDEtaJEu3AUsegtqyLFhNaWOOAvIHAj9wROAPDcpHfBVuQtjE6C/XwB89jxQVQE1ejLUaTdAv36P/QnIJYXQ330MlTkoeJ+iYqH3bgWWvGPWQa+hUMddAf3+o0BBXv12tAP91X/cZdJhn9oD6pQboeX+zh/cVXZuAFbPBSTItqbSQmD1p+6BQXxq8LJVnwBFuwDHB8SnAeOvBor3AbnfhG7rqEuA4j3Au/e7wXr8NeZAA7t/ABLTgXFXAAtfAXasBjKHAEdfAXzyKFC8197+lO0H1s0BuvYH4ursz7pZQPFud39k2ZFXuEF+27L67cR3AQ6/GFj6BrBzrXtgIPc//wtQug9QHuCIS4Ft3wKLZrgHFKN+BHzxV6DE4v50VrY+JzmgHnMpsOFLYP7f3b/hsT8GSuTv9OswlhNF2oCmjcuATd8C5UX1l61dCGxZCVSUNd1ObAJw2InQc19yA6so2usGBSHt/LAYqCg1/xj1t5+YbM8EYNtyv3UzyPLi4MeL9gArZwElBe7RsGSsBTuBbn1DtyNBP3977Ug4cwyggeRu7n35Ulk9xw20rX2Gkm3LgW0rgPKS+ssKd7jb4t9I2RbJOkOR4Cn7u/wDwFcJ7Mt1g3DfI9zlPQYD+7YCO1a5bcn13lyg9xi7+5O3EshbBVSG2J/9ecH7IxfJSkPJOAQo3A7slMxau9cFW4Beo9zl6X2BmARg3WzAqXKX790I9KxeTm3zOUXHQcnnsEUO+LQbhHevB5J7hLecWo12tJVLpOu4fa4Z/dwvrwFHuJmu/KP7YTH04ncD/gEGyBoILYGiaB/ajZSuU3sA+7Y1upqaeC2QNQjKG+1mvptbqfTbEqOmmrKxioqBlgxzUwNH+1IalQOewIOp/G3AgHHubaXcSyC5n5qJNjXsDFOOVN4YaMletiwJvV7dbXUfBJKrt1e+nPfvDJ4qIAcj/NJu28+pshRaysY5Y4D1c93MVMr5K94Jbzm1Hp7+sIMH17gEqJh4IDUD+l8PmExWTboOqCwHls4MXjexC9SxF0Mv/G/7zY+S/tLjrgA2LXUzs0boWX83X+I6oz/QY6DbH9TRLH0LWPo2dFpPIHuo2wcWSlQsUFmnEiHryuMiby0wfAqQPQzYvgrIOhTo2sctGbelFe8CK96DloOB7kPqb7OfZDdDJgE9DgV2rgG6D3b77iQ7FbJfVY3sL7XN5yS2LweGTwUGToDyeKE3LnD7ccNdTnRQznOVICqBaPF7QFW56fPTy2dB9R4evF5CGtTpN0GvnAt8Xz0wqD0C6/FXAVWV0AteD//IbOd6qPgkYNiJ6JikjL3Fff+HnxF6FVkmg7kCRce7jwvpu134MnDoycAZ9wB9jwS2LAtdjm51GijY5m7boZNCryJ9ft+8AQycCJx0J9DrcPdLWgZ2CXlu3UAq++/fX2qbzymxGzDmEnew2YcPQH/6kDtwSQaihbOcWn9Ak9PCS4TruJnrnq1NryOBdcrPodd9BSz7EO0XWK8013r2s6FL1k09PzmjY49flEE8DfW5FmwH4lOA2MTaPum0LKAgYNT29pXuxW/ijcCmxWg3Hi+Q0EBfnpCRv4Gjf4/5CbBlaW2/4MAJ7nvir5KkyP423hVAlj8nKcPLSO0d1SPzpVtC+lcHHAes+ajp5dR6OFq4lTJX+dLxRrnX0n8lt+UfSb1lAbdDKdoDvXWVGQVshuYnpEINmwC9+Vt3udyf8nN3UNM371vfjXr75JH9kG1V1be95nETWKVfUqbNNBVYu/Q0fa1mf6TNnkOBfmOgtwV8kUu70n7g64bsB7S0T+b999+WL7M0IPswwBvj7mt6H2DgeCBvTeh2pD9290ZgWPXn1CUHyBkNbPyqdp20Xu7rSMY35GR3QFBDfbg29kfeL//+yKjTzKG1+5OWA/Q9OnhqTV2p2dV/nzFuII1OALZWj5SW8rCUgQee4H5WMgBKBjnJNA9qu8+pYCsQl+yW72V9+ZuSQWUyGC2c5UStTGnddM9xYWEhUlNTse/P1yAlPqbxBg8/HerwKUGP6e3fQ7/3ONTxl0MNOjp42fcLoOf+033ueb+GXvohsP6r2nmk4y8Bsge7fS/rFkF//Y6bMYw+HZ4xU6Cry8c17c18EshbH97ex0aHtZoacRrUyNOCX2fHOjOn1TPpZ9BVFUEd8Hr5x8BymU8HqDPvhJbbGxYD6TlQYy8AUrq7pa+ivdDffwmsnVf7Wufc4456DuB8+Qrww6LwtjWh8c+nxqGnQA09JXifdq0Hvn4dOPJid0qNfPnJ0f+mJcCaWbXzA0+5FVg9q3ZqTtA81xJ3Kk/gPFeZmpMu85G12wcr81xl/mg44sPsyzxkItQhweV1vWcDsOw/wKjzqwccKaB8P7B1mTvIxb8/x/3MnU8p02vEUdOA1F61fbCr3nffBz8pLx4m81yzq+e5zmzWPFd1+oPobPT7d7f95yR9sodMBBLS3cGP8lmtfL92JHJTyw+yz8n/PV5QUICUlJRWa3/fI1c1GSeabKu0Al1+8UKrbWtEBteIEmZwjSRhB9dIEW5wjSCd7Uu7WcE1gnS2z6nNgutDV9oJrnfOiOjg2nEHNBEREUWojjugiYiIIo4UQ3ULBySFUVDt8BhciYjIHo4WNhhciYjIHgZXg32uREQU0ebOnYszzzwT2dnZUErhrbfeqldmvueee5CVlYX4+HicfPLJWLu2dc/WxeBKREQRfYam4uJijBw5Ek8++WTI5Q8//DD+/Oc/4+mnn8bChQuRmJiISZMmoawsjB+ROUAsCxMRUUSfuH/y5MnmEropjcceewy//vWvcfbZZ5vH/vGPf6BHjx4mw73ooovQGpi5EhFRh1RYWBh0KS9v/jm8N2zYgB07dphSsJ/Mxx07dizmz2+989EzuBIRkTVyAj1t4SJycnJMIPRfpk+f3uztkcAqJFMNJPf9y1oDy8JERNQhRwvn5uYGnaEpNjZyztjGzJWIiDqklJSUoMuBBNfMzExznZeXF/S43Pcvaw0MrkRE1Gl/z7Vfv34miH766ac1j0n/rYwaHjduHFoLy8JERGRNYJ/pgWru84uKirBu3bqgQUxLly5Feno6evfujZtvvhm/+c1vcMghh5hge/fdd5s5sVOnTkVrYXAlIqKI9vXXX2PixIk192+99VZzPW3aNMyYMQN33HGHmQv7k5/8BPn5+Rg/fjxmzpyJuLi4VtsmBlciIrJHWyjrNnOe64QJExo92b+ctemBBx4wl7bC4EpERPZISdex0EaEY3AlIiJr5OfmdEt/co4n7iciIqK6mLkSEZE9LAsbDK5ERGSPVHS1hTYiHMvCRERE7Zm5eqb9EZ6A8zxGOufZG9DZqMsfQ2ei5/wOnY1e9Cg6ncT49t4C6iA4oMnFsjAREdnDPleDZWEiIiLLmLkSEVFEn1u4I2JwJSIie1gWNlgWJiIisoyZKxERWcOysIvBlYiI7JFZNI6FNiIcgysREVkjv/ym2/YX5zok9rkSERFZxsyViIisYZ+ri8GViIjs4VQcg2VhIiIiy5i5EhGRNSwLuxhciYjIGo4WdrEsTEREZBkzVyIissdR7qWlbUQ4BlciIrKGfa4uloWJiIgsY+ZKRETWaK3MpaVtRDoGVyIisoZlYReDKxER2Z2K47S8jUjH4BqOoSdAHTIWSM8GcldCf/K3mkVqzBlAnxFAWiawcg70gv803d7IU6GGjAfikoCSfOjZ/wB2bXSXpfWAOu5SoGsOULwPeuGbwObvWnHnOonsMUDmCCAxA9i7HlgR4nOITgSO/AlQXggsfi50O6k5wPALgx/zRANbvwbWf1z7WM44IHs0EJ0AlO8HVr8D7N9md5+6jwIyhgLx3YCCjcDa/9UuG3IBkJQV/C327QtAZXHD7WUcBmQeAcQkA1UlwKbZQP762vem36lAci+gqhTYthDY9V1kf04xScCg04G03kBlKbDpS2DHUvv7RBQCg2s4JAAunQmVPQRITAtapAt3AYveghpybFhNqSPOAjIHQn/wBCDPTUoHfFXVCz1Qp1wHrP8K+v0/A9lDoE78MfSb0911qWEVRe6XZ5e+QGxy6HUOORUoygOi4xtupyAX+OIPwV/0R98I7FpZ+1i/E4DU3sCyV4GyfUBsCqB9sK6yCNi6EEjt4waKunI/B/K+Ca+tjOFA5uHA+veAkl1AVALgja5dPmAKUJ4PfPOUG8wHn+vu2/4tiNjP6dCpQOk+YN7jbjAfcRFQuhco2Gxxh6gu9rm6OFo4HBuXAZu+BcqL6i9buxDYshKoKGu6ndgE4LAToee+VBssi/YCpYXu7axDgLhE6G8+cANu7nJg+1qogUdZ3qFOaPcaYM/3boYSStdDgKh4IK+Z2VjmcPcLunCrez8qDug1Fljznht8hGRYFY1kjAdq3zo3s5RMskUU0PMYN1OVwCokcy0vcG/HpgLJUpX5AnCqgOIdwJ7VQMYwROznFJcGpPYCNswGnEq3qpC3ws2aqXU5CrqFF85zpebJ6Od+eQ04ws10HR/ww2Loxe+6t6XsvG97cKlv7xb3cTpw3lhgwMnAd68BKb2a99zMkcD2gOwwpaf7GXYfCmSNdjPWnauAjXPafhRG9tFAz3FucN+xGNizKvR6cV2gYhKhE7sD/U42FRLkbwQ2zwGcCiAhwy0nS8D1kyDcfSQi9nOSfZUsObBMLtlw9uH2tpeoEQyubSkuASomHkjNgP7XAyaTVZOuAyrLgaUzgahYoKIkuDxSXgoVHddum9wp9D8R2PGtm9k050tb+vUkA8pbXvtYVBxUVBx0fDqw6Gm3dHnYjwBfBbD5S7SZLV8ApXvcQJ+SAww4w83QJNutS7JtkdIbWPFybRm4zwRgw0duX2VVefBzqsoAbwwi9nOSbQ+1T1FtvE8HIZ5b2MWycFuSICp/OIvfc//hy4Cl5bOgeg93l8tjEnwDqJg4oDKMkjM1/MUr5cHc+c1/rmRDe9YClQEHPL5K93rjXDeYSda49Su3nNmWira7AV2y5YJNwK5vgfRBodeV7RTbF7kBRi5yO61/7XLJGgPJgZ60H7GfU0Xofapqw306yPtcdQsvkY6Za1vaU90f1JC924BRk92ynb/E2LUXsDu3TTavU0rr62Y1425y7yuvO5DnmJuBr//ecF+pZD4ZQ4AV/w1+vDgPHVJjh/ql+6Alw22IlIBjEt2+Tn//bkJ3oHQ3Ivdz2gnEJrmjuf1BN6kHUMyBgdQ2mLmGQ4KdN8q9Vsq97fGGWBZwO5SiPdBbV0GNnux+cSSkQg2bAL35W3f59rVAeQnU6NMATxTQaxiQNQh63aK229eIpdwvZPPeB9zestAt3379nHuRjLNkj3u7Tgk+SPdh7qCbfT8EP15WAL1vA9BnvPsZySjenke4g3Raa5/kOnCfJCNL7ee+vjwuZeHuI4C9a0M3o6uA3auArCPd58pFbu+rnoYjA5tkwE+v6n1KzAS6DgF2LY/gzykfKNgC9Jvg7lNylrvujmWtsE8UqKWDmbR/UFMz3HfffVBKBV2GDBmC9sTMNQwS7NThU2rvX/U49Pbvod973MxJVYOOrl0mwfL7BdBz/+neP+/X0Es/NNNrhJ41A2r8JVCX/t4t90rgXFY9L0870B8/DXXcJVAjTgGK8836nIYThj7jofoeV3v/+Duh8zcBy14OLm9KSVSqAhX7ax874lpg8zxg54rgUqP0/4Wy6m1g0GRg3M/dtqWvL3eB/X3qeTSUDFjyO/Ln0IW5wLp3zTLEn+4+LqVpGZy0LyC4DjoH2L/VLf+KzbOAPicBI692B2FJMNo8u3b99e8D/U4BRl/nvkcyzcf2NJw2/5zeAgZPcbNf+bf2wyxOw+nEfa7Dhg3DJ598UnM/Kqp9w5vSuundKCwsRGpqKgoKCpCSkoLOwnn2BnQ2nmueRGei5/wOnU58JxygVtr5xgWoE36FzqS1v8f97a896zIkR7ds4Nj+ygoc8r+Xwt5WyVzfeustLF3acU4SwrIwERF1yAFNhYWFQZfy8jojwAOsXbsW2dnZ6N+/Py699FJs3ty+VQoGVyIissZxlJWLyMnJMdmw/zJ9+vSQrzl27FjMmDEDM2fOxFNPPYUNGzbguOOOw/79Ad0KbYx9rkRE1CH7XHNzc4PKwrGxdaZXVZs8eXLN7REjRphg26dPH7zxxhu4+uqr0R4YXImIqENKSUk5oP7htLQ0DBo0COvWhTipShthWZiIiDrVSSSKioqwfv16ZGVlob0wuBIRUUQH19tvvx1z5szBxo0bMW/ePJxzzjnwer24+OKL0V5YFiYiooi2ZcsWE0j37NmDjIwMjB8/HgsWLDC32wuDKxERWeNoZS4tbaM5XnvtNXQ0DK5ERGTNgZy+sK6WPr8jYJ8rERGRZcxciYjIGv6eq4vBlYiIrHFgoc/V/BJUZGNZmIiIyDJmrkREZI2Nk0DoFj6/I2BwJSIiayQwOgyuDK5ERGQPM1cX+1yJiIgsY+ZKRETWONWXlmjp8zsCBlciIrKGZWEXy8JERESWHdSZqy6tbO9NoKYkxqOz0YkJ6Gw8R93a3ptAHYSjm3/i/VBtRLqDOrgSEZFdLAu7WBYmIiKyjJkrERFZLgujxW1EOgZXIiKyhmVhF8vCREREljFzJSIiuz85B/7kHIMrERFZwx9LdzG4EhGRNTLH1WnxPNfIz1zZ50pERGQZM1ciIrJGW+hz1exzJSIiqsU+VxfLwkRERJYxcyUiIms4oMnF4EpERNZIf6lmnyvLwkRERLYxcyUiImt44n4XgysREVnDPlcXy8JERESWMXMlIiJrOKDJxeBKRETWsM/VxeBKRETWMHN1sc+ViIjIMmauYVAjJkINGQd06wlsWgHnvb/WLoyOg5p4KVS/EUBVJfS3s6C/eq/hxjJ6w3P8RW5bpUXQi96BXr3AXZbWHZ5jzgMy+wNR0cCebXDm/QfYvr71dzLSZYwEug0D4rsCBRuB9e/ULht8PpCYBWin9rHlM4DK4obb63YYkDkGiE4GqkqA3NlA/g+A8gKDzgHiugIeL1BRDOQtAXZ/Z3+fugyDSh0MxKYDxZuht3wUvDxtCFT6SCA6Eagqg877EijaFLIpNeASICre5ASGdqC/n1G7QmJPqO5jgehUoKoYOm8+UJxrf5+o02NZ2MXgGgZdnA/99ftQOYdCJXUJWqZOuAgqLhHOC3cBCcnwTL0V2L+nNmAGiomH56yboBf+D/q/nwPd+8Jz9s3QBbuB7euA2AToTcuhP/snUF4MNfRYeM68Cc4//g8oK2q7HY5EEii3LwRSegPRSfWXb/kC2PlNeG11Gw70GA2sfx8o3QVEJQCeaHeZBOjNs4DSvW6gikt3g3fZXqBoq919qiqB3r0EKrGnG0ADpR0KlT4ceusnQPkewBsPeBr/56y3fgoUbay/IDoZqtckt62izUBSb6hep0D/8C+gcr/dfaJOr72m4jz55JN45JFHsGPHDowcORJPPPEEjjrqKLQXloXDsf4b4IelJtMMEhUDNehIOAveBipKgfyd0Ms+gxo6PnQ7WQMAXxX08rnuzz7kbYBevwRqWPX6eRuhV3zuBlKtoVd84X6ZS5ZLjctfB+SvB6pKW9iQAnqOczNVCaxCMteKgurlGijdU5sB+sWmwrr9G9xg6Curt40q4wjovHluYBW+0gMPhEk5QNluN7AKuS7dCZU6qGXbT9RGXn/9ddx666249957sWTJEhNcJ02ahJ07d6K9MHNtibQeUN5oYFdt+UzvzoU6YnLo9VWIozGloLr2qvtV7eoqGUscsHe7vW0+WGWNBbKPBioK3TLunlWh14vrAhWdCJ3QA+hzMqA8bpk5dy7gVNSuN/BskyUrTxR0yS43sLeVmFSoqATouG5Qmce7f1dFudA75wNOZYNPU1nHATjevAd69+KAsm+oLEEBsV1bbReo85LvMm2hDVFYWBj0eGxsrLnU9eijj+Laa6/FVVddZe4//fTTeO+99/D888/jrrvuQntg5toSMbHQFWXBfXnlJUBMXOj1d/wARMeaPlzTX5c1AGrA6NDrSwl50rXQi98HSoL/wKiZtnwJLH8eWPaMWx7OmQikDQi9blT1ZyHl5VWvACtfcrPSnBOC11v3NrDkL9Cr3wD2rQWcKrQZr7uNUi7WG/8LveE/QEwyVI9jGnyK3vYZ9LpXode9BL1vOVSvU4G4DHdh8Rb3dlJfN6jKdUImIAeORAfyY+m6ZRf/aOGcnBykpqbWXKZPn17v9SoqKrB48WKcfPLJNY95PB5zf/78+WgvzFxboqIciI5xsxt/gI1NACTghlJWDOfdv8Bz7PlQR50J7NsOvXIelAxgqhtYpS92+zrohQEDc+jAFAdk/oWb3MFH6YNDZ5u+6sxv+yIzSKjmdv/TgXpjhbTbz5o+yB38JOu1hersVO9eWlMyltuq50kNP6d0R+3twnVAcl+o5H7QZbtMyVv6W6XUjOwTgJI8oHC9+3dN1I5yc3ORkpJScz9U1rp79274fD706NEj6HG5v3r1arQXBteWyM8DHB/QrRewy+2vUnJ7TyMDW7avh/Pvh2ruqtOuhd76fZ3A+nPovdugZ73Uqpt/0JL+7oaU7YVubhYqI4hjgwe6taqK/OZvY1PvQdEm6ICRxqrvVOiCgL9LojBJmuFYaENIYA0MrpGEh6bhkCN4b5TUGtyymbntBaoqoNd+Dc/RZ5ugiNTuUCNOhF75RcNtdctxR3V6o81AJtVzMPSyT9xl0XHuaOL8POhP/9Fmu9c5KDfIyZ+0qr5tPrdYILVv9UhaBSTnABnD3VJuKNrn9sdmHuk+Vy5y25/lxme4JWPzWgpI7QekDwEKN7bePpkMMmD/ZBsL1kJ1HQl4YszF3N7fwDZEJQHxWdXteIDk/iZz1YHrx3VzX0NGRXc73C095zO4UvNpKevqll/C1a1bN3i9XuTl5QU9LvczMzPRXpi5hkEdOQWesWfW3Pde/1foLWvgvPlH6NmvAideBs9VD7nBVua5BkzDMcFy21rorz9w2xp5otvPKl90O34wbaDYHYkqjysZUdytF1T/0TVtSAarv2+jkmOkyh4LlT2u9v6Ym6D35wLr3wOyjgb6p7uPlxe6g5MCg+shU4H9W4EdX7n3ZaRw7xOB4T92A5nMb82d4y6Tz63nsWbgkxl1IQOkZNneNdZ3SXU73C3V+u8PuQa6eBv05nfMSGGVOR5q4CXuNu7f5A5o8q/b/wLo3d+4JWBPFFTmMWYglOm+qC4Do6x2JKXKGAvEd3dL3cVboTe9A+g27EcmOkAxMTEYM2YMPv30U0ydOtU85jiOuX/jjTeivSitG6uR1Y7Yks7kgoKCiE3RQ/E98RN0Nt6f/Q2dif76T+hsdGICOhvPoT9t702gdv4e97f/yuG3I0EqPi1Q4ivHJUv+EPa2ylScadOm4ZlnnjFzWx977DG88cYbps+1bl9sW2HmSkREEX2GpgsvvBC7du3CPffcY04iMWrUKMycObPdAqtgcCUioog/cf+NN97YrmXgujigiYiIyDJmrkREZA1P3O9icCUiImv4e64uloWJiIgsY+ZKRETWsCzsYnAlIiJrGFxdLAsTERFZxsyViIis4YAmF4MrERFZIyfUdVpY1m36pLwdH8vCREREljFzJSKiDvl7rpGMwZWIiKxp7u+xhtLS53cEDK5ERGQNM1cX+1yJiIgsY+ZKRETW8CQSLgZXIiKyRuKittBGpGNZmIiIyDJmrkREZLksrFrcRqQ7qIOrLvOhs6l88Ep0JtF3z0Bnoze/hM7GV/A/dDY6KQWdSZWvuE1eh2VhF8vCRERElh3UmSsREdnF0cIuBlciIrKGJ5FwsSxMRERkGTNXIiKyRn4uTvMn5xhciYjIHvmhc4c/ls7gSkRE9jBzdbHPlYiIyDJmrkREZA1HC7sYXImIyBrOc3WxLExERGQZM1ciIrKG5xZ2MbgSEZE1LAu7WBYmIiKyjJkrERFZw3muLgZXIiKyhlNxXCwLExHRQaNv375QSgVdfv/731t/HWauRER0UA1oeuCBB3DttdfW3E9OTrb+GgyuRETUIafiFBYWBj0eGxtrLi0lwTQzMxOtiWVhIiKynrk6LbyInJwcpKam1lymT59uZRulDNy1a1eMHj0ajzzyCKqqqmAbM1ciIuqQcnNzkZKSUnPfRtZ600034fDDD0d6ejrmzZuHX/7yl9i+fTseffRR2MTgGgY1+kR4DjsW6NYTesNyOG/9pXZhTBw8p1wONWAkUFUB55vPoOe/G7qhhGR4Jl4ElTMIiIkH8nfB+fIt6PXLgl9v7OnwjDwBiE8GivbB9/6zwPYNrbuTXTLgPe1yqJ4DgMoKOIs+gjP/gwZXV6OOh/eYyUByOlCyH74PX4b+/hvAGwXvJbdDZWQDUdHA/nz4FsyE/mZO625/Z5Q0CCpxABCTBpRug95d/R56E6CyzgxeV3mr15ndcHuJA6FShprnwymD3vc1ULoleJ3oVKjM04NfzyIV0xcqujfgTQaqdsIp+ap2oScJnvjhgDcV0A501Q7o0hUAfKEba3L9KKj4EVDRPdzlFRugy7+3vUfwqEOgVBd58wBUwNGbofWO6uVeeNQgKNXVjIF19FZovamR9ppav7nttT35LVZt6fdcJbAGBteG3HXXXXjooYcaXWfVqlUYMmQIbr311prHRowYgZiYGPz0pz81WbGN4O3H4BqOonw489+F6nOoG0wCeE66BIhPhO+ZXwAJKfD+6DY4hXugV8yv3050LPTOzXDm/Nu0qQaMgOeMn8L30oPAnu1ue8edC9VrEHxv/BHI3wmkdAV89ksWQZRC1IU3w1mzBL7XHzeBNurSX0Dv3we9fEH91UefAO/Rk1D136eAHZuBxBSzb4bjg+/Dl4Bd28wXGrplI+ryO+HbvR061/YXWyfnK4Uu/A4qLssNiDWPl0BveT1gRQ9Uz3OhSzY2EVgPhd79OVC5D/DEAar+P3+VfjRQvgutRTvlJsCpqAwo2YYAnoQx0FV7oYsXACoanoSxQOwg6PJVIdtqan0VPxxKxcAp/BjwxMKTOA5wSqAr6xxQtIgEgQr4HDlALpNwAK9nOBxdDo19JvDKtvkc+XcUDa9nJByUQeu80PvUxPrNba89aAsDknQz17/ttttw5ZVXNrpO//79Qz4+duxYUxbeuHEjBg8eDFsYXMOg1y4x16p7TnBwjYqBGnIUfK9MB8pLzcVZ8ik8w4+DL1RwLdgN/dWHte1KxrpvB1TWAGgJrnGJUEecCt+Me93AKgr3tP4Ods0CumbCmfOWCY7YswPO0rnwjJ4AX93gqhS8E86F7+2/uYFVFBcGz/7euaX+P5P07gCDa/OU5rrXMenBwbWuhF7ul3xJ9edRj4JKGwm9Z54bWIUjgaCO5CFAZYEJ3oiWTKwVVLkHkSbbRHBwhScBuvJb929GV5hMVHm7NPxF2+j6XqjobDjFX8iLAk4VdPkGqJjeloOrZI+BBzWF0MiHUqnQugBKdYfP+cbdBlSZTNOjsuALGQw9Tazf1PKDV0ZGhrkciKVLl8Lj8aB79+5Wt4nBtSXSM6Gk9Lmz+ktQyO2jp4T3/IRkID0Lepf7fJXVH/BVQh061i0L+6qgV38F54s33aDXWpQKvq6+rXrIl3YdXbOgkqR02BfeKVcBHg/0uu/g+/hVoKL2C9t74c1Q/YeZ90fnbYZe7R6gkH0qcSBQsqHhqfdRKVDeeOiYdKj0sW4gLtsGvW8JoCvddbyJUMlDoHe8b67bgy5fDxWdA+0rMNmZisqCrtx0YOt7EqGkVO6rPfDTTiGU95BW3gsPFFLgaDk4ToBSMma0KGCjiwDVu4HnNrV+c9trHx15Ks78+fOxcOFCTJw40YwYlvu33HILLrvsMnTpYveAksG1JaTMKwFFyp/VdHmJ6YdtksdrSsJ6zVdAXvUXQnwiVGwCVJfu8D37K5PJes+7CaqyrOF+XBv27ADyd8Mz4Rw4s980WaZn5HFAbHy9VVV8onvdbyiqnr3P3Paeex28p14C37vP16zne/0xN0DnDILqM9j0R1Mr8CYCcZnQOxo5ePHGmCtl1nP70VW38VBdxkDvdSsTEnR1/jLAab/PSVfthCd+FFTK6SaI6Mrt0BWbD2x9FQWtJbsL+JY2BxKt+5XnUYOhUQINKa1L9uoL2gZtMs6GtsHbxPpNLe8YOvKv4sTGxuK1117Dfffdh/LycvTr188E18B+WFs61qcSaSrLgegYQI4mqwOskoAUkME1GFjPvt4dAPXhi7WPV5SbK+fLt922K8vhLP7EZLG+1gyujg9VbzwO7ymXIOrmPwGF++As+wKewyfUW1XXbOO7QGlRzW0JsPVX1tCb10ANPQqecZPhfPFO6+3DQUolDQAq9gGV+Q2v5Lh99rpgBeCU19yWAGsk9HOrFib7bS/Rpk9Ul62GrthYPRhpOFT84dCli5u/vgms3uo+UR3wddd64xfcgU3x1f2vQgKhJ2gbVKPb0NT6zW2P6pJRwgsW1B9H0hoYXFti7w7A5wOkL9affXbvDeza2nhgPes6wOuF8+Zfgsq9/vJwu9i1Db5X/lBz13PSBSYw1rNnO3Rl87Ib5fUC6T1sbCXVldgfulBGyDaiqhC6OsCGIhktYrpB9Ty/+oEot+rQ8zzorf9BmzB9yl4zotdVaYKmJ/Fo6NIDWN8pdsvknhTAKXB3S/p5fftbMbCmVAdW/7/pkuogmFhbylVJMkihgVaaWr+57bWPjlwWbks8iUQ4JDP1RpnAaI7w/berKkxZ1zN+qju1Jq07PKNPgvPd3MYDa0ysG1jrjgIu2A1n4wp4xp1pBkshMQ2ew0+Cs25p6+9j915uFu7xQg0ZY8rCvs//V3+9qkro7+bDc8wUIC4BiE0wt/UaGWQBoEdvqH7D3Gk4ygM1cCTUYePg/LC89feh01EBmYr/dsA/WRlFLCNuixsZJSyklFiywZ2Go6TSEu3erh4wpfctht7+julvlQuK1gJlee7tNtknBfiKTLYpU3Xcx71QMX0A6U8Npcn1fdCV2+CJk/5j+fcqXS79oCs2tVJgTa0OrIH/ph1ovRMeT7/qLDoeHtUTjq4e1FVPU+s3t732oS39F+mYuYbBM+4MeI49u/b+rc9Ab14N3+uPwPnkZXhOvQLe6/7gzg+Vea4BI4U9590MveV76IXvA9kD4DlktMn8vDc+XrOOs+A9d7ncfu/v8Jw6Dd4b/mRGH+uVC6AXzWz9fZTS7ZgTTVCUAUi+N/5cM+rXe/Gt0Ju/d0vB8rX10cvwTr4CUT/7gxtsv1/qDmgyxyEeeE48D0pGIMvIYTlg+PjVkFN6qHEqdThU6oja+70vgZagt/PjgIFMm2oHJQU+N2MidPlOoDqrlTmtqstRUD2nusG2dIsJqu7CCsAXUI2Q9mQdX6n9fYodBE9c7XQHb+oZ0FW74RTPg1OyCJ64Q6HiDnX/dnx74ZR8EzA4eCy0by90+VoTPJtaX5d+B8SPgCflVLM/Zp6r1ZHCIhYeT09o7cDrGVf72joPjv4ejl4LDwZVL/PPS60d2evxDDejirV2+4qbWr+p5dRxKK2b/uU8Ob+jnHqqoKAgrAm9kaLqkavR2eiyVhxV3A6i756BzsbZ/BI6G53aeb4X/HRS59qnwsJidE0/o9W+x/1x4ud9fonYOnOYm6vcKcPjm6ZHdMxh5kpERAfFaOG2xOBKRETWcECTiwOaiIiILGPmSkRE1sgoHt3Scwt3gsyVwZWIiKxxGj4RZ9ha+vyOgGVhIiIiy5i5EhGRNRzQ5GJwJSIieyz0uaITBFeWhYmIiCxj5kpERNZwQJOLwZWIiKzhVBwXy8JERESWMXMlIiJrWBZ2MbgSEZE18kNruoV13ZY+vyNgcCUiIms4z9XFPlciIiLLmLkSEZE1/D1XF4MrERFZw7Kwi2VhIiIiy5i5EhGRNcxcXQyuRERkuc9Vt7iNSHdQB1dd5kNn49tXhc4kRnW+P1Fn4cPobLSn8/UwRaWehc4kylvY3ptwUOl831xERNRuWBZ2MbgSEZE1PHG/i8GViIiskf5Wp8V9rpEfXTtfRwkREVE7Y+ZKRETWsCzsYnAlIiJr+JNzLpaFiYiILGPmSkRE1vD3XF0MrkREZA3nubpYFiYiooPGb3/7WxxzzDFISEhAWlpayHU2b96MKVOmmHW6d++OX/ziF6iqat7Z75i5EhGRNY6Fea5OK85zraiowAUXXIBx48bhueeeq7fc5/OZwJqZmYl58+Zh+/btuOKKKxAdHY3f/e53Yb8OgysREdk9cb9ueRut5f777zfXM2bMCLn8o48+wsqVK/HJJ5+gR48eGDVqFB588EHceeeduO+++xATExPW67AsTEREHVJhYWHQpby8vNVfc/78+Rg+fLgJrH6TJk0yr79ixYqw22FwJSIi62Vhp4UXkZOTg9TU1JrL9OnTW337d+zYERRYhf++LAsXy8JERGT3DE1oeRsiNzcXKSkpNY/HxsaGXP+uu+7CQw891Gibq1atwpAhQ9BWGFyJiKhDDmhKSUkJCq4Nue2223DllVc2uk7//v3Dem0ZyLRo0aKgx/Ly8mqWhYvBlYiIIlpGRoa52CCjiGW6zs6dO800HPHxxx+bID906NCw22FwJSIiaxxtIXNtxTM0yRzWvXv3mmuZdrN06VLz+MCBA5GUlIRTTz3VBNHLL78cDz/8sOln/fWvf40bbrihwbJ0KAyuRERkjfwWq+7Av+d6zz334MUXX6y5P3r0aHM9a9YsTJgwAV6vF++++y6uu+46k8UmJiZi2rRpeOCBB5r1OgyuRER00JgxY0aDc1z9+vTpg/fff79Fr8PgSkRE1mgLPxmnEfkYXA9Elwx4T7scqucAoLICzqKP4Mz/oMHV1ajj4T1mMpCcDpTsh+/Dl6G//wbwRsF7ye1QGdlAVDSwPx++BTOhv5mDNpfaBdHnToOn/2Dzl+2sW4nK/8wAivfXW9U7/hR4jzwOKisHzqplqHzhsfrrjJ0A78QpUKldTBuVb/4TzoolrboLp59+Ou688xcYPvwwVFZWYu7cz3Hzzbdi69atNeucffZZeOSRh9CzZ08sWfINrrnmJ1izZk2DbTa1fnPba5buo4CMoUB8N6BgI7D2f7XLhlwAJGUBOuBr7NsXgMri0G01tn5UPNB7ApDSC/DGAGUFwNZ5QP4PsE0lDoJK7AdEpwFl2+Ds+dxd4E2Ap8eUOit7q9eZ23B7CQOgkg81z4dTBid/MVBW/XnHZMCTOhqITgF0FXTxBujCZdb3iSLr9IdthcG1uZRC1IU3w1mzBL7XHzeBNurSX0Dv3we9fEH91UefAO/Rk1D136eAHZuBxBQgurpT3PHB9+FLwK5t7pdet2xEXX4nfLu3Q+d+36a7JYFVlD94s9nH6EuvR/Q5V6DypSfrrasL9qHq47fhGTQMKjW93nLv0RPhPeE0VP7zL9BbNwFJKVAx4Q8EOFCpqSl46KFHMGfOHPOTVU888TjeeOM1HHvscWb5oEGD8PLL/8SFF15iTm32q1/9Em+//V8MGzbCDGyoq6n1m9tes1UWAVsXAql9gJik+stzPwfyvgm/vYbWl4BastNdLq+Z1h8YMAVY8TJQthc2aV8JdOEKqLhMKG987QJfCZxt/wpY0wNP1lTokk0NtqUSB0AlDYGz90ugch/giXMDsrsUnq7HQxetgt61yg3eGScBvmLo4nVW94koFJ6hqbm6ZgFdM+HMecsER+zZAWfpXHhGT6i/rlLwTjjXZKomsIriQiB/l3tbRsTt3BKQTVQfraW7w7/bkuraHb5lC4GKcqC8DL6lC6CyeoVc1/nuazjLFwPFRSEaUog67TxUvflPN7CKokLovdX73IpeffU1009SXFyMkpISPPbYnzF27FFmgIK47LJLMWvWbLz33nvmNGoPPvgbM9T+uOPc4FtXU+s3t71m27cOyF8PVJWiVZUXADsWu4FVSMYqQVUyXdvKtrgXp/HT2Kn4XuZvSZfmNrQGVMoIN1OVwCqcMhM83cXRUN5Yk62af1cSVMt2uBkztcnvueoWXiIdM9fmUir4uvq26hEiEHXNgkpKhcrsC++UqwCPB3rdd/B9/CpQUVazmvfCm6H6D4OKiobO2wy9unXLp6H45nwA78ixcFbKsHQF7+hxcFY2IyuqprpnQaWkQfXqh9gfXQ14vPCtXoaqt18Byls5SNRxwgnHm7Oy+LPIESOGY+nS2rKg/ITUypWrzOOzZ8+u9/ym1m9ue9ZlHw30HAeUF7rBcc8qO+tLmTi+K1C6G+1FslJdsrHh3ruoZJP5qph0qC5HmTxBl22DLlhiSsDQFXCK17vt7F8BeBNNtuzkf9XWu3LQYVnYxeDaXHt2APm74ZlwDpzZb5os0zPyOCA2oMRVTcUnutf9hqLq2fvMbe+518F76iXwvft8zXq+1x9zA3TOIKg+g4GqCrQ1Z8P3ppwb+5tnzH29aR0qP3mn+Q0luOVLKRmX/+luczv68hsRNfVSVL3+LNqK+0sW9+OCCy6qeUzmsOXn5wetJ/eTk5NDttHU+s1tz6otXwClewCnCkjJAQacATiVbrbbkvWVBxg4Bdi7Bih2z0rT5qT/NLYHdEEjB3cet5tBxWbC2fmh+1D6sUDaGOh9C819XbIZni5HQaUcBqU8cIrWAGXb22Yf6KDHsnBzOT5UvfE4VI8+iLr5T4ia+v/gLPsCKKlfItVSYpWnfPkuUFpkLnJbDRpVv10phWxeAySmwjNuMtqUUoj5f3eZAFv+y2vMRW7H/L87m99WuZuR+z59xy0bFxeZ296h7lwymy655GLs359vLsuX12aQhx12GD744F3ceONNpi/Ur6ioyJz8O5Dc37+//qCtcNZvbntWFW0HfBVul0LBJmDXt0D6oJatbwLrmW4A3vAx2otkm6bUWxl84BJEV5orR7JSKTE75ea2iuvpLo9Khqfb8XAKlsDZ+jp82/4LFZUKlRri3x512BP3RzJmrgdi1zb4XvlDzV3PSRe4gbGuPduhK5uXhSrpH0wP/kWGVpeQCJWegarPPzKjn0XVFx8h7sQzgMSk0H2rDdC7mr/PB+qVV141l0ASWD/55EPcddev8PLLrwQt+/bb7zBq1Mia+1FRURg69FB8993ykO03tX5z22tVze2jqru+CaxnuAOC1r4dPKq4jamE/m4ptzGV+6Gl/NsQ6Vv1lQD+PlunDLrkB6jkodAF7hl5qHX4w2NLtPT5HQEz1wPRvRcQHWP6E9WQMaYs7Ps8YJqEX1Ul9Hfz4TlmChAnpa4Ec1uvqS539egN1W+YOw1HeaAGjoQ6bBycH9r4y7m4CM6uHYgaf7K7LVHRiDr2FOh9e0IHVo/HXU+uVfXt6kFDqKyEb/GX8Epgjk8w+y23fa08DUfIKcsksP761/dgxozaM7D4vfTSyzjxxImYPHmy+cHj//u/X2H37t2YOzf0VI+m1m9ue82nqke/qtrb8n57Y4HUfoBHjo2VW+btPgLYuzZ0M02t7w+snujqwGphpHNj+2S+dlTA7YCvodgsU/JtbJSwywddvBGe5KFm8JJc5LYu3eIurtgLeOKBuOqxEJ5YqIR+0BXVg5+o1TBzdSkdxrAs+ZFYKXcVFBSE9QsFkaLywcZ/RaEhngnnwjPmRBNUZACS88nr0FvcvivvxbdCb/7eLQWL6Bh4J18BNfhwN9h+v7RmQJPK6gvP6VdAyQhk+RgKdsP5+jM4Sw58MIxvXyNH841QPbIRdfZl8OT0M1+2ztaNqPrfK2bEb9T5V5l1qv79grmOmnSuuQRy1q1CxV9/696JiXXnzA4fI6N8TGCtevvlmpJxc8T/6bWw133++WcxbdoVZqRwoKFDh5ufrhJTp56Nhx/+PXr16mXmpV599bU181LHjx9vysnJybUjShtbP5zloTgLHw5vh3qOg5IBSAF0YS6w7l1g0FQgvnoalBmgtATYHZDtDToH2L8V2L7IHaDU2PrJvaAO/RG0lIMDM9Zti9znh8HJDu/XQlTKcHhShgfvU3kenF2fusul31T7oPfVn9bm6TYBunwn9P6V1Y15odKOdEcWy3NKt9YOaBJxPd3Xikpyl5ftcJc3MVLZz9vrEnQmrf097m9/XMr1iFItm3pXpcsxv/CvER1zGFw7mQMNrh1Vc4JrpAg7uEaQcINrJGFwPbD2x6ZcZyW4Lix8KqJjDvtciYjIGqf6v5Zo6fM7Ava5EhERWcbMlYiIrNFKQ6uWjhaO/AFNDK5ERGSNtjDaV3eC4MqyMBERkWXMXImIyBoZjKQ4oInBlYiI7OEZmlwsCxMREVnGzJWIiKxxlAPVwtHCLAsTEREFYJ+ri8GViIisYXB1sc+ViIjIMmauRERkDUcLuxhciYjIGgc+KPha3EakY1mYiIjIMmauRERkjZwXWLe4LBz55xZmcCUiIms4z9XFsjAREZFlzFyJiMjygCZPi9uIdAyuRERkUcun4kgbkY5lYSIiIssO6sw1+u4Z6Gyi0bnoR19q702gMHjbewOow3C0lHQ9FtqIbAd1cCUiIrt4hiYXgysREVmj4YNuYeYqbUQ69rkSEdFB47e//S2OOeYYJCQkIC0tLeQ6Sql6l9dee61Zr8PMlYiIrHFPAOFYaKN1VFRU4IILLsC4cePw3HPPNbjeCy+8gNNOO63mfkOBuCEMrkREdNCc/vD+++831zNmND6gVYJpZmbmAb8Oy8JERNQhFRYWBl3Ky8vb7LVvuOEGdOvWDUcddRSef/55aN28gM/MlYiIrNFaBjSpFrchcnJygh6/9957cd9996G1PfDAAzjxxBNNv+xHH32E66+/HkVFRbjpppvCboPBlYiIOmSfa25uLlJSUmoej42NDbn+XXfdhYceeqjRNletWoUhQ4aE9fp33313ze3Ro0ejuLgYjzzyCIMrERFFvpSUlKDg2pDbbrsNV155ZaPr9O/f/4C3Y+zYsXjwwQdNWbqhAF8XgysREVme56pa3EZzZGRkmEtrWbp0Kbp06RJ2YBUMrkREZI3WFs7QpFtvKs7mzZuxd+9ec+3z+UzgFAMHDkRSUhLeeecd5OXl4eijj0ZcXBw+/vhj/O53v8Ptt9/erNdhcCUiooPGPffcgxdffDGoT1XMmjULEyZMQHR0NJ588knccsstZoSwBN1HH30U1157bbNeR+kwxhfLEOjU1FQUFBSEVf8mIqKOpbW/x/3td0sZC49qWd7m6CrsLlwY0TGHmSsREXXIqTiRjMGViIgOmjM0tRWeoYmIiMgyZq5ERGR5tLBqcRuRjsGViIgskj7XlrcR6VgWJiIisoyZKxERWeOWdJWFNiIbgysREVnD4OpiWZiIiMgyZq5ERGSN/FycavGJ+yM/c2VwJSIia1gWdrEsTEREZBkzVyIissbGeYE1zy1MRERU97zAjoU2IhuDKxERWWOjv1Szz5WIiIjqYuZKRETWMHN1MbgSEZE1Nuao6k4wz5VlYSIiIsuYuRIRkTUsC7sYXImIyBoGVxfLwkRERJYxcyUiIotsZJ2Rn7kyuBIRkTUsC7tYFiYiIrKMmSsREVnDea4uBlciIrJGawsn7jdtRDYGVyIiskh+Lk61OHeNdOxzJSIisoyZKxERWeOO9FUtbCPyM1cGVyIisqjlwZVlYSIiIqqHmSsREdljoSwMloWJiIhqaQslXc2yMBEREdXFzJWIiCzigCbB4EpERBZpC4N9WRYmIiKiA8lc/RN6CwsLw1mdiIg6GP/3d+ufoEGGI0V+5tkmwXX//v3mOicnp7W3h4iIWpF8n6emplpvNyYmBpmZmdixY4eV9qQtaTNSKR3GYYzjONi2bRuSk5OhVEvPvEFERG1NvuolsGZnZ8PjaZ0ewbKyMlRUVFhpSwJrXFwcOnVwJSIiovBxQBMREZFlDK5ERESWMbgSERFZxuBKRERkGYMrERGRZQyuREREljG4EhERwa7/D2LZ+Kya8v0xAAAAAElFTkSuQmCC",
|
||
"text/plain": [
|
||
"<Figure size 640x480 with 2 Axes>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
},
|
||
{
|
||
"data": {
|
||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAGgCAYAAAC0SSBAAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAITBJREFUeJzt3Q10VNW99/FfYl4gkcDlRcFLxFtLoTUWQUqptkC0EYr1JQilhIqutndRKcpLrrUpBZKWUhp6WaULKq14wWub3mrbhxZZRcJrtbZceGgrIRAkj6Kg0hQhIQ4mgcyz9o4TQoKbiYY5c06+n7UOmTk5IfufmTm/OXvvMychHA6HBQDAe0h8r28AAGAQFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERRAB9m+fbsSEhLs14j7779f11xzjaftAj4oggKeW7t2rd3BmuX5559v833zKTOZmZn2+5///Oc9aSPQmREUiBtdunRRSUlJm/U7duzQkSNHlJqaKr957LHHVFFR4XUzgA+EoEDcGD9+vJ5++mmdOXPmvPUmPG688Ub17dtXfpOcnOzLgANaIigQN6ZMmaLjx4+rtLS0eV19fb1+/etfKy8vr013lOn7v+uuu9r8P++88466d++u6dOnO3+f6cqaOXOmfvGLX2jQoEH2iMYE0h//+Mc22/71r3/V5z73OWVkZOjyyy/Xrbfeqr/85S8XrelCYxSNjY1avny5rr/+evs7+/Tpo3Hjxmn37t32+6NHj9aQIUMu+P+Zdo4dO/aivxfoSAQF4obZoX7qU5/SL3/5y+Z1f/jDH1RdXa0vfvGLbXbyX/rSl+z333rrrfO+t379etXU1NjvX4zp1po9e7bd9jvf+Y4NKrPTLisra95m3759+sxnPqO///3v+sY3vqH58+fr5Zdf1pgxY7Rz58521/mVr3zF/k4z7vKDH/xA3/zmN21gRILn3nvv1YsvvnheG4xdu3bp4MGDUdUFdChzPQrAS2vWrDHXRAnv2rUrvGLFinC3bt3CoVDIfm/SpEnh7Oxse3vAgAHh22+/vfnnKioq7M89+uij5/1/d955Z/iaa64JNzY2On+v+Vmz7N69u3nd4cOHw126dAnn5uY2r7v77rvDKSkp4crKyuZ1r7/+um3nqFGjmtdt27bN/n/ma8R9991n2x2xdetWu81DDz3Upj2R9p48edK24ZFHHjnv++Zn0tPTw7W1tc66gI7GEQXiyhe+8AWdPn1azzzzjE6dOmW/tu52ivjIRz6iT37yk7brKMIcXZijjKlTp9qjjosxRzCmuyni6quvtt1Zzz77rM6ePWuXTZs26e6779aHPvSh5u369etn22VmaZmjl2j95je/se1auHBhm+9F2mu6zUwbzJFV5Lpiph2/+tWvbDvS09Oj/n1ARyAoEFdMf/1nP/tZO4D929/+1u4gJ06c+J7bT5s2TX/60590+PBhe98Mhjc0NNjum2gMHDjwggEUCoVUVVVlF3PbjA209tGPftSON7z22mtR11dZWamrrrpKPXv2dG5n6nr11Vf13HPP2fubN2/WsWPHoq4L6EgEBeKOeadujgpWrVplB5B79OjxntuasQszsyhyVPHzn/9cw4cPv+CO3U/MgPWVV15p6zHMVzPry4QoEGsEBeJObm6uEhMT7eDue3U7RZh35rfffrsNCnNUYY4u2vOu+6WXXmqzzgwYp6Wl2aMbs5jbFzoX4sCBA7adZlA6Wtdee61ef/31NgPwrV122WW2djPj68SJE1q3bp2dFWbWA7FGUCDumOmnjz76qAoLC3XHHXdcdHsTDOXl5Xr44YftjrT1DCmXP//5z9qzZ0/zfdON9Lvf/U633Xab/b/MYm6bda+88krzdqYbyHSPffrTn7ZTZqN1zz332HGHoqKiNt+LjEe0rMuEhJnmW1tby2wneCbJu18NvLf77rsv6m3NEUWvXr3s+ITpqrriiiui/tmsrCzbzfPQQw/ZE+N+8pOf2PUtd+SLFi2y53aYUJgxY4aSkpL005/+VHV1dSouLm5XXdnZ2TYAfvzjH9ujGTMV14xzmLEI8z1zXkfE0KFDbftMXWY8ZNiwYe36XUBH4YgCvpeSkqLJkyfb2+0d7DUnt/3oRz/Sk08+qQULFtiuLDM+8vGPf7x5m+uuu87uyM1O+/vf/74NkQEDBmjbtm121lV7rVmzRkuXLrXnYpijoMWLF9uZXjfddNMFB7XfT11AR0owc2Q79H8EPDBnzhw9/vjjevPNN+2YQjTMdNSvf/3rWrFiheKVOYPb1Ga6vczUXcALHFHA98xHdphZQab/P9qQ8APzHs6EnznqISTgJcYo4Fv/+Mc/7PkFZmaQ+eiNWbNmKQjefvtt/f73v7ddW3v37rUD6YCXCAr4lpnpZM7ANoPXZnD4hhtuUBCYk/zM1Fhz/si3vvUt3XnnnV43CZ0cYxQAACfGKAAATgQFAKBjxijMyUVmiTAnCZmPITAnOkXzKZ0AgPhhRh3MJzSbD6k0H0XTIUEROdEIABAc5mNr+vfv3zGD2a2PKMxVx8zcbnNmqfn0ziAwqfqxj33MzqYxR0x+F7R6DGryB2qKf+bj+M0nBJw8edJeA6VDjijM5+Bc6CLxJiTMRygE5YlgTtgy9QThiRC0egxq8gdq8o9ohg4YzAYAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMApKdoN6+rq7BJRU1NjvyYmJtolCCJ1UE/8oiZ/oKb41546EsLhcDiaDQsLC1VUVNRmfUlJidLS0trXQgCAp0KhkPLy8lRdXa2MjIyOCYoLHVFkZmbqjTfeUK9evRQEDQ0NKi0tVU5OjpKTk+V3Qasn6DWVlZWpsbFRQXm3mpWVxeMUx+rr67Vo0aKogiLqrqfU1FS7tGaeBEF5IgS1pqDVE9SazM4nCDuglnic4ld7aghGZxsA4JIhKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIA/g+J/yv5HY38+Vlf+8EolfzdZ3Zd0178t/zeNWTtGs/4wS88eetbrJiLAZs2apb59+3rdDFwEj1NsRH2Fu1ia9n+m6ckXnzxvXU1djV1eOfmKdhzeocPVhzX2w2M9ayOCzVwe8tixY143AxfB49RJg2LjoY3nhcSN/W7U2GvH6vKUy1UVqtKeN/boz0f+7GkbAaAzibug2FS5qfn2h3t+WDu/ulOXJV523jbmyGLvsb0KAnOB87lz5+rhhx/WgAEDFARBrAnozOJujOJM45nm2yffOWm7mlrLSM3QzVffrCDYs2ePVq9erVGjRqmyslJBEMSagM4s7oJiWL9hzbf/GfqnPrLiI7rxZzfqa898TY/938d06K1DCpKRI0dq/fr1qqqqsjvWiooK+V0QawI6s7jrevrSx7+klbtWavfru+39xnCjHZcwS8Snr/60VnxuhYb0HaJ4Nn/+fJ04cSKqbbOysrRr1y6NHj1aO3bs0KBBgxSPglgTAJ8FRVJikrZO26rvP/99/ddf/0vH3m47o+H5V59XzpM52jdjn/qk91G8WrNmjY4ePdqunzEzOMrLy+N2pxrEmgD4rOvJ6JbaTYtvXaw38t9Q2QNlevzOx3XfkPvULaVb8zZmBlTrKbTx5siRIwqHwxddamtrNWbMGPszRUVFys3NVbwKYk0AfBgUEQkJCbruiuv05aFf1tq71+rFB15UYsK5Jr90/CX53alTpzRu3Dht375dS5Ys0YIFC+R3fq3JDMKvWrWqzfr9+/dr+fLlnrQJbfE4xV7cdT098bcn9M6ZdzTl+il2dlNL6cnpNijMuIXRo0sP+d2hQ4e0d+9eLVu2THPmzFEQ+LWmefPmaePGjQqFQs3r9u3bp1tuuUWnT5/WhAkTlJmZ6WkbwePkhbgLipdPvqyiHUWa/exsO2h9w5U3qGfXnjp++rh+Xf7r86bPjvvwOPnd0KFD7Y61d+/eCgq/1vTUU09p/Pjxys/PV58+TWNf2dnZ9ryQTZs2sfOJEzxOsRd3QRFhjio2/7/NdrmQfx/27xp9zWgFgd92qEGtqVu3bvad6h133KFt27bZdWfOnNHmzZs1fPhwr5uHd/E4xV7cBcXskbN1/RXXa+vLW7X7jd16s/ZNVb1dpbPhs+qT1kc3XnWjHdie8NEJXjcVAZSenq4NGzborrvusn3hpaWl9ggJ8YXHqZMHhRl3uOdj99gF8ELXrl1tFwbiG49T7MT1rCcAgPcICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAEDHXAq1rq7OLhE1NTX2a0NDg12CIFIH9cSvINeUmBic922RWnic4ld76kgIh8PhaDYsLCxUUVFRm/UlJSVKS0trXwsBAJ4KhULKy8tTdXW1MjIyOuaIoqCgQHPnzj3viCIzM1Pl5eVKSUlRUBI2KytLOTk5Sk5OVhDeAZWWlqqsrEyNjY0KgqA9Ri0fJ2qKbw0Bez3V19dHvW3UQZGammqX1swfLAh/tJbMEzsoT26Dx8gfqMkfGgPyempPDcHobAMAXDIEBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKGLszJkzmjp1qgYPHqyDBw963Ry8h1mzZqlv374KkiDWhNggKGJ86cFJkybZ64xXVFRozJgxOnDggNfNwgWY6wgfO3ZMQRLEmhAbBEWM1NXVacKECVq3bl3zhcxra2ttWOzbt8/r5gHAeyIoYmTy5MnasGGDCgoKlJuba9dt2rRJp0+fVnZ2to4cOeJ1EwHfXOs5Pz9f5eXlXjel00jyugGdxezZs/WJT3xC8+bN0/3332/XjRw5UqWlpVq/fr369+/vdROBuHf27FlNmzbNdt8mJiZq6dKlXjepUyAoYsR0MZmltREjRtgFwMUnguTl5enpp5/WjBkzVFxc7HWTOg2CAoAvmIkgZoyve/fuSkhI0IMPPhj1bK+BAwde8vYFGUEBwBfjEtu3b2+evbVy5cqof3bixIkExQfEYDaAuGfGI7Zs2aKePXuqR48e2rlzp8LhcFTLhbp80T4EBfCuPXv2aNWqVW3W79+/X8uXL5cfBammYcOGaevWrUpKSlJOTo5eeOEFr5vUadD1BLzLzEjbuHGjQqFQ8zpzjsstt9xipzGb82AyMzPlJ0GraciQIdq2bZtuvfVWe4Rx0003ed2kToGgAN711FNPafz48XaOfp8+few6c46LOaPenPPipx1qkGvKysqyYde7d2+vm9Jp0PUEvKtbt2723bfZkVZVVTVPydy8ebM958WPgliTQUjEFkEBtJCenm7PoDd94L169bLdG8OHD5efBbEmxBZdTx5Yu3atXRCfunbtartlgiSINSF2OKIAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAICOuRRqXV2dXSJqamrs18TERLsEQaSOhoYGBUGkjqA8PkF8jFrWEsSaiouL1djYqKA897KysgLzempPHQnhcDgczYaFhYUqKipqs76kpERpaWntayEAwFOhUEh5eXmqrq5WRkZGxxxRFBQUaO7cuecdUWRmZqq8vFwpKSkK0juGnJwcJScnKwjv6kpLS1VWVha4d3VBeYxaPk5BrCmIz72ygNRUX18f9bZRB0VqaqpdWjN/sCD80VoyL9agvGANHiN/CGJNQXzuNQakpvbUEIzONgDAJUNQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCIsbOnDmjqVOnavDgwTp48KDXzUEnMmvWLPXt29frZsCHCIoYX3pw0qRJ9jrjFRUVGjNmjA4cOOB1s9BJmGsjHzt2zOtmwIcIihipq6vThAkTtG7duuYLmdfW1tqw2Ldvn9fNA4D3RFDEyOTJk7VhwwYVFBQoNzfXrtu0aZNOnz6t7OxsHTlyxOsm4iJHgzNnztThw4e9bgoQcwRFjMyePVuLFi3S4sWLm9eNHDlSpaWlmj59uvr37+9p++C2Z88erV69WqNGjVJlZaXXzQFiKim2v67zMl1MZmltxIgRdkF8M6G+fv163XXXXTYstm7dqkGDBnndLCAmCAp0evPnz9eJEyei2jYrK0u7du3S6NGjtWPHDsICnQJBgU5vzZo1Onr0aLt+xsweKi8vJyjQKTBGgU7PTCQIh8MXXSKz1IyioqLmSQlA0BEUQBROnTqlcePGafv27VqyZIkWLFggvwzCr1q1qs36/fv3a/ny5Z60Cf5D1xMQhUOHDmnv3r1atmyZ5syZI7+YN2+eNm7cqFAo1LzOnLdzyy232KnZ5tyezMxMT9uI+EdQAFEYOnSoDYvevXvLT5566imNHz9e+fn56tOnj11nztsx54WY83gICUSDricgSn4LCaNbt272iMKEQ1VVVfPnjW3evNlO+QWiQVAAAZeenm4/FSAnJ0e9evXSli1bNHz4cK+bBR8hKDywdu1aO4sGiJWuXbvarqZ//vOfthsNaA+CAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnJIUpbq6OrtE1NTU2K+JiYl2CYJIHQ0NDQqCSB35+flKTk5WUGoqLS1VcXGxGhsbFZTnXVZWVmCed0F/7uUHpKbjx49r8eLFUW2bEI7y4s2FhYUqKipqs76kpERpaWntbyUAwDOhUEh5eXmqrq5WRkZGxwTFhY4oMjMz9e1vf1spKSkK0ju7nJycQLxjiLwDCko9LWsqKysL3BFFEB8naorvI4p+/fpFFRRRdz2lpqbapTXzYg3KCzbCPAmC8EQIaj0Gzzt/oKb41Z4agjG4AAC4ZAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAngiLGzpw5o6lTp2rw4ME6ePCg180BEAeOH5eKi6XbbpOuukrq0sVcA0jq108aNUp6+GHpueek6C4z1/GivnARPrj6+npNnjxZ69ats/fHjBmjrVu32tAA0Dn97GfS3LnS22+3/d6bbzYtJiR++EPpjTekvn1j30aCIkbMZWTvuecebdiwwV520FxKtra21obFli1bdN1113ndRAAxtnSp9I1vnLufkCBlZ0sjR0qXXy699Zb0t79Jzz8vvfOOd+0kKGLEHEmYkCgoKNDrr7+uJ554Qps2bdLYsWOVnZ2tPXv2qH///l43E0CM7N8vFRScu9+rl/T730s33dR229pa6cknpa5d5QnGKGJk9uzZWrRokRYvXty8buTIkfZi7dOnT/dlSJhrVufn56u8vNzrpuB9dIPOnDlThw8fVlD4raYf/1g6e/bc/VWrLhwShjm6eOABqXt3eYKgiBHTxTRv3rw260eMGKHvfve78puzZ8/q3nvv1bJly7RmzRqvm4N2Mkewq1ev1qhRo1RZWakg8FtNW7acu/0v/yJNmKC4RVDgfc3cmjJlikpKSjRjxgwVm+ka8BVzNLt+/XpVVVXZHWtFRYX8zm81HT167vbAgVJii73xgQNN4xWtl/vv96SpjFGg/SZNmmRnbnXv3l0JCQl68MEHo/q5WbNmaaB5ReCSmj9/vk6cOBHVtllZWdq1a5dGjx6tHTt2aNCgQYpHQaypJRMC8YygQLvHJbZv325vV1dXa+XKlVH/7MSJEwmKGDBdgUdbvl2NwrFjx+xYU7zuVINY07/+q/TSS023zVdzjkQkMK64omlGlLFwoRQKeddOg64ntEtiYqKdztuzZ0/16NFDO3fuVDgcjmox4zS49I4cORLV4xGZnm0UFRUpNzdX8SqINd1667nbZhqsmfEU0bOn9B//0bR4NdOpJYIC7TZs2DB7omBSUpJycnL0wgsveN0ktNOpU6c0btw4e3S4ZMkSLViwQH7nt5pmzpQuu+zc/a99remciXhEUOB9GTJkiLZt26YuXbrYIwz4y6FDh7R37147a+2RRx5REPitpuuuk1pOeDRnYA8fLt1xh1RYKH3ve9JXvyrV1MhzjFHgfTODhvv27VPv3r29bgraaejQoXbHGqTHzo81FRRI6elNZ2fX1TWdV/HMM03LhZiT8rxAUOAD8dOLEsF/7PxY00MPmZmETZ/5tHmzZGb1mgleyclSnz6SGYu/+WbpzjtNGHrTRoLCA2vXrrULABjmU2LN7CazxCPGKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAB0zDWz6+rq7BJRU1NjvyYmJtolCCJ1NDQ0KAgidQSlnpa15OfnK9lcfT4gNZWWlgbycQpiTcXFxWpsbJTf1dfXR71tQjgcDkezYWFhoYqKitqsLykpUVpaWvtaCADwVCgUUl5enqqrq5WRkdExRxQFBQWaO3fueUcUmZmZKi8vV0pKioJyRJGVlaWcnJxAvFuNvFMNSj0GNflDkGsqKyvrdEcUUQdFamqqXVozf7Ag/NFaMk/soDy5g1iPQU3+EMSaGgOyz2tPDcEYXAAAXDIEBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0ERY2fOnNHUqVM1ePBgHTx40OvmwKislC6/XEpIaFpuu01qfeFHcz8n59w26enSSy951WLwWoopgiLGV5SaNGmSvXxsRUWFxowZowMHDnjdLFx7rfSf/3nufmmptHLl+dusWCFt3nzu/g9/KA0cGLs24jy8lmKLoIiRuro6TZgwQevWrWu+Pm1tba19gu/bt8/r5mH6dGn8+HP3H3lEirxLNV/N/Yhx46QHHoh9G2HxWoo9giJGJk+erA0bNthrj+fm5tp1mzZt0unTp5Wdna0jR4543USsXi316tV0OxSSpk0zeyXp3nul06eb1vfsKT3+uKfN7Ox4LcUeQREjs2fP1qJFi7R48eLmdSNHjrQXa58+fbr69+8vP15zNz8/X+Xl5QqEfv2kRx89d3/nTmnECOl///fcOvP9q65SELpuZs6cqcOHD8tvgvhaindJXjegszCHxWZpbcSIEXbxm7Nnz2ratGm2jzgxMVFLly5VIEyaJE2dKv3iF033X3zx3Pfy8qQvfEFBsGfPHq1evVrr16/X1q1bda0Zp/GJoL2W/IAjCryv2SZTpkyxITFjxgwVFxcrUMzAtTm6aOnKK9sOcPuYeQduQqKqqkqjRo2yA8LAe+GIAu1mZpuYgcTu3bsrISFBDz74YFQ/N2vWLA30w0wh08f91lvnrzP3X3lFuuEGxbv58+frxIkTUW2blZWlXbt2afTo0dqxY4cGDRp0ydsH/yEo0O5xie3bt9vb1dXVWtmOd9kTJ06M/6BoaGgavDaD2Bdav3u3lJqqeLZmzRodPXq0XT9z7NgxO9ZEUOBC6HpCu5jxiC1btqhnz57q0aOHdu7cqXA4HNVyoX7luLNwofS3v527//Wvn7tdViZ9+9uKd2bWTzSPR2RKqVFUVNQ8gwhojaBAuw0bNswOgCYlJSknJ0cvvPCCAsHU0XK85ctfbhqv+MpXzq1btkx67jn53alTpzRu3Dh7dLhkyRItWLDA6yYhjhEUeF+GDBmibdu2qUuXLvYIw/feflu67z4znavp/jXXSD/6UdNt8/VDH2q63djYtF1trfzs0KFD2rt3r5YtW6ZHWp5MCFwAYxR438xAqDkTtnfv3vK9/Hyz92y6nZgoPfGE1K1b033zOVD//d/S6NFNQfLyy9KcOdJjj8mvhg4dasMiEI8dLjmOKPCBBGJHs3Gj9NOfnrtvQmDUqPO3ufnm8z/Gw5zFvWGD/CwQjx1igqDwwNq1a+1gIuKE+ewm83hEFvOBfxfyve+dv93tt8e6pWiF11JsEBQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOCUpCjV1dXZJaKmpsZ+TUxMtEsQROpoaGhQEETqCEo9BjX5Q5BrSgzY/i4aCeEoLzhbWFiooqKiNutLSkqUlpbWvhYCADwVCoWUl5en6upqZWRkdMwRRUFBgebOnXveEUVmZqbKy8uVkpKioCRsVlaWysrK1NjYKL8LWj0GNfkDNcW/+vr6qLeNOihSU1Pt0pr5gwXhjxbkmoJWj0FN/kBN8as9NQSjsw0AcMkQFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIATQQEAcCIoAABOBAUAwImgAAA4ERQAACeCAgDgRFAAAJwICgCAE0EBAHAiKAAATgQFAMCJoAAAOBEUAAAnggIA4ERQAACcCAoAgBNBAQBwIigAAE4EBQDAiaAAADglKUp1dXV2iaiurrZfGxoaFBSJiYkKhUKqr69XY2Oj/C5o9RjU5A/UFP8i++5wOHzxjcNRWrhwofnfWFhYWFgUnKWysvKi+/8E88/7OaI4efKkBgwYoFdffVXdu3dXENTU1CgzM1OvvfaaMjIy5HdBq8egJn+gpvhneoWuvvpqnThxQj169OiYrqfU1FS7tGZCIgh/tJZMPUGqKWj1GNTkD9Tkjy61i24Tk5YAAHyLoAAAXJqgMN1QCxcuvGB3lF8Fraag1WNQkz9QU7DqiXowGwDQOdH1BABwIigAAE4EBQDAiaAAADgRFAAAJ4ICAOBEUAAAnAgKAIBc/j+c6Peoa2v/OgAAAABJRU5ErkJggg==",
|
||
"text/plain": [
|
||
"<Figure size 640x480 with 1 Axes>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"my_policy = [\n",
|
||
" A_RIGHT,\n",
|
||
" A_RIGHT,\n",
|
||
" A_RIGHT,\n",
|
||
" A_DOWN,\n",
|
||
" A_DOWN, # First row\n",
|
||
" A_UP,\n",
|
||
" A_DOWN,\n",
|
||
" A_DOWN,\n",
|
||
" A_LEFT, # Second row\n",
|
||
" A_UP,\n",
|
||
" A_RIGHT,\n",
|
||
" A_DOWN, # Third row\n",
|
||
" A_UP,\n",
|
||
" A_LEFT,\n",
|
||
" A_RIGHT,\n",
|
||
" A_RIGHT,\n",
|
||
" A_RIGHT, # Fourth row\n",
|
||
" A_UP,\n",
|
||
" A_LEFT,\n",
|
||
" A_DOWN,\n",
|
||
" A_RIGHT,\n",
|
||
" A_UP, # Fifth row\n",
|
||
"]\n",
|
||
"\n",
|
||
"V_my_policy = policy_evaluation(policy=my_policy, P=P, R=R, gamma=gamma)\n",
|
||
"\n",
|
||
"plot_values(V=V_my_policy, title=\"Value function: my policy\")\n",
|
||
"plot_policy(policy=my_policy, title=\"My policy\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e61f5ee8-f9cd-4fbc-96c0-0a8d661bd1e5",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Exercise 10.** (optional) How can we find an optimal policy?\n",
|
||
"(We will discuss this question next week, but you can already start thinking about it!)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "00ae548b",
|
||
"metadata": {},
|
||
"source": [
|
||
"To find an optimal policy $π^*$ (a policy that yields the highest possible expected return from every state), we generally use one of two main dynamic programming algorithms:\n",
|
||
"\n",
|
||
"1. **Policy Iteration**: This method alternates between two steps until convergence:\n",
|
||
"\n",
|
||
"- *Policy Evaluation*: Calculate the value function Vπ(s) for the current specific policy (as we did in Exercise 8).\n",
|
||
"\n",
|
||
"- *Policy Improvement*: Update the policy to be greedy with respect to the current values. For every state s, we choose the action a that maximizes the expected next value:\n",
|
||
" $$π_{new}(s) = argmax_{a} \\sum_{s\\prime} P({s \\prime}∣s,a)[R(s)+ \\gamma V_{\\pi}({s\\prime})]$$\n",
|
||
"\n",
|
||
"1. **Value Iteration**: Instead of evaluating a specific policy until convergence every time, we iteratively update the value function directly using the *Bellman Optimality Equation*:\n",
|
||
" $$V_{k+1}(s) = max_a (R(s)+ \\gamma \\sum_{s\\prime} P(s\\prime∣s,a)V_k(s\\prime))$$\n",
|
||
"\n",
|
||
" Once the values converge to the optimal values $V^{*}$, we simply extract the optimal policy by acting greedily towards those values."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "studies",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.13.9"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|