Reinforcement Learning: Balancing a CartPole using Q-Learning
\n",
"
\n",
"Nazar Khan\n",
" CVML Lab\n",
" University of The Punjab\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This tutorial helps you get started with OpenAI Gymnasium (the updated version of OpenAI Gym) for reinforcement learning. This tutorial will provide a visual, hands-on experience, where you can see how an agent learns in a simple environment. We'll use **CartPole** as the example environment, which is one of the classic environments in RL.\n",
"\n",
"### **Getting Started with OpenAI Gymnasium: A Visual Tutorial**\n",
"\n",
"In this tutorial, you will learn how to:\n",
"1. Install OpenAI Gymnasium and dependencies.\n",
"2. Understand the CartPole environment.\n",
"3. Create and train a reinforcement learning agent using Q-learning.\n",
"4. Visualize how the agent learns over time.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"### **Step 1: Install OpenAI Gymnasium and Dependencies**\n",
"\n",
"First, you need to install **OpenAI Gymnasium** (Gym’s newer version) and some other dependencies.\n",
"\n",
"#### Install the necessary libraries:\n",
"```bash\n",
"pip install gymnasium[all] numpy matplotlib\n",
"```\n",
"\n",
"- `gymnasium[all]`: This installs all the environments (including the classic CartPole environment) and necessary dependencies.\n",
"- `numpy`: For array and matrix manipulations.\n",
"- `matplotlib`: For visualizing the training process."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"### **Step 2: Import Libraries and Set Up the CartPole Environment**\n",
"\n",
"Let’s start by importing the necessary libraries and initializing the **CartPole** environment."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
">>>>\n",
"Observation Space: Box([-4.8 -inf -0.41887903 -inf], [4.8 inf 0.41887903 inf], (4,), float32)\n",
"Action Space: Discrete(2)\n"
]
}
],
"source": [
"import gymnasium as gym\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from matplotlib.animation import FuncAnimation\n",
"import time\n",
"\n",
"# Create the CartPole environment\n",
"env = gym.make(\"CartPole-v1\", render_mode='rgb_array')\n",
"print(env)\n",
"\n",
"# Reset the environment to start\n",
"observation, info = env.reset()\n",
"print(\"Observation Space:\", env.observation_space)\n",
"print(\"Action Space:\", env.action_space)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- `gym.make(\"CartPole-v1\")`: This initializes the CartPole environment.\n",
"- `render_mode='human'`: This ensures that the environment renders a visual representation for human viewers.\n",
"- `env.reset()`: Resets the environment to its initial state.\n",
"\n",
"The output should display information about the observation and action spaces. For CartPole:\n",
"- **Observation space** is a continuous space with 4 elements (Cart position, Cart velocity, Pole angle, Pole velocity).\n",
"- **Action space** is discrete: 0 (move left) or 1 (move right)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"### **Step 3: Define Q-Learning Algorithm**\n",
"\n",
"We'll now define a simple **Q-learning** algorithm for training the agent to balance the pole.\n",
"\n",
"#### Key elements for Q-learning:\n",
"1. **Q-table**: A table that stores Q-values for each state-action pair.\n",
"2. **Learning Rate (α)**: Determines how quickly the agent updates its Q-values.\n",
"3. **Discount Factor (γ)**: Determines the importance of future rewards.\n",
"4. **Exploration-Exploitation (ε)**: Determines the agent's strategy of exploration (random actions) versus exploitation (choosing the best-known action)."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Shape of Q-table: (24, 24, 24, 24, 2)\n"
]
}
],
"source": [
"# Parameters for Q-Learning\n",
"alpha = 0.1 # Learning rate\n",
"gamma = 0.99 # Discount factor\n",
"epsilon = 0.1 # Exploration rate\n",
"n_episodes = 30000 # Number of episodes for training\n",
"\n",
"# Initialize Q-table (for discrete states)\n",
"n_actions = env.action_space.n\n",
"q_table = np.zeros((24, 24, 24, 24, n_actions)) # For CartPole, discretized states (4D)\n",
"print(\"Shape of Q-table: \", q_table.shape)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"- **Discretizing the continuous state space**: CartPole's state space is continuous, but we’ll discretize it to make Q-learning feasible. Here, the 4 dimensions of the state space are divided into 24 bins each."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"### **Step 4: Discretize the Continuous State Space**\n",
"\n",
"To apply Q-learning, we need to convert the continuous state space into discrete states. We’ll use `numpy`'s `linspace` to create bins."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"# Define state space boundaries and number of bins for each dimension\n",
"state_bins = [\n",
" np.linspace(-2.4, 2.4, 24), # Cart position\n",
" np.linspace(-3.0, 3.0, 24), # Cart velocity\n",
" np.linspace(-0.5, 0.5, 24), # Pole angle\n",
" np.linspace(-2.0, 2.0, 24) # Pole velocity\n",
"]\n",
"\n",
"def discretize_state(state):\n",
" \"\"\"\n",
" Discretize the continuous state to an index in the Q-table.\n",
" \"\"\"\n",
" state_discretized = []\n",
" for i, (s, bins) in enumerate(zip(state, state_bins)):\n",
" state_discretized.append(np.digitize(s, bins) - 1)\n",
" return tuple(state_discretized)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- `np.digitize(s, bins)` maps each continuous state value to a bin index.\n",
"- This discretizes the 4-dimensional state space into 4 indices, each ranging from 0 to 23 (as we have 24 bins for each dimension)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"---\n",
"\n",
"### **Step 5: Train the Agent with Q-learning**\n",
"\n",
"Now we will implement the Q-learning training loop. In each episode, the agent will:\n",
"1. Choose an action based on an ε-greedy policy.\n",
"2. Take the action and observe the new state and reward.\n",
"3. Update the Q-table using the Q-learning update rule."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Episode 0/30000, Total Reward: 10.0\n",
"Episode 50/30000, Total Reward: 13.0\n",
"Episode 100/30000, Total Reward: 9.0\n",
"Episode 150/30000, Total Reward: 11.0\n",
"Episode 200/30000, Total Reward: 11.0\n",
"Episode 250/30000, Total Reward: 10.0\n",
"Episode 300/30000, Total Reward: 16.0\n",
"Episode 350/30000, Total Reward: 14.0\n",
"Episode 400/30000, Total Reward: 9.0\n",
"Episode 450/30000, Total Reward: 10.0\n",
"Episode 500/30000, Total Reward: 12.0\n",
"Episode 550/30000, Total Reward: 9.0\n",
"Episode 600/30000, Total Reward: 10.0\n",
"Episode 650/30000, Total Reward: 15.0\n",
"Episode 700/30000, Total Reward: 10.0\n",
"Episode 750/30000, Total Reward: 15.0\n",
"Episode 800/30000, Total Reward: 13.0\n",
"Episode 850/30000, Total Reward: 8.0\n",
"Episode 900/30000, Total Reward: 15.0\n",
"Episode 950/30000, Total Reward: 15.0\n",
"Episode 1000/30000, Total Reward: 15.0\n",
"Episode 1050/30000, Total Reward: 11.0\n",
"Episode 1100/30000, Total Reward: 16.0\n",
"Episode 1150/30000, Total Reward: 14.0\n",
"Episode 1200/30000, Total Reward: 18.0\n",
"Episode 1250/30000, Total Reward: 15.0\n",
"Episode 1300/30000, Total Reward: 19.0\n",
"Episode 1350/30000, Total Reward: 20.0\n",
"Episode 1400/30000, Total Reward: 11.0\n",
"Episode 1450/30000, Total Reward: 18.0\n",
"Episode 1500/30000, Total Reward: 37.0\n",
"Episode 1550/30000, Total Reward: 17.0\n",
"Episode 1600/30000, Total Reward: 14.0\n",
"Episode 1650/30000, Total Reward: 11.0\n",
"Episode 1700/30000, Total Reward: 16.0\n",
"Episode 1750/30000, Total Reward: 12.0\n",
"Episode 1800/30000, Total Reward: 16.0\n",
"Episode 1850/30000, Total Reward: 21.0\n",
"Episode 1900/30000, Total Reward: 13.0\n",
"Episode 1950/30000, Total Reward: 29.0\n",
"Episode 2000/30000, Total Reward: 14.0\n",
"Episode 2050/30000, Total Reward: 19.0\n",
"Episode 2100/30000, Total Reward: 27.0\n",
"Episode 2150/30000, Total Reward: 57.0\n",
"Episode 2200/30000, Total Reward: 74.0\n",
"Episode 2250/30000, Total Reward: 32.0\n",
"Episode 2300/30000, Total Reward: 57.0\n",
"Episode 2350/30000, Total Reward: 88.0\n",
"Episode 2400/30000, Total Reward: 59.0\n",
"Episode 2450/30000, Total Reward: 42.0\n",
"Episode 2500/30000, Total Reward: 54.0\n",
"Episode 2550/30000, Total Reward: 36.0\n",
"Episode 2600/30000, Total Reward: 44.0\n",
"Episode 2650/30000, Total Reward: 76.0\n",
"Episode 2700/30000, Total Reward: 35.0\n",
"Episode 2750/30000, Total Reward: 55.0\n",
"Episode 2800/30000, Total Reward: 34.0\n",
"Episode 2850/30000, Total Reward: 60.0\n",
"Episode 2900/30000, Total Reward: 76.0\n",
"Episode 2950/30000, Total Reward: 35.0\n",
"Episode 3000/30000, Total Reward: 51.0\n",
"Episode 3050/30000, Total Reward: 39.0\n",
"Episode 3100/30000, Total Reward: 36.0\n",
"Episode 3150/30000, Total Reward: 79.0\n",
"Episode 3200/30000, Total Reward: 63.0\n",
"Episode 3250/30000, Total Reward: 50.0\n",
"Episode 3300/30000, Total Reward: 76.0\n",
"Episode 3350/30000, Total Reward: 87.0\n",
"Episode 3400/30000, Total Reward: 33.0\n",
"Episode 3450/30000, Total Reward: 85.0\n",
"Episode 3500/30000, Total Reward: 42.0\n",
"Episode 3550/30000, Total Reward: 38.0\n",
"Episode 3600/30000, Total Reward: 55.0\n",
"Episode 3650/30000, Total Reward: 61.0\n",
"Episode 3700/30000, Total Reward: 41.0\n",
"Episode 3750/30000, Total Reward: 64.0\n",
"Episode 3800/30000, Total Reward: 68.0\n",
"Episode 3850/30000, Total Reward: 76.0\n",
"Episode 3900/30000, Total Reward: 89.0\n",
"Episode 3950/30000, Total Reward: 67.0\n",
"Episode 4000/30000, Total Reward: 156.0\n",
"Episode 4050/30000, Total Reward: 65.0\n",
"Episode 4100/30000, Total Reward: 163.0\n",
"Episode 4150/30000, Total Reward: 117.0\n",
"Episode 4200/30000, Total Reward: 51.0\n",
"Episode 4250/30000, Total Reward: 75.0\n",
"Episode 4300/30000, Total Reward: 55.0\n",
"Episode 4350/30000, Total Reward: 51.0\n",
"Episode 4400/30000, Total Reward: 193.0\n",
"Episode 4450/30000, Total Reward: 48.0\n",
"Episode 4500/30000, Total Reward: 180.0\n",
"Episode 4550/30000, Total Reward: 56.0\n",
"Episode 4600/30000, Total Reward: 203.0\n",
"Episode 4650/30000, Total Reward: 57.0\n",
"Episode 4700/30000, Total Reward: 89.0\n",
"Episode 4750/30000, Total Reward: 105.0\n",
"Episode 4800/30000, Total Reward: 110.0\n",
"Episode 4850/30000, Total Reward: 72.0\n",
"Episode 4900/30000, Total Reward: 73.0\n",
"Episode 4950/30000, Total Reward: 127.0\n",
"Episode 5000/30000, Total Reward: 44.0\n",
"Episode 5050/30000, Total Reward: 67.0\n",
"Episode 5100/30000, Total Reward: 76.0\n",
"Episode 5150/30000, Total Reward: 107.0\n",
"Episode 5200/30000, Total Reward: 154.0\n",
"Episode 5250/30000, Total Reward: 71.0\n",
"Episode 5300/30000, Total Reward: 52.0\n",
"Episode 5350/30000, Total Reward: 80.0\n",
"Episode 5400/30000, Total Reward: 78.0\n",
"Episode 5450/30000, Total Reward: 79.0\n",
"Episode 5500/30000, Total Reward: 99.0\n",
"Episode 5550/30000, Total Reward: 114.0\n",
"Episode 5600/30000, Total Reward: 70.0\n",
"Episode 5650/30000, Total Reward: 98.0\n",
"Episode 5700/30000, Total Reward: 307.0\n",
"Episode 5750/30000, Total Reward: 82.0\n",
"Episode 5800/30000, Total Reward: 130.0\n",
"Episode 5850/30000, Total Reward: 21.0\n",
"Episode 5900/30000, Total Reward: 72.0\n",
"Episode 5950/30000, Total Reward: 57.0\n",
"Episode 6000/30000, Total Reward: 44.0\n",
"Episode 6050/30000, Total Reward: 95.0\n",
"Episode 6100/30000, Total Reward: 58.0\n",
"Episode 6150/30000, Total Reward: 103.0\n",
"Episode 6200/30000, Total Reward: 59.0\n",
"Episode 6250/30000, Total Reward: 72.0\n",
"Episode 6300/30000, Total Reward: 133.0\n",
"Episode 6350/30000, Total Reward: 21.0\n",
"Episode 6400/30000, Total Reward: 44.0\n",
"Episode 6450/30000, Total Reward: 73.0\n",
"Episode 6500/30000, Total Reward: 92.0\n",
"Episode 6550/30000, Total Reward: 82.0\n",
"Episode 6600/30000, Total Reward: 57.0\n",
"Episode 6650/30000, Total Reward: 30.0\n",
"Episode 6700/30000, Total Reward: 93.0\n",
"Episode 6750/30000, Total Reward: 57.0\n",
"Episode 6800/30000, Total Reward: 170.0\n",
"Episode 6850/30000, Total Reward: 62.0\n",
"Episode 6900/30000, Total Reward: 52.0\n",
"Episode 6950/30000, Total Reward: 143.0\n",
"Episode 7000/30000, Total Reward: 69.0\n",
"Episode 7050/30000, Total Reward: 113.0\n",
"Episode 7100/30000, Total Reward: 115.0\n",
"Episode 7150/30000, Total Reward: 94.0\n",
"Episode 7200/30000, Total Reward: 103.0\n",
"Episode 7250/30000, Total Reward: 116.0\n",
"Episode 7300/30000, Total Reward: 91.0\n",
"Episode 7350/30000, Total Reward: 87.0\n",
"Episode 7400/30000, Total Reward: 48.0\n",
"Episode 7450/30000, Total Reward: 110.0\n",
"Episode 7500/30000, Total Reward: 106.0\n",
"Episode 7550/30000, Total Reward: 117.0\n",
"Episode 7600/30000, Total Reward: 158.0\n",
"Episode 7650/30000, Total Reward: 45.0\n",
"Episode 7700/30000, Total Reward: 61.0\n",
"Episode 7750/30000, Total Reward: 172.0\n",
"Episode 7800/30000, Total Reward: 207.0\n",
"Episode 7850/30000, Total Reward: 97.0\n",
"Episode 7900/30000, Total Reward: 55.0\n",
"Episode 7950/30000, Total Reward: 69.0\n",
"Episode 8000/30000, Total Reward: 87.0\n",
"Episode 8050/30000, Total Reward: 197.0\n",
"Episode 8100/30000, Total Reward: 70.0\n",
"Episode 8150/30000, Total Reward: 101.0\n",
"Episode 8200/30000, Total Reward: 31.0\n",
"Episode 8250/30000, Total Reward: 141.0\n",
"Episode 8300/30000, Total Reward: 161.0\n",
"Episode 8350/30000, Total Reward: 45.0\n",
"Episode 8400/30000, Total Reward: 53.0\n",
"Episode 8450/30000, Total Reward: 111.0\n",
"Episode 8500/30000, Total Reward: 170.0\n",
"Episode 8550/30000, Total Reward: 175.0\n",
"Episode 8600/30000, Total Reward: 89.0\n",
"Episode 8650/30000, Total Reward: 118.0\n",
"Episode 8700/30000, Total Reward: 92.0\n",
"Episode 8750/30000, Total Reward: 119.0\n",
"Episode 8800/30000, Total Reward: 98.0\n",
"Episode 8850/30000, Total Reward: 163.0\n",
"Episode 8900/30000, Total Reward: 56.0\n",
"Episode 8950/30000, Total Reward: 88.0\n",
"Episode 9000/30000, Total Reward: 96.0\n",
"Episode 9050/30000, Total Reward: 83.0\n",
"Episode 9100/30000, Total Reward: 73.0\n",
"Episode 9150/30000, Total Reward: 151.0\n",
"Episode 9200/30000, Total Reward: 155.0\n",
"Episode 9250/30000, Total Reward: 98.0\n",
"Episode 9300/30000, Total Reward: 157.0\n",
"Episode 9350/30000, Total Reward: 113.0\n",
"Episode 9400/30000, Total Reward: 118.0\n",
"Episode 9450/30000, Total Reward: 93.0\n",
"Episode 9500/30000, Total Reward: 113.0\n",
"Episode 9550/30000, Total Reward: 45.0\n",
"Episode 9600/30000, Total Reward: 106.0\n",
"Episode 9650/30000, Total Reward: 110.0\n",
"Episode 9700/30000, Total Reward: 116.0\n",
"Episode 9750/30000, Total Reward: 168.0\n",
"Episode 9800/30000, Total Reward: 139.0\n",
"Episode 9850/30000, Total Reward: 115.0\n",
"Episode 9900/30000, Total Reward: 59.0\n",
"Episode 9950/30000, Total Reward: 67.0\n",
"Episode 10000/30000, Total Reward: 197.0\n",
"Episode 10050/30000, Total Reward: 102.0\n",
"Episode 10100/30000, Total Reward: 151.0\n",
"Episode 10150/30000, Total Reward: 115.0\n",
"Episode 10200/30000, Total Reward: 135.0\n",
"Episode 10250/30000, Total Reward: 136.0\n",
"Episode 10300/30000, Total Reward: 156.0\n",
"Episode 10350/30000, Total Reward: 204.0\n",
"Episode 10400/30000, Total Reward: 92.0\n",
"Episode 10450/30000, Total Reward: 119.0\n",
"Episode 10500/30000, Total Reward: 101.0\n",
"Episode 10550/30000, Total Reward: 117.0\n",
"Episode 10600/30000, Total Reward: 107.0\n",
"Episode 10650/30000, Total Reward: 94.0\n",
"Episode 10700/30000, Total Reward: 88.0\n",
"Episode 10750/30000, Total Reward: 112.0\n",
"Episode 10800/30000, Total Reward: 114.0\n",
"Episode 10850/30000, Total Reward: 159.0\n",
"Episode 10900/30000, Total Reward: 102.0\n",
"Episode 10950/30000, Total Reward: 111.0\n",
"Episode 11000/30000, Total Reward: 53.0\n",
"Episode 11050/30000, Total Reward: 111.0\n",
"Episode 11100/30000, Total Reward: 26.0\n",
"Episode 11150/30000, Total Reward: 58.0\n",
"Episode 11200/30000, Total Reward: 22.0\n",
"Episode 11250/30000, Total Reward: 81.0\n",
"Episode 11300/30000, Total Reward: 49.0\n",
"Episode 11350/30000, Total Reward: 19.0\n",
"Episode 11400/30000, Total Reward: 52.0\n",
"Episode 11450/30000, Total Reward: 57.0\n",
"Episode 11500/30000, Total Reward: 66.0\n",
"Episode 11550/30000, Total Reward: 85.0\n",
"Episode 11600/30000, Total Reward: 76.0\n",
"Episode 11650/30000, Total Reward: 92.0\n",
"Episode 11700/30000, Total Reward: 36.0\n",
"Episode 11750/30000, Total Reward: 107.0\n",
"Episode 11800/30000, Total Reward: 43.0\n",
"Episode 11850/30000, Total Reward: 108.0\n",
"Episode 11900/30000, Total Reward: 106.0\n",
"Episode 11950/30000, Total Reward: 78.0\n",
"Episode 12000/30000, Total Reward: 46.0\n",
"Episode 12050/30000, Total Reward: 69.0\n",
"Episode 12100/30000, Total Reward: 77.0\n",
"Episode 12150/30000, Total Reward: 35.0\n",
"Episode 12200/30000, Total Reward: 137.0\n",
"Episode 12250/30000, Total Reward: 96.0\n",
"Episode 12300/30000, Total Reward: 38.0\n",
"Episode 12350/30000, Total Reward: 109.0\n",
"Episode 12400/30000, Total Reward: 55.0\n",
"Episode 12450/30000, Total Reward: 62.0\n",
"Episode 12500/30000, Total Reward: 133.0\n",
"Episode 12550/30000, Total Reward: 70.0\n",
"Episode 12600/30000, Total Reward: 101.0\n",
"Episode 12650/30000, Total Reward: 45.0\n",
"Episode 12700/30000, Total Reward: 80.0\n",
"Episode 12750/30000, Total Reward: 67.0\n",
"Episode 12800/30000, Total Reward: 49.0\n",
"Episode 12850/30000, Total Reward: 70.0\n",
"Episode 12900/30000, Total Reward: 66.0\n",
"Episode 12950/30000, Total Reward: 70.0\n",
"Episode 13000/30000, Total Reward: 69.0\n",
"Episode 13050/30000, Total Reward: 112.0\n",
"Episode 13100/30000, Total Reward: 83.0\n",
"Episode 13150/30000, Total Reward: 21.0\n",
"Episode 13200/30000, Total Reward: 101.0\n",
"Episode 13250/30000, Total Reward: 62.0\n",
"Episode 13300/30000, Total Reward: 114.0\n",
"Episode 13350/30000, Total Reward: 110.0\n",
"Episode 13400/30000, Total Reward: 87.0\n",
"Episode 13450/30000, Total Reward: 154.0\n",
"Episode 13500/30000, Total Reward: 141.0\n",
"Episode 13550/30000, Total Reward: 71.0\n",
"Episode 13600/30000, Total Reward: 128.0\n",
"Episode 13650/30000, Total Reward: 121.0\n",
"Episode 13700/30000, Total Reward: 30.0\n",
"Episode 13750/30000, Total Reward: 50.0\n",
"Episode 13800/30000, Total Reward: 116.0\n",
"Episode 13850/30000, Total Reward: 78.0\n",
"Episode 13900/30000, Total Reward: 123.0\n",
"Episode 13950/30000, Total Reward: 27.0\n",
"Episode 14000/30000, Total Reward: 77.0\n",
"Episode 14050/30000, Total Reward: 56.0\n",
"Episode 14100/30000, Total Reward: 70.0\n",
"Episode 14150/30000, Total Reward: 75.0\n",
"Episode 14200/30000, Total Reward: 50.0\n",
"Episode 14250/30000, Total Reward: 53.0\n",
"Episode 14300/30000, Total Reward: 82.0\n",
"Episode 14350/30000, Total Reward: 115.0\n",
"Episode 14400/30000, Total Reward: 55.0\n",
"Episode 14450/30000, Total Reward: 79.0\n",
"Episode 14500/30000, Total Reward: 141.0\n",
"Episode 14550/30000, Total Reward: 78.0\n",
"Episode 14600/30000, Total Reward: 137.0\n",
"Episode 14650/30000, Total Reward: 62.0\n",
"Episode 14700/30000, Total Reward: 93.0\n",
"Episode 14750/30000, Total Reward: 111.0\n",
"Episode 14800/30000, Total Reward: 59.0\n",
"Episode 14850/30000, Total Reward: 89.0\n",
"Episode 14900/30000, Total Reward: 82.0\n",
"Episode 14950/30000, Total Reward: 70.0\n",
"Episode 15000/30000, Total Reward: 74.0\n",
"Episode 15050/30000, Total Reward: 81.0\n",
"Episode 15100/30000, Total Reward: 59.0\n",
"Episode 15150/30000, Total Reward: 59.0\n",
"Episode 15200/30000, Total Reward: 67.0\n",
"Episode 15250/30000, Total Reward: 87.0\n",
"Episode 15300/30000, Total Reward: 54.0\n",
"Episode 15350/30000, Total Reward: 108.0\n",
"Episode 15400/30000, Total Reward: 82.0\n",
"Episode 15450/30000, Total Reward: 76.0\n",
"Episode 15500/30000, Total Reward: 79.0\n",
"Episode 15550/30000, Total Reward: 95.0\n",
"Episode 15600/30000, Total Reward: 90.0\n",
"Episode 15650/30000, Total Reward: 101.0\n",
"Episode 15700/30000, Total Reward: 90.0\n",
"Episode 15750/30000, Total Reward: 128.0\n",
"Episode 15800/30000, Total Reward: 134.0\n",
"Episode 15850/30000, Total Reward: 97.0\n",
"Episode 15900/30000, Total Reward: 93.0\n",
"Episode 15950/30000, Total Reward: 117.0\n",
"Episode 16000/30000, Total Reward: 87.0\n",
"Episode 16050/30000, Total Reward: 129.0\n",
"Episode 16100/30000, Total Reward: 113.0\n",
"Episode 16150/30000, Total Reward: 86.0\n",
"Episode 16200/30000, Total Reward: 184.0\n",
"Episode 16250/30000, Total Reward: 96.0\n",
"Episode 16300/30000, Total Reward: 90.0\n",
"Episode 16350/30000, Total Reward: 103.0\n",
"Episode 16400/30000, Total Reward: 144.0\n",
"Episode 16450/30000, Total Reward: 87.0\n",
"Episode 16500/30000, Total Reward: 150.0\n",
"Episode 16550/30000, Total Reward: 162.0\n",
"Episode 16600/30000, Total Reward: 125.0\n",
"Episode 16650/30000, Total Reward: 94.0\n",
"Episode 16700/30000, Total Reward: 63.0\n",
"Episode 16750/30000, Total Reward: 110.0\n",
"Episode 16800/30000, Total Reward: 53.0\n",
"Episode 16850/30000, Total Reward: 80.0\n",
"Episode 16900/30000, Total Reward: 77.0\n",
"Episode 16950/30000, Total Reward: 128.0\n",
"Episode 17000/30000, Total Reward: 87.0\n",
"Episode 17050/30000, Total Reward: 126.0\n",
"Episode 17100/30000, Total Reward: 105.0\n",
"Episode 17150/30000, Total Reward: 255.0\n",
"Episode 17200/30000, Total Reward: 119.0\n",
"Episode 17250/30000, Total Reward: 38.0\n",
"Episode 17300/30000, Total Reward: 130.0\n",
"Episode 17350/30000, Total Reward: 125.0\n",
"Episode 17400/30000, Total Reward: 126.0\n",
"Episode 17450/30000, Total Reward: 131.0\n",
"Episode 17500/30000, Total Reward: 87.0\n",
"Episode 17550/30000, Total Reward: 187.0\n",
"Episode 17600/30000, Total Reward: 206.0\n",
"Episode 17650/30000, Total Reward: 176.0\n",
"Episode 17700/30000, Total Reward: 113.0\n",
"Episode 17750/30000, Total Reward: 150.0\n",
"Episode 17800/30000, Total Reward: 295.0\n",
"Episode 17850/30000, Total Reward: 120.0\n",
"Episode 17900/30000, Total Reward: 185.0\n",
"Episode 17950/30000, Total Reward: 163.0\n",
"Episode 18000/30000, Total Reward: 256.0\n",
"Episode 18050/30000, Total Reward: 148.0\n",
"Episode 18100/30000, Total Reward: 134.0\n",
"Episode 18150/30000, Total Reward: 190.0\n",
"Episode 18200/30000, Total Reward: 135.0\n",
"Episode 18250/30000, Total Reward: 141.0\n",
"Episode 18300/30000, Total Reward: 144.0\n",
"Episode 18350/30000, Total Reward: 117.0\n",
"Episode 18400/30000, Total Reward: 199.0\n",
"Episode 18450/30000, Total Reward: 133.0\n",
"Episode 18500/30000, Total Reward: 114.0\n",
"Episode 18550/30000, Total Reward: 178.0\n",
"Episode 18600/30000, Total Reward: 225.0\n",
"Episode 18650/30000, Total Reward: 213.0\n",
"Episode 18700/30000, Total Reward: 172.0\n",
"Episode 18750/30000, Total Reward: 142.0\n",
"Episode 18800/30000, Total Reward: 102.0\n",
"Episode 18850/30000, Total Reward: 113.0\n",
"Episode 18900/30000, Total Reward: 118.0\n",
"Episode 18950/30000, Total Reward: 147.0\n",
"Episode 19000/30000, Total Reward: 129.0\n",
"Episode 19050/30000, Total Reward: 180.0\n",
"Episode 19100/30000, Total Reward: 97.0\n",
"Episode 19150/30000, Total Reward: 135.0\n",
"Episode 19200/30000, Total Reward: 169.0\n",
"Episode 19250/30000, Total Reward: 124.0\n",
"Episode 19300/30000, Total Reward: 134.0\n",
"Episode 19350/30000, Total Reward: 102.0\n",
"Episode 19400/30000, Total Reward: 127.0\n",
"Episode 19450/30000, Total Reward: 250.0\n",
"Episode 19500/30000, Total Reward: 105.0\n",
"Episode 19550/30000, Total Reward: 130.0\n",
"Episode 19600/30000, Total Reward: 166.0\n",
"Episode 19650/30000, Total Reward: 86.0\n",
"Episode 19700/30000, Total Reward: 202.0\n",
"Episode 19750/30000, Total Reward: 171.0\n",
"Episode 19800/30000, Total Reward: 159.0\n",
"Episode 19850/30000, Total Reward: 247.0\n",
"Episode 19900/30000, Total Reward: 135.0\n",
"Episode 19950/30000, Total Reward: 108.0\n",
"Episode 20000/30000, Total Reward: 96.0\n",
"Episode 20050/30000, Total Reward: 24.0\n",
"Episode 20100/30000, Total Reward: 266.0\n",
"Episode 20150/30000, Total Reward: 138.0\n",
"Episode 20200/30000, Total Reward: 141.0\n",
"Episode 20250/30000, Total Reward: 119.0\n",
"Episode 20300/30000, Total Reward: 29.0\n",
"Episode 20350/30000, Total Reward: 189.0\n",
"Episode 20400/30000, Total Reward: 127.0\n",
"Episode 20450/30000, Total Reward: 140.0\n",
"Episode 20500/30000, Total Reward: 191.0\n",
"Episode 20550/30000, Total Reward: 135.0\n",
"Episode 20600/30000, Total Reward: 189.0\n",
"Episode 20650/30000, Total Reward: 129.0\n",
"Episode 20700/30000, Total Reward: 66.0\n",
"Episode 20750/30000, Total Reward: 218.0\n",
"Episode 20800/30000, Total Reward: 112.0\n",
"Episode 20850/30000, Total Reward: 142.0\n",
"Episode 20900/30000, Total Reward: 104.0\n",
"Episode 20950/30000, Total Reward: 134.0\n",
"Episode 21000/30000, Total Reward: 76.0\n",
"Episode 21050/30000, Total Reward: 130.0\n",
"Episode 21100/30000, Total Reward: 91.0\n",
"Episode 21150/30000, Total Reward: 134.0\n",
"Episode 21200/30000, Total Reward: 149.0\n",
"Episode 21250/30000, Total Reward: 24.0\n",
"Episode 21300/30000, Total Reward: 101.0\n",
"Episode 21350/30000, Total Reward: 198.0\n",
"Episode 21400/30000, Total Reward: 68.0\n",
"Episode 21450/30000, Total Reward: 86.0\n",
"Episode 21500/30000, Total Reward: 147.0\n",
"Episode 21550/30000, Total Reward: 83.0\n",
"Episode 21600/30000, Total Reward: 68.0\n",
"Episode 21650/30000, Total Reward: 84.0\n",
"Episode 21700/30000, Total Reward: 29.0\n",
"Episode 21750/30000, Total Reward: 96.0\n",
"Episode 21800/30000, Total Reward: 58.0\n",
"Episode 21850/30000, Total Reward: 133.0\n",
"Episode 21900/30000, Total Reward: 110.0\n",
"Episode 21950/30000, Total Reward: 117.0\n",
"Episode 22000/30000, Total Reward: 84.0\n",
"Episode 22050/30000, Total Reward: 135.0\n",
"Episode 22100/30000, Total Reward: 78.0\n",
"Episode 22150/30000, Total Reward: 133.0\n",
"Episode 22200/30000, Total Reward: 39.0\n",
"Episode 22250/30000, Total Reward: 88.0\n",
"Episode 22300/30000, Total Reward: 92.0\n",
"Episode 22350/30000, Total Reward: 94.0\n",
"Episode 22400/30000, Total Reward: 99.0\n",
"Episode 22450/30000, Total Reward: 89.0\n",
"Episode 22500/30000, Total Reward: 99.0\n",
"Episode 22550/30000, Total Reward: 132.0\n",
"Episode 22600/30000, Total Reward: 69.0\n",
"Episode 22650/30000, Total Reward: 96.0\n",
"Episode 22700/30000, Total Reward: 155.0\n",
"Episode 22750/30000, Total Reward: 133.0\n",
"Episode 22800/30000, Total Reward: 113.0\n",
"Episode 22850/30000, Total Reward: 124.0\n",
"Episode 22900/30000, Total Reward: 145.0\n",
"Episode 22950/30000, Total Reward: 103.0\n",
"Episode 23000/30000, Total Reward: 95.0\n",
"Episode 23050/30000, Total Reward: 84.0\n",
"Episode 23100/30000, Total Reward: 107.0\n",
"Episode 23150/30000, Total Reward: 170.0\n",
"Episode 23200/30000, Total Reward: 218.0\n",
"Episode 23250/30000, Total Reward: 162.0\n",
"Episode 23300/30000, Total Reward: 135.0\n",
"Episode 23350/30000, Total Reward: 52.0\n",
"Episode 23400/30000, Total Reward: 155.0\n",
"Episode 23450/30000, Total Reward: 188.0\n",
"Episode 23500/30000, Total Reward: 146.0\n",
"Episode 23550/30000, Total Reward: 93.0\n",
"Episode 23600/30000, Total Reward: 189.0\n",
"Episode 23650/30000, Total Reward: 97.0\n",
"Episode 23700/30000, Total Reward: 270.0\n",
"Episode 23750/30000, Total Reward: 162.0\n",
"Episode 23800/30000, Total Reward: 84.0\n",
"Episode 23850/30000, Total Reward: 51.0\n",
"Episode 23900/30000, Total Reward: 175.0\n",
"Episode 23950/30000, Total Reward: 133.0\n",
"Episode 24000/30000, Total Reward: 110.0\n",
"Episode 24050/30000, Total Reward: 129.0\n",
"Episode 24100/30000, Total Reward: 66.0\n",
"Episode 24150/30000, Total Reward: 104.0\n",
"Episode 24200/30000, Total Reward: 111.0\n",
"Episode 24250/30000, Total Reward: 72.0\n",
"Episode 24300/30000, Total Reward: 92.0\n",
"Episode 24350/30000, Total Reward: 112.0\n",
"Episode 24400/30000, Total Reward: 104.0\n",
"Episode 24450/30000, Total Reward: 130.0\n",
"Episode 24500/30000, Total Reward: 36.0\n",
"Episode 24550/30000, Total Reward: 164.0\n",
"Episode 24600/30000, Total Reward: 180.0\n",
"Episode 24650/30000, Total Reward: 137.0\n",
"Episode 24700/30000, Total Reward: 180.0\n",
"Episode 24750/30000, Total Reward: 108.0\n",
"Episode 24800/30000, Total Reward: 52.0\n",
"Episode 24850/30000, Total Reward: 86.0\n",
"Episode 24900/30000, Total Reward: 158.0\n",
"Episode 24950/30000, Total Reward: 114.0\n",
"Episode 25000/30000, Total Reward: 110.0\n",
"Episode 25050/30000, Total Reward: 87.0\n",
"Episode 25100/30000, Total Reward: 99.0\n",
"Episode 25150/30000, Total Reward: 116.0\n",
"Episode 25200/30000, Total Reward: 48.0\n",
"Episode 25250/30000, Total Reward: 96.0\n",
"Episode 25300/30000, Total Reward: 137.0\n",
"Episode 25350/30000, Total Reward: 133.0\n",
"Episode 25400/30000, Total Reward: 106.0\n",
"Episode 25450/30000, Total Reward: 87.0\n",
"Episode 25500/30000, Total Reward: 134.0\n",
"Episode 25550/30000, Total Reward: 107.0\n",
"Episode 25600/30000, Total Reward: 126.0\n",
"Episode 25650/30000, Total Reward: 102.0\n",
"Episode 25700/30000, Total Reward: 203.0\n",
"Episode 25750/30000, Total Reward: 250.0\n",
"Episode 25800/30000, Total Reward: 224.0\n",
"Episode 25850/30000, Total Reward: 107.0\n",
"Episode 25900/30000, Total Reward: 84.0\n",
"Episode 25950/30000, Total Reward: 124.0\n",
"Episode 26000/30000, Total Reward: 138.0\n",
"Episode 26050/30000, Total Reward: 188.0\n",
"Episode 26100/30000, Total Reward: 105.0\n",
"Episode 26150/30000, Total Reward: 145.0\n",
"Episode 26200/30000, Total Reward: 128.0\n",
"Episode 26250/30000, Total Reward: 135.0\n",
"Episode 26300/30000, Total Reward: 68.0\n",
"Episode 26350/30000, Total Reward: 177.0\n",
"Episode 26400/30000, Total Reward: 226.0\n",
"Episode 26450/30000, Total Reward: 126.0\n",
"Episode 26500/30000, Total Reward: 117.0\n",
"Episode 26550/30000, Total Reward: 107.0\n",
"Episode 26600/30000, Total Reward: 114.0\n",
"Episode 26650/30000, Total Reward: 98.0\n",
"Episode 26700/30000, Total Reward: 103.0\n",
"Episode 26750/30000, Total Reward: 143.0\n",
"Episode 26800/30000, Total Reward: 174.0\n",
"Episode 26850/30000, Total Reward: 149.0\n",
"Episode 26900/30000, Total Reward: 198.0\n",
"Episode 26950/30000, Total Reward: 155.0\n",
"Episode 27000/30000, Total Reward: 105.0\n",
"Episode 27050/30000, Total Reward: 157.0\n",
"Episode 27100/30000, Total Reward: 177.0\n",
"Episode 27150/30000, Total Reward: 189.0\n",
"Episode 27200/30000, Total Reward: 33.0\n",
"Episode 27250/30000, Total Reward: 127.0\n",
"Episode 27300/30000, Total Reward: 173.0\n",
"Episode 27350/30000, Total Reward: 251.0\n",
"Episode 27400/30000, Total Reward: 238.0\n",
"Episode 27450/30000, Total Reward: 250.0\n",
"Episode 27500/30000, Total Reward: 166.0\n",
"Episode 27550/30000, Total Reward: 221.0\n",
"Episode 27600/30000, Total Reward: 218.0\n",
"Episode 27650/30000, Total Reward: 133.0\n",
"Episode 27700/30000, Total Reward: 242.0\n",
"Episode 27750/30000, Total Reward: 166.0\n",
"Episode 27800/30000, Total Reward: 181.0\n",
"Episode 27850/30000, Total Reward: 160.0\n",
"Episode 27900/30000, Total Reward: 145.0\n",
"Episode 27950/30000, Total Reward: 171.0\n",
"Episode 28000/30000, Total Reward: 181.0\n",
"Episode 28050/30000, Total Reward: 34.0\n",
"Episode 28100/30000, Total Reward: 221.0\n",
"Episode 28150/30000, Total Reward: 175.0\n",
"Episode 28200/30000, Total Reward: 86.0\n",
"Episode 28250/30000, Total Reward: 97.0\n",
"Episode 28300/30000, Total Reward: 46.0\n",
"Episode 28350/30000, Total Reward: 109.0\n",
"Episode 28400/30000, Total Reward: 144.0\n",
"Episode 28450/30000, Total Reward: 182.0\n",
"Episode 28500/30000, Total Reward: 107.0\n",
"Episode 28550/30000, Total Reward: 158.0\n",
"Episode 28600/30000, Total Reward: 177.0\n",
"Episode 28650/30000, Total Reward: 165.0\n",
"Episode 28700/30000, Total Reward: 131.0\n",
"Episode 28750/30000, Total Reward: 245.0\n",
"Episode 28800/30000, Total Reward: 275.0\n",
"Episode 28850/30000, Total Reward: 138.0\n",
"Episode 28900/30000, Total Reward: 146.0\n",
"Episode 28950/30000, Total Reward: 170.0\n",
"Episode 29000/30000, Total Reward: 235.0\n",
"Episode 29050/30000, Total Reward: 500.0\n",
"Episode 29100/30000, Total Reward: 356.0\n",
"Episode 29150/30000, Total Reward: 179.0\n",
"Episode 29200/30000, Total Reward: 238.0\n",
"Episode 29250/30000, Total Reward: 133.0\n",
"Episode 29300/30000, Total Reward: 113.0\n",
"Episode 29350/30000, Total Reward: 163.0\n",
"Episode 29400/30000, Total Reward: 169.0\n",
"Episode 29450/30000, Total Reward: 355.0\n",
"Episode 29500/30000, Total Reward: 233.0\n",
"Episode 29550/30000, Total Reward: 132.0\n",
"Episode 29600/30000, Total Reward: 172.0\n",
"Episode 29650/30000, Total Reward: 154.0\n",
"Episode 29700/30000, Total Reward: 269.0\n",
"Episode 29750/30000, Total Reward: 211.0\n",
"Episode 29800/30000, Total Reward: 179.0\n",
"Episode 29850/30000, Total Reward: 121.0\n",
"Episode 29900/30000, Total Reward: 121.0\n",
"Episode 29950/30000, Total Reward: 157.0\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAHFCAYAAAAUpjivAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB3tElEQVR4nO3dd1wT9/8H8FfYiIiCTCfuAW6LewGOOmtbbbXWjl+/rVatrV1qq7bWUVtHp3ZYR23VWrVDWxUcKE5ErXtUwQmigOzN5/cHJRJIIIFL7pK8no8HD+VyuXvnuNy97zNVQggBIiIiIgtlI3cARERERMbEZIeIiIgsGpMdIiIismhMdoiIiMiiMdkhIiIii8Zkh4iIiCwakx0iIiKyaEx2iIiIyKIx2SEiIiKLxmSHZKFSqfT62bdvX4Xbmj9/Pn777bcqxzNnzhyD465Rowa6deuG9evXV2n/SjJnzhyoVCrJtpeRkYGFCxeiffv2qF69OlxcXNCuXTvMnz8fGRkZku2nqvbt26f3eVlVffr0QZ8+fSr1Xqn/PpXZt66f2NhYg7e5evXqSr+3Kho2bIjnnnvOpPsk+djJHQBZp8OHD2v8PnfuXOzduxd79uzRWN6qVasKtzV//nw88cQTGDFihJQh6vTEE09g2rRpEEIgJiYG8+fPx5gxYyCEwJgxY0wSg7m4e/cuQkJCcPXqVUyZMgWLFi0CAOzZswcfffQR1q9fj/DwcHh7e8scKdChQ4cy5+Vjjz2Gxo0b49NPP5V0X19//XWl3/t///d/GDhwoITRGG7Hjh1wc3Mrs9zX19fgbQ0ePBiHDx+u1HuJ9MVkh2TRpUsXjd89PT1hY2NTZrkSeXt7q+Ps2rUrunfvjoYNG+Kbb74xi2SnoKAA+fn5cHR0NPq+nn32WVy8eBF79+5Fjx491MtDQ0MxePBg9O3bF+PHj8eOHTuMHktJWVlZcHZ21lhWo0aNMuefo6MjatasWe55KYRAdnZ2me2VR58kXpe6deuibt26lX6/FDp27IjatWtLsi1PT094enpKsi0iXViNRYqVlJSEiRMnok6dOnBwcECjRo0wc+ZM5OTkqNdRqVTIyMjAmjVr1EXpxdUD9+7dw8SJE9GqVStUr14dXl5e6NevHw4cOCBpnA0aNICnpyfu3r2rsTw1NRVvvvkm/P394eDggDp16mDq1KkaVTdPPvkkWrdurfG+oUOHQqVSYdOmTeplJ06cgEqlwp9//mnQZ4uNjYVKpcKiRYvw0Ucfwd/fH46Ojti7dy8AYPv27WjXrh0cHR3h7++vswRj06ZNCAoKgpubG6pVq4ZGjRrhhRdeKPe4HD9+HLt27cKLL76okegU69GjB1544QXs3LkT0dHRAID27dujZ8+eZdYtKChAnTp1MHLkSPWy3NxcfPTRR2jRogUcHR3h6emJ559/Hvfu3dN4b8OGDTFkyBBs2bIF7du3h5OTEz744INyYy+PSqXCpEmTsGLFCrRs2RKOjo5Ys2YNAOCDDz5AUFAQ3N3dUaNGDXTo0AErV65E6fmWS1djFf+dPv30UyxZsgT+/v6oXr06unbtiiNHjmi8V1s1VvFn3LFjBzp06ABnZ2e0aNECP/zwQ5n4IyMj0bVrVzg5OaFOnTp4//338f3330talVTyvJs3bx7q168PJycndOrUCbt379ZYV1s11smTJzFkyBB4eXnB0dERfn5+GDx4MG7duqVeJzs7G9OnT9f4fr366qt48OCBxvbz8vLw9ttvw8fHB9WqVUOPHj1w7NgxrXHHx8fj5ZdfRt26deHg4AB/f3988MEHyM/P11hv+fLlaNu2LapXrw5XV1e0aNECM2bMqNpBI6NiyQ4pUnZ2Nvr27YurV6/igw8+QJs2bXDgwAEsWLAAp06dwvbt2wEUVYf169cPffv2xfvvvw+g6AkdKEqWAGD27Nnw8fFBeno6tm7dij59+mD37t2VbjNRWkpKCpKSkjSe/jMzM9G7d2/cunULM2bMQJs2bXDu3DnMmjULZ86cQXh4OFQqFUJCQvDrr78iLi4Ovr6+yM/PR0REBJydnREWFoYnn3wSABAeHg47Ozt1zIZ+ts8//xzNmjXDp59+iho1aqBp06bYvXs3hg8fjq5du2LDhg0oKCjAokWLyiRthw8fxujRozF69GjMmTMHTk5OuH79epkqx9LCwsIAoNzqxREjRuDbb79FWFgYOnbsiOeffx6vvfYarly5gqZNm6rX27VrF+7cuYPnn38eAFBYWIjhw4fjwIEDePvtt9GtWzdcv34ds2fPRp8+fXD8+HGNkpYTJ07gwoULeO+99+Dv7w8XF5dyY6/Ib7/9hgMHDmDWrFnw8fGBl5cXgKKb/Msvv4z69esDAI4cOYLJkyfj9u3bmDVrVoXb/eqrr9CiRQssW7YMAPD+++/j0UcfRUxMjNZqo5L++ecfTJs2De+++y68vb3x/fff48UXX0STJk3Qq1cvAMDp06cRGhqKZs2aYc2aNahWrRpWrFiBdevWGfT5i0sHS1KpVLC1tdVY9uWXX6JBgwZYtmwZCgsLsWjRIgwaNAgRERHo2rWr1m1nZGQgNDQU/v7++Oqrr+Dt7Y34+Hjs3bsXaWlpAIpK00aMGIHdu3dj+vTp6NmzJ06fPo3Zs2fj8OHDOHz4sLrk8qWXXsLatWvx5ptvIjQ0FGfPnsXIkSPV2yoWHx+PRx55BDY2Npg1axYaN26Mw4cP46OPPkJsbCxWrVoFANiwYQMmTpyIyZMn49NPP4WNjQ3+/fdfnD9/3qBjSCYmiBRg/PjxwsXFRf37ihUrBADxyy+/aKz38ccfCwBi165d6mUuLi5i/PjxFe4jPz9f5OXlieDgYPHYY49pvAZAzJ49u8JtABATJ04UeXl5Ijc3V1y+fFkMGzZMuLq6iuPHj6vXW7BggbCxsRFRUVEa7//1118FAPHXX38JIYT4999/BQCxdu1aIYQQkZGRAoB4++23hb+/v/p9oaGholu3bgZ/tpiYGAFANG7cWOTm5mq8JygoSPj5+YmsrCz1stTUVOHu7i5KXho+/fRTAUA8ePCgwuNT0iuvvCIAiIsXL+pc58KFCwKAmDBhghBCiPv37wsHBwcxY8YMjfVGjRolvL29RV5enhBCiPXr1wsAYvPmzRrrRUVFCQDi66+/Vi9r0KCBsLW1FZcuXTIo/uL3Dh48WGMZAOHm5iaSkpLKfW9BQYHIy8sTH374ofDw8BCFhYXq13r37i169+6t/r347xQYGCjy8/PVy48dOyYAiPXr16uXzZ49W5S+dDdo0EA4OTmJ69evq5dlZWUJd3d38fLLL6uXPfnkk8LFxUXcu3dPI85WrVoJACImJqbcz1S8b20/jRs3LvN5dJ1fISEh6mWrVq3S2Pfx48cFAPHbb7/pjGPHjh0CgFi0aJHG8o0bNwoA4ttvvxVCPDy/Xn/9dY31fvrpJwFA47rx8ssvi+rVq2scQyEenv/nzp0TQggxadIkUbNmzXKPEykPq7FIkfbs2QMXFxc88cQTGsuLe0+ULgrXZcWKFejQoQOcnJxgZ2cHe3t77N69GxcuXKh0bF9//TXs7e3h4OCAZs2a4e+//8b69evRsWNH9Trbtm1DQEAA2rVrh/z8fPXPgAEDNHqZNW7cGA0bNkR4eDiAotKQwMBAPPPMM4iJicHVq1eRk5ODyMhIhISEVPqzDRs2DPb29urfMzIyEBUVhZEjR8LJyUm93NXVFUOHDtV4b+fOnQEAo0aNwi+//ILbt29X+tiVJv6r3imulvHw8MDQoUOxZs0aFBYWAgCSk5Px+++/49lnn4WdXVFh9LZt21CzZk0MHTpU4/i2a9cOPj4+ZXrxtWnTBs2aNZMs7n79+qFWrVpllu/ZswchISFwc3ODra0t7O3tMWvWLCQmJiIhIaHC7Q4ePFijdKRNmzYAgOvXr1f43nbt2qlLlADAyckJzZo103hvREQE+vXrp9HexsbGBqNGjapw+yWFh4cjKipK40dbj0hd59f+/ftRUFCgddtNmjRBrVq18M4772DFihVaS0yKSxVL96Z68skn4eLior4+FFfXjh07VmO9UaNGqc+lYtu2bUPfvn3h5+encU4NGjQIQNGxA4BHHnkEDx48wNNPP43ff/8d9+/f13WYSEGY7JAiJSYmwsfHp0zbBC8vL9jZ2SExMbHCbSxZsgQTJkxAUFAQNm/ejCNHjiAqKgoDBw5EVlZWpWMbNWoUoqKicOjQIXzzzTdwdXXFU089hStXrqjXuXv3Lk6fPg17e3uNH1dXVwghNC6QwcHB6otzeHg4QkNDERgYCG9vb4SHh+PgwYPIysrSSHYM/Wyle7okJyejsLAQPj4+ZdYtvaxXr1747bffkJ+fj2effRZ169ZFQEBAhd3ti2+8MTExOtcpbqdRr1499bIXXngBt2/fVleDrV+/Hjk5ORo3trt37+LBgwdwcHAoc4zj4+PL3ICk7umjbXvHjh1D//79AQDfffcdDh48iKioKMycORMA9DrnPDw8NH4vroqpzHuL31/yvYmJiVp7vhnaG65t27bo1KmTxk9AQECZ9XSdX7m5uUhPT9e6bTc3N0RERKBdu3aYMWMGWrduDT8/P8yePRt5eXnqz2FnZ1emYbNKpYKPj4/6+lD8b+k47Ozsyhyvu3fv4s8//yxzPhW3qSs+p8aNG4cffvgB169fx+OPPw4vLy8EBQWpz1dSJrbZIUXy8PDA0aNHIYTQSHgSEhKQn5+vV0+QdevWoU+fPli+fLnG8tJ19Yby9PREp06dABT1xmrZsiV69+6N119/Hdu2bQMA1K5dG87OzlobiBa/Xiw4OBgrV67EsWPHcPToUbz33nsAikoPwsLCcP36dVSvXl2jTZChn6100lirVi2oVCrEx8eXWVfbsuHDh2P48OHIycnBkSNHsGDBAowZMwYNGzbU2fYiNDQUM2bMwG+//aazq3RxaUBoaKh62YABA+Dn54dVq1ZhwIABWLVqFYKCgjR6MNWuXRseHh46e3G5urqW+/mrStv2NmzYAHt7e2zbtk2jNKOqY0BJycPDo0ybLED731wKus4vBwcHVK9eXef7AgMDsWHDBgghcPr0aaxevRoffvghnJ2d8e6778LDwwP5+fm4d++eRsIjhEB8fLy6NLI4oYmPj0edOnXU6+Xn55d5YKpduzbatGmDefPmaY3Jz89P/f/nn38ezz//PDIyMrB//37Mnj0bQ4YMweXLl9GgQQM9jgyZGkt2SJGCg4ORnp5e5kaxdu1a9evFSj+9FlOpVGW6V58+fbrMWCpV1bNnTzz77LPYvn27ettDhgzB1atX4eHhUeYJuFOnTmjYsKH6/cHBwVCpVHj//fdhY2OjbkwaEhKCvXv3IiwsDL169dKohqrqZ3NxccEjjzyCLVu2IDs7W708LS1N3eNLG0dHR/Tu3Rsff/wxgKJeM7p06tQJ/fv3x8qVK3Hw4MEyr0dGRuKHH37AwIEDNaoAbW1tMW7cOHUj4OPHj5fp+TVkyBAkJiaioKBA6/Ft3ry5XsdBSiqVCnZ2dhrVUFlZWfjxxx9NHosuvXv3xp49ezRKvgoLCzV6/klJ1/nVs2fPMo2ZtVGpVGjbti2WLl2KmjVr4sSJEwAefv9LN6zevHkzMjIy1K8XN9T/6aefNNb75ZdfyjSwHjJkCM6ePYvGjRtrPadKJjvFXFxcMGjQIMycORO5ubk4d+5chZ+J5MGSHVKkZ599Fl999RXGjx+P2NhYBAYGIjIyEvPnz8ejjz6qUaUTGBiIffv24c8//4Svry9cXV3RvHlzDBkyBHPnzsXs2bPRu3dvXLp0CR9++CH8/f3LXOiqau7cudi4cSPef/99hIeHY+rUqdi8eTN69eqF119/HW3atEFhYSFu3LiBXbt2Ydq0aQgKCgJQVDUXEBCAXbt2oW/fvqhWrRqAomQnKSkJSUlJWLJkicb+pPhsc+fOxcCBAxEaGopp06ahoKAAH3/8MVxcXNS9vQBg1qxZuHXrFoKDg1G3bl08ePAAn332Gezt7dG7d+9y97F27VqEhISgf//+mDJlivomtGfPHnz22Wdo0aIFVq9eXeZ9L7zwAj7++GOMGTMGzs7OGD16tMbrTz31FH766Sc8+uijeO211/DII4/A3t4et27dwt69ezF8+HA89thjeh0HqQwePBhLlizBmDFj8L///Q+JiYn49NNPTTKekb5mzpyJP//8E8HBwZg5cyacnZ2xYsUK9XAINjb6Pf9GR0dr7R3WqlUrdW9IoChxDQ0NxRtvvIHCwkJ8/PHHSE1NLbfr/7Zt2/D1119jxIgRaNSoEYQQ2LJlCx48eKAuAQwNDcWAAQPwzjvvIDU1Fd27d1f3xmrfvj3GjRsHAGjZsiWeeeYZLFu2DPb29ggJCcHZs2fVvRJL+vDDDxEWFoZu3bphypQpaN68ObKzsxEbG4u//voLK1asQN26dfHSSy/B2dkZ3bt3h6+vL+Lj47FgwQK4ubmpS5RIgeRsHU1UrHRvLCGESExMFK+88orw9fUVdnZ2okGDBmL69OkiOztbY71Tp06J7t27i2rVqgkA6l4uOTk54s033xR16tQRTk5OokOHDuK3334T48ePFw0aNNDYBgzojfXqq69qfe2tt94SAERERIQQQoj09HTx3nvviebNmwsHBwfh5uYmAgMDxeuvvy7i4+M13vv6668LAGLevHkay5s2bSoAiNOnT2ss1/ezFfeK+eSTT7TG/Mcff4g2bdoIBwcHUb9+fbFw4cIyvX22bdsmBg0aJOrUqSMcHByEl5eXePTRR8WBAwcqPF7Fx2H+/PmiXbt2olq1aqJatWqiTZs24qOPPhLp6ek639etWzcBQIwdO1br63l5eeLTTz8Vbdu2FU5OTqJ69eqiRYsW4uWXXxZXrlxRr6etR5W+dPXG0nUO/PDDD6J58+bC0dFRNGrUSCxYsECsXLmyTE8nXb2xtP2dSp+bunpjafuMpfcjhBAHDhwQQUFBwtHRUfj4+Ii33npL3cuxoh535fXGAiDCwsI0Ps/HH38sPvjgA1G3bl3h4OAg2rdvL3bu3KmxzdK9sS5evCiefvpp0bhxY+Hs7Czc3NzEI488IlavXq3xvqysLPHOO++IBg0aCHt7e+Hr6ysmTJggkpOTNdbLyckR06ZNE15eXsLJyUl06dJFHD58WDRo0KBML8579+6JKVOmCH9/f2Fvby/c3d1Fx44dxcyZM9Xn6po1a0Tfvn2Ft7e3cHBwEH5+fmLUqFFlvqOkLCohSo12RUREVqV///6IjY3F5cuXJdlebGws/P398cknn+DNN9+UZJtEVcFqLCIiK/LGG2+gffv2qFevHpKSkvDTTz8hLCwMK1eulDs0IqNhskNEZEUKCgowa9YsxMfHQ6VSoVWrVvjxxx/xzDPPyB0akdGwGouIiIgsGrueExERkUVjskNEREQWjckOERERWTQ2UEbRCKJ37tyBq6ur5MPKExERkXEIIZCWlgY/P79yB8VksgPgzp07GhMREhERkfm4efMm6tatq/N1Jjt4OGngzZs3ywwhTkRERMqUmpqKevXqlZn8tzQmO3g4g3GNGjWY7BAREZmZipqgsIEyERERWTQmO0RERGTRmOwQERGRRWOyQ0RERBaNyQ4RERFZNCY7REREZNGY7BAREZFFY7JDREREFo3JDhEREVk0JjtERERk0WRNdubMmQOVSqXx4+Pjo35dCIE5c+bAz88Pzs7O6NOnD86dO6exjZycHEyePBm1a9eGi4sLhg0bhlu3bpn6oxAREZFCyV6y07p1a8TFxal/zpw5o35t0aJFWLJkCb788ktERUXBx8cHoaGhSEtLU68zdepUbN26FRs2bEBkZCTS09MxZMgQFBQUyPFxiIiISGFkT3bs7Ozg4+Oj/vH09ARQVKqzbNkyzJw5EyNHjkRAQADWrFmDzMxM/PzzzwCAlJQUrFy5EosXL0ZISAjat2+PdevW4cyZMwgPD5fzY1mdrNwCCCEqXE8Igey8qiWiWbkP359fUIjc/EKdrxtKCKHx/uy8giptT5v8gkLcTMrU63iVVFEsxX8DKY6xsRUWKj9GU5LyWGTnFaCwsOjcKn2+FJ9DpZcnZ+QatI+8gkLkFRRWvKICZOUWIDuvQPHfi+y8AuTmFyK/oBCZufk6Y72bmo2CQs1rx4PMXI2/qRBCvaz0dSY1Ow/5Ov52FV1fCgoFcvIL1Nsv3kfxMS7+HEkZuRrxa7tOm5rss55fuXIFfn5+cHR0RFBQEObPn49GjRohJiYG8fHx6N+/v3pdR0dH9O7dG4cOHcLLL7+M6Oho5OXlaazj5+eHgIAAHDp0CAMGDNC6z5ycHOTk5Kh/T01NNd4HtAIX41MxcNkBPNmxLj55sm256/7vx2iEnb+LyHf6om6tagbv68SNZIz8+hCe794Qs4a0Qp9P9yElKw/R74XCwc4Gf5+Jw4SfTmDmoy3xUq9GBm9/+pYz2BB1E39N6Ykaznbo8fFeAMD6l7qga2MPg7dXmhACTWb+rf49duFgvd63LPwyloVfKfr/6HYY0b6OxusX4lIx6LMDGN2pHpIyc6t0jE1h+FcHcSEuFSdnhcLVyV7ucGS15lAsZv9xDp891Q7D29Wp+A3lSM7IRfu5YXjE3x3D2/lh5tazWPxkWzzesS7upmYjaP5u9bpzRwRgXJcGePaHY9h/+R4aebpgz7Q+Fe6jsFCgx8d7kFcgcGxGMOxsZX9m1mnW72ex9vB1AEALH1dcjE/D8fdCULu6o8yRafr+wDV8tP1CmeUn3g+Fu4uD+vdBnx3Ahbii+1XxteOtTf9gU3RR043iv7X/9L/U7xna1g9fPN0eAPDT0euYufWsxvuLnb71AMO+PIhxXRpg7ogAjdcW/HUB3+y/BgBwsLPRmbhsmdgNI78+pP792MxgeFZ3RK9Fe5GZV4DjM0NkO19kPUuDgoKwdu1a7Ny5E9999x3i4+PRrVs3JCYmIj4+HgDg7e2t8R5vb2/1a/Hx8XBwcECtWrV0rqPNggUL4Obmpv6pV6+exJ/MuizfdxUA1F+48oSdvwsA+OV45dpVfbrzEgBg1cFYAMCt5CykZefjRlIGAOD1X04BAOb9VfbCoY8NUTcBAMsjrmLjf/8v2t75Sm2vtJxKPt0UJzoAMHXjqTKvf/3f32Dj8ZtVPsamcOZ2CvILBY5eS5I7FNnN/qOoHeJrG05VeVthF4r+9sdiktQ3tWmb/gEAbDlxW2Pd938ren3/5XsAgGv3MvTaR3puPu6m5iApIxf30w0rETK14kQHAC7GFzV/+Pus7nuDXLQlOgAQdl4z1uJEp6SS193iv3VJf/5zR/3/4nNCm6VhlwEAPx65Xua14kQHQLklNE99e0Tj923/xCEnvxB3UrLxIDMPcSnZOt9rbLImO4MGDcLjjz+OwMBAhISEYPv27QCANWvWqNdRqVQa7xFClFlWWkXrTJ8+HSkpKeqfmzdv6lyXiIiIzJuiyh9dXFwQGBiIK1euqHtllS6hSUhIUJf2+Pj4IDc3F8nJyTrX0cbR0RE1atTQ+CEiIiLLpKhkJycnBxcuXICvry/8/f3h4+ODsLAw9eu5ubmIiIhAt27dAAAdO3aEvb29xjpxcXE4e/aseh0yvdTsPERfTza4AS4REZExyNpA+c0338TQoUNRv359JCQk4KOPPkJqairGjx8PlUqFqVOnYv78+WjatCmaNm2K+fPno1q1ahgzZgwAwM3NDS+++CKmTZsGDw8PuLu7480331RXi5E8hnweiRtJmfji6fYY2tZP7nCIiMgEym9gIi9Zk51bt27h6aefxv379+Hp6YkuXbrgyJEjaNCgAQDg7bffRlZWFiZOnIjk5GQEBQVh165dcHV1VW9j6dKlsLOzw6hRo5CVlYXg4GCsXr0atra2cn0sq3cjKRMAsP10HJMdIiKSnazJzoYNG8p9XaVSYc6cOZgzZ47OdZycnPDFF1/giy++kDg6IiIisgSKarNDJKWUzDwkGThYmi5sfkREVL4KOkrLSvZBBYmMpe2HuwAAF+cOhJM9qzXJugkwYyfdKhrSRa9tlGq1o6TkhyU7ZPHupso3kBURmda9tBz2BKUymOyQUeXmF2L9sRu4kZgpdyhEZOH+PhOHzvPC8c7m03KHQgrDZIeM6rsD1zB9yxn0+mSv3KEYTEElsESKpLSqsSX/TXmg5KlSDFG6WkjplFRtVRqTHTKqI9cSTbKf4lJrll4TGZeC72dkAGur6mOyQ2Smjl5LxIR10YiXcXI9IiJdlJRPsTcWGY3SirirQonFs6P/m2E4I7cAbs72MkdDZCaUdAe2MKUvk7kFumdINzWW7JAsFJg7lEvJ18fbyWz8TUTKs/Dvi3KHoMZkh8yKEktYiIjMnaVfWpnsUJUp/Uui5FIZa8fklchySDEwobEw2SGLIACcuZWCnHzl1BETESnV/XRpptIxF0x2yCL8+c8dDP0yUu4wiIjMQud54XKHYFJMdsgibDlxW+4QiMgcKLiqRU6mqO2Xs0kBkx0iGW2OvoXw83flDoOIyKJxnB2ShVm0GS7xBGisJ5Jpm/4BAMQuHFyl7fBZleSgtMb/Cgun6szsi63kUZlZskNERGTlzCyvMhiTHTIaBSf5FoWHmUxJyd2LiXRhskMWQcnFp0REJC8mO2TxmAaRtSivzIXPA2TNmOyQUfECa3ysVKBi/LoRacdkh6qMF1giIlIyJjskC3MrjWCbTCKyNDeTMhF9PUnuMEyC4+wQ6YHVcUQWgl9mtZ6L9gIAdr3eS5LtKbmnHkt2iIiIrNjZ2ykWX3rNZIeIiMhCxKVkyR2CIjHZIdKh9INOXkEhzt1J4Zg+ErL0p0ml47msbJX5ery09rjkcehLyV9nJjtUZbpOcEu7jL6+8RQGfx6J5RFX5Q5FL0q+8JAybIy6WaX3K+07zuQNOHs7Ve4QFInJDlk8qS6A207HAQBW7FNWsqPr0/GyTxVZfShW7hAU641fTmHCumirSaBO3UyROwSjYm8sIiLSmzWUGGbnFWDLidsAgNsPslC3VjWZIzK+++k5codgVCzZIaMSFlK+YBmfgogMZSUFOxaPyQ6ZrZ3n4o227coUXSekZWPdkevIyMk3QkS6WcOTNhFRVbAai8zWK+tOyB2ChlErDiM2MRMnbzzA4lFt5Q6HiLSxkC6AQghFD+KnNCzZIZJIbGImAGD3xbsyR2I+WEVAVDmz/zgndwgGk7NZA5Mdsni5BYVyh0BEJKm1h69Lti3JHjq0FDQppfCJyQ4ZlRKe3Dccq9pYIkREZN6Y7JAsTJntJ2fmVup9JWNUyMMJERFVApMdMhollOoAyonDmJRSVExE0jG7BsgKvtYy2SFZmFsCouRwlRwbWTazHF1Yj5jN8WMpQZqJh90wBJMdIiKyCJaco5y8kSx3CGaNyQ4REZHCVbbtIRVhskNkIEt+eiTrUpk2IebWjIQqZg1/UyY7ZFZU7BdlUazhIktkzZTS/onJDlWZEnoMKOT7BMD0X275jz5VRkpWHh5ffkjuMIisApMdIguglKcn0t93+68h+jobnUqJiX/lWMP1g8kOkR7MsostKVpmboFB6z/IzK3wPOTNnkg7JjtkRMpIEJQRBVHl7b5wF+0+DMOs381v8kciJWCyQ6QDG0OTUizacQkA8OMR6SZ/JN3knJ27JF6BpMNkh4jIQijjFl3k/J1UHL6aKHcYFoMPX1VjJ3cARFKw5suAkm5wZF3Ka0L06OcHAACH3u0Hv5rOJopIegrobEoSYMkOVZmuRpP7r9w3WQzG7v6uhO71RFK7m5pt9H3cSs4y+j4s1Zw/2EZLKkx2yGhy8wt1PvkxdyCSX1KGZU1BYIxSTjk7Yip5Yk19fbX3X7lDAMBkh4iIrIkeT1psHyOdL/Yw2SEyG4aMs8MxeYgUjN9P2ch56JnskMlcik+TOwRp8FpJZohJOOlyPi7VJPs58K/p2nGWxmSHTGbaplOy7LcyF3lzapBsPpESkRKtjIwxyX4STNAgXhd2PSejKjk4V05eofH2Y8VPrdb7yYmsiAxPNZ/uvISjMZYxVpJiSnYWLFgAlUqFqVOnqpcJITBnzhz4+fnB2dkZffr0wblzml3xcnJyMHnyZNSuXRsuLi4YNmwYbt26ZeLoieRz7V4GMiyg1waRUihlBGUpbTt9x+D3fLn3X0TFWsZktYpIdqKiovDtt9+iTZs2GssXLVqEJUuW4Msvv0RUVBR8fHwQGhqKtLSHbT+mTp2KrVu3YsOGDYiMjER6ejqGDBmCggLDJtkjKs2MarKw+2KC3CFUCnu9kLW4l5Yj6/6/O2Caqiqlkj3ZSU9Px9ixY/Hdd9+hVq1a6uVCCCxbtgwzZ87EyJEjERAQgDVr1iAzMxM///wzACAlJQUrV67E4sWLERISgvbt22PdunU4c+YMwsPD5fpIViPmfgY2HLuBAst7CCIiAiDNQ8+aQ7HoPC8cS3ZdqvrGzJicjzayJzuvvvoqBg8ejJCQEI3lMTExiI+PR//+/dXLHB0d0bt3bxw6dAgAEB0djby8PI11/Pz8EBAQoF5Hm5ycHKSmpmr8kOH6froP7245gz//Mbx4VCnSWf1DREY2+7+RkD9XyJgz1kjWZGfDhg04ceIEFixYUOa1+Ph4AIC3t7fGcm9vb/Vr8fHxcHBw0CgRKr2ONgsWLICbm5v6p169elX9KGQgJbQnXnMoFgGzd+Kno5Y5kzQriJTN1NWk5tTDkEhqsiU7N2/exGuvvYZ169bByclJ53qlv6BCiAq/tBWtM336dKSkpKh/bt68aVjwpDiVyZ2Kn7Zmbj2r9fXY+xlViEg6ufnG68VG1sOaeyySYR5kWtY0IoCMyU50dDQSEhLQsWNH2NnZwc7ODhEREfj8889hZ2enLtEpXUKTkJCgfs3Hxwe5ublITk7WuY42jo6OqFGjhsYPUWlbTt6u1PukvKU8yMxFwJydssdBZBZKnPQxCnlYMUef7LS8tkWyJTvBwcE4c+YMTp06pf7p1KkTxo4di1OnTqFRo0bw8fFBWFiY+j25ubmIiIhAt27dAAAdO3aEvb29xjpxcXE4e/aseh0ic77pbz8Tx5Idokr4JuKq3CHoRd8SN211FcYqrbubKm/PMWOQbVBBV1dXBAQEaCxzcXGBh4eHevnUqVMxf/58NG3aFE2bNsX8+fNRrVo1jBkzBgDg5uaGF198EdOmTYOHhwfc3d3x5ptvIjAwsEyDZ1IWNh8gMk8cLkBaB65UfgqFvZeMM+REfGqWUbYrJ0WPoPz2228jKysLEydORHJyMoKCgrBr1y64urqq11m6dCns7OwwatQoZGVlITg4GKtXr4atra2MkVMxa2wmwFsBkXkz5XUruQrtY5Iy8iSM5KGzt43TQ1nO24Gikp19+/Zp/K5SqTBnzhzMmTNH53ucnJzwxRdf4IsvvjBucFRlUpTmROqYSM6UCYYV5m9EVElsGK4Mso+zQ0REZCqGph5V7bJ/K9nyqoTMEZMdsgh8diIifcz6/Rx+PnpD7jDIxJjsEGnBomeyVuk5+fhq778W3XV7xtYzcodAJsZkh4jIQpRX4aJv/j5v+wV8svMSghfvkyIkIkVgskNERGrHY5MAAIV6JEcsADUdTvdRNUx2yPLxgkxEZNWY7BBJIK/g4SjH+jwRE1kCayhsYPs96ch5KJnsEEngXtrD4dUzc/NljITIejEtIV2Y7BDpgQ93ZA54mj7E7yyVxGSHqIryC6SfqPPotUS8tuEk7qdVfih5ImOwhqorY/rt5G28u/m0Ua4bpJuiposgUipdF/jVB2Mwd/sFfPpkG0n3N/rbI5Juj5TH1DkDk5TKqUwvqHN3UuBga4Om3q5lXpu68RQAoHNDdzzesW5VwyM9Mdkhkyk5W7LUMycXSNwqeNvpOL3Wm/PneQDAtF/+kXT/RGSeUrLyMPjzSADAtfmP6lyvKhOAmishY0Urq7HIIiSUaCBMRGWxDYtp3E9/eC2S8pCzYK5qmOyQUSni+irxVYJdUcmSSX12W/v3hdWHysBkh4xLCdc5JcRARFaneDRqkh+THSIiqhQrL7Sp0P1062uXo1RMdshkHmQ9/OLL2VDN2Cz3k5lWZm4+7qZmyx2G0fA8kZ451RgxUTQtJjtkMndT2YiY9PfIvN0Imr8btx9kyR2KRYi+kSx3CCZVlbYyJfOQB5bSa0oB2ZXUvXANwWSHiBQpPado2o3DVxNljsR0CgoFEtON81Cw79I9o2zX0r216bTcIZAEmOwQSaAyA48RlfbM90fR8aNwnLmVIncoZskYZRfn41KNsFUyNSY7RAZiYkPGcvhaUSnW+qgbMkdCUpHqesHLTtUw2SEiIoujgCYqACoeZ6gqcd5IzKj8m60Mkx0ikg+fVslKGdojVVtS9PmefyWKpqw8C5uolMkOWTxL7uZORGQMXebvlnybnBuLiIjU5CzwsvbpHaRkzg9aiRkW0uX+P0x2iCRQ8ubEewURkbIw2SFZyDm4VGXkFz7MYPjkS8ZmST1vzLH3YlW+49dLNRo2x89viezkDoDIEijxeqbAkCrFUj5HaZbwucy5msZYzt3Wb1weffOpwkKB07dTLKLBsJzPiUx2iPRQ0ZNeUUkVL/wkDTlLPlkSIZ3yLhv6Xi2+3vcvPt11Gc72tpLEZK1YjUVGxSc/ItMxtzQlOSMXA5ftxzcRV+UORYOSEr7vI2MAAFl5BTJHYt6Y7BCZwO+nbmPH2Ti5wyArVTzPmNJ8ve9fXIxPw4K/L8odClk4VmORxZO7MXRieg5e23AKAHD2gwEm2y/L1MyX1AULC/66IO0GJZKbL207FHYeIF1YskMWT+6qtJJP1YW8GJMepE7Po68nS7xFIvPCZIcsHvMLMndyJ+yk3caoGxi38qjOakKBihNX/m1Ng8kOERGpWUpVkCmSiHc2n8GBK/fx3f5rRt8XVQ2THSKJKagjB1mI2MRMvdazjDRFfoYmfCVLdvT9/ltITmk2mOwQkaJZY/JYuutzyYa8uy/cNXU4ZAAmMbrJeWiY7BBJjBc7MqYX1xxHxOV7codBZqSgkBclJjtEeqhwkLEqlj5YSjsJQ1lhoY0klNK7qqqnrTEH75N7yAmpSPEp+ny6F3kF1nmNKcZxdogMZN2XDCIyhBKqYW8mZQHIkjsMWbFkh4xK15OfJUxqR2QsFd0gr91LN00gpBcF5DNUASY7JIsv9/5rsn1ZaQ0RWbBtp+OQlp0ndxgmd/Df+1i04yLyLeBhiePrmBarsYj0EPeg/CJgPtmRlPRpb5KQlgNXJ3sTRKMcY78/CgDwq+mMZ7o0kDmaqrmakKHYOcssEUt2iPSQkcsZh0laSmjLYa5uJmsfd0gJZSUl/65CCJ0ly5tP3ELPj/eYJihiskPGpYSLD5G5YSJkmNO3HmDA0v24rufgi8aUkJaj97rJmXmKiNkaMNkhIkXjjV+7HyJjjNLQ3xwfUMb/cAyX7qZJtj1Dj0HJc3Tt4esGvXdZ+BUD90aVwWSHyMjYQJqM4aejN7D6YKzJ92vo+WyKMaRSs+Vt+1LeeEFVTdaNORaRNWGyQ4pTUCiwMeqGRXav5WVLE/NA7fQ9T07fTjFqHOasot5OqVbYm82aMdkhxdkQdQPvbD6Dfosj5A6FSFKFhQIZevTAkethPjE9B9fuZcizcxOb9ss/codAJsRkhxRH6qHwTTGeBUuaSR+jvjmM1rN34m5qtsbknrrsvnAXl+J1t0U59O99KcPDxzsuSro9Y5FiKoiw85xQ1dTkrNLnODtEErCUeXjIuI7/l8j/fSYOMRX0wvnn5gO8uOZ4ueskZuQiPiUbPm5OksSXmJ5r0PqshiRzoVey0759e70bSZ04caJKARGZI5bskJRUKhUuxKXqtW5cSpZkyQ5VDr/+yqdXsjNixAj1/7Ozs/H111+jVatW6Nq1KwDgyJEjOHfuHCZOnGiUIImIqPKssUfgL8dvVbjO9C2nEdLSG8EtvTWWS5m8sNRXGfRKdmbPnq3+///93/9hypQpmDt3bpl1bt68KW10RAqUmVuAjJx8uDiyFpgqLydP96jcvD2axvpjN7H+2E3ELhwsdyhkZAY3UN60aROeffbZMsufeeYZbN68WZKgiJTuhdVRcodgNSz1ybiwguIWayiMkar6V+qxfKzh2Fsbg5MdZ2dnREZGllkeGRkJJyfWG5N1OBqTJHcIFsGc0phJP5/AY18fREGhCW6FlTww5nQ8q8Ka2shZ0Uc1KoPL4adOnYoJEyYgOjoaXbp0AVDUZueHH37ArFmzJA+QyBoIYV0XcHO07XQcgKJ5mNrXryVzNA+xFEIB+N1VPINLdt59912sXbsWJ0+exJQpUzBlyhScPHkSq1evxrvvvmvQtpYvX442bdqgRo0aqFGjBrp27Yq///5b/boQAnPmzIGfnx+cnZ3Rp08fnDt3TmMbOTk5mDx5MmrXrg0XFxcMGzYMt25V3DCNiKgypEourLHRsLXh31iTnA90BiU7+fn5+OCDD9C1a1ccPHgQSUlJSEpKwsGDBzFq1CiDd163bl0sXLgQx48fx/Hjx9GvXz8MHz5cndAsWrQIS5YswZdffomoqCj4+PggNDQUaWkPB9maOnUqtm7dig0bNiAyMhLp6ekYMmQICgp0N/4j62KKCw4f7MhQ5Z2WKqh4ozQjltquzJIYlOzY2dnhk08+kSyRGDp0KB599FE0a9YMzZo1w7x581C9enUcOXIEQggsW7YMM2fOxMiRIxEQEIA1a9YgMzMTP//8MwAgJSUFK1euxOLFixESEoL27dtj3bp1OHPmDMLDwyWJkcgUssrpmUOWp6JmP5ZUpZmcad1zUFnS39KcGVyNFRISgn379kkeSEFBATZs2ICMjAx07doVMTExiI+PR//+/dXrODo6onfv3jh06BAAIDo6Gnl5eRrr+Pn5ISAgQL0OycsUMx6bE11HY/2xGyaNg+T14bbziiwL2H0xQfJtfrX3X8m3aS5MMVUN6cfgBsqDBg3C9OnTcfbsWXTs2BEuLi4arw8bNsyg7Z05cwZdu3ZFdnY2qlevjq1bt6JVq1bqZMXbW3OwJ29vb1y/fh0AEB8fDwcHB9SqVavMOvHx8Tr3mZOTg5ycHPXvqan6jVRKpmGNRcLZRijZsb6jaBrM3x/S52EmM7fiiU8NpQJQUCiQkZuPGk72km+fjMOs5saaMGECAGDJkiVlXlOpVAZXcTVv3hynTp3CgwcPsHnzZowfPx4REQ9nuy49TYUQosKpKypaZ8GCBfjggw8MipOoPPp+h5mAULHyzpl7aTmoV6uaftvR4+RTaglDVR5shn8VibO3U3Ho3X4SRlQ5rKpSPoOrsQoLC3X+VKYtj4ODA5o0aYJOnTphwYIFaNu2LT777DP4+PgAQJkSmoSEBHVpj4+PD3Jzc5GcnKxzHW2mT5+OlJQU9Q9HfpZOVq7y2p7wQmTerPHv92v0rUolKMpMaYzj7O2iEvmw83cl/9xSlUBIsR1rPP+NweBkx9iEEMjJyYG/vz98fHwQFhamfi03NxcRERHo1q0bAKBjx46wt7fXWCcuLg5nz55Vr6ONo6Ojurt78Q9Jo+WsHXKHYJZYNUJEZDyVmtwnIyMDERERuHHjBnJzczVemzJlit7bmTFjBgYNGoR69eohLS0NGzZswL59+7Bjxw6oVCpMnToV8+fPR9OmTdG0aVPMnz8f1apVw5gxYwAAbm5uePHFFzFt2jR4eHjA3d0db775JgIDAxESElKZj0YKIHWRO7ueExFZN4OTnZMnT+LRRx9FZmYmMjIy4O7ujvv376NatWrw8vIyKNm5e/cuxo0bh7i4OLi5uaFNmzbYsWMHQkNDAQBvv/02srKyMHHiRCQnJyMoKAi7du2Cq6urehtLly6FnZ0dRo0ahaysLAQHB2P16tWwtbU19KOREVhLgUVF7cjkYC3H3trdSs4EINCxgbvcoRAplsHJzuuvv46hQ4di+fLlqFmzJo4cOQJ7e3s888wzeO211wza1sqVK8t9XaVSYc6cOZgzZ47OdZycnPDFF1/giy++MGjfZD140ydpSXNGVdST6Yvd+nXZfm3DKQDArtd7VTUkyRh7yAmlPVsYM5z76bkVr0QVMrjNzqlTpzBt2jTY2trC1tYWOTk5qFevHhYtWoQZM2YYI0YiKuX3U7flDoGMLD4126D1z95O0fnareSsqoajt/vpOei2cI/J9kfmQ85egQYnO/b29uoie29vb9y4UTQYmpubm/r/RGQYQy8BxU/zZHpSFVqkZJluZOFME/aS/O7ANcSl6JeoSVFCo4RSHiXEQOUzuBqrffv2OH78OJo1a4a+ffti1qxZuH//Pn788UcEBgYaI0aSwbzt53Hgyn1sndgdzg5s/1QRXuvIUFfvZcgdQoVSs/MMH7SvVDKoxPZsFUlMz6l4JTLYvVT5jqvBJTvz58+Hr68vAGDu3Lnw8PDAhAkTkJCQgG+//VbyAEke3x2IwcX4NGw9yeoSfaTlSD9KrDUwxxuhNek6f7fcIcii3+KIilcig22R8X5icMlOp06d1P/39PTEX3/9JWlApCwFHACGyGpl5BYgr6AQ9rban4t5dSifECz1VQqDS3a+++47XLlyxRixEAGwzrmxmFOaD2v7U43+5rDcIZRR+hphqu9PRk4+vt1/FTcSM8uNp5i1nStKZnCys3jxYrRo0QJ+fn54+umn8c033+DixYvGiI3Iaih17iKiEzceyB1CGV+WmEndlI9G8/66gPl/XcTAz/abcK8kBYOTnYsXL+L27dtYvHgx3NzcsHTpUrRu3Ro+Pj546qmnjBEjmTGWWGg+3elqo5KcwbE0iJTuyNVEAKbt3UbSqNTcWD4+Pnj66aexePFifPbZZ3j22WeRmJiIX3/9Ver4iKpMSQmXrqfQNYevmzQOImPJL1TQF47oPwY3UP77778RERGBffv24Z9//kHr1q3Rq1cvbN68GT179jRGjERW4dTNB8jMyUe3JrXlDoWo0lZGxui9rvW1ziO5GJzsDB48GJ6enpg2bRp27twJNzc3Y8RFZHVGfHUQAHBsZrDMkRBJx9hTRyhBuSMoMKNTBIOrsZYsWYLu3bvjk08+QfPmzTF69GgsX74cFy5cMEZ8RFbnXpo0A2/dSspE5JX7kmyLHlLqvVupcZXHDEMmM2VwsjN16lRs2bIF9+7dQ1hYGHr27Inw8HC0bdtWPdggEclvy8nbeGblUQxcth87z8XLHQ6RcXBgStJDpRooA8DJkycRHh6OXbt2Yc+ePSgsLETdunWljI2IJHAxPg0v/xgtdxiVxlGWleXzPRxnrTRdZyinnVAOg5OdYcOGwd3dHZ07d8ZPP/2EZs2a4ccff0RSUhKioqKMESNZGXMZcybBwFmpiYzNFGnh8n1XTbAXy7Dgb45BpxQGN1Bu1qwZ/ve//6FXr16oUaOGMWIiMgvX7uueyHHk1wex6ZVusLVhqYSlMbTB7fk7qahTyxluzgZOqKkgQfPD8dGIQIS28pY7lDL4DSN9GFyy8+mnn2LIkCGoUaMGsrP5ZEukzYkbD3D+TqrcYSiSNfTOKRYVm4RHPz+Anh/vkTuUKrmbmoOX1h4vs9yK/pTlK6eq1Rqnv1Eig5OdwsJCzJ07F3Xq1EH16tVx7do1AMD777+PlStXSh4gWR/pLw7mdUW29IvjznN3db6WkJqNJIWPJm3I2RR+oeizpmbnGycYMyfFmc4mXaQPg5Odjz76CKtXr8aiRYvg4OCgXh4YGIjvv/9e0uCIyPLcSs7Uujw9Jx+PzN+NDnPDTBwRGdPVe7qrey3F3osJ2HT8ptxhUDkMTnbWrl2Lb7/9FmPHjoWtra16eZs2bTghKJEErOlJteRHvZmkPQmypmqvqjL1nE3WdK6WVrKn1ZnbKXjr19OI0dKOz1w6XFg6g5Od27dvo0mTJmWWFxYWIi8vT5KgiMwB78FVZ46H8HZyltwhaLX6UKzWmy0Zx7Lwsl3w2dVcuQxOdlq3bo0DBw6UWb5p0ya0b99ekqCIyDj2XkrAU98e1lmKQhWbtukfuUPQ6sztFKw+FGvy/aZl52HcyqP4Jcryq3FKJucZuWyHZU4M7no+e/ZsjBs3Drdv30ZhYSG2bNmCS5cuYe3atdi2bZsxYiSyGPqUZCzeddlo+39+VdFYWNM2/YNfXu5qtP1IiYMKKtu3+6/hwJX7OHDlPkZ1rlfp7fxy/CZ6N/OEdw0ng95n6Q36SRoGl+wMHToUGzduxF9//QWVSoVZs2bhwoUL+PPPPxEaGmqMGMmMKaOaQp6LYWXr6ot78BgTi9u1yy8oRG5+oWTbK3kjzs4zbXsaU0nNqlzzhdz8QuQXPvyOvP3raQz7MlKqsBSDyZgyGFyyAwADBgzAgAEDyiyPiopC586dqxwUWRA2bCEz0m9xBJIzcnH8/RA42tlW/AYDrDoYK+n2zFleQSE6zwtHSqlE6W4qk3AyDoNLdtLT05GVpdlA79SpUxg6dCi6dOkiWWBExWLuZ2Dy+pO4EKesQfoqU3LDnkXKdiMpE2k5+UZp6PsgS9njB5nSreSsMomO2eFX2azonezcunUL3bt3h5ubG9zc3PDGG28gMzMTzz77LDp37gxHR0dERlpeESTJb/wPx/DnP3cw4quDcodCRsQ8kMzddweuyR0C6aB3Nda7776L9PR0fPbZZ9i8eTM+++wzREREoG3btrh8+TL8/f2NGSdZsRv/9RzKqXRbCvnuokotyZEzKjY4thxCcHToksobHZzkpXeys3fvXvzyyy/o3r07nnjiCfj5+eHJJ5/Eu+++a8z4iMzWxzsu4tq9DHw5hkMyECy22mPryduy7l+u3PnkzQfy7JgqRe9qrPj4eDRu3BgA4OPjA2dnZwwfPtxogZH1spQH/4P/JiIuJRuPLz8sdyhkIKl60FjKuWzp5m47j2v30g16j77tungOKINBDZRLTg9hY2MDJyfDxkMg0ofUNT8KrUmS1bV7GRi38qhiq9lK4r3C8ijtb7oyMqbcNoGX76Yhy8RTcZC09K7GEkIgODgYdnZFb8nKysLQoUM1JgMFgBMnTkgbIZHEhBD4QYpuwMrPE8p14Mp9XLufgcae1eUOhUh25bU96r90P/xru5gwGpKa3snO7NmzNX5nFRaZq8NXEzF323mNZTcSM1Hfo5pMEcnHDAp2TMbYpVw81OYt5n4GEx4zVulkh8hc3XpQdiLHXp/sRSvfGjJEQ0okVTsLJpPGp7QqMVImgwcVJOvy/m9n8favypz4UGrnDRy08GYyJ9OsyL20nDIlJoWF2jOAyk6vQfIZ/8MxuUPAF3v+lTsEMgNMdqhCvxy/hYS0bLnDqDRj3ULf2XzGSFu2DL+fuo3O88Ix6/dzGsvn/XVB/X+lPpWviLgqyXZKlhCZQ2NwQ93WUkqqDyl7KFU2hsqo7DxgJD8mO6SXQunmRiQr8fHfFwEAPx65LnMk+imZi2w5Ie/YMZbq+wPX8OrPJ1Cgo3RP6RIzDJ/yQ6kJvbWp1ESgRGQpzPOmQ+bpo+1FpXpt6rjJHIluO87Gyx0CGQFLdsiolHArtcTqA5Lf3osJePXnE3iQyQk+DZWp0DFr9l++h1fWRcsdBhmBXiU7n3/+ud4bnDJlSqWDITIJ5j5mRQkj0N7S0hj9+dVRAICazvaY91hghdtgzl01pnhoWXMo1uj7IHnolewsXbpUr42pVComO0SkOKsOxiA5Mw9vhDar1PvLuwneTc3R+ZoC8jRFqkwCW/mJgPWXniP9pKbMcZVBr2QnJibG2HEQqSnhSZ5Mz5gP7h/8WTSI5Mj2ddBQx8BwvCkRWS622SHrI2MydeeB+XbhN5RKgVmrUtuKEJFxVao31q1bt/DHH3/gxo0byM3VbJy3ZMkSSQIjskT/tzZK7hBIJiw5qprfTnI4AKo8g5Od3bt3Y9iwYfD398elS5cQEBCA2NhYCCHQoUMHY8RICsDRbaVxM8l0A6Dpg41myVy8u8X4g3gqsDCSJGJwNdb06dMxbdo0nD17Fk5OTti8eTNu3ryJ3r1748knnzRGjETS4g2etCivt48Sq+TMmYpNt8nEDE52Lly4gPHjxwMA7OzskJWVherVq+PDDz/Exx9/LHmARKRc64/dwP7L9+QOQ7GYIxnum4iryC+QZ8h2lnRaLoOTHRcXF+TkFHW19PPzw9WrD+eQuX//vnSRkUWozMWDFxzjyJP4BnL2dgqmbzmDZ3VMBskbvSZLOK/jUqSphr18N03nawv+voifjt6QZD9KkFSJKSZIega32enSpQsOHjyIVq1aYfDgwZg2bRrOnDmDLVu2oEuXLsaIkczYmdspcodA/xnx1UFJt3fHRBMwSlnlwQSsarou2CPJdrafiSv39YvxupMhYzLG+bFk12XpN0oGM7hkZ8mSJQgKCgIAzJkzB6Ghodi4cSMaNGiAlStXSh4gUVVZwAO1JM7dSZU7BEmkZedh3Mqj+CXqpqTbLe88qew90BJKcwwh1QSf64/JU7JjjLZEGbnSD1RIhjO4ZKdRo0bq/1erVg1ff/21pAERGR2f7hVJ38Tg2/3XcODKfRy4ch+jOtczblBkEKlLDy2BtSW8SmVwyU6jRo2QmJhYZvmDBw80EiEiUj65r8OVqTZIzcqTPhATsIbhG1htTUplcLITGxuLgoKyo5Dm5OTg9m0O+kRmwPLvOXqzhqfOkzeS1f8PO39X66SexsI2QmQNSa450Lsa648//lD/f+fOnXBzc1P/XlBQgN27d6Nhw4aSBkfKYcqbotQ3CGu4oSuRUm70S8Iua/x/SdhlxC4cXGY9nidElkvvZGfEiBEAigbXKh5np5i9vT0aNmyIxYsXSxocEZkvixo4zoI+Cul21gjVcEyilUHvaqzCwkIUFhaifv36SEhIUP9eWFiInJwcXLp0CUOGDDFmrESSYLGyeSv511vw14VyRz5WEjMJ06ql5Ujfc4p/d2UwuM1OTEwMateubYxYiIwuO68A72w2/hw71kAJ1/Bv9l/D7gsJssaglOo6UiY+XCmDwckOAERERGDo0KFo0qQJmjZtimHDhuHAgQNSx0YkuT0X5b0xkn5+PByr/n9FyURiRo4k+zT2TYlJUeVExSZhwd8X5A6DzJzByc66desQEhKCatWqYcqUKZg0aRKcnZ0RHByMn3/+2aBtLViwAJ07d4arqyu8vLwwYsQIXLp0SWMdIQTmzJkDPz8/ODs7o0+fPjh37pzGOjk5OZg8eTJq164NFxcXDBs2DLdu3TL0o5EVYJGyJmPe4A29uZeM5f3fz5WzpoHbleAjVrb9kUW1W5LJkysO45uIa3KHQWbO4GRn3rx5WLRoETZu3IgpU6bgtddew8aNG7Fw4ULMnTvXoG1FRETg1VdfxZEjRxAWFob8/Hz0798fGRkZ6nUWLVqEJUuW4Msvv0RUVBR8fHwQGhqKtLSHw4lPnToVW7duxYYNGxAZGYn09HQMGTJEaxd5qhxjP5XeTMrEkrDLSEyX5imdyJh2nos3+D2x9zMqXokszo1E0w11QLoZPILytWvXMHTo0DLLhw0bhhkzZhi0rR07dmj8vmrVKnh5eSE6Ohq9evWCEALLli3DzJkzMXLkSADAmjVr4O3tjZ9//hkvv/wyUlJSsHLlSvz4448ICQkBUFT6VK9ePYSHh2PAgAGGfkSSwahvDiMuJRsnrifDu4aT3OEQii7SdWo5w9ZGWaUTcpTO5eRrPji9/GO0wduIik2ueCULlZNfAEc7W4Pek56Tj0wLmGrhTkq23CEQKlGyU69ePezevbvM8t27d6NevaoN3Z6SUtTtz93dHUBRY+j4+Hj0799fvY6joyN69+6NQ4cOAQCio6ORl5ensY6fnx8CAgLU65SWk5OD1NRUjR+SV9x/F4Qj1xKx+cTDKkgp5tphzZXhtp68hV6f7MWrP50w6n5MkbhoK5XUNgN8ebH8+U/5E1fqI90IPX3MRWW+xwGzd+KReWXvNUSVoXfJzgsvvIDPPvsM06ZNw5QpU3Dq1Cl069YNKpUKkZGRWL16NT777LNKByKEwBtvvIEePXogICAAABAfX1RU7O3trbGut7c3rl+/rl7HwcEBtWrVKrNO8ftLW7BgAT744INKx2qN5GrrkpVX9apIc+marCTFbSR2VKK6ppixyoNKnxOV+fOuPXwdL/bw13t9bcmRPkqXCFmrW8lZaObtKncYZMX0LtlZs2YNsrKyMGHCBGzYsAFnzpzB1KlT8dprr+Hs2bPYuHEjXn755UoHMmnSJJw+fRrr168v85qq1KOZEKLMstLKW2f69OlISUlR/9y8Ke3syZZoY9RNXLuXLncYJLEtJ24rJhmM1bNtw6/Rmp0P/k0w/Lw0ZPC4tOw8pFRyPq7vDsRU6n2WZsYWDvdA8tI72Sl5QXzssccQGRmJxMREJCYmIjIyEsOHD690EJMnT8Yff/yBvXv3om7duurlPj4+AFCmhCYhIUFd2uPj44Pc3FwkJyfrXKc0R0dH1KhRQ+OHyvfZ7ivotzhC7jCqjGNeaPp2/zWEnb9bqfdWVHKj7WHj4x0Xda4/Zf3JSsWRL0FVZ3ne2XzaoPXP3UnBuJVHcfrWA+MEZIYemOnkrWQ5DGqzU1FpiqGEEJg0aRK2bNmCPXv2wN9fs1jZ398fPj4+CAsLUy/Lzc1FREQEunXrBgDo2LEj7O3tNdaJi4vD2bNn1euQ+WAqYnrn4yrXZs2Qv9WdB1n4/dRtLN93tVL7Ko+xewruOGtYVd5T3x7BgSv3MfJr7W0Gicj0DOqN1axZswoTnqSkJL239+qrr+Lnn3/G77//DldXV3UJjpubG5ydnaFSqTB16lTMnz8fTZs2RdOmTTF//nxUq1YNY8aMUa/74osvYtq0afDw8IC7uzvefPNNBAYGqntnEZG8en+yF3kFlUtlt52+g0cDfSWNx5hVd2nZRQ2RjV3iRET6MyjZ+eCDDzRmO6+q5cuXAwD69OmjsXzVqlV47rnnAABvv/02srKyMHHiRCQnJyMoKAi7du2Cq+vDxm5Lly6FnZ0dRo0ahaysLAQHB2P16tWwtTWsqyPJr3QqrZT2JFQ1uhIdfUpl/jpT+UbSpAz8HpPcDEp2nnrqKXh5eUm2c32+ACqVCnPmzMGcOXN0ruPk5IQvvvgCX3zxhWSxEVFZhYUCNgobd6fkKMU3kzKxNPwyXurZCC192RaPiIro3WZH6vY6RGReYu5noP3cMHy554re65tCyUbn//sxGltO3MbQLyIN346OZy9e+4jMX6V6YxEZC88y5Zq3/QJSsvLw6a7Lcoei06X4osbWFbWXKX41N78QneeFY/Q3h40cmXVjwkhy0zvZKSwslLQKi0gf0vcAlHRzFiHugXkPZ7/qYCySM3Ir9d61h2NxLy0HR2OScOrmg0ptI+z8Xey7lACA818RKZXB00UQmRJLFI1v4/GbkkzLIafvIys3K/aNpIcDGT7IrFzCBADPrYoC8LAnFmni95jkxmSHFMUYhd330nJwn7Oplyu/UHM6BCVWO6Tn5GPw5we0vmZorqbt3lt6Uf+lEVh1MMbsE0EluHqPJV4kLyY7ZPEuxqeh00fhlZ7fiLQz9cP6T0eu49wdaSftLe8zXL6bjg/+PC/p/ohIHkx2SFGMef9MZxWD3i5oGVVZ7sKe8ubAKi9p0beUKi7FvNsuKd0NPec+IzIGJjtEJLsrd9MqXGdTqQlA9VVee5GS3dbnbqtaKU4+Sw7LNX2rYXOMEUnJoEEFiYiMIXTp/iq9/9fomzh3J0XvtjvFq0lZFZeRWyDdxiwQG2+TnJjskGzyCwphZ2vawkU2NVWWE9cf6DWHVEJa+VVM99NzceDKfanCqhwhf1WfkvHQkJyY7JBskjJy4VXDSWMZL4jmQ4ob+9Jw/QYo3HfpXtV3pgWTX9PJreREsERSYJsdks0/t1IQfT1JY5mxL4cL/7pg5D1YrvJym8JSpTOSd9eWeHPFjZ2zJax6EkydyqWt0TuRqbBkh2Tz0trjAIDTc/qbbJ932OPGKL47cA3n41IxunM9dGtcGz8ejpV0+/pUdRmi+Ma75eRtybYpBEfoJlIqJjsku5TMPLlDoCpa8PdFAMDvp+4gamYI5klcgjZj6xlJt2cMzHOIlIvVWKRovIEo093UbMzYoj0B6f7xHuRZafsMNlAmUiYmO0Skl5I38qkbTiFRx+Sbufkcb4aIlIXJDilK6YatUjYgJelc0mMQQKUw1TxfnOySSLmY7JDsfi1nZNyv9v5rwkioPPfTKz8ruDVgqkOkXEx2SHaf7b6i8zXOlqwc0deT5Q5BUmHn70q6PRbsECkXkx1SNDb4pKrSVb1UPPQBEVk+JjukaHxapqpKNdGcTBxUkEi5mOwQkUW6l5YDIQT+ufnAJPsb/0OU5FVjRCQNDipIRIi+noxujWvrvX6Sjm7nStJ5XjjGBtU32f4uxKVySgQihWLJDila5L8yz2RtJcZ8d1TnaysirpowEmn9dPSG3CEQkQIw2SGici38+yJSszmlBxGZLyY7ZDV6Ltordwhm67X1J+UOgYio0pjskNVIzzFNrxxLtPfSPblDICKqNCY7REREZNGY7BAREZFFY7JDREREFo3JDhEREVk0JjtERERk0TiCMhGprYi4it9O3pY7DCIiSTHZISK1hX9flDsEIiLJsRqLiAAAw76MlDsEIiKjYLJDRACA07dS5A6BiMgomOwQERGRRWOyQ0RERBaNyQ4RERFZNCY7ZJA7D7LkDoGIiMggTHbIILn5hXKHQEREZBAmO0RERGTRmOwQERGRRWOyQwYRcgdARERkICY7REREZNGY7BAREZFFY7JDAID8gkJ8tfdfnLyRLHcoREREkuKs5wQA+PnYDXyy8xIAIHbhYJ3rCcFWO0REZF5YsmOlUrLy8E3EVfUggVfupuv1PqY6RERkbpjsWKkZW85gwd8X8cTyQ3KHQkREZFRMdqzU/sv3AAB3UrINeh9rsYiIyNww2SGDTPwpGnkFnDKCiIjMB5MdK/HPzQc4+O/9Km/n8t10bDt9R4KIiIiITIO9sazE8K8OAgCOzgiGdw2nKm0rI6dAipCIiIhMgiU7VuZu6n9tdFSV38Z7v52VJhgiIiITYLJDREREFo3JjpVKy86XOwQiIiKTkDXZ2b9/P4YOHQo/Pz+oVCr89ttvGq8LITBnzhz4+fnB2dkZffr0wblz5zTWycnJweTJk1G7dm24uLhg2LBhuHXrlgk/hbIJIfDpfyMjA4AKKlyKT5MxIiIiItOSNdnJyMhA27Zt8eWXX2p9fdGiRViyZAm+/PJLREVFwcfHB6GhoUhLe3iznjp1KrZu3YoNGzYgMjIS6enpGDJkCAoK2IgWAPZcTMCXe//VWFZ6/qviUZQNceZWSpXiIiIiMhVZe2MNGjQIgwYN0vqaEALLli3DzJkzMXLkSADAmjVr4O3tjZ9//hkvv/wyUlJSsHLlSvz4448ICQkBAKxbtw716tVDeHg4BgwYYLLPolQJaTlllkXFaiY73RbuwZig+gZtd+iXkVWKi4iIyFQU22YnJiYG8fHx6N+/v3qZo6MjevfujUOHiqY4iI6ORl5ensY6fn5+CAgIUK+jTU5ODlJTUzV+rIVKBWw+Ubaar7Dw4dDI3RfuMWVIRERERqXYZCc+Ph4A4O3trbHc29tb/Vp8fDwcHBxQq1Ytnetos2DBAri5ual/6tWrJ3H0ylF6eoevSlVpFcvIfVjtd7sS1VpERERKpdhkp5hKpTkgjBCizLLSKlpn+vTpSElJUf/cvHlTkljNwd9ntSeBf/7DUZGJiMgyKTbZ8fHxAYAyJTQJCQnq0h4fHx/k5uYiOTlZ5zraODo6okaNGho/REREZJkUm+z4+/vDx8cHYWFh6mW5ubmIiIhAt27dAAAdO3aEvb29xjpxcXE4e/aseh1rJ8BpyomIyLrJ2hsrPT0d//77sA1JTEwMTp06BXd3d9SvXx9Tp07F/Pnz0bRpUzRt2hTz589HtWrVMGbMGACAm5sbXnzxRUybNg0eHh5wd3fHm2++icDAQHXvLCIiIrJusiY7x48fR9++fdW/v/HGGwCA8ePHY/Xq1Xj77beRlZWFiRMnIjk5GUFBQdi1axdcXV3V71m6dCns7OwwatQoZGVlITg4GKtXr4atra3JPw8REREpj0qI0v11rE9qairc3NyQkpJice13Jv18AttOx8kdBhEREWIXDpZ0e/revxXbZoeq7lZyJhMdIiKyekx2LNiNpEy5QyAiIpIdkx1LZvUVlEREREx2iIiIyMIx2bFgLNghIiJiskNEREQWjsmOBeOgAkREREx2iIiIyMIx2SEiIiKLxmTHgq09HCt3CERERLJjsmPBdp2/K3cIREREsmOyQ0RERBaNyQ4RERFZNCY7REREZNGY7BAREZFFs5M7AJJeYaGASiV3FERERMrAZMfCZOcVYOCy/WjsWV3uUIiIiNSe7dpAtn0z2bEwkVfuIzYxE7GJmXKHQkREpFbdUb6Ug212iIiIyKIx2SEiIiKjk3NualZjWYir99IRvDhC7jCIiIgUhyU7FuLtX0/LHQIREZEiMdmxEFm5BXKHQESkSNUcbOUOgWTGZMdCcFwdIiLtqjmwxYa1Y7JDREQWjQ+DxGTHQtxKzpI7BCIiIkVismMhUrLy5A6BiMzAEx3ryh2CyQk5+zxbEEc7G4S09JJ1JOTKYkUmEZEVcbDjMy5VjkoFfD++MwBg7eHrMkdjGJ71REREZNGY7JiJ5Ixc7L98D4WFLI8losrr1thD7hBMTqkNlEd1sr4qRbkw2TETQ76IxLM/HMPPx25oLBesjCYiPY0Jqo9WvjVk2//rIc1Mvs+V4zuZfJ/6mvdYIDZP6ApfNye5Q7F4THbMxO0HRb2tdp6LVy9Lzc5Dj4/3Ys4f5+QKi4jMSEvfGlDJWMwh1eB+TxrQyDq4pbck+zQGe1sbdGzgjg3/6yJ3KHop+Wz9eAfzKpVismPGNh67idsPsrD6UKzcoRARmczEvk0MWl+htVhqDTxc5A7BYHVqOeu97rC2fnB1ssPz3RoaL6AKMNkxMyqVClfupiEzNx9C1jlkiaiyejXzNPo+hrfzK7OshpOdLDf+bo098Mek7pJtz85G6emLZSpZKNi5YS2d67WtV1Pj9w+Ht8apWf3hVUO+6jomO2bm6LVEhC7djwHL9kOl+OcVItKmX3PjJzu2WqqrnOzlmSPqxxeD0KZuTVn2TdIpec/p2dQTq57vjHUvBpVZ7/WQpmWW2cqcoDLZMTM5+YUAgJtJWZj31wWZoyGiyiiv3czkfoZV0Ri+b6NuXiu5b3RK6Y01ycDqN6Xr29wLdbVUZ9Vzr6bxuxIezJnsEBFVUpu6bpV6n39t3W00xkk0Oq2uSm57W+u77Cuh06qbsz1e7t2oytsxJGHq2bR2lfdXGY09q8uy3/JY31lPRCSjXs08y61O8nJ1wrguDRBYp3KJVEX8ajpjbFB9/F8Pf43llz8aZJT90UOuTvZV3sbEvo3xz+z+8K7hKEFE0jNFe7TKYLJDRFRJukoMnMtJZpz0mK5h7ogAvD+kVWXDqtC8xwLxXqnt6zONhLESMEMZWi2llGosqbg5Vz1pMpYXSyXRSsFkh4gUq20lq4mk3H9AHf0H4Rsc6IvODWvhyPTgKu9b3xt0A49qFa9UvM1KxlJs1fOdq7gF45FqDB9zNCW4bINgUymd7/fWUrKjhJ7DTHaISLF+n9RD1v0/390fr/RurPP1kiU4p2aF4quxHbDplW5wq2bcJ+9Nr3RV/9/bhN15tfXwAgC//0YA1tYO2ZAG128NaF6puADgVQka/wa38KryNj57ql2Vt6EPH7eHDYM9XbVXaSmx7YxcmOxQmbp7ufz2andsmyzvzc1cnHg/VO4QjG5IG98qvX/RE20kiaN/Kx+ty5t5V8fCxwPxz+z+ODojGDWrOejcxtC2Zce8qYrODd0l3Z422hpR66rq2vNmHxx4uy9a+ZUtBXsjVP8pIl7t2wSHp/fDmTn9K6wOMWTUYVP2Bmqt5RiUVLu6NG1tnO0f/i10dXYzdXVXTSMn+VXBZMcMZOUWGHX7bw9sUaUnKqm0q1cTAQppE6BkC0YGwt1F943V2D4c3rrMMg8jxPPlmA6Vfq+djQqjOtXDE1qmFQhpqf3p3dXRrsyy9vVrar3Bn3g/FLte741GntXh5mxfbunKlOCm+OLp9g9js1VeAxJtn9HFsWy1kIuWYwQUjd9Tz70axgaV7Ummq5t9+Bu9tJ43vm7OcHWyx/tDWuGpzvW0vlelUqFLIw+seKZjiWVaVzWISgXsnNqr0u9/a0BzNPFyrXog5TC0Z1lzb+PGYy6Y7JiBqNgko27fwc5G1skBzVGIjPPtPP1IfQCGPdlK6dmuDcssM3a1jT46NqiFPdN6I/yNXoh+r6jka8HIQPwxqTuuzn8UfZp7wruGI5Y91V7r+w9O74dFj2uWBukaxr8yyeb7Q1qhbi1nTB/UUuc6y0a3M3i7VfHtuI7wdXPSOjCcLgNa6z73n+pcD9sm99BZreJb82FS2MTLVWv7Dl3q1Cx/eoLykgDnctrzlP5bNvdxxZQSVW/D2/mhQ/2aWkcMLn3cnulSlOy18JEmwXiu1PQKIS291QlnsxJJTHklV0tHt6vw2JXWs2lt1KnpbNAcZIAyuvjrwmTHDBizJ0Fjz/8u5jI/bP7wnHJnJtamVzPTjl+hrapR1w1FH1XttupS6ubh7Vq1diOlG75+NCJA/f/iovGGFTTE3TyhGxp5VkcTL1d18mVva4M2dWvC1kaF1c8/gqMzQlDd0Q7bJvfA890bon6Jwc9qONljVOd62PhfErn4ybZV+kylvdjDH5Hv9EM992poWFv7ZxnRvo6k+yytZamHmv6tfXB4ejAe8de/WsyunHF6VCoVAuq46SwpfjTAF6/2bfxwJvIS152+FYwqvWNqT71jBIAXuvtj68RuAMqvXqmoSvizp9pjy8Tu+GpMhzLJRw8d49isfK4znuvWEN+O66j1dX3NGdYaozo9TDi+LzGD+1sDmuOF7v747dXuOhsA92haG638auDgu/20llzq0tK36D2PG5js6KKEJIjJjhkwZn1zE6+iBmzdG9eGvQzF6408XTCkjS/6tVDezMTHZgbrbLDo56b7SckY7Y4alDMIXWV0auheYQ+O4vr+kJZe+HpsB7g52+PHFx8BAAwu1Z5mwcjACquy1r/UBRfnDiyzvKFHNfRt/vA4B9ZxUz8hA8DeaX3w6ytdJS1NC6jjhtlDW2u9CQY18kDswsGSXei18apiclja8HZ+aKTHOVJ6ZNvSJvbR3hjb3cUBs4fq3xV+SBtf1HN31rhRA4CNjQpvDWihdSby5c+UnxhUd7RDl0buaF+/prpBdJ/mnmhU20XrPGCzhrZC+/ra528qPTTAOwNbwMPFATMH6/6MXjWcMGdYa2yf0gO1qtljrpbq3GJ1ajpjzrDWGm2YDH1oLa5+0nX9d3Wyx6yhrdCu1DxUJZVs29VMgtImbdsoeSyV3MVf/1SPquzkjWS8si4aMwe3wrASDRYLCwVO305BfkEhnlhxGABw/L0QJGXkopm3K3ILpGuz82THupjUrwkSM3Kx8dhNvDWw6AnMwc4Glz8aBP/pf0m2L33sfqN3uUPnSyGkpRfCLyQY/D4vVycE1nXD7ouGvTegjhsm92uCQ1cTEX092eD9atNOy7xCVW3o2MXfHZ/reG1sUH116Urx32dQgI/Ov1XD2i44/l5ImfNnUt8msFEB3m5O6NrYo9x47G1VyCsQZUaZreXigE4u7th5Ll6PT2UYfc+8Rp4uuHYvQ/L9S+Wzp9pDCKFx/Ps098TWk7cN2s6U4KZ4kJWH0JbeWBx2Sb08+r0Qg76n1RzssP+tvhW+p09zL2w5cRtO9jYVztulUqmw/qUu6v8DRW2Fdk8ruoZ8tfdfveMrbUKfxnildyO9PmNrPzeceD9U+7qlSjDq1tKWXOpXzDG0bdUa6Jf25Zj2WBp2GUdjknA9MVO93NZGhYJC/WL6emzZdnQ+bg8TdyVP0Mpkx4Re/jEaCWk5mLL+pEay8+rPJ/D3Wc0LeaePwo0Swyf/Fc038HBBh1JPPSqVCl883R6T15/EoACfMjEZg1SJzqLH2+DtzafLLO/W2APfj++Mhu9uN2h7FY0C6lvTqdwb4LT+zRF4Lh7/+zFaY7mnqyPWv9QFIUsidG67uqMdmnlXx4kbD9TLAuu6YcP/umjUvZfuafHji4+ggbsLen2yt9zYi3VrUn5VXOm/TcnfSxZLH3i7b5nX5z8WCO8ajujVzLPM9ARdG3ng8LXEMvs7+G4/XIhLQy8Dhrg/PL0friZkoLFX5Uq+6taqhn9upVS43pYJ3fDFnn+xMjLGqIP96TKgtTd2nrur9bXiUo7Sf69hbf3g5myPvAKBl9Ye16tXlJO9LeY/FggAGslOyW376NnVXZ/v9tA2vnBztkdLX/1KHbRtU5/9zB7aGiO+OmjwtqVYtzJ+eK4TejWVdiRiXzdnLHqiLSasi9ZIdoa388OWExUnxQF1ahjc9kdJWI1lAtfupSN48T4kpOWol5V8CjFFUqGvoW39cGXeIEzrr393UX14ujqWqe/W5btnO2FYWz8sHa1/m4lROnpt/PR/5Te8HBzoq7WEpKIGhq393BD2em/1jUGbkJbeePqRelgwMlC9vcGBvmjiVR1X5g3Cq301qwyuzBuEmAWP4uSsUGye0K3M9ro08ii3GqJnU0/U96hWZrtAUXuWtuUUd5dmSBW7tpg6NKiJ4JbeWudh+vmlIFyZNwj13IsunAMCirp2e7k6oXczT503kk5aulv7ujmjR9Pa8C2nWrE8c4a1xuA2vvi5gvOkZjUHvD+kFa7MG2TwCLHt69fUuvzPST0wONAXI/Vop7NCSxXPuheDMKSNr87kS6VSoU9zL4S28saVeYMkG3jutZCmGNbWTz0vmK9b5avkVCoVejfz1Fmt16VR+aWBupT+3pdX1aNNZZuYSDV4Xr8W3uq2UcauGnIvZ8iEkpaPrVr7I7mxZMcE+i0u+xT/yc5LcHdxqPJYIob4+zX9GvjZ29qgiZcr1rzwCF5cHYV8PYs4SzoyPRhdFuxW//7x44Fo7Fkdqw/FVvje0FbeCG3ljdTsPPWy9wa3xNOP1Efr2TvVv++5mIBDVx+WENSsZo8HmXka2yq+cY5o54ffTt0BUFRl9/lT7dDK1w31PaohPScf7209o34deHgD1zZOxek5/QEUFf+Wfn1aiadnGxsVFows6t0zoLUPDly5hwGti27s9rY2eGtAC3y196p6/eLEoHTbqW4VVP+UZmujmWA806U+OjaohTXPd0a7D8MM2pahdk7thTspWWjho7t3n0qlgr2tCr9N7I6DVxPL7d1TUv9W3ljxTAe8su6EVOHC09URXxnQxd2QSTT3vdkHl+6moY+OUsLAum74amwHLAu/rPX1AD832NqoULeWM1QqFWxUQMmvYo+mtXU2kK1K3BWp4WSPz59uj9z8Quw8F1/phEQfw9r6wcHORq9pKkSJ4sbuFZRampPKJju7p/WWdHsVtfcClNEQWRcmO0aUnJGL9nN131ymbzmD6VvOmCye0j0xKtK7mScufTQIx2KSsCLiKgSA/ZfvVfi+nVN7adTjAlA3QPZ0dcS9EiVc5XEsMfbH6M714OJoh49GBODw1USM79YQTz1SH69vPIXBgUUJ45YJ3bQmlkBR98txXRvg893/4q0BzTXG86nuaIclo9pBpVKp2zg0+6/h9tOP1MfRa0no28ITT3Ssh7yCQo22BQNae2NYWz+0r18TozvXQzUH7V8pdxcHDG+n+wm+vN5RFbVlKC2g1KBm7eoVVVeWHPTukVKlJC4OtsgwYDwnXde05j6uaK5nQ0iP6o4a1bkVUalUGBhguoeDqmpY2wUNq9Cw3NnBFuc+GKDYdhAOdjaSD5ZYmo2NCo8Gms/f3Dg3+8r9/asyerLUc6AZeg0zBiY7RvT4ikNyh6CmbXA1fdjaqNC1sYdG49LLd9Nw9nYKHOxs8OWef3ExPk3jPaV7OpScs6a+ezV1slPc1kMXRztb/Px/QcgvFOrZgp/p0kDdU8fe1gbfPfuwK2ajUl/ukjd0lUqFjg3cseaFR7Tuy8ZGhaWj2+H/evrj6r0MBP33tOpkb4sVJbqP2tpofjY7Wxt8XmLAuMqy0fK4Vdxu6n+9Gml5h26le7qUrCaJeKsPjsYklak6KX2NHvPfWD5KNPPRlpj31wXMeLSF3KFIYni7OlgWfkXrPGBy3iReD2mGF9ccL9Ojylx1aeSOI9eSsGRUW8zcelay7apUhiU5Hi6OuJ+eK9n+ASC4hTdmQr/P9EJ3f/x1RnvTifA3euGfmynqGgd9JoetSIf6Ncsd58hUmOwYkZJ6b2gb9baymnm7qge08qzuiNHfHil3/ZK38SWj2uKtTafxSp9GehWLVtSItrTVz3fGV3v/xfRHW6Ktlh5MFWnt54bWfqYfxVnbs9vXYzsgOTOv3AHs+jT3xL5L9zTaGNnaqPBs1wZYe/g63h7YHDYlSgYaeLjoHCivmXd1XL6brviRrF/q1QgjO9SBh0TD7svNv7YLTs0KVSf0uvRrUdSrUD02lhaPNHTHsdgkg9uoaBPc0hvR74XIOlq3IZpWMFLwT//XBalZeajl4oCw83fx99l4jXGWKqua/cNSUUf7ipODr8Z2wPQtp8u0oerb3BN7L91DM2/NhzZ9qp1Kl6SXp1NDd2yb3AMf/nkeL/Twx/ESg9Y28XLVGAG6fb2aGNzGt0rHSSlVikx2rEAr3xo6q1eqKqiRBxY93gb+ni548r9u817lVMk08HDBLyUmMZRan+Ze6NO86pP5mZquXiYV3WiWjW6HX6NvlakO+nB4AD4cHqDjXQ8VJ0VvDWiOwYG++OOfO3qVAso9w7SlJDrFyptXq9jiJ9thU/TNcquOlj/TAVtP3pZscEJzOs79W3njoxEBOqtgbG1UqPXf92nh423QsUEtDGmj/VgaUqL23pBWmL7lDNxdHMq9zro6Fb3WxKs6Nr1StgPC0tHtsOn4rTJjBpUewFMKAXXc1NfhurWc8X1kjNZpJVQqlUFt2rRxlKB0SAoqIZTcpMg0UlNT4ebmhpSUFNSoId20CYZ2dzYGL1dHbJ7QTa9SlKpKzc5DQYFQX1A6zwvHvbQc9GvhhR+e61zBu61T8TlSt5YzIt/pZ/L9CyFwKznL4PMjMT0HL6yOwhOd6mFciQEAiSxBek4+xq08ikEBPvhfL92z3he7mZSJOjWdNUpRi+29mIBPdl7C4lFtDW43CRS1/Xxu1TGM7FAX48vp0Rpx+R7e2vQPEjNyMXd4AMYE6V8VHZeSBQ8Xx0pXWxVfxzxcHBD934jU3+6/ir/PxuPHF4NQ3YDRmw2l7/2byQ7MP9lp6lUdVxLSNZad/WCAUU8wfdxMysSWE7fxbNcG6gSINP1v7XHsOn8XHwxrXe6FjIhIqSavP4k//7mDGY+20Cs5lBKTHQMYK9lJycxD2w93VbheSEtvhF/QPmAYUNRbKD0nHwCwdWI39SzLJWcgnr7lNCIu3cMnT7ZFp4a14Ggnf4MwqlheQSGu3ktHc29Xow9URkRkDPkFhfhXpuuY1SU7X3/9NT755BPExcWhdevWWLZsGXr21G9cGWMlO0RERGQ8+t6/ldFyqIo2btyIqVOnYubMmTh58iR69uyJQYMG4caNG3KHRkRERDKziJKdoKAgdOjQAcuXL1cva9myJUaMGIEFCxZU+H6W7BAREZkfqynZyc3NRXR0NPr376+xvH///jh0SPugfjk5OUhNTdX4ISIiIstk9snO/fv3UVBQAG9vzVFjvb29ER+vfZTIBQsWwM3NTf1Tr572SSSJiIjI/Jl9slOsdAtwIYTOVuHTp09HSkqK+ufmzZumCJGIiIhkYPYjKNeuXRu2trZlSnESEhLKlPYUc3R0hKOj+YwMSkRERJVn9iU7Dg4O6NixI8LCNGcXDwsLQ7duZYfkJiIiIuti9iU7APDGG29g3Lhx6NSpE7p27Ypvv/0WN27cwCuvvCJ3aERERCQzi0h2Ro8ejcTERHz44YeIi4tDQEAA/vrrLzRowDl7iIiIrJ1FjLNTVRxnh4iIyPxYzTg7REREROVhskNEREQWjckOERERWTQmO0RERGTRLKI3VlUVt9HmHFlERETmo/i+XVFfKyY7ANLS0gCAc2QRERGZobS0NLi5uel8nV3PARQWFuLOnTtwdXXVOZ9WZaSmpqJevXq4efMmu7TrgcdLfzxW+uOx0h+Plf54rPRnzGMlhEBaWhr8/PxgY6O7ZQ5LdgDY2Nigbt26Rtt+jRo1+GUwAI+X/nis9MdjpT8eK/3xWOnPWMeqvBKdYmygTERERBaNyQ4RERFZNCY7RuTo6IjZs2fD0dFR7lDMAo+X/nis9MdjpT8eK/3xWOlPCceKDZSJiIjIorFkh4iIiCwakx0iIiKyaEx2iIiIyKIx2SEiIiKLxmTHiL7++mv4+/vDyckJHTt2xIEDB+QOyajmzJkDlUql8ePj46N+XQiBOXPmwM/PD87OzujTpw/OnTunsY2cnBxMnjwZtWvXhouLC4YNG4Zbt25prJOcnIxx48bBzc0Nbm5uGDduHB48eGCKj1hp+/fvx9ChQ+Hn5weVSoXffvtN43VTHpsbN25g6NChcHFxQe3atTFlyhTk5uYa42NXSkXH6rnnnitznnXp0kVjHWs5VgsWLEDnzp3h6uoKLy8vjBgxApcuXdJYh+dWEX2OFc+tIsuXL0ebNm3UgwB27doVf//9t/p1szynBBnFhg0bhL29vfjuu+/E+fPnxWuvvSZcXFzE9evX5Q7NaGbPni1at24t4uLi1D8JCQnq1xcuXChcXV3F5s2bxZkzZ8To0aOFr6+vSE1NVa/zyiuviDp16oiwsDBx4sQJ0bdvX9G2bVuRn5+vXmfgwIEiICBAHDp0SBw6dEgEBASIIUOGmPSzGuqvv/4SM2fOFJs3bxYAxNatWzVeN9Wxyc/PFwEBAaJv377ixIkTIiwsTPj5+YlJkyYZ/Rjoq6JjNX78eDFw4ECN8ywxMVFjHWs5VgMGDBCrVq0SZ8+eFadOnRKDBw8W9evXF+np6ep1eG4V0edY8dwq8scff4jt27eLS5cuiUuXLokZM2YIe3t7cfbsWSGEeZ5TTHaM5JFHHhGvvPKKxrIWLVqId999V6aIjG/27Nmibdu2Wl8rLCwUPj4+YuHChepl2dnZws3NTaxYsUIIIcSDBw+Evb292LBhg3qd27dvCxsbG7Fjxw4hhBDnz58XAMSRI0fU6xw+fFgAEBcvXjTCp5Je6Ru4KY/NX3/9JWxsbMTt27fV66xfv144OjqKlJQUo3zeqtCV7AwfPlzne6z1WAkhREJCggAgIiIihBA8t8pT+lgJwXOrPLVq1RLff/+92Z5TrMYygtzcXERHR6N///4ay/v3749Dhw7JFJVpXLlyBX5+fvD398dTTz2Fa9euAQBiYmIQHx+vcUwcHR3Ru3dv9TGJjo5GXl6exjp+fn4ICAhQr3P48GG4ubkhKChIvU6XLl3g5uZmtsfWlMfm8OHDCAgIgJ+fn3qdAQMGICcnB9HR0Ub9nFLat28fvLy80KxZM7z00ktISEhQv2bNxyolJQUA4O7uDoDnVnlKH6tiPLc0FRQUYMOGDcjIyEDXrl3N9pxismME9+/fR0FBAby9vTWWe3t7Iz4+XqaojC8oKAhr167Fzp078d133yE+Ph7dunVDYmKi+nOXd0zi4+Ph4OCAWrVqlbuOl5dXmX17eXmZ7bE15bGJj48vs59atWrBwcHBbI7foEGD8NNPP2HPnj1YvHgxoqKi0K9fP+Tk5ACw3mMlhMAbb7yBHj16ICAgAADPLV20HSuA51ZJZ86cQfXq1eHo6IhXXnkFW7duRatWrcz2nOKs50akUqk0fhdClFlmSQYNGqT+f2BgILp27YrGjRtjzZo16kZ+lTkmpdfRtr4lHFtTHRtzP36jR49W/z8gIACdOnVCgwYNsH37dowcOVLn+yz9WE2aNAmnT59GZGRkmdd4bmnSdax4bj3UvHlznDp1Cg8ePMDmzZsxfvx4REREqF83t3OKJTtGULt2bdja2pbJPBMSEspkqZbMxcUFgYGBuHLlirpXVnnHxMfHB7m5uUhOTi53nbt375bZ171798z22Jry2Pj4+JTZT3JyMvLy8sz2+Pn6+qJBgwa4cuUKAOs8VpMnT8Yff/yBvXv3om7duurlPLfK0nWstLHmc8vBwQFNmjRBp06dsGDBArRt2xafffaZ2Z5TTHaMwMHBAR07dkRYWJjG8rCwMHTr1k2mqEwvJycHFy5cgK+vL/z9/eHj46NxTHJzcxEREaE+Jh07doS9vb3GOnFxcTh79qx6na5duyIlJQXHjh1Tr3P06FGkpKSY7bE15bHp2rUrzp49i7i4OPU6u3btgqOjIzp27GjUz2ksiYmJuHnzJnx9fQFY17ESQmDSpEnYsmUL9uzZA39/f43XeW49VNGx0saaz63ShBDIyckx33PKoObMpLfirucrV64U58+fF1OnThUuLi4iNjZW7tCMZtq0aWLfvn3i2rVr4siRI2LIkCHC1dVV/ZkXLlwo3NzcxJYtW8SZM2fE008/rbW7Yt26dUV4eLg4ceKE6Nevn9buim3atBGHDx8Whw8fFoGBgYrvep6WliZOnjwpTp48KQCIJUuWiJMnT6qHIjDVsSnuyhkcHCxOnDghwsPDRd26dRXT5VWI8o9VWlqamDZtmjh06JCIiYkRe/fuFV27dhV16tSxymM1YcIE4ebmJvbt26fRXTozM1O9Ds+tIhUdK55bD02fPl3s379fxMTEiNOnT4sZM2YIGxsbsWvXLiGEeZ5TTHaM6KuvvhINGjQQDg4OokOHDhpdHC1R8VgL9vb2ws/PT4wcOVKcO3dO/XphYaGYPXu28PHxEY6OjqJXr17izJkzGtvIysoSkyZNEu7u7sLZ2VkMGTJE3LhxQ2OdxMREMXbsWOHq6ipcXV3F2LFjRXJysik+YqXt3btXACjzM378eCGEaY/N9evXxeDBg4Wzs7Nwd3cXkyZNEtnZ2cb8+AYp71hlZmaK/v37C09PT2Fvby/q168vxo8fX+Y4WMux0nacAIhVq1ap1+G5VaSiY8Vz66EXXnhBfe/y9PQUwcHB6kRHCPM8p1RCCGFYWRARERGR+WCbHSIiIrJoTHaIiIjIojHZISIiIovGZIeIiIgsGpMdIiIismhMdoiIiMiiMdkhIiIii8Zkh4jMVmxsLFQqFU6dOmW0fTz33HMYMWKE0bZPRMbHZIeIZPPcc89BpVKV+Rk4cKBe769Xrx7i4uIQEBBg5EiJyJzZyR0AEVm3gQMHYtWqVRrLHB0d9Xqvra2tehZmIiJdWLJDRLJydHSEj4+Pxk+tWrUAACqVCsuXL8egQYPg7OwMf39/bNq0Sf3e0tVYycnJGDt2LDw9PeHs7IymTZtqJFJnzpxBv3794OzsDA8PD/zvf/9Denq6+vWCggK88cYbqFmzJjw8PPD222+j9Iw6QggsWrQIjRo1grOzM9q2bYtff/3ViEeIiKqKyQ4RKdr777+Pxx9/HP/88w+eeeYZPP3007hw4YLOdc+fP4+///4bFy5cwPLly1G7dm0AQGZmJgYOHIhatWohKioKmzZtQnh4OCZNmqR+/+LFi/HDDz9g5cqViIyMRFJSErZu3aqxj/feew+rVq3C8uXLce7cObz++ut45plnEBERYbyDQERVY/DUoUREEhk/frywtbUVLi4uGj8ffvihEKJopupXXnlF4z1BQUFiwoQJQgghYmJiBABx8uRJIYQQQ4cOFc8//7zWfX377beiVq1aIj09Xb1s+/btwsbGRsTHxwshhPD19RULFy5Uv56Xlyfq1q0rhg8fLoQQIj09XTg5OYlDhw5pbPvFF18UTz/9dOUPBBEZFdvsEJGs+vbti+XLl2ssc3d3V/+/a9euGq917dpVZ++rCRMm4PHHH8eJEyfQv39/jBgxAt26dQMAXLhwAW3btoWLi4t6/e7du6OwsBCXLl2Ck5MT4uLiNPZnZ2eHTp06qauyzp8/j+zsbISGhmrsNzc3F+3btzf8wxORSTDZISJZubi4oEmTJga9R6VSaV0+aNAgXL9+Hdu3b0d4eDiCg4Px6quv4tNPP4UQQuf7dC0vrbCwEACwfft21KlTR+M1fRtVE5Hpsc0OESnakSNHyvzeokULnet7enriueeew7p167Bs2TJ8++23AIBWrVrh1KlTyMjIUK978OBB2NjYoFmzZnBzc4Ovr6/G/vLz8xEdHa3+vVWrVnB0dMSNGzfQpEkTjZ969epJ9ZGJSGIs2SEiWeXk5CA+Pl5jmZ2dnbph8aZNm9CpUyf06NEDP/30E44dO4aVK1dq3dasWbPQsWNHtG7dGjk5Odi2bRtatmwJABg7dixmz56N8ePHY86cObh37x4mT56McePGwdvbGwDw2muvYeHChWjatClatmyJJUuW4MGDB+rtu7q64s0338Trr7+OwsJC9OjRA6mpqTh06BCqV6+O8ePHG+EIEVFVMdkhIlnt2LEDvr6+GsuaN2+OixcvAgA++OADbNiwARMnToSPjw9++ukntGrVSuu2HBwcMH36dMTGxsLZ2Rk9e/bEhg0bAADVqlXDzp078dprr6Fz586oVq0aHn/8cSxZskT9/mnTpiEuLg7PPfccbGxs8MILL+Cxxx5DSkqKep25c+fCy8sLCxYswLVr11CzZk106NABM2bMkPrQEJFEVEKUGkSCiEghVCoVtm7dyukaiKhK2GaHiIiILBqTHSIiIrJobLNDRIrFWnYikgJLdoiIiMiiMdkhIiIii8Zkh4iIiCwakx0iIiKyaEx2iIiIyKIx2SEiIiKLxmSHiIiILBqTHSIiIrJoTHaIiIjIov0/pTCfQ4lbo2YAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"#env = gym.make(\"CartPole-v1\")#, render_mode='human')\n",
"# Training loop\n",
"rewards = []\n",
"\n",
"for episode in range(n_episodes):\n",
" state, _ = env.reset() # Reset environment to start a new episode\n",
" total_reward = 0\n",
" done = False\n",
" \n",
" while not done:\n",
" # Discretize the state\n",
" state_discretized = discretize_state(state)\n",
" \n",
" # Exploration vs Exploitation: Choose action\n",
" if np.random.rand() < epsilon:\n",
" action = env.action_space.sample() # Explore: Random action\n",
" else:\n",
" action = np.argmax(q_table[state_discretized]) # Exploit: Best known action\n",
" \n",
" # Step in the environment\n",
" next_state, reward, terminated, truncated, _ = env.step(action)\n",
" \n",
" # Discretize next state\n",
" next_state_discretized = discretize_state(next_state)\n",
" \n",
" # Q-learning update rule\n",
" q_table[state_discretized + (action,)] = q_table[state_discretized + (action,)] + \\\n",
" alpha * (reward + gamma * np.max(q_table[next_state_discretized]) - q_table[state_discretized + (action,)])\n",
" \n",
" total_reward += reward\n",
" state = next_state\n",
" \n",
" if terminated or truncated:\n",
" done = True\n",
" \n",
" rewards.append(total_reward)\n",
" \n",
" if episode % 50 == 0:\n",
" print(f\"Episode {episode}/{n_episodes}, Total Reward: {total_reward}\")\n",
"\n",
"# Plot the reward curve over episodes\n",
"plt.plot(rewards)\n",
"plt.xlabel('Episode')\n",
"plt.ylabel('Total Reward')\n",
"plt.title('Total Rewards Over Training Episodes')\n",
"plt.savefig(\"training_rewards.png\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- **Q-learning update**: After each action, we update the Q-value for the state-action pair using the Q-learning rule.\n",
"- **ε-greedy policy**: The agent explores the environment by taking random actions with probability ε and exploits the best-known action with probability 1-ε.\n",
"- **Reward visualization**: After training, we plot the total reward achieved by the agent in each episode."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"#### When does an episode end?\n",
"\n",
"In our \"CartPole-v1\" environment, an **episode** ends when one of the following conditions is met:\n",
"\n",
"1. **Pole Angle Exceeds Limit**: \n",
" The pole's angle exceeds a threshold (±12 degrees from vertical). This threshold is measured in radians internally.\n",
"\n",
"2. **Cart Position Exceeds Bounds**: \n",
" The cart moves too far to the left or right from the center. Specifically, the cart's position exceeds ±2.4 units from the center of the track.\n",
"\n",
"3. **Maximum Episode Steps**: \n",
" The environment has a maximum step limit (500 steps for \"CartPole-v1\"). If this limit is reached without any of the above failures, the episode ends successfully.\n",
"\n",
"**Rewards**: \n",
" For every step the pole remains upright, the agent receives a reward of +1. \n",
" Therefore, the total reward in an episode reflects how long the pole was balanced.\n",
"\n",
"**Done Flag**: \n",
" When the episode ends, the environment returns `done=True`. This indicates that the episode has concluded, and the agent should reset the environment before continuing."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"### **Step 6: Visualize the Trained Agent and Compare with an Untrained Agent**\n",
"\n",
"To see how the agent performs, you can run the trained agent in the environment for a few episodes and visualize its behavior."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAF7CAYAAAD4/3BBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAArmUlEQVR4nO3de3CUdZ7v8U/n0p0QkjYX0p2WwGQG0MEAZydxkBxH7kFqEFGrYMYtD9RQlhegTAGlA/4hs2URdUpZ97jD7s5YoIxurC2N45bIEheIw8lhhQjHgA6DI2iQND1g0p2EpDuX3/nDtWeaW6dDJ/10eL+qnir6eb799Pf5VUh/8lxtxhgjAAAAC0lJdAMAAAAXI6AAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLSWhA+dWvfqWSkhJlZGSorKxMv//97xPZDgAAsIiEBZQ33nhDVVVVevLJJ3X48GH96Ec/0sKFC/Xll18mqiUAAGARtkQ9LHD69On6wQ9+oK1bt4bnff/739eSJUtUXV2diJYAAIBFpCXiQ0OhkBobG/Xzn/88Yn5lZaUaGhouqQ8GgwoGg+HX/f39+vrrr5Wfny+bzTbk/QIAgGtnjFF7e7s8Ho9SUq5+ECchAeXcuXPq6+uTy+WKmO9yueT1ei+pr66u1i9+8Yvhag8AAAyh5uZmjR079qo1CQko37p474cx5rJ7RDZs2KC1a9eGX/v9fo0bN07Nzc3KyckZ8j4BAMC1CwQCKi4uVnZ2dtTahASUgoICpaamXrK3xOfzXbJXRZIcDoccDscl83NycggoAAAkmYGcnpGQq3jsdrvKyspUV1cXMb+urk4VFRWJaAkAAFhIwg7xrF27Vg888IDKy8s1Y8YM/cu//Iu+/PJLPfzww4lqCQAAWETCAsqyZct0/vx5/d3f/Z1aWlpUWlqqnTt3avz48YlqCQAAWETC7oNyLQKBgJxOp/x+P+egAACQJGL5/uZZPAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHLiHlA2bdokm80WMbnd7vByY4w2bdokj8ejzMxMzZo1S8eOHYt3GwAAIIkNyR6UW265RS0tLeGpqakpvOy5557TCy+8oJdeekkHDx6U2+3W/Pnz1d7ePhStAACAJDQkASUtLU1utzs8jRkzRtI3e0/+/u//Xk8++aTuvfdelZaW6pVXXtGFCxf0+uuvD0UrAAAgCQ1JQDlx4oQ8Ho9KSkr0k5/8RJ9//rkk6eTJk/J6vaqsrAzXOhwOzZw5Uw0NDVdcXzAYVCAQiJgAAMDIFfeAMn36dL366qv6j//4D/3617+W1+tVRUWFzp8/L6/XK0lyuVwR73G5XOFll1NdXS2n0xmeiouL4902AACwkLgHlIULF+q+++7TlClTNG/ePL377ruSpFdeeSVcY7PZIt5jjLlk3l/bsGGD/H5/eGpubo532wAAwEKG/DLjrKwsTZkyRSdOnAhfzXPx3hKfz3fJXpW/5nA4lJOTEzEBAICRa8gDSjAY1KeffqqioiKVlJTI7Xarrq4uvDwUCqm+vl4VFRVD3QoAAEgSafFe4fr163XXXXdp3Lhx8vl8evrppxUIBLR8+XLZbDZVVVVp8+bNmjhxoiZOnKjNmzdr1KhRuv/+++PdCgAASFJxDyinT5/WT3/6U507d05jxozRbbfdpgMHDmj8+PGSpMcff1xdXV169NFH1draqunTp2v37t3Kzs6OdysAACBJ2YwxJtFNxCoQCMjpdMrv93M+CgAASSKW72+exQMAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACwn5oDywQcf6K677pLH45HNZtPbb78dsdwYo02bNsnj8SgzM1OzZs3SsWPHImqCwaDWrFmjgoICZWVlafHixTp9+vQ1bQgAABg5Yg4onZ2dmjZtml566aXLLn/uuef0wgsv6KWXXtLBgwfldrs1f/58tbe3h2uqqqpUW1urmpoa7d+/Xx0dHVq0aJH6+voGvyUAAGDEsBljzKDfbLOptrZWS5YskfTN3hOPx6Oqqio98cQTkr7ZW+JyufTss8/qoYcekt/v15gxY7Rjxw4tW7ZMknTmzBkVFxdr586dWrBgQdTPDQQCcjqd8vv9ysnJGWz7AABgGMXy/R3Xc1BOnjwpr9erysrK8DyHw6GZM2eqoaFBktTY2Kienp6IGo/Ho9LS0nDNxYLBoAKBQMQEAABGrrgGFK/XK0lyuVwR810uV3iZ1+uV3W5Xbm7uFWsuVl1dLafTGZ6Ki4vj2TYAALCYIbmKx2azRbw2xlwy72JXq9mwYYP8fn94am5ujluvAADAeuIaUNxutyRdsifE5/OF96q43W6FQiG1trZeseZiDodDOTk5ERMAABi54hpQSkpK5Ha7VVdXF54XCoVUX1+viooKSVJZWZnS09MjalpaWnT06NFwDQAAuL6lxfqGjo4OffbZZ+HXJ0+e1JEjR5SXl6dx48apqqpKmzdv1sSJEzVx4kRt3rxZo0aN0v333y9JcjqdWrlypdatW6f8/Hzl5eVp/fr1mjJliubNmxe/LQMAAEkr5oBy6NAhzZ49O/x67dq1kqTly5dr+/btevzxx9XV1aVHH31Ura2tmj59unbv3q3s7Ozwe7Zs2aK0tDQtXbpUXV1dmjt3rrZv367U1NQ4bBIAAEh213QflEThPigAACSfhN0HBQAAIB4IKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHJiDigffPCB7rrrLnk8HtlsNr399tsRy1esWCGbzRYx3XbbbRE1wWBQa9asUUFBgbKysrR48WKdPn36mjYEAACMHDEHlM7OTk2bNk0vvfTSFWvuvPNOtbS0hKedO3dGLK+qqlJtba1qamq0f/9+dXR0aNGiRerr64t9CwAAwIiTFusbFi5cqIULF161xuFwyO12X3aZ3+/Xyy+/rB07dmjevHmSpN/+9rcqLi7W+++/rwULFsTaEgAAGGGG5ByUffv2qbCwUJMmTdKDDz4on88XXtbY2Kienh5VVlaG53k8HpWWlqqhoeGy6wsGgwoEAhETAAAYueIeUBYuXKjXXntNe/bs0fPPP6+DBw9qzpw5CgaDkiSv1yu73a7c3NyI97lcLnm93suus7q6Wk6nMzwVFxfHu20AAGAhMR/iiWbZsmXhf5eWlqq8vFzjx4/Xu+++q3vvvfeK7zPGyGazXXbZhg0btHbt2vDrQCBASAEAYAQb8suMi4qKNH78eJ04cUKS5Ha7FQqF1NraGlHn8/nkcrkuuw6Hw6GcnJyICQAAjFxDHlDOnz+v5uZmFRUVSZLKysqUnp6uurq6cE1LS4uOHj2qioqKoW4HAAAkgZgP8XR0dOizzz4Lvz558qSOHDmivLw85eXladOmTbrvvvtUVFSkU6dOaePGjSooKNA999wjSXI6nVq5cqXWrVun/Px85eXlaf369ZoyZUr4qh4AAHB9izmgHDp0SLNnzw6//vbckOXLl2vr1q1qamrSq6++qra2NhUVFWn27Nl64403lJ2dHX7Pli1blJaWpqVLl6qrq0tz587V9u3blZqaGodNAgAAyc5mjDGJbiJWgUBATqdTfr+f81EAAEgSsXx/8yweAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOTE/LBBAfJn+vm/+YbNJsslmsyW0HwCwAgIKkECmv0+f792mYODPyv3O3yhn7M1KH3WDUu2ZSkmzE1YAXLcIKEACBdvPKdTRqk7fSXX6TsrWmK5RBeOU7Z6gzPwblXmDR47sfKVlZCW6VQAYVgQUIIEunGtWd1tL+LXp61Hn2T+p8+yfJEmjCsYpM7dIBTf9T+XceHOi2gSAYcdJskAChS741dvdccXlF859qfMn/ksdvpPD2BUAJB4BBUiQvlC3Qu3noxfabLLZ+K8K4PrCbz0gQXq629X55y+i1mUXTVL+xOnD0BEAWAcBBUiQns42dXhPRK1Lc2TJnnXD0DcEABZCQAESwJh+9VwIJLoNALAsAgqQAKa/Tx1/jn7ia1rGaBXcfPswdAQA1kJAARKgvzckX9N/Rq1LTc9QztjvD0NHAGAtBBRgmBlj1N8b+sst7q/GZlNKSurQNwUAFkNAARKg3fv5gOqyPdycDcD1iYACJIDvaPTDO7KlyFP246FvBgAsiIACJEDX+dMDqksf5RziTgDAmggowDDrbjsrY0zUulR7pniWMYDrFQEFGGZtX34s098bta5k5v+SuMU9gOsUv/2AYdb6p0MDuoInI9czDN0AgDURUIBh1Bu8MLDLiyXZUlJks3GQB8D1KaaAUl1drVtvvVXZ2dkqLCzUkiVLdPz48YgaY4w2bdokj8ejzMxMzZo1S8eOHYuoCQaDWrNmjQoKCpSVlaXFixfr9OmBnTQIJLOu1hb1hrqi1hXeMkvpGaOHoSMAsKaYAkp9fb1WrVqlAwcOqK6uTr29vaqsrFRnZ2e45rnnntMLL7ygl156SQcPHpTb7db8+fPV3t4erqmqqlJtba1qamq0f/9+dXR0aNGiRerrG9hflkCyOn/igELt56LWjXZPVEp6xjB0BADWZDMDuZzgCv785z+rsLBQ9fX1uuOOO2SMkcfjUVVVlZ544glJ3+wtcblcevbZZ/XQQw/J7/drzJgx2rFjh5YtWyZJOnPmjIqLi7Vz504tWLAg6ucGAgE5nU75/X7l5OQMtn1gWJn+Pp36/W917g//J2rthMpHlFvyN8PQFQAMn1i+v6/pHBS/3y9JysvLkySdPHlSXq9XlZWV4RqHw6GZM2eqoaFBktTY2Kienp6IGo/Ho9LS0nDNxYLBoAKBQMQEJJuervYBPcE4I9cj++jcYegIAKxr0AHFGKO1a9fq9ttvV2lpqSTJ6/VKklwuV0Sty+UKL/N6vbLb7crNzb1izcWqq6vldDrDU3Fx8WDbBhKm03dSHWej3+I+x3MTV/AAuO4NOqCsXr1aH3/8sf71X//1kmUXX3lgjIl6NcLVajZs2CC/3x+empubB9s2kBDGGPUGO9UX7Ixam2rPUEpq+jB0BQDWNaiAsmbNGr3zzjvau3evxo4dG57vdrsl6ZI9IT6fL7xXxe12KxQKqbW19Yo1F3M4HMrJyYmYgGRi+nrV09kWtc6WmvbNHWS5vBjAdS6mgGKM0erVq/XWW29pz549KikpiVheUlIit9uturq68LxQKKT6+npVVFRIksrKypSenh5R09LSoqNHj4ZrgJGmp7tdga/+ELVuVH6x8if8cBg6AgBrS4uleNWqVXr99df1u9/9TtnZ2eE9JU6nU5mZ3/zVV1VVpc2bN2vixImaOHGiNm/erFGjRun+++8P165cuVLr1q1Tfn6+8vLytH79ek2ZMkXz5s2L/xYCFtDb3aH2lj9GrUtNz1D6qBuGviEAsLiYAsrWrVslSbNmzYqYv23bNq1YsUKS9Pjjj6urq0uPPvqoWltbNX36dO3evVvZ2dnh+i1btigtLU1Lly5VV1eX5s6dq+3btys1NfXatgawIGOM+rovDKzYZpMthRs8A8A13QclUbgPCpKJ6e9Ty+H39NWhd65al2ofpe9VPiznjTcPU2cAMLyG7T4oAKLr7+uNGk4kKSU1TaMLvzsMHQGA9RFQgCFm+noGVmizKTXdPrTNAECSIKAAQ+zC+YE9CDOPq3cAIIyAAgyx5v96a0B1hZNnDnEnAJA8CCjAEDLGKNR+fkC19qwbhrYZAEgiBBRgCPVcaJMx/VHr7NkFEnePBYAwAgowhM6f+FD9vaGodTfeerdsKdwHCAC+RUABhpC/+eiAruLJvMEtiT0oAPAtAgowRPp7e2T6ox/ekb55SCAPCASAvyCgAEOkq/WMers7otYV3jJb9qy8YegIAJIHAQUYIl9/9qG621qi1mXmeZSa7hiGjgAgeRBQgCFgTL8G+pirlDQHDwgEgIvwWxEYAn3BLvV0tUety8wvVmauexg6AoDkQkABhkC79zO1nToStS4rv1gZNxBQAOBiBBQgzowx6u8Nqb83GLU21Z6hlDTOPwGAixFQgHgz/eoLdkYtS0lzKH10HpcXA8BlEFCAOOvp7tD5zz6MWufIKVBeyQ+GoSMASD4EFCDO+kNd6vB+FrUuJd0h++jcYegIAJIPAQWII2PMgO8em5KSxvN3AOAKCChAXBm1ffFx1KqUNIc85YuHoR8ASE4EFCCOTH+/Th+sjVpnS0lRVkHxMHQEAMmJgAIkhE2p9sxENwEAlkVAAeKou80rDeAO94W3zBryXgAgmRFQgDgKfPWpBpJQ8r5XPvTNAEASI6AAcXS26T8HVGfPzh/iTgAguRFQgDjpDV6QMdEvMR6VXyybjf96AHA1/JYE4uTCuS/U39sTta7wlllKSU0bho4AIHkRUIA4Of/H/1JfqCtqXcYNbok9KABwVfyWBOLAmH71dLdLUQ7x2FJSlZKazgMCASAKAgoQB8H28+rt7ohalz+pQo6cMcPQEQAkNwIKEAedvlMKtp+PWpdxg0up9oxh6AgAkltMAaW6ulq33nqrsrOzVVhYqCVLluj48eMRNStWrJDNZouYbrvttoiaYDCoNWvWqKCgQFlZWVq8eLFOnz597VsDJEh3m1e9XYGodanpDh4QCAADEFNAqa+v16pVq3TgwAHV1dWpt7dXlZWV6uzsjKi788471dLSEp527twZsbyqqkq1tbWqqanR/v371dHRoUWLFqmvr+/atwgYZn093QM6vDMqv1hZBeOHoSMASH4xXeu4a9euiNfbtm1TYWGhGhsbdccdd4TnOxwOud3uy67D7/fr5Zdf1o4dOzRv3jxJ0m9/+1sVFxfr/fff14IFC2LdBiChei741dV6JmqdI2eMHDe4hqEjAEh+13QOit/vlyTl5eVFzN+3b58KCws1adIkPfjgg/L5fOFljY2N6unpUWVlZXiex+NRaWmpGhoaLvs5wWBQgUAgYgKsorvtrNrPHI9al2rPVBoPCASAARl0QDHGaO3atbr99ttVWloanr9w4UK99tpr2rNnj55//nkdPHhQc+bMUTAYlCR5vV7Z7Xbl5uZGrM/lcsnr9V72s6qrq+V0OsNTcTGPqYc1mP7+Ad37JCU9Q5l5nmHoCABGhkHfznL16tX6+OOPtX///oj5y5YtC/+7tLRU5eXlGj9+vN59913de++9V1yfMeaK94bYsGGD1q5dG34dCAQIKbCE/r5eXTgf/QTv9FE5co6bOgwdAcDIMKg9KGvWrNE777yjvXv3auzYsVetLSoq0vjx43XixAlJktvtVigUUmtra0Sdz+eTy3X54/MOh0M5OTkRE2AFfaEL+vMn9VHrUtIcysgpGIaOAGBkiCmgGGO0evVqvfXWW9qzZ49KSkqivuf8+fNqbm5WUVGRJKmsrEzp6emqq6sL17S0tOjo0aOqqKiIsX0gcYwxMn096uvpjlqb5hjF5cUAEIOYDvGsWrVKr7/+un73u98pOzs7fM6I0+lUZmamOjo6tGnTJt13330qKirSqVOntHHjRhUUFOiee+4J165cuVLr1q1Tfn6+8vLytH79ek2ZMiV8VQ+QLDoHcHjHlpKmor9ZOAzdAMDIEVNA2bp1qyRp1qxZEfO3bdumFStWKDU1VU1NTXr11VfV1tamoqIizZ49W2+88Yays7PD9Vu2bFFaWpqWLl2qrq4uzZ07V9u3b1dqKn9hIrkM5PCOzWZT1hjufwIAsbAZY0yim4hVIBCQ0+mU3+/nfBQkjDH9avzNKpn+q99gMCXNrh/87H/zgEAA171Yvr95Fg8wSKGO1uhFktzTuPkgAMSKgAIMUtuXH2sgOyCdxaVRawAAkQgowCD5ju6VTH/UOkdOPod3ACBGBBRgEPp6umUGEE5GuyfIljLo+yECwHWLgAIMQlerV/29oah1uSVlSkmzD0NHADCyEFCAQfAd26ueTn/Uusw8DzdoA4BBIKAAMTL9/f+99+TqJ8im2jOVkmbn/BMAGAQCChCj3u529QUvRK27YfxUZTjHDENHADDyEFCAGLW3nNCFr6Pf4t7hLFSqY9QwdAQAIw8BBYiBMUahjq/V29UetTYlzaEUruABgEEhoAAxMP296gt1Ra3LzBur7KJJw9ARAIxMBBQgBj0XAurwnYxaZ8+6gfNPAOAaEFCAGATbzytw+pOodSnpGUpzZA1DRwAwMhFQgAEyxqi/Nxi1LiU9Q9nu7w1DRwAwchFQgAEy/X3q+vqrqHWp9kzljJ08DB0BwMhFQAEGqC/UpdMfvh21LiUtXRk3uIe+IQAYwQgowAAYY755OOBAnl48mqcXA8C1IqAAA9Tt90UvsqWosHTu0DcDACMcAQUYoK8GcHjHZrNxgiwAxAG3ucSIZoxRX19fXNZ14dyXUWv6jZFJTVdvb2/M609NTeXQEAD8NwIKRrSzZ8+quLj4mtczOtOu3z29VJmO9KvW/fqdD/XK6iyZqz/o+BI2m02BQEAZGRnX0CUAjBwEFIx4g9mbcbE5/2Oi0lKjHxH96I9e9fTE/nnsOQGASAQUYACW/Ohmpaam60zwO7rQ55RklJXql9t+Un+dLc6cj/4QQQBAdAQUIIqU/w4gH7XPl79njHpMhiQje0q3fKHvaFr2XknSh5+e1oVgT+IaBYARhIACRFHsytUnwbuVGRon6S+7S4L9WToT/OaKnSmj9+nDP3ylLgIKAMQFAQWIYur0R5Rxw2T9dTj5ixSdCU7QqJSAPj+zSz290W/kBgCIjvugAFHYbLYoJ7HadL69S53doWHrCQBGOgIKcBVZGenKyrBHrft/n3nV7AsMQ0cAcH0goABX8cPv36gfft8Tta7ZF1Bre9cwdAQA1wcCCnAVzqwMTS/8SDeknZV0ubuvGRWknZIn9ZD6+mO8OxsA4IpiCihbt27V1KlTlZOTo5ycHM2YMUPvvfdeeLkxRps2bZLH41FmZqZmzZqlY8eORawjGAxqzZo1KigoUFZWlhYvXqzTp0/HZ2uAOGvr6Napr3wa17tD6v5CKaZLUr+kfqXbujUmvVn5F97QoT98kehWAWBEiekqnrFjx+qZZ57RhAkTJEmvvPKK7r77bh0+fFi33HKLnnvuOb3wwgvavn27Jk2apKefflrz58/X8ePHlZ2dLUmqqqrSv//7v6umpkb5+flat26dFi1apMbGRqWmpsZ/C4FrsO/IKX30xzMqLnSq2HVEo13/U6kOt/KdmfpOXpcmFJ7Wn9q79KXPn+hWAWBEsRkT61NDIuXl5emXv/ylfvazn8nj8aiqqkpPPPGEpG/2lrhcLj377LN66KGH5Pf7NWbMGO3YsUPLli2TJJ05c0bFxcXauXOnFixYMKDPDAQCcjqdWrFihez26Ccw4vrV1dWlHTt2xHWdjvRU5edkKt85SgXOUeoO9er/Hrv2vYArV64kpAMY0UKhkLZv3y6/36+cnJyr1g76Pih9fX36t3/7N3V2dmrGjBk6efKkvF6vKisrwzUOh0MzZ85UQ0ODHnroITU2NqqnpyeixuPxqLS0VA0NDVcMKMFgUMFgMPw6EPjmaokHHnhAo0ePHuwm4Drw9ddfxz2gBHv6dOZ8h86c74jregncAEa6jo4Obd++fUC1MQeUpqYmzZgxQ93d3Ro9erRqa2s1efJkNTQ0SJJcLldEvcvl0hdffHN83uv1ym63Kzc395Iar9d7xc+srq7WL37xi0vml5eXR01guL5d7efKSmw2m8rLy3maMYAR7dsdDAMR81U8N910k44cOaIDBw7okUce0fLly/XJJ5+El198QytjTNQntUar2bBhg/x+f3hqbm6OtW0AAJBEYg4odrtdEyZMUHl5uaqrqzVt2jS9+OKLcrvdki79i9Xn84X3qrjdboVCIbW2tl6x5nIcDkf4yqFvJwAAMHJd831QjDEKBoMqKSmR2+1WXV1deFkoFFJ9fb0qKiokSWVlZUpPT4+oaWlp0dGjR8M1AAAAMZ2DsnHjRi1cuFDFxcVqb29XTU2N9u3bp127dslms6mqqkqbN2/WxIkTNXHiRG3evFmjRo3S/fffL0lyOp1auXKl1q1bp/z8fOXl5Wn9+vWaMmWK5s2bNyQbCAAAkk9MAeXs2bN64IEH1NLSIqfTqalTp2rXrl2aP3++JOnxxx9XV1eXHn30UbW2tmr69OnavXt3+B4okrRlyxalpaVp6dKl6urq0ty5c7V9+3YurwQAAGHXfB+URPj2PigDuY4a1zev16uioqJEtxGVzWbThQsXuIoHwIgWy/c3z+IBAACWQ0ABAACWQ0ABAACWQ0ABAACWM+hn8QDJwOFwaMmSJYluIyqbzaaUFP5eAIBvEVAwouXm5qq2tjbRbQAAYsSfbAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHJiCihbt27V1KlTlZOTo5ycHM2YMUPvvfdeePmKFStks9kipttuuy1iHcFgUGvWrFFBQYGysrK0ePFinT59Oj5bAwAARoSYAsrYsWP1zDPP6NChQzp06JDmzJmju+++W8eOHQvX3HnnnWppaQlPO3fujFhHVVWVamtrVVNTo/3796ujo0OLFi1SX19ffLYIAAAkPZsxxlzLCvLy8vTLX/5SK1eu1IoVK9TW1qa33377srV+v19jxozRjh07tGzZMknSmTNnVFxcrJ07d2rBggUD+sxAICCn0ym/36+cnJxraR8AAAyTWL6/B30OSl9fn2pqatTZ2akZM2aE5+/bt0+FhYWaNGmSHnzwQfl8vvCyxsZG9fT0qLKyMjzP4/GotLRUDQ0NV/ysYDCoQCAQMQEAgJEr5oDS1NSk0aNHy+Fw6OGHH1Ztba0mT54sSVq4cKFee+017dmzR88//7wOHjyoOXPmKBgMSpK8Xq/sdrtyc3Mj1ulyueT1eq/4mdXV1XI6neGpuLg41rYBAEASSYv1DTfddJOOHDmitrY2vfnmm1q+fLnq6+s1efLk8GEbSSotLVV5ebnGjx+vd999V/fee+8V12mMkc1mu+LyDRs2aO3ateHXgUCAkAIAwAgWc0Cx2+2aMGGCJKm8vFwHDx7Uiy++qH/+53++pLaoqEjjx4/XiRMnJElut1uhUEitra0Re1F8Pp8qKiqu+JkOh0MOhyPWVgEAQJK65vugGGPCh3Audv78eTU3N6uoqEiSVFZWpvT0dNXV1YVrWlpadPTo0asGFAAAcH2JaQ/Kxo0btXDhQhUXF6u9vV01NTXat2+fdu3apY6ODm3atEn33XefioqKdOrUKW3cuFEFBQW65557JElOp1MrV67UunXrlJ+fr7y8PK1fv15TpkzRvHnzhmQDAQBA8okpoJw9e1YPPPCAWlpa5HQ6NXXqVO3atUvz589XV1eXmpqa9Oqrr6qtrU1FRUWaPXu23njjDWVnZ4fXsWXLFqWlpWnp0qXq6urS3LlztX37dqWmpsZ94wAAQHK65vugJAL3QQEAIPkMy31QAAAAhgoBBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWE5aohsYDGOMJCkQCCS4EwAAMFDffm9/+z1+NUkZUNrb2yVJxcXFCe4EAADEqr29XU6n86o1NjOQGGMx/f39On78uCZPnqzm5mbl5OQkuqWkFQgEVFxczDjGAWMZP4xlfDCO8cNYxocxRu3t7fJ4PEpJufpZJkm5ByUlJUU33nijJCknJ4cfljhgHOOHsYwfxjI+GMf4YSyvXbQ9J9/iJFkAAGA5BBQAAGA5SRtQHA6HnnrqKTkcjkS3ktQYx/hhLOOHsYwPxjF+GMvhl5QnyQIAgJEtafegAACAkYuAAgAALIeAAgAALIeAAgAALCcpA8qvfvUrlZSUKCMjQ2VlZfr973+f6JYs54MPPtBdd90lj8cjm82mt99+O2K5MUabNm2Sx+NRZmamZs2apWPHjkXUBINBrVmzRgUFBcrKytLixYt1+vTpYdyKxKuurtatt96q7OxsFRYWasmSJTp+/HhEDWM5MFu3btXUqVPDN7qaMWOG3nvvvfByxnFwqqurZbPZVFVVFZ7HWA7Mpk2bZLPZIia32x1ezjgmmEkyNTU1Jj093fz61782n3zyiXnsscdMVlaW+eKLLxLdmqXs3LnTPPnkk+bNN980kkxtbW3E8meeecZkZ2ebN9980zQ1NZlly5aZoqIiEwgEwjUPP/ywufHGG01dXZ356KOPzOzZs820adNMb2/vMG9N4ixYsMBs27bNHD161Bw5csT8+Mc/NuPGjTMdHR3hGsZyYN555x3z7rvvmuPHj5vjx4+bjRs3mvT0dHP06FFjDOM4GB9++KH5zne+Y6ZOnWoee+yx8HzGcmCeeuopc8stt5iWlpbw5PP5wssZx8RKuoDywx/+0Dz88MMR826++Wbz85//PEEdWd/FAaW/v9+43W7zzDPPhOd1d3cbp9Np/umf/skYY0xbW5tJT083NTU14ZqvvvrKpKSkmF27dg1b71bj8/mMJFNfX2+MYSyvVW5urvnNb37DOA5Ce3u7mThxoqmrqzMzZ84MBxTGcuCeeuopM23atMsuYxwTL6kO8YRCITU2NqqysjJifmVlpRoaGhLUVfI5efKkvF5vxDg6HA7NnDkzPI6NjY3q6emJqPF4PCotLb2ux9rv90uS8vLyJDGWg9XX16eamhp1dnZqxowZjOMgrFq1Sj/+8Y81b968iPmMZWxOnDghj8ejkpIS/eQnP9Hnn38uiXG0gqR6WOC5c+fU19cnl8sVMd/lcsnr9Saoq+Tz7Vhdbhy/+OKLcI3dbldubu4lNdfrWBtjtHbtWt1+++0qLS2VxFjGqqmpSTNmzFB3d7dGjx6t2tpaTZ48OfzLnHEcmJqaGn300Uc6ePDgJcv4mRy46dOn69VXX9WkSZN09uxZPf3006qoqNCxY8cYRwtIqoDyLZvNFvHaGHPJPEQ3mHG8nsd69erV+vjjj7V///5LljGWA3PTTTfpyJEjamtr05tvvqnly5ervr4+vJxxjK65uVmPPfaYdu/erYyMjCvWMZbRLVy4MPzvKVOmaMaMGfre976nV155RbfddpskxjGRkuoQT0FBgVJTUy9Jpj6f75KUiyv79iz1q42j2+1WKBRSa2vrFWuuJ2vWrNE777yjvXv3auzYseH5jGVs7Ha7JkyYoPLyclVXV2vatGl68cUXGccYNDY2yufzqaysTGlpaUpLS1N9fb3+4R/+QWlpaeGxYCxjl5WVpSlTpujEiRP8TFpAUgUUu92usrIy1dXVRcyvq6tTRUVFgrpKPiUlJXK73RHjGAqFVF9fHx7HsrIypaenR9S0tLTo6NGj19VYG2O0evVqvfXWW9qzZ49KSkoiljOW18YYo2AwyDjGYO7cuWpqatKRI0fCU3l5uf72b/9WR44c0Xe/+13GcpCCwaA+/fRTFRUV8TNpBYk4M/dafHuZ8csvv2w++eQTU1VVZbKyssypU6cS3ZqltLe3m8OHD5vDhw8bSeaFF14whw8fDl+O/cwzzxin02neeust09TUZH76059e9vK5sWPHmvfff9989NFHZs6cOdfd5XOPPPKIcTqdZt++fRGXIl64cCFcw1gOzIYNG8wHH3xgTp48aT7++GOzceNGk5KSYnbv3m2MYRyvxV9fxWMMYzlQ69atM/v27TOff/65OXDggFm0aJHJzs4Of58wjomVdAHFGGP+8R//0YwfP97Y7Xbzgx/8IHzJJ/5i7969RtIl0/Lly40x31xC99RTTxm3220cDoe54447TFNTU8Q6urq6zOrVq01eXp7JzMw0ixYtMl9++WUCtiZxLjeGksy2bdvCNYzlwPzsZz8L/78dM2aMmTt3bjicGMM4XouLAwpjOTDf3tckPT3deDwec++995pjx46FlzOOiWUzxpjE7LsBAAC4vKQ6BwUAAFwfCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMBy/j/cZ4Jc+5x79gAAAABJRU5ErkJggg==",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"env = gym.make(\"CartPole-v1\", render_mode='rgb_array')\n",
"# Test the trained agent\n",
"frames=[]\n",
"for episode in range(10):\n",
" state, _ = env.reset()\n",
" done = False\n",
" while not done:\n",
" # Discretize the state\n",
" state_discretized = discretize_state(state)\n",
" \n",
" # Choose the action with the highest Q-value (exploitation)\n",
" action = np.argmax(q_table[state_discretized])\n",
" \n",
" # Step in the environment\n",
" state, reward, terminated, truncated, _ = env.step(action)\n",
" \n",
" # Render the environment (to visualize the agent)\n",
" im = env.render()\n",
" frames.append(im)\n",
" \n",
" if terminated or truncated:\n",
" done = True\n",
"\n",
"fig = plt.figure()\n",
"img = plt.imshow(frames[0])\n",
"\n",
"def update(frame):\n",
" img.set_data(frames[frame])\n",
" return img\n",
"\n",
"ani = FuncAnimation(fig, update, frames=len(frames), interval=50)\n",
"ani.save(\"CartPoleTrained.mp4\", fps=10, writer='ffmpeg')\n",
"plt.show()\n",
"\n",
"env.close()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"- **Visualizing agent behavior**: After training, the agent will follow the learned policy and you will see it balancing the pole using the **CartPole-v1** environment.\n",
"- **`env.render()`**: This will render the environment so you can see the agent's movements during each step.\n"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAF7CAYAAAD4/3BBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAArm0lEQVR4nO3dfXBUdZ7v8c/pPDQhJG0eoDstIRNHcAYT2DU4kFxHnoOsyPhQFxy3vFBDWToKZRYoHfAPmS0vUaeEcdcddnfWEkXdeLcUx1mQIS4Sh0qxg1HGgC6LI2iQNBFMupMQupPO7/7h2mPLQ+ex+3R4v6pOFX3Ot7u/51ck/cnpc37HMsYYAQAA2Igj0Q0AAAB8GwEFAADYDgEFAADYDgEFAADYDgEFAADYDgEFAADYDgEFAADYDgEFAADYDgEFAADYDgEFAADYTkIDyq9+9SsVFxdr1KhRKisr0+9///tEtgMAAGwiYQHllVdeUVVVlR555BG9//77+uEPf6iFCxfqs88+S1RLAADAJqxE3Sxw+vTpuu6667Rly5bIuu9///u69dZbVV1dnYiWAACATaQm4k1DoZAaGhr0s5/9LGp9ZWWl6uvrz6sPBoMKBoORx729vfryyy+Vl5cny7KGvV8AADB4xhi1t7fL6/XK4bj0lzgJCSinT59WOByW2+2OWu92u+Xz+c6rr66u1s9//vN4tQcAAIZRU1OTxo8ff8mahASUr3376Icx5oJHRNatW6fVq1dHHvv9fk2YMEFNTU3Kzs4e9j4BAMDgBQIBFRYWKisrK2ZtQgJKfn6+UlJSzjta0tLSct5RFUlyOp1yOp3nrc/OziagAACQZPpyekZCruJJT09XWVmZamtro9bX1taqoqIiES0BAAAbSdhXPKtXr9bdd9+tadOmqby8XP/8z/+szz77TPfdd1+iWgIAADaRsICydOlSnTlzRn/7t3+r5uZmlZSUaOfOnSoqKkpUSwAAwCYSNg/KYAQCAblcLvn9fs5BAQAgSfTn85t78QAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANsZ8oCyYcMGWZYVtXg8nsh2Y4w2bNggr9erjIwMzZo1S4cPHx7qNgAAQBIbliMo1157rZqbmyNLY2NjZNuTTz6pTZs26ZlnntGBAwfk8Xg0f/58tbe3D0crAAAgCQ1LQElNTZXH44ksY8eOlfTV0ZNf/vKXeuSRR3T77berpKREzz//vM6ePauXX355OFoBAABJaFgCytGjR+X1elVcXKw777xTn3zyiSTp2LFj8vl8qqysjNQ6nU7NnDlT9fX1F329YDCoQCAQtQAAgJFryAPK9OnT9cILL+h3v/udfv3rX8vn86miokJnzpyRz+eTJLnd7qjnuN3uyLYLqa6ulsvliiyFhYVD3TYAALCRIQ8oCxcu1B133KHS0lLNmzdPO3bskCQ9//zzkRrLsqKeY4w5b903rVu3Tn6/P7I0NTUNddsAAMBGhv0y48zMTJWWluro0aORq3m+fbSkpaXlvKMq3+R0OpWdnR21AACAkWvYA0owGNRHH32kgoICFRcXy+PxqLa2NrI9FAqprq5OFRUVw90KAABIEqlD/YJr167VLbfcogkTJqilpUWPPfaYAoGAli1bJsuyVFVVpY0bN2rixImaOHGiNm7cqNGjR+uuu+4a6lYAAECSGvKAcuLECf34xz/W6dOnNXbsWM2YMUP79+9XUVGRJOmhhx5SV1eX7r//frW2tmr69OnavXu3srKyhroVAACQpCxjjEl0E/0VCATkcrnk9/s5HwUAgCTRn89v7sUDAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsh4ACAABsp98B5Z133tEtt9wir9cry7L0+uuvR203xmjDhg3yer3KyMjQrFmzdPjw4aiaYDCoVatWKT8/X5mZmVq8eLFOnDgxqB0BAAAjR78DSmdnp6ZOnapnnnnmgtuffPJJbdq0Sc8884wOHDggj8ej+fPnq729PVJTVVWl7du3q6amRvv27VNHR4cWLVqkcDg88D0BAAAjhmWMMQN+smVp+/btuvXWWyV9dfTE6/WqqqpKDz/8sKSvjpa43W498cQTuvfee+X3+zV27Fht27ZNS5culSSdPHlShYWF2rlzpxYsWBDzfQOBgFwul/x+v7KzswfaPgAAiKP+fH4P6Tkox44dk8/nU2VlZWSd0+nUzJkzVV9fL0lqaGhQd3d3VI3X61VJSUmk5tuCwaACgUDUAgAARq4hDSg+n0+S5Ha7o9a73e7INp/Pp/T0dOXk5Fy05tuqq6vlcrkiS2Fh4VC2DQAAbGZYruKxLCvqsTHmvHXfdqmadevWye/3R5ampqYh6xUAANjPkAYUj8cjSecdCWlpaYkcVfF4PAqFQmptbb1ozbc5nU5lZ2dHLQAAYOQa0oBSXFwsj8ej2trayLpQKKS6ujpVVFRIksrKypSWlhZV09zcrEOHDkVqAADA5S21v0/o6OjQxx9/HHl87NgxHTx4ULm5uZowYYKqqqq0ceNGTZw4URMnTtTGjRs1evRo3XXXXZIkl8ulFStWaM2aNcrLy1Nubq7Wrl2r0tJSzZs3b+j2DAAAJK1+B5R3331Xs2fPjjxevXq1JGnZsmXaunWrHnroIXV1den+++9Xa2urpk+frt27dysrKyvynM2bNys1NVVLlixRV1eX5s6dq61btyolJWUIdgkAACS7Qc2DkijMgwIAQPJJ2DwoAAAAQ4GAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbKffAeWdd97RLbfcIq/XK8uy9Prrr0dtX758uSzLilpmzJgRVRMMBrVq1Srl5+crMzNTixcv1okTJwa1IwAAYOTod0Dp7OzU1KlT9cwzz1y05qabblJzc3Nk2blzZ9T2qqoqbd++XTU1Ndq3b586Ojq0aNEihcPh/u8BAAAYcVL7+4SFCxdq4cKFl6xxOp3yeDwX3Ob3+/Xss89q27ZtmjdvniTpxRdfVGFhod566y0tWLCgvy0BAIARZljOQdm7d6/GjRunSZMm6Z577lFLS0tkW0NDg7q7u1VZWRlZ5/V6VVJSovr6+gu+XjAYVCAQiFoAAMDINeQBZeHChXrppZe0Z88ePfXUUzpw4IDmzJmjYDAoSfL5fEpPT1dOTk7U89xut3w+3wVfs7q6Wi6XK7IUFhYOddsAAMBG+v0VTyxLly6N/LukpETTpk1TUVGRduzYodtvv/2izzPGyLKsC25bt26dVq9eHXkcCAQIKQAAjGDDfplxQUGBioqKdPToUUmSx+NRKBRSa2trVF1LS4vcbvcFX8PpdCo7OztqAQAAI9ewB5QzZ86oqalJBQUFkqSysjKlpaWptrY2UtPc3KxDhw6poqJiuNsBAABJoN9f8XR0dOjjjz+OPD527JgOHjyo3Nxc5ebmasOGDbrjjjtUUFCg48ePa/369crPz9dtt90mSXK5XFqxYoXWrFmjvLw85ebmau3atSotLY1c1QMAAC5v/Q4o7777rmbPnh15/PW5IcuWLdOWLVvU2NioF154QW1tbSooKNDs2bP1yiuvKCsrK/KczZs3KzU1VUuWLFFXV5fmzp2rrVu3KiUlZQh2CQAAJDvLGGMS3UR/BQIBuVwu+f1+zkcBACBJ9Ofzm3vxAAAA2yGgAAAA2yGgAAAA2yGgAAAA2yGgAAAA2yGgAAAA2yGgAAAA2yGgAAAA2yGgAAAA2yGgAAAA2yGgAAAA2+n3zQIB4FK+ur2XkYyRMUaWZclycCNQAP1DQAEwKL3hHvWGu9XbE1JvT0ihjladPf2pOr/4VJ1fHNcY93dVPPP/EFIA9AsBBcCAnf3ycwVOfKRz/lPqaj2pc20+9XS1R9U4x+Qp1NkqZ1Z+groEkIwIKAAGrPWT93Sy4beXrDn75ec6e+YEAQVAv3CSLIBh1dMVUPdZf6LbAJBkCCgABiyn+C+VkVcYs667q129Pd1x6AjASEFAATBgo1xupWVkx6zrbDmunmBHHDoCMFIQUAAMmCM1TZYj9q8R/2cfKNTZNvwNARgxCCgABiXbe40cqekx63p7uv9njhQAiI2AAmBQXEVT5UgbFbPuXFuzZHrj0BGAkYCAAmBQRrnGypESe8aCz+r/n3p7QnHoCMBIQEABMCiW5VDa6Cti1plwt/iCB0BfEVAADJq7dI4kK2Zdzzmu5AHQNwQUAIOW5ZnYp7qO5o+HuRMAIwUBBcCgpWZk9eUAipoPvjn8zQAYEQgoAIZESh+u5Ok+G5DhSh4AfUBAATBoliNFEyqWxqwzplfBwBdx6AhAsiOgABgSGXnjY9aYcLfOHP3POHQDINkRUAAMmmVZSunDbLKmN6zA5/8Vh44AJLt+BZTq6mpdf/31ysrK0rhx43TrrbfqyJEjUTXGGG3YsEFer1cZGRmaNWuWDh8+HFUTDAa1atUq5efnKzMzU4sXL9aJEycGvzcAEiZ11BjlXDUtZp0xveoN98ShIwDJrF8Bpa6uTg888ID279+v2tpa9fT0qLKyUp2dnZGaJ598Ups2bdIzzzyjAwcOyOPxaP78+Wpvb4/UVFVVafv27aqpqdG+ffvU0dGhRYsWKRwOD92eAYgrR9oojRlXHLOu51yHus7wBwmAS7PMIO7e9cUXX2jcuHGqq6vTjTfeKGOMvF6vqqqq9PDDD0v66miJ2+3WE088oXvvvVd+v19jx47Vtm3btHTpVyfVnTx5UoWFhdq5c6cWLFgQ830DgYBcLpf8fr+ys2Pf6h1AfHz5pwP601u/jlnnLpmjCf/rzjh0BMBO+vP5PahzUPx+vyQpNzdXknTs2DH5fD5VVlZGapxOp2bOnKn6+npJUkNDg7q7u6NqvF6vSkpKIjXfFgwGFQgEohYA9pOaka30MbmJbgPACDDggGKM0erVq3XDDTeopKREkuTz+SRJbrc7qtbtdke2+Xw+paenKycn56I131ZdXS2XyxVZCgsLB9o2gGE0KnusRufF/vnsCXUpHDoXh44AJKsBB5SVK1fqgw8+0L/+67+et82yoqeUNMact+7bLlWzbt06+f3+yNLU1DTQtgEMo9TRLqVnxT6CEur4UqGz/jh0BCBZDSigrFq1Sm+88YbefvttjR//57kPPB6PJJ13JKSlpSVyVMXj8SgUCqm1tfWiNd/mdDqVnZ0dtQCwH8tyyOFIjVnXfvKIOluOxaEjAMmqXwHFGKOVK1fqtdde0549e1RcHH3GfnFxsTwej2prayPrQqGQ6urqVFFRIUkqKytTWlpaVE1zc7MOHToUqQGQnCzLUnpWnhx9mPbe9PYw7T2Ai4r9p843PPDAA3r55Zf1m9/8RllZWZEjJS6XSxkZGbIsS1VVVdq4caMmTpyoiRMnauPGjRo9erTuuuuuSO2KFSu0Zs0a5eXlKTc3V2vXrlVpaanmzZs39HsIIK6uKJqq00fqdfb0Z5esCwZOq7enWylpzjh1BiCZ9CugbNmyRZI0a9asqPXPPfecli9fLkl66KGH1NXVpfvvv1+tra2aPn26du/eraysrEj95s2blZqaqiVLlqirq0tz587V1q1blZKSMri9AZBw6ZlX9OnGgaeP1Cv/mnKluC781S6Ay9ug5kFJFOZBAeztyI5fKnDiw5h11/7vR5WR4415Ej2AkSFu86AAwIW4S+f16TyUcPBsHLoBkIwIKACG3Oj8CbIcsb+y7Tj1iZR8B3EBxAEBBcCQSx+dLcuK/evlzNH9XMkD4IIIKAASpuvLzzmCAuCCCCgAhsV3bry7T3Vdbc3D3AmAZERAATAsxhRc3ae6tuN/HOZOACQjAgqAYeFISetT3RcfvTPMnQBIRgQUAMPCcqQop/i62IXGKNwdHP6GACQVAgqAYWE5HMq9+vqYdb29YZ1r5TwUANEIKACGiSVnVn7MqnCwU583/DYO/QBIJgQUAMPCsiw5UtOUmhH7dhSmN6wkvOsGgGFEQAEwbNJGX6Gc4r+MWdfb061wqCsOHQFIFgQUAMMmJc2pjCsKYtb1BDsU9LfEoSMAyYKAAmDYWI4UOdLSY9YF/S3y9+HuxwAuHwQUAMPKkZouK8acKKY3rHD3Oe7LAyCCgAJgWGVf+T25Cq+NWdfd2aae4Nk4dAQgGRBQAAyrVOcYpY4aE7Ou49QnCvpPxaEjAMmAgAJgWFkOhywr9q+aYKBF3WcDcegIQDIgoAAYdrnfLVP6mNyYdeHuczK9nIcCgIACIA4y8gqV4hwds669+WOFe7gvDwACCoA4SBs1Rg5Hasy60//1e/Wc64hDRwDsjoACIC7GFEyU+nAuCgBIBBQAcZJz1XVypMQ+ihL0t3BfHgAEFADxMWbsd/p0BCVw8r8lEVCAyx0BBUB8WA5ZfSjz/fF3EkdQgMseAQVA3IwpmNinunA3V/IAlzsCCoC48V53cx+qjM6ebhr2XgDYGwEFQNw4s8fGLjJGJxt+O/zNALA1AgqAuLAs66tp7/twJU+os5UreYDLHAEFQNw4Up0q+Mu/illnesNM2AZc5ggoAOLGcqRodF5hzLqe4Fm1HnsvDh0BsKt+BZTq6mpdf/31ysrK0rhx43TrrbfqyJEjUTXLly//6lDuN5YZM2ZE1QSDQa1atUr5+fnKzMzU4sWLdeLEicHvDQBbsyxLqX24J09v9zkFTnwUh44A2FW/AkpdXZ0eeOAB7d+/X7W1terp6VFlZaU6Ozuj6m666SY1NzdHlp07d0Ztr6qq0vbt21VTU6N9+/apo6NDixYtUjgcHvweAbC19MwrlNXHy405DwW4fMU+W+0bdu3aFfX4ueee07hx49TQ0KAbb7wxst7pdMrj8VzwNfx+v5599llt27ZN8+bNkyS9+OKLKiws1FtvvaUFCxb0dx8AJJHUUWM0Om+C2puPXrKu51yHujvblD4mJ06dAbCTQZ2D4vf7JUm5ublR6/fu3atx48Zp0qRJuueee9TS0hLZ1tDQoO7ublVWVkbWeb1elZSUqL6+/oLvEwwGFQgEohYAycmRNkrpWXkx64IdX6qr9WQcOgJgRwMOKMYYrV69WjfccINKSkoi6xcuXKiXXnpJe/bs0VNPPaUDBw5ozpw5Cga/mhnS5/MpPT1dOTnRfxW53W75fL4Lvld1dbVcLldkKSyMfZIdAHuyLEuyYk96H2o/rc7Tn8WhIwB21K+veL5p5cqV+uCDD7Rv376o9UuXLo38u6SkRNOmTVNRUZF27Nih22+//aKvZ4z56hfXBaxbt06rV6+OPA4EAoQUIIll5k/QqJwCnWttvmRdb3dQveGePt0FGcDIMqAjKKtWrdIbb7yht99+W+PHj79kbUFBgYqKinT06FffN3s8HoVCIbW2tkbVtbS0yO12X/A1nE6nsrOzoxYAyWt03niNcl345/2bgu2nFQ6ejUNHAOymXwHFGKOVK1fqtdde0549e1RcXBzzOWfOnFFTU5MKCgokSWVlZUpLS1NtbW2kprm5WYcOHVJFRUU/2weQjBxpo5SS5oxZ9+XHf9DZL5mCALgc9eu46QMPPKCXX35Zv/nNb5SVlRU5Z8TlcikjI0MdHR3asGGD7rjjDhUUFOj48eNav3698vPzddttt0VqV6xYoTVr1igvL0+5ublau3atSktLI1f1ABjZLMtS+phcWSmpMuGeS9aa3t5LfgUMYGTqV0DZsmWLJGnWrFlR65977jktX75cKSkpamxs1AsvvKC2tjYVFBRo9uzZeuWVV5SVlRWp37x5s1JTU7VkyRJ1dXVp7ty52rp1q1JSUga/RwCSQs5V03Tm6H8q1PHlJetCna2S6ZUsfj8AlxPLJOFMSIFAQC6XS36/n/NRgCTV2xPS4Vf/r861XfpEWafLre/f+rDSRo2JU2cAhkt/Pr+5Fw+AhHCkpstyxD4qEvSfkulllmngckNAAZAw7ilzJSv2ryET7mbae+AyQ0ABkDBj3N/t08mvHS3Hh78ZALZCQAGQMKOyx/ZpVtkvPnwnDt0AsBMCCoDE6cPXO5LUcepPw9wIALshoABIqPE/uPgtMP7MqPssNwkFLicEFAAJlX3l92LWmN5etX3WGIduANgFAQVAwliWpdS+zG9ietVy6D+GvyEAtkFAAZBQjtQ0jSmY1Kda09s7zN0AsAsCCoCEcqQ6dUXRlJh1vT0hBTvOxKEjAHZAQAGQUJYjRRk53ph14e5zOnu6KQ4dAbADAgqAhLIsq0+TtfV0tavt0z/GoSMAdkBAAZBwqRlZGnVFQcw6E+5Wb08oDh0BSDQCCoCEG3WFR64JJTHrus8GFOpsG/6GACQcAQVAwjlS0pSaPjpmXWfLcbU3H41DRwASjYACIOEsy5IjNU2WI+WSdb3hkHq7z3FnY+AyQEABYAtXfOcvNDq/KGZdd1e7TLg7Dh0BSCQCCgBbSM+8QqmjMmPW+U8cVqizNQ4dAUgkAgoAW3Ckpsf8ikeSzn7xqXq6OuLQEYBESk10AwBGnnA4PKDzRLKL/kKBk0fUG+qK8fo96unpGWh7EQ6HQw4Hf6cBdsRPJoAhd9NNNykjI6PfS9mNfyXfqdMxX//+5Us0JjNzQO/xzeXv//7v4zAaAAaCIygAhlw4HB7QEY7Pv/Ar1BOOWfeD73n12jsfDvooSi83HwRsi4ACwNZau8eprcetnt50OR2dyk//XBUlhUpLcehcopsDMGz4igeArdT8xyH19n51/sqJc5PU2DFT/915vT7uuk5Hzk7Xwfa56ui5Qnmu2BO7AUheBBQAtrL73Y9ljHQqWKQPOyrUGc5Rr1IlWeoxTvl7xmm//xZNvuo7iW4VwDAioACwlfazIZ3tzdJ77ZUKK/2CNd0mQ1Nnb4pzZwDiiYACwFaMkY40nZFkXbrQspSSEqMGQNIioACwlV5j9GLtBzHrHJal8fnZcegIQCIQUADYzp8+/zJmzaj0FK24+bo4dAMgEQgoAGwnpbdNkzPekqULz4niUI9+mLtdV+ZnxbkzAPHSr4CyZcsWTZkyRdnZ2crOzlZ5ebnefPPNyHZjjDZs2CCv16uMjAzNmjVLhw8fjnqNYDCoVatWKT8/X5mZmVq8eLFOnDgxNHsDYETwd3SpoWGHvp+5X6McHf8TVIwsE5IJfaGrzIs61fKZjvvaEt0qgGHSr4naxo8fr8cff1xXX321JOn555/Xj370I73//vu69tpr9eSTT2rTpk3aunWrJk2apMcee0zz58/XkSNHlJX11V86VVVV+u1vf6uamhrl5eVpzZo1WrRokRoaGpSSEvtGYQBGvnOhHh09cUYLpx9Ua9enOnI6R762Xp07+4VC/j9qh+8TfXbKr1OtnYluFcAwscxA7uj1Dbm5ufrFL36hn/zkJ/J6vaqqqtLDDz8s6aujJW63W0888YTuvfde+f1+jR07Vtu2bdPSpUslSSdPnlRhYaF27typBQsW9Ok9A4GAXC6Xli9frvT0C1+GCCBx/v3f/10nT54c1Gt487I05btunfafjSwdXaEh6vAr5eXlKi0tHdLXBHBxoVBIW7duld/vV3b2pU9yH/BU9+FwWP/2b/+mzs5OlZeX69ixY/L5fKqsrIzUOJ1OzZw5U/X19br33nvV0NCg7u7uqBqv16uSkhLV19dfNKAEg0EFg8HI40AgIEm6++67NWbMmIHuAoBh0tDQMOiAcvJMu06eaR+iji5sxowZuvPOO4f1PQD8WUdHh7Zu3dqn2n4HlMbGRpWXl+vcuXMaM2aMtm/frsmTJ6u+vl6S5Ha7o+rdbrc+/fRTSZLP51N6erpycnLOq/H5fBd9z+rqav385z8/b/20adNiJjAA8ZcsP5eFhYX6wQ9+kOg2gMvG1wcY+qLfV/Fcc801OnjwoPbv36+f/vSnWrZsmT788MPIdsuKnjjJGHPeum+LVbNu3Tr5/f7I0tTU1N+2AQBAEul3QElPT9fVV1+tadOmqbq6WlOnTtXTTz8tj8cjSecdCWlpaYkcVfF4PAqFQmptbb1ozYU4nc7IlUNfLwAAYOQa9DwoxhgFg0EVFxfL4/GotrY2si0UCqmurk4VFRWSpLKyMqWlpUXVNDc369ChQ5EaAACAfp2Dsn79ei1cuFCFhYVqb29XTU2N9u7dq127dsmyLFVVVWnjxo2aOHGiJk6cqI0bN2r06NG66667JEkul0srVqzQmjVrlJeXp9zcXK1du1alpaWaN2/esOwgAABIPv0KKKdOndLdd9+t5uZmuVwuTZkyRbt27dL8+fMlSQ899JC6urp0//33q7W1VdOnT9fu3bsjc6BI0ubNm5WamqolS5aoq6tLc+fO1datW5kDBQAARAx6HpRE+HoelL5cRw0g/ubMmaO333470W3EtGnTJv3N3/xNotsALhv9+fzmXjwAAMB2CCgAAMB2CCgAAMB2CCgAAMB2BnwvHgC4mBtuuEEulyvRbcT09Z3ZAdgPV/EAAIC44CoeAACQ1AgoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdggoAADAdvoVULZs2aIpU6YoOztb2dnZKi8v15tvvhnZvnz5clmWFbXMmDEj6jWCwaBWrVql/Px8ZWZmavHixTpx4sTQ7A0AABgR+hVQxo8fr8cff1zvvvuu3n33Xc2ZM0c/+tGPdPjw4UjNTTfdpObm5siyc+fOqNeoqqrS9u3bVVNTo3379qmjo0OLFi1SOBwemj0CAABJzzLGmMG8QG5urn7xi19oxYoVWr58udra2vT6669fsNbv92vs2LHatm2bli5dKkk6efKkCgsLtXPnTi1YsKBP7xkIBORyueT3+5WdnT2Y9gEAQJz05/N7wOeghMNh1dTUqLOzU+Xl5ZH1e/fu1bhx4zRp0iTdc889amlpiWxraGhQd3e3KisrI+u8Xq9KSkpUX19/0fcKBoMKBAJRCwAAGLn6HVAaGxs1ZswYOZ1O3Xfffdq+fbsmT54sSVq4cKFeeukl7dmzR0899ZQOHDigOXPmKBgMSpJ8Pp/S09OVk5MT9Zput1s+n++i71ldXS2XyxVZCgsL+9s2AABIIqn9fcI111yjgwcPqq2tTa+++qqWLVumuro6TZ48OfK1jSSVlJRo2rRpKioq0o4dO3T77bdf9DWNMbIs66Lb161bp9WrV0ceBwIBQgoAACNYvwNKenq6rr76aknStGnTdODAAT399NP6p3/6p/NqCwoKVFRUpKNHj0qSPB6PQqGQWltbo46itLS0qKKi4qLv6XQ65XQ6+9sqAABIUoOeB8UYE/kK59vOnDmjpqYmFRQUSJLKysqUlpam2traSE1zc7MOHTp0yYACAAAuL/06grJ+/XotXLhQhYWFam9vV01Njfbu3atdu3apo6NDGzZs0B133KGCggIdP35c69evV35+vm677TZJksvl0ooVK7RmzRrl5eUpNzdXa9euVWlpqebNmzcsOwgAAJJPvwLKqVOndPfdd6u5uVkul0tTpkzRrl27NH/+fHV1damxsVEvvPCC2traVFBQoNmzZ+uVV15RVlZW5DU2b96s1NRULVmyRF1dXZo7d662bt2qlJSUId85AACQnAY9D0oiMA8KAADJJy7zoAAAAAwXAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALAdAgoAALCd1EQ3MBDGGElSIBBIcCcAAKCvvv7c/vpz/FKSMqC0t7dLkgoLCxPcCQAA6K/29na5XK5L1limLzHGZnp7e3XkyBFNnjxZTU1Nys7OTnRLSSsQCKiwsJBxHAKM5dBhLIcG4zh0GMuhYYxRe3u7vF6vHI5Ln2WSlEdQHA6HrrzySklSdnY2/1mGAOM4dBjLocNYDg3GcegwloMX68jJ1zhJFgAA2A4BBQAA2E7SBhSn06lHH31UTqcz0a0kNcZx6DCWQ4exHBqM49BhLOMvKU+SBQAAI1vSHkEBAAAjFwEFAADYDgEFAADYDgEFAADYTlIGlF/96lcqLi7WqFGjVFZWpt///veJbsl23nnnHd1yyy3yer2yLEuvv/561HZjjDZs2CCv16uMjAzNmjVLhw8fjqoJBoNatWqV8vPzlZmZqcWLF+vEiRNx3IvEq66u1vXXX6+srCyNGzdOt956q44cORJVw1j2zZYtWzRlypTIRFfl5eV68803I9sZx4Gprq6WZVmqqqqKrGMs+2bDhg2yLCtq8Xg8ke2MY4KZJFNTU2PS0tLMr3/9a/Phhx+aBx980GRmZppPP/000a3Zys6dO80jjzxiXn31VSPJbN++PWr7448/brKyssyrr75qGhsbzdKlS01BQYEJBAKRmvvuu89ceeWVpra21rz33ntm9uzZZurUqaanpyfOe5M4CxYsMM8995w5dOiQOXjwoLn55pvNhAkTTEdHR6SGseybN954w+zYscMcOXLEHDlyxKxfv96kpaWZQ4cOGWMYx4H4wx/+YL7zne+YKVOmmAcffDCynrHsm0cffdRce+21prm5ObK0tLREtjOOiZV0AeUHP/iBue+++6LWfe973zM/+9nPEtSR/X07oPT29hqPx2Mef/zxyLpz584Zl8tl/vEf/9EYY0xbW5tJS0szNTU1kZrPP//cOBwOs2vXrrj1bjctLS1GkqmrqzPGMJaDlZOTY/7lX/6FcRyA9vZ2M3HiRFNbW2tmzpwZCSiMZd89+uijZurUqRfcxjgmXlJ9xRMKhdTQ0KDKysqo9ZWVlaqvr09QV8nn2LFj8vl8UePodDo1c+bMyDg2NDSou7s7qsbr9aqkpOSyHmu/3y9Jys3NlcRYDlQ4HFZNTY06OztVXl7OOA7AAw88oJtvvlnz5s2LWs9Y9s/Ro0fl9XpVXFysO++8U5988okkxtEOkupmgadPn1Y4HJbb7Y5a73a75fP5EtRV8vl6rC40jp9++mmkJj09XTk5OefVXK5jbYzR6tWrdcMNN6ikpEQSY9lfjY2NKi8v17lz5zRmzBht375dkydPjvwyZxz7pqamRu+9954OHDhw3jb+T/bd9OnT9cILL2jSpEk6deqUHnvsMVVUVOjw4cOMow0kVUD5mmVZUY+NMeetQ2wDGcfLeaxXrlypDz74QPv27TtvG2PZN9dcc40OHjyotrY2vfrqq1q2bJnq6uoi2xnH2JqamvTggw9q9+7dGjVq1EXrGMvYFi5cGPl3aWmpysvL9d3vflfPP/+8ZsyYIYlxTKSk+oonPz9fKSkp5yXTlpaW81IuLu7rs9QvNY4ej0ehUEitra0XrbmcrFq1Sm+88YbefvttjR8/PrKeseyf9PR0XX311Zo2bZqqq6s1depUPf3004xjPzQ0NKilpUVlZWVKTU1Vamqq6urq9Hd/93dKTU2NjAVj2X+ZmZkqLS3V0aNH+T9pA0kVUNLT01VWVqba2tqo9bW1taqoqEhQV8mnuLhYHo8nahxDoZDq6uoi41hWVqa0tLSomubmZh06dOiyGmtjjFauXKnXXntNe/bsUXFxcdR2xnJwjDEKBoOMYz/MnTtXjY2NOnjwYGSZNm2a/vqv/1oHDx7UVVddxVgOUDAY1EcffaSCggL+T9pBIs7MHYyvLzN+9tlnzYcffmiqqqpMZmamOX78eKJbs5X29nbz/vvvm/fff99IMps2bTLvv/9+5HLsxx9/3LhcLvPaa6+ZxsZG8+Mf//iCl8+NHz/evPXWW+a9994zc+bMuewun/vpT39qXC6X2bt3b9SliGfPno3UMJZ9s27dOvPOO++YY8eOmQ8++MCsX7/eOBwOs3v3bmMM4zgY37yKxxjGsq/WrFlj9u7daz755BOzf/9+s2jRIpOVlRX5PGEcEyvpAooxxvzDP/yDKSoqMunp6ea6666LXPKJP3v77beNpPOWZcuWGWO+uoTu0UcfNR6PxzidTnPjjTeaxsbGqNfo6uoyK1euNLm5uSYjI8MsWrTIfPbZZwnYm8S50BhKMs8991ykhrHsm5/85CeRn9uxY8eauXPnRsKJMYzjYHw7oDCWffP1vCZpaWnG6/Wa22+/3Rw+fDiynXFMLMsYYxJz7AYAAODCkuocFAAAcHkgoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANshoAAAANv5/yhpqEK7Su8kAAAAAElFTkSuQmCC",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"env = gym.make(\"CartPole-v1\", render_mode='rgb_array')\n",
"# Test an untrained agent that acts randomly\n",
"frames=[]\n",
"for episode in range(10):\n",
" state, _ = env.reset()\n",
" done = False\n",
" while not done:\n",
" # Discretize the state\n",
" state_discretized = discretize_state(state)\n",
" \n",
" # Act randomly\n",
" action = np.random.randint(0,2) #randomly generates either 0 or 1\n",
" \n",
" # Step in the environment\n",
" state, reward, terminated, truncated, _ = env.step(action)\n",
" \n",
" # Render the environment (to visualize the agent)\n",
" im = env.render()\n",
" frames.append(im)\n",
" \n",
" if terminated or truncated:\n",
" done = True\n",
"\n",
"fig = plt.figure()\n",
"img = plt.imshow(frames[0])\n",
"\n",
"def update(frame):\n",
" img.set_data(frames[frame])\n",
" return img\n",
"\n",
"ani = FuncAnimation(fig, update, frames=len(frames), interval=50)\n",
"ani.save(\"CartPoleUntrained.mp4\", fps=10, writer='ffmpeg')\n",
"plt.show()\n",
"\n",
"env.close()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"### **Final Thoughts**\n",
"\n",
"- **Q-learning**: This basic Q-learning algorithm allows you to see how the agent improves over time. As the agent learns, the total reward should increase.\n",
"- **Exploration vs Exploitation**: You'll observe that early in training, the agent explores a lot, but over time, it starts exploiting what it has learned.\n",
"- **Visualization**: The plots and animations show how the agent's performance improves, and the rendered environment shows its progress in real-time.\n",
"\n",
"This tutorial provided a hands-on, visual approach for teaching reinforcement learning. It allows you to experiment with Q-learning in a simple environment and understand how the agent learns through interaction with the environment."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"### **Next Steps:**\n",
"- Modify the exploration rate (`epsilon`) and observe how it impacts learning.\n",
"- Experiment with the number of episodes and see how the agent’s performance changes.\n",
"- Try training the agent in different environments in Gymnasium, such as **MountainCar-v0** or **FrozenLake-v1**.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "cvml",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}