Homework #1 Finding the Optimal State-Value Function Problem Description The game DieN is played in the following way:
1. You will be given a die with N sides. You will know the size of N, and can assume that N is a value greater than 1 and less than or equal to 30.
2. You will be given a bit mask vector isBadSide representing the sides of a die that will make you lose. The vector will be of size N, and 1 indexed. (there is no 0 side)
3. You start with 0 dollars.
4. At any time you have the option to roll the die or to quit the game
a. If you decide to roll:
i. And you roll a number not in is Bad Side, you receive that many dollars. (eg. if you roll the number 2 and 2 is not active — meaning the second element of the vector is 0 — in is Bad Side, then you receive 2 dollars) Repeat step 4.
ii. And you roll a number in is Bad Side, then you lose all money obtained in previous rolls and the game ends.
b. If you decide to quit:
i. You keep all money gained from previous rolls and the game ends. Procedure ? For this problem, determine an optimal policy for playing the game DieN for N sides. You will be given N and the array is Bad Side which indicates which sides are bad. As you will see, the optimal policy for this game will depend on your current bankroll. ? You can try solving this problem either by creating an MDP of the game (state, action, transition, reward function, and assume a gamma of 1) and then calculating the optimal state-value function or you can plug-in values and solve directly using the Bellman Equations. ? What is the expected amount of dollars for this game if you follow an optimal policy?
That is, what is the optimal state-value function for the initial state of the game? Provide answers for the problems you are given in the “Solve for Code” tab on the Heroku site. Your answer must be correct to 3 decimal places. Examples The following examples can be used to verify your calculation is correct. ? Input: N = 21, is Bad Side = {1,1,1,1,0,0,0,0,1,0,1,0,1,1,0,1,0,0,0,1,0}, Output: 7.3799 ? Input: N = 22, is Bad Side = {1,1,1,1,1,1,0,1,0,1,1,0,1,0,1,0,0,1,0,0,1,0}, Output: 6.314 ? Input: N = 6, is Bad Side = {1,1,1,0,0,0}, Output: 2.5833