2.1 Markov decision -- type 3 gains

An electronic store stocks a certain type of VCR. At the end of each week, an order is placed for early delivery the following Monday. A maximum of fourunits is stocked. Let the states be the number of units on hand at the end of the sales week: $Eb=\{0,1,2,3,4\}$ . Two possible actions:

• Order two units, at a cost of $150 each
• Order four units, at a cost of $120 each

Units sell for $200. If demand exceeds the stock in hand, the retailer assumes a penalty of $40 per unit (in losses due to customer dissatisfaction, etc.).Because of turnover, return on sales is considered two percent per week, so that discount is $\alpha =1/1.02$ on a weekly basis.

In state 0, there are three possible actions: order 0, 2, or 4. In states 1 and 2 there are two possible actions: order 0 or 2. In states 3 and 4, the only actionis to order 0. Customer demand in week $n+1$ is represented by a random variable $D_{n+1}$ . The class is iid, uniformly distributed on the values 0, 1, 2, 3, 4. If X _n is the state at the end of week n , then $\{X_n,D_{n+1}\}$ is independent for each n .

Analyze the system as a Markov decision process with case 3 gains, depending upon current state, action, and demand. Determine the transition probability matrixPA (properly padded) and the gain matrix (also padded). Sample calculations are as follows:

• State 0, action 0: $p_{00}\left(0\right)=1$ (all other $p_{0k}\left(0\right)=0$ )
• State 0, action 2: $p_{00}\left(2\right)=P(D\ge 2)=3/5,p_{01}\left(2\right)=P(D=1)=1/5$ , etc.
• State 2, action 2: $p_{2j}\left(k\right)=1/5,k=0,1,2,3,4$

For state = i , action = a , and demand = k , we seek $g(i,a,k)$

$\begin{array}{cccccc}g(0,0,k)=-40k& 0& -40& -80& -120& -160\\ g(0,2,k)=-300+200min\{k,2\}-40max\{k-2,0\}& -300& -100& 100& 60& 20\\ g(0,4,k)=-480+200k& -480& -280& -80& 120& 320\end{array}$

1. Complete the transition probability table and the gain table.
2. Determine an optimum infinite-horizon strategy with no discounting.
3. Determine an optimum infinite-horizon strateby with discounting (alpha = 1/1.02).
4. The manager decides to set up a six-week strategy, after which new sales conditions may be established. Determine an optimum strategy for the six-week period.

Data file

Transition probabilities and gains

The procedure