Communications in Mathematical Sciences

Volume 12 (2014)

Number 5

Markov control processes with rare state observation: theory and application to treatment scheduling in HIV–1

Pages: 859 – 877



Stefanie Winkelmann (Department of Mathematics and Computer Science, Freie Universität, Berlin, Germany)

Christof Schütte (Department of Mathematics and Computer Science, Freie Universität, Berlin, Germany )

Max von Kleist (Department of Mathematics and Computer Science, Freie Universität, Berlin, Germany )


Markov Decision Processes (MDP) or Partially Observable MDPs (POMDP) are used for modelling situations in which the evolution of a process is partly random and partly controllable. These MDP theories allow for computing the optimal control policy for processes that can continuously or frequently be observed, even if only partially. However, they cannot be applied if state observation is very costly and therefore rare (in time). We present a novel MDP theory for rare, costly observations and derive the corresponding Bellman equation. In the new theory, state information can be derived for a particular cost after certain, rather long time intervals. The resulting information costs enter into the total cost and thus into the optimization criterion. This approach applies to many real world problems, particularly in the medical context, where the medical condition is examined rather rarely because examination costs are high. At the same time, the approach allows for efficient numerical realization. We demonstrate the usefulness of the novel theory by determining, from the national economic perspective, optimal therapeutic policies for the treatment of the human immunodeficiency virus (HIV) in resource-rich and resource-poor settings. Based on the developed theory and models, we discover that available drugs may not be utilized efficiently in resource-poor settings due to exorbitant diagnostic costs.


information costs, hidden state, Bellman equation, optimal therapeutic policies, diagnostic frequency, resource-poor, resource-rich

2010 Mathematics Subject Classification

49N30, 60J27, 60J28, 90C40, 93B07, 93E20

Full Text (PDF format)