Markov Processes for Stochastic Modeling

Kijima, Masaaki

• Introduction: Stochastic processes; the Markov property; some examples; transition probabilities; the strong Markov property; exercises. Discrete-time Markov chains: First time passages; classification of states; recurrent Markov chains; finite Markov chains; time-reversable Markov chains; the rate of convergence to stationary; absorbing Markov chanins and their applications; Lossy Markov chains; exercises. Monotone Markov chains: Preliminaries; distribution classes of interest; stochastic ordering relations; monotone Markov chains; unimodality of transition probabilities; first-passage-time distributions; bounds for quasi-stationary distributions; renewal processes in discrete time; comparability of Markov chains; exercises. Continuous-time Markov chains: transition probability functions; finite Markov chains in continuous time; uniformization; more on finite Markov chains; absorbing Markov chains in continuous time; calculation of transition probability functions; stochastic monotonicity; semi-Markov processes; exercises. Birth-death processes: Boundary classifications; birth-death polynomials; finite birth-death processes; the Karlin-McGregor representation theorem; asymptotics of birth-death polynomials; quasi-stationary distributions; the decay parameter; the M/M/1 queue; exercises. Appendix A Review of matrix theory: Nonnegative matrices; ML-matrices; infinite matrices. Appendix B Generating functions and Laplace transforms: Generating functions; Laplace transforms. Appendix C Total positivity: TP functions; the variation-diminishing property.