Discrete Stochastic Processes

• 1 Introduction and Probability Review.- 1.1 Introduction.- 1.2 Probability Review.- 1.3 Conditional Probabilities.- 1.4 Random Variables.- 1.5 Expectations.- 1.6 Transforms.- 1.7 Weak Law of Large Numbers.- 1.8 Strong Law of Large Numbers.- 1.9 Summary.- Table of Standard Random Variables.- Exercises.- Notes.- 2 Poisson Processes.- 2.1 Introduction.- 2.2 Definition and Properties of the Poisson Process.- 2.3 Combinations and Subdivisions of Independent Poisson Processes.- 2.4 Non-Homogeneous Poisson Processes.- 2.5 Order Statistics and Conditional Arrival Epochs.- 2.6 Summary.- Exercises.- Notes.- 3 Renewal Processes.- 3.1 Introduction.- 3.2 Strong Law of Large Numbers for Renewal Processes.- 3.3 Expected Number of Renewals.- 3.4 Renewal Reward Processes; Time Averages.- 3.5 Renewal Reward Processes; Ensemble Averages.- 3.6 Applications of Renewal Reward Theory.- 3.7 Delayed Renewal Processes.- 3.8 Summary.- Exercises.- Notes.- 4 Finite State Markov Chains.- 4.1 Introduction.- 4.2 Classification of States.- 4.3 The Matrix Representation.- 4.4 Perron—Frobenius Theory.- 4.5 Markov Chains with Rewards.- 4.6 Markov Decision Theory and Dynamic Programming.- 4.7 Summary.- Exercises.- Notes.- 5 Markov Chains with Countably Infinite State Spaces.- 5.1 Introduction and Classification of States.- 5.2 Branching Processes.- 5.3 Birth Death Markov Chains.- 5.4 Reversible Markov Chains.- 5.5 The M/M/1 Sampled Time Markov Chain.- 5.6 Round-Robin and Processor Sharing.- 5.7 Semi-Markov Processes.- 5.8 Example—M/G/1 Queue.- 5.9 Summary.- Exercises.- 6 Markov Processes with Countable State Spaces.- 6.1 Introduction.- 6.2 The Kolmogorov Differential Equations.- 6.3 Uniformization.- 6.4 Birth Death Processes.- 6.5 Reversibility for Markov Processes.- 6.6 Jackson Networks.- ClosedJackson Networks.- 6.7 Summary.- Exercises.- 7 Random Walks and Martingales.- 7.1 Introduction.- 7.2 The G/G/1 Queue.- 7.3 Detection, Decisions, and Hypothesis Testing.- 7.4 Threshold Crossing Probabilities.- 7.5 Wald’s Identity and Walks with Two Thresholds.- 7.6 Martingales and Submartingales.- 7.7 Stopped Processes and Stopping Rules.- 7.8 The Kolmogorov Inequalities.- 7.9 Summary.- Exercises.- Notes.

' ... very good book that will help students to understand many of the concepts used in describing stochastic processes. I recommend it highly either as a main text or as supplementary reading.' Interfaces, 27:4 (1997)