Optimal Control

FRANK L. LEWIS is the Moncrief-O'Donnell Professor and Head of the Advanced Controls, Sensors, and MEMS Group in the Automation and Robotics Research Institute of the University of Texas at Arlington. Dr. Lewis is also a Fellow of the IEEE. DRAGUNA L. VRABIE is Graduate Research Assistant in Electrical Engineering at the University of Texas at Arlington, specializing in approximate dynamic programming for continuous state and action spaces, optimal control, adaptive control, model predictive control, and general theory of nonlinear systems. VASSILIS L. SYRMOS is a Professor in the Department of Electrical Engineering and the Associate Vice Chancellor for Research and Graduate Education at the University of Hawaii at Manoa.

• Preface xi1 STATIC OPTIMIZATION 11.1 Optimization without Constraints 11.2 Optimization with Equality Constraints 41.3 Numerical Solution Methods 15Problems 152 OPTIMAL CONTROL OF DISCRETE-TIME SYSTEMS 192.1 Solution of the General Discrete-Time Optimization Problem 192.2 Discrete-Time Linear Quadratic Regulator 322.3 Digital Control of Continuous-Time Systems 532.4 Steady-State Closed-Loop Control and Suboptimal Feedback 652.5 Frequency-Domain Results 96Problems 1023 OPTIMAL CONTROL OF CONTINUOUS-TIME SYSTEMS 1103.1 The Calculus of Variations 1103.2 Solution of the General Continuous-Time Optimization Problem 1123.3 Continuous-Time Linear Quadratic Regulator 1353.4 Steady-State Closed-Loop Control and Suboptimal Feedback 1543.5 Frequency-Domain Results 164Problems 1674 THE TRACKING PROBLEM AND OTHER LQR EXTENSIONS 1774.1 The Tracking Problem 1774.2 Regulator with Function of Final State Fixed 1834.3 Second-Order Variations in the Performance Index 1854.4 The Discrete-Time Tracking Problem 1904.5 Discrete Regulator with Function of Final State Fixed 1994.6 Discrete Second-Order Variations in the Performance Index 206Problems 2115 FINAL-TIME-FREE AND CONSTRAINED INPUT CONTROL 2135.1 Final-Time-Free Problems 2135.2 Constrained Input Problems 232Problems 2576 DYNAMIC PROGRAMMING 2606.1 Bellman’s Principle of Optimality 2606.2 Discrete-Time Systems 2636.3 Continuous-Time Systems 271Problems 2837 OPTIMAL CONTROL FOR POLYNOMIAL SYSTEMS 2877.1 Discrete Linear Quadratic Regulator 2877.2 Digital Control of Continuous-Time Systems 292Problems 2958 OUTPUT FEEDBACK AND STRUCTURED CONTROL 2978.1 Linear Quadratic Regulator with Output Feedback 2978.2 Tracking a Reference Input 3138.3 Tracking by Regulator Redesign 3278.4 Command-Generator Tracker 3318.5 Explicit Model-Following Design 3388.6 Output Feedback in Game Theory and Decentralized Control 343Problems 3519 ROBUSTNESS AND MULTIVARIABLE FREQUENCY-DOMAIN TECHNIQUES 3559.1 Introduction 3559.2 Multivariable Frequency-Domain Analysis 3579.3 Robust Output-Feedback Design 3809.4 Observers and the Kalman Filter 3839.5 LQG/Loop-Transfer Recovery 4089.6 H∞ DESIGN 430Problems 43510 DIFFERENTIAL GAMES 43810.1 Optimal Control Derived Using Pontryagin’s Minimum Principle and the Bellman Equation 43910.2 Two-player Zero-sum Games 44410.3 Application of Zero-sum Games to H∞ Control 45010.4 Multiplayer Non-zero-sum Games 45311 REINFORCEMENT LEARNING AND OPTIMAL ADAPTIVE CONTROL 46111.1 Reinforcement Learning 46211.2 Markov Decision Processes 46411.3 Policy Evaluation and Policy Improvement 47411.4 Temporal Difference Learning and Optimal Adaptive Control 48911.5 Optimal Adaptive Control for Discrete-time Systems 49011.6 Integral Reinforcement Learning for Optimal Adaptive Control of Continuous-time Systems 50311.7 Synchronous Optimal Adaptive Control for Continuous-time Systems 513APPENDIX A REVIEW OF MATRIX ALGEBRA 518A.1 Basic Definitions and Facts 518A.2 Partitioned Matrices 519A.3 Quadratic Forms and Definiteness 521A.4 Matrix Calculus 523A.5 The Generalized Eigenvalue Problem 525References 527Index 535