Complex Valued Nonlinear Adaptive Filters

Danilo Mandic, Department of Electrical and Electronic Engineering, Imperial College London, LondonDr Mandic is currently a Reader in Signal Processing at Imperial College, London. He is an experienced author, having written the book Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability (Wiley, 2001), and more than 150 published journal and conference papers on signal and image processing. His research interests include nonlinear adaptive signal processing, multimodal signal processing and nonlinear dynamics, and he is an Associate Editor for the journals IEEE Transactions on Circuits and Systems and the International Journal of Mathematical Modelling and Algorithms. Dr Mandic is also on the IEEE Technical Committee on Machine Learning for Signal Processing, and he has produced award winning papers and products resulting from his collaboration with industry. Su-Lee Goh, Royal Dutch Shell plc, HollandDr Goh is currently working as a Reservoir Imaging Geophysicist at Shell in Holland. Her research interests include nonlinear signal processing, adaptive filters, complex-valued analysis, and imaging and forecasting. She received her PhD in nonlinear adaptive signal processing from Imperial College, London and is a member of the IEEE and the Society of Exploration Geophysicists.

• Preface xiiiAcknowledgements xvii1 The Magic of Complex Numbers 11.1 History of Complex Numbers 21.2 History of Mathematical Notation 81.3 Development of Complex Valued Adaptive Signal Processing 92 Why Signal Processing in the Complex Domain? 132.1 Some Examples of Complex Valued Signal Processing 132.2 Modelling in C is Not Only Convenient But Also Natural 192.3 Why Complex Modelling of Real Valued Processes? 202.4 Exploiting the Phase Information 232.5 Other Applications of Complex Domain Processing of Real Valued Signals 262.6 Additional Benefits of Complex Domain Processing 293 Adaptive Filtering Architectures 333.1 Linear and Nonlinear Stochastic Models 343.2 Linear and Nonlinear Adaptive Filtering Architectures 353.3 State Space Representation and Canonical Forms 394 Complex Nonlinear Activation Functions 434.1 Properties of Complex Functions 434.2 Universal Function Approximation 464.3 Nonlinear Activation Functions for Complex Neural Networks 484.4 Generalised Splitting Activation Functions (GSAF) 534.5 Summary: Choice of the Complex Activation Function 545 Elements of CR Calculus 555.1 Continuous Complex Functions 565.2 The Cauchy–Riemann Equations 565.3 Generalised Derivatives of Functions of Complex Variable 575.4 CR-derivatives of Cost Functions 626 Complex Valued Adaptive Filters 696.1 Adaptive Filtering Configurations 706.2 The Complex Least Mean Square Algorithm 736.3 Nonlinear Feedforward Complex Adaptive Filters 806.4 Normalisation of Learning Algorithms 856.5 Performance of Feedforward Nonlinear Adaptive Filters 876.6 Summary: Choice of a Nonlinear Adaptive Filter 897 Adaptive Filters with Feedback 917.1 Training of IIR Adaptive Filters 927.2 Nonlinear Adaptive IIR Filters: Recurrent Perceptron 977.3 Training of Recurrent Neural Networks 997.4 Simulation Examples 1028 Filters with an Adaptive Stepsize 1078.1 Benveniste Type Variable Stepsize Algorithms 1088.2 Complex Valued GNGD Algorithms 1108.3 Simulation Examples 1139 Filters with an Adaptive Amplitude of Nonlinearity 1199.1 Dynamical Range Reduction 1199.2 FIR Adaptive Filters with an Adaptive Nonlinearity 1219.3 Recurrent Neural Networks with Trainable Amplitude of Activation Functions 1229.4 Simulation Results 12410 Data-reusing Algorithms for Complex Valued Adaptive Filters 12910.1 The Data-reusing Complex Valued Least Mean Square (DRCLMS) Algorithm 12910.2 Data-reusing Complex Nonlinear Adaptive Filters 13110.3 Data-reusing Algorithms for Complex RNNs 13411 Complex Mappings and M¨obius Transformations 13711.1 Matrix Representation of a Complex Number 13711.2 The M¨obius Transformation 14011.3 Activation Functions and M¨obius Transformations 14211.4 All-pass Systems as M¨obius Transformations 14611.5 Fractional Delay Filters 14712 Augmented Complex Statistics 15112.1 Complex Random Variables (CRV) 15212.2 Complex Circular Random Variables 15812.3 Complex Signals 15912.4 Second-order Characterisation of Complex Signals 16113 Widely Linear Estimation and Augmented CLMS (ACLMS) 16913.1 Minimum Mean Square Error (MMSE) Estimation in C 16913.2 Complex White Noise 17213.3 Autoregressive Modelling in C 17313.4 The Augmented Complex LMS (ACLMS) Algorithm 17513.5 Adaptive Prediction Based on ACLMS 17814 Duality Between Complex Valued and Real Valued Filters 18314.1 A Dual Channel Real Valued Adaptive Filter 18414.2 Duality Between Real and Complex Valued Filters 18614.3 Simulations 18815 Widely Linear Filters with Feedback 19115.1 The Widely Linear ARMA (WL-ARMA) Model 19215.2 Widely Linear Adaptive Filters with Feedback 19215.4 The Augmented Kalman Filter Algorithm for RNNs 19815.5 Augmented Complex Unscented Kalman Filter (ACUKF) 20015.6 Simulation Examples 20316 Collaborative Adaptive Filtering 20716.1 Parametric Signal Modality Characterisation 20716.2 Standard Hybrid Filtering in R 20916.3 Tracking the Linear/Nonlinear Nature of Complex Valued Signals 21016.4 Split vs Fully Complex Signal Natures 21416.5 Online Assessment of the Nature of Wind Signal 21616.6 Collaborative Filters for General Complex Signals 21717 Adaptive Filtering Based on EMD 22117.1 The Empirical Mode Decomposition Algorithm 22217.2 Complex Extensions of Empirical Mode Decomposition 22617.3 Addressing the Problem of Uniqueness 23017.4 Applications of Complex Extensions of EMD 23018 Validation of Complex Representations – Is This Worthwhile? 23318.1 Signal Modality Characterisation in R 23418.2 Testing for the Validity of Complex Representation 23918.3 Quantifying Benefits of Complex Valued Representation 243Appendix A: Some Distinctive Properties of Calculus in C 245Appendix B: Liouville's Theorem 251Appendix C: Hypercomplex and Clifford Algebras 253Appendix D: Real Valued Activation Functions 257Appendix E: Elementary Transcendental Functions (ETF) 259Appendix F: The O Notation and Standard Vector and Matrix Differentiation 263Appendix G: Notions From Learning Theory 265Appendix H: Notions from Approximation Theory 269Appendix I: Terminology Used in the Field of Neural Networks 273Appendix J: Complex Valued Pipelined Recurrent Neural Network (CPRNN) 275Appendix K: Gradient Adaptive Step Size (GASS) Algorithms in R 279Appendix L: Derivation of Partial Derivatives from Chapter 8 283Appendix M: A Posteriori Learning 287Appendix N: Notions from Stability Theory 291Appendix O: Linear Relaxation 293Appendix P: Contraction Mappings, Fixed Point Iteration and Fractals 299References 309Index 321