Cybernetical Intelligence

Prof. Dr. Kelvin K. L. Wong, is a distinguished expert in medical image processing and computational science, earning his Ph.D. from The University of Adelaide. With a strong academic background from Nanyang Technological University and The University of Sydney, he has been at the forefront of merging the fields of cybernetics and artificial intelligence (AI). He is renowned for coining the term “Cybernetical Intelligence” and is the inventor and founder of Deep Red AI.

• Preface xvAbout the Author xixAbout the Companion Website xxi1 Artificial Intelligence and Cybernetical Learning 11.1 Artificial Intelligence Initiative 11.2 Intelligent Automation Initiative 41.2.1 Benefits of IAI 51.3 Artificial Intelligence Versus Intelligent Automation 51.3.1 Process Discovery 61.3.2 Optimization 71.3.3 Analytics and Insight 81.4 The Fourth Industrial Revolution and Artificial Intelligence 91.4.1 Artificial Narrow Intelligence 101.4.2 Artificial General Intelligence 121.4.3 Artificial Super Intelligence 131.5 Pattern Analysis and Cognitive Learning 141.5.1 Machine Learning 151.5.1.1 Parametric Algorithms 161.5.1.2 Nonparametric Algorithms 171.5.2 Deep Learning 201.5.2.1 Convolutional Neural Networks in Advancing Artificial Intelligence 211.5.2.2 Future Advancement in Deep Learning 221.5.3 Cybernetical Learning 231.6 Cybernetical Artificial Intelligence 241.6.1 Artificial Intelligence Control Theory 241.6.2 Information Theory 261.6.3 Cybernetic Systems 271.7 Cybernetical Intelligence Definition 281.8 The Future of Cybernetical Intelligence 30Summary 32Exercise Questions 32Further Reading 332 Cybernetical Intelligent Control 352.1 Control Theory and Feedback Control Systems 352.2 Maxwell’s Analysis of Governors 372.3 Harold Black 392.4 Nyquist and Bode 402.5 Stafford Beer 422.5.1 Cybernetic Control 422.5.2 Viable Systems Model 422.5.3 Cybernetics Models of Management 432.6 James Lovelock 432.6.1 Cybernetic Approach to Ecosystems 432.6.2 Gaia Hypothesis 442.7 Macy Conference 442.8 McCulloch–Pitts 452.9 John von Neumann 472.9.1 Discussions on Self-Replicating Machines 472.9.2 Discussions on Machine Learning 48Summary 48Exercise Questions 49Further Reading 503 The Basics of Perceptron 513.1 The Analogy of Biological and Artificial Neurons 513.1.1 Biological Neurons and Neurodynamics 523.1.2 The Structure of Neural Network 533.1.3 Encoding and Decoding 563.2 Perception and Multilayer Perceptron 573.2.1 Back Propagation Neural Network 593.2.2 Derivative Equations for Backpropagation 593.3 Activation Function 613.3.1 Sigmoid Activation Function 613.3.2 Hyperbolic Tangent Activation Function 623.3.3 Rectified Linear Unit Activation Function 623.3.4 Linear Activation Function 64Summary 65Exercise Questions 67Further Reading 674 The Structure of Neural Network 694.1 Layers in Neural Network 694.1.1 Input Layer 694.1.2 Hidden Layer 704.1.3 Neurons 704.1.4 Weights and Biases 714.1.5 Forward Propagation 724.1.6 Backpropagation 724.2 Perceptron and Multilayer Perceptron 734.3 Recurrent Neural Network 754.3.1 Long Short-Term Memory 764.4 Markov Neural Networks 774.4.1 State Transition Function 774.4.2 Observation Function 784.4.3 Policy Function 784.4.4 Loss Function 784.5 Generative Adversarial Network 78Summary 79Exercise Questions 80Further Reading 815 Backpropagation Neural Network 835.1 Backpropagation Neural Network 835.1.1 Forward Propagation 855.2 Gradient Descent 855.2.1 Loss Function 855.2.2 Parameters in Gradient Descent 885.2.3 Gradient in Gradient Descent 885.2.4 Learning Rate in Gradient Descent 895.2.5 Update Rule in Gradient Descent 895.3 Stopping Criteria 895.3.1 Convergence and Stopping Criteria 905.3.2 Local Minimum and Global Minimum 915.4 Resampling Methods 915.4.1 Cross-Validation 935.4.2 Bootstrapping 935.4.3 Monte Carlo Cross-Validation 945.5 Optimizers in Neural Network 945.5.1 Stochastic Gradient Descent 945.5.2 Root Mean Square Propagation 965.5.3 Adaptive Moment Estimation 965.5.4 AdaMax 975.5.5 Momentum Optimization 97Summary 97Exercise Questions 99Further Reading 1006 Application of Neural Network in Learning and Recognition 1016.1 Applying Backpropagation to Shape Recognition 1016.2 Softmax Regression 1056.3 K-Binary Classifier 1076.4 Relational Learning via Neural Network 1086.4.1 Graph Neural Network 1096.4.2 Graph Convolutional Network 1116.5 Cybernetics Using Neural Network 1126.6 Structure of Neural Network for Image Processing 1156.7 Transformer Networks 1166.8 Attention Mechanisms 1166.9 Graph Neural Networks 1176.10 Transfer Learning 1186.11 Generalization of Neural Networks 1196.12 Performance Measures 1206.12.1 Confusion Matrix 1206.12.2 Receiver Operating Characteristic 1216.12.3 Area Under the ROC Curve 122Summary 123Exercise Questions 123Further Reading 1247 Competitive Learning and Self-Organizing Map 1257.1 Principal of Competitive Learning 1257.1.1 Step 1: Normalized Input Vector 1287.1.2 Step 2: Find the Winning Neuron 1287.1.3 Step 3: Adjust the Network Weight Vector and Output Results 1297.2 Basic Structure of Self-Organizing Map 1297.2.1 Properties Self-Organizing Map 1307.3 Self-Organizing Mapping Neural Network Algorithm 1317.3.1 Step 1: Initialize Parameter 1327.3.2 Step 2: Select Inputs and Determine Winning Nodes 1327.3.3 Step 3: Affect Neighboring Neurons 1327.3.4 Step 4: Adjust Weights 1337.3.5 Step 5: Judging the End Condition 1337.4 Growing Self-Organizing Map 1337.5 Time Adaptive Self-Organizing Map 1367.5.1 TASOM-Based Algorithms for Real Applications 1387.6 Oriented and Scalable Map 1397.7 Generative Topographic Map 141Summary 145Exercise Questions 146Further Reading 1478 Support Vector Machine 1498.1 The Definition of Data Clustering 1498.2 Support Vector and Margin 1528.3 Kernel Function 1558.3.1 Linear Kernel 1558.3.2 Polynomial Kernel 1568.3.3 Radial Basis Function 1578.3.4 Laplace Kernel 1598.3.5 Sigmoid Kernel 1598.4 Linear and Nonlinear Support Vector Machine 1608.5 Hard Margin and Soft Margin in Support Vector Machine 1648.6 I/O of Support Vector Machine 1678.6.1 Training Data 1678.6.2 Feature Matrix and Label Vector 1688.7 Hyperparameters of Support Vector Machine 1698.7.1 The C Hyperparameter 1698.7.2 Kernel Coefficient 1698.7.3 Class Weights 1708.7.4 Convergence Criteria 1708.7.5 Regularization 1718.8 Application of Support Vector Machine 1718.8.1 Classification 1718.8.2 Regression 1738.8.3 Image Classification 1738.8.4 Text Classification 174Summary 174Exercise Questions 175Further Reading 1769 Bio-Inspired Cybernetical Intelligence 1779.1 Genetic Algorithm 1789.2 Ant Colony Optimization 1819.3 Bees Algorithm 1849.4 Artificial Bee Colony Algorithm 1869.5 Cuckoo Search 1899.6 Particle Swarm Optimization 1939.7 Bacterial Foraging Optimization 1969.8 Gray Wolf Optimizer 1979.9 Firefly Algorithm 199Summary 200Exercise Questions 201Further Reading 20210 Life-Inspired Machine Intelligence and Cybernetics 20310.1 Multi-Agent AI Systems 20310.1.1 Game Theory 20510.1.2 Distributed Multi-Agent Systems 20610.1.3 Multi-Agent Reinforcement Learning 20710.1.4 Evolutionary Computation and Multi-Agent Systems 20910.2 Cellular Automata 21110.3 Discrete Element Method 21210.3.1 Particle-Based Simulation of Biological Cells and Tissues 21410.3.2 Simulation of Microbial Communities and Their Interactions 21510.3.3 Discrete Element Method-Based Modeling of Biological Fluids and Soft Materials 21610.4 Smoothed Particle Hydrodynamics 21810.4.1 SPH-Based Simulations of Biomimetic Fluid Dynamic 21910.4.2 SPH-Based Simulations of Bio-Inspired Engineering Applications 220Summary 221Exercise Questions 222Further Reading 22311 Revisiting Cybernetics and Relation to Cybernetical Intelligence 22511.1 The Concept and Development of Cybernetics 22511.1.1 Attributes of Control Concepts 22511.1.2 Research Objects and Characteristics of Cybernetics 22611.1.3 Development of Cybernetical Intelligence 22711.2 The Fundamental Ideas of Cybernetics 22711.2.1 System Idea 22711.2.2 Information Idea 22911.2.3 Behavioral Idea 23011.2.4 Cybernetical Intelligence Neural Network 23111.3 Cybernetic Expansion into Other Fields of Research 23411.3.1 Social Cybernetics 23411.3.2 Internal Control-Related Theories 23711.3.3 Software Control Theory 23711.3.4 Perceptual Cybernetics 23811.4 Practical Application of Cybernetics 24011.4.1 Research on the Control Mechanism of Neural Networks 24011.4.2 Balance Between Internal Control and Management Power Relations 24011.4.3 Software Markov Adaptive Testing Strategy 24211.4.4 Task Analysis Model 244Summary 245Exercise Questions 246Further Reading 24712 Turing Machine 24912.1 Behavior of a Turing Machine 25012.1.1 Computing with Turing Machines 25112.2 Basic Operations of a Turing Machine 25212.2.1 Reading and Writing to the Tape 25312.2.2 Moving the Tape Head 25412.2.3 Changing States 25412.3 Interchangeability of Program and Behavior 25512.4 Computability Theory 25612.4.1 Complexity Theory 25712.5 Automata Theory 25812.6 Philosophical Issues Related to Turing Machines 25912.7 Human and Machine Computations 26012.8 Historical Models of Computability 26112.9 Recursive Functions 26212.10 Turing Machine and Intelligent Control 263Summary 264Exercise Questions 265Further Reading 26513 Entropy Concepts in Machine Intelligence 26713.1 Relative Entropy of Distributions 26813.2 Relative Entropy and Mutual Information 26813.3 Entropy in Performance Evaluation 26913.4 Cross-Entropy Softmax 27113.5 Calculating Cross-Entropy 27213.6 Cross-Entropy as a Loss Function 27313.7 Cross-Entropy and Log Loss 27413.8 Application of Entropy in Intelligent Control 27513.8.1 Entropy-Based Control 27513.8.2 Fuzzy Entropy 27613.8.3 Entropy-Based Control Strategies 27713.8.4 Entropy-Based Decision-Making 278Summary 279Exercise Questions 279Further Reading 28014 Sampling Methods in Cybernetical Intelligence 28314.1 Introduction to Sampling Methods 28314.2 Basic Sampling Algorithms 28414.2.1 Importance of Sampling Methods in Machine Intelligence 28614.3 Machine Learning Sampling Methods 28714.3.1 Random Oversampling 28814.3.2 Random Undersampling 29014.3.3 Synthetic Minority Oversampling Technique 29014.3.4 Adaptive Synthetic Sampling 29214.4 Advantages and Disadvantages of Machine Learning Sampling Methods 29314.5 Advanced Sampling Methods in Cybernetical Intelligence 29414.5.1 Ensemble Sampling Method 29514.5.2 Active Learning 29714.5.3 Bayesian Optimization in Sampling 29914.6 Applications of Sampling Methods in Cybernetical Intelligence 30214.6.1 Image Processing and Computer Vision 30214.6.2 Natural Language Processing 30414.6.3 Robotics and Autonomous Systems 30714.7 Challenges and Future Directions 30814.8 Challenges and Limitations of Sampling Methods 30914.9 Emerging Trends and Innovations in Sampling Methods 309Summary 310Exercise Questions 311Further Reading 31215 Dynamic System Control 31315.1 Linear Systems 31415.2 Nonlinear System 31615.3 Stability Theory 31815.4 Observability and Identification 32015.5 Controllability and Stabilizability 32115.6 Optimal Control 32315.7 Linear Quadratic Regulator Theory 32415.8 Time-Optimal Control 32615.9 Stochastic Systems with Applications 32815.9.1 Stochastic System in Control Systems 32915.9.2 Stochastic System in Robotics and Automation 32915.9.3 Stochastic System in Neural Networks 330Summary 331Exercise Questions 331Further Reading 33216 Deep Learning 33316.1 Neural Network Models in Deep Learning 33516.2 Methods of Deep Learning 33616.2.1 Convolutional Neural Networks 33716.2.2 Recurrent Neural Networks 34016.2.3 Generative Adversarial Networks 34216.2.4 Deep Learning Based Image Segmentation Models 34516.2.5 Variational Auto Encoders 34816.2.6 Transformer Models 35016.2.7 Attention-Based Models 35216.2.8 Meta-Learning Models 35416.2.9 Capsule Networks 35716.3 Deep Learning Frameworks 35816.4 Applications of Deep Learning 35916.4.1 Object Detection 36016.4.2 Intelligent Power Systems 36116.4.3 Intelligent Control 362Summary 362Exercise Questions 363References 364Further Reading 36517 Neural Architecture Search 36717.1 Neural Architecture Search and Neural Network 36917.2 Reinforcement Learning-Based Neural Architecture Search 37117.3 Evolutionary Algorithms-Based Neural Architecture Search 37417.4 Bayesian Optimization-Based Neural Architecture Search 37617.5 Gradient-Based Neural Architecture Search 37817.6 One-shot Neural Architecture Search 37917.7 Meta-Learning-Based Neural Architecture Search 38117.8 Neural Architecture Search for Specific Domains 38317.8.1 Cybernetical Intelligent Systems: Neural Architecture Search in Real-World 38417.8.2 Neural Architecture Search for Specific Cybernetical Control Tasks 38517.8.3 Neural Architecture Search for Cybernetical Intelligent Systems in Real-World 38617.8.4 Neural Architecture Search for Adaptive Cybernetical Intelligent Systems 38817.9 Comparison of Different Neural Architecture Search Approaches 389Summary 391Exercise Questions 391Further Reading 392Final Notes on Cybernetical Intelligence 393Index 399