M. Ümit Uyar is a Professor at the City College of the City University of New York, USA. Dr. Uyar is an IEEE Fellow, author, co-author and co-editor of seven books, holder of seven U.S. patents, and developer of AI and game theory-based algorithms for applications in topology control in mobile networks and personalized cancer treatment.

• About the Author xiiiPreface xvAcknowledgments xviiAbout the Companion Website xix1 Introduction 11.1 Artificial Intelligence 11.2 A Historical Perspective 11.3 Principles of AI 21.4 Applications That Are Impossible Without AI 21.5 Organization of This Book 32 Artificial Neural Networks 72.1 Introduction 72.2 Applications of ANNs 72.3 Components of ANNs 82.3.1 Neurons 82.3.2 Sigmoid Activation Function 92.3.3 Rectilinear Activation Function 92.3.4 Weights of Synapses 102.4 Training an ANN 112.5 Forward Propagation 122.5.1 Forward Propagation from Input to Hidden Layer 132.6 Back Propagation 132.6.1 Back Propagation for a Neuron 132.6.2 Back Propagation – from Output to Hidden Layer 152.6.3 Back Propagation – from Hidden Layer to Input 162.7 Updating Weights 172.8 ANN with Input Bias 172.9 A Simple Algorithm for ANN Training 182.10 Computational Complexity of ANN Training 182.11 Normalization of ANN Inputs and Outputs 192.12 Concluding Remarks 202.13 Exercises for Chapter 2 203 ANNs for Optimized Prediction 233.1 Introduction 233.2 Selection of ANN Inputs 243.3 Selection of ANN Outputs 243.4 Construction of Hidden Layers 253.5 Case Study 1: Sleep-Study Example 253.5.1 Using Matrices for ANN Training 263.5.2 Forward Propagation 283.5.3 Back Propagation 283.5.4 Updating Weights 293.5.5 Forward Propagation with New Weights 293.5.6 Back Propagation with New Weights 303.5.7 Using Normalized Input and Output Values 313.5.8 Reducing Errors During Training 343.5.9 Implementation of Sleep-Study ANN in Python 343.5.10 Implementation of Sleep-Study ANN in Matlab 373.6 Case Study 2: Prediction of Bike Rentals 413.6.1 Python Script for Bike Rentals Using an ANN 413.6.2 Matlab Script for Bike Rentals Using an ANN 463.7 Concluding Remarks 483.8 Exercises for Chapter 3 484 ANNs for Financial Stock Trading 514.1 Introduction 514.2 Programs that Buy and Sell Stocks 514.3 Technical Indicators 514.3.1 Simple Moving Average 524.3.2 Momentum 534.3.3 Exponential Moving Average 544.3.4 Bollinger Bands 544.4 A Simple Algorithmic Trading Policy 554.5 A Simple ANN for Algorithmic Stock Trading 574.5.1 ANN Inputs and Outputs 574.5.2 ANN Architecture 584.6 Python Script for Stock Trading Using an ANN 594.7 Matlab Script for Stock Trading Using an ANN 634.8 Concluding Remarks 654.9 Exercises for Chapter 4 655 ANNs for Alzheimer’s Disease Prognosis 675.1 Introduction 675.2 Alzheimer’s Disease 675.3 A Simple ANN for AD Prognosis 685.4 Python Script for AD Prognosis Using an ANN 715.5 Matlab Script for AD Prognosis Using an ANN 755.6 Concluding Remarks 805.7 Exercises for Chapter 5 816 ANNs for Natural Language Processing 836.1 Introduction 836.2 Impact of Text Messages on Stock Markets 846.3 A Simple ANN for NLP 856.3.1 ANN Inputs and Outputs 856.3.2 Keywords 856.3.3 Formation of Training Data 866.3.4 ANN Architecture 886.4 Python Script for NLP Using an ANN 896.5 Matlab Script for NLP Using an ANN 926.6 Concluding Remarks 966.7 Exercises for Chapter 6 977 Convolutional Neural Networks 997.1 Introduction 997.1.1 Training CNNs 1007.2 Variations of CNNs 1017.3 Applications of CNNs 1017.4 CNN Components 1027.5 A Numerical Example of a CNN 1027.6 Computational Cost of CNN Training 1087.7 Concluding Remarks 1127.8 Exercises for Chapter 7 1128 CNNs for Optical Character Recognition 1158.1 Introduction 1158.2 A Simple CNN for OCR 1158.3 Organization of Training and Reference Files 1178.4 Python Script for OCR Using a CNN 1198.5 Matlab Script for OCR Using a CNN 1248.6 Concluding Remarks 1308.7 Exercises for Chapter 8 1309 CNNs for Speech Recognition 1339.1 Introduction 1339.2 A Simple CNN for Speech Recognition 1349.3 Organization of Training and Reference Files 1369.4 Python Script for Speech Recognition Using a CNN 1389.5 Matlab Script for Speech Recognition Using a CNN 1449.6 Concluding Remarks 1509.7 Exercises for Chapter 9 15010 Recurrent Neural Networks 15110.1 Introduction 15110.2 One-to-One Single RNN Cell 15310.2.1 A Simple Alphabet and One-Hot Encoding 15610.2.2 Forward and Back Propagation 15710.3 A Numerical Example 15810.4 Multiple Hidden Layers 16310.5 Embedding Layer 16510.5.1 Forward and Back Propagation with Embedding 16710.5.2 A Numerical Example with Embedding 16810.6 Concluding Remarks 17210.7 Exercises for Chapter 10 17211 RNNs for Chatbot Implementation 17511.1 Introduction 17511.2 Many-to-Many RNN Architecture 17511.3 A Simple Chatbot 17611.4 Python Script for a Chatbot Using an RNN 17911.5 Matlab Script for a Chatbot Using an RNN 18311.6 Concluding Remarks 18811.7 Exercises for Chapter 11 18912 RNNs with Attention 19112.1 Introduction 19112.2 One-to-One RNN Cell with Attention 19112.3 Forward and Back Propagation 19312.4 A Numerical Example 19512.5 Embedding Layer 20012.6 A Numerical Example with Embedding 20212.7 Concluding Remarks 20712.8 Exercises for Chapter 12 20713 RNNs with Attention for Machine Translation 20913.1 Introduction 20913.2 Many-to-Many Architecture 21013.3 Python Script for Machine Translation by an RNN-Att 21113.4 Matlab Script for Machine Translation by an RNN-Att 21613.5 Concluding Remarks 22313.6 Exercises for Chapter 13 22314 Genetic Algorithms 22514.1 Introduction 22514.2 Genetic Algorithm Elements 22614.3 A Simple Algorithm for a GA 22714.4 An Example of a GA 23014.5 Convergence in GAs 23114.6 Concluding Remarks 23214.7 Exercises for Chapter 14 23215 GAs for Dietary Menu Selection 23515.1 Introduction 23515.2 Definition of the KP 23615.3 A Simple Algorithm for the KP 23815.4 Variations of the KP 23915.5 GAs for KP Solution 24015.6 Python Script for Dietary Menu Selection Using a GA 24215.7 Matlab Script for Dietary Menu Selection Using a GA 24515.8 Concluding Remarks 24815.9 Exercises for Chapter 15 24816 GAs for Drone Flight Control 25116.1 Introduction 25116.2 UAV Swarms 25116.3 UAV Flight Control 25216.4 A Simple GA for UAV Flight Control 25316.4.1 Virtual Force-Based Fitness Function 25416.4.2 FGA Progression 25516.4.3 Chromosome for FGA 25716.5 Python Script for UAV Flight Control Using a GA 26016.6 Matlab Script for UAV Flight Control Using a GA 26416.7 Concluding Remarks 27016.8 Exercises for Chapter 16 27117 GAs for Route Optimization 27317.1 Introduction 27317.2 Definition of the TSP 27417.3 A Simple Algorithm for the TSP 27617.4 Variations of the TSP 27717.5 GA Solution for the TSP 27717.6 Python Script for Route Optimization Using a GA 27917.7 Matlab Script for Route Optimization Using a GA 28417.8 Concluding Remarks 28717.9 Exercises for Chapter 17 28918 Evolutionary Methods 29118.1 Introduction 29118.2 Particle Swarm Optimization 29118.2.1 Applications of PSO 29218.2.2 PSO Operation 29318.2.3 Remarks for PSO 29818.3 Differential Evolution 29818.3.1 Different Versions of DE 29918.3.2 Applications of DE 29918.3.3 A Simple Algorithm for DE 29918.3.4 Numerical Example: Maximum of sinc by DE 30218.3.5 Remarks for DE 30518.4 Grammatical Evolution 30618.4.1 A Simple Algorithm for GE 30618.4.2 Definition of GE 30718.4.3 A Simple GA to Implement GE 31418.4.4 Remarks on GE 315Appendix A ANNs with Bias 317A.1 Introduction 317A.2 Training with Bias Input 317A.3 Forward Propagation 318A.3.1 Forward Propagation from Input to Hidden Layer 319A.3.2 Neuron Back Propagation with Bias Input 319Appendix B Sleep Study ANN with Bias 321B.1 Inclusion of Bias Term in ANN 321B.1.1 Inclusion of Bias in Matrices 321B.1.2 Forward Propagation with Biases 322Appendix C Back Propagation in a CNN 327Appendix D Back Propagation Through Time in an RNN 331D.1 Back Propagation in an RNN 331D.2 Embedding Layer 335Appendix E Back Propagation Through Time in an RNN with Attention 337E.1 Back Propagation in an RNN-Att 337E.2 Embedding Layer 340Bibliography 343Index 353

M. Ümit Uyar is a Professor at the City College of the City University of New York, USA. Dr. Uyar is an IEEE Fellow, author, co-author and co-editor of seven books, holder of seven U.S. patents, and developer of AI and game theory-based algorithms for applications in topology control in mobile networks and personalized cancer treatment.

• About the Author xiiiPreface xvAcknowledgments xviiAbout the Companion Website xix1 Introduction 11.1 Artificial Intelligence 11.2 A Historical Perspective 11.3 Principles of AI 21.4 Applications That Are Impossible Without AI 21.5 Organization of This Book 32 Artificial Neural Networks 72.1 Introduction 72.2 Applications of ANNs 72.3 Components of ANNs 82.3.1 Neurons 82.3.2 Sigmoid Activation Function 92.3.3 Rectilinear Activation Function 92.3.4 Weights of Synapses 102.4 Training an ANN 112.5 Forward Propagation 122.5.1 Forward Propagation from Input to Hidden Layer 132.6 Back Propagation 132.6.1 Back Propagation for a Neuron 132.6.2 Back Propagation – from Output to Hidden Layer 152.6.3 Back Propagation – from Hidden Layer to Input 162.7 Updating Weights 172.8 ANN with Input Bias 172.9 A Simple Algorithm for ANN Training 182.10 Computational Complexity of ANN Training 182.11 Normalization of ANN Inputs and Outputs 192.12 Concluding Remarks 202.13 Exercises for Chapter 2 203 ANNs for Optimized Prediction 233.1 Introduction 233.2 Selection of ANN Inputs 243.3 Selection of ANN Outputs 243.4 Construction of Hidden Layers 253.5 Case Study 1: Sleep-Study Example 253.5.1 Using Matrices for ANN Training 263.5.2 Forward Propagation 283.5.3 Back Propagation 283.5.4 Updating Weights 293.5.5 Forward Propagation with New Weights 293.5.6 Back Propagation with New Weights 303.5.7 Using Normalized Input and Output Values 313.5.8 Reducing Errors During Training 343.5.9 Implementation of Sleep-Study ANN in Python 343.5.10 Implementation of Sleep-Study ANN in Matlab 373.6 Case Study 2: Prediction of Bike Rentals 413.6.1 Python Script for Bike Rentals Using an ANN 413.6.2 Matlab Script for Bike Rentals Using an ANN 463.7 Concluding Remarks 483.8 Exercises for Chapter 3 484 ANNs for Financial Stock Trading 514.1 Introduction 514.2 Programs that Buy and Sell Stocks 514.3 Technical Indicators 514.3.1 Simple Moving Average 524.3.2 Momentum 534.3.3 Exponential Moving Average 544.3.4 Bollinger Bands 544.4 A Simple Algorithmic Trading Policy 554.5 A Simple ANN for Algorithmic Stock Trading 574.5.1 ANN Inputs and Outputs 574.5.2 ANN Architecture 584.6 Python Script for Stock Trading Using an ANN 594.7 Matlab Script for Stock Trading Using an ANN 634.8 Concluding Remarks 654.9 Exercises for Chapter 4 655 ANNs for Alzheimer’s Disease Prognosis 675.1 Introduction 675.2 Alzheimer’s Disease 675.3 A Simple ANN for AD Prognosis 685.4 Python Script for AD Prognosis Using an ANN 715.5 Matlab Script for AD Prognosis Using an ANN 755.6 Concluding Remarks 805.7 Exercises for Chapter 5 816 ANNs for Natural Language Processing 836.1 Introduction 836.2 Impact of Text Messages on Stock Markets 846.3 A Simple ANN for NLP 856.3.1 ANN Inputs and Outputs 856.3.2 Keywords 856.3.3 Formation of Training Data 866.3.4 ANN Architecture 886.4 Python Script for NLP Using an ANN 896.5 Matlab Script for NLP Using an ANN 926.6 Concluding Remarks 966.7 Exercises for Chapter 6 977 Convolutional Neural Networks 997.1 Introduction 997.1.1 Training CNNs 1007.2 Variations of CNNs 1017.3 Applications of CNNs 1017.4 CNN Components 1027.5 A Numerical Example of a CNN 1027.6 Computational Cost of CNN Training 1087.7 Concluding Remarks 1127.8 Exercises for Chapter 7 1128 CNNs for Optical Character Recognition 1158.1 Introduction 1158.2 A Simple CNN for OCR 1158.3 Organization of Training and Reference Files 1178.4 Python Script for OCR Using a CNN 1198.5 Matlab Script for OCR Using a CNN 1248.6 Concluding Remarks 1308.7 Exercises for Chapter 8 1309 CNNs for Speech Recognition 1339.1 Introduction 1339.2 A Simple CNN for Speech Recognition 1349.3 Organization of Training and Reference Files 1369.4 Python Script for Speech Recognition Using a CNN 1389.5 Matlab Script for Speech Recognition Using a CNN 1449.6 Concluding Remarks 1509.7 Exercises for Chapter 9 15010 Recurrent Neural Networks 15110.1 Introduction 15110.2 One-to-One Single RNN Cell 15310.2.1 A Simple Alphabet and One-Hot Encoding 15610.2.2 Forward and Back Propagation 15710.3 A Numerical Example 15810.4 Multiple Hidden Layers 16310.5 Embedding Layer 16510.5.1 Forward and Back Propagation with Embedding 16710.5.2 A Numerical Example with Embedding 16810.6 Concluding Remarks 17210.7 Exercises for Chapter 10 17211 RNNs for Chatbot Implementation 17511.1 Introduction 17511.2 Many-to-Many RNN Architecture 17511.3 A Simple Chatbot 17611.4 Python Script for a Chatbot Using an RNN 17911.5 Matlab Script for a Chatbot Using an RNN 18311.6 Concluding Remarks 18811.7 Exercises for Chapter 11 18912 RNNs with Attention 19112.1 Introduction 19112.2 One-to-One RNN Cell with Attention 19112.3 Forward and Back Propagation 19312.4 A Numerical Example 19512.5 Embedding Layer 20012.6 A Numerical Example with Embedding 20212.7 Concluding Remarks 20712.8 Exercises for Chapter 12 20713 RNNs with Attention for Machine Translation 20913.1 Introduction 20913.2 Many-to-Many Architecture 21013.3 Python Script for Machine Translation by an RNN-Att 21113.4 Matlab Script for Machine Translation by an RNN-Att 21613.5 Concluding Remarks 22313.6 Exercises for Chapter 13 22314 Genetic Algorithms 22514.1 Introduction 22514.2 Genetic Algorithm Elements 22614.3 A Simple Algorithm for a GA 22714.4 An Example of a GA 23014.5 Convergence in GAs 23114.6 Concluding Remarks 23214.7 Exercises for Chapter 14 23215 GAs for Dietary Menu Selection 23515.1 Introduction 23515.2 Definition of the KP 23615.3 A Simple Algorithm for the KP 23815.4 Variations of the KP 23915.5 GAs for KP Solution 24015.6 Python Script for Dietary Menu Selection Using a GA 24215.7 Matlab Script for Dietary Menu Selection Using a GA 24515.8 Concluding Remarks 24815.9 Exercises for Chapter 15 24816 GAs for Drone Flight Control 25116.1 Introduction 25116.2 UAV Swarms 25116.3 UAV Flight Control 25216.4 A Simple GA for UAV Flight Control 25316.4.1 Virtual Force-Based Fitness Function 25416.4.2 FGA Progression 25516.4.3 Chromosome for FGA 25716.5 Python Script for UAV Flight Control Using a GA 26016.6 Matlab Script for UAV Flight Control Using a GA 26416.7 Concluding Remarks 27016.8 Exercises for Chapter 16 27117 GAs for Route Optimization 27317.1 Introduction 27317.2 Definition of the TSP 27417.3 A Simple Algorithm for the TSP 27617.4 Variations of the TSP 27717.5 GA Solution for the TSP 27717.6 Python Script for Route Optimization Using a GA 27917.7 Matlab Script for Route Optimization Using a GA 28417.8 Concluding Remarks 28717.9 Exercises for Chapter 17 28918 Evolutionary Methods 29118.1 Introduction 29118.2 Particle Swarm Optimization 29118.2.1 Applications of PSO 29218.2.2 PSO Operation 29318.2.3 Remarks for PSO 29818.3 Differential Evolution 29818.3.1 Different Versions of DE 29918.3.2 Applications of DE 29918.3.3 A Simple Algorithm for DE 29918.3.4 Numerical Example: Maximum of sinc by DE 30218.3.5 Remarks for DE 30518.4 Grammatical Evolution 30618.4.1 A Simple Algorithm for GE 30618.4.2 Definition of GE 30718.4.3 A Simple GA to Implement GE 31418.4.4 Remarks on GE 315Appendix A ANNs with Bias 317A.1 Introduction 317A.2 Training with Bias Input 317A.3 Forward Propagation 318A.3.1 Forward Propagation from Input to Hidden Layer 319A.3.2 Neuron Back Propagation with Bias Input 319Appendix B Sleep Study ANN with Bias 321B.1 Inclusion of Bias Term in ANN 321B.1.1 Inclusion of Bias in Matrices 321B.1.2 Forward Propagation with Biases 322Appendix C Back Propagation in a CNN 327Appendix D Back Propagation Through Time in an RNN 331D.1 Back Propagation in an RNN 331D.2 Embedding Layer 335Appendix E Back Propagation Through Time in an RNN with Attention 337E.1 Back Propagation in an RNN-Att 337E.2 Embedding Layer 340Bibliography 343Index 353

M. Ümit Uyar is a Professor at the City College of the City University of New York, USA. Dr. Uyar is an IEEE Fellow, author, co-author and co-editor of seven books, holder of seven U.S. patents, and developer of AI and game theory-based algorithms for applications in topology control in mobile networks and personalized cancer treatment.

• About the Author xiiiPreface xvAcknowledgments xviiAbout the Companion Website xix1 Introduction 11.1 Artificial Intelligence 11.2 A Historical Perspective 11.3 Principles of AI 21.4 Applications That Are Impossible Without AI 21.5 Organization of This Book 32 Artificial Neural Networks 72.1 Introduction 72.2 Applications of ANNs 72.3 Components of ANNs 82.3.1 Neurons 82.3.2 Sigmoid Activation Function 92.3.3 Rectilinear Activation Function 92.3.4 Weights of Synapses 102.4 Training an ANN 112.5 Forward Propagation 122.5.1 Forward Propagation from Input to Hidden Layer 132.6 Back Propagation 132.6.1 Back Propagation for a Neuron 132.6.2 Back Propagation – from Output to Hidden Layer 152.6.3 Back Propagation – from Hidden Layer to Input 162.7 Updating Weights 172.8 ANN with Input Bias 172.9 A Simple Algorithm for ANN Training 182.10 Computational Complexity of ANN Training 182.11 Normalization of ANN Inputs and Outputs 192.12 Concluding Remarks 202.13 Exercises for Chapter 2 203 ANNs for Optimized Prediction 233.1 Introduction 233.2 Selection of ANN Inputs 243.3 Selection of ANN Outputs 243.4 Construction of Hidden Layers 253.5 Case Study 1: Sleep-Study Example 253.5.1 Using Matrices for ANN Training 263.5.2 Forward Propagation 283.5.3 Back Propagation 283.5.4 Updating Weights 293.5.5 Forward Propagation with New Weights 293.5.6 Back Propagation with New Weights 303.5.7 Using Normalized Input and Output Values 313.5.8 Reducing Errors During Training 343.5.9 Implementation of Sleep-Study ANN in Python 343.5.10 Implementation of Sleep-Study ANN in Matlab 373.6 Case Study 2: Prediction of Bike Rentals 413.6.1 Python Script for Bike Rentals Using an ANN 413.6.2 Matlab Script for Bike Rentals Using an ANN 463.7 Concluding Remarks 483.8 Exercises for Chapter 3 484 ANNs for Financial Stock Trading 514.1 Introduction 514.2 Programs that Buy and Sell Stocks 514.3 Technical Indicators 514.3.1 Simple Moving Average 524.3.2 Momentum 534.3.3 Exponential Moving Average 544.3.4 Bollinger Bands 544.4 A Simple Algorithmic Trading Policy 554.5 A Simple ANN for Algorithmic Stock Trading 574.5.1 ANN Inputs and Outputs 574.5.2 ANN Architecture 584.6 Python Script for Stock Trading Using an ANN 594.7 Matlab Script for Stock Trading Using an ANN 634.8 Concluding Remarks 654.9 Exercises for Chapter 4 655 ANNs for Alzheimer’s Disease Prognosis 675.1 Introduction 675.2 Alzheimer’s Disease 675.3 A Simple ANN for AD Prognosis 685.4 Python Script for AD Prognosis Using an ANN 715.5 Matlab Script for AD Prognosis Using an ANN 755.6 Concluding Remarks 805.7 Exercises for Chapter 5 816 ANNs for Natural Language Processing 836.1 Introduction 836.2 Impact of Text Messages on Stock Markets 846.3 A Simple ANN for NLP 856.3.1 ANN Inputs and Outputs 856.3.2 Keywords 856.3.3 Formation of Training Data 866.3.4 ANN Architecture 886.4 Python Script for NLP Using an ANN 896.5 Matlab Script for NLP Using an ANN 926.6 Concluding Remarks 966.7 Exercises for Chapter 6 977 Convolutional Neural Networks 997.1 Introduction 997.1.1 Training CNNs 1007.2 Variations of CNNs 1017.3 Applications of CNNs 1017.4 CNN Components 1027.5 A Numerical Example of a CNN 1027.6 Computational Cost of CNN Training 1087.7 Concluding Remarks 1127.8 Exercises for Chapter 7 1128 CNNs for Optical Character Recognition 1158.1 Introduction 1158.2 A Simple CNN for OCR 1158.3 Organization of Training and Reference Files 1178.4 Python Script for OCR Using a CNN 1198.5 Matlab Script for OCR Using a CNN 1248.6 Concluding Remarks 1308.7 Exercises for Chapter 8 1309 CNNs for Speech Recognition 1339.1 Introduction 1339.2 A Simple CNN for Speech Recognition 1349.3 Organization of Training and Reference Files 1369.4 Python Script for Speech Recognition Using a CNN 1389.5 Matlab Script for Speech Recognition Using a CNN 1449.6 Concluding Remarks 1509.7 Exercises for Chapter 9 15010 Recurrent Neural Networks 15110.1 Introduction 15110.2 One-to-One Single RNN Cell 15310.2.1 A Simple Alphabet and One-Hot Encoding 15610.2.2 Forward and Back Propagation 15710.3 A Numerical Example 15810.4 Multiple Hidden Layers 16310.5 Embedding Layer 16510.5.1 Forward and Back Propagation with Embedding 16710.5.2 A Numerical Example with Embedding 16810.6 Concluding Remarks 17210.7 Exercises for Chapter 10 17211 RNNs for Chatbot Implementation 17511.1 Introduction 17511.2 Many-to-Many RNN Architecture 17511.3 A Simple Chatbot 17611.4 Python Script for a Chatbot Using an RNN 17911.5 Matlab Script for a Chatbot Using an RNN 18311.6 Concluding Remarks 18811.7 Exercises for Chapter 11 18912 RNNs with Attention 19112.1 Introduction 19112.2 One-to-One RNN Cell with Attention 19112.3 Forward and Back Propagation 19312.4 A Numerical Example 19512.5 Embedding Layer 20012.6 A Numerical Example with Embedding 20212.7 Concluding Remarks 20712.8 Exercises for Chapter 12 20713 RNNs with Attention for Machine Translation 20913.1 Introduction 20913.2 Many-to-Many Architecture 21013.3 Python Script for Machine Translation by an RNN-Att 21113.4 Matlab Script for Machine Translation by an RNN-Att 21613.5 Concluding Remarks 22313.6 Exercises for Chapter 13 22314 Genetic Algorithms 22514.1 Introduction 22514.2 Genetic Algorithm Elements 22614.3 A Simple Algorithm for a GA 22714.4 An Example of a GA 23014.5 Convergence in GAs 23114.6 Concluding Remarks 23214.7 Exercises for Chapter 14 23215 GAs for Dietary Menu Selection 23515.1 Introduction 23515.2 Definition of the KP 23615.3 A Simple Algorithm for the KP 23815.4 Variations of the KP 23915.5 GAs for KP Solution 24015.6 Python Script for Dietary Menu Selection Using a GA 24215.7 Matlab Script for Dietary Menu Selection Using a GA 24515.8 Concluding Remarks 24815.9 Exercises for Chapter 15 24816 GAs for Drone Flight Control 25116.1 Introduction 25116.2 UAV Swarms 25116.3 UAV Flight Control 25216.4 A Simple GA for UAV Flight Control 25316.4.1 Virtual Force-Based Fitness Function 25416.4.2 FGA Progression 25516.4.3 Chromosome for FGA 25716.5 Python Script for UAV Flight Control Using a GA 26016.6 Matlab Script for UAV Flight Control Using a GA 26416.7 Concluding Remarks 27016.8 Exercises for Chapter 16 27117 GAs for Route Optimization 27317.1 Introduction 27317.2 Definition of the TSP 27417.3 A Simple Algorithm for the TSP 27617.4 Variations of the TSP 27717.5 GA Solution for the TSP 27717.6 Python Script for Route Optimization Using a GA 27917.7 Matlab Script for Route Optimization Using a GA 28417.8 Concluding Remarks 28717.9 Exercises for Chapter 17 28918 Evolutionary Methods 29118.1 Introduction 29118.2 Particle Swarm Optimization 29118.2.1 Applications of PSO 29218.2.2 PSO Operation 29318.2.3 Remarks for PSO 29818.3 Differential Evolution 29818.3.1 Different Versions of DE 29918.3.2 Applications of DE 29918.3.3 A Simple Algorithm for DE 29918.3.4 Numerical Example: Maximum of sinc by DE 30218.3.5 Remarks for DE 30518.4 Grammatical Evolution 30618.4.1 A Simple Algorithm for GE 30618.4.2 Definition of GE 30718.4.3 A Simple GA to Implement GE 31418.4.4 Remarks on GE 315Appendix A ANNs with Bias 317A.1 Introduction 317A.2 Training with Bias Input 317A.3 Forward Propagation 318A.3.1 Forward Propagation from Input to Hidden Layer 319A.3.2 Neuron Back Propagation with Bias Input 319Appendix B Sleep Study ANN with Bias 321B.1 Inclusion of Bias Term in ANN 321B.1.1 Inclusion of Bias in Matrices 321B.1.2 Forward Propagation with Biases 322Appendix C Back Propagation in a CNN 327Appendix D Back Propagation Through Time in an RNN 331D.1 Back Propagation in an RNN 331D.2 Embedding Layer 335Appendix E Back Propagation Through Time in an RNN with Attention 337E.1 Back Propagation in an RNN-Att 337E.2 Embedding Layer 340Bibliography 343Index 353