Multimodal Intelligent Sensing in Modern Applications

Masood Ur Rehman, PhD, MSc, is a Senior Lecturer at the James Watt School of Engineering, University of Glasgow, UK and leads the Antennas & Radio-wave Propagation group. He received his MSc and PhD in Electronic Engineering from Queen Mary University of London, UK, in 2006 and 2010, respectively. Ahmed Zoha, PhD, MSc, is a Senior Lecturer at the James Watt School of Engineering, University of Glasgow, UK and leads the Distributed Learning and Intelligence group. He received his PhD degree in Electrical and Electronic Engineering from the 5G Innovation Centre at the University of Surrey, UK, and his MSc in Communication Engineering from the Chalmers University of Technology, Sweden. Muhammad Ali Jamshed, PhD, MSc, is with University of Glasgow, UK, since 2021. He is a visiting Research Fellow at the University of Sussex. He is endorsed by the Royal Academy of Engineering under exceptional talent category and was nominated for the Departmental Prize for Excellence in Research in 2019 at the University of Surrey. Naeem Ramzan, PhD, is a Full Professor in Computing Engineering and Chair of Affective and Human Computing for Smart Environment Research Centre and Co-lead of Visual Communication Cluster in AVCN at the University of the West of Scotland, Paisley, UK. He received his PhD in Electronic Engineering from Queen Mary University of London, UK in 2008.

• ContentsAbout the Editors xvList of Contributors xixPreface xxiii1 Advances in Multi-modal Intelligent Sensing 1Masood Ur Rehman, Muhammad Ali Jamshed, and Tahera Kalsoom1.1 Multi-modal Intelligent Sensing 11.2 Sensors for Multi-modal Intelligent Sensing 31.2.1 Sensor Types 31.2.2 Integration of Multiple Sensor Types for Enhanced Sensing Capabilities 51.2.2.1 Advantages of Multiple Sensor Integration 51.2.2.2 Key Considerations for Multiple Sensor Integration 61.2.2.3 Concurrent Data Acquisition Methods 91.2.2.4 Data Analysis Tools for Multi-modal Sensing 111.2.2.5 Considerations for Data Fusion and Synchronization 131.3 Applications of Multi-modal Intelligent Sensing 141.3.1 Healthcare and Medical Monitoring 141.3.2 Automotive and Transportation Systems 151.3.3 Environmental Monitoring and Conservation 161.3.4 Smart Cities and Infrastructure Management 171.3.5 Industrial Automation 181.4 Challenges and Opportunities in Multi-modal Sensing 181.4.1 Data Security and Privacy 191.4.2 Interoperability and Standardization 191.4.3 Energy Efficiency and Power Management 201.4.4 Coverage 211.4.5 Summary 21References 222 Antennas for Wireless Sensors 29Abdul Jabbar, Muhammad Ali Jamshed, and Masood Ur Rehman2.1 Wireless Sensors: Definition and Architecture 292.1.1 Wireless Sensor Node Architecture 302.1.2 Operating Systems 322.1.3 Classification of Wireless Sensors 322.2 Multi-modal Wireless Sensing 342.3 Antennas: The Sensory Gateway for Wireless Sensors 352.4 Fundamental Antenna Parameters 362.4.1 Bandwidth and Operating Frequency 362.4.2 Gain 372.4.3 Radiation Pattern 372.4.4 Polarization 382.5 Key Operating Frequency Bands for Sensing Antennas 392.6 Fabrication Methods for Sensing Antennas 402.6.1 Printed Circuit Board (PCB) Antennas 402.6.2 On-Chip and Integrated Antenna Fabrication 412.6.3 Stitching and Embroidery for Flexible Textile Antennas 412.7 Antenna Types for Wireless Sensing Networks 422.7.1 Flexible Antennas 432.7.2 Omnidirectional Antennas 452.7.3 Directional Antennas 462.8 Advantages of Electronic Beamsteering Antennas in Sensing Systems 462.9 Summary 49References 493 Sensor Design for Multimodal Environmental Monitoring 55Muhammad Ali Jamshed, Bushra Haq, Syed Ahmed Shah, Kamran Ali, Qammer H. Abbasi, Mumraiz Khan Kasi, and Masood Ur Rehman3.1 Environment and Forests 563.2 Methods to Combat Deforestation 563.2.1 Combating Deforestation Using Wireless Sensor Networks 573.2.2 Sensor Types for Combating Deforestation 583.3 Design of a WSN to Combat Deforestation 593.3.1 Stage 1: System Requirements 593.3.1.1 Key Performance Indicators 623.3.2 Stage 2: Architecture 633.3.3 Stage 3: System Implementation 653.3.3.1 Type of Sensors 653.3.3.2 Processing Boards 663.3.3.3 Communication Modules 673.3.3.4 Batteries 673.3.3.5 Energy Harvesting Circuit and Equipment 693.3.3.6 Weather Protection 703.3.3.7 Wireless Communication Link 713.3.3.8 Data Processing Algorithms 743.4 Summary 76References 764 Wireless Sensors for Multi-modal Health Monitoring 81Nadeem Ajum, Shagufta Iftikhar, Tahera Kalsoom, and Masood Ur Rehman4.1 Wearable Sensors 824.1.1 Electrocardiography (ECG) Sensors 834.1.2 Electroencephalography (EEG) Sensors 834.1.3 Electrooculography (EOG) Sensors 844.1.4 Electrodermal Activity (EDA) Sensors 864.1.5 Respiratory (RESP) Sensors 864.1.6 Motion Sensors 864.1.7 Temperature (TEMP) Sensors 874.1.8 Pressure Sensors 874.1.9 Hydration Sensors 884.1.10 Lactate Sensors 884.1.11 Photoplethysmography (PPG) Sensors 894.1.12 Continuous Glucose Monitoring (CGM) Sensors 894.2 Flexible Sensors 894.3 Multi-modal Healthcare Sensing Devices 904.3.1 Wearable Sensing Devices for Healthcare 904.3.1.1 Wearable Devices in Detection 904.3.1.2 Wearable Devices in Monitoring 924.3.1.3 Wearable Devices in Rehabilitation 934.3.1.4 Wearable Devices in Personalized Medicine 944.3.1.5 Wearable Devices in Skin Patches 944.3.1.6 Wearable Devices for Body Fluid Monitoring 954.3.1.7 Wearable Devices in Monitoring Body Temperature 964.3.1.8 Wearable Devices as Contact Lens 964.3.1.9 Wearable Devices in Daily Use Objects 974.3.2 Implantable Sensing Devices for Healthcare 974.3.2.1 Implantable Cardioverter Defibrillators 984.3.2.2 Bioinks and 3D Print Implants 984.3.2.3 Deep Brain Stimulation 994.3.2.4 Biosensor Tattoos 994.4 AI Methods for Multi-modal Healthcare Systems 1004.4.1 Supervised Learning 1004.4.2 Unsupervised Learning 1014.4.3 Semi-supervised Learning 1014.4.4 Reinforcement Learning 1024.5 Summary 102References 1035 Sensor Design for Industrial Automation 109Abdul Jabbar, Tahera Kalsoom, and Masood Ur Rehman5.1 Multimodal Sensing in Industrial Automation 1095.1.1 IIoT and Multimodal Sensing 1115.1.2 Advanced Robotics 1135.1.3 Big Data Analytics 1145.1.4 Cloud Computing 1145.1.5 Artificial Intelligence 1155.1.6 Augmented Reality 1165.2 Sensors for Realizing Industrial Automation 1165.2.1 RF Sensors 1175.2.2 Vision Sensors 1185.2.3 Localization and Tracking Sensors 1195.2.4 Infrared Sensors 1195.2.5 Proximity Sensors 1195.2.6 IMU Sensors 1205.2.7 Level Sensors 1205.2.8 Temperature Sensors 1205.2.9 Pressure Sensors 1215.3 Design Considerations for Effective Multimodal Industrial Automation 1215.3.1 Design of AI-Assisted Multimodal Sensing 1225.3.2 Design of Radar Sensing Networks 1235.3.2.1 Transmitter and Receiver Antennas 1235.3.2.2 Data Collection and Interface 1235.3.2.3 Signal Processing 1245.3.2.4 Housing and Enclosure 1245.4 Challenges and Opportunities of Multimodal Sensing in Industrial Automation 1245.5 Summary 126References 1266 Hybrid Neuromorphic-Federated Learning for Activity Recognition Using Multi-modal Wearable Sensors 133Ahsan Raza Khan, Habib Ullah Manzoor, Fahad Ayaz, Muhammad Ali Imran, and Ahmed Zoha6.1 Multi-modal Human Activity Recognition 1346.2 Machine Learning Methods in Multi-modal Human Activity Recognition 1376.2.1 Centralized Learning-based HAR Systems 1376.2.2 Federated Learning-based HAR Systems 1386.3 System Model 1396.3.1 Federated Learning Framework 1406.3.2 Spiking Neural Network 1416.3.3 Proposed S-LSTM Model 1446.4 Simulation Setup 1466.4.1 Dataset Description 1466.4.1.1 UCI Dataset 1476.4.1.2 Real-World Dataset 1486.4.2 Performance Metrics 1496.5 Results and Discussion 1506.5.1 UCI Results 1516.5.2 Real-World Dataset Results 1546.5.3 Energy Efficiency Comparison 1576.5.4 Personalized Model Comparison 1596.6 Summary 159References 1617 Multi-modal Beam Prediction for Enhanced Beam Management in Drone Communication Networks 165Iftikhar Ahmad, Ahsan Raza Khan, Rao Naveed Bin Rais, Muhammad Ali Imran, Sajjad Hussain, and Ahmed Zoha7.1 Drone Communication 1667.2 Beam Management 1677.3 System Model 1687.3.1 Problem Formulation for Beam Prediction 1707.3.2 Proposed Stacked Vision-Assisted Beam Prediction Model 1707.4 Simulation and Analysis 1717.4.1 Description of the Dataset 1737.4.2 Configuration for Simulation 1737.4.2.1 YOLO-v5 Training Process 1747.4.3 Results and Analysis 1757.5 Summary 178References 1788 Multi-modal-Sensing System for Detection and Tracking of Mind Wandering 181Sara Khosravi, Haobo Li, Ahsan Raza Khan, Ahmed Zoha, and Rami Ghannam8.1 Mind Wandering 1828.2 Multi-modal Wearable Systems for Mind-Wandering Detection and Monitoring 1848.2.1 Wearable Eye Trackers for Gaze Measurements 1858.2.2 Wearable GSR and PPG Sensors for Physiological Measurements 1868.3 Design of Multi-modal Wearable System 1878.3.1 Selection of Sensor 1878.3.2 Selection of Participant Groups and Testing Environment 1888.3.3 Data Collection Process 1898.3.4 Machine Learning and Multisensory Fusion 1908.4 Results and Discussion 1948.5 Summary 197References 1979 Adaptive Secure Multi-modal Telehealth Patient-Monitoring System 201Muhammad Hanif , Ehsan Ullah Munir, Muhammad Maaz Rehan, Saima Gulzar Ahmad, Tassawar Iqbal, Nasira Kirn, Kashif Ayyub, and Naeem Ramzan9.1 Healthcare Systems 2029.1.1 Traditional Healthcare Systems 2039.1.2 Multi-modal Telehealth Systems 2039.2 Security in Healthcare Systems 2059.2.1 Prevailing Techniques for Secure Telehealth 2059.2.2 Challenges in Ensuring Healthcare Security 2089.2.3 Strategies to Enhance Healthcare Data Security 2099.2.4 Key Security Features for Telehealth Systems 2109.2.4.1 Encryption 2109.2.4.2 Authentication 2119.2.4.3 Access Control 2119.2.4.4 Audit Trails 2119.2.4.5 Data Integrity Checks 2119.2.4.6 Secure Communication Protocols 2119.2.4.7 Security Audits and Penetration Testing 2119.2.4.8 Cyber Resilience 2129.2.4.9 Zero-Trust-Based Micro-segmentation 2129.3 Blockchain-Powered ZTS for Enhanced Security of Telehealth Systems 2139.3.1 Zero-Trust Security 2139.3.2 Blockchain 2149.4 Cyber-resilient Telehealth-Enabled Patient Management System 2179.4.1 Assessment and Planning 2189.4.2 Infrastructure Setup 2199.4.3 Security Controls Implementation 2199.4.4 Training and Awareness 2199.4.5 Advantages and Limitations 2219.5 Summary 222References 22210 Advances in Multi-modal Remote Infant Monitoring Systems 227Najia Saher, Omer Riaz, Muhammad Suleman, Dost Muhammad Khan, Nasira Kirn, Sana Ullah Jan, Rizwan Shahid, Hassan Rabah, and Naeem Ramzan10.1 Remote Patient Monitoring 22810.2 Remote Infant Monitoring (RIM) System 22910.2.1 Contactless Remote Patient Monitoring (RPM) Systems 23010.2.2 Contact-Based Remote Patient Monitoring (RPM) Systems 23010.3 Disease-Specific Remote Infant Monitoring Systems 23210.3.1 Respiration and Apnea Monitoring System 23210.3.1.1 Emerging Sensing Techniques for Respiratory Diseases 23310.3.2 Heart and Blood-Related Diseases Monitoring Systems 23810.3.2.1 Emerging Sensing Techniques for Heart and Blood Diseases 23810.3.3 Infant Monitoring Systems for Various Diseases 24110.4 Challenges in Remote Infant Monitoring Systems 24110.5 Summary 245References 24611 Balancing Innovation with Ethics: Responsible Development of Multi-modal Intelligent Tutoring Systems 253Romina Soledad Albornoz-De Luise, Pablo Arnau-González, Ana Serrano-Mamolar, Sergi Solera-Monforte, and Yuyan Wu11.1 Intelligent Tutoring Systems and Ethical Considerations 25311.2 The Promise and Perils of ITS 25511.2.1 Advantages of ITS in Education 25511.2.1.1 Personalized Learning 25511.2.1.2 Increased Accessibility 25611.2.1.3 Continuous Assessment and Feedback 25611.2.2 Potential Risks and Challenges 25611.2.2.1 Bias in Algorithms 25711.2.2.2 Privacy Concerns 25711.2.2.3 Socioeconomic Disparities in Access 25711.2.2.4 Dependency on Technology 25811.3 Ethical Frameworks for ITS 25811.3.1 Utilitarian Perspective 25811.3.2 Deontological Perspective 25911.3.3 Virtue Ethics 26011.4 Bias and Fairness in ITS 26111.5 Privacy and Security Concerns 26311.5.1 Self-Reported Sources 26311.5.2 Captured Data Sources 26411.6 Socioeconomic Disparities in Access 26511.7 Dependency on Technology 26711.8 Summary 268References 26912 Road Ahead for Multi-modal Intelligent Sensing in the Deep Learning Era 275Ahmed Zoha, Naeem Ramzan, Muhammad Ali Jamshed, and Masood Ur Rehman12.1 Future Challenges and Perspectives for Intelligent Multi-modal Sensing 27612.1.1 Semantic Gaps and Cross-modality Representation 27612.1.2 Concept Drift and Data Quality 27812.1.3 Computational Demands and Model Scalability 27912.1.4 Interpretability 28012.1.5 Ethical Considerations 28112.2 Summary 282References 282Index 285