Generative AI for Connected and Autonomous Vehicles

Kyungtae Han, PhD, is a Senior Principal Researcher at Toyota Motor North America and was previously a Senior Research Scientist at Intel Labs. With over 90 publications and more than 100 patents spanning generative AI, connected and automated vehicles, cyber-physical systems, and edge computing, his work has earned the SAE Vincent Bendix Award, IEEE Best Application Award, and Intel Achievement Award.

• Preface xxxvAcknowledgments xxxixHow to Read This Book xliStatement on the Use of Generative AI xlvCompanion Resources xlviiAcronyms xlixI Foundations of Generative Intelligence for CAVs 11 Introduction to Generative AI for CAV Systems 31.1 Introduction: The Promise of Generative AI in CAVs 41.2 The Paradigm Shift: Generative vs. Discriminative AI 51.3 Transformative Capabilities in CAVs 101.4 Generative Model Families 131.5 The Generative AI Ecosystem 211.6 Summary and Roadmap 292 Large Language Models: Architecture and Operation 392.1 Introduction 402.3 The Rise of Transformers in Language Modeling 472.4 The LLM Generation Pipeline 512.5 Stage 1: Tokenization and Embeddings 542.6 Stage 2: Positional Encoding 572.7 Stage 3: Attention and Contextualization 602.8 Stage 4: Refinement and Stacking 662.9 Stage 5: Output and Sampling 712.10 Model Limitations and Comparative Landscape 742.11 Summary 81II Multimodal Representation and World Grounding 863 Diffusion and Flow Matching 873.1 Introduction: Why Diffusion for CAV Systems 883.2 Conceptual Foundations 903.3 Mathematical Foundations 923.4 Architectural Choices in Diffusion and Transport Models 1013.5 Applications in CAV Systems 1043.6 Limitations and Considerations 1073.7 Summary 1084 GANs, VAEs, and Autoregressive Models 1114.1 Introduction 1124.2 Generative Adversarial Networks 1144.3 Variational Autoencoders 1234.4 Autoregressive Models for Sequential Prediction 1344.5 Comparative Analysis 1424.6 Summary 1435 Vision-Language Models for Scene Understanding 1495.1 Introduction 1505.2 Architectural Principles of VLMs 1515.3 Stage I: Visual Feature Extraction 1555.4 Stage II: Vision–Language Adaptation 1605.5 Stage III: Integration into the Language Model 1645.6 Training Methodologies for VLMs 1695.7 Deployment in Connected and Autonomous Vehicles 1785.8 Case Study: Semantic Filtering of V2X Messages 1815.9 Summary 1856 Video Large Language Models for Temporal Reasoning in Driving Environments 1896.1 Introduction 1906.2 Fundamental Design Principles 1916.3 Stage I: Spatio-Temporal Encoding 1946.4 Stage II: Spatio-Temporal Fusion 1986.5 Stage III: Integration with Language Models 2016.6 Token Compression Techniques 2076.7 Representative Architectures 2126.8 Applications in Connected and Autonomous Vehicles 2156.9 Summary 217III Language-Based Interaction, Memory, and Tool Use 2207 Prompt Engineering 2217.1 Introduction and Motivation 2227.2 Foundations of Prompt Engineering 2247.3 Design Principles for Effective Prompting 2287.4 Prompt Engineering in Practice 2347.5 Prompt Fragility and Safety Considerations in CAV Systems 2447.6 Summary 2458 Retrieval-Augmented Generation for Context Grounding 2498.1 Why RAG Matters in Connected and Autonomous Vehicles 2508.2 Architectural Foundations of RAG 2548.3 RAG vs. Rule-Based Information Systems in CAVs 2588.4 Mathematical Modeling of RAG 2608.5 Practical RAG Implementation for CAV Systems 2638.6 Real-Time Feasibility and System Constraints 2678.7 Applied Use Cases of RAG in CAV Systems 2698.8 Summary 2729 Function Calling and Tool Use 2759.1 Introduction: Why CAVs Need Function Calling 2769.2 System Architecture of Function Calling 2799.3 Theoretical Foundations of Function Calling 2839.4 Function Calling in Practice 2889.5 Function Schema Library and Design Principles 2949.6 Summary 302IV Agentic Intelligence and Coordination 30610 GenAI-Enhanced Agent Architectures 30710.1 Introduction 30810.2 Classical Agent Foundations and Architecture 31010.3 GenAI Enhancement Concepts 31310.4 Mathematical Formalization 32010.5 ReAct Framework 32310.6 Individual Agent Applications in CAVs 32610.7 Limitations and Challenges of GenAI-Enhanced Agents 32810.8 Summary 33211 GenAI-Enhanced Multi-Agent Systems 33511.1 Introduction 33611.2 Multi-Agent Systems Foundations and Architecture 33711.3 GenAI Enhancement Concepts 34211.4 Mathematical Formalization 34811.5 GenAI-Enhanced Applications in CAVs 35411.6 Implementation Considerations and Practical Challenges 35611.7 Summary 36112 Orchestration Frameworks for Agentic AI 36512.1 Introduction 36612.2 Graph-Based Orchestration with LangGraph 36812.3 Dialogue-Based Orchestration with AutoGen 37812.4 Framework Comparison and Selection 38712.5 Summary 390V Communication Protocols and Collective Intelligence 39213 Communication Protocols for Agentic AI 39313.1 Introduction 39413.2 Protocol Classification Framework 39613.3 Protocol Evolution and Development 40213.4 Protocol Examples and CAV Applications 40313.5 Summary 40614 Model Context Protocol for Agentic Systems 40914.1 Introduction 41014.2 Context Provision in CAV Systems 41114.3 MCP Architecture 41514.4 Server Features 42314.5 Client Features 42814.6 Mathematical Foundations 43214.7 Practical Implementation 43514.8 MCP in CAV Systems 43914.9 Summary 44415 Agent-to-Agent Protocols for Collaborative Intelligence 44715.1 Introduction 44815.2 A2A Architecture 44915.3 Mathematical Foundations for Agent Collaboration 45815.4 Security and Trust 46415.5 A2A in CAV Systems 46615.6 Limitations and Challenges 47015.7 Summary 47316 Integrated CAV Agent Architectures with MCP and A2A 47716.1 Introduction 47816.2 Protocol Comparison and Decision Framework 47916.3 System Architecture and Mathematical Framework 48216.4 Implementation and Execution Patterns 48616.5 Summary 495VI Adaptation and Deployment 49717 Supervised Fine-Tuning 49917.1 The Necessity of Fine-Tuning in CAV Applications 50017.2 From Foundation Models to Specialization 50217.3 Architectural Perspective on Fine-Tuning 50417.4 Mathematical Formulation of Fine-Tuning 50717.5 Practical Fine-Tuning 50917.6 Parameter-Efficient Fine-Tuning Methods 51517.7 Use Cases and Deployment Modes 52417.8 Summary 52818 Reinforcement Learning Fine-Tuning 53118.1 Introduction 53218.2 Mathematical Foundation 53618.3 Policy Optimization Methods in Detail 54418.4 RLHF Methodology 55918.5 Summary 56219 Knowledge Distillation for Edge Deployment 56719.1 Introduction: The Deployment Challenge 56819.2 The Teacher-Student Paradigm 56819.3 Core Knowledge Distillation Mechanisms 57019.4 Advanced Distillation Techniques 57719.5 Deployment Considerations for CAV Applications 58319.6 Summary 585VII Validation and Safety 58820 Simulation Platforms for GenAI-Driven CAVs 58920.1 Introduction: Why Simulation Is Indispensable for GenAI-CAVs 59020.2 Simulation as a System-Level Substrate for GenAI-CAVs 59220.3 Scenario Generation and Coverage Expansion 59520.4 Simulation Platforms for GenAI-CAV Development 59720.5 Scaling Simulation: From Single-Agent to Multi-Agent Systems 60220.6 Simulation–Evaluation Boundary 60520.7 Summary 60621 Evaluation of Generative AI Systems in CAVs 60921.1 Introduction: The Role of Evaluation in GenAI-Driven CAVs 61021.2 Evaluation Dimensions for GenAI Systems in CAV Applications 61121.3 Fundamental Evaluation Metrics and Methodologies 61521.4 Evaluation of Vision-Language Models for CAV Applications 63721.5 Evaluation of Retrieval-Augmented Generation Systems 64121.6 Multi-Turn Dialogue and Conversational Agents 64821.7 Summary 65022 Ethical and Safety Considerations 65522.1 Introduction: The Ethical Stakes of Generative AI in CAVs 65622.2 Foundational Ethical Principles and Challenges 65822.3 Case Studies: Ethical Challenges in Practice 67622.4 Regulatory and Safety Frameworks 67922.5 Ethical Engineering Methods 68322.6 Broader Environmental and Societal Considerations 68922.7 Summary 690Bibliography 692