Detecting, Troubleshooting, and Preventing Congestion in Storage Networks

Paresh Gupta, CCIE No. 36645, has almost two decades of experience in the computer industry. Currently, as a senior leader of Technical Marketing Engineering for Cisco, he drives the technical and market evolution of products, technologies, and solutions such as SAN Analytics, Nexus Dashboard, UCS, MDS, and Nexus switches. He has been testing and validating congestion in storage networks for many years. In his multiple roles, he has invented/patented many ideas, developed many features, and trained thousands of people in sales, partner, and customer communities. Paresh is the creator of the fullblown traffic monitoring apps for Cisco UCS Servers (UTM) and MDS switches (MTM). Hundreds of organizations use Paresh’s apps in production around the world.  Edward Mazurek, CCIE No. 6448, has more than 40 years of experience in the computer networking industry. The first 18 were with IBM, supporting products such as Virtual Machine (VM) and VTAM. He has spent the last 22+ years with the Cisco TAC. As a principal engineer, he supports data center networking technologies, including storage-area networking, Fibre Channel, and FCoE in the MDS, UCS, and Nexus 5000, 6000, 7000, and 9000 series. He holds two CCIEs—SNA/IP Integration (2000) and Storage Area Networking (SAN)—and is presently a CCIE Emeritus. Ed has spearheaded the congestion-handling mechanisms on the Cisco MDS 9000 switches and Nexus 9000 switches. Based on his deep understanding, Ed developed an app that is being used by other Cisco engineers and continues to be in high demand by various partners. In addition to inventing multiple features on Cisco products, Ed holds multiple patents in the field of network congestion.

• Introduction xxxiiChapter 1 Introduction to Congestion in Storage Networks 1Types of Storage in a Data Center 1Storage Protocols, Transports, and Networks 6Storage Networks 21Congestion in Storage Networks: An Overview 28NVMe over Fabrics 43Quality of Service (QoS) 46Summary 51References 52Chapter 2 Understanding Congestion in Fibre Channel Fabrics 55Fibre Channel Flow Control 55Congestion Spreading in Fibre Channel Fabrics 67Frame Flow Within a Fibre Channel Switch 86The Effects of Bit Errors on Congestion 92B2B Credit Loss and Recovery 112Fibre Channel Counters Summary 123Summary 127References 127Chapter 3 Detecting Congestion in Fibre Channel Fabrics 129Congestion Detection Workflow 129Congestion Detection Metrics 135Congestion Detection Metrics on Cisco MDS Switches 137Automatic Alerting 168Detecting Congestion Using Remote Monitoring Platforms 177Detecting Congestion Due to Slow Drain and Overutilization 192Slow Drain and Overutilization at the Same Time 194Detecting Congestion on long-distance links 195Summary 195References 196Chapter 4 Troubleshooting Congestion in Fibre Channel Fabrics 199Troubleshooting Methodology and Workflow 199Hints and Tips for Troubleshooting Congestion 214Cisco MDS NX-OS Commands for Troubleshooting Congestion 219Case Study 1: Finding Congestion Culprits and Victims in a Single-Switch Fabric 242Case Study 2: Credit Loss Recovery Causing Frame Drops 271Case Study 3: Overutilization on a Single Device Causing Massive Congestion Problems 297Case Study 4: Long-Distance ISLs Causing Congestion 323Summary 336References 337Chapter 5 Solving Congestion with Storage I/O Performance Monitoring 339Why Monitor Storage I/O Performance? 339How and Where to Monitor Storage I/O Performance 340Cisco SAN Analytics Architecture 344Understanding I/O Flows in a Storage Network 347I/O Flow Metrics 350I/O Operations and Network Traffic Patterns 358Summary 379References 379Chapter 6 Preventing Congestion in Fibre Channel Fabrics 381An Overview of Eliminating or Reducing Congestion 382Link Capacity 386Congestion Recovery by Disconnecting the Culprit Device 387Congestion Recovery by Dropping Frames 388Traffic Segregation 398Congestion Prevention Using Rate Limiters on Storage Arrays 433Congestion Prevention Using Dynamic Ingress Rate Limiting on Switches 436Preventing Congestion by Notifying the End Devices 457Network Design Considerations 469Summary 475References 476Chapter 7 Congestion Management in Ethernet Storage Networks 479Ethernet Flow Control 479Understanding Congestion in Lossless Ethernet Networks 506Detecting Congestion in Lossless Ethernet Networks 511Troubleshooting Congestion in Lossless Ethernet Networks 534Preventing Congestion in Lossless Ethernet Networks 547Lossless Traffic with VXLAN 565Summary 569References 570Chapter 8 Congestion Management in TCP Storage Networks 573Understanding Congestion in TCP Storage Networks 574Storage I/O Performance Monitoring 587Preventing Congestion in TCP Storage Networks 597Detecting Congestion in TCP Storage Networks 615Troubleshooting Congestion in TCP Storage Networks 625iSCSI and NVMe/TCP in a Lossless Network 630iSCSI and NVMe/TCP with VXLAN 631Fibre Channel over TCP/IP (FCIP) 631Modified TCP Implementations 637Summary 638References 639Chapter 9 Congestion Management in Cisco UCS Servers 641Cisco UCS Architecture 641Understanding Congestion in a UCS Domain 644Detecting Congestion in a UCS Domain 645The UCS Traffic Monitoring (UTM) App 648Summary 668References 669  9780137887231, TOC, 1/17/2024