Maintaining Mission Critical Systems in a 24/7 Environment

Peter M. Curtis is the Founder of PMC Group One, LLC. He is a graduate of the New York Institute of Technology with a Bachelor of Technology degree in Electro-Mechanical Computer Technology and a Master of Science degree in Energy Management. Mr. Curtis has decades of experience working in the Mission Critical Facilities Engineering industry in the areas of banking and finance, defense, electric and water utilities, energy management, and education. He has in-depth expertise in computer-integrated systems, online interface, and facilities operations-maintenance management.

• Foreword xviiPreface xxiAcknowledgments xxiii1 An Overview of Reliability and Resiliency in Today’s Mission Critical Environment 11.1 Introduction 11.2 Risk Assessment 51.2.1 Levels of Risk 61.3 Capital Costs versus Operation Costs 71.4 Critical Environment Workflow and Change Management 91.4.1 Change Management 101.5 Testing and Commissioning 111.6 Documentation and Human Factor 161.7 Education and Training 201.8 Corporate Knowledge Transfer – the Means to Securing Tomorrow’s Critical Infrastructure 211.9 Operation and Maintenance 241.10 Employee Certification 251.11 Standards and Benchmarking 251.12 What is a Mission Critical Engineer 261.13 Conclusion 281.14 An Overview of Reliability and Resiliency in Today’s Mission Critical Environment - Questions to Consider 282 Energy and Cyber Security and its Effect on Business Resiliency 312.1 Introduction 312.2 Risks Related to Information Security 362.3 Electro Magnetic Pulse and Solar Flares 422.4 How Risks Are Addressed 472.5 Use of Distributed Energy Resources and Generation 522.6 Documentation and Its Relation to Information Security 552.7 Smart Grid 572.8 Conclusion 602.9 Energy Security and Its Effect on Business Resiliency – Questions to Consider 603 Mission Critical Engineering with an Overview of Green Technologies 633.1 Introduction 633.2 Companies’ Expectations: Risk Tolerance and Reliability 653.3 Identifying the Appropriate Redundancy in a Mission Critical Facility 673.4 Improving Reliability, Maintainability, and Proactive Preventative Maintenance 693.5 The Mission Critical Facilities Manager and the Importance of the Boardroom 713.6 Quantifying Reliability and Availability 713.6.1 Review of Reliability Terminology 723.7 Design Considerations for the Mission Critical Data Center 733.7.1 Data Center Certification 743.8 The Evolution of Mission Critical Facility Design 763.9 Human Factors and the Commissioning Process 773.10 Short Circuit & Coordination Studies 793.11 Introduction to Direct Current in the Data Center 843.11.1 Advantages of DC Distribution 853.11.2 Lighting Updates 873.11.3 DC Storage Options 873.11.4 Renewable Energy Integration 883.11.5 DC and Combined Cooling, Heat & Power 893.11.6 Safety Issues 913.11.7 Maintenance 913.11.8 Education & Training 923.11.9 Future Vision 933.12 Containerized Systems Overview 933.13 Mission Critical Engineering with an Overview of Green Technologies - Questions to Consider 954 Mission Critical Electrical System Maintenance & Safety 1034.1 Introduction 1034.2 The History of the Maintenance Supervisor and the Evolution of the Mission Critical Facilities Engineer 1054.3 Internal Building Deficiencies and Analysis 1074.4 Evaluating Your System 1084.5 Choosing a Maintenance Approach 1104.5.1 Annual Preventive Maintenance 1114.6 Safe Electrical Maintenance 1124.6.1 Standards and Regulations 1124.6.2 Electrical Safety: NFPA 70E Arc Flash Mitigation 1144.6.3 Personal Protective Equipment (PPE) 1174.6.4 Lockout/Tagout 1264.7 Maintenance of Typical Electrical Distribution Equipment 1274.7.1 Thermal Scanning and Thermal Monitoring 1274.7.2 15 KV Class Equipment 1294.7.3 480 Volt Switchgear 1304.7.4 Motor Control Centers and Panel Boards 1314.7.5 Automatic Transfer Switches 1314.7.6 Automatic Static Transfer Switches (ASTS) 1324.7.7 Power Distribution Units 1324.7.8 277/480 Volt Transformers 1334.7.9 Uninterruptible Power Systems 1334.8 Being Proactive in Evaluating the Test Reports 1344.9 Designing for Safety and Reliability 1354.10 Conclusion 1365 Standby Generators 1375.1 Introduction 1375.2 The Necessity for Standby Power 1385.3 Emergency, Legally Required, and Optional Systems 1405.4 Standby Systems That Are Legally Required 1415.5 Optional Standby Systems 1425.6 Understanding Your Power Requirements 1425.7 Management Commitment and Training 1425.7.1 Lockout/ Tagout (LOTO) 1435.7.2 Training 1445.8 Standby Generator Systems Maintenance Procedures 1445.8.1 Maintenance Record Keeping and Data Trending 1455.8.2 Engine 1455.8.3 Coolant System 1455.8.4 Electrical / Control System 1465.8.5 Generator 1465.8.6 Automatic and Manual Switchgear 1475.8.7 Load Bank Testing 1475.9 Documentation Plan 1485.9.1 Proper Documentation and Forms 1485.9.2 Record keeping 1485.10 Emergency Procedures 1495.11 Cold Start 1505.12 Non-Linear Load Concerns 1515.12.1 Line Notches and Harmonic Current 1515.12.2 Voltage / Frequency Drop 1525.12.3 Voltage / Frequency Rise 1525.12.4 Frequency Fluctuation 1535.12.5 Synchronizing to Parallel 1545.12.6 Automatic Transfer Switch 1545.13 Conclusion 1556 Fuel Systems Design and Maintenance 1576.1 Introduction 1576.2 Brief Discussion on Diesel Engines 1586.3 Bulk Storage Tank Selection 1596.3.1 Aboveground Tanks 1596.3.2 Modern Underground Tanks and Piping Systems 1606.3.3 Fuel Receiving Tanks 1616.3.4 Generator Sub-Base Tanks 1616.4 Codes and Standards 1626.5 Recommended Practices for all Tanks 1636.6 Fuel Distribution System Configuration 1686.7 Day Tank Control System 1706.8 Diesel Fuel and a Fuel Quality Assurance Program 1746.9 Conclusion 1867 Power Transfer Switch Technology, Applications, and Maintenance 1877.1 Introduction 1877.2 Transfer Switch Technology and Applications 1897.3 Types of Power Transfer Switches 1917.3.1 Manual Transfer Switches 1917.3.2 Automatic Transfer Switches 1917.4 Control Devices 2047.4.1 Time Delays 2047.4.2 In-Phase Monitor 2057.4.3 Test Switches 2067.4.4 Exercise Clock 2077.4.5 Current, Voltage and Frequency Sensing 2077.5 Design Features 2077.5.1 Close Against High In-Rush Currents 2087.5.2 Withstand and Closing Rating (WCR) 2087.5.3 Carry Full Rated Current Continuously 2087.5.4 Interrupt Current 2097.6 Additional Characteristics and Ratings of ATS 2097.6.1 NEMA Classification 2097.6.2 System Voltage Ratings 2097.6.3 ATS Sizing 2097.6.4 Seismic Requirement 2107.7 Installation & Commissioning, Maintenance, and Safety 2107.7.1 Installation & Commissioning 2107.7.2 Maintenance & Safety 2127.7.3 Maintenance Tasks 2147.7.4 Drawings and Manuals 2157.7.5 Testing & Training 2157.8 General Recommendations 2187.9 Conclusion 2198 The Static Transfer Switch 2218.1 Introduction 2218.2 Overview 2228.2.1 Major Components 2228.3 Typical Static Switch One Line 2238.3.1 Normal Operation 2238.3.2 Bypass Operation 2248.3.3 STS and STS/transformer Configurations 2258.4 STS Technology and Application 2258.4.1 General Parameters 2258.4.2 STS Location and Type 2268.4.3 Advantages and Disadvantages of the Primary and Secondary STS/Transformer Systems 2268.4.4 Monitoring, Data Logging, and Data Management 2278.4.5 Downstream Device Monitoring 2278.4.6 STS Remote Communication 2288.4.7 Security 2288.4.8 Human Engineering and Eliminating Human Errors 2298.4.9 Reliability and Availability 2308.4.10 Repairability and Maintainability 2318.4.11 Fault Tolerance and Abnormal Operation 2328.5 Testing 2328.6 Conclusion 2339 The Fundamentals of Power Quality 2359.1 Introduction 2359.2 ElectricityBasics 2379.2.1 Basic Circuit 2389.2.2 Power Factor 2389.3 Transmission of Power 2419.3.1 Life Cycle of Electricity 2419.3.2 Single-Phase and Three-Phase Power Basics 2439.3.3 Unreliable Power versus Reliable Power 2459.4 Understanding Power Problems 2459.4.1 Power Quality Standards 2469.4.2 Power Quality Transients 2499.4.3 RMS Variations 2509.4.4 Causes of Power Line Disturbances 2559.4.5 Power Line Disturbance Levels 2619.5 Tolerances of Critical Loads 2619.5.1 CBEMA Curve 2639.5.2 ITIC Curve 2639.5.3 Purpose of Curves 2659.6 Power Monitoring 2659.7 The Impact of Alternative Energy Generation 2689.8 Conclusion 26910 UPS Systems: Applications and Maintenance with an Overview of Green Technologies 27310.1 Introduction 27310.1.1 Green and Reliability Overview 27310.2 Purpose of UPS Systems 27510.3 General Description of UPS Systems 27910.3.1 What is a UPS system? 27910.3.2 How does a UPS system work? 27910.3.3 Static UPS Systems 28010.3.4 Online 28110.3.5 Double Conversion 28210.3.6 Double Conversion UPS Power Path 28210.4 Components of a Static UPS System 28410.4.1 Power Control Devices 28410.5 Online - Line Interactive UPS Systems 29110.6 Offline (Standby) 29210.7 The Evolution of Static UPS Technology 29310.7.1 Emergence of the IGBT 29310.7.2 Two and Three-Level Rectifier/Inverter Topology 29410.7.3 Silicon Carbide Replaces Silicon as UPS Semiconductor of Electricity 29510.8 Rotary UPS Systems 29910.8.1 UPSs Using Diesel 30010.8.2 Hybrid UPS Systems 30110.9 Redundancy, Configurations, and Topology 30110.9.1 N 30210.9.2 N+1 30210.9.3 Isolated Redundant 30310.9.4 N+2 30310.9.5 2N 30410.9.6 2(N+1) 30510.9.7 Distributed Redundant / Catcher UPS 30510.9.8 “Eco-Mode” for Static UPS 30610.9.9 Availability Calculations 30710.10 Energy Storage Devices 30810.10.1 Battery 30810.10.2 Flywheel Energy 31410.11 UPS Maintenance & Testing 31610.11.1 Physical Preventive Maintenance (PM) 31810.11.2 Protection Settings, Calibration, and Guidelines 31810.11.3 Functional Load Testing 31910.11.4 Steady State Load Test 31910.11.5 Steady State Load Test at 0%, 50% and 100% load: 32010.11.6 Harmonic Analysis and Testing 32010.11.7 Filter Integrity and Testing 32110.11.8 Transient Response Load Test 32210.11.9 Module Fault Test 32210.11.10 Battery Run Down Test 32210.12 Static UPS and Maintenance 32310.12.1 Examples of Semi-Annual Checks and Services for UPS Systems 32410.13 UPS Management 32410.14 Conclusion 32511 Data Center Cooling Systems 32711.1 Introduction 32711.2 Background Information 33011.3 Cooling within Datacom Rooms 33111.4 Cooling Process 33211.4.1 Cooling Process in Datacom Space 33211.4.2 Direct Expansion (DX) Systems 33311.4.3 Chilled Water Systems 33411.5 Cooling Final Dissipation 33411.5.1 Air Cooled System 33511.5.2 Water Side 33511.6 The Refrigeration Process 33711.6.1 Refrigeration Equipment – Compressors 33711.6.2 Refrigeration Equipment – Chillers 33811.6.3 Heat Rejection Equipment 34211.6.4 Energy Recovery Equipment 35311.6.5 Heat Exchangers 36011.7 Components Inside Datacom Room 36311.7.1 Computer Room Cooling Units 36311.8 Conclusion 37312 Data Center Cooling Efficiency, Concepts, & Technologies 37512.1 Introduction 37512.2 Heat Transfer Inside Data Centers 37912.2.1 Heat Generation 37912.2.2 Heat Return 38012.2.3 Cooling Air 38012.3 Cooling and Other Airflow Topics 38112.3.1 Leakage 38112.3.2 Mixing and its Relationship to Efficiency 38212.3.3 Re-circulation 38212.3.4 Venturi Effect 38212.3.5 Vortex Effect 38312.3.6 CRAC/CRAH Types 38312.3.7 Potential CRAC Operation Issues 38312.3.8 Sensible vs. Latent Cooling 38412.3.9 Humidity Control 38612.3.10 CRAC Fighting / Too Many CRACs 38712.4 Design Approaches for Data Center Cooling 38812.4.1 Hot Aisle/Cold Aisle 38812.4.2 Cold Aisle Containment 38812.4.3 In-Row Cooling with Hot Aisle Containment 38812.4.4 Overhead Supplemental Cooling 38912.4.5 Chimney or Ducted Returns 38912.4.6 Advanced Active Airflow Management for Server Cabinets 39012.5 Additional Considerations 39012.5.1 Active Air Movement 39012.5.2 Adaptive Capacity 39012.5.3 Liquid Cooling 39112.5.4 Cold Storage 39212.6 Hardware & Associated Efficiencies 39212.6.1 Server Efficiency 39212.6.2 Server Virtualization 39212.6.3 Multi-Core Processors 39312.6.4 Blade Servers 39312.6.5 Energy Efficient Servers 39312.6.6 Power Managed Servers 39312.6.7 Effect of Dynamic Server Loads on Cooling 39312.7 Best Practices 39412.8 Efficiency Problem Solving 39412.9 Conclusion 39612.10 Conversions, Formulas, Guidelines 39613 Raised Access Floors 39713.1 Introduction 39713.1.1 What is an Access Floor? 39713.1.2 What are the Typical Applications for Access Floors? 39913.1.3 Why use an Access Floor? 39913.2 Design Considerations 40013.2.1 Determine the Structural Performance Required 40013.2.2 Determine the Required Finished Floor Height 40313.2.3 Determine the Understructure Support Design Type Required 40413.2.4 Determine the Appropriate Floor Finish 40513.2.5 Air Flow Requirements 40613.3 Safety Concerns 40913.3.1 Removal & Reinstallation of Panels 40913.3.2 Removing Panels 40913.3.3 Stringer Systems 41113.3.4 Protection of the Floor from Heavy Loads 41213.3.5 Grounding the Access Floor 41713.3.6 Fire Protection 41813.3.7 Zinc Whiskers 41913.4 Panel Cutting (For all Steel Panels or Cement Filled Panels that do not Contain an Aggregate) 41913.4.1 Safety Requirements for Cutting Panels 41913.4.2 Guidelines for Cutting Panels 42013.4.3 Cutout Locations in Panels; Supplemental Support for Cut Panels 42013.4.4 Saws and Blades for Panel Cutting 42013.4.5 Interior Cutout Procedure: 42113.4.6 Round Cutout Procedure 42113.4.7 Installing Protective Trim Around Cut Edges 42113.4.8 Cutting and Installing the Trim 42213.5 Access Floor Maintenance 42313.5.1 Best Practices for Standard High Pressure Laminate Floor Tile (HPL) and for Vinyl Conductive & Static Dissipative Tile 42313.5.2 Damp Mopping Procedure for HPL and Conductive & Static Dissipative Vinyl Tile 42313.5.3 Cleaning the Floor Cavity 42413.6 Troubleshooting 42413.6.1 Making Pedestal Height Adjustments 42513.6.2 Rocking Panel Condition 42513.6.3 Panel Lipping Condition (Panel Sitting High) 42513.6.4 Out-of-Square Stringer Grid (Twisted Grid) 42613.6.5 Tipping at Perimeter Panels 42713.6.6 Tight Floor or Loose Floor: Floor Systems Laminated with HPL Tile 42713.7 Additional Design Considerations 42813.7.1 LEED Certification 42813.7.2 Energy Efficiency - Hot and Cold Air Containment 42813.7.3 Airflow Distribution and CFD Analysis 42913.8 Conclusion 43714 Fire Protection in Mission Critical Infrastructures 43914.1 Introduction 43914.2 Hazard Analysis 44114.3 Alarm and Notification 44114.4 Early Warning Detection 44414.4.1 Wireless Detection 44514.5 Fire Suppression 44514.5.1 Hybrid Fire Suppression Systems 44814.5.2 Protecting Lithium Ion Batteries 44914.6 Systems Design 45014.6.1 Stages of a Fire 45014.6.2 Fire and Building Codes 45114.7 Fire Detection 45214.8 Fire Suppression Systems 46114.8.1 Water Mist Systems 46714.8.2 Carbon Dioxide Systems 47014.8.3 Clean Agent Systems 47214.8.4 Inert Gas Agents 47214.8.5 IG-541 47314.8.6 IG-55 47414.8.7 Chemical Clean Agents 47414.8.8 Portable Fire Extinguishers 47914.8.9 Clean Agents and the Environment 47914.9 Conclusion 48015 Managing Through Pandemics 48115.1 Executive Summary: COVID-19’s Impact on Critical Infrastructure Globally 48115.2 Architectural Solutions and Air Purification Systems 48215.2.1 HVAC Systems 48215.2.2 UV Technology 48215.2.3 Bipolar Ionization 48515.2.4 Copper Doorknobs 48515.2.5 Architectural Improvements to be Considered 48615.3 Building Equipment Solutions and Technology 48715.3.1 Cleaning vs. Disinfecting vs. Sanitizing 48715.3.2 Intensify Cleaning Frequency and Measures 48715.3.3 IR Scans 48815.3.4 Rethinking the flush, the sink, and the hand dryer 48815.3.5 Technology 48915.4 Operations, Maintenance and Training 49115.4.1 Personal Protection 49115.4.2 Change in Operation 49115.4.3 Data Center Betterment Opportunities 49215.5 Site Protection: Safeguarding the Staff and Visitors 49315.6 The Workforce of Tomorrow 49415.7 Assessment Tasks - HVAC and Air Handling Units Filter Upgrades 49515.8 Managing Through Pandemics -Questions to Consider 49615.9 Conclusion 497Appendix A Policies and Regulations 499A.1 Introduction 499A.2 Industry Policies & Regulations 501A.2.1 USA PATRIOT Act 503A.2.2 Sarbanes-Oxley Act (SOX) 505A.2.3 Comprehensive Environmental Response, Compensation, and Liability Act of 1980(also known as the Superfund Act) 506A.2.4 Executive Order 13423: Strengthening Federal Environmental, Energy and Transportation Management 507A.2.5 ISO27000 Information Security Management System (ISMS) 508A.2.6 The National Strategy for the Physical Protection of Critical Infrastructures and Key Assets 513A.2.7 2009 National Infrastructure Protection Plan 514A.2.8 North American Electric Reliability Corporation (NERC) Critical Infrastructure Protection Program 514A.2.9 U.S. Security & Exchange Commission (SEC) 516A.2.10 Sound Practices to Strengthen the Resilience of the U.S. Financial System 516A.2.11 C4I Command, Control, Communications, Computers, and Intelligence 517A.2.12 Basel II Accord 519A.2.13 National Institute of Standards and Technology (NIST) 519A.2.14 Business Continuity Management Agencies and Regulating Organizations 521A.2.15 FFIEC - Federal Financial Institutions Examination Council 523A.2.16 National Fire Prevention Association 1600 – Standards on Disaster/Emergency Management and Business Continuity Programs 524A.2.17 Private Sector Preparedness Act 525A.3 Data Protection 526A.4 Encryption 528A.4.1 Protecting Critical Data through Security and Vaulting 529A.5 Business Continuity Plan (BCP) 529A.6 Conclusion 531Appendix B Consolidated List of Key Questions 535Appendix C Airflow Management (A System Approach) 553C.1 Introduction 553C.2 Control is the Key 555C.3 Obtaining Control 558C.4 Air Management Technologies 565C.5 Conclusion 570Glossary 573References 595Index 609