Next-Generation Video Coding and Streaming

Benny Bing has worked in academia for over 20 years. He has published over 80 research papers and 12 books, and has 6 video patents licensed to industry. He has served as a technical editor for several IEEE journals and an IEEE Communications Society Distinguished lecturer. He also received the National Association of Broadcasters (NAB) Technology Innovation Award for demonstrations of advanced media technologies.

• Preface xvii1 Digital Video Delivery 11.1 Broadband TV Landscape 21.1.1 Internet TV Providers 21.1.2 Netflix 31.1.3 Hulu 31.1.4 Amazon 31.1.5 YouTube 31.1.6 ESPN3 41.1.7 HBO 41.1.8 CBS 41.1.9 Sony 41.1.10 Retail Giants 41.2 Internet TV Delivery Platforms 51.2.1 Cloud TV 51.2.2 Content Delivery Network 61.2.3 Free CDN 61.2.4 Video Transcoding 71.3 Second Screen Device Adoption 71.3.1 Mobile Video 81.3.2 Mobile Versus Traditional TV 81.3.3 Over-the-Air Digital TV 81.3.4 Non-Real-Time TV Delivery 91.3.5 NRT Use Cases 91.3.6 Cable Wi-Fi Alliance 91.4 Screen and Video Resolution 101.4.1 Aspect Ratios 111.4.2 Video Resolution 111.4.3 Visual Quality 131.4.4 Matching Video Content to Screen Size 131.5 Stereoscopic 3D TV 141.5.1 Autostereoscopic 3D 141.5.2 Anaglyph 3D 141.6 Video Coding Standards 151.6.1 Exploiting Video Content Redundancies 151.6.2 High-Quality Versus High-Resolution Videos 161.6.3 Factors Affecting Coded Video Bit Rates 161.6.4 Factors Affecting Coded Frame Sizes 171.7 Video Streaming Protocols 181.7.1 Video Streaming over HTTP 191.7.2 Adaptive Bit Rate Streaming 191.7.3 Benefits and Drawbacks of Adaptive Streaming 201.7.4 HTTP Progressive Download 201.7.5 HTML5 201.8 TV Interfaces and Navigation 211.8.1 Streaming Adapters 211.8.2 Streaming Boxes 211.8.3 Media-Activated TV Navigation 221.8.4 Smartphone and Tablet TV Navigation 221.8.5 Digital Living Network Alliance 221.8.6 Discovery and Launch 231.8.7 UltraViolet 23References 24Homework Problems 242 Video Coding Fundamentals 292.1 Sampling Formats of Raw Videos 292.1.1 Color Subsampling 302.1.2 YUV Versus RGB Color Space 312.1.3 Bit Rate and Storage Requirements 312.2 Impact of Video Compression 322.2.1 Rate-Distortion Optimization 322.2.2 Partitions in a Video Frame 332.2.3 Video Coding Standards 342.2.4 Profiles and Levels 342.3 General Video Codec Operations 342.3.1 Transform Coding 352.3.2 Quantization 352.3.3 Deblocking Filter 372.4 Transform Coding 382.4.1 Orthonormal Transforms 382.4.2 Discrete Cosine Transform 402.4.3 Discrete Sine Transform 442.4.4 Asymmetric DST 442.4.5 Comparison of KLT ADST and DCT 442.4.6 Hybrid Transforms 462.4.7 Wavelet Transform 462.4.8 Impact of Transform Size 462.4.9 Impact of Parallel Coding 472.5 Entropy Coding 472.5.1 Variable Length Codes 472.5.2 Golomb Codes 482.5.3 Arithmetic Coding Overview 482.5.4 Nonadaptive Arithmetic Coding 492.5.5 Steps in Nonadaptive Arithmetic Coding 492.5.6 Context-Based Adaptive Arithmetic Coding 502.5.7 Code Synchronization 502.6 MPEG (H.26x) Standards 512.6.1 MPEG Frames 512.6.2 I Frames 512.6.3 P Frames 522.6.4 B Frames 522.6.5 Intracoded P and B Frames 522.7 Group of Pictures 532.7.1 GOP Length 532.7.2 Closed GOP 532.7.3 Error Resiliency in a Closed GOP 542.7.4 Decoding Sequence 552.7.5 Open GOP 552.7.6 Variable GOP Length 562.7.7 Random Access of MPEG Frames 562.8 Motion Estimation and Compensation 572.8.1 Motion Estimation 572.8.2 Motion Search in P Frames 582.8.3 Motion Search in B Frames 582.8.4 Fractional (Subsample) Motion Search 592.8.5 Motion Compensation 602.8.6 Computational Complexity 612.8.7 Motion Search Algorithms 632.8.8 Accelerating Motion Search 652.8.9 Impact of Video Resolution 662.9 Non-MPEG Video Coding 662.9.1 Motion JPEG 662.9.2 Dirac 672.9.3 WebM Project 672.10 Constant and Variable Bit-Rate Videos 672.10.1 CBR Encoding 682.10.2 VBR Encoding 682.10.3 Assessing Bit Rate Variability 692.10.4 Scene Change Detection 702.10.5 Adaptive Scene Change Detection 712.10.6 I Frame Size Prediction 722.11 Advanced Audio Coding 722.11.1 Low and High Bit Rate AAC 742.11.2 High-Efficiency and Low-Complexity AAC 742.11.3 MPEG Surround 742.12 Video Containers 742.12.1 MPEG-4 752.12.2 MP4 Access Units 752.12.3 Binary Format for Scenes 752.12.4 MP4 Overheads 762.12.5 MPEG-2 TS 762.12.6 MPEG-2 TS Structure 762.12.7 MPEG-2 TS Audio and Video PESs 772.12.8 MPEG-2 TS IP/Ethernet Encapsulation 772.13 Closed Captions 77References 78Homework Problems 783 H.264/AVC Standard 833.1 Overview of H.264 833.1.1 Fundamental H.264 Benefits 843.1.2 H.264 Applications 843.2 H.264 Syntax and Semantics 843.2.1 Profiles and Levels 853.2.2 Baseline Extended Main Profiles 853.2.3 High Profiles 853.3 H.264 Encoder 893.3.1 H.264 Slice Types 893.3.2 H.264 Intraprediction 903.3.3 Intraprediction for 4 × 4 Blocks 913.3.4 Intraprediction for 16 × 16 Macroblocks 923.3.5 Intra Pulse Code Modulation Mode 933.3.6 H.264 Interprediction 933.4 Rate Distortion Optimization 943.4.1 RDO under VBR 953.4.2 RDO under CBR 953.4.3 In-Loop Deblocking Filter 963.5 Video Coding and Network Abstraction Layers 963.5.1 Video Coding Layer 963.5.2 Network Abstraction Layer 973.5.3 Hypothetical Reference Decoder 973.5.4 Supplemental Enhancement Information 983.6 Error Resilience 983.6.1 Slice Coding 983.6.2 Data Partitioning 993.6.3 Slice Groups 1003.6.4 Redundant Slices 1013.6.5 Flexible Macroblock Ordering 1013.6.6 FMO Types 1023.6.7 FMO Overhead 1033.6.8 Arbitrary Slice Ordering 1033.7 Transform Coding 1043.7.1 Transform Types 1043.7.2 Hadamard Transforms 1053.7.3 Transform Implementation 1063.8 Entropy Coding 1063.8.1 Context-Adaptive Binary Arithmetic Coding 1063.8.2 CABAC Performance 1073.8.3 Context-Adaptive Variable-Length Coding 1073.9 Motion Vector Search 1083.9.1 Motion Search Options 1083.10 Multiple Reference Slices 1093.10.1 Motivations for Using More Reference Slices 1093.10.2 Switching Reference Slices 1093.11 Scalable Video Coding 1093.11.1 Temporal Scalability 1103.11.2 Spatial Scalability 1103.11.3 Video Quality Scalability 1103.11.4 Disadvantages of SVC 110References 111Homework Problems 1114 H.265/HEVC Standard 1154.1 H.265 Overview 1154.1.1 Fundamental H.265 Benefits 1164.1.2 H.265 Applications 1184.1.3 Video Input 1184.2 H.265 Syntax and Semantics 1184.2.1 Parameter Set Structure 1194.2.2 NAL Unit Syntax Structure 1194.2.3 Reference Frame Sets and Lists 1194.2.4 H.265 GOP Structure 1204.2.5 Support for Open GOPs and Random Access 1214.2.6 Video Coding Layer 1224.2.7 Temporal Sublayers 1224.2.8 Error Resilience 1234.2.9 RTP Support 1244.3 Profiles Levels and Tiers 1244.3.1 Profiles 1244.3.2 Levels 1254.3.3 Range Extensions 1264.4 Quadtrees 1264.4.1 Variable Block Size Quadtree Partitioning 1274.4.2 Coding Tree Units 1284.4.3 Splitting of Coding Blocks 1294.4.4 Frame Boundary Matching 1304.4.5 Prediction Blocks and Units 1304.4.6 Transform Blocks and Units 1324.4.7 Determining the Quadtree Depth 1324.4.8 Coding Unit Identification 1334.5 Slices 1334.5.1 Tiles 1344.5.2 Dependent Slice Segments 1354.5.3 Wavefront Parallel Processing 1364.5.4 Practical Considerations for Parallel Processing 1374.6 Intraprediction 1374.6.1 Prediction Block Partitioning 1384.6.2 Intra-Angular Prediction 1384.6.3 Intra-DC and Intra-Planar Prediction 1404.6.4 Adaptive Smoothing of Reference Samples 1404.6.5 Filtering of Prediction Block Boundary Samples 1414.6.6 Reference Sample Substitution 1414.6.7 Mode Coding 1424.7 Interprediction 1434.7.1 Fractional Sample Interpolation 1434.7.2 Motion Vector Prediction 1454.7.3 Merge Mode 1464.7.4 Skip Mode 1474.7.5 Advanced MV Prediction 1484.7.6 Restrictions on Motion Data 1484.7.7 Practical Considerations 1494.8 Transform Scaling and Quantization 1494.8.1 Alternative 4 × 4 Transform 1504.8.2 Scaling 1514.8.3 Quantization 1514.9 Entropy Encoding 1514.9.1 H.265 Binarization Formats 1524.9.2 Context Modeling 1524.9.3 CABAC Throughput Issues 1544.9.4 CABAC Encoding 1544.9.5 CABAC Decoding 1554.9.6 Coefficient Scanning 1554.9.7 Coefficient Coding 1564.10 In-Loop Filters 1564.10.1 In-Loop Deblocking Filter 1574.10.2 Sample-Adaptive Offset Filter 1584.11 Special H.265 Coding Modes 161References 162Homework Problems 1625 Assessing and Enhancing Video Quality 1655.1 Introduction 1655.1.1 Subjective Metrics 1665.1.2 Limitations of Subjective Metrics 1665.1.3 Objective Metrics 1665.1.4 Types of Objective Metrics 1675.1.5 References for Objective Metrics 1675.1.6 Network Impact 1685.2 Distortion Measure 1695.2.1 Sum of Absolute Differences 1695.2.2 Sum of Absolute Transformed Differences 1695.3 Peak Signal to Noise Ratio 1705.3.1 Combined PSNR 1705.3.2 Impact of Video Resolution and QP on PSNR 1725.3.3 Limitations of PSNR 1735.4 Structural Similarity Index 1735.5 Observable Versus Perceptual Visual Artifacts 1755.5.1 Limited Information Provided by PSNR 1765.5.2 Observable Artifacts and Link Quality 1765.5.3 Combined Spatial and Temporal Video Quality Assessment 1765.6 Error Concealment 1775.6.1 Error Resilience 1775.6.2 Impact on Visual Artifacts 1785.6.3 Types of Error Concealment 1795.6.4 Comparison of EC Methods 1795.6.5 Increasing Frame Rate Using EC 1795.6.6 Actions Performed After EC 1805.7 Color Science 1805.7.1 Color Reception 1805.7.2 Color Reproduction 180References 181Homework Problems 1816 Coding Performance of H.262 H.264 and H.265 1836.1 Coding Parameters 1846.1.1 Coding Block Size 1846.1.2 Transform Block Size 1876.1.3 TMVP SAO AMP 1886.2 Comparison of H.265 and H.264 1896.2.1 Absolute Coding Efficiency 1896.2.2 Relative Coding Gain 1906.2.3 Videos with Different Levels of Motion 1916.3 Frame Coding Comparison 1926.3.1 I Frame Coding Efficiency Quality and Time 1936.3.2 P Frame Coding Efficiency Quality and Time 1956.3.3 B Frame Coding Efficiency Quality and Time 1976.3.4 Overall Frame Coding Efficiency Quality and Time 1996.4 Impact of Coding Block Size on Frame Coding Efficiency 2016.4.1 Impact of Transform Block Size on Frame Coding Efficiency 2016.4.2 Impact of Coding Block Size on Frame Encoding Time 2036.4.3 Impact of Transform Block Size on Frame Encoding Time 2036.4.4 Impact of CU Size on Encoding Time 2036.4.5 Decoding Time 2056.5 Summary of Coding Performance 2056.6 Error Resiliency Comparison of H.264 and H.265 2056.6.1 H.264 Error Resiliency 2086.6.2 H.265 Error Resiliency 2126.7 H.264/H.265 Versus H.262 2146.7.1 Performance Comparison 2146.7.2 H.262 Frame Coding Efficiency 2156.7.3 Impact of GOP Size 218References 219Homework Problems 2197 3D Video Coding 2217.1 Introduction 2217.1.1 3D Video Transmission and Coding 2227.1.2 View Multiplexing 2227.1.3 View Expansion and Display 2237.1.4 View Packing Methods 2237.2 Multiview Coding 2247.2.1 MVC Bitstream 2247.2.2 2D to 3D Conversion 2257.2.3 H.264 Multiview Coding Extension 2257.2.4 MVC Inter-view Prediction 2257.2.5 MVC Inter-view Reordering 2277.2.6 MVC Profiles 2277.2.7 Comparing MVC with 2D H.264 Video Coding 2277.3 Correlation Between Left and Right Views in S3D VIDEOS 2287.4 View Expansion Via Sample Interpolation 2307.4.1 Impact of Sample Interpolation 2307.4.2 Inter-view Versus Intraview Sample Interpolation 2337.4.3 Interframe Versus Intraview Sample Interpolation 2357.4.4 Impact of Quantization on Interpolated S3D Videos 2357.5 Anaglyph 3D Generation 2357.5.1 H.264 Coding Efficiency for Anaglyph Videos 2387.5.2 Delta Analysis 2397.5.3 Disparity Vector Generation 242References 243Homework Problems 2448 Video Distribution and Streaming 2458.1 Adaptive Video Streaming 2468.1.1 Playlists and Bandwidth Estimation 2478.1.2 Quality (Bitstream) Switching 2478.2 Video Quality and Chunk Efficiency 2488.2.1 Video Quality for Different VBR Chunk Durations 2488.2.2 VBR Chunk Bit Rate Versus Chunk Duration 2508.2.3 VBR Chunk Efficiency Versus Chunk Duration 2508.2.4 Capped VBR Chunk Efficiency Versus Chunk Duration 2528.2.5 CBR Chunk Efficiency Versus Chunk Duration 2538.2.6 Instantaneous and Average Rates for Different Chunk Durations 2548.3 Apple HLS 2578.3.1 Overview of HLS Operation 2578.3.2 GOP Structure 2588.3.3 Super and Dynamic Playlists 2598.3.4 Media Control 2608.4 HLS Over 4G and 802.11 2618.4.1 Startup Delay 2618.4.2 Switching Quality Levels 2638.4.3 One-Level Versus Unfragmented HLS 2658.4.4 Multi-Level HLS 2668.4.5 Duplicate Video Chunks with Audio 2688.4.6 Duplicate Video Chunks 2698.4.7 Duplicate Audio Chunks 2718.4.8 Duplicate Chunk Suppression 2728.4.9 Server-Based Chunk Suppression 2728.4.10 Custom App Chunk Suppression 2748.5 Impact of Varying Chunk Duration 2748.5.1 Impact of Varying Quality Levels 2768.5.2 Summary of HLS Performance 2778.6 Microsoft Silverlight Smooth Streaming 2808.6.1 Overview of MSS Operation 2808.6.2 MSS Streaming over 802.11n and 802.16 2818.6.3 802.16 MSS Streaming 2838.6.4 802.11n MSS Streaming 2848.6.5 Comparison of HLS and MSS Streaming 2878.7 Traffic Rate Shaping 2878.7.1 Impact of Shaping and Scene Complexity on Quality Switching 2888.7.2 Impact of Shaping on Quality Switch Delay 2908.7.3 Impact of Shaping on Playback Duration 2918.7.4 Impact of Shaping on Start of Playback 2918.7.5 Impact of Shaping and Scene Complexity on Duplicate Chunks 2928.7.6 Impact of Unshaped Traffic on Quality Switching 2938.8 Adobe HTTP Dynamic Streaming 2948.9 MPEG-DASH (ISO/IEC 23009) 2948.9.1 DASH Process 2968.9.2 DASH Media Formats 2968.9.3 DASH for HTML5 2978.9.4 DASH Industry Forum 2978.10 Aggregate Adaptive Stream Bandwidth Prediction 2978.10.1 Permanence Time 2988.10.2 Prediction Model Implementation 2988.11 Limitations of Client-Based Adaptive Streaming 2988.11.1 Limitations of Fixed-Size Chunks 3008.11.2 Server-Based Adaptive Streaming 3018.11.3 Linear Broadcast Systems 3028.11.4 Adaptive Streaming and Scalable Video Coding 3028.12 Tips for Efficient Adaptive Streaming 3028.12.1 Quality Levels and Chunk Duration 3028.12.2 Encoder Efficiency 3038.12.3 Bit Rates of Quality Levels 3038.12.4 Server Bandwidth Shaping 3038.12.5 Server Bandwidth Estimation 3048.12.6 Analyzing Network Congestion 304References 305Homework Problems 305Glossary 311Index 317

Select Guide Rating