Camera Image Quality Benchmarking

JONATHAN B. PHILLIPS, is a Staff Image Scientist at Google, USA. He is a United States delegate to the technical committee ISO/TC 42 Photography and a major contributor to the IEEE Camera Phone Image Quality (CPIQ) initiative. HENRIK ELIASSON, PHD, is an image sensor and image analysis specialist at Eclipse Optics, Sweden. He is a senior member of SPIE.

• About the Authors xvSeries Preface xviiPreface xixList of Abbreviations xxiiiAbout the CompanionWebsite xxvii1 Introduction 11.1 Image Content and Image Quality 21.1.1 Color 31.1.2 Shape 81.1.3 Texture 101.1.4 Depth 111.1.5 Luminance Range 121.1.6 Motion 151.2 Benchmarking 181.3 Book Content 22Summary of this Chapter 24References 252 Defining Image Quality 272.1 What is Image Quality? 272.2 Image Quality Attributes 292.3 Subjective and Objective Image Quality Assessment 31Summary of this Chapter 32References 333 Image Quality Attributes 353.1 Global Attributes 353.1.1 Exposure, Tonal Reproduction, and Flare 353.1.2 Color 393.1.3 Geometrical Artifacts 403.1.3.1 Perspective Distortion 403.1.3.2 Optical Distortion 423.1.3.3 Other Geometrical Artifacts 423.1.4 Nonuniformities 433.1.4.1 Luminance Shading 453.1.4.2 Color Shading 453.2 Local Attributes 453.2.1 Sharpness and Resolution 453.2.2 Noise 493.2.3 Texture Rendition 503.2.4 Color Fringing 503.2.5 Image Defects 513.2.6 Artifacts 513.2.6.1 Aliasing and Demosaicing Artifacts 523.2.6.2 Still Image Compression Artifacts 533.2.6.3 Flicker 533.2.6.4 HDR Processing Artifacts 553.2.6.5 Lens Ghosting 553.3 Video Quality Attributes 563.3.1 Frame Rate 563.3.2 Exposure and White Balance Responsiveness and Consistency 583.3.3 Focus Adaption 583.3.4 Audio-Visual Synchronization 583.3.5 Video Compression Artifacts 593.3.6 Temporal Noise 603.3.7 Fixed Pattern Noise 603.3.8 Mosquito Noise 60Summary of this Chapter 60References 614 The Camera 634.1 The Pinhole Camera 634.2 Lens 644.2.1 Aberrations 644.2.1.1 Third-Order Aberrations 654.2.1.2 Chromatic Aberrations 664.2.2 Optical Parameters 674.2.3 Relative Illumination 694.2.4 Depth of Field 704.2.5 Diffraction 714.2.6 Stray Light 734.2.7 Image Quality Attributes Related to the Lens 744.3 Image Sensor 754.3.1 CCD Image Sensors 754.3.2 CMOS Image Sensors 774.3.3 Color Imaging 814.3.4 Image Sensor Performance 824.3.5 CCD versus CMOS 894.3.6 Image Quality Attributes Related to the Image Sensor 904.4 Image Signal Processor 914.4.1 Image Processing 914.4.2 Image Compression 984.4.2.1 Chroma Subsampling 984.4.2.2 Transform Coding 984.4.2.3 Coefficient Quantization 994.4.2.4 Coefficient Compression 1004.4.3 Control Algorithms 1014.4.4 Image Quality Attributes Related to the ISP 1014.5 Illumination 1024.5.1 LED Flash 1034.5.2 Xenon Flash 1034.6 Video Processing 1034.6.1 Video Stabilization 1034.6.1.1 Global Motion Models 1044.6.1.2 Global Motion Estimation 1054.6.1.3 Global Motion Compensation 1064.6.2 Video Compression 1074.6.2.1 Computation of Residuals 1074.6.2.2 Video Compression Standards and Codecs 1094.6.2.3 Some Significant Video Compression Standards 1104.6.2.4 A Note On Video Stream Structure 1114.7 System Considerations 111Summary of this Chapter 112References 1135 Subjective Image Quality Assessment—Theory and Practice 1175.1 Psychophysics 1185.2 Measurement Scales 1205.3 PsychophysicalMethodologies 1225.3.1 Rank Order 1235.3.2 Category Scaling 1235.3.3 Acceptability Scaling 1245.3.4 Anchored Scaling 1255.3.5 Forced-Choice Comparison 1255.3.6 Magnitude Estimation 1255.3.7 Methodology Comparison 1265.4 Cross-Modal Psychophysics 1265.4.1 Example Research 1275.4.2 Image Quality-Related Demonstration 1285.5 Thurstonian Scaling 1295.6 Quality Ruler 1315.6.1 Ruler Generation 1345.6.2 Quality Ruler Insights 1355.6.2.1 Lab Cross-Comparisons 1355.6.2.2 SQS2 JND Validation 1365.6.2.3 Quality Ruler Standard Deviation Trends 1395.6.2.4 Observer Impact 1415.6.3 Perspective from Academia 1425.6.4 Practical Example 1445.6.5 Quality Ruler Applications to Image Quality Benchmarking 1475.7 Subjective Video Quality 1485.7.1 Terminology 1495.7.2 Observer Selection 1495.7.3 Viewing Setup 1505.7.4 Video Display and Playback 1515.7.5 Clip Selection 1525.7.6 Presentation Protocols 1545.7.7 Assessment Methods 1565.7.8 Interpreting Results 1585.7.9 ITU Recommendations 1595.7.9.1 The Double-Stimulus Impairment Scale Method 1605.7.9.2 The Double-Stimulus Continuous Quality Scale Method 1605.7.9.3 The Simultaneous Double-Stimulus for Continuous Evaluation Method 1605.7.9.4 The Absolute Category Rating Method 1615.7.9.5 The Single Stimulus Continuous Quality Evaluation Method 1615.7.9.6 The Subjective Assessment of Multimedia Video Quality Method 1615.7.9.7 ITU Methodology Comparison 1625.7.10 Other Sources 162Summary of this Chapter 162References 1636 Objective Image Quality Assessment—Theory and Practice 1676.1 Exposure and Tone 1686.1.1 Exposure Index and ISO Sensitivity 1686.1.2 Optoelectronic Conversion Function 1696.1.3 Practical Considerations 1706.2 Dynamic Range 1706.3 Color 1716.3.1 Light Sources 1716.3.2 Scene 1746.3.3 Observer 1766.3.4 Basic Color Metrics 1786.3.5 RGB Color Spaces 1806.3.6 Practical Considerations 1816.4 Shading 1816.4.1 Practical Considerations 1826.5 Geometric Distortion 1826.5.1 Practical Considerations 1846.6 Stray Light 1846.6.1 Practical Considerations 1856.7 Sharpness and Resolution 1856.7.1 The Modulation Transfer Function 1866.7.2 The Contrast Transfer Function 1916.7.3 Geometry in Optical Systems and the MTF 1936.7.4 Sampling and Aliasing 1946.7.5 System MTF 1956.7.6 Measuring the MTF 1986.7.7 Edge SFR 1986.7.8 Sine Modulated Siemens Star SFR 2016.7.9 Comparing Edge SFR and Sine Modulated Siemens SFR 2036.7.10 Practical Considerations 2046.8 Texture Blur 2046.8.1 Chart Construction 2066.8.2 Practical Considerations 2066.8.3 AlternativeMethods 2076.9 Noise 2076.9.1 Noise and Color 2076.9.2 Spatial Frequency Dependence 2096.9.3 Signal to Noise Measurements in Nonlinear Systems and Noise Component Analysis 2116.9.4 Practical Considerations 2126.10 Color Fringing 2136.11 Image Defects 2146.12 Video Quality Metrics 2146.12.1 Frame Rate and Frame Rate Consistency 2156.12.2 Frame Exposure Time and Consistency 2156.12.3 Auto White Balance Consistency 2166.12.4 Autofocusing Time and Stability 2166.12.5 Video Stabilization Performance 2176.12.6 Audio-Video Synchronization 2186.13 Related International Standards 218Summary of this Chapter 221References 2217 Perceptually Correlated Image Quality Metrics 2277.1 Aspects of Human Vision 2277.1.1 Physiological Processes 2277.2 HVS Modeling 2327.3 Viewing Conditions 2327.4 Spatial Image Quality Metrics 2347.4.1 Sharpness 2357.4.1.1 Edge Acutance 2357.4.1.2 Mapping Acutance to JND Values 2377.4.1.3 Other Perceptual Sharpness Metrics 2397.4.2 Texture Blur 2397.4.3 Visual Noise 2407.5 Color 2447.5.1 Chromatic Adaptation Transformations 2447.5.2 Color Appearance Models 2457.5.3 Color and Spatial Content—Image Appearance Models 2477.5.4 Image Quality Benchmarking and Color 2497.6 Other Metrics 2517.7 Combination of Metrics 2527.8 Full-Reference Digital Video Quality Metrics 2527.8.1 PSNR 2537.8.2 Structural Similarity (SSIM) 2567.8.3 VQM 2607.8.4 VDP 2627.8.4.1 Further Considerations 2637.8.5 Discussion 265Summary of this Chapter 267References 2678 Measurement Protocols—Building Up a Lab 2738.1 Still Objective Measurements 2738.1.1 Lab Needs 2748.1.1.1 Lab Space 2748.1.1.2 Lighting 2758.1.1.3 Light Booths 2788.1.1.4 Transmissive Light Sources 2798.1.1.5 Additional Lighting Options 2808.1.1.6 Light Measurement Devices 2818.1.2 Charts 2828.1.2.1 Printing Technologies for Reflective Charts 2828.1.2.2 Technologies for Transmissive Charts 2868.1.2.3 Inhouse Printing 2868.1.2.4 Chart Alignment and Framing 2878.1.3 Camera Settings 2898.1.4 Supplemental Equipment 2898.1.4.1 RealWorld Objects 2908.2 Video Objective Measurements 2938.2.0.2 Visual Timer 2938.2.0.3 Motion 2948.3 Still Subjective Measurements 2978.3.1 Lab Needs 2978.3.2 Stimuli 2988.3.2.1 Stimuli Generation 2988.3.2.2 Stimuli Presentation 3018.3.3 Observer Needs 3028.3.3.1 Observer Selection and Screening 3028.3.3.2 Experimental Design and Duration 3038.4 Video Subjective Measurements 304Summary of this Chapter 305References 3059 The Camera Benchmarking Process 3099.1 Objective Metrics for Benchmarking 3099.2 Subjective Methods for Benchmarking 3119.2.1 Photospace 3129.2.2 Use Cases 3139.2.3 Observer Impact 3149.3 Methods of Combining Metrics 3159.3.1 Weighted Combinations 3169.3.2 Minkowski Summation 3169.4 Benchmarking Systems 3179.4.1 GSMArena 3179.4.2 FNAC 3189.4.3 VCX 3189.4.4 Skype Video Capture Specification 3199.4.5 VIQET 3209.4.6 DxOMark 3219.4.7 IEEE P1858 3239.5 Example Benchmark Results 3249.5.1 VIQET 3249.5.2 IEEE CPIQ 3259.5.2.1 CPIQ Objective Metrics 3279.5.2.2 CPIQ Quality Loss Predictions from Objective Metrics 3379.5.3 DxOMark Mobile 3389.5.4 Real-World Images 3399.5.5 High-End DSLR Objective Metrics 3399.6 Benchmarking Validation 345Summary of this Chapter 348References 34910 Summary and Conclusions 353References 357Index 359