IEEE P1858 CPIQ Overview Margaret Belska P1858 CPIQ Chair, CPIQ CASC Chair February 3, 2015
What is CPIQ? ¾ CPIQ = Camera Phone Image Quality Scope expanded to include all mobile cameras, not just phones ¾ History: Launched in 2006 under International Imaging Industry Association (I3A) Transitioned in 2012 to IEEE standards development as Work Group P1858 ¾ Participating companies: Cross-industry: Mobile carriers, OS vendors, handset manufactures, chipset vendors, component vendors, test labs, test software and equipment vendors, and others Global representation: Currently >20 member companies representing Europe, US, and Asia ¾ Relationship to ISO: CPIQ has a liaison relationship with ISO in order to maintain consistency across imaging standards from different organizations 2
Why do we need CPIQ? ¾ Reviewers and consumers starting to understand that megapixels image quality ¾ Need an alternative way to asses and communicate image quality ¾ CPIQ goals are to: Standardize image quality test metrics and methodologies across the industry Correlate objective results with human perception Combine the data into a meaningful consumer rating system 3
No Common Language ¾ Everyone in the industry defines image quality differently ¾ This makes working together a challenge Component Handset ISP OS Test Lab Carrier 4
Many Ways to Measure IQ ¾ Many conditions X many metrics = endless combinations The same test can be done under many different conditions Test targets, light sources, light levels, color temperatures, distances, etc. all have an impact There are many metrics to measure the same thing. Color alone can be measured in almost twenty different ways! ΔE* ΔC* ab = ( (a 2 *-a 1 *) 2 + (b 2 *-b 1 *) 2 ) 1/2 uv = ( (L ΔE* 2 *-L 94 = 1 *) ( (ΔL*) 2 + (u 2 *-u + (ΔC* 1 *) 2 + (v S 2 *-v C ) 2 + 1 *) (ΔH* 2 ) 1/2 L* = 116f(Y/Y n ) 16 Su* H ) 2 = 1/2 13L*(u u n ) where S C = 1 C* ΔH = ( (ΔE) + 0.045 C* ; S 2 + (ΔL) H = ΔE* 1 + 2 + (ΔC) ab = 0.015 ΔC ( (Lv* C*; 2 ) 2 *-L 94 = 1/2 ab = (a* 2 + b* 2 ) = 13L*(v ( 1 *) 2 (ΔC* L* S + (a = 2 *-a v 116f(Y/Y C ) 1 *) 2 2 + (ΔH* S (b 2 *-b 1 *) H 2 ) ) 2 1/2 ) 1/2 R/B, R/G, B/G color ratios n ) n ) 16 ΔH* = ( (ΔE* ab ) 2 (ΔL*) 2 (ΔC*) 2 ) 1/2 a* = 500[f(X/X n ) f(y/y n )] ΔC* = ( a 1 * 2 + b 1 * 2 ) 1/2 ( a 2 * 2 + b 2 * 2 ) 1/2 b* = 200[f(Y/Y n ) f(z/z n )] Chroma % = 100% mean((a* 2 Y = 0.2126R i_meas + b* i_meas2 ) 1/2 ) / linear + 0.7152G mean((a i_ideal * 2 linear + + b i_ideal * 2 ) 1/2 ) 0.0722B linear If Company A measures 10 and Company B measures 20, who s to say who s right? 5
P1858 Published Standards ¾ Draft spec published Sep 2014; final Rev1 spec targeted for August 2015 ¾ Will include metrics for: Spatial Frequency Response Lateral Chromatic Aberration Color Uniformity Lens Geometric Distortion Texture Blur Visual Noise (under development) Color Saturation & White Balance (under development) Video Quality (under development) Auto Focus (under development) 6
Subjective Correlation ¾ Now we are all measuring the same thing, but what does it mean? ¾ Need to correlate objective results with perceived quality ¾ This is where CPIQ and ISO standards differ Now Company A measures 10 and Company B measures 10, but is that good? 7
The Quality Ruler Method ¾ ISO 20462 Part 3 The Quality Ruler Method used to correlate objective measurements with subjective perception Standardization of anchored pair comparison method based on JND units A Just Noticeable Difference (JND) is the smallest statistically measurable difference of perception Typically, defined when half of the people perceive a difference and the other half are guessing 50% perceive a change 50% guessing 75% of judgments correct, 25% incorrect 8
Anchored Pair Comparison ¾ Image references (anchors) form basis of category scale Anchors step in quality from high to low Calibrated to numerical scale of 30 JND values in sharpness ¾ Test images are compared to anchors for position of closest match in quality Anchors Test Image 31 24 16 8 2 16? Excellent Very Good Good Fair Poor Not Worth Keeping 9
Softcopy Quality Ruler ¾ Simultaneous viewing of ruler and test image on monitor ¾ Controlled environment: monitor, viewing distance (chin/ head rest), ambient lighting ¾ Facilities available at several participating companies.
JNDs for Published Standards Texture Blur Spatial Frequency Response Color Uniformity Standard Quality Scale (SQS) JNDS 35 30 25 20 15 10 5 0 y = 21.5x + 11.7 R² = 0.9552 0 0.2 0.4 0.6 0.8 1 1.2 Texture Acutance JND loss 35 30 25 20 15 10 5 Data Fit 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.88592 - Acutance Lens Geometric Distortion Lateral Chromatic Aberration 11
Too Much Information ¾ So we have all this data, now what? ¾ Goldmine for the image scientist and engineer ¾ Overwhelming and meaningless for the average consumer (and executive) who just want to know: Is this camera good? Will it give me good quality in the situations I care about? ¾ Need a concise and meaningful way to answer these questions. 12
ICAP - IEEE Conformity Assessment Program ¾ CPIQ Conformity Assessment Steering Committee (CASC) Formed 2014, 13 member companies ¾ CPIQ CASC Objectives: Create a meaningful, easy to understand consumer rating system for mobile cameras Enable carriers, manufacturers and reviewers to effectively convey the image quality of mobile cameras Enable consumers to select the right mobile camera for their needs Create and manage a mobile camera certification program to award ratings 13
Benefits ¾ Consumers can: Make informed, educated decisions Push the industry towards better devices Have a relevant and understandable way to compare devices ¾ Carriers can: Prevent negative user experience by helping to set expectations Market to specific segments (e.g. Selfies, printing, HD, 4K) ¾ Consumer protection in the form of independent verification of results CPIQ Certification Program by independent 3 rd party test labs 11 Nov. 2014 14
From Specs to Stars 15
CPIQ Next Steps ¾ Complete Color Saturation and Visual Noise by June 2015 ¾ Continue work on Video, Auto Focus, and White Balance for Rev 2 ¾ Add even more metrics: Exposure/Sensitivity Latency Vignetting/Luma Shading Veiling Glare Tone/Contrast Flash Dynamic Range Artifacts ¾ Expand to feature testing: HDR/WDR Image/Video Stabilization Panorama And many others 16
ICAP Next Steps ¾ Develop the Consumer Rating System formula ¾ Conduct Consumer Rating System validation study ¾ Prepare test spec and documentation ¾ Develop certification program guidelines ¾ Administer certification programs ¾ Market the Consumer Rating system to build brand awareness 17
How to Join ¾ To join the IEEE P1858 Working Group and/or the CPIQ Conformity Assessment Steering Committee, contact: icap-team@ieee.org 18