Dealing with the Complexities of Camera ISP Tuning

Size: px

Start display at page:

Download "Dealing with the Complexities of Camera ISP Tuning"

Ernest Wood
5 years ago
Views:

1 Dealing with the Complexities of Camera ISP Tuning Clément Viard, Sr Director, R&D Frédéric Guichard, CTO, co-founder 1

2 Dealing with the Complexities of Camera ISP Tuning > Basic camera image processing > First revolution: optical and sensor defects corrections > Second revolution: miniaturization > ISP, A key differentiator in image and video processing 2

3 Introduction «Basic» digital image processing in a camera 3

4 RAW data from Sensor 4

5 After «demosaicing» 5

6 After exposure adaptation 6

7 After «white ballance» 7

8 After color rendering 8

9 After sharpening 9

10 A digital revolution in cameras Optical and sensor defect corrections 10

11 Geometric distortion 11

12 Geometric distortion 12

13 Volume anamorphos perspective error with wide angle lenses 13

14 Volume anamorphos perspective error with wide angle lenses 14

15 Optical vignetting 15

16 Chromatic aberation 16

17 Chromatic aberation 17

18 Tone mapping 18

19 Tone mapping 19

20 Tone mapping (with HDR sensor) 20

21 Tone mapping (with HDR sensor) DxO Labs 2016 PREPARED FOR: RENAULT 21

22 Clipping Recovering saturated area 22

23 Clipping Recovering saturated area 23

24 Stabilisation and Electronic Rolling Shutter (ERS) effect Yaw (Rotation) Pitch (Rotation) ERS vertical Yaw Z Pitch Translation X Translation Y Translation Z X Roll Y ERS horizontal 24

25 Range of Digital Corrections with Advanced ISPs > Optical aberrations > Distortion > Blur (spherical aberration, field curvature, coma, astigmatism, motion) > Chromatic aberrations > Vignetting > Flare, veiling glare > Light & Sensor > Noise > Dynamic range > Contrast > Atmospheric haze > Sensor limitations > Field non uniformity (color, black level, ) > Defective pixels, Dust > Clipping > Metamerism > Others > Video stabilization > Electronic rolling shutter 25

26 Miniaturization the second camera revolution Will they eventually reach same performance? 26

27 Sensor Miniaturization Challenge 22 times less light 4.5 stops 27

28 Performance improvement over 10 years > CMOS Sensor improvement : +1.5 stops +1.5 stops DxOMark sensor score, APS-C cameras > Digital processing gain: +3 to +4 stops 28

29 Nikon D70s, ISO 3200 jpg 29

30 Nikon D70s, ISO 3200 RAW + DxO Optics Pro v3 30

31 Nikon D70s, ISO 3200 RAW + DxO Optics Pro v7 31

32 Nikon D70s, ISO 3200 RAW + DxO Optics Pro v9 32

33 Nikon D70s, ISO 3200 RAW + DxO Optics Pro v11 33

34 Nikon D70s, ISO 3200 jpg 34

Sensor color response changes with CRA Photosite 350

35 Sensor color response changes with CRA Photosite Color response as a function of the angles

36 ISP A key differentiator in image and video processing 36

37 «ISP» as a key differentiator > Enables camera hardware and scene artefact corrections > Only remaining differentiator since access to best sensor and lens is now ubiquitous > Paradigm shift, cameras are designed considering possible digital corrections > Computing power requirement consistently increases Delivering 2,000 Ops/pixel with 240 Mpix/s (24 frames at 10Mpix per second) requires 500 GOps/s > Very significant investment within the Mobile industry (e.g. iphone camera engineering team ~ 800 people) 37

38 Same HW, same ISP, same BOM same image quality? DxO Mark Mobi e score (Photos a d Videos) 38

39 «Camera Tuning» For correcting each image, ISP requires 10,000+ register settings to be adapted to the situation. > Per serie Calibration > Per unit Calibration > On the fly parameter estimations 39

40 Image Quality Evaluation Challenge > What is Image Quality? > Perception of how a picture looks good > In essence a subjective matter that can be modeled with engineering tools > What influences Image Quality > Shooting conditions: illumination, illuminant, dynamic > Content has a strong influence on Image Quality perception and image processing > Image Quality perception is different between imaging expert, professional photographers and consumer

41 Image Quality Evaluation Challenge - Subjective vs. objective > Objective evaluation > Objective means a measurement that is neutral, operator independent: 2 cm, 15 meters, 150 kg, 2 g, -5 C, 50 C, > A device must provide figures (metrics) that are related to image quality > Normalizations may be necessary to have comparable metrics (cameras with different resolutions) > Only addresses a set of metrics (some artifacts may be ignored) > Subjective evaluation > Subjective means real people give an opinion like: big, small, heavy, light, cold, hot, > People can judge the quality of photographs > Methodology is key to get non-biased results

42 Camera IQ assessments Thousands of videos/photos are required to characterize IQ along different dimensions Temporal behavior Dynamic behavior Photogrammetry and 3D Specifying AND verifying image quality targets are tough challenges 42

43 Just Noticeable Difference (JND), a Matter of Statistics > The smallest statistically measurable difference of perception, e.g., smallest perceivable distance between 2 parallel lines > Typically, defined when half of the people perceive a difference and the other half are guessing (50% JND) 50% perceive a change 50% guessing

44 Anchored Scaling Concept Anchors C D E F?? Test image

45 JND MTF (%) Objective/subjective correlation example: Sharpness JND mapping After extensive visual experiments, one could show that the Acutance objective metric linearly correlates with perception of sharpness (expressed with JND unit) CSF MTF Frequency (cycle/pixel) Acutance

46 Back to the real world! > Unfortunately the assumptions to build a complete multi-variate analysis based scoring system are not met > Only few metrics are linearly correlated to perception > They are not strictly orthogonal to each others > New issues comes with every technological improvement > Since we can t build a unified scoring system based on a set of perceptual metrics, a Multi-Modal approach is required

Recommendation for building a relevant Image or video score > Multi-modal testing approach > Objective measurements in lab environment > Perceptually-correlated metrics when available >

47 Recommendation for building a relevant Image or video score > Multi-modal testing approach > Objective measurements in lab environment > Perceptually-correlated metrics when available > Perceptual analysis from natural scenes > Multi-level testing results > Overall, Photo, and Video scores > Top-level scores (open scale) non technical > Detailed reports for engineering 47

48 Example of natural scene set used for mobile camera application 48

49 Conclusion > Eventually, all industries would benefit from Smartphone technology breakthroughs > ISP can fix many optical and sensor defects > ISP technology improvements are a key enabler for miniaturization > Mastering tuning complexity and IQ evaluation is a key differentiator > DxO has developed robust methods to deal with this problem 49

Thank you! cviard@dxo.com 1 / DxO USA 1407 Minnesota Street San Francisco, CA 94107-3519 USA info.usa@dxo.

50 Thank you! 1 / DxO USA 1407 Minnesota Street San Francisco, CA USA info.usa@dxo.com Tel: / DxO France 3, rue Nationale Boulogne-Billancourt France info.eu@dxo.com Tel: / DxO Korea 16th Floor, Gangnam Building Seocho-dong, Seocho gu Seoul Korea info.kr@dxo.com Tel: / DxO Taiwan Tai Centre 2/F-A170 Dunhua North Road Taipei Taiwan Tel: / DxO Japan 3F Metlife Kabutocho Bldg 5-1 Kabutocho Chuo-ku Nihonbashi, Tokyo Japan info.jp@dxo.com Tel: +81 (0)

DxO Analyzer Stabilization Module

DxO Analyzer Stabilization Module This Module includes essential hardware and software to perform stabilization performance testing. Users can analyze optical and digital stabilization for photo and video. It also measures the performance