Pixel race. Resolution. f/2.8 For a 0.8 µm pixel pitch, the f-number needs to be lowered to f/2.0, according to the definition on the previous slide.

Transcription

1 Pixel race Sense and sensitivity 29 International Image Sensor Workshop Mats Wernersson and Henrik Eliasson Not a race for more pixels: it s a race for smaller pixels! Why do pixels shrink? Becase we can! Easy to sell larger pixel conts Demand for smaller cameras Rev Rev 2 The Pixel Race - Becase we can! Predicted pixel race How to make sense of the megapixel race Time line shows when Sony Ericsson Camera Phones were introdced on the market The maximm sefl pixel cont is ltimately limited by optical diffraction and photon shot noise. Slightly longer time between pixel nodes Rev 3 Rev 4 Resoltion Resoltion calclated as the prodct of diffraction limited optical and pixel geometric. The optical is a polychromatic nder D65 lighting. f/2.8 For a µm pixel pitch, the f-nmber needs to be lowered to f/2., according to the definition on the previos slide. f/2. Dashed lines are the Nyqist freqencies for the respective pixel pitch..4 µm. µm.4 µm. µm This sets an absolte limit for the resoltion of a certain sensor/lens f- nmber combination. µm µm Setting this limit at an of % means that f/2.8 shold not be sed for pixel pitches below. µm, in order to take advantage of the increased pixel cont.. 5 µm. 5 µm Spatial freqency [cy/mm] Spatial freqency [cy/mm] Rev 5 Rev 6 Rev

2 Resoltion Decreasing the pixel pitch even frther, the f-nmber mst keep on decreasing. Lowering the f-nmber is very challenging for the optical designer: aberrations increase, especially off-axis, meaning that it might not be possible to maintain diffraction limited optics for smaller f-nmbers...4 µm f/.4. µm µm 5 µm Spatial freqency [cy/mm] Sharpness Same optical format, varios pixel pitch So far we have only discssed the limiting resoltion of the camera system. The impression of sharpness is critically dependent on lower spatial freqencies. Using SQF (sbjective qality factor), we see that there is only a marginal increase in perceived sharpness when the pixel size is decreased. 45 cm viewing distance, /2.8 optical format SQF vales.4 µm 92% 5 µm 93%. SQF range.4 µm f/2.. µm µm 5 µm Spatial freqency [cy/mm] Rev 7 Rev 8 Photon shot noise and pixel size same sensor size Illminance: 5 lx Photon shot noise and pixel size same sensor size Illminance: lx t = /7 s Gain: db S/N: 9 2. µm t = /7 s Gain: db S/N: 73.4 µm t = /5 s Gain: db S/N: 69 µm t = /35 s Gain: db S/N: 56 t = /5 s Gain: db S/N: µm t = /5 s Gain: db S/N: 55.4 µm t = /5 s Gain: 4 db S/N: 39 µm t = /5 s Gain: 8 db S/N: 24 Rev 9 Rev How to make sense of the megapixel race The maximm sefl pixel cont is ltimately limited by optical diffraction and photon shot noise. However, there is a fair chance to maintain a reasonable image qality between pixel generations if the optical format is kept (and disregarding other implications arising from smaller pixels). The dark side Smaller cameras: Keeping the pixel cont means smaller sensor (optical format) => image qality for a given megapixel cont will decrease with newer pixel generations. Aberration and process technology limitations are being replaced by physical limitations (diffraction, photon shot noise). So far, technology has been limiting the image qality. G. Agranov et al., 27 International Image Sensor Workshop, pp 37-3, Ognqit USA. Rev Rev 2 Rev 2

3 ..4 µm. µm µm 5 µm. f/.4. Normalized spatial freqency f/f N f/2..4 µm. µm µm 5 µm. Resoltion If the pixel cont is maintained, bt the sensor size is decreasing, becase of a smaller pixel pitch, the resoltion will qickly become optics-limited for a diffraction-limited, ideal, system. To keep the same resoltion and sharpness, the f-nmber mst be decreased. If it is not possible to do this, the image qality of cameras with the same pixel nmber will decrease with new sensor generations.. f/2.8.4 µm f/2.8. µm µm 5 µm. Normalized spatial freqency Nf/f f/.4 f/2. Normalized spatial freqency f/f N Sharpness Same pixel cont decreasing optical format with smaller pixel pitch viewing distance 5 µm 82% Very good µm 85% V.g. Ex. µm 89% Excellent.4 µm 9% Excellent. 8 MP f/2..4 µm. µm ( )d(log) = = 2 cycles/deg Granger & Cpery, Photographic Science and Engineering, Vol. 6, 973, pp µm 5 µm. Normalized spatial freqency f/f N Rev 3 Rev 4 Sharpness Same pixel cont decreasing optical format with smaller pixel pitch viewing distance f/2..4 µm ( )d(log) = = 2 cycles/deg Granger & Cpery, Photographic Science and Engineering, Vol. 6, 973, pp Photon shot noise and pixel size same pixel cont, different sensor size Illminance: 5 lx. µm 5 µm 63% Fair Good µm 69% Good. µm 76% Good V.g..4 µm 8% Very good 2 MP µm 5 µm.. Normalized spatial freqency f/f N t = /7 s Gain: db S/N: 9 2. µm t = /7 s Gain: db S/N: 73.4 µm t = /5 s Gain: db S/N: 69 µm t = /35 s Gain: db S/N: 56 Rev 5 Rev 6 Photon shot noise and pixel size same pixel cont, different sensor size Illminance: lx How to make sense of the megapixel race The maximm sefl pixel cont is ltimately limited by optical diffraction and photon shot noise. However, there is a fair chance to maintain a reasonable image qality between pixel generations if the optical format is kept (and disregarding other implications arising from smaller pixels). Bt if the pixel cont is kept constant while pixel size is decreased, image qality will sffer! t = /5 s Gain: db S/N: µm t = /5 s Gain: db S/N: 55.4 µm t = /5 s Gain: 4 db S/N: 39 µm t = /5 s Gain: 8 db S/N: 24 Rev 7 Rev 8 Rev 3

4 N N N N The best compromise between physical size and image qality The best compromise between physical size and image qality How large sensor can we se? A mobile phone mst be slim The camera thickness depends on the sensor format The sensor format dictates the focal length of the lens The maximm sensor format that can be sed ~/2.8 inch /2.8 =.357 inch = Magnm Sensor Lens Thickness ~ Sensor format Rev 9 Rev 2 There are some claims that digital zoom will become more sable with an increasing pixel resoltion. More pixels shold atomatically lead to a larger f/2.8.4 µm 2MP There are some claims that digital zoom will become more sable with an increasing pixel resoltion. More pixels shold atomatically lead to a larger f/2.8.4 µm 2MP x 9% ~ Excellent. ( )d(log) = = 2 cycles/deg 2x 79% ~ Very Good. ( )d(log) = = 2 cycles/deg.. Normalized spatial freqencyf/f Normalized spatial freqencyf/f Rev 2 Rev 22 There are some claims that digital zoom will become more sable with an increasing pixel resoltion. More pixels shold atomatically lead to a larger f/2.8.4 µm 2MP There are some claims that digital zoom will become more sable with an increasing pixel resoltion. More pixels shold atomatically lead to a larger f/2.8.4 µm 2MP 4x 57% ~ Fair. ( )d(log) = = 2 cycles/deg 8x 22% - Off scale -. ( )d(log) = = 2 cycles/deg.. Normalized spatial freqencyf/f Normalized spatial freqencyf/f Conclsion: Decent performance with perhaps 2x digital zoom for a good system (not confirmed). Conclsion: Decent performance with perhaps 2x digital zoom for a good system (not confirmed). Rev 23 Rev 24 Rev 4

5 Field of view vs CRA In prsit of smaller cameras, especially the height is important. A conseqence of decreasing the height is a shorter focal length. Not only does this have a bad effect on image qality (shading), bt also the ser experience is negatively affected, leading to perspective distortion (e.g., close distance portraits). Can image processing help? Distortion correction Noise redction Sharpening F = m F = 36 m Rev 25 Rev 26 Distortion correction Impossible to fix perspective distortion. Neglectable distortion Perspective distortion Straight lines Optical distortion Crved lines Noise redction problem solved? While adaptive noise redction algorithms in general do a good job at preserving sharp edges, the effect on low contrast detail can be horrible. Images copyright I3A the International Imaging Indstry Association Rev 27 Rev 28 Digital sharpening Effect of digital sharpening Introdces artefacts and amplifies noise EDOF? Same problem..2. Example from existing camera System Diffraction Digital sharpening can increase perceived sharpness sbstantially, bt at the expense of noise amplification and introdction of artefacts. Optical. Spatial freqency f/f N Rev 29 Rev 3 Rev 5

6 Digital processing The impact on image qality de to the ISP is increasing. This leads to improvements bt also artefacts that are difficlt to qantify. CPIQ Since the megapixel race seems ot of control, and no-one (except marketing people...) really wants it, an indstry-wide effort needs to be made in order to move the focs away from pixel cont to image qality. The Camera Phone Image Qality (CPIQ) initiative was started by the I3A in 26. Crrently developing image qality measrement methods for: Sharpness Color shading Distortion Lateral chromatic aberration Textre sharpness Sbjective image qality assessment Two different cameras: same sensor, different optics and ISP. Rev 3 Rev 32 Participating companies in Phase 2: AMD OmnivisionTechnologies, Inc. Aptina Palm, Inc. CDM Optics Philips Lmileds Lighting Co. CPIQ Eastman Kodak Co. Sensata Technologies Foveon, Inc. Sony Ericsson Mobile Commnications Freescale Semicondctor, Inc. Sprint FUJIFILM Corporation STMicroelectronics Hewlett-Packard Texas Instrments, Inc. Phase 2 finished April 29. Image Engineering Tessera Motorola, Inc. Verizon Wireless Phase 3 started atmn 29. Open enrollment Nethra Imaging Vista Point Technologies Nokia Corporation Zoran Corporation is taking place at this moment. Phase 3 will contine to develop metrics for: Spatial actance Color consistency and white balance Lminance shading Straylight Signal to noise ratio A method for combining all metrics into a single consmer-oriented rating system is the ltimate goal. Hot Topics Wafer Level Cameras Important for cost and size redction Concerns abot optical performance and yield High Dynamic Range May be the next big step in image qality improvement Tre HDR reqires proper infra strctre (not jst sensor) Adaptive tone mapping for pleasant representation Challenging for small pixels? HD Video A new pixelrace adding one dimension High data rates pt high demands on infra strctre Rev 33 Rev 34 Appendix Original scene radiance image Rendered into srgb by calclating the CIEXYZ vales from the spectral data at each sample followed by a chromatic adaptation to D65 and conversion to srgb according to IEC cont. Rev 35 Rev 36 Rev 6

7 The images were simlated nder the following conditions: The scene was captred sing a Nikon D7 camera with a 28 mm/2.8 Nikkor lens. The relative spectral sensitivity of this camera was previosly measred. From the camera spectral data, a radiance scenewas calclated sch that each pixel contains the radiance spectrm from 4 p to 7 nm. Using only 3 color channels, the spectral accracy is still reasonably good. The sensor images were calclated from the same spectral QE in all cases. All noise sorces except photon shot noise were set to zero. The raw images were processed by a very simple image pipe: white balance, color correction and gamma correction. The color matrix was calclated sing a white-point preserving least sqares method. The sensor parameters that were varied were: - Pixel size - Fll well capacity - Conversion gain - ADC voltage swing For the m pixel, a fll well capacity of 2 electrons was assmed, together with a conversion gain of 6 v/e- and an ADC swing of 4 mv. We are ths assming that the fll well capacity is considerably better than estimated from a simple scaling of fll well vs pixel area. The signal to noise ratio was calclated on the rendered images sing the Photoshop histogram for the lminosity channel. Rev 37 Rev 7