An Engineer s Perspective on of the Retina. Steve Collins Department of Engineering Science University of Oxford

Transcription

1 An Engineer s Perspective on of the Retina Steve Collins Department of Engineering Science University of Oxford

2 Aims of the Talk To highlight that research can be: multi-disciplinary stimulated by user requirements stimulated by opportunities created by new research results To suggest two topics that might lead to a discussion and/or investigation in a school

3 Plan for Talk A challenge in digital camera design An engineers view of the retina Research into even better digital cameras

4 Economics of Cameras To create a good quality image you need a million or more picture elements (pixels) to be affordable each pixel therefore needs to be very small and hence very simple Since assembling components is relatively expensive the whole camera should be made in a single chip Both of these aims are possible if we use the manufacturing processes originally developed to make computer chips

5 Light Sensing in Si Computer chips are made from silicon Photons of visible light can create electron-hole pairs in silicon If these pairs are created in the field within a p-n junction they generate a current however the current is femtoamps or less To detect this tiny current we need to use it to discharge a capacitor for long enough for it to create a detectable voltage change

6 When M1 stops conducting the photocurrent (I ph ) can discharge the capacitor C pixel The resulting voltage is then detected after a fixed time This voltage is proportional to the light intensity A pixel circuit

7 The voltage in the pixel is detected using M2, M3 and M4 (which is shared) These transistors connect each pixel in turn to the single output Only 3 transistors and a photodiode are needed in each pixel A pixel circuit

8 Small is beautiful Using 180nm long transistors the pixel circuit can fit into 4µm by 4µm 1 million pixels fit into an area of 4mm by 4mm In this case small is beautiful for two reasons: Millions of cameras can be made at a low cost Cameras fit unobtrusively into other products e.g. digital cameras in mobile phones now the dominant market for digital camera chips

9 We each almost certainly have at least one Digital Cameras Low-cost and convenience means digital cameras are now common We also see them on streets and in trains

10 Cameras in Cars In the mid-90s car makers spotted the potential of digital cameras

11 Cars Today The huge market meant several companies worked on developing cameras for cars After 15 years of R&D you can buy automatically activated windscreen wipers a camera to help when reversing blindspot cameras However they are too expensive to be included in mass market cars

12 The Problem The problem is that the dynamic range of scenes that need to be imaged is too large for cameras Consumers accept the resulting inconvenience but drivers can t accept the resulting danger We need a solution that is inexpensive, effective and creates displayable images

13 The Problem s Source To solve the problem start with its origin A model of a visible scene O(x,y)=L(x,y).R(x,y) R(x,y) object reflectance the important information (2 decades max) L(x,y) illumination variations varies by up to 4 decades in a scene dominates the dynamic range contains relatively little information

14 An existing solution The human visual system deals with this dynamic range problem effortlessly This is interesting for two reasons: In many applications captured images are displayed to the HVS and we only need to preserve the information that it requires We may want to replicate it s strategy

15 The Mammalian Retina Three layers of cells form the information path rod/cones bipolar cells ganglion cells Two sets of cells form lateral connections amacrine and horizontal cells

16 Amacrine/Ganglion Cells The function of the Amacrine + Ganglion output system varies between species not that well understood can t be doing anything that is universally useful Their inputs from the Bipolar cells are the effective inputs to the HVS

17 Bipolar Cells The Bipolar cells are the effective sensors for the rest of the system The function of the Bipolar cells is the same in all species They must do something that is universally beneficial Their inputs are from the Rods or Cones and the Horizontal cells

18 The Horizontal Cells Excited by the cones (or rod) cells Form a laterally connected network similar to a network of leaky resistors The result is a response that represents a locally weighted average of the cone responses

19 The Cone The horizontal cells bias the cones so that the response of each cone encodes differences between its response and that of its neighbour This creates a non-linear relationship between the amount of light and the cone s response There are different opinions about the exact form of the non-linear relationship Some believe that it is logarithmic, and this is convenient for modelling and image processing

20 Bipolar Cells Excited by the nearest few cone cells Inhibited by the local horizontal cells these represent the average cone response over a slightly wider area The result is a centre-surround receptive field

21 The function of the retina The non-linear logarithmic response compresses the input dynamic range and splits the product into a sum Log(R(x,y)L(x,y))=Log R(x,y) + Log L(x,y) The centre surround receptive fields then reduces the slowly varying component of the input signal

22 Effect of Spatial Filters The intensity of the illuminant varies relatively slowly across a scene The bipolar cells therefore attenuate the relatively unimportant illuminant dependant part of the image

23 An example image To see the effects of the bipolar cell receptive fields consider this image B is obviously brighter than A

24 An optical illusion Add two uniform grey bars These bars show that A and B are actually the same shade of grey

25 Lessons learnt The processing in the retina means that everything is relative We only need to preserve local differences between pixel responses

26 Summary The dynamic range of a scene is dominated by the unimportant illuminant intensity changes The retina compresses an image using logarithmic sensors and spatial filtering This means that we are only sensitive to local changes in intensity This feature of the HVS can be exploited to create displayable wide dynamic range images using Tone-Mapping Algorithms

27 Tone Mapping Tone Mapping Algorithms are needed to display wide dynamic range images Local Tone mapping algorithms tend to have three stages convert the pixel intensities onto a log scale calculate the local average response at each pixel the best algorithms don t average across any edges create an image from the difference between the pixel response and the local average response This preserves useful information by being very similar to the processing in the retina

28 Local Tone Mapping Local tone mapping results can be spectacular However, currently too computationally expensive for real time implementation in cars One aim is to create a camera that performs local tone-mapping

29 A Future Problem There will soon be so many cameras in a car that the driver will suffer sensory overload Our aim is to add intelligence to each camera For example add the automatic detection of pedestrians who might be in danger preferably using pixel level information to reduce the processing load Humans use colour as a key descriptor Can we use colour usefully to automatically detect pedestrians and alert the driver if someone might step in front of the car?

30 The Colour Problem Colour imaging depends upon the responses of three sensors (usually referred to as RGB) The output from each sensor depends on the spectrum of the illuminant light Daylight is particularly important, but unfortunately, its spectrum changes!

31 Effect of Spectral Changes In daylight colours can look very different at different times of the same day! In contrast we see the same colours at any time of day An ability known as colour constancy

32 Colour in the retina Again the retina has a capability that we need to replicate The retina contains three types of cones with peak responses at 559nm, 531nm and 419nm Bipolar cells respond to one of three combinations of cones an achromatic channel encodes lightness the local difference between Red and Green the local difference between Blue and Yellow (=G+R)

33 What should we do? It seems to be a good idea to mimic the function of the bipolar layer compress dynamic range and achieve colour constancy But is this really a good idea? To decide this consider the following:

34 These two peacocks are obviously the same Two Peacocks

35 Optical Illusions Place the same birds onto a background Now one looks blue and the other looks yellow This context dependence means that it is not sensible to replicate the strategy of the retina in detail

36 An alternative solution We need a descriptor of colour that is independent of the illuminator and the background This mean any solution should be: only based on information from each pixel independent of illuminant intensity compatible with achieving a wide dynamic range To develop a solution we have looked at daylight

37 Daylight Sunlight at the top of the atmosphere has a spectrum similar to a blackbody at a temperature of 5750K This light is scattered and absorbed by the atmosphere hence: depends upon atmospheric conditions depends upon the path length through the atmosphere which itself depends upon solar elevation (or the time of day)

38 Model for Daylight There is an international model for daylight Each spectrum characterised by an equivalent black-body temperature We have based our method on assuming a black-body spectrum

39 A useful descriptor If the illuminator has a blackbody-like spectrum then illuminator dependent terms are removed using data from three sensor responses to create F = log( R2 ) { α log( R1 ) + (1 α)log( 3)} 1 R The coefficients α and (1- α) remove the illuminator intensity dependence If α is chosen correctly the effect of illuminator spectral changes can also be removed

40 Data from a camera In ideal situations the proposed feature separates some colours However colours exist in a 2D space We need a second descriptor

41 A Second Descriptor A second independent descriptor needs 4 sensors with different spectral responses Cameras only have 3 sensors (R,G and B) A simple ratio (B/R) or the equivalent difference between logarithmic responses (log(b)-log(r)) is at least independent of intensity

42 Second Descriptor Results Results from a camera taken on two days Daylight not a blackbody There is also noise in the pixel data Despite everything the results suggest that F 1 is almost independent of the illuminant These features might be useful for detecting pedestrians

43 Initial Region Identification Image of a finger and a colour checker chart The chart has a colour called skin and one called dark skin Initial results show that a simple test on the two features picks out the finger, the two skin patches and some black

44 Summary We need inexpensive cameras with easily processed outputs Processing might take two forms local tone mapping to create displayable images colour descriptor extraction for machine vision Both forms of processing are easier with pixel responses that are proportional to the logarithm of the incident light

45 The Oxford Pixel Very similar to a conventional pixel Only additional device is M p2 which can isolate the photodiode from the output circuit

46 Operation of the Pixel Reset pixel with M p2 conducting Photodiode discharges source of M p2 Reference voltage increases the gate voltage of M p2

47 Operation of the Pixel M p2 will stop conducting when V ref (t) V diode = V th,mp2 Effective integration time depends on the photocurrent

48 Temporal Response Monitor the output from a pixel As expected the effective integration time depends upon Vout (V) lux 840 lux photocurrent Time (s)

49 Pixel Response This photocurrent dependent effective integration time gives a logarithmic response with a user controlled slope 2.0 Output Voltage (V) mV/decade V ref1 with S=0.25 V ref2 with S= mV/decade Illumination (lux) 10 5

50 Logarithmic Image Image from our lab window in February Dynamic Range is 104dB (>10 5 ) So good that it is difficult to display

51 Summary Imaging WDR scenes is potentially lucrative but surprisingly challenging The surprise is because the HVS is very good at imaging scenes effortlessly The example of the HVS suggests a logarithmic pixel is a very good idea A logarithmic pixel has been developed It now has to be built into useful cameras

52 The Probable Outcome

53 Research and Applications The application of cameras in cars is an example of research emerging from a user requirement Applications also emerge from technical opportunities created by research

54 A future opportunity Three facts and a new technique create a future opportunity for a technological advance The first fact is that silicon can detect photons with wavelengths between around 650nm and 900nm that are invisible

55 Plants The second fact is that healthy plants have an invisible reflectance edge between 670nm and 750nm. Finally, lack of nutrients or fungal infection or insect damage all reduce this edge We could image in the visible and around 750nm to gather this information Effect of insect damage on the reflectance of rice leaves

56 Smart Farming Information on the health of each plant would help a farmer to farm individual plants a widely discussed concept know as Smart Farming reduce the amount of fertiliser and pesticides needed Problem is gathering the information despite the variations in intensity and spectrum of daylight same problem as faced by car makers However, in this case we will have four sensors Currently working on obtaining two illuminant independent features using (RGB + NIR data)

57 The Future The logarithmic pixel is only the start Need inexpensive cameras that perform localtone-mapping to create displayable images Also need cameras that highlight possible dangers using colour information There is a possibility that cameras sensitive to R,G,B+NIR could be applied to smart farming

58 Conclusion The HVS is a fantastic system However it is not perfect a fact that is exploited in creating optical illusions As an engineer my aim is to create systems that are even better than the HVS by: processing colour information in a way that doesn t depend on context gathering useful invisible information

59 Question for the Audience Can or Optical illusions and the HVS The colour response of digital cameras form the basis for a discussion and/or investigation at a school?

60 Thank you for your attention