approximately 10-3 cd/m² to 10 cd/m², i.e. from the scotopic light

Transcription

1 IMPLEMENTATION OF A RETINA MODEL EXTENDED TO MESOPIC VISION Decuypere J. 1, Capron J.-L. 1,2, Dutoit T. 1, Renglet M. 1 1 Université de Mons - UMONS, Mons, Belgium, 2 UCLouvain, Louvain-la-Neuve, Belgium justine.decuypere@umons.ac.be Abstract Mesopic lighting conditions can be physiologically defined as the light levels where both cones and rods of the retina participate in visual perception. At night, the luminance of artificially lighted outdoor environments falls typically within that range. This calls for a photometric model that describes the mesopic spectral sensitivity of the eye at various adaptation levels. However, mesopic issues are not limited to spectral sensitivity changes. The interactions of signals from cones and rods lead to other variations in the characteristics of vision that a computational model of the retina can describe. We develop such a model in order to analyse mesopic visual scenes and simulate the signals transmitted by the retina to the brain. Keywords: Mesopic, Retina, Modelisation 1 Introduction The mesopic domain spans from approximately 10-3 cd/m² to 10 cd/m², i.e. from the scotopic light levels to the photopic light levels (Fig. 1). Many lighting applications fall typically within that range, especially night-time outdoor lighting. However, the basis of the current photometry is the CIE 1924 photopic luminous efficiency function V(λ). This curve only describes the cones spectral sensitivity whereas the human retina contains two types of photoreceptors: cones and rods. The rods sensitivity is different from the cones sensitivity and can be defined by the CIE 1951 scotopic luminous efficiency function V'(λ). In mesopic conditions, cones and rods operate simultaneously, which results in changes in the spectral sensitivity according to the adaptation level. This means that lighting units are unable to assess light in mesopic environments. A mesopic photometric model has recently been validated by CIE in order to define mesopic luminance values and improve night-time driving performance (CIE, 2010). Figure 1. Luminous ranges Nevertheless, mesopic issues are not limited to spectral sensitivity changes. The interactions of signals from cones and rods lead to other variations in the characteristics of vision, such as changes in spatial and temporal contrastt sensitivities and acuity, visual delay and colour sensitivity (Barbur, 2010; Stockman, 2006). We develop a retina model that can take these changes into account. Our model simulates the retina cells processing and results in the retina output transmitted to the cerebral areas. Since considerable processing and filtering of visual information occurs at the earliest stage in the visual system (Wässle, 2004), a computational model of the retina can depict correctly many characteristics of the mesopic perception. 2 Physiology of the retina 2.1 Structure of the retina The retina is a layered neural structure that converts lighting inputs into electrical signals and processes the latter before sending them to the brain. The retina is composed five major layers: the photoreceptors layer, the outer plexiform layer, the inner nuclear layer, the inner plexiform layer and the ganglion cells layer. In summary, bipolar and ganglion cells mediate a direct pathway whereas horizontal and amacrine cells deal with lateral inhibition.

2 The photoreceptors layer is made up by cones and rods. We can distinguish between three types of cones according to their spectral sensitivity (Fig. 2). The L cones are sensitive to the red part of the visible spectrum (long wavelengths), the M cones are sensitive to the green part of the spectrum (medium wavelengths) and the S cones are sensitive to the blue part of the spectrum (short wavelengths). Rods have only one type of visual photopigment. Rods and cones are also differently positioned in the visual field (Curcio, 1990). In the foveola, only L and M cones are present. Their density is very high and quickly decreases with the eccentricity. S cones are far less numerous and their density is nearly constant across the whole retina. Rods are absent from the fovea. Their density reaches a maximum at about from the centre of vision. The eye spectral sensitivity depends not only on the light levels (photopic, mesopic or scotopic) but also on the considered retinal eccentricity. The V(λ) curve describes foveal vision in photopic conditions while the V'(λ) curve describes the scotopic vision, realised by rods. Figure 2. Spectral Sensitivities (replotted from All other retinal neurons are composed by dendrites, a cellular corpus and an axon. The neuron receives inputs from its dendrites and transports them via its axon. The retinal neurons are interconnected by inhibitory and excitatory synapses. The synapses can be chemical (release of a neurotransmitter) or electrical (gap junctions). In the retina, connections between different layers are often managed by chemical synapses while cells from the same layer are often electrically connected. Their positioning inside the visual field is also a function of the eccentricity (Dacey, 1993) as well as the size of their dendritic field. The density decreases and the receptive field size increases when the location of the cell moves away from the fovea (Dacey, 1992). 2.2 Cone pathways through the retina Cones response reflects the energy they absorb, which depends on the intensity and spectral content of the stimulus. The response signal is a graded hyperpolarisation of the membrane which manages the neurotransmitter (glutamate) release at the synaptic terminal of the cones. This release is high in darkness and decreases in light, according to the membrane polarisation. ON-centre bipolar cells depolarize when the retina is stimulated by light whereas OFF-centre cells hyperpolarize (Masland, 2001). A light stimulus can be conveyed through each layer via different circuitries. It consists of many parallel channels that carry different features of the same information. The three major pathways involve either parasol or midget or small bistratified ganglion cells (Fig. 3). Parasol ganglion cells have a diffuse receptive field. Via parasol bipolar cells, they receive messages from many L and M cones in the centre and surround of their receptive field. There are two types of parasol ganglion cells: with an ON-centre and with an OFF-centre (the surround of their concentric receptive field has an antagonist response). Temporally, an ON-centre cell is activated when the stimulus that reaches it is brighter than the previous stimulus whereas an OFF-centre cell is activated when the stimulus is darker than before. Spatially, an ON-centre cell is activated when a stimulus reaching its centre is brighter than the background or when a stimulus reaching the surround is darker than the stimulus reaching the centre. Parasol ganglion cells project their output on the Magnocellular layers of the Lateral Geniculate Nucleus (LGN). This pathway is called Magnocellular pathway (MCa highly convergent pathway) because of the big size of its constitutive cells. The MC-pathway is circuitry that detects quickly luminance changes.

3 Figure 3. Layered structure of the retina Midget ganglion cells have a small receptive field. In the fovea, there is a one-by-one connection between a cone, a midget bipolar cell and a midget ganglion cell. The convergence is minimal. The receptive field centre of a midget bipolar cell is composed by only one cone (L or M) and the surround is composed by several L and M cones. The same cone communicates with the centre of an ON- by horizontal centre and the centre of an OFF-centre bipolar cell. The surround, which is influenced cells, is spectrally mixed, composed by responses of L and M cones. The comparison of the centre and surround reactions allows bipolar cells to detect red/green opponency. There are four types of midget bipolar and ganglion cells: +M/-L (green ON-centre, red OFF-surround), -M/+L, +L/-M and L/+M. As the receptive field size grows up with eccentricity, the centre becomes spectrally mixed and our retina becomes colour blind in the periphery of the visual field. The Midget ganglion cells project on the Parvocellular layers of the LGN. This pathway, called Parvocellular Pathway (PC-Pathway), manages red/green colour opponency and high spatial resolution. Small bistratified ganglion cells receive inputs from ON-centre S bipolar cells (bipolar cells specialised in S cone signals processing) and from OFF-centre diffuse bipolar cells. The comparison of both reactions provides blue/yellow opponency. These receptive field are coextensive instead of concentric: it has no centre-surround opponency. The small bistratified ganglion cells project on the Koniocellular layers of the LGN. This pathway, called the KC-pathway, is responsible of the blue/yellow opponency.

4 2.2 Rod pathways through the retina There are no rod ganglion cells, so the rod messages use the same ganglion cells as the cone messages. Rod signals were mainly recorded in the MC-pathway, but also weakly reach the PCcan be transmitted to pathway (Lee, 1997) and the KC-pathway (Field, 2009). Rod-driven information these pathways via different circuitries (Sharpe, 1999). The first circuitry is responsible for scotopic luminous events. Many rods converge to a rod ON-centre bipolar cell. This cell transmits its signal to A2 amacrine cells, which transfer it to ON-centre and OFFcentre cone bipolar cells. Then, the message follows the cone pathway to ON- and OFF-centre ganglion cells and to the brain. This circuitry is slow but its high convergence allows it to detect single- lighting conditions photon absorption. It works well in scotopic conditions but saturates when the become brighter. The second circuitry takes over in mesopic conditions. Rod signals are transmitted to cones pathways through gap junctions. It is the earliest possible stage to infiltrate the cones pathways. Rod spherules contact neighbouring L and M cone pedicles, which allows electrical transmission. Rod information is driven to ON- and OFF-centre bipolar cells and thence to ON- and OFF-centree ganglion cells. This circuitry is less-sensitive but faster than the previous one because of its shorter integration times. 3 Modelisation Computational models of the retina already exist but are often limited to photopic vision, considering only the cone inputs to the following retina neurons (Shah, 1993; Herault, 1996; Wohrer, 2007). We implement a monochromatic model of the retina that simulates the processing of cones and rods signals in the PC and MC pathways in mesopic conditions. In these conditions, the rod-driven signals are transmitted to cones via gap junctions. The model takes an image as input and processes it to build maps of the PC and MC pathways responses. The model is implemented in C++ using the Open CV library. 3.1 From photoreceptors to outer plexiform layer Visual adaptation is a mechanismm that allows our retina to see across a large luminance range. The adaptation state is controlled by pupil size, by photoreceptor pigment bleaching and regeneration, by slow neural adaptation and by fast neural adaptation (Hood, 1986). Each of those mechanisms has its own temporal behaviour. At a given adaptation level, the response of a photoreceptor follows an S- shaped curve that can be described by the Naka-Rushton equation (Equation 1) introduced in 1966 (Irawan, 2005). In this equation, R(I) is the response of the cell to the luminance stimulus I, B is the amplitude of the curve depending on photopigment bleaching, σ defines the horizontal position of the curve, representative of the neutrally-driven adaptation and n is a sensitivity control. The luminance values I are calculated with V(λ) for cone vision and with V'(λ) for rod vision. This means that the photoreceptor is more sensitive to a range of luminance values around its adaption luminance. If the stimulus is too bright or too dim compared to the adaptation level, the response is compressed (Fig. 4b). (1) Figure 4. Computational processing in the photoreceptors layer

5 To implement the time-dependent visual adaptation and dynamic response of rods and cones, we use the Pattanaik model (Pattanaik, 2000). This model considers neural adaptation and chemical adaptation (pigment bleaching). Pupil size is neglected due to its small influence. These mechanisms have different time constants. The pigment bleaching and regeneration mechanism is slow and spatially localized. Neural adaptation stems from horizontal feedbacks (Shah, 1993), so that it is a spatial adaptation. It is faster than pigment bleaching. We model this spatial network with a Gaussian filter whose spatial extent is slightly larger than the horizontal cells receptive fields. Both mechanisms are slower for rods than for cones (Cao, 2008). The adaptation model organisation is shown on Figure 4 (a,b). In mesopic conditions, rods responses are provided to cones via gap junctions. The number of rods coupled to one cone depends on the eccentricity. To account for this factor, we map the density of photoreceptors according to Curcio tables (1990) before sending the cones responses to horizontal and bipolar cells (Fig. 4c). 3.2 Bipolar cells processing In the outer plexiform layer (OPL), cones, rods and horizontal cells messages are transmitted to the bipolar cells. The photoreceptors outputs are coupled via gap junctions, so that the final cone response transmitted to horizontal and bipolar cells is an averaging of the neighbouring cones and rods. This diffusion process is described by the convolution of the photoreceptors output with a Gaussian operator (Equation 2). The cells have also a temporal latency that is reflected in the model by the convolution of the signal with a temporal exponential low-pass filter (Equation 3). The message is then transmitted to horizontal cells. They have the same functioning as the cones, which can be modelled using a spatial Gaussian filter and a temporal low-pass filter. Their feedback pathway to cones is the basis of the negative surround of the receptive field of bipolar cells (Kamermans, 1999). As the connectivity is higher in the horizontal cells network than in the cones network, the Gaussian filter is larger. The horizontal celll signal is temporally more low-pass than the cone signal because the signal undergoes one more cellular integration.,, exp, exp, 0, 0 0 The receptive fields of bipolar cells of the PC and MC pathways are concentric, with a centre-surround organisation. The centre input is provided by cones, the antagonist surround by horizontal cells. Spatially, it is a difference of Gaussians and temporally, the surround is slower than the centre. This spatio-temporal kernel is expressed by Equation 4, in which σ c and σ s represent respectively the Gaussian widths of the centre and the surround and τ c and τ s, the time constants of the centre and the surround. The Gaussian width depends not only on the type of bipolar cell (midget or diffuse) but also on the eccentricity (Cröner, 1994). One midget bipolar cell receives inputs from one cone in its centre while the diffuse bipolar cells receive inputs from several cones. The α c and α s epresents the relative weight of the centre and surround.,,.,,,.,.,,,., (2) (3) (4) Figure 5. Computational processing in bipolar cells

6 Figure 5 shows the operations applied to the photoreceptors outputs in order to transform them into bipolar cells outputs. In this simplified version of the model, the gain control occurs in the bipolar cells layer. It is expressed by gain amplification and by a saturation function (Fig. 5c). In a future more complex model, it should come from the amacrine cells feedbacks (Wohrer, 2007). As we do not model this processing, bipolar cells outputs are simply provided to ganglion cells which transform them into spikes. 4 Results Our model takes an image as input and processes it to mimic the retina mechanisms. The model needs the angle of the visual field encompassed in the input image. Figure 6 presents results obtained with a simplified image of a square positioned in front of the fovea (the image is 2 wide). In the first test (Fig. 6a), the stimulus luminance is 1 cd/m² while the background luminance is 0,01 cd/m². Both are mesopic light levels but the vision is nearly photopic because there are few rods in that central part of the visual field. In the second test (Fig. 6b), the stimulus luminance is cd/m² while the background luminance is 10 cd/m². Those are photopic light levels. The retina model is adapted to the background luminance at t < 0 s. At t = 0 s, the image is displayed to the model. The outputs of the cones, of the horizontal cells and of the midget and diffuse bipolar cells are shown as a function of time. We can see the averaging created by Gaussian diffusion in the cones and in the horizontal cells layers. As the PC pathway receives only inputs from one cone in the fovea, its output is simply the difference between the cones output and the horizontal cells output. The MC pathway receives inputs from several cones. Visually, its output is more blurred. In the first milliseconds, the responses of the PC and the MC pathways are spatially low-pass. After 200 ms, the response of the PC pathway becomes band-pass. Starting from that moment, edges are detected. This behaviour is representative of cones foveal vision. It generates a high spatial acuity. Although the stimulus is 100 times higher than the background in both tests, the amplitude of the reaction is not the same. Visual perception decreases in mesopic conditions. Rods signals could have increased the response but they are nearly absent from the fovea. Figure 6. Results on a simplified image

7 Figure 7 shows results obtained with a test image where the stimulus is a set of strips brought together in squares. The strips luminance is 1 cd/m² and the background luminance is 0,01 cd/m². The image encompasses 40 of the visual field. The spatial resolution is high in the fovea and decreases with eccentricity in the PC pathway (Fig. 7b). The strips of the first test image are separately perceived in the central area but are seen as squares in the periphery of the visual field. This is due to the variable Gaussian filter. In the periphery of the image (20 from the visual axis) ), the rods density is high so that they provide a higher response, especially to the diffuse bipolar cells (Fig. 7d) At an eccentricity of 5, the response of a cone and a rod is described as a function of the stimulus intensity and the adaptation state of the photoreceptor (Fig. 7a, b). This adaptation state rises with time. It would have gone down if the stimulus had been lower than the background. Since the stimulus luminance gets closer to the adaptation luminance, the response decreases. Figure 7. Results in the visual field Figure 8 displays visual scenes of real night-time situations. The input of the model is a high-dynamicimages allow storing range picture taken in the city of Mons, Belgium. High-dynamic-range (HDR) more accurately physical values of luminance. The photopic and scotopic luminances are calculated from the RGB coordinates (Decuypere, 2009). The photopic luminance Y is provided by the CIE XYZ colour space. Y is calibrated with a luminance meter. The scotopic luminance Y' is approximated by the empirical formula from Larson (1997). For more accurate results, multispectral images should be used because they enable real luminance calculations from spectral sensitivities of cones and rods pixel-by-pixel.

8 We can see the band-pass behaviour that works as an edges detector (Fig. 8a). For example, the outline of windows is particularly emphasized. We can also see the periphery blur that occurs more in the MC pathways than in the PC pathway (Fig. 8b). Figure 8. Results on real night-time scenes 5 Conclusion We propose a computational model that simulates the retina behaviour in mesopic conditions. The temporal and spatial constants used in the processing are directly related to the visual neurosciences field. Up to now, we have investigated the PC and MC pathways using non-chromatic fixed stimuli. In the future, it is planned to implement the coloured pathways adding the KC pathway to the model and the red/green opponency to the already studied PC pathway. We will use multispectral sequences instead of HDR fixed images. The current model can analyse the effect of changes in luminance, spectral characteristics and light distribution on the spatial and temporal contrast sensitivity of the retina. This improves our understanding of mesopic vision.. References BARBUR J.L. & STOCKMAN A Photopic, Mesopic and Scotopic Vision and Changes in Visual Performance. In: Darlene A. Dartt, editor. Encyclopedia of the Eye, Vol. 3. Oxford: Academic Press, CAO D., POKORNY J., SMITH V.C. & ZELE A.J Rod contributions to color perception: Linear with rod contrast. Vision Research, 48,

9 CIE CIE 191:2010. Recommended System for Mesopic Photometry Based on Visual Performance. Vienna: CIE. CRONER L.J. & KAPLAN E Receptive Fields of P and M Ganglion Cells Across the Primate Retina. Vision Research, 35 (1), CURCIO C.A., SLOAN K.R., KALINA R.E. & HENDRICKSON A.E Human Photoreceptor Topography. The Journal of Comparative Neurology, 292, DACEY D.M. & PETERSEN M.R Dendritic field size and morphology of midget and parasol ganglion cells of the human retina. Proc. Natl. Acad. Sci. USA, 89, DACEY D. M The Mosaic of Midget Ganglion Cells in the Human Retina. The Journal of Neuroscience, 13 (12), DECUYPERE J., CAPRON J.-L. & RENGLET M Influence of Mesopic Lighting Conditions on Pedestrian Visual Field in Urban Environments. Proceedings of Lux Europa 2009, FIELD G., GRESCHNER M., GAUTHIER J. L., RANGEL C, SHLENS J., SHER A., MARSHAK D. W., LITKE A. M. and CHICHILNISKY E.J High-sensitivity rod photoreceptor input to the blue-yellow color opponent pathway in macaque retina. Nature Neuroscience, 12 (9), HERAULT J A model of colour processing in the retina of vertebrates: From photoreceptors to colour opposition and colour constancy phenomena. Neurocomputing, 12, HOOD D. C. & FINKELSTEIN M. A Sensitivity to light. In Handbook of Perception and Human Performance: Sensory Processes and Perception, Boff K. R., Kauffman L., Thomas J. P. (Eds). John Wiley & Sons, Inc., 1986, ch. 5. IRAWAN P., FERWEDA J. A. & MARSCHNER S. (2005). Perceptually Based tone Mapping of High Dynamic Range Image Streams. In Proceedings of Rendering Techniques, KAMERMANS M. & SPEKREREIJSE H The feedback pathway from horizontal cells to cones. A mini review with a look ahead. Vision Research, 39 (15), LARSON G. W., RUSHMEIER H. & Piatko C A visibility matching tone reproduction operator for high dynamic range scenes. IEEE Transactions on Visualization and Computer Graphics, 3 (4), LEE B., SMITH V. C., POKORNY J. & KREMERS J Rod inputs to macaque ganglion cells. Vision Research, 37, MASLAND R. H The fundamental plan of the retina. Nature Neuroscience, 4 (9), PATTANAIK S.N., TUMBLIN J.E., YEE H. & GREENBERG D.P Time-Dependent Visual Adaptation for Realistic Real-Time Image Display. Proceedings of SIGGRAPH 2000, SHAH S. & LEVINE M.D Information Processing in Primate Retinal Cone Pathways: A Model. Technical Report TR-CIM SHARPE L.T. & STOCKMAN A Rod pathways: the importance of seeing nothing. Trends in Neurosciences, 22 (11), STOCKMAN, A. & SHARPE, L.T Into the twilight zone: the complexities of mesopic vision and luminous efficiency. Ophthalmic and Physiological Optics, 26, WÄSSLE H Parallel Processing in the Mammalian Retina. Nature Reviews Neuroscience, 5, WOHRER A., KORNPROBST P. & VIEVILLE T Virtual Retina: a biological retina model and simulator; with contrast gain control. Report INRIA. Acknowledgements J. Decuypere is supported by the Fonds National de la Recherche Scientifique (FNRS).