Frog Vision Template matching as a strategy for seeing (ok if have small number of things to see) Template matching in spiders? Template matching in frogs? The frog s visual parameter space PSY305 Lecture 4 JV Stone 1
Primate Vision The richness of a representation depends on what it is to be used for. Simple organisms construct simple representations. 2
Jumping Spider Recall from PSY241 that the jumping spider has eight eyes, each of which is a lens/camera type eye such as our own. Most of these eyes have fish-eye like lenses, giving them a wide field of view but blurry vision. The two forward looking eyes, though, have very high resolution (almost as good as a cat). The jumping spider uses its low-resolution, wide field-of view eyes to detect the presence of moving objects and prey, and then it quickly reorients its body to image the object of interest with its high-resolution eyes (akin to foveating an object). The jumping spider is capable of discriminating its prey (flies) from mates (other spiders) using vision alone. 3
Jumping Spider Mate recognition is achieved via template matching. Eyes are fixed in carapace, and cannot move, but retinae can. Each retina has )-shaped region, so that two retina together yield )( or X-shaped region in visual field. The two retinae move in a conjugate manner (i.e. movements of both eyes are the same). Retinae are scanned over image to see if object is a mate Land (1969). 4
Jumping Spider Evokes courtship Evokes Prey capture Drees (1952) cited in Land (1969) 5
What the frog s eye tells the frog s brain (Lettvin Maturana, McCulloch and Pitts, 1959) Seminal paper. McCulloch and Pitts produced seminal 1943 connectionist paper, so were accustomed to thinking about building models. McCulloch, W. S. and Pitts, W. H. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5:115-133. 6
Frog Brain View from above Optic tectum Side vew Optic tectum Front Tail 7
Frog Visual System No fovea. Retina projects to superior colliculus, also called tectum. Requires movement to see food. 1 million rods and cones project to 0.5 million retinal ganglion cells. These project to tectum. (human has 126M receptors and 1M ganglion cells which project to LGN) 8
Frog Visual System Eye Optic tectum or Superior colliculus Spinal chord Optic nerve Eyes have overlapping views in tiny region of scene (unlike humans). Complete cross-over of optic nerve at chiasm. 9
Method Frog faces inside of hemisphere 14 inches in diameter. Stimulus moved about with magnets on outside of hemisphere. Stimulus on inside of hemisphere 10
Optic angle and image size The size of the image of an object on the retina is proportional to the angle subtended by that object. Eye Same sized retinal images An optic angle of one degree can be subtended by a small nearby object or by a large distant object. Sun or moon = 0.5 degree Fingernail at arm s length = 1 degree. 1 degree Retinal image 11
Four Concentric Ganglion Cell Receptive Field (RF) Types 1 Sustained contrast detectors (2 degrees, unmyelinated) 2 Net convexity detectors (1-3 degrees, unmyelinated) 3 Moving edge detectors (12 degrees, myelinated) 4 Net dimming detectors (15 degrees, myelinated) 30 times as many of (1,2) as (3,4) - reflects RF size. Types 1-4 reflect depth in tectum (with 1 at surface). Myelinated fibres transmit information more rapidly than unmyelinated fibres - consistent with detection of static/changing contrast above. 12
1 Sustained Contrast Detectors Figure 2. Unmyelinated, 2 degrees RF size. Sustained response to any static contrast edge. Response to moving edge in one direction only (2a top trace). Not respond to general changes in illumination. 13
1 Sustained Contrast Detectors are Sensitive to Motion Direction 3 degree disc moved across RF in one direction, then the other. Direction of motion of disc across RF Action potentials Motion Luminance Within RF Time Figure 2a in paper 14
1 Sustained Contrast Detectors Response to Static Edge 3 degree disc moved into RF and stopped => sustained response. Sustained firing Luminance Within RF Timebase = 50ms Time Figure 2b in paper 15
2 Net Convexity Detectors: bug detectors? Figure 3. So-called bug detectors. Unmyelinated. Transient response to <3 degree moving disc. Response small if <1 degree. Response large if motion is jerky (like a fly?). No response to moving straight edge. No response to whole-field motion of array of dots, unless one dot moves wrt whole-field motion (i.e. responds to moving dot only if motion is relative to whole-field). Whole-field motion can be caused by frog moving. 16
2 Net Convexity Detectors Responses to Moving Disc and Edge Figure 3e and 3f in paper. Time Luminance in RF (up => dim, here) Response to 1 degree disc at 3 speeds (left to right). Response to straight edge at 2 speeds (no response). Also respond to static 1 degree disc (figure 3a). 17
3 Moving-Edge Detectors See figure 4. 12 degree RF. Responds only if edge (bar) is moving. Firing rate increases with speed of edge. 18
3 Moving-Edge Detectors Time Transient Firing From figure 4e in paper. Luminance within RF (up => dim). 19
4 Net Dimming Detectors See Figure 5 in paper. 15 degree RF size Sustained response to general (lack of) illumination Illumination Level (up => dimmer) Firing Figure 5d in paper. Dark 20
Recoding the Retinal Image Different cell types encode different types of information. Output of any single cell is inherently ambiguous in terms of what it signals about the retinal image (NEXT SLIDE). For example, may need population of bug detector cells to signal exact location of moving dot - see population coding. May also need to take account of output of moving edge detectors - these provide contextual information 21
Cell s Output Does Not Specify Exactly What is in Cell s RF A spot of light here or here gives identical input to neuron and identical firing rate. Edge of retina 22
What a tectal neuron sees Consider the dendrite of a tectal cell extending up through four sheets (p257), where each sheet of neurons encodes one of four different types of feature (i.e. parameter) in the same retinal location. Each retinal stimulus generates a different signature, depending on how it stimulates neurons in each of the of 4 sheets. 23
What a tectal neuron sees if tectum had only two layers Image on retina Receptive field of bug detector Retinotopic map Bug detector sheet Path of tectal dendrite through tectum Edge detector sheet 24
Definition: Population and Assembly Coding Population coding: If a population of cells all respond to the same parameters then their collective output is called a population code. For example, all cells within a tectal sheet respond to same parameters => population code. Assembly coding: If two cell populations respond to different parameters then their collective output is called an assembly code. For example, cells in different tectal sheets respond to different parameters => assembly code. 25
Recoding the Image (A little speculative connectionism). 26
What a tectal neuron would see if tectum had only two layers To spinal chord Tectal neuron +1-1 Synaptic strength Bug detecting neuron Edge detecting neuron Retinal image 27
What a tectal neuron would see if tectum had only two layers Assume that the tectal neuron has a connection strength of +1 to the bug detector and -1 to the edge detector. If bug detector neuron is firing (denote this as +9) and edge detector is too (also +9) then may not infer a bug is present (because bug detector s output is a false alarm ). This would yield a total input to the tectal neuron of 1x9 + (-1x9) = 0. But if bug detector neuron is firing but edge detector is not firing then may infer a bug is present. This would yield a total input to the tectal neuron of +1x9 + (-1x0) = +9, and the tectal neuron would fire, +1 and the frog would stick out its tongue. -1 28
From Image Space to Feature (or Parameter) Space Degree of bugness == output of bug detector Degree of edgeness == output of moving-edge detector Any image stimulates each detector to different extent, and therefore represents different point in 2D bug-edge parameter space. Thus detectors implement recoding of retinal image space into useful 2D feature space. If have three detector types then parameter space is 3D. 29
2D Parameter Space Very like a bug Unlike an edge A bit like an edge and a bug Degree of bugness x x x Very like a contrast edge Unlike a bug Degree of Edgeness Each point in parameter space represents the shape of a retinal stimulus presented to the same retinal location. 30
Degree of bugness Tectal Neuron s RF in 2D Parameter Space Very like a bug Unlike an edge x x Degree of Edgeness x Each tectal neuron responds according to output of retinal neurons which detect lower order parameters (bug and edges), so the RF of the tectal neuron can be defined as a region or parameter field (yellow disc) in parameter space which corresponds to bugand-not-edge features. 31
Two types of feature detector neurons define a 2D feature space Degree of bugness x N=7 1 2 3 4 5 6 7 x x Degree of Edgeness Require M=N k neurons to tile 2D parameter space. If k=2 and N=7 (as here) then need M= N 2 =49 tectal neurons to tile 2D parameter space. Can t draw 4D feature space, but we know that M=N k neurons are required to code each of k parameters over N values of each parameter. Therefore require M=N k to tile 4D parameter space. Would need N 4 =7 4 =2401 tectal cells per retinal location to tile 4D space. 32
Summary 1 The four tectal sheets of neurons essentially provide a recoding of the retinal image. The retinal image is specified in terms of luminance at each receptor - this description is redundant and not useful to frog. Tectal neurons recode each small region on retina in terms of 4 basic features or parameters, so we have 4 different average firing rates per retinal location. Unlike the retinal receptor outputs, the outputs of these four feature detectors are useful. Taken together they make up the frog s representation of its visual world. 33
Summary 2 Each point on retina represented in 4D parameter space. Bug detector probably insufficient to signal presence of bug, but requires contextual information from other 3 cell types (parameters). Bug detector is easily fooled 34
References Essential Lettvin, J.Y., Maturana, H.R., McCulloch, W.S., and Pitts, W.H., What the Frog s Eye Tells the Frog s Brain, Proc. Inst. Radio Engr. 47:1940-1951, 1959 (supplied as handout). Background Reading: Land, MF, Structure of the retinae of the principal eyes of jumping spiders (Salticidae: dendryphantinae) in relation to visual optics, J Exp Biol 1969 51: 443-470. Land, MF, Movements of the retinae of jumping spiders (Salticidae: dendryphantinae) in response to visual stimuli, J Exp Biol 1969 51: 471-493. Land, MF, Animal Eyes, Oxford University Press, 2002. 35