Probing sensory representations with metameric stimuli

Size: px

Start display at page:

Download "Probing sensory representations with metameric stimuli"

Melvyn Turner
5 years ago
Views:

1 Probing sensory representations with metameric stimuli Eero Simoncelli HHMI / New York University 1

2 Retina Optic Nerve LGN Optic Visual Cortex Tract Harvard Medical School. All rights reserved. This content is excluded from our Creative Commons license. For more information, see Where does all that visual information go? [figure: Hubel 95] 2

3 Destiny of sensory information Sensory input Discard Act Remember (brain) 3

4 Metamers Two stimuli that are physically different, but appear the same to a human observer Classic example: trichromatic color perception Another example: texture perception 4

5 Spectral nature of light Diagram of a prism removed due to copyright restrictions. Please see the video. This image is in the public domain. [Newton, 1665] 5

6 Perceptual color matching experiment Arbitrary test light Mixture of 3 primary lights power power wavelength Courtesy of David Brainard. Used with permission. wavelength [Young, Helmholtz, Grassman, etc, 1800 s; slide c/o D. Brainard] 6

7 Perceptual color matching experiment Arbitrary test light Mixture of 3 primary lights power power wavelength Courtesy of David Brainard. Used with permission. wavelength [Young, Helmholtz, Grassman, etc, 1800 s; slide c/o D. Brainard] 7

8 Theory (Grassman, 1853): the visual system performs a linear projection of the wavelength spectrum onto a three-dimensional response space L N Harvard Medical School. All rights reserved. This content is excluded from our Creative Commons license. For more information, see Predicts/explains perceptual metamers - lights that appear identical, but have physically distinct wavelength spectra (1800 s) Codified in CIE standards for color representation (1931) Underlying mechanism (cone photoreceptors) verified (1987) 8

9 Figure removed due to copyright restrictions. Please see the video. Source: Figure 3 from Baylor, D. A., B. J. Nunn, and J. L. Schnapf. "Spectral sensitivity of cones of the monkey Macaca fascicularis." The Journal of Physiology 390, no. 1 (1987): Courtesy of Denis Baylor. Used with permission. [Baylor, Nunn & Schnapf, 1987] 9

Visual texture Springer. All rights reserved.

For more information, see https://ocw.mit.edu/help/faq-fair-use/.

"A parametric texture model based on joint statistics of complex

10 Visual texture Springer. All rights reserved. This content is excluded from our Creative Commons license. For more information, see Source: Portilla, Javier, and Eero P. Simoncelli. "A parametric texture model based on joint statistics of complex wavelet coefficients." International journal of computer vision 40, no. 1 (2000): Homogeneous, with repeated structures 10

11 11

For more information, see https://ocw.mit.edu/help/faq-fair-use/. Source: Portilla, Javier, and Eero P. Simoncelli.

12 Julesz (1962) Hypothesis: Two textures with identical Nth-order pixel statistics will appear the same (for some N). Hand-constructed counter-examples (N=3): Springer. All rights reserved. This content is excluded from our Creative Commons license. For more information, see Source: Portilla, Javier, and Eero P. Simoncelli. "A parametric texture model based on joint statistics of complex wavelet coefficients." International journal of computer vision 40, no. 1 (2000): Julesz 78 Yellott 93 12

Physiologically-inspired Julesz-style texture model Springer. All rights reserved. This content is excluded from our Creative Commons license. For more information, see https://ocw.mit.

13 Physiologically-inspired Julesz-style texture model Springer. All rights reserved. This content is excluded from our Creative Commons license. For more information, see Source: Portilla, Javier, and Eero P. Simoncelli. "A parametric texture model based on joint statistics of complex wavelet coefficients." International journal of computer vision 40, no.1 (2000): Harvard Medical School. All rights reserved. This content is excluded from our Creative Commons license. For more information, see [Portilla & Simoncelli, 2000] 13

14 Physiologically-inspired Julesz-style texture model + Joint statistics Springer. All rights reserved. This content is excluded from our Creative Commons license. For more information, see Source: Portilla, Javier, and Eero P. Simoncelli. "A parametric texture model based on joint statistics of complex wavelet coefficients." International journal of computer vision 40, no.1 (2000): [Portilla & Simoncelli, 2000] 14

15 Texture synthesis + Joint statistics Springer. All rights reserved. This content is excluded from our Creative Commons license. For more information, see Source: Portilla, Javier, and Eero P. Simoncelli. "A parametric texture model based on joint statistics of complex wavelet coefficients." International journal of computer vision 40, no.1 (2000): Joint statistics + [Portilla & Simoncelli, 2000] 15

16 Texture synthesis + Joint statistics Joint statistics Springer. All rights reserved. This content is excluded from our Creative Commons license. For more information, see Source: Portilla, Javier, and Eero P. Simoncelli. "A parametric texture model based on joint statistics of complex wavelet coefficients." International journal of computer vision 40, no.1 (2000): [Portilla & Simoncelli, 2000] 16

Images original image Model responses Springer. All rights reserved. This content is excluded from our Creative Commons license. For more information, see https://ocw.mit.edu/ help/faq-fair-use/.

17 Images original image Model responses Springer. All rights reserved. This content is excluded from our Creative Commons license. For more information, see help/faq-fair-use/. Source: Portilla, Javier, and Eero P. Simoncelli. "A parametric texture model based on joint statistics of complex wavelet coefficients." International journal of computer vision 40, no. 1 (2000): noise seed Images with identical model responses synthesized 17

18 Experimental logic Original Photographic Image Ventral Model Model Responses Responses Random Seed Image Statistical Image Generator Retina Optic Nerve LGN Optic Tract Visual Cortex Harvard Medical School. All rights reserved. This content is excluded from our Creative Commons license. For more information, see If model captures the same properties as the visual system, images with identical model responses should appear identical to a human. 18

19 Pairs of images with identical model responses: Springer. All rights reserved. This content is excluded from our Creative Commons license. For more information, see Source: Portilla, Javier, and Eero P. Simoncelli. "A parametric texture model based on joint statistics of complex wavelet coefficients." International journal of computer vision 40, no. 1 (2000): Top: original, Bottom: synthesized [Portilla & Simoncelli 2000] 19

20 outpainting Central square of each image is original texture. Surround is synthesized. Springer. All rights reserved. This content is excluded from our Creative Commons license. For more information, see Source: Portilla, Javier, and Eero P. Simoncelli. "A parametric texture model based on joint statistics of complex wavelet coefficients." International journal of computer vision 40, no. 1 (2000):

For more information, see https://ocw.mit.edu/help/faq-fair-use/. Source: Portilla, Javier, and Eero P.

22 Can we generalize to inhomogeneous stimuli? Can we make the model more physiological? Springer. All rights reserved. This content is excluded from our Creative Commons license. For more information, see Source: Portilla, Javier, and Eero P. Simoncelli. "A parametric texture model based on joint statistics of complex wavelet coefficients." International journal of computer vision 40, no.1 (2000):

23 Retina Optic Nerve LGN Optic Tract Visual Cortex (V1) Harvard Medical School. All rights reserved. This content is excluded from our Creative Commons license. For more information, see [figure: Hubel 95] 23

24 Dorsal pathway: V1->V3->V5 position, motion, action Retina Optic Nerve LGN Optic Tract Visual Cortex Harvard Medical School. All rights reserved. This content is excluded from our Creative Commons license. For more information, see Ventral pathway: V1->V2->V4-> IT spatial form, recognition, memory [Ungerleider & Mishkin, 1982] 24

25 Retina Optic Nerve LGN Optic Tract Visual Cortex Harvard Medical School. All rights reserved. This content is excluded from our Creative Commons license. For more information, see Visual neurons responds to content within a small region of the visual input known as the Receptive Field (RF) In each visual area, we assume RFs cover the entire visual field 25

26 Inhomogeneity - RF sizes grow with eccentricity Retinal ganglion (midget) cell receptive fields (macaque, magnified x10) [Perry et.al., 1984; Watanabe & Rodiek, 1989] loss of resolution A H U D B O J W T R H O P F Q B K Z Y V J G M A C L P Y S F R K Modified Snellen acuity chart (threshold, x10) [after Anstis, 1973] Vision Research. All rights reserved. This content is excluded from our Creative Commons license. For more information, see Source: Anstis, Stuart M. "A chart demonstrating variations in acuity with retinal position." Vision research 14, no. 7 (1974):

28 RF sizes grow with eccentricity Retina Reprinted by permission from Macmillan Publishers Ltd: Nature. Source: Freeman, Jeremy, and Eero P. Simoncelli. "Metamers of the ventral stream." Nature neuroscience 14, no. 9 (2011): [Freeman & Simoncelli 2011, data from Gattass et. al., 1981; Gattass et. al., 1988; Perry et. al., 1984] 28

29 V1 V4 V2 IT Reprinted by permission from Macmillan Publishers Ltd: Nature. Source: Freeman, Jeremy, and Eero P. Simoncelli. "Metamers of the ventral V1 V2 V4 IT stream." Nature neuroscience 14, no. 9 (2011): [Freeman & Simoncelli, 2011] 29

30 V1 simple cell V1 complex cell Figure removed due to copyright restrictions. Please see the video. Source: Hubel, David H., and Torsten N. Wiesel. "Receptive fields, binocular interaction and functional architecture in the cat's visual cortex." The Journal of physiology 160, no. 1 (1962): [Hubel & Wiesel, 1962] linear weights rectifying nonlinearity + 30

31 Local texture representation in the ventral stream V2 receptive fields V1 cells Joint statistics Local correlational statistics can be re-expressed as a subunit model... 31

32 Canonical computation in the ventral stream V2 complex cell V2 receptive fields V1 cells Substantial information loss => model predicts metamers 32

33 Canonical sensory computation Linear filter (determines pattern selectivity) Rectifying nonlinearity Local pooling (e.g., average, max) Local gain control Noise Cascaded... [eg. Douglas, 1989; Heeger, Simoncelli & Movshon 1996; Heeger & Carandini 2014] 33

34 View-tuned cells Composite features Complex cells Simple cells Input image Reprinted by permission from Macmillan Publishers Ltd: Nature. Source: Koch, Christof, and Tomaso Poggio. "Predicting the visual world: Silence is golden." Nature neuroscience 2, no. 1 (1999): [Koch & Poggio, 1999; cf. Fukishima, 1980; Serre, Oliva, Poggio 2007; etc] 34

35 Synthesizing Ventral Stream Metamers Original image Source Unknown. All rights reserved. This content is excluded from our Creative Commons license. For more information, see [Freeman & Simoncelli, 2011] 35

36 Synthesizing Ventral Stream Metamers Original image Model responses Synthesized image Source Unknown. All rights reserved. This content is excluded from our Creative Commons license. For more information, see [Freeman & Simoncelli, 2011] 36

Synthesizing Ventral Stream Metamers Original image Model responses Synthesized image 3.1 1.4 12.5... Source Unknown. All rights reserved.

37 Synthesizing Ventral Stream Metamers Original image Model responses Synthesized image Source Unknown. All rights reserved. This content is excluded from our Creative Commons license. For more information, see [Freeman & Simoncelli, 2011] 37

38 Courtesy of Elsevier, Inc., Used with permission. Source: Cohen, Michael A., Daniel C. Dennett, and Nancy Kanwisher. "What is the bandwidth of perceptual experience?" Trends in Cognitive Sciences 20, no. 5 (2016):

Courtesy of Elsevier, Inc., http://www.sciencedirect.com. Used with permission. Source: Cohen, Michael A., Daniel C.

39 Courtesy of Elsevier, Inc., Used with permission. Source: Cohen, Michael A., Daniel C. Dennett, and Nancy Kanwisher. "What is the bandwidth of perceptual experience?" Trends in Cognitive Sciences 20, no. 5 (2016):

40 1.0 Proportion correct chance Model RF size (diam / eccentricity) Scaling (radius / eccentricity) Reprinted by permission from Macmillan Publishers Ltd: Nature. Source: Freeman, Jeremy, and Eero P. Simoncelli. "Metamers of the ventral stream." Nature neuroscience 14, no. 9 (2011): [Freeman & Simoncelli, 2011] 40

1.0 Proportion correct 0.9 0.8 0.7 0.6 0.5 0.

41 1.0 Proportion correct chance Model RF size (diam / eccentricity) Scaling (radius / eccentricity) [Freeman & Simoncelli, 2011] 41

42 1.0 Proportion correct chance Model RF size (diam / eccentricity) Scaling (radius / eccentricity) [Freeman & Simoncelli, 2011] 42

43 1.0 Proportion correct chance Model RF size (diam / eccentricity) Scaling (radius / eccentricity) [Freeman & Simoncelli, 2011] 43

44 [Freeman & Simoncelli, 2011] 44

45 RF sizes grow with eccentricity Retina Reprinted by permission from Macmillan Publishers Ltd: Nature. Source: Freeman, Jeremy, and Eero P. Simoncelli. "Metamers of the ventral stream." Nature neuroscience 14, no. 9 (2011): [Freeman & Simoncelli 2011, from Gattass et. al., 1981; Gattass et. al., 1988; Perry et. al., 1984] 45

46 Reprinted by permission from Macmillan Publishers Ltd: Nature. Source: Freeman, Jeremy, and Eero P. Simoncelli. "Metamers of the ventral stream." Nature neuroscience 14, no. 9 (2011): Macaque Physiology [Allman & Kaas, 1971; Allman & Kaas, 1974; Gattass et.al., 1981; van Essen et.al., 1984; Maguire & Baizer, 1984; Burkhalter & van Essen, 1986; Gattass et.al., 1987; Desimone & Schein, 1987; Gattass et.al., 1988; Cavanaugh et. al., 2002] [Freeman & Simoncelli, 2011] 46

47 Proportion correct a Extended presentation chance ms 400 ms r 2 = 0.91, 0.89 r 2 = 0.94, 0.91 r 2 = 0.95, 0.85 r 2 = 0.97, b Directed attention Undirected attention Directed attention Proportion correct chance r 2 = 0.91, 0.94 r 2 = 0.94, 0.98 r 2 = 0.95, 0.97 r 2 = 0.97, Reprinted by permission from Macmillan Publishers Ltd: Nature. Source: Freeman, Jeremy, and Eero P. Simoncelli. "Metamers of the ventral stream." Nature neuroscience 14, no. 9 (2011): Scaling (diameter / eccentricity) of receptive fields in synthesis model [Freeman & Simoncelli, 2011] 47

48 Mid-ventral model V1 model 0.3 r 2 = 0.91, 0.94 r 2 = 0.94, Proportion correct chance r 2 = 0.95, 0.85 r 2 = 0.97, Reprinted by permission from Macmillan Publishers Ltd: Nature. Source: Freeman, Jeremy, and Eero P. Simoncelli. "Metamers of the ventral stream." Nature neuroscience 14, no. 9 (2011): Scaling (diameter / eccentricity) of receptive fields in synthesis model [Freeman & Simoncelli, 2011] 48

49 V2 model Main experiment V1 model Extended presentation Directed attention Scaling (diameter / eccentricity) S1 S2 S3 S4 Average V4 V2 V1 Reprinted by permission from Macmillan Publishers Ltd: Nature. Source: Freeman, Jeremy, and Eero P. Simoncelli. "Metamers of the ventral stream." Nature neuroscience 14, no. 9 (2011): Macaque Physiology [Allman & Kaas, 1971; Allman & Kaas, 1974; Gattass et.al., 1981; van Essen et.al., 1984; Maguire & Baizer, 1984; Burkhalter & van Essen, 1986; Gattass et.al., 1987; Desimone & Schein, 1987; Gattass et.al., 1988; Cavanaugh et. al., 2002] [Freeman & Simoncelli, 2011] 49

50 Reading [Freeman & Simoncelli, 2011] 50

51 Camouflage c Nature. All rights reserved. This content is excluded from our Creative Commons license. For more information, see Source: Freeman, Jeremy, and Eero P. Simoncelli. "Metamers of the ventral stream." Nature neuroscience 14, no. 9 (2011): [Freeman & Simoncelli, 2011] 51

52 Can we drive individual V2 neurons using local texture stimuli? Reprinted by permission from Macmillan Publishers Ltd: Nature. Source: Freeman, Jeremy, Corey M. Ziemba, David J. Heeger, Eero P. Simoncelli, and J. Anthony Movshon. "A functional and perceptual signature of the second visual area in primates." Nature neuroscience 16, no. 7 (2013): Top: synthetic textures, full model Bottom: spectral noise (matched only for V1 statistics) [Freeman, et. al. 2013] 52

53 25 V Firing Rate (ips) 0 25 V Time from stimulus onset (ms) Reprinted by permission from Macmillan Publishers Ltd: Nature. Source: Freeman, Jeremy, Corey M. Ziemba, David J. Heeger, Eero P. Simoncelli, and J. Anthony Movshon. "A functional and perceptual signature of the second visual area in primates." Nature neuroscience 16, no. 7 (2013): [Freeman, et. al. 2013] 53

54 15% of V1 neurons significantly positively modulated Proportion of cells V1 63% of V2 neurons significantly positively modulated Proportion of cells V2 Modulation index Reprinted by permission from Macmillan Publishers Ltd: Nature. Source: Freeman, Jeremy, Corey M. Ziemba, David J. Heeger, Eero P. Simoncelli, and J. Anthony Movshon. "A functional and perceptual signature of the second visual area in primates." Nature neuroscience 16, no. 7 (2013): [Freeman, et. al. 2013] 54

55 Firing Rate (ips) Texture category 0 0 Reprinted by permission from Macmillan Publishers Ltd: Nature. Source: Freeman, Jeremy, Corey M. Ziemba, David J. Heeger, Eero P. Simoncelli, and J. Anthony Movshon. "A functional and perceptual signature of the second visual area in primates." Nature neuroscience 16, no. 7 (2013): [Freeman, et. al. 2013] 55

56 V2d V3d V2d V3d V1 V1 V2v Subject 1, Right hemisphere V3v V4 V2v V3v Subject 2, Right hemisphere V Coherence Texture (9 sec) Spectral Noise (9 sec) Time Reprinted by permission from Macmillan Publishers Ltd: Nature. Source: Freeman, Jeremy, Corey M. Ziemba, David J. Heeger, Eero P. Simoncelli, and J. Anthony Movshon. "A functional and perceptual signature of the second visual area in primates." Nature neuroscience 16, no. 7 (2013): [Freeman, et. al. 2013] 56

57 Predicting discriminability Different families Different exemplars (same statistics) Courtesy of Proceedings of the National Academy of Science. Used with permission. Source: Ziemba, Corey M., Jeremy Freeman, J. Anthony Movshon, and Eero P. Simoncelli. "Selectivity and tolerance for visual texture in macaque V2." Proceedings of the National Academy of Sciences 113, no. 22 (2016): E3140-E3149. [Ziemba, Freeman, Movshon, Simoncelli - unpublishd] 57

58 Anesthetized macaque V1: 102 neurons V2: 103 neurons Example V1 neuron Stimuli presented for 100ms within a 4 aperture 20 repetitions each Firing rate (ips) Time from stimulus onset (ms) Courtesy of Proceedings of the National Academy of Science. Used with permission. Source: Ziemba, Corey M., Jeremy Freeman, J. Anthony Movshon, and Eero P. Simoncelli. "Selectivity and tolerance for visual texture in macaque V2." Proceedings of the National Academy of Sciences 113, no. 22 (2016): E3140-E3149. [Ziemba, Freeman, Movshon, Simoncelli - unpublishd] 58

59 Anesthetized macaque V1: 102 neurons V2: 103 neurons Example V2 neuron Stimuli presented for 100ms within a 4 aperture 20 repetitions each Firing rate (ips) Time from stimulus onset (ms) Courtesy of Proceedings of the National Academy of Science. Used with permission. Source: Ziemba, Corey M., Jeremy Freeman, J. Anthony Movshon, and Eero P. Simoncelli. "Selectivity and tolerance for visual texture in macaque V2." Proceedings of the National Academy of Sciences 113, no. 22 (2016): E3140-E3149. [Ziemba, Freeman, Movshon, Simoncelli - unpublishd] 59

60 Variance across families (%) 80 0 V1 n = V2 n = Courtesy of Proceedings of the National Academy of Science. Used with permission. Source: Ziemba, Corey M., Jeremy Freeman, J. Anthony Movshon, and Eero P. Simoncelli. "Selectivity and tolerance for visual texture in macaque V2." Proceedings of the National Academy of Sciences 113, no. 22 (2016): E3140-E3149. Variance across samples (%) Variance across exemplars (%) 60

61 Decoding Family classification 1 n = 1, 3, 10, 30, 100 Exemplar identification V2 performance Chance V1 performance Courtesy of Proceedings of the National Academy of Science. Used with permission. Source: Ziemba, Corey M., Jeremy Freeman, J. Anthony Movshon, and Eero P. Simoncelli. "Selectivity and tolerance for visual texture in macaque V2." Proceedings of the National Academy of Sciences 113, no. 22 (2016): E3140-E

62 Portraits of Javier Portilla, Jeremy Freeman, Josh McDermott, Corey Ziemba and Tony Movshon removed due to copyright restrictions. Please see the video. 62

63 MIT OpenCourseWare Resource: Brains, Minds and Machines Summer Course Tomaso Poggio and Gabriel Kreiman The following may not correspond to a particular course on MIT OpenCourseWare, but has been provided by the author as an individual learning resource. For information about citing these materials or our Terms of Use, visit:

A Primer on Human Vision: Insights and Inspiration for Computer Vision

A Primer on Human Vision: Insights and Inspiration for Computer Vision Guest&Lecture:&Marius&Cătălin&Iordan&& CS&131&8&Computer&Vision:&Foundations&and&Applications& 27&October&2014 detection recognition