Multi-sensory integration using sparse spatio-temporal encoding

Transcription

1 Proceedings of International Joint Conference on Neural Networks, Dallas, Texas, USA, August -9, 0 Multi-sensory integration using sparse spatio-temporal encoding A. Ravishankar Rao, Guillermo Cecchi Abstract The external world consists of objects that stimulate multiple sensory pathways simultaneously, such as auditory, visual and touch. Our brains receive and process these sensory streams to arrive at a coherent internal representation of the world. Though much attention has been paid to these streams individually, their integration is comparatively less well understood. In this paper we propose the principle of sparse spatio-temporal encoding as a foundation to build a framework for multi-sensory integration. We derive the dynamics that govern a network of oscillatory units that achieves phase synchronization, and is capable of binding related attributes of objects. We simulate objects that produce simultaneous visual and auditory input activations. We demonstrate that our system can bind features in both these sensory modalities. We examine the effect of varying a tuning function that governs the ability of the units to synchronize, and show that by broadening this function we reduce the ability of the network to disambiguate mixtures of objects. Thus, our model offers the potential to study brain disorders such as autism, which may arise from a disruption of synchronization. I. INTRODUCTION Objects in the natural world typically excite multiple senses simultaneously. Our sensory apparatus has evolved to explore different facets of external objects, spanning the senses of vision, hearing, touch, olfaction and taste. Our brains learn to integrate the information from these senses in order to create rich, multimodal representations of objects. The individual senses such as vision and hearing, and their early cortical processing pathways have been studied in great detail. However, comparatively less effort has been devoted to studying the integration and interaction between these senses. One of the challenges is that the dimensionality of the problem is increased considerably. Furthermore, there is no single cortical area where such interactions take place, and multiple divergent pathways need to be investigated. Examples of such integration areas include the superior colliculus, which receives both auditory and visual input [], [], the superior temporal sulcus [] and the pre-frontal []. Imaging studies have also identified tri-sensory areas which respond to a combination of tactile, audio and visual inputs []. Obtaining detailed cortical recordings from such integration areas is challenging, as it involves training animals to perform specific behavioral tasks and recording from select neurons []. In contrast, techniques such as optical imaging can obtain activity spread over a larger cortical area, and have been applied to understand the localized processing of information in the visual []. There are many neuroscientific [], theoretical and modeling issues that need to be examined when one considers The authors are at the T.J. Watson IBM Research Center, Yorktown Heights, NY 09, USA. ravirao@us.ibm.com, and gcecchi@us.ibm.com multi-sensory integration. Some of the modeling issues consist of understanding the right spatio-temporal abstractions of neural behavior, and developing a principled approach to explore interactions between the units in the system. We build on an earlier model we developed based on sparse spatiotemporal encoding of sensory inputs [], []. We tested this model on visual objects, showed its capability to bind visual features related to a single object, and demonstrated its ability to separate combinations of objects. We extend this model by including an additional simulated auditory stream as an input, and investigate the interactions between the auditory and visual input streams. Our results show that the same framework of sparse spatio-temporal encoding can be applied successfully to a combination of sensory streams. The value in creating a model for multisensory integration is that it allows us to explore higher level issues in brain function. This becomes relevant when we consider the functioning of both normal brains and those that exhibit certain deficits. For instance, one of the disease models for autism involves the inability to achieve proper temporal binding for features from multiple sensory streams []. The architecture and model we propose in the current paper has the potential to investigate such issues in multi-sensory processing. The remainder of this paper is organized as follows. In Section II we describe the computational foundation of our model. In Section III we present results that demonstrate the capability of our model to encode joint audio-visual stimuli arising from simulated objects, and to separate mixtures of these objects into their constituents. We examine the implications of our findings, and their relationship to existing literature in Section IV. II. METHODS In earlier work, we used the principle of sparse spatiotemporal encoding to derive the dynamics of a network for processing sensory information [], []. We now extend this model, developed for a single sensory modality, to a combination of two sensory modalities. We describe the system with the aid of Figure, which consists of a two layer system with two input streams, the audio and visual inputs. We name these streams visual and audio for the sake of concreteness, to illustrate the key concepts. Our method should be applicable to other combinations of sensory inputs such as tactile and visual for instance, or even to tri-sensory inputs. Let x denote units in the lower layer visual, u denote units in the lower layer auditory, and y denote units in the upper layer association. The visual is connected by a weight matrix W to the association //$.00 0 IEEE 0

2 y Upper Layer, Association W nj x j [ + cos(φ j θ n )] Lower Layer, Auditory V W Lower Layer, Visual Δy n j + j V nj u j [ + cos(ξ j θ n )] αy n γ k y k [ + cos(θ k θ n )] () u x Lower Layer, Auditory u Feed-forward connectivity (A) Lateral connectivity y (B) Feedback connectivity (C) Upper Layer, Association Upper Layer, Association x Lower Layer, Visual Fig.. This figure shows the connections from visual and auditory cortices to a higher level area, termed the association. (A) shows feedforward connections, (B) shows lateral connections and (C) shows feedback connections. The auditory is connected by a weight matrix V to the association. Each unit is considered to be an oscillator with an amplitude, frequency and phase of oscillation. If all the units have a similar nominal frequency, their behavior can be described in terms of phasors of the form x n e iφn for the visual, u n e iξn for the auditory and y n e iθn for the association layer. Here, x n and u n denote the amplitudes of units in the visual and auditory cortices and y n denotes amplitudes of units in the association. Similarly, φ n and ξ n are the phases of the n th unit in the visual and auditory cortices. and θ n refers to phases of in the upper layer association. Equations - describe the instantaneous evolution of the system, starting with a set of initial conditions. Δθ n W nj x j sin(φ j θ n ) j + V nj u j sin(ξ j θ n ) j γ y k sin(θ k θ n ) () k Δφ n W jn y j sin(θ j φ n ) () j Δξ n V jn y j sin(θ j ξ n ) () j In this paper, we assume the initial conditions for the lower layer consist of pixel values of a -D visual image constituting a visual stimulus and a -D audio image consisting of a simultaneously presented auditory stimulus. We choose this representation to aid the interpretation of the system s function. In general, the two lower layers could represent any cortical areas. The initial values for the upper layer, y can be set to zero. The values of the phases are randomized. The update rules in Equations - are applied, upon which the system exhibits transients which then settle down, say after 00 iterations. The synaptic weights W are modified only after this settling period as follows. ΔW ij y i x j [ + cos(φ j θ i )] () A similar update rule is used for V as follows. ΔV ij y i u j [ + cos(ξ j θ i )] () The network configuration consists of dynamical units arranged as follows: (a) Lower layers designated by u and x which receive multisensory audio and visual input respectively. The amplitude output of these units depends only on their inputs, whereas the phase is a function of their natural frequency and feedback interactions with a top layer; (b) A top layer designated by y that receive inputs from the bottom u and x layers via feed-forward connections. Top layer units determine their individual amplitude and phase dynamics by integrating the input amplitudes weighted by a function of relative phase differences; (c) The bottom layer receives feedback input from the top layer, which affects only the phase of the bottom layer s units. This behavior has been described by Equations -. In our simulation, the lower layer visual consists of x units, each of which receives a visual intensity value as input. Similarly, the lower layer auditory consists of x units, each of which receives an auditory intensity value as input. 0

3 Object # Object # Object # Object # Fig.. This figure shows the representation of four objects in the visual. The objects differ in shape as well as gray level. Audio Obj. # Audio Obj. # Audio Obj. # Audio Obj. # Fig.. A representation of objects in the auditory. This figure shows idealized tonotopic maps associated with the visual objects shown in Figure The upper layer y consists of units. There are all-to-all connections between units in the lower layers x and the upper layer y, and similarly between u and y. Furthermore, the units in the upper layer possess all-to-all lateral connections. Finally, there are all-to-all feedback connections from y to x and from y to u. Learning leads to a winner-take-all dynamics upon presentation of one of the learned inputs. We choose an input set consisting of simple visual objects such as a square, triangle, cross, circle and so on, as shown in Figure. These visual objects are also associated with corresponding audio objects, as shown in Figure. The interpretation here is that each object generates a paired visual and auditory input pattern. The auditory objects are idealized representations of tonotopic maps, where different frequencies are represented in an ordered spatial fashion []. When an input is presented, we pair the auditory and visual representations, and present the corresponding stimulus at the Audio object # V u y x W Paired presentation of the visual and auditory representations Upper Layer, Association Visual object # Fig.. When an object is presented as a stimulus, we initialize the lower layers to its visual and auditory representations as shown. We have used the visual and auditory representations of object # shown in Figures and respectively. lower layer, as depicted in Figure. The network operates in two stages consisting of learning and performance. During the learning stage, a randomly selected object is presented as input, and the network activity is allowed to settle. Following this the Hebbian learning rules in equations and are applied. The process is repeated over, 000 trials. The system typically shows a winner-takeall behavior at the upper layer y for each input presented. Furthermore, after the training, a unique winner is associated with each input. Note that the network training is done in an unsupervised fashion. As shown in Figures - of [], when two inputs are combined and presented to the lower layer x, it results in two units, termed the winners, being activated in the upper layer y. These units are identified as the two units with the highest and second-highest amplitude respectively. Furthermore, the phases of the winners in layer y are synchronized with the phases of units in the lower layer x that correspond to the two individual inputs. As explained in [], the interpretation of this behavior is that different units can be simultaneously active while having phases that are maximally apart from each other. We define a measure termed the separation accuracy, which captures the ability of the network to correctly identify mixtures of inputs. Suppose unit i in the upper layer is the winner for an input x, and unit j is the winner for input x. If units i and j in the upper layer are also winners when the input presented is the mixture x + x, then we say the separation is performed correctly, otherwise not. The ratio of the total number of correctly separated cases to the total number of cases investigated is the separation accuracy. A related measure concerns the ability of the network to perform segmentation. The accuracy of phase segmentation is measured by computing the fraction of the units of the lower layer that correspond to a given object and are within some tolerance level of the phase of the upper layer unit that represents the same object. III. RESULTS Figures and show the behavior of the system when objects and are presented simultaneously. In Figure we examine the superposition of the visual aspects of these two objects, whereas in Figure we examine the superposition of the auditory cues associated with the same two objects. In Figure, the third row shows that the two winners in the upper layer are approximately degrees out of phase with each other. The phasors representing the winners have been color coded in blue and red so that they can be compared against the phasors in the lower layer. We can readily observe that the lower layer phasors in blue correspond to visual object #, and are synchronized with the upper layer winner, also represented in blue. Similarly, the red phasors show that there is a close phase similarity between the components of visual object # and the winner in the upper layer that represents this object. Furthermore, Figure shows that this congruence also extends to the auditory representations of the same two objects. Thus, there is a phase synchronization 0

4 between the units in the lower layer auditory and visual maps corresponding to a given object and also the upper layer winner that represents the composite audio-visual object. Similarly, Figures and show the ability of the network to separate a mixture of objects and, involving both the audio and visual representations of these objects. A. Varying the tuning function for integration of phase information From Equation we note that the amplitudes of the upper layer, y n are a function of the phase differences between the input unit j in the lower layer, and the target unit n in the upper layer as follows. Δy n [ + cos(φ j θ n )] () This is plotted in Figure 9(A). We now vary the tuning function shown by making it both broader (Figure 9(B)) and narrower (Figure 9(C)) than the original, and examining its effect on the network s function. We use the concepts of separation accuracy and segmentation accuracy to quantify the desired network function. The tuning function can be characterized by a measure such as the full-width at half maximum (FWHM). We vary the FWHM of the tuning function and measure its effect on network performance. The resulting relationship is shown in Figures 0 and. The FWHM of the original tuning curve in equation, as plotted in Figure 9(A), is.. When the tuning width is increased from. to., we observe from Figure 0 that both the separation and segmentation accuracy decline, indicating poorer network performance. A similar decline is observed in Figure as the tuning width is decreased from.to.. B. Varying the number of iterations for settling It would appear that as the tuning function is made narrow, it would allow for a finer discrimination capacity between objects, as a unit in the network would be responsive only to other units in the network with very similar phases. Since we did not see this effect directly in Figure, we must examine other variables that affect the system dynamics. One such variable is the number of iterations used for settling. Recall that the learning rules in equations and are applied only after the network settles. We offer the intuition that a narrow tuning function should also be combined with a longer settling time in order to improve the network performance. Figures and show that as the number of iterations representing settling time is increased, it is accompanied by an increase in separation and segmentation accuracies. This demonstrates that there is a tradeoff between the width of the tuning function and the separation and segmentation accuracies. IV. DISCUSSION In Figures 0 and we examine the effect of varying the tuning function on the separation and segmentation accuracy. For the selected settling time, which is 00 iterations, the best performance is achieved with the original tuning function shown in Equation. Visual obj. # visual obj. # Phase of first winner with first winner Superposed objects Phase of second winner with second winner Fig.. Illustrating the behavior of phase information in the visual stream. This shows the superposition of objects and, and their corresponding visual maps. The grayscale image of the superposed objects is normalized before being displayed. The phase of the first winner corresponding to object in the upper layer is 0. radians. The phase of the second winner in the upper layer corresponding to object is.. The activity in the lower layer units of the visual map is displayed as a vector field. The magnitude of the vector reflects the amount of activity in the unit, and the direction encodes the phase of the unit. 0

5 Audio obj. # Audio obj. # Visual obj. # Visual obj. # Superposed objects Superposed objects Phase of first winner Phase of second winner Phase of first winner Phase of second winner with first winner with second winner with first winner with second winner Fig.. Illustrating the behavior of phase information in the auditory stream. This shows the superposition of objects and, and the corresponding auditory maps. The phase of the first winner corresponding to object in the upper layer is 0. radians. The phase of the second winner in the upper layer corresponding to object is.. The phases of the lower layer in the auditory maps are shown in the bottom row. Fig.. This shows the superposition of objects and, and the corresponding visual maps. The phase of the first winner corresponding to object in the upper layer is. radians. The phase of the second winner in the upper layer corresponding to object is 0. radians. The lower layer units depicted are from the visual map. 0

6 Tuning function Amplitude Phase difference in radians (A) Original tuning function, defined by y =(+cos(x)) Audio obj. # Audio obj. # Superposed objects Tuning function Amplitude Phase difference in radians (B) Broader tuning function, defined by y =tanh(0( + cos(x))) Tuning function Amplitude Phase difference in radians (C) Narrower tuning function. The curve is defined by y =(+cos(x)) / Phase of first winner with first winner Phase of second winner with second winner Fig.. This shows the superposition of objects and, and the corresponding auditory maps. The phase of the first winner corresponding to object in the upper layer is. radians. The phase of the second winner in the upper layer corresponding to object is 0. radians. The lower layer units depicted are from the auditory map. Fig. 9. Different tuning functions investigated. The tuning function affects the integration of phase information from multiple inputs, and hence influences the performance of the overall network. When the tuning function is made broader, the performance of the network declines. This behavior is in agreement with findings in autistic subjects, as reported by Foss-Feig et al [9]. In this study, it was shown that autistic subjects integrate multi-sensory cues over a longer binding window. This has been implicated as one of the mechanisms that may explain the behavior of autistic subjects. The tuning function we have used in our model can be considered to represent a temporal binding window. Thus, our model directly demonstrates that a wider tuning function, or temporal binding window adversely affects the ability of the oscillatory network to correctly identify combinations of audio-visual sensory inputs. Further effort is required to tailor our model to the specific experimental protocol reported in Foss-Feig et al [9], and this is the subject of future research. We also show that narrowing the tuning function has a similar effect in reducing the network performance (Figure ). Nakamura [0] presents an overview of techniques for the processing of synchronously delivered multimodal signals such as an audio-visual input stream. According to his terminology, the method presented in our paper would be considered a form of an early integration model, where the input signals are directly transmitted to a bi-modal classifier 09

7 Separation accuracy Full width half maximum of tuning curve (A) Variation in separation accuracy as tuning width is increased Segmentation accuracy Full width half maximum of tuning curve (B) Variation in segmentation accuracy as tuning width is increased Separation accuracy Number of iterations (settling time) (A) Variation in separation accuracy as the number of settling iterations is increased Segmentation accuracy Number of iterations (settling time) (B) Variation in segmentation accuracy as the number of settling iterations is increased Fig. 0. Results showing variation of separation accuracy with the tuning function as the full-width half maximum measure of tuning (FWHM) is increased from the value of. corresponding to Equation. Fig.. Results showing variation of separation accuracy with the number of settling iterations. The FWHM of the tuning function here is. Separation accuracy Full width half maximum of tuning curve (A) Variation in separation accuracy as tuning width is decreased from. to. Segmentation accuracy Full width half maximum of tuning curve (B) Variation in segmentation accuracy as tuning width is decreased from. to. Separation accuracy Number of iterations (settling time) (A) Variation in separation accuracy as the number of settling iterations is increased Segmentation accuracy Number of iterations (settling time) (B) Variation in segmentation accuracy as the number of settling iterations is increased Fig.. Results showing variation of separation accuracy with the tuning function as the full-width half maximum measure of tuning (FWHM) is decreased from the value of. corresponding to Equation. Fig.. Results showing variation of separation accuracy with the number of settling iterations. The FWHM of the tuning function here is. 0

8 (which is the upper layer in our network). However, one difference in our model is that we utilize feedback connections to modify the phase of the lower layer that receives the input. Such feedback is not present in the models described by Nakamura [0]. We briefly review studies in the field of neuroscience that identify brain regions where multi-modal, or cross-modal information is integrated. Fuster et al. [] showed that cells in the prefrontal in monkeys are capable of associating visual and auditory stimuli over time. De Gelder and Bertelson [] point out that different types of relationships can exist during multisensory integration. The pairs of individual stimuli that are used to evoke a multisensory response could be arbitrary or naturalistic. An arbitrary pairing is created specifically for an experiment, and could consist of a high frequency tone paired with a rectangle or a low frequency tone paired with a square. In the present paper, we have chosen to go this route, so that we can first establish the viability of our model using synchronizing network elements. Future research will consist of using audio-visual stimuli occurring in natural environments. Iarocci and McDonald [] review the relationship between sensory integration and autism. Brock et al. [] suggest that brain development in autism is impaired by a lack of integration amongst brain areas that need to interact with each other to solve behavioral tasks. They propose that this impairment takes the form of a deficit in temporal binding. Though they did not propose a formal computational model for how this might happen, our earlier work [], [] provided a computational framework for achieving temporal binding. The research presented in the current paper examines the effect of changing one of the synchronization enablers, namely the tuning function, on the ability of the network to bind audio-visual features. Other ways of affecting synchronization include the reduction of functional connectivity within brain networks []. Chou et al. [] present a self-organizing map based approach to integrate audio and visual inputs. They mainly explore spatial organization issues and not the temporal interactions as we have presented. There is increasing interest in investigating the neural correlates of multi-sensory perception using techniques such as event-related potentials [] and fmri []. Our understanding of multi-sensory integration is still at an early stage. Many effects, such as that of the interaction of feedback pathways, and the role of direct connections between primary sensory areas are still being investigated. Further research is required to build computational models that capture both the spatial organization within the multisensory cortical areas and the temporal interactions involved in binding features across sensing modalities. These computational models will need to be verified and validated against experimental findings in neuroscience. V. CONCLUSION In this paper we presented a computational model for multi-sensory integration of two input streams consisting of auditory and visual information. The dynamics of this model are derived from the principles of sparse spatio-temporal encoding. The model is capable of grouping, or binding related object features in the two sensory streams through phase synchrony. Our model can also identify the components of audio-visual objects that have been combined or mixed. We investigate the performance of our model by varying the tuning function that governs phase synchronization, and show that broader tuning functions disrupt the ability of the model units to effectively integrate multi-modal inputs. This behavior has the potential to serve as a foundation to explore deficits in brain function such as autism. Acknowledgement We appreciate helpful comments from the reviewers. REFERENCES [] B. E. Stein and T. R. Stanford, Multisensory integration: current issues from the perspective of the single neuron, Nature Reviews Neuroscience, vol. 9, no., pp., 00. [] M. Casey, A. Pavlou, and A. Timotheoue, Audio-visual localization with hierarchical topographic maps: Modeling the superior colliculus, Neurocomputing, 0. [] J. M. Fuster, M. Bodner, J. K. Kroger et al., Cross-modal and crosstemporal association in neurons of frontal, Nature, vol. 0, no., pp. 0, 000. [] Y. Xiao, R. Rao, G. Cecchi, and E. Kaplan, Improved mapping of information distribution across the cortical surface with the support vector machine, Neural Networks, vol., no., pp., 00. [] A. R. Rao, G. A. Cecchi, C. C. Peck, and J. R. Kozloski, Unsupervised segmentation with dynamical units, IEEE Trans. Neural Networks, Jan 00. [] A. Rao and G. Cecchi, An objective function utilizing complex sparsity for efficient segmentation in multi-layer oscillatory networks, International Journal of Intelligent Computing and Cybernetics, vol., no., pp. 0, 00. [] G. Iarocci and J. McDonald, Sensory integration and the perceptual experience of persons with autism, Journal of autism and developmental disorders, vol., no., pp. 90, 00. [] E. Formisano, D. Kim, F. Di Salle, P. van de Moortele, K. Ugurbil, and R. Goebel, Mirror-symmetric tonotopic maps in human primary auditory, Neuron, vol. 0, no., pp. 9 9, 00. [9] J. H. Foss-Feig, L. D. Kwakye, C. J. Cascio, C. P. Burnette, H. Kadivar, W. L. Stone, and M. T. Wallace, An extended multisensory temporal binding window in autism spectrum disorders, Experimental Brain Research, vol. 0, no., pp. 9, 00. [0] S. Nakamura, Statistical multimodal integration for audio-visual speech processing, Neural Networks, IEEE Transactions on, vol., no., pp., 00. [] B. De Gelder and P. Bertelson, Multisensory integration, perception and ecological validity, Trends in cognitive sciences, vol., no. 0, pp. 0, 00. [] J. Brock, C. C. Brown, J. Boucher, G. Rippon et al., The temporal binding deficit hypothesis of autism, Development and psychopathology, vol., no., pp. 09, 00. [] P. J. Uhlhaas and W. Singer, Neural synchrony in brain disorders: relevance for cognitive dysfunctions and pathophysiology, Neuron, vol., no., pp., 00. [] S. M. Chou, A. P. Paplinski, and L. Gustafsson, Speaker-dependent bimodal integration of chinese phonemes and letters using multimodal self-organizing networks, in Neural Networks, 00. IJCNN 00. International Joint Conference on. IEEE, 00, pp.. [] P. Jing, T. Yin, and Y. Bo, Phase synchrony measurement of erp based on complex wavelet during visual-audio multisensory integration, in Industrial Control and Electronics Engineering (ICICEE), 0 International Conference on. IEEE, 0, pp. 0. [] K. O. Bushara, T. Hanakawa, I. Immisch, K. Toma, K. Kansaku, and M. Hallett, Neural correlates of cross-modal binding, Nature neuroscience, vol., no., pp. 90 9, 00.