Temporal Recalibration: Asynchronous audiovisual speech exposure extends the temporal window of multisensory integration

Transcription

1 Temporal Recalibration: Asynchronous audiovisual speech exposure extends the temporal window of multisensory integration Argiro Vatakis Cognitive Systems Research Institute, Athens, Greece

2 Multisensory integration in time The brain seems to be able to compensate for small asynchronies in order to maintain the different modalities aligned in time.

3 Spence & Squire (2003)

4 Point of Subjective Simultaneity (PSS) Proportion of Vision First' Responses PSS Sound first PSS (ms) Condition 1 Condition 2 Vision first Sound First SOA (ms) Vision First v PSS: average time one stimulus has to lead another for simultaneity to be perceived v Sensitive to differences in the neural processing latencies between the auditory and visual stimuli

5 Just Noticeable Difference (JND) 0.99 Proportion of Vision First' Responses JND (ms) Poor performance Good performance Condition 1 Condition 2 Sound First SOA (ms) Vision First v JND: smallest temporal interval between two stimuli for correct judgment of stimulus order on 75% of trials v Sensitive to changes in the online temporal resolution

6 Fujisaki et al. (2004) Vroomen et al. (2004)

7 But we are rarely exposed to a single event Temporal window = PSS ± JND

8 Multisensory integration in time: Adaptation to asynchrony Can we produce a temporal realignment between the senses through exposure to asynchronous stimuli? Navarra et al. (2005) & Vatakis et al. (2008)

9 Temporal Recalibration Experiments

10 Multisensory integration in time: Adaptation to asynchrony Count music pattern breaks Temporal order judgments (TOJ) on light flash noise burst pairs (-500/+500 ms) Monitoring audiovisual speech Monitoring audiovisual music Count male names (i.e., Peter)... Table, bird, PETER

11 Synchronous condition VIDEO Asynchronous condition SOUND VIDEO 300 ms SOUND

12 Just Noticeable Difference (ms) Synch Asynch (300 ms) Just Noticeable Difference (ms) Synch Asynch (300 ms) Exposure to asynchronous audiovisual speech widens the window of simultaneity. This effect spans across different stimulus domains (speech => flash/noise) The effects are replicated with desynchronized musical stimuli (a hand playing on a piano).

13 Just Noticeable Difference (ms) Synch Asynch (1000 ms) But it breaks down if the asynchrony is too large, thus ruling out a general distraction/ attentional effect

14

15 Multisensory integration in time: Summary v Onsets in one modality can attract perceived onset time in another modality. v This realignment persists in time (adaptation) and across stimulus types (speech/music -> beeps/flashes). v It could be argued that the perceptual system adapted to the most important (?)/temporally coherent (?; i.e., speech) stimulus.

16 Can we produce a temporal realignment between the senses through exposure to asynchronous speech stimuli? Vatakis et al. (2007)

17 Target audiovisual speech... \ABA\, \AGA\ Monitoring audiovisual speech... Table, bird, PETER Count male names (i.e., Peter)

18 Temporal Recalibration: Audiovisual speech Target audiovisual speech... \ABA\, \AGA\ Superimposed audiovisual speech... Table, bird, PETER Monitoring audiovisual speech... Table, bird, PETER Projected... \ABA\, \AGA\

19 Exposure to asynchronous audiovisual speech does not alter the window of simultaneity in terms of JND. Exposure to asynchronous audiovisual speech widens the window of simultaneity in terms of PSS. This effect spans across different complexity stimulus domains (brief speech -> continuous speech)

20 v the senses realign in time up to small asynchronies Temporal Recalibration v realignment creates adaptation across stimulus types v is it the complexity of the stimulus that leads to this adaptation? v proof of the existence of a wide, flexible temporal window Visual stimulus Auditory stimulus Meredith et al. Time (1985) Wide window for multisensory integration

21 Thank You