Can Sounds Alter What We See?

Three decades of psychological research show that our visual and auditory senses work together. Famously, an experiment by Robert Sekuler (1997) found that the presence or absence of a clicking sound determined whether people interpreted two moving dots as crossing paths or bouncing off each other. Similarly, Ladan Shams and colleagues (2000) showed that presenting multiple beeps led participants to perceive multiple visual flashes when only one flash was actually shown. Research therefore shows that sounds can bias how we interpret dynamic visual events.

Can sounds influence how we recognize visual objects?

A new study by Jamal Williams and colleagues at UC San Diego published this month in Psychological Science investigated whether naturalistic sounds (for example, the sound of a cat meowing or a kettle whistling) can influence how ambiguous objects are perceived. To do this, the researchers created silhouette morphs between four pairs of objects: a cat and a kettle, a plane and a raven, a hammer and a seal, and a goat and a Vespa. In a preliminary study, participants rated the morphed silhouettes as appearing more like one category or the other, in order to identify the most ambiguous points along each morph continuum. These ambiguous objects served as the target stimuli in three subsequent experiments.

In the first experiment, the researchers tested whether presenting a sound consistent with one of the morph categories (e.g. a cat's meow) while an ambiguous object (e.g. a morph between a cat and a kettle) was slowly revealed would lead observers to categorize the ambiguous object as more cat-like (compared to presenting a kettle's whistle or an unrelated sound). When asked to recreate the observed ambiguous object using a slider, participants were significantly biased to produce an object more similar to the sound-congruent category (i.e. producing a more cat-like morph) than the sound-incongruent category. These results suggest that people integrate auditory information when performing visual recognition.

A post-perceptual or pre-perceptual effect?

In the next two experiments, the researchers tested two alternative accounts of this effect. First, they considered the possibility that the effect was post-perceptual: that the sounds were affecting observers' responses rather than their perceptions. It could be, for example, that the visual object remained ambiguous, but the sound of a cat's meow made observers more likely to respond that they had seen a cat. To account for this, in experiment 2, sounds were presented after the visual object was revealed (during the reporting phase of the trial). Here, they found no effect of the sound in biasing responses. Therefore, the effect found in experiment 1 was unlikely to reflect a post-perceptual decision effect.

In the third experiment, the researchers examined whether pre-perceptual contexts (i.e. top-down factors) could be driving the effect observed in experiment 1. It could be that the context of hearing a cat's meow prepares participants to expect to see a cat and thus categorize ambiguous stimuli as more cat-like. To test this, the researchers presented the sounds before the onset of the visual stimulus. This way, the sound still provided a context for top-down effects to occur, but not at the same time as the visual object was shown. Interestingly, the researchers found no significant differences in this condition either: the presence of a cat's meow prior to the onset of the visual stimulus did not make it appear more cat-like.

Sounds affect visual processing during a short multisensory integration window

Comparing results across the three experiments suggests that the temporal window where sounds are most likely to affect visual recognition happens during the period in which the visual object is being revealed. Sounds presented earlier or later than that do not have the same effect. Therefore, the findings of Williams and colleagues highlight the importance of multisensory processes in perception, demonstrating that sounds can indeed alter how we recognize visual objects.