Using hearing and vision for motion prediction, motion perception, and localization.

What is it about?

When crossing a busy street, with cars honking and engines rumbling, we must continuously keep track of what we see and hear. At the same time, our vision and hearing are often interrupted; our view of an incoming car might be obstructed by a bus, and the sound of construction works might obfuscate the sound of the engine. This paper examines how humans use multisensory information to localize moving objects, and how does this depend on expected sensory interference (e.g., occlusion).

Featured Image

Photo by Mimi Thian on Unsplash

Photo by Mimi Thian on Unsplash

Why is it important?

Surprisingly, we found that observers never combine vision and hearing to track occluded moving objects, always relying almost exclusively on a single sense. Instead, observers flexibly switch between senses in anticipation of sensory interruptions; using hearing or vision depending on (a) which sense will be interrupted and on (b) which sense conveys the most reliable information. Our study provides insights into how our senses work together to help us interact with objects in the world.