Multisensory integration

Our senses can guide one another in very subtle ways. E.g. the image below.


Audition and Vision

Audition informs vision (when we see two balls cross paths, we'll either see them bounce or cross depending on sound).

Vision informs audition (e.g. the McGurk effect).

Taste Perception and Multisensory Integration


When subjects are asked to identify tastes with and without their nostrils open, the results differ dramatically - Mozell et al. (1969).


The responses from taste and smell are first combined in the orbital frontal cortex (OFC), and many neurons in this area respond to not only taste and smell, but also taste and vision. They're referred to as polysensory areas.



Vision can influence our taste, as DuBose et al. found in 1980. For instance Cherry-flavoured carbonated drinks are perceived as ‘orange-flavoured’ when coloured orange, or wine experts are fooled by the colour of the wine.

Measuring Perceptual Phenomenon

Solving the Criterion Problem

Some participants will be conservative and cautious when asked if they can detect a difference, and will only answer “yes” if they are sure that they see it. Conversely other participants will be liberal and bold, and will answer “yes” if they think they have even the barest hint of a difference.

We avoid this problem by using a 2 Alternative Forced Choice paradigm; give people two stimuli, and ask which has the difference. We repeat this many times, and if they cannot see it they will be correct 50% by chance.


Measuring “impaired” Sensory Perception

Hearing loss is a natural consequence of ageing, but it can also be due to damage or disease.



Qualia is a term used in philosophy to refer to individual instances of subjective, conscious experience.

It's the way it feels to experience pain, or see red, or smell a rose; how do I know my experience of red is like yours?

Four Properties of Qualia

  • Ineffable: They cannot be communicated other than be direct experience
  • Intrinsic: Non-relational, do no change depending on the experiences' relationship to other things
  • Private: Interpersonal comparisons of qualia are impossible
  • Apprehensible in Consciousness: To experience a quale is to know one is experiencing it, and to know all there is to know of that quale.

We have no device or machine to measure qualia of any sort, and perception studies largely side-step the issue. Psychophysics can tell us what can be detected and what can be
discriminated, but nothing about the experience directly.

Bottom Up v Top Down Processing

Bottom up: Information flows from lower to higher levels of analysis, integrating simple sensory attributes into large structures.

Top down: Processing starts at higher levels to lower levels of analysis, using prior knowledge and experience to steer lower processes.

So far our discussion of anatomy implies a bottom-up approach, but only because we haven't focussed on the top-down details.

The majority of nerve fibres that project to the thalamus do not come from
the sensory periphery, but are “decending” fibres from cortex; Retinal input accounts for 10% of synapses onto lateral geniculate neurons of the thalamus, and cortical visual areas account for 30% of synapses.

One contrast of bottom up vs top down is a policeman collecting facts, vs Sherlock Holmes thinking in plots.
Or seeing a blurred image and not understanding it until we see the unblurred one, and hence only being able to see the actual one.

Sensory Adaptation

Because the world is vastly varied and we encounter huge ranges of things (0.001 to 130,000 lux, for instance) we rarely encounter our whole possible range at the same time. Because of this we use the same neurons to encode different intensity ranges - but we have to allow them to adapt the "ambient" local environment level of that stimulus.

This also explains how cells in the eyes adapt to certain colours, enabling us to overcompensate for black-white when we've stared at something for a while.