What is it about?

While it is crystal clear that communication can draw on many semiotic resources, research in the humanities has hitherto strongly focused on its verbal manifestations. “Multimodality” labels a variety of approaches and theories trying to remedy this bias by investigating how for instance visuals, music, and sound contribute to meaning-making. The contours of what is developing into a new discipline begin to be discernible. This handbook chapter provides a brief survey of various perspectives on multimodality, addresses the thorny issue of what should count as a mode, and makes suggestions for further development of the fledgling discipline.

Featured Image

Why is it important?

Communication is increasingly multimodal. The discipline of multimodality, however, is still in its early stages. One of the biggest problems is that many multimodality scholars have a background in linguistics, and over-extend the analogy between verbal communication and other modes of communication -- e.g., via visuals, music, and/or sounds. Crucially, language has a grammar and a vocabulary, while visuals and sounds do not -- at least not in the literal, strict sense of what "grammar" and "vocabulary" mean (things may be different for music). That said, these modes may have typical "structures" and "components." Each mode communicates in its own, unique way. Multimodality scholars therefore need to develop expertise in at least two different modes, and thus be knowledgeable about minimally two types of monomodal discourse in order to be able to theorize how these can combine to create multimodal meaning. A multimodal medium such as film requires knowledge about at least the modes of language, (moving) images, music, and sound.

Perspectives

A hotly debated issue in the young discipline of multimodality is what counts as a mode. Systemic Functional Linguistic approaches take a much broader view of this than cognitivist-oriented approaches. My own proposal, to be situated within the latter, is to keep "modes" as closely as possible connected to varieties of sensory perception. My list of mode candidates consists of the following: (1) spoken language; (2) written language; (3) visuals; (4) music; (5) sound; (6) taste; (7) smell; (8) touch; (9) bodily behavior. The last mode comprises gestures, postures, facial expressions, and (manner of) movement

Dr Charles Forceville
Universiteit van Amsterdam

Read the Original

This page is a summary of: Multimodality, May 2021, Taylor & Francis, DOI: 10.4324/9781351034708-45.
You can read the full text:

Read

Resources

Contributors

The following have contributed to this page