What is multimodality

1
What is multimodality?

• Bella Dicks
• School of Social Sciences
• Cardiff University

2
science centre exhibition hall
all the modes employed contribute to meaning
3
modes and media
Kress, G.and Van Leeuwen, T. (2001). Multi-modal
Discourse. London Arnold.

• Modes - abstract, non-material resources of
  meaning-making (e.g. the languages of image,
  sound, gesture, colour, texture, narrative)
• Media - specific material forms in which modes
  are realised (e.g. a photograph, a computer
  screen, an audio recording, paint, a wall,
  clothing, a chisel, a poem, a block of wood)

4
multimodality

• people communicate through
• speech, accent, gesture, facial expression,
  gaze-direction, shape, weight, size, body
  posture, position movement, dress (colour,
  shape, texture, etc.), hairstyle, etc.
• environments/objects communicate through
• colour, texture, shape, position,
  opaqueness/light, weight, movement, sound, etc.

5
multimodality in the research process

• observations/elicitations of the field (data)
• recording of data
• analysis of data records
• representation/dissemination of research findings
• N.B. are data recorded, represented, reproduced,
  reduced, transformed?

6
1. Multimedia in the field

• Material environments, e.g.
• Relative position of objects
• Walls and corridors
• Colours, shapes and sizes
• Social interaction, e.g.
• Relations of proximity
• Sequencing of events
• Turn-taking in conversation

7
modalities of the field setting
Exhibits Hall
Science Theatre
8
framing a multimodal principle

• framing works across diverse media e.g. in
  interior design in sound compositions, in
  paintings.
• introduces breaks and discontinuities contrasts
  and continuities uses framelines and empty space

9
framing

• In writing e.g. punctuations, prepositions and
  paragraphs
• In images e.g. lines and borders
• In sound e.g. pacing, intonation and rhythm
• In material environments e.g. object
  positioning, corridors and walls
• In movement e.g. jumps, gestures and flows
• In web-pages e.g. pop-up boxes, graphic layout
  and hyperlinks

10

• Framing parcels up the conceptual world () It
  marks off, but in doing so it establishes, at the
  same time, the elements which may be joined. ()
  Without framing, no meaning, we might say.
• Gunther Kress (2003) Literacy in the new media
  age, p. 123/4/5.

11
audio recordings framing whats said

• Pauses
• Hesitations
• Ordering of items
• Soundscapes

12
2. recording multimodality

• What are the affordances of different recording
  media?
• photographs v. writing
• photographs v. video
• writing v. sound

13
multimodal data-records

• still images deploy
• colour shape light directionality
• perspective position borders size
• voice recordings deploy
• accent expression tone pitch
• language volume cadence rhythm

14
framing childrens movement
   
15
   
16
     
17
recording media

• E.g. cameras embody their own framing devices
• edge v. centre of photograph
• composition of frame
• angle and position of camera
• choice of what to record
• decision about when to press shutter
• Etc.etc.

18
data records

• photographs
• record spatial framing e.g. boundaries transits
• audio recordings
• record temporal framing e.g. sequencing of items
  pauses, etc.
• video recordings
• record spatial and temporal framings
• fieldnotes
• record spatial and temporal framings

19
fieldnotes

• When you enter the Science Centre, you see a
  huge, well-lit hall in front of you with high
  ceilings and a gallery above. It is a white space
  with big white pillars and vast windows, but
  there's also lots of colour, noise and movement.
• On entering, immediately ahead of you is a
  plastic ball, suspended seemingly by magic in
  front of a bright yellow solid pyramid. Beyond
  are more yellows and reds and greens and blues.
• All around are a variety of different
  brightly-coloured devices or machines, housed in
  brightly coloured casings and consoles. Many of
  them move, produce sounds, create visual effects,
  and so forth when activated by a user
• There are lots of children moving around
  excitedly, flitting from exhibit to exhibit,
  sometimes bumping into each other.