Image-Language Interaction in Multimodal Literacy Pedagogy

1
Image-Language Interaction in Multimodal Literacy
Pedagogy

• Len Unsworth
• Professor in English and Literacies Education
• School of Education
• University of New England, Armidale, 2351
  Australia.
• Phone 61 (0)2 6773 2677
• FAX 61 (0)2 6773 2445
• Email len.unsworth_at_une.edu.au
• Homepagehttp//fehps.une.edu.au/Education/staff/U
  nsworth/

2
(No Transcript)
3
Zervos, K. (2001). Childhood in Richmond.
Retrieved 11th August, 2004, from
http//wordcircuits.com/gallery/childhood/index.ht
ml
4
Metamorphosis
5
Lemke, J. (2006). Towards critical multimedia
literacy Technology, research and politics. In
M. McKenna, L. Labbo, R. Kieffer D. Reinking
(Eds.), International Handbook of Literacy and
Technology (Vol. II, pp. 3-14). Mahwah, N.J.
Lawrence Erlbaum.

• We need to understand
• How linguistic and visual-graphic modes of
  meaning-making are integrated.
• How dynamic, cumulative meaning-making in real
  time differs from making meaning with static
  images or text.

6
(No Transcript)
7
One of Australias greatest natural resources
Like no other.
Welcome to the new Crown Plaza
8

• One of Australias greatest natural resources

• Like no other.

• Welcome to the new Crown Plaza

9
The Consensus - image/text relations and literacy
In order to understand the role of print in the
digital age, it is essential to have a solid
grasp of the growing integrative relationship of
print and graphics" (p.22) Dresang, E. (1999).
Radical Change Books for youth in a digital age.
New York Wilson.
The changing nature of society underlines the
imperative for increased metatextual awareness
and visual literacy, including an understanding
of pictureword combinations. Russell, G. (2000).
Print-based and visual discourses in schools
implications for pedagogy. Discourse studies in
the cultural politics of education, 21(2),
205-217.
... no text is an image. No text has the exact
set of meaning-affordances as any image. No
image or visual representation means in all and
only the same ways that some text can mean. It
is this essential incommensurablility that
enables genuine new meanings to be made from the
combinations of modalities. Lemke, J. (2002).
Travels in hypermodality. Visual Communication,
1(3), 299-325.
10
...many contemporary texts make use of image and
of writing at the same time, using both to carry
meaning in specific ways. In that context, a
theory of reading which relates to the graphic
material of 'letters' alone is no longer able to
explain how we derive meaning from texts. Kress,
G. (2003). Literacy in the New Media Age. London
Routledge.
Serial cognitive processing of linear print
text no longer adequately characterizes
contemporary reading and writing, which now
involve parallel processing of multimodal
text-image information sources. Luke, C. (2003).
Pedagogy, connectivity, multimodality and
interdisciplinarity. Reading Research Quarterly,
38(10), 356-385.
it is the visual/verbal interface that is at
the heart of literacy learning and development
for both computer-users and those without access
to computers (p.63) Andrews, R. (2004). Where
next in research on ICT and literacies. Literacy
Learning The Middle Years, 12(1), 58-67.
11
An agenda for discussion

• Meaning at the intersection of image and language
  in literary and informational texts

• Reading image/language interaction in curriculum
  area texts

• The role of images in group reading comprehension
  tests for primary schools - Implications for the
  new national literacy tests in Australia in 2008

• Image/language interaction in student multimedia
  text composition

• Multimodal literacy in government English
  syllabus and curriculum documents.

• Metalanguage and image/text relations in
  multimodal literacy pedagogy

• Implications for developing multimodal literacy
  pedagogy

• Conclusion

12
The construction of meaning at the intersection
of image and language in literary and
informational texts.
13
Zoo - Anthony Browne
Browne, A. (1994). Zoo. London Random House.
14
In respect to interpersonal meaning,
verbiage/image relations are more concerned with
evaluation than interaction . Martin argues that
a key function of images is to co-articulate
attitude (including Affect, Judgment and
Appreciation). In doing so, images operate in a
similar way to imagery, provoking an evaluative
reaction in readers, and the images are typically
positioned to do this so that they foreshadow the
value positions to be constructed in the
subsequent verbiage. One example is taken from
Nelson Mandelas The Illustrated Long Walk to
Freedom . In the section dealing with the 1976
Soweto uprising, the well-known photo of the body
of thirteen-year-old Hector Pieterson being
carried from the fray, is positioned as a full
page image on page 147 preceded by its caption in
the right hand margin of the previous page. The
main text dealing with the Soweto uprising then
appears overleaf on page 148. The photo previews
and amplifies the reaction induced by Mandelas
verbal imagery. In SFL terms Martin suggests
that the photo functions as an evaluative
interpersonal Theme, naturalizing the stance from
which the remaining verbiage can be read.
Martin, J. R. (2002). Fair trade Negotiating
meaning in multimodal texts. In P. Coppock (Ed.),
The semiotics of writing Transdisciplinary
perspectives on the technology of writing. (pp.
311-338). Begijnhof, Belgium Brepols Indiana
University Press.
15
Bringing them home National inquiry into the
separation of aboriginal and torres strait
islander children from their families. Sydney
Human Rights and Equal Opportunities Commission
(1997).
16
(No Transcript)
17
Reading image/language interaction in curriculum
area texts
18
(No Transcript)
19
Divergence (misrepresentation)
The text and images are set out on the double
page spread in a portrait orientation to simulate
the levels of the various zones occurring at
successive depths in the ocean.
Pictorially these levels are allocated equal
proportions of the layout space and hence are not
to scale, since the text states that the sunlit
level is 150 metres deep the twilight zone
starts at 150 metres below sea level and finishes
around 1000 metres below the surface the
midnight zone starts at the end of the twilight
zone and continues to the sea bed, possibly being
as deep as 4000 metres and in some trenches more
than 6000 metres.
This non-scale depiction of the sea levels used
in the layout is also repeated in the diagrams
showing the three main ocean depth zones
beginning each descriptive paragraph.
20
The image and its correspondingly numbered
caption text seem to emphasise different causes
of the death of aquatic animals
21
Image text relations - interpreting ideational
meanings
These arrows are not explained in the text. What
meanings are to be inferred?
These arrows are also not explained. What do
they mean?
These arrows pointing from the gases to the Earth
are not explained in the text
Although not stated directly, we infer that this
band represents gases such as carbon dioxide
22
Compositional Interaction Weak Framing
23
The role of images in group reading comprehension
tests for primary schools - NSW Basic Skills
TestImplications for the new national literacy
tests in Australia in 2008
24
NSW BST
25
Classifying items targeting images

• Question descriptions as indicated in the
  explanatory material for teachers which
  accompanied the test results, were used as a
  starting point for identifying items that
  targeted images.

26
Each item was then further described in terms of

• 1. the explicitness of the item in targeting
  either text, image, or both, by explicit
  reference to the text or image in the wording of
  the item and,
• 2. text / image dependence, that is, whether the
  correct answer could be derived from the text
  alone (text-specific T), from the image alone
  (image-specific I), or from a combination of
  meanings from text or/and image (text-image TI)

27
Description of BST test items in terms of their
explicitness and text/image dependence.
28
Distribution of items targeting images
29
Top 30 of items in difficulty level
Items involving images
30
Image/Text Items and Difficulty Level
In seeking to explain the relative difficulty of
these image-targeted items it is useful to
consider the nature of the image/text relations
in the three texts Mapping Islands Tobwabba Art
Gallery and Telling the Time Using Water.
31
Complementarity Image and text contribute
different meanings
32
Types of image/text relations

• The discussion to follow focuses on the analysis
  of image-text relations.
• Two types of intermodal relationships were
  examined
• 1. Concurrence, that is, where there is
  ideational equivalence between image and text in
  participant-process-phenomenon configuration, in
  a relationship where one mode elaborates on the
  meaning of the other by further specifying or
  describing it while no new element is introduced
  by the text or image.
• 2. Complementarity, where the meanings in the
  image/s and text are different but complementary
  that is, meaning is conveyed through different
  modes.

33
Relating types of image/text relations to item
difficulty
34
(No Transcript)
35
(No Transcript)
36
Image/language interaction in student multimedia
text composition
37
(No Transcript)
38
Taken - Machinima by Grade 6 Student, Nathan
Thomas, A. (in press). Machinima Composing 3D
Multimedia Narratives. In Unsworth, L. (Ed). New
Literacies and the English Curriculum Multimodal
Perspectives. London Continuum
Nathan is in a grade 6 class in Victoria,
Australia, and his class has been using the 3D
animation/gaming software Kahootz to construct
multimedia narratives. Kahootz is a program that
has been mandated by the Victorian government for
use in all public schools in the state, and has
been used by many teachers now since 2004. One
of the features of the Kahootz software
(developed by the Australian Childrens
Television Foundation) is that the 3D animations
can be converted into machinima, and this feature
has proven most popular. Nathan created his
machinima, Taken, in class, and it was selected
by staff of the ACTF as one of a number of
examples of exemplary work.
39
Taken
40
The machinima consists of a number of animated
scenes with the text of the poem scrolling over
it, as Nathan reads the poem in a slow and sombre
tone.
41
Inter-modal meaning in Taken
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49
Zahras Blog
http//naturallysweet.blogspot.com/2004_10_01_natu
rallysweet_archive.html
Humphrey, S. (in press) Getting the reader on
side - Exploring adolescent online political
discourse.
50
(No Transcript)
51
Multimodal literacy in government English
syllabus and curriculum documents.
52
Image/Text Relations in NSW English K-6
Reading and viewing are linked throughout the
syllabus because the processes of reading and
viewing have much in common. They both involve
decoding and interpreting texts that is, they
are both based on understanding codes.
Furthermore, in many texts language and visual
images are interrelated for example, in
newspapers, textbooks, films, TV series and
documentaries (NSW English k-6 Syllabus, p.8).
53
Queensland and other States
Queensland English 1-10 syllabus indicates that
textual resources include linguistic, visual,
gestural, spatial and audio dimensions, and that
Students use language in ways that demonstrate
explicit knowledge and understanding of how
textual resources work interactively for
particular purposes in texts. (p.6). Similar
statements in Western Australia English syllabi
and those of other Australian States (for example
in the State of Tasmania Essential Learnings
Document,) emphasize the language/image link in
multimodal literacy in English.
54
English - National Curriculum for England
In the National Curriculum for English in the
United Kingdom concern with image/language
interaction appears to be confined to the section
of the curriculum dealing with Media and moving
image texts. In this section dealing with
reading in Key Stage 3 for example, it is
required that pupils should be taught how
meaning is conveyed in texts that include print,
images and sometimes sounds.
55
English - National Curriculum for England
Many books and other media now available in
schools cannot be read by attention to writing
alone. Much learning in the curriculum is
presented through images, often in the
double-page spreads of books, which are designed
to use layout, font size and shape and colour to
add to the information or stories contained in
the words. Such designed double-page spreads,
whether in picture or information book, make use
of spatial arrangements to convey ideas. We read
them differently from the way we read continuous
print When children use multimodal ways of
presenting ideas (often pictures plus words) they
use their knowledge of spatial organisation as
well as print conventions (Qualifications and
Curriculum Authority, 20045).
56
Meta-language and image/text relations in
multimodal literacy pedagogy
57
The consensus - metalanguage
There is also a strong consensus that the
knowledge readers need to have about how images
and text make meanings, both independently and
interactively, requires a metalanguage, or a
grammar, for describing these meaning-making
resources.
58
The quest for an appropriate metalanguage
facilitating the kind of metasemiotic work
required in multimodal literacy pedagogy,
although intensified in recent years, has been on
the agenda of some scholars over a significant
period of time. For example, Eliza Dresang , in
her book, Radical Change Books for Youth in a
Digital Age, reports a relevant quote from
McLuhan McLuhan spoke of one of the concepts of
Radical Change, the interaction between print and
electronic media, when he said Our job is not to
wreck the book but save it by teaching grammars
of new media quoted in Neill, S. (1971) Books
and Marshall McLuhan Library Quarterly
41(4)p.311.
59
Perry Nodelman (1988) believed the narrative art
of childrens picturebooks might be better
explicated if there were the possibility of a
system underlying visual communication that is
something like a grammar - something like the
system of relationships and contexts that makes
verbal communication possible (Nodelman,
1988ix.) Nodelman, P. (1988). Words about
pictures the narrative art of children's picture
books. Athens University of Georgia Press.
60
More recently Gees (2003) work on What Video
Games Have to Teach Us About Learning and
Literacy notes that in multimodal texts the
images often communicate different meanings from
the words and that the combination of the two
modes communicates things that neither of the
modes does separately. But he also notes that
for learning to be critical as well as active
The learner needs to learn not only how to
understand and produce meanings in a particular
semiotic domain that are recognizable to those
affiliated with the domain, but, in addition, how
to think about the domain at a meta level as a
complex system of interrelated parts For
active learning, the learner must, at least
unconsciously, understand and operate within the
internal and external design grammars of the
semiotic domain he or she is learning. But for
critical learning, the learner must be able
consciously to attend to, reflect on, critique,
and manipulate those design grammars at a
metalevel .
61
A pedagogy of multiliteracies
The orientation of our work in this unit aligns
with the agenda for a pedagogy of multiliteracies
as proposed in the Harvard Educational Review by
the New London Group (1996). This group
Courtney Cazden (Harvard, USA) Bill Cope
(University of Technology Sydney) Norman
Fairclough (Lancaster University, UK) Jim Gee
(Clark University,USA) Mary Kalantzis (James
Cook University of North Queensland) Gunther
Kress (University of London, UK) Allan Luke
(University of Queensland) Carmen Luke
(University of Queensland) Sarah Michaels (Clark
University, USA) Martin Nakata (James Cook
University of North Queensland) advise that the
multiliteracies for the future will need to be
built on
... an educationally accessible functional
grammar that is, a metalanguage that describes
meaning in various realms. These include the
textual and the visual, as well as the
multi-modal relations between the different
meaning-making processes that are now so critical
in media texts and texts of electronic
multimedia. (The New London Group 199677).
62
Visual literacy but no visual grammar in the
curriculum.
63
Literacy Curriculum and Pedagogy Lack of
consensus on metalanguage
Language
Images (Viewing)
Contestation of Metalanguage
Paucity of Metalanguage
Traditional grammar Functional
Visual Grammar(KressvanLeeuwen 1996)
Kress, G., van Leeuwen, T. (1996). Reading
images A grammar of visual design. London
Routledge.
64
Three different kinds of simultaneous
meaning-making in visual and verbal texts

• representational/ideational structures
  verbally and visually construct
• the nature of events, the objects and
  participants involved, and the
• circumstances in which they occur.
• interactive/interpersonal verbal and visual
  resources construct the
• nature of relationships among speakers/listeners,
  writers/readers, and
• viewers and what is viewed.
• compositional meanings are concerned with the
  distribution of the
• information value or relative emphasis among
  elements of the text
• and image.

65
Systemic Functional Linguistics
66
(No Transcript)
67
(No Transcript)
68
Teaching young children about images - The Deep
69
Using drama to teach knowledge about images in
the infants classroom - Demand
be a demand Jacqueline A.
70
(No Transcript)
71
Burn, A., Durran, J. (2006). Digital Anatomies
Analysis as Production in Media Education. In D.
Buckingham R. Willett (Eds.), Digital
Generations Children, Young People and New
Media. Malwah, New Jersey Erlbaum.
Yr 8 (12-13yrs) working on Baz Luhrmans film of
Romeo and Juliet using Adobe Premiere editing
software to re-arrange library shots to show
Romeo being pursued across Verona as he tries to
reach the dead Juliet.
The camera tilts up on Tybalt, this is because
the last shot was Romeo and he is on the floor
and the camera was going upwards from the floor
focussing on Tybalts reaction of what Romeo had
just said. This tells the audience that Tybalt
is the one in control and gives you the sense of
power. (Danielle)
The final shot is of a new character to the
sequenceSamson. The camera is placed at an
oblique angle to him. He is not an important
character, he is at the side of the action. His
emotion, his expression of fear and anxiety,
needs to be acknowledged - not felt - by the
audience. (Joe)
72
CONTENTS 1. Changing dimensions of school
literacies 2. Learning about language as a
resource for literacy development 3. Describing
visual literacies 4. Distinguishing the
literacies of school science and humanities 5.
Exploring multimodal meaning-making in literature
for children 6. Developing multiliteracies in
the early school years 7.Developing
multiliteracies in content area teaching 8.
Teaching multiliteracies in the English classroom
73
(No Transcript)
74
Literacy Curriculum and Pedagogy Lack of
consensus on metalanguage
Language
Images (Viewing)
Contestation of Metalanguage
Paucity of Metalanguage
Traditional grammar Functional
Visual Grammar(KressvanLeeuwen 1996)
Meaning at the Intersection of Language and Image
?
75
Implications for developing multimodal literacy
pedagogy

• While the research on an evolving metalanguage
  of multimodality is in the very early stages and
  emerging descriptions remain quite tentative,
  there are at least two firm, practical
  implications for teachers, curriculum authorities
  and government and non-government school systems

76
(No Transcript)
77
(No Transcript)
78
Conclusion

• It is hoped that this presentation, by
  stimulating critically constructive responses to,
  and envisioning beyond, what is presented here,
  will encourage further collaborative work among
  teachers, curriculum authorities, school systems,
  teacher educators and researchers, in exploring
  the nature and role of a metalanguage that will
  facilitate development of the multiliteracies
  pedagogies appropriate to the multimedia world of
  our children in the twenty first century.

79
Bibliography

• Unsworth, L. (2006) e-literature for Children
  Enhancing Digital Literacy Learning. London
  Routledge.
• Unsworth, L., Thomas, A., Simpson, A. and Asha,
  J. (2005) Childrens Literature and
  Computer-based Teaching. London
  McGraw-Hill/Open University Press.
• Unsworth, L. (2001) Teaching multiliteracies
  across the curriculum Changing contexts of text
  and image in classroom practice. Buckingham Open
  University Press.
• Unsworth, L. (in press). Multiliteracies and
  metalanguage Describing image/text relations as
  a resource for negotiating multimodal texts. In
  D. Leu, Corio, J., Knobel, M., Lankshear, C.
  (Ed.), Handbook of research on new literacies.
  New Jersey Erlbaum.
• Unsworth, L. (2007). Multiliteracies and
  Multimodal Text Analysis in Classroom Work with
  Childrens Literature. In T. Royce W. Bowcher
  (Eds.), Perspectives on the Analysis of
  Multimodal Discourse. New Jersey Erlbaum. (pp
  331-360).
• Unsworth, L. (in press/2008) Multiliteracies,
  e-Literature and English Teaching. Language and
  Education, 22(2).
• Unsworth, L. and Ortigas, I. (in press) Exploring
  the Narrative Art of David Wiesner Using a
  Grammar of Visual Design and Learning Experiences
  on the World Wide Web. L1-Educational Studies in
  Language Literature
• Unsworth, L. (2006) Multiliteracies and a
  metalanguage of image/text relations
  Implications for teaching English as a first or
  additional language in the 21st century. In K.
  Cadman K. O'Regan (Eds.), Tales out of school
  Identity and English language teaching. Special
  edition of TESOL in Context. Series S(1)
  pp147-162.
• Unsworth, L. (2006) Towards a metalanguage for
  multiliteracies education Describing the
  meaning-making resources of language-image
  interaction. English Teaching Practice and
  Critique, 5(1)55-76. http//education.waikato.ac
  .nz/research/files/etpc/2006v5n1art4.pdf
• Unsworth, L., Thomas, A. and Bush, R. (2004) The
  role of images and image-text relations in group
  basic skills tests of literacy for children in
  the primary years. Australian Journal of Language
  and Literacy, 27(1)46-65.