Summarizing, Explaining, and Diagramming: The Differential Effects on Reading Times, TextBase Repres

1
Summarizing, Explaining, and Diagramming The
Differential Effects on Reading Times, Text-Base
Representations, and Mental Models of Science
Text

• Janice Gobert
• Senior Research Scientist Research Associate
• The Concord Consortium Dept. of Learning
  Teaching
• 37 Thoreau Street Graduate School of
  Education
• Concord, MA 01742 Harvard University
• jgobert_at_concord.org Cambridge, MA 01238
• Check us out at www.concord.org
• This research is funded by the the National
  Science Foundation under grant No. REC-9806141
  awarded to the author. Any opinions, findings,
  and conclusions expressed are those of the
  presenter and do not necessarily reflect the
  views of the NSF.

2
Background to the Research Study 1

• Builds on two studies addressing the on-line
  processing of science text and conceptual gains
  made when different orienting tasks are given to
  students.
• Study 1 Gobert Clement (1999 WTC, 1997) --
  effects of either summarizing or diagramming vs.
  read only on
• 1) students understanding of science text using
  summaries or diagrams generated at 3 different
  points in the text (intermediate representations)
• 2) post-test measures reflecting conceptual
  understanding (situation models).
• On the post-test -- diagram group summary group
  and summary group read only group as predicted
  on both the understanding of spatial and
  causal/dynamic aspects of the domain.
• On students intermittent tasks (i.e., summaries
  and diagrams generated during reading)--
  summaries contained more semantic information
  than did the diagrams. Puzzling at first, but.
• Interpreted as follows For summary group,
  summarizing was a low-level task, as it involved
  same media as text, and thus, did not elicit the
  development of rich mental models from which
  inferences could be made. Superior performance on
  the post-test by the diagram group, reflecting
  students situation model understanding (Kintsch,
  1988) also supports these findings, i.e., that
  diagramming elicited a deeper processing of the
  text, seen at post-test.

3
Background to the Research (contd) Study 2

• Study 2 Gobert (1997) -- whether the learning
  advantages obtained by the diagram group
  summary group in Study 1 were due to the
  translation from textual into a visual
  representation (constructing diagrams) or due to
  inferencing in general.
• same experimental design except the two orienting
  conditions used were diagramming and explaining
  (a higher level task than summarizing).
• Results
• diagram group explanation group on the amounts
  of semantic information generated on the
  intermittent tasks during reading AND
• diagram group explanation group on the
  conceptual post-test measures reflecting mental
  models of the domain.
• Interpretation generating explanations and
  diagrams both promoted inference-making on the
  textual material and development of rich
  situation models, as evidenced by equally good
  performance on the post-test and equivalent
  amounts of semantic information on the
  explanations and diagrams.
• Thus, it was not the visual representation
  driving the superior performance in Study 1.

4
Ok, so the visual medium hypothesis was out the
window, what now?

• Studies 1 and 2 provided a motivation for
  investigating the reading time performance,
  intermittent task performance, and subsequent
  conceptual understanding in three different
  orienting conditions in order to track potential
  differences in text comprehension.
• Orienting tasks for Study 3 summarizing,
  explaining, and diagramming.
• IF reading time performance of the explanation
  diagram group but summary group..
• it is possible that similar processes are being
  used in the explanation and diagram conditions in
  setting up mental models of the text.

5
Research Design Study 3

• Subjects 30 5th grade students from a suburban
  town in eastern Massachusetts were selected for
  minimal prior knowledge of the domain on the
  basis of a written pre-test.
• Data Collection
• Students interviewed individually
• sessions 45 minutes and 1 hour.
• All sessions were video-taped using two cameras,
  one focussed on the student and interviewer, the
  other was positioned over the students desk to
  record the diagrams, etc.
• The Text Source
• 3 pages in total 5 sections 1) introduction, 2)
  layers of the earth, 3) the movement and
  processes inside the earth, 4) how mountains form
  and how volcanes erupt, and 5) how the sea floor
  spreads.
• text presented on a Macintosh computer using
  Select-the-Text for reading times (Goldman
  Saul, 1990).
• reading times were recorded for each sentence and
  for each section of text, both in milliseconds.

6
Research Design Study 3 (contd)

• The three orienting intermittent tasks were
  to draw a diagram at 3 specific points in the
  text, to summarize at 3 specific points in the
  text, and to explain at 3 specific points in the
  text.
• Orienting instructions, given BEFORE the
  respective section
• After this paragraph you will be asked to give
  a summary OR give an explanation, OR draw a
  diagram of. the different layers of the earth
  (task 1)
• .. the movement and processes inside the
  earth (task2)
• the movement and processes inside the earth
  when mountains form (task 3).
• The intermittent tasks (1-3) were designed and
  ordered in order to allow for a learnable
  progression of models where simple models provide
  conceptual leverage for building more complex
  models (Raghavan Glaser, 1993 White, 1993).
• While drawing, summarizing, or explaining, the
  students could not look back at the text.

7
Coding of Students Intermittent Data, e.g.,
Spatial Scoring- Mountain Formation (task 3)

• Based on a propositional analysis of the text
  read by the students,
• Scoring of Spatial Components are
• crust
• LOC on surface/ 1st layer 1 point
• PRT plates 2 points
• ATT up to 96 miles thick 1 point
• ATT plates, moving 1 point
• PRT crust, continents 1 point
• mantle
• PART magma 2 points
• LOC below crust/ 2nd layer 1 point
• magma
• ATT liquid 1 point
• ATT hot 1 point
• core
• LOC center of earth/ interior layer 1
  point
• ATT hot mass 1 point total spatial (13)
  

8
Coding of Students Intermittent Data, e.g.,
Causal Scoring for Mountain Formation (task 3)

• Based on a propositional analysis of the text
  read by the students,
• Scoring of Causal/Dynamic Components
• core heats magma 2 points
• if currents are shown/mentioned 2 points
• currents of magma rise to top of mantle 2
  points
• mantle moves 2 points
• plates collide/forced together 2 points
• one plate moves under another 2 points
• crust/land breaking/crumbling 2 points
• cyclical pattern causes mountain 2 points
  total (16)
• For more info on how a semantic-based coding
  scheme can be used for diagrams and
  explanations, see Gobert Clement (JRST, 99)
  Gobert (IJSE, 2000).

9
Coding of Students Post-test Data

• Post-test All of the post-test items assessed
  either spatial or causal/dynamic knowledge.
• Examples of spatial/static items are
• Where is the thinnest part of the crust, on the
  top of mountains or at the sea floor?
• If the continents were all together, would the
  rest of the earths surface be water?
• and spatial features of students post-test
  diagrams depicting volcanic eruption and sea
  floor spreading (these diagrams were NOT
  requested during reading).
• Examples of causal/dynamic items are
• Plates moving apart causes. . .?
• Why is rock from the floor of the Atlantic
  Ocean newer (i.e. younger) than rock from the
  middle of the North American Continent because.
• and causal and dynamic features of students
  post-test diagrams depicting volcanic eruption
  and sea floor spreading.

10
Hypotheses Regarding Reading Times

• For part 1- Introduction- no orienting task
  given
• expected no differences in reading times
• provided an analysis of whether the three groups
  baseline reading times were equivalent
• For part 2- The Layers of the Earth consisting
  primarily of spatial information
• expected diagram and explanation group summary
  group since task instructions would elicit a
  deeper level processing of the text,
• interaction of diagramming or explaining with
  text requires setting up a spatial mental model
  AND that this would be similar in these two
  conditions.
• For parts 3- The Movement and processes inside
  the Earth and part 4- How Mountains Form
  consisting of several different types of
  information (spatial, causal, dynamic)
• expected diagram and explanation groups summary
  group since their task instructions would elicit
  a deeper level processing of the text, and
  processing required to annotate spatial mental
  model with causal dynamic information would be
  similar.
• For an elaboration on mental model building from
  text in this domain, see Gobert, Int. J. Sci.
  Ed., 2000.
• For part 5- How volcanoes erupt and how the sea
  floor spreads, no orienting task given
• served as an exploratory analysis of whether the
  students would apply the same strategy used in
  reading the 3 previous sections of text.

11
Hypotheses Regarding Post-test Measures

• Based on vanDijk and Kintschs text comprehension
  model (van Dijk Kintsch, 1983) simple recall
  and recognition tasks are best supported by a
  good text-base and inference tasks are best
  supported by higher level, mental models or
  situation models (Johnson-Laird, 1983 Kintsch,
  1987).(as well as based on date from Study 1)
• hypothesized that generating either diagrams or
  explanations while reading would promote the
  development of rich mental models reflected by a
  superior understanding of the domain compared to
  the summary group.

12
Regarding Intermittent Tasks.

• A corollary of the hypothesis for the post-test
  regarding the intermittent data is .
• For the intermittent tasks--the summary group
  would generate summaries with information that
  more closely reflected the surface structure of
  the text when compared to the explanation group
  thus,
• Summary explanation on RECALL of propositions of
  propositions from text.
• Explanationsummary on PARAPHRASE and INFERENCE
  of propositions from text.
• This analysis is presently being done and will
  not be discussed today.

13
Overview of Results

• There are four sets of quantitative data
• 1) reading time data for each section of the
  text,
• 2) intermittent data generated by the student at
  3 points during the text (these data will not be
  discussed today) and
• 3) performance on the post-test.

14
Analysis of Reading Times

• (Re prior knowledge of the domain, there was no
  statistically significant differences found
  across the three groups for prior knowledge (F
  2.243, p0.125), so all subsequent analyses were
  done NOT using prior knowledge as a covariate.
  (Subjects were selected for minimal p.k.).
• Reading time for each section of text was used as
  opposed to sentence reading times because
  multiple types of knowledge, (i.e., spatial,
  causal, dynamic) were reflected in many of the
  sentences of the text.
• READING TIME ANALYSES
• Part 1 Introduction-- No orienting task given
  here
• Anova, total time part 1 as the d.v. yielded, as
  expected, no statistically significant
  differences across groups (F .92, p.414)
• Thus, no baseline differences in reading times
  across groups-GOOD.

15
Table 1aTotal Reading Time for Part 2 Layers
of the Earth
0 summary 1 explanation 2 diagram
16
Figure 1Total Reading Time by group for Part 2
Layers of the Earth
Significant Diagsum Diagexp
17
Total Reading Time for Part 3 Movement in the
Layers of the Earth
0 summary 1 explanation 2 diagram
18
Figure 2Total Reading Time by group for Part 3
Movement of the Layers of the Earth
Significant Diagsum Diagexp
19
Total Reading Time for Part 4 Movement in the
Layers of the Earth When Mountains are Formed
0 summary 1 explanation 2 diagram
20
Figure 3Total Reading Time by group for Part 4
Movement in the Layers of the Earth When
Mountains are Formed
Significant Diagsum Diagexp
21
Analysis 5 Part 5 of the Text, Why the
Atlantic Ocean is getting wider.

• Here, no orienting task given
• Exploratory question whether students might
  utilize the same reading processes they had were
  prompted to do in tasks 2, 3, and 4.
• Anova using total time on part 5 as the d.v.
  yielded no statistically significant differences
  between the groups (F.243, p.786).
• Thus, students did not employ the same processing
  strategies they had used in the prior three
  sections of text where the orienting tasks were
  given.

22
Analysis of Post-Test Data

• spatial knowledge and causal/dynamic knowledge
  entered as variables into a manova.
• No statistically significant differences were
  found (Fmult .60, p.67). Univariate not
  significant for
• for spatial knowledge (Funiv .007, p.99) or
• causal/dynamic knowledge (Funiv .397, p.68)
• Analysis of Specific Post-Test Items (desperate
  times calls for desperate measures.)
• Anovas on each of the 14 post-test items yielded
  no statistical significant results, but some
  means in the expected direction.

23
Overview of Results

• The diagram group consistently took longer than
  either the explanation and summary group to read
  sections of the text where orienting tasks were
  given.
• It was expected the reading time explanation
  group diagram group because it was assumed that
  higher level processing is required to construct
  mental models (regardless of medium of
  information received)- this did not happen!
• Rather the summary group explanation group on
  reading time, and the diagram group than
  summary or explanation.
• BUT, the post-test does not reflect any gains for
  the diagram group despite their longer time spent
  reading the text?

24
Some Speculations

• Speculations about this are...
• 1) the summary task (as well as the other two)
  said Include as much information as you can and
  as such, may have elicited a higher level
  processing of the test than is typical with
  summarization tasks.
• 2) through the act of articulating summaries and
  explanations during their reading of the text,
  students were reifying what they had read, and
  thus performed equally well on the post-test.
• Additional Analyses underway to tease this out
  are
• 1) analyses of the three summaries and
  explanations generated during reading are being
  coded for propositions recalled, inferred, and
  paraphrased. If summarizing was interpreted as a
  high level task by the students we would expect
  to see similar proportions of propositions
  recalled, summarized, and inferred when compared
  to the explanation group.
• BUT, if summarizing was interpreted as a lower
  level tasks (as it was designed to), we would
  expect explanationsummary on propositions
  paraphrased inferred but summaryexplanation on
  propositions recalled.
• (The summaries, explanations, and diagrams are
  being coded for inclusion of semantic features
  but it is likely that this is not fine grained
  enough to pick up difference. Additional analyses
  may be required of all the video data in the
  three conditions to track possible differences
  between the three groups while they summarized,
  explained, or diagrammed).

25
Conclusions

• These data are consistent with other studies on
  self-explanations (Cote Goldman, 1998 Cote, et
  al, 1998) in which it was found that the
  relationship between self-explanations and
  learning are not simple and linear.
• Cote et al speculate that differences might be
  due to the types of self-explanations that are
  generated. In terms of the present study, a fine
  grained analysis of the propositions recalled and
  inferred across the summary and explanation
  groups may yield important information about what
  features students are attending to in
  constructing these representations.
• Further research needs to be conducted with texts
  that are specifically designed to tease out
  potential group differences. The text used here
  had many sentences which contained multiple types
  of information, i.e., spatial, causal, and
  dynamic, and combinations of the three, thus
  sentence by sentence reading time comparisons
  would not be fruitful. The domain, being highly
  spatial, causal, and dynamic may not be conducive
  to designing natural texts for this purpose.
• Regarding Summarizing, Explaining, and
  Diagramming, there are differential effects on
  reading times, there may be differences in
  Text-Base Representations and there are no
  tractable differences in mental models of science
  text.