1
Design Research for Advancing the Integration
of Digital Technologies into Teaching and Learning
2
EPIT LPSL _at_ UGA
3
(No Transcript)
4
Agenda

• Critique the state-of-the-art of educational
  technology research.
• Describe attributes of design research.
• Encourage new thinking about why and how we do
  research and evaluation.

5

• And opening the window of his cell, he pointed to
  the vast church of Notre-Dame, which outlining
  darkly its two towers against the starry sky,
  with its stone flanks and its enormous back,
  appeared a gigantic two-headed sphinx crouching
  in the midst of the city. For some time the
  archdeacon considered the enormous edifice in
  silence, then with a sigh, extending his right
  hand towards the printed book which lay open upon
  his table, and with his left hand extended
  towards Notre-Dame, his eyes sadly wandered from
  the book to the church. 'Alas!' he said, 'this
  will kill that.'"

6

• And opening the window of his cell, he pointed to
  the vast church of Notre-Dame, which outlining
  darkly its two towers against the starry sky,
  with its stone flanks and its enormous back,
  appeared a gigantic two-headed sphinx crouching
  in the midst of the city. For some time the
  archdeacon considered the enormous edifice in
  silence, then with a sigh, extending his right
  hand towards the printed book which lay open upon
  his table, and with his left hand extended
  towards Notre-Dame, his eyes sadly wandered from
  the book to the church. 'Alas!' he said, 'this
  will kill that.'"

7
Only a fool would blithely welcome any technology
without giving serious thought to what the
technology will do, but also what it will undo.
- Neil Postman
8
Bad News
Oh...no!

• Educational researchers have been unable to
  provide compelling evidence that the integration
  of technology into higher education enhances
  teaching and learning.

9
Good News

• There are new strategies for conducting design
  research that can improve educational research
  so that it can become a more socially responsible
  enterprise.

Thank goodness!
10
August 6, 1999

• The Failure of Educational Research
• Vast resources going into education research are
  wasted.
• They educational researchers employ weak
  research methods, write turgid prose, and issue
  contradictory findings.

11
August 6, 1999

• The Failure of Educational Research
• Too much useless work is done under the banner of
  qualitative research.
• Qualitative research. yields .little that can
  be generalized beyond the classrooms in which it
  is conducted.

12
People increasingly want to know What is the
value of educational research?
13
College of EducationThe University of Georgia

• Ranked 22nd of 187 education colleges in the USA
• 220 faculty members in 9 departments
• 5,000 students in 32,000 student university

14
Research Productivity 1997-2001Refereed Journal
Articles - in-cites.com

1. U. of Wisconsin
2. U. of Georgia
3. U. of Michigan
4. Indiana U.
5. U. of Maryland

15
Georgia vs. Wisconsin

• 7,824
• 44,073
• 51
• 49th

• 8,604
• 42,232
• 78
• 7th

• Per pupil
• Salary
• HS Grad.
• Ranking

16
Youll find The Impact of Educational Research in
Fiction.
17
U.S. Dept. of Educations Position

• Theres been no improvement in education over
  the last 30 years, despite a 90 percent increase
  in real public spending per pupil.
• Promotes randomized controlled trials as used in
  medical research.

Secretary of Education Rod Paige
18
What Works Position

• Once we have dozens or hundreds of randomized or
  carefully matched experiments going on each year
  on all aspects of educational practice, we will
  begin to make steady, irreversible progress.
• NCLB funds scientifically based research.

Robert Slavin
19
It Wont Work Position

• Double blind experiments impossible in education
• Implementation variance reduces treatment
  differences
• Causal agents are unspecified in education
• Goals, beliefs, and intentions of students and
  teacher affect treatments
• Medical knowledge is not applied in many cases

David Olson
20
Randomized controlled trials are the only way
well ever be able to prove what works in
education!
Randomized controlled trials promotes
pseudoscience and will limit effective change!
21

• Ellen Condliffe Lagemann argues that educational
  researchers, in a misguided effort to be
  scientific, have turned away from the pragmatic
  vision of John Dewey.
• She criticizes the excessive emphasis on
  quantitative measurement.

22

• Kieran Egan argues that progressive ideas from
  Herbert Spencer, John Dewey, and Jean Piaget are
  responsible for the general ineffectiveness of
  our schools.
• He also assails the notion that education can be
  improved through research as traditionally
  conceived.

23
Thomas Kuhn The Structure of Scientific
Revolutions

• "I'm not sure that there can now be such a thing
  as really productive educational research. It is
  not clear that one yet has the conceptual
  research categories, research tools, and properly
  selected problems that will lead to increased
  understanding of the educational process. There
  is a general assumption that if you've got a big
  problem, the way to solve it is by the
  application of science. All you have to do is
  call on the right people and put enough money in
  and in a matter of a few years, you will have it.
  But it doesn't work that way, and it never will."

24
Lee Cronbach

• We cannot store up generalizations and constructs
  for ultimate assembly into a network.
• When we give proper weight to local conditions,
  any generalization is a working hypothesis, not a
  conclusion.

25
Educational technology research is just as poor
as educational research in general.
26
(No Transcript)
27
Chewing Gum More Effective than Interactive
Multimedia CD-ROM

• Dr. Ken Allen at NYU wanted to compare CD-ROM
  with lectures
• Wrigleys wanted to fund chewing gum study
• Combined study
• Gum chewers B-Abstainers C
• CD-ROM no better

28
Ed. Tech Research Reality

• Isolated researchers conduct individual studies
  rarely linked to a research agenda or concerned
  with any relationship to practice.
• Studies are presented at conferences attended by
  other researchers and published in journals few
  people read.
• Occasional literature reviews and meta-analyses
  are published.

29
Ed. Tech Research Reality

• Most instructional technology studies claim to
  have predictive goals (testing theories) and use
  quasi-experimental designs with quantitative
  measures.
• Research reviewers usually must reject 75 percent
  or more of the published studies to find the few
  worthy of further review or inclusion in
  meta-analyses.

30
Bernard et al. (2004) Meta-analysis How Does
Distance Education Compare to Classroom
Instruction?

• a very small but positive mean effect size for
  interactive distance education over traditional
  classroom instruction on student achievement
• small negative effect for retention rate

31
DE Research from 1985-2002

• 1,010 potential studies retrieved
• 232 studies met all criteria
• 599 independent effect sizes
• 47,341 students (achievement)

32
Results Overall Effects

• 325 independent outcomes (total achievement)
• Hedges g 0.0122, p lt .001
• Range of findings from 2.17 to 2.66
• 177 outcomes with low methodology removed
• Hedges g 0.017, p gt .05
• Significantly heterogeneous

33
Distribution of Effect Sizes
Hedgesg
Effect Sizes Ordered by Magnitude
325 independent outcomes (achievement) Hedges g
0.0122, p lt .001
34

• Abundant technology has not led to extensive use
  of computers for tradition-altering classroom
  instruction.
• The small percentage of computer-using
  instructors only use it to maintain existing
  classroom practices.

35
Instructors have legitimate concerns.

• Is it simple enough for me to integrate quickly?
• It it versatile?
• Will it motivate students?
• Is it aligned with skills Im expected to teach.
• Is it reliable?
• It it breaks, who will help?
• Will it weaken my classroom authority?

36
Todd Oppenheimer

• Our American desperation for objective
  information is illustrated nowhere more
  gorgeously than in the field of education. I am
  speaking of our tendency to promote any new
  concept by invoking volumes of quantitative
  research that ostensibly proves its value.
  technology advocates have played it expertly
  when it comes to claims about what computers will
  do for student achievement. As it turns out, the
  vast bulk of their research is surprisingly
  questionable.

37
Sir John Daniel - UNESCO

• the futile tradition of comparing test
  performances of students using new learning
  technologies with those who study in more
  conventional waysis a pointless endeavor because
  any teaching and learning system, old or new, is
  a complex reality. Comparing the impact of
  changes to small parts of the system is unlikely
  to reveal much effect and indeed, no significant
  difference is the usual result of such research.

38
Part of the problem stems from the failure to
distinguish between research goals and methods.
39
Ed. Tech research goals

• Theoretical
• Predictive
• Interpretivist
• Postmodern
• Design
• Action

40
Methods should not be selected until goals
research questions are clear

• Quantitative
• Qualitative
• Critical Theory
• Historical
• Literature Review
• Mixed-methods

41
What is the rationale for design research in the
higher education?
42
Experimental approaches to educational research
wont work!
MedicalCures
GeneticsResearch
43
Pasteurs Quadrant approach to research is
needed (Stokes, 1997).
Research is inspired by
44
The traditional educational research methodology
is expressed as follows
Lit. Review
45
Design research methodology can be expressed as
follows
Development of prototype solutions
ID of problems with practitioners
Derivation of design principles
46
Good Design ResearchDesign-Based Research
Collective

• Goals of designing learning environments and
  theories are intertwined
• Development and research occur in continuous
  cycles
• Research on designs leads to sharable theories
  relevant to practitioners
• Research must account for how designs function in
  authentic settings
• Development of accounts relies on methods that
  connect actions to outcomes

47
Design Research Scenario

• Geosciences faculty wish to explore feasibility
  of incorporating more inquiry-based learning
• Desire to utilize digital libraries

48
http//dlese.org
49
Teaching with technology works when learning
tasks are authentic!
50
Task-Orientation
Academic
Authentic

• textbook problems
• abstract context
• easily solvable
• one right answer

• ill-structured problems
• meaningful context
• time required
• multiple solutions

51
A key quality indicator of technology in higher
education is alignment.

• objectives
• content
• pedagogical dimensions
• instructor role
• student role
• technology role
• assessment strategies

52
Because education is a design profession (not a
discipline per se), we should pursue design
research that integrates the desire to solve
problems with the search for knowledge.
53
John DeweyLogic The Theory of Inquiry

• Design knowledge is warranted assertibility. It
  designates a potentiality rather than an
  actuality and involves recognition that all
  special conclusions of special inquiries are
  parts of an enterprise that is continually
  renewed, or is a going concern.

54
Sometimes our warranted assertions will allow us
to build marvels.
55
Other times we may fail.
56
But in the end, higher education will improve.
57
Todays Lecture Constructivism
58
Design Research Challenges

• Sampling bias
• Response bias
• Researcher bias
• Overwhelming data
• Confounded variables
• Dissemination
• Scaling up

Ann Brown
Brown, A. L. (1992). Design experiments
Theoretical and methodological challenges in
creating complex interventions in classroom
settings. Journal of the Learning Sciences, 2,
141-178.
59
Keeping pedagogy ahead of technology is an
ongoing struggle.
60
In my experience, this is especially true in
higher education!
61
Myth Faculty are easily prepared to teach with
technology.
62
RealityMost instructors are ill-prepared for the
demands of teaching with technology.
63
What about our students? With they engage in
online learning with a higher level of commitment
than normal.
64

• Colleges and universities are about to be beset
  by a new generation of learners whose skills and
  expectations derive from growing up on the net.

65
(No Transcript)
66
(No Transcript)
67
How much do our students really study?
68
Latest research is dismaying.

• Work expectation 10 15 hrs in class, 25 30
  hrs studying
• Reality20 study 5 hrs per week or less25
  6 10 hrs48 11 30 hrs 7 gt 30 hrs

69
Traditional Learning Domains

• Cognitive
• Affective
• Psychomotor

70
Cognitive Domain
What we say we value
Evaluation
Synthesis
Analysis
Application
What we teach and test
Comprehension
Knowledge
71
Affective Domain
Characterization by Value Set
Organization
Valuing
Responding
Receiving
72
Psychomotor Domain
Non-discursive Communication
Skilled Movements
Physical Activities
Perceptual Skills
Basic Fundamental Movement
Reflex Movement
73
Conative Domain

• Will
• Desire
• Level of effort
• Drive
• Striving
• Mental energy
• Self-determination
• Intention

74
History of the Conative Domain

• Orexis (Greek) Striving desire the conative
  aspect of mind
• Aristotle distinguished the conative from the
  cognitive (thinking) and affective (emotional)
  traits

75
cognitive
affective
conative
76
Can we restore the conative domain to its proper
place in higher education today?
77
Conative Domain

• Amazon search yields only one contemporary book
  about the conative domain.

78
Cognitive Affective Conative

• To act
• Willing
• Volition
• Ethics
• Doing

• To know
• Thinking
• Thought
• Epistemology
• Knowing

• To feel
• Feeling
• Emotion
• Esthetics
• Caring

79
How do we judge the quality of Design Research?
80
Good Design ResearchDesign-Based Research
Collective

• We suggest that the value of design-based
  research should be measured by its ability to
  improve educational practice.

81
The status of research deemed educational would
have to be judged, first in terms of its
disciplined quality and secondly in terms of its
impact. Poor discipline is no discipline. And
excellent research without impact is not
educational. - Charles W. Desforges (2000)
82
Quality is controversial because judging it is
often so subjective.
Yang Tae-young
Paul Hamm
83
Quality is controversial because judging it is
often so subjective.
84
Quality is controversial because judging it is
often so subjective.
85
Flyer
86
Spencer
87
(No Transcript)
88
Thank You!

• Professor Tom Reeves
• The University of Georgia
• Instructional Technology
• 604 Aderhold Hall
• Athens, GA
• 30602-7144 USA
• treeves_at_coe.uga.edu
• http//it.coe.uga.edu/treeves
• http//www.evaluateitnow.com