Technology: Access to the Future Regional Meetings

1
(No Transcript)
2
Technology Access to the FutureRegional
Meetings

• Ellen B. Mandinach
• Naomi Hupert
• EDC Center for Children and Technology
• WWW.edc.org/CCT
• November 21, 2003
• January 23, 2004
• February 20, 2004
• March 19, 2004

3
(No Transcript)
4
A paradox gradually became evident The more a
technology, and its usages, fits the prevailing
educational philosophy and its pedagogical
application, the more it is welcome and embraced,
but the less of an effect it has. When some
technology can be smoothly assimilated into
existing educational practices without
challenging them, its chances of stimulating a
worthwhile change are very small.Salomon and
Almog, 1998
5
According to Papert (1987), if an instructional
technology is harmless that it is easily
integrated into existing pedagogical practices
without many changes, then it will be equally
harmless in making an instructional difference.
6

• Question
• What is the difference between elephants mating
  and the implementation of educational technology?
• Answer
• There is a lot of dust and noise, and nothing
  happens for a very long period of time!

7
Think of a pointillist painting by Seurat or an
impressionist work by Monet

• Step up to the painting.
• Step all the way back.
• Compare what you see.
• That is precisely what evaluation and assessment
  require - taking multiple perspectives of the
  same phenomena and getting different feedback.

8
(No Transcript)
9

• Question
• How can the accountability issue be addressed?
• Answer
• With great difficulty
• Use a number of measurement strategies, asking
  different, but related questions, all concerned
  with various aspects of the learning process and
  outcomes.

10
Mandinach Cline (1994)
11
Assessment, Evaluation, and Research

• Different purposes, questions, and objectives
• Assessment - the measurement of learner
  performance. Can be part of an evaluation, but
  not synonymous.
• Evaluation - the examination of a system,
  product, or program - formative, summative, or
  both - to determine how it is functioning, being
  implemented, and how it can be improved.
• Research - encompasses assessment, evaluation,
  and much more.

12
(No Transcript)
13
Evaluation and ResearchSome Tradeoffs

• Is likely to be most effective when it is both
  formative and summative.
• Should be used for planning as well as a
  systematic research tool.
• Must consider the information desired by the
  stakeholders.
• Must use measures that will maximize the
  potential for detecting impact.
• Must use standardized tests or targeted
  assessments.
• Must use experimental versus other designs.
• Must use comparison/control or matched groups.

14
Some Caveats
All parties involved need to change their
conceptions of proof of successful implementation
of technology and its impact on teaching and
learning.
What does it mean to say, it works?

• Need for continuous adjustments in
• pedagogical philosophy.
• assessment techniques.
• strategies, roles, priorities, and schedules.

15
Is there enough implementation of the technology
to enable measurement?

• There must be enough technology implementation
  to produce the desired outcomes.
• If you are only using technology for a small
  amount of instructional time, there will be
  limited exposure for each student each day.
  Therefore the targeted outcomes are not likely to
  show substantial effects size.

16
Evaluation/Research Issues

• Experimental paradigm
• eliminate possible explanations
• verify hypothesized causal relationship
• control for contaminating influences
• random assignment of students to experimental and
  control groups
• controlled application of the stimuli
• Barriers to and issues in the use of an
  experimental
• design.
• selection, acquisition, and installation of
  hardware and software
• training and ongoing support of teachers
• enlisting support, encouragement, and
  participation of students, teachers,
  administrators, school board members, and parents
• actual classroom implementation (for as long as
  it takes)

17
Quasi-Experiment Versus Formative Experiment

• Not sufficiently powerful
• Not sufficiently sensitive to changes
• Inadequate evaluative question -
• Does it work?

18
Rarely does one study produce an unequivocal and
durable result multiple methods, applied over
time and tied to evidentiary standards, are
essential to establishing a base of scientific
knowledge.Shavelson Towne, 2002
19
Methodological Implications for Technology-Based
Educational Reform Efforts

• Longitudinal Design
• Multiple Methods
• Hierarchical Analyses
• System Dynamics

20
Longitudinal Design
Sacrifice the quasi-experimental design in favor
of ongoing longitudinal data collection and
analyses that are carried out continuously and
indefinitely.

• Reasons that quasi-experiments are difficult to
  implement in classroom settings
• pre and post measures
• experimental and control groups
• random assignments of students
• control of how and when the stimulus is
  administered

21
Multiple Methods
Educational innovation efforts often generate
multiple outcomes that require the triangulation
of traditional and nontraditional data
collection and assessment methods. For example

• Think aloud protocols
• Classroom observations
• Peer observations
• In-depth interviews with teachers, students, and
  administrators
• Content analyses of essays
• Paper and pencil assessments
• Performance assessments
• Notebooks and portfolios
• Performance on traditional tests,
• assignments, and projects
• Many data items routinely generated in the
  operation of any school (e.g., GPA, tardiness,
  attendance, drop-out rates, course taking
  patterns)
• Gather data at multiple time points
• Case studies
• Bottom line Identify consistent patterns across
  data sources

22
HIERARCHICAL ANALYSES
Examine impact at different levels of analysis

• Student Learning - Processes and Outcomes
• Classroom Dynamics - the Changing Patterns of
  Interactions between Students and Teachers
• The School as a Social Organization - its
  Structure and Functions

23
SYSTEM DYNAMICS

• A school is an interdependent, multilayered
  system. One has to understand what is happening
  across layers and levels.
• Reform efforts take place in real world contexts
  that are composed of many interrelated, dynamic
  factors.
• Any attempt at educational reform will have
  multiple components and multiple impacts, and
  they will interact across levels of organization
  over time.
• Asking a single question about impact or outcome
  is naive.
• Thus, we need to approach
• educational reform systemically.

24
The character of education not only affects the
research enterprise, but also necessitates
careful consideration of how the understanding or
use of results can be impeded or facilitated by
conditions at different levels of the system.
Organizational, structural, and leadership
qualities all influence how the complex education
system works in practice.Shavelson Towne,
2002
25
(No Transcript)
26
How do you know if the technology is working?

• Questions - better, more, what ifs, what and how
  they ask
• Deeper understanding
• More engaged and on task
• How they interact
• Challenged
• Argumentation
• Application
• Independence and self-directedness
• Planfulness and organizational skills
• Increased problem solving, logical analysis
• Reponses in discussion
• Decision making
• See the light bulb go on - the aha experience
• See the obvious
• Students in charge of their own learning
• Success is being able to handle failure and learn
  from it

27
Mandinach Cline (1994)
28
(No Transcript)
29
WHAT Does NCLB Want?

• To determine with scientific rigor
• WHAT WORKS.
• Translation The impact of the intervention must
  be to
• INCREASE TEST SCORES!

30
What is Required by the What Works Clearinghouse
and NCLB

• Randomized, controlled, experimental studies,
  using the medical model of research.
• Not matched comparisons.
• Not quasi-experimental designs.
• Must establish causality, ruling out plausible
  explanations.
• Small, focused interventions.
• Limited teacher professional development
  components.
• Short-term.
• School patterns are not changed.
• Students are the unit of assignment, not
  classrooms or schools.
• No contextualization.
• Foremost, there must be valid and reliable
  evidence that the intervention improves student
  achievement through scientific evidence.

31
The Medical Model as the Gold Standard

• The Institute for Educational Sciences (IES) in
  the US Department of Education invokes the
  medical model of research as the standard toward
  which all research should strive.
• Yet is this gold standard achievable?
• Is it the right gold standard or a silver bullet?
• For example, can an instructional intervention
  be examined in the same way as a course of
  pharmaceutical treatment?

32
Research and Evaluation Methodology Required by
NCLBRandomized Field Trials (RFTs)

• The rationale for RFTs is the quest for
  unambiguous information in education.

33
To be scientific, the design must allow direct,
empirical investigation of an important question,
account for the context in which the study is
carried out, align with a conceptual framework,
reflect careful and thorough reasoning, and
disclose results to encourage debate in the
scientific community.Shavelson Towne, 2002
34
The Six Guiding Principles of Scientific
Inquiry(Not the Seven Deadly Sins)

• Pose significant questions that can be
  investigated empirically (ruling out counter
  interpretations and bringing evidence to bear on
  alternative explanations)
• Link research to relevant theory
• Use methods that permit direct investigation of
  the question
• Provide a coherent and explicit chain of
  reasoning
• Replicate and generalize across studies
• Disclose research to encourage professional
  scrutiny and critique

35
Which Really is the Driving Factor -Research
Questions or Methods?

• The question should drive the research
  methodology, not the research methodology driving
  the questions.
• Unfortunately, all too often the reverse has been
  happening because of political pressures.
• Mandated questions, methods, and potentially
  answers as well.

36
The Question Should Drive the Research Design

• What is happening (e.g., descriptions of
  population characteristics)?
• Is there a systematic effect (i.e., systematic
  means causal)?
• How or why does it happen?
• Need to account for contextual factors.
• Replicability of patterns across groups and time.

37
Evaluation

• Should be meaningful and constructive. The
  results and information should benefit the
  students, teachers, school, and district.
• Should not be punitive.
• Should be informative, providing information on
  what is going on, how to improve, or other
  important questions.

38
Designing Evaluations

• Use targeted evaluations.
• Match your goals to data collection activities -
  that is, let the questions drive the methods.
• Use measurable components.
• Consider the design before it is implemented.
• Be flexible. Things change and the evaluation
  design must change accordingly.

39
Numerous Caveats to RFTs

• Fidelity of implementation
• Variability of treatment
• Overlap between treatment and control groups
• Adequacy of outcome measures
• Multiple treatment interference
• Relevance of control condition to policy issues
• External validity

40
NCLB GoalsImpact on Students

• Primary Goal - To improve student academic
  achievement through the use of technology in
  elementary schools and secondary schools.
• Additional Goals
• To assist every student in crossing the digital
  divide by ensuring that every student is
  technologically literate by the time the student
  finishes the eighth grade, regardless of the
  students race, ethnicity, gender, family income,
  geographic location, or disability.
• To encourage the effective integration of
  technology resources and systems with teacher
  training and curriculum development to establish
  research-based instructional methods that can be
  widely implemented as best practices by State
  educational agencies and local educational
  agencies.

41
NCLB QuestionsImpact on Students

• Is academic achievement improving with effective
  technology use?
• Are students acquiring 21st century skills
  through effective technology use?
• Are students more engaged in learning through
  effective technology use?

42
Necessary Conditions

• Effective Practice - Is classroom practice
  characterized by powerful, research-based
  strategies that effectively and appropriately use
  technology?
• Educator Proficiency - Are educators proficient
  in implementing, assessing, and supporting a
  variety of technology-based teaching and learning
  practices?
• Robust Access Anywhere Anytime - Do students and
  staff have robust access to technology anywhere,
  any time, to support effective designs for
  teaching and learning?
• Digital Age Equity - Is the digital divide being
  monitored and addressed through resources and
  strategies aligned to 21st century vision?
• Vision and Leadership - Is there a 21st century
  vision? Is the education system transforming
  into a high-performance learning organization?

43
(No Transcript)
44
Information and Communication Technology Literacy

• ICT literacy is more than just the mastery of
  technical skills. It also includes
• Cognitive skills.
• The application of cognitive skills and
  knowledge.
• ICT literacy is seen as a continuum of skills and
  abilities from simple, everyday tasks to complex
  applications.

45
A Working Definition

• ICT literacy is using digital technology,
  communications tools, and/or networks to access,
  manage, integrate, evaluate, and create
  information in order to function in a knowledge
  society.
• Source ICT Literacy Panel, 2002.

46
ICT Proficiency Skills

• ACCESS - knowing about and knowing how to collect
  and/or retrieve information.
• MANAGE - applying an existing organizational or
  classification scheme.
• INTEGRATE - interpreting and representing
  information. It involves summarizing, comparing,
  and contrasting.
• EVALUATE - making judgments about the quality,
  relevance, usefulness, or efficiency of
  information.
• CREATE - generating information by adapting,
  applying, designing, inventing, or authoring
  information.

47
Three Proficiencies

• Cognitive Proficiency - the desired foundational
  skills of everyday life at school, at home, and
  at work. Literacy, numeracy, problem solving,
  and spatial/visual literacy demonstrate these
  proficiencies.
• Technical Proficiency - the basic components of
  digital literacy. It includes a foundational
  knowledge of hardware, software applications,
  networks, and elements of digital technology.
• ICT Proficiency - the integration and application
  of cognitive and technical skill. Seen as
  enablers, they allow individuals to maximize the
  capabilities of technology. At the highest
  level, ICT proficiencies result in innovation,
  individual transformation, and societal change.

48
The SETDA Technology Literacy Working Definition

• Technology literacy is the ability to responsibly
  use appropriate technology to communicate, solve
  problems, and access, manage, integrate,
  evaluate, and create information to improve
  learning in all subject areas and to acquire
  lifelong knowledge and skills in the 21st century.

49
Sources of 21st Century Skill Definitionswww.ncre
l.org/engauge/skills/sources.htm

• The enGauge 21st-Century Skills
• National Education Technology Standards
• SCANS (Secretarys Commission on Achieving
  Necessary Skills)
• A Nation of Opportunity Building Americas 21st
  Century Workforce
• Preparing Students for the 21st Century
• Standards for Technological Literacy, Content for
  the Study of Technology
• Being Fluent with Information Technology
• Information Literacy Standards for Student
  Learning
• Growing Up Digital

50
Interactive Assessment and Evaluation

• Assessment can be used both as a teaching tool
  and an evaluation mechanism.
• Assessment and instruction should be iterative -
  a feedback loop to provide information to both
  student and instructor - continuous and
  interactive.
• Assessments should assist students to evaluate
  their learning processes and outcomes. They
  should help to facilitate learning.
• Techniques should capitalize on the affordances
  of the technology.
• Online assessment should not be restricted by
  time constraints or resources.

51
Interactive Assessment and Evaluation(Continued)

• Techniques should be creative (e.g., games,
  puzzles, competitions).
• Group assessments can be effective tools.
  Consider collaborative learning groups and team
  projects.
• Information from assessments should provide the
  teacher with information about a students
  strengths and weaknesses.
• Design thought provoking questions that stimulate
  interesting debate and threaded discussions.

52
Interactive Assessment and Evaluation(Continued)

• One form of assessment effective online is
  project-based learning - intensive, long-term,
  and focused on specific, real-world topics and
  authentic problems - accomplished individually or
  collaboratively.
• Another form is a short, iterative instruction -
  assessment feedback cycle.
• Use multiple methods.
• Focus on the formative, not just summative -
  processes, not just outcomes, with chances for
  reassessment, acknowledging that every student
  has a different learning curve.

53
Interactive Assessment, Evaluation, and
Instruction Issues

• Collaborative learning necessitates new paradigms
  for learning, instruction, assessment, and
  honesty.
• Rules for assessment need to be made clear to the
  students. If assessments are conducted properly,
  cheating may become a moot issue.
• Expectations should be made clear in terms of
  participation online, including threaded
  discussions.
• Teacher training is a key issue in terms of the
  technology, paradigm shifts, and role changes.

54
Interactive Assessment, Evaluation, and
Instruction More Issues

• Encourage interaction among student and teacher.
  
• Provide opportunities for peer review, not just
  teacher feedback.
• Encourage active participation and learning.
• Provide prompt feedback.
• Recognize diversity in learning - different
  paths, different paces, different learning
  styles.
• Tailored to individual needs.
• Provides a learning trail.

55
Some Existing K-12 Products and Companies

• Smarterkids.com
• LeapFrog
• K12 - William Bennetts diagnostic tests
• Classroom Connect NOW
• ETS K-12 Works
• Ignite! - Neil Bushs company
• Many others
• A mega-market now at 105 billion (NY Times,
  1/21/01)

56
A Major Caution

• Many of the companies are jumping onto the
  bandwagon and the testing craze.
• Tests should be professionally developed and
  psychometrically sound. Many companies are
  cutting corners in the rush to market.
• Consider in the selection process the pedagogical
  theory that underlies the tests and the
  psychometric models. Are the tests even grounded
  in theory????
• BE CAREFUL!

57
Interactive Assessment, Evaluation, and
Instruction A Paradigm Shift

• Any time, any where, any pace.
• Immediate knowledge and infinite resources.
• Global learning.
• Multimedia.
• Interactive rather than lecturing or didactic.
• Learner-directed, not teacher-directed.
• Active versus passive. Student engagement.
• Role changes and shifts in responsibility for
  learning.
• Collaboration versus individual learning.
• Guide on the side versus sage of the stage.
  The teacher as a facilitator and coach, not as
  the primary transmitter of information.
• Intellectual exploration.

58
Recommendations

• Implementation
• Dont depend on the technology itself to improve
  teaching.
• Teachers change their practices more readily when
  they can work in teams and have the support of
  administrators.
• Teachers are more likely to use technology
  effectively when they have a computer at home on
  which to practice.
• Technical support for teachers should be
  available at the building level, not just at the
  district level.

59
Recommendations (continued)

• Use the technology to support existing activities
  instead of designing the activities around
  technology. Develop activities for the sake of
  learning, not for the chance to use the
  technology.
• Dont rely on traditional tests to determine what
  students have learned. Create activities that
  allow students to demonstrate what theyve
  discovered through their projects to enable a
  more accurate assessment of their knowledge and
  skills.
• Be patient. Teachers need time to learn how the
  technology can be used in the classroom.
  Students then need time to discover the knowledge
  by themselves and in collaboration with
  classmates.

60
Recommendations (continued)

• Teacher Professional Development
• Training should focus on how technology can
  enhance learning and be embedded in real
  projects rather than simply on how to work the
  machines.
• Teachers need time to rework lessons to use
  technology effectively and experiment with new
  teaching styles.
• If teachers are encouraged to share with other
  teachers what they are trying in the classroom,
  momentum for change can grow.
• Teachers need professional development
  opportunities and continuing support from a
  facilitator or coach to learn how to use the
  technology.

61
Agenda for Research and Action

• Conduct targeted case studies that address
  effective implementations in diverse educational
  settings.
• Conduct research that addresses the ramping up
  issue.
• Provide collaborative outreach and research
  opportunities to schools to begin to bridge the
  digital divide.
• Shift the focus of teacher professional
  development to include technology as an important
  component of the emerging educational paradigms
  and pedagogical philosophies.

62
(No Transcript)
63

• The Center for Children and Technology
• A division of the Education Development Center
• WWW.edc.org/CCT
• Resource
• Identifying and Implementing Educational
• Practices Supported By Rigorous
• Evidence A User Friendly Guide
• www.excelgov.org/evidence