Identifying and Measuring the Value of 21st Century Skills

1
Identifying and Measuring the Value of 21st
Century Skills

• CoSN Partnership for 21st Century
  SkillsFountain Hills Unified School District,
  Arizona

2
The Issue

• School Goal Our graduates will be successful in
  college and the workplace
• What do they need to learn?
• What is the value of investment in these skills?
  (a cost-benefit analysis)
• How do we measure achievement?

3
Agenda

• Rich Kaestner - CoSN
• Overview of 21st century skills
• Value of Investment assessment methodology
• Margaret Honey and Stuart Kahl P21
• Use of computer technology
• International perspective
• Measuring achievement
• Tim Leedy and Bianca Lochner Fountain Hills USD
• Costing, justifying and selling the project
• Open Discussion

4
Framework for 21st Century Learning
5
21st century student outcomes

• Core Subjects and 21st Century Themes
• Global Awareness
• Financial, Economic, Business and Entrepreneurial
  Literacy
• Civic Literacy
• Publication date 07/23/07 Health Literacy
• Learning and Innovation Skills
• Creativity and Innovation Skills
• Critical Thinking and Problem Solving Skills
• Communication and Collaboration Skills

6
21st century student outcomes, cont.

• Information, Media and Technology Skills
• Information Literacy
• Media Literacy
• ICT Literacy
• Life and Career Skills
• Flexibility Adaptability
• Initiative Self-Direction
• Social Cross-Cultural Skills
• Productivity Accountability
• Leadership Responsibility

7
Value of Investment

• The business of schools is education
• Value of Investment Total benefit of proposed
  project towards school mission or goals vs.
  Anticipated costs
• ROI (used by business) is only effective as it
  allows resources to be diverted to school mission
• Need to determine and measure qualitative
  benefits(Such as student achievement of
  important 21st century skills)
• Costs can be estimated
• Measuring benefits is more difficult

8
Why Measure Qualitative Benefits

• Evaluate proposed projects
• Projects competing for funding
• Sell a proposed project
• Specifically, what we expect to accomplish
• Succinctly describe or justify a project
• Specific terms
• Sustain a project
• Going back for continued funding

9
Methodology Determining Costs and Benefits

• Estimate project costs (Project TCO)
• Planning, curriculum, training, technology,
  support
• Adjust cost with savings, productivity
  enhancements, cost avoidance and additional
  revenue
• Measure (score) benefits
• Align with school mission, goals, mandates
• State benefits in measurable terms
• Effect of project on applicable goal(s) ( or -)
• Importance of affected goal
• Assess risk
• Weigh anticipated risk-weighted benefits vs.
  estimated net cost . . .
• Was it worth it? - Evaluate Results
• Actual costs
• Actual benefits

10
Achievement Stating the Benefits in Measurable
Terms

• Standards and AssessmentAssessment of 21st
  century skills should
• Support a balance of assessments, including
  high-quality standardized testing along with
  effective classroom formative and summative
  assessments
• Emphasize useful feedback on student performance
  that is embedded into everyday learning
• Require a balance of technology-enhanced,
  formative and summative assessments that measure
  student mastery of 21st century skills
• Enable development of portfolios of student work
  that demonstrate mastery of 21st century skills
  to educators and prospective employers
• Enable a balanced portfolio of measures to assess
  the educational systems effectiveness at
  reaching high levels of student competency in
  21st century skills

11
ICT Assessments

• ICT Higher Order Cognitive Abilities
• Critical thinking and problem solving
• Communication and presentation
• Interpersonal and self-direction
• Creativity and innovation

12
Australia ICT Performance Assessment

• Information and Communication Technology
  (ICT) Literacy Assessment
• Measures and reports on the ability of
  students to use ICT appropriately to access,
  manage, integrate and evaluate information,
  develop new understandings and communicate with
  others in order to participate effectively in
  society.
• MCEETYA Performance Measurement and
  Reporting Taskforce
• National Assessment Program Information
  and Communication Technology Literacy 2005 Years
  6 and 10 An Assessment Domain for ICT Literacy
• http//www.mceetya.edu.au/mceetya/default.a
  sp?id12183

13
Australia ICT Performance Assessment
14
NAEP Problem Solving in Technology-Rich
Environments

• Objective Last of 3 studies that explored use
  of
• new technologies in administering National
• Assessment of Educational Progress (NAEP).
• What makes this different is that it measures
• problem solving with technology.
• Problem Solving in Technology-Rich Environments
• A report from the NAEP Technology-Based
  Assessment Project, Research and
• Development Series, August, 2007
• US Department of Education - Institute for
  Education Sciences

15
NAEP TRE

• 8th Grade Level (13 years)
• 2 extended scenarios
• TRE Search scenario
• Students locate and synthesize information
• about scientific helium balloons from simulated
• Web environment.
• TRE Simulation scenario
• Students experiment to solve problems of
• increasing complexity about relationships
• among buoyancy, mass, and volume.

16
NAEP TRE

• Methodology - Evidenced Centered Design
• This methodology is used to develop an
• interpretive framework for translating multiple
• actions captured from each student into
• inferences about what populations of
• students know and can do.
• Mislevy, R.J., Almond, R.G., Lukas, J.F. (2003).
  
• A Brief Introduction to Evidenced-Centered Design
  (rr-03-16)
• Princeton, NJ Education Testing Service

17
NAEP TRE

• Guiding principles
• TRE should use the computer to do what cannot
  easily
• be done on paper.
• 2. TRE should represent the type of problem
  solving done
• with computers in educational and work
  environments.
• To the degree possible, TRE should allow the
• disentangling of component skills.
• TRE should be positioned so it can inform the
• development of a future assessment of emerging
  skills
• or of a more traditional subject matter.
• TRE should be an assessment, not instruction,
• but students should be able to learn from it
  incidentally.

18
UK KS3 ICT Assessment

• Build on 11 ICT Capabilities
• A. Identifying problems and defining tasks
• B. Searching and selecting information
• C. Organising and structuring information
• D. Analysing and interpreting information
• E. Combining and refining information
• F. Modelling
• G. Controlling events and devices
• H. Exchanging information
• I. Presenting information
• J. Reviewing, testing and evaluating
• K. Assessing the impact of ICT

19
UK KS3 ICT Assessment

• Redesigned as formative assessment
• Ten to fifteen minutes
• Automatically marked with onscreen reports that
  can be printed
• Teachers can access student reports
• Teachers can use tasks and reports to provide
  evidence of students progress and identify
  strengths and weaknesses

20
UK KS3 ICT Assessment

• Why the change?
• Problems with reliability (evaluation study by
  Andrew Boyle)
• Innovation is challenging and risky. Rather
  than table the project the formative strategy
  uses what was built and helps create human
  capacity
• Misalignment between teachers judgments of
  students competencies, test results, and limited
  numbers of students performing comfortably with
  the test standards
• Too much testing..

21
Selected Points from P21 White Paper on
Assessment
P21 Paper F.A. Characteristics
in Literature

• clarifying learning intensions/criteria
  (standards)
• rich information, elicit evidence of learning,
  during learning
• feedback to advance learning
• process used by teachers and students, activating
  students

• focus on 21st C skills
• make thinking visible, structured, real time,
  authentic
• data used to inform instruction
• build capacity of teachers and students

22
Route 21

• http//www.21stcenturyskills.org/route21

23
VOI MethodologyFountain Hills Unified

• A funding initiative for laptops for all high
  school students went to a vote in 2006 and failed
• Lack of focus on educational benefits of laptops
  was partially to blame for the failure
• No justification for expenses tied to educational
  outcomes
• District did not do its homework and communicate
  the value of the project to the community
• Analyze costs and anticipated benefits to the
  district
• Process forced us to define the plan and present
  objectives with cost data to support decision
• Quantifiable information for Board Members and
  constituents on student technology

24
Benefits of VOI Process for Fountain Hills Unified

• Channeled decision making thought processes into
  segments tied to goals of the project
• Maintain high student achievement levels as
  indicated by standardized test scores
• To attract new students from surrounding areas,
  home school students, and private/charter schools
• Provide students with 21st Century skills needed
  to be successful in college and the workforce
• Enabled administration to see and evaluate data
  that would have been overlooked
• Produced a report that verified total cost of
  ownership and supported decision to purchase
  fixed classroom computers vs. laptops

25
Quantitative Benefits that Affect Mission, Goals,
Mandates

• Student Achievement
• 21st Century Life Skills
• Enhanced Curriculum through Online Learning
• Increased Student Enrollment
• Enhanced Teacher Productivity and Retention

26
Open Discussion

• CoSNRich Kaestner richk_at_alyrica.netwww.cosn.org
  www.edtechvoi.org
• Partnership for 21st Century SkillsMargaret
  Honey mhoney_at_wgen.netStuart Kahl
  skahl_at_measuredprogress.orgwww.21stcenturyskills.o
  rg
• Fountain Hills USDTim Leedy tleedy_at_fhusd.orgBian
  ca Lochner blochner_at_fhusd.orgwww.fountainhillssch
  ools.org

27
Thank You CoSN TCO/VOI Project Sponsors!