Measuring the Effectiveness of Online Programs

1
Measuring the Effectiveness of Online Programs

Liz Pape, CEO Virtual High School www.goVHS.org
2
Todays Discussion

• Need for online program effectiveness measures
• Sample of measures in use
• Discussion of online ed. terms is there a
  common language?

3
About Virtual High School (VHS)

• A non profit collaborative of 300 high schools
  offering innovative online courses
• Currently in 9th year of online course delivery
• 10M and 5 years of RD effort

4
VHS Thought Leadership

• The Virtual High School Teaching Generation V,
  Andrew Zucker, Robert Kozma
• Guide to Online High School Courses
  (www.nea.org/technology/onlinecourseguide.html)
• Virtual Schooling Issues in the Development of
  E-Learning Policy, Donovan R. Walling (editor)
• Virtual Realities A School Leaders Guide to
  Online Education, Ismat Adbal-Haqq (editor)
• Virtual Schools Planning for Success, Tom Clark,
  Zane Berge, editors (April, 2005)

5
VHS Practice Leadership

• Blackboard Course Content Award 2005
• USDLA 21st Century Best Practices Award 2005
• USDLA Best Practices K-12 Course
• NCTET Community Builder Award 2004
• Stockholm Challenge 2001 Award

6
What can online students learn?The Three Cs on
Online Education

• Content/curriculum
• Access to courses not otherwise available
• Learning with technology, instead of about
  technology
• Information literacy skills
• Online collaboration
• Online teams and group projects
• Multimedia presentation skills
• Critical thinking, project-based learning
• Global citizenship skills
• Extending the learning experience beyond the
  classroom walls
• Online discussions with students of different
  cultures

7
Evaluating your Online Program

• Why measure?
• What is measured?
• Mission statement
• Annual goals
• Strategic Plan
• How is it measured?
• Hard data points
• Self-reported data
• When?
• Continuous improvement loop

8
Sample Online Program Measures of Effectiveness
9
Mission and Focus

• The VHS mission is to develop and deliver
  standards-based, student-centered online courses
  to expand students educational opportunities and
  21st century skills and to offer professional
  development to teachers to expand the scope and
  depth of their instructional skills
• The focus includes
• Effective online course design and delivery
• Scheduled
• Asynchronous
• Instructor-led
• Professional development for online teachers
• Quality control/quality assurance
• Course design/delivery standards (Guide to Online
  High School Courses)
• Teacher mentoring
• On-going professional development
• Infrastructure for course delivery
• Online Grading System
• Online Registration
• 21st Century Skills development for Millennial
  Generation students

10
Goals the first place to start

• Program growth goals students, courses, schools
• Quality goals course quality, membership
  services, preparation of teachers

11
VHS SCHOOLS
12
VHS COURSES
13
VHS STUDENTS
14
Measuring Effectiveness

• Course Quality
• Professional Development
• Program Services

15
VHS Performance Metrics-Course Quality

• AP pass rates
• Course completion rates
• of students taking AP exams
• Credit recovery rate
• of external evaluations

16
VHS Surveys-Course Quality

• Satisfaction with VHS course quality
• 92 superintendents
• 100 principals
• 98 Site Coordinators
• Course features, teacher satisfaction
• 92 engagement in course work
• 78 collaborative learning
• 97 21st century skills development

17
VHS Performance Metrics-Professional Development

• Professional development grad rate
• 1st-semester teacher success rate
• of teachers remaining with FA
• of external evaluations

18
VHS Surveys-Professional Development

• Satisfied with VHS professional development
• 99 principals
• Applying VHS pedagogy to f2f teaching
• 79 teachers

19
VHS Performance Metrics-Program and Services
Quality

• Membership retention rates
• Seat utilization
• of external evaluations

20
VHS Surveys-Program and Services Quality

• Satisfaction with electronic communications
  email
• Teachers 99, SCs 96
• Website
• Teachers 97, SCs 98
• Newsletter/Annual Report
• Superintendents 86/68
• Principals 78/79
• Teachers 79/54
• SCs 84/54

21
Terminology

• What do we really mean when we say
• Course completion rate
• AP exam participation rate
• Drop-out rate
• Student interactions/online discussions
• Online teacher
• Skilled online teacher
• Effective online course
• Highly interactive online course

22
Are online students learning and engaged?
23
Thank You!

• Liz Pape, CEO, Virtual High School
• LPape_at_goVHS.org
• 978.450.0411