Sarah Giersch

1
Supporting Meaningful Learningwith Online
ResourcesDeveloping a Review Process

• Sarah Giersch
• National Science Digital Library
• Heather Leary, Mimi Recker, Bart Palmer
• Utah State University

2
Introduction

• Content creation on the web - users
• Dissemination of web content
• Increased use of online resources in classrooms
• Educational Digital Libraries/Repositories
• Review user generated content for classrooms

3
National Science Digital Library (NSDL)

• http//nsdl.org
• Free online library for education and research in
  Science, Technology, Engineering, and Mathematics

4
Instructional Architect (IA)

• http//ia.usu.edu
• NSDL service
• Allows teachers to easily package online
  resources

• Professional

development workshops for teachers
5
Instructional Architect (IA)
6
Goals

• IA Goals
• Create a rubric to assess the quality of IA
  projects and resources
• Identify high quality projects and resources
• Establish reputation with users
• Provide further functionality to IA users in
  their search, discovery, and (re)use of existing
  IA projects
• Establish better teacher design capacity for
  career development and support of more meaningful
  learning opportunities in the classroom

7
Goals

• IA Goals cont.
• Capitalize on social networking technology to
  enhance the reviews and to encourage interaction
  around IA projects
• Provide a level of vetting for existing and new
  projects that has not been done to date
• NSDL Goals
• Establish a collection of peer-reviewed,
  teacher-designed IA projects to the NSDL with
  contextual metadata for better discovery and
  (re)use

8
Literature Review

• Search within computer science, library and
  information science, education (specifically
  online learning objects), and digital libraries
• Articles needed to included or reference rubrics
  for online educational resources
• Wanted to re-use other peoples work
• Noted motivation and process for creating rubric
  and how they generated reviews

9
Motivation, Process, Reviews

• Motivation to create a rubric
• Identify high quality resources
• Establish reputation of digital
  library/repository
• Collection development (inclusion/exclusion)
• Process for rubric creation
• Time consuming
• Identify stakeholders and users

10
Motivation, Process, Reviews

• Process for rubric creation cont.
• Test rubric with users for refinement
• Gather usability information
• How reviews are created
• Review boards
• From users (with and without incentives)

11
Methodology

• Identified 12 rubrics from lit review
• Yield of 200 review criteria
• No standard vocabulary for review criteria
• Definitions contained multiple ideas, not easily
  categorized
• Local criteria meaningless out of context from
  the site
• Narrowed to 104 through deductive reasoning
• Used card sort technique

12
Card Sort Analysis

• 9 participants sorted 104 review criteria into
  114 groups
• Developed 6 meta-categories sorted the groups
  to those
• Meta-categories
• Interface design accessibility
• Technical reliability
• Content
• Pedagogy
• Administrative
• Other
• Mapped criteria to meta-categories. see paper

13
Card Sorting
Snow, K., Ballaux, B, Christensen-Dalsgaard, B.,
etc al. (2008). Considering the user perspective
Research into usage and communication of digital
information. D-Lib Magazine, 14(5/6).
http//www.dlib.org/dlib/may08/ross/05ross.html
14
This chart shows how quickly the agreement among
study participants breaks down about how to
classify the 104 review criteria.
15
Next Steps

• Tried to be thorough in our re-use
• Suggest effort be put into standardized
  definitions of review criteria
• Doing a study (right now) with in-service
  teachers (online/f2f)
• Testing the feasibility of the rubric embedded in
  our PD workshop
• Does using it add to the teachers understanding
  and creation of quality online resources

16
What is Quality anyway?

• Quality
• Seems like you know it when you see it
• Buta guide for users who create content seems to
  produce higher quality and better standards
• Reality check
• Standardized rubric
• Support meaningful learning
• Does anyone have teacher-created content we can
  review?

17
Activity

• Why should we have review rubrics? (or should
  we?)
• What are your needs for evaluating online
  resources?
• IA Review Rubric http//dlconnect.usu.edu/image/r
  ubric.pdf
• Resources to evaluate
• http//ia.usu.edu/viewproject.php?projectia3740
• http//ia.usu.edu/viewproject.php?projectia4000
  
• http//ia.usu.edu/viewproject.php?projectia4018

18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
Activity Follow-up

• How meaningful was the rubric in terms of
  learning about quality?
• What did you learn?
• How can educator peer-review be sustainable?
• How can this rubric be re-used for your
  evaluation of online educational resources?

22
Contacts

• Sarah Giersch, National Science Digital Library
• sgiersch_at_gmail.com
• Heather Leary, Utah State University
• heatherleary_at_gmail.com