Title: Quality Matters: Peer Review of Online Courses
1Using the Quality Matters Rubric to Improve
Online Coursesed bowen, Executive Dean Distance
Learning, Dallas TeleCollege
This session will address how faculty and
administrators could use the Quality Matters
rubric to improve their online course(s).
Participants will become familiar with the rubric
tool developed by the nationally recognized,
FIPSE-funded Quality Matters (QM) project. The QM
rubric provides a research-supported framework
with annotations and examples for applying
quality practices to specific course design
standards.
2Why Adopt
What's Involved
Implementation Strategies
3Quality Matters Is a Process A Set of Standards
4Quality Matters Peer Course Review Process
Faculty Course Developers
Institutions
National Standards Research Literature
Course
Rubric
Faculty Reviewers
Training
Peer Course Review
Feedback
Blueprint for improvement
Instructional Designers
5Underlying Principles of QM
- The QM toolset and process are
- based in national standards of best practice, the
research literature and instructional design
principles - designed to promote student learning
- integral to a continuous quality improvement
process - part of a faculty-driven, peer review process
- Course does not have to be perfect but better
than just good enough. (Standards met at about
85 level or better.)
6Underlying Principles of QM Continued
- Process designed to ensure all reviewed courses
will eventually meet expectations - Collegial review process, not an evaluation
process - Review team must include an external peer
reviewer - Course faculty or instructor considered part of
the review team
7Whats In It For Institutions
- External validation
- Strengthen accreditation package
- Raise QA as a priority activity
- Access to a sustainable, replicable, scalable QA
process - Inform online course training practices
- Provide professional development activities
8Whats In It For Faculty
- Improve online courses
- External quality assurance
- Expand professional community
- Review other courses and gain new ideas for own
course - Participation useful for professional development
plan and portfolio - Receive 150 for each completed peer course review
9(No Transcript)
10Why Adopt
What's Involved
Implementation Strategies
11Quality MattersPeer Course Review Process
1. Course Selected
2. Trained Review Team Assigned
3. MOUs and Instructor Worksheet
4. Pre-Review Discussion
5. Individuals complete reviews
6. Team Discussion(s)
7. Final Review Report
12Quality Matters Peer Course ReviewerCertification
Process
Quality Matters Certification
Course Review Experience
Training
- Name on QM website
- Use of QM Logo
- Eligible for Peer Reviewer Pool
Assigned to Peer Review Team
Attend QM Training
Submit Report
Demonstrate Competencies
Trained
Certified
Kane 1/15/05
13The Rubric
- Eight general standards
- Course Overview and Introduction
- Learning Objectives
- Assessment and Measurement
- Resources and Materials
- Learner Interaction
- Course Technology
- Learner Support
- Accessibility
Key components must align.
14Key Standards That Must Align
- Objectives
- Standard II.1 Measurable outcomes
- Standard II.2 Content mastery, critical
thinking, learning skills addressed. - Assessment and Measurement
- Standard III.1 Measures objectives consistent
with learning activities - Learner Interactions and Activities
- Standard V.1 Help students achieve the
objectives - Course Materials
- Standard IV.1 Deep and comprehensive enough for
students to achieve the objective - Course Technology
- Standard VI.1 Tools and media support the
objectives
15General Standard ICourse Overview and
Introduction
- 1.1 Navigational instructions make the
- organization of the course easy to
- understand.
- 1.2 A statement introduces the student to the
- course and to the structure of the student
- learning, and, in the case of a hybrid
- course, clarifies the relationship between
- the face-to-face and online components.
16Rubric Scoring
- Team of three (3) reviewers
- One score per standard based on team majority
- Assigned point value not sliding scale
17The Peer Review Team
- 3 Faculty Peer Reviewers
- All must be experienced online instructors
- All must attend QM training
- One MUST be external to the course developers
institution - One must be a subject matter expert (SME) The SME
could also be the external reviewer. - AND
- Faculty Course Developer
- access to rubric prior to review
- involved in pre-review discussions
- consulted during review
18Faculty Developer
- Part of the review team.
- Provides access to the course.
- Completes Instructor Worksheet
- Part of the initial team discussion
- Receives compiled report
- Returns Faculty Response Form
19Peer Reviewers
- Establish Team Calendar
- Review the course individually
- Complete the online web review form
- Discuss review with Team as needed
- Complete an Exit Interview
- Receive
- Recognition as Certified Peer Reviewer.
- Compensation
20Team Chair
- Reviewer Roles Responsibilities plus.
- Organizes Team calendar
- Confirm Instructor Worksheet is used
- Creates draft report from compiled reviews
- Convenes Team discussions
- Reviews, edits and submits Team Report
- Receives
- Recognition as Certified Peer Reviewer and Chair.
- Compensation
21Overall Course Review Results
- Upon initial review
- 51 meet expectations
- 19 do not meet expectations - missing at least
one essential 3-point element(s) - 30 do not meet expectations - missing at least
one essential 3 point element(s) and a minimum of
68 points
22Why Adopt
What's Involved
Implementation Strategies
23Range of Implementation Options
Formal QM Peer Review
Internal QM Peer Review
Internal Review
Dean Review
Colleague Review
Cost
Self Administered
24(No Transcript)
25- Additional Optional Services and Fees
- Independent (Subscriber-Run) Peer Reviews
- Subscribers may manage their own MOL-recognized
Peer Reviews after training. Institutions are
responsible for selecting and managing peer
reviewers per Quality Matters program guidelines
and for compensating peer reviewers. Subscribers
are given access to the Quality Matters Rubric
for the review team and recognition in the
Quality Matters Registry of courses that meet
standards. The annual subscriber fee includes
registration of up to 30 independent reviewed
courses for Large Institutions, 20 for Mid-Sized
Institutions, and 10 for Small Institutions.
(Subscribers pay a 50 per-course registration
fee for any additional courses.) - To be eligible for recognition in the Quality
Matters Registry, an on-line course must have
successfully completed a subscriber-run review
that has met all Quality Matters guidelines
(which may or may not involve additional costs to
the Institution). - Guidelines include
- Review managed by a QM-trained Institutional
Representative (one seat in training included in
annual subscriber fee) - Review completed by a review team of three
Qualified Quality Matters-Trained Reviewers. Team
make-up must include a reviewer from outside the
institution a reviewer trained as a Quality
Matters Master Reviewer and a reviewer assigned
as team chair. (A single reviewer may fill one or
all of these roles.) - Reviewers used the Quality Matters materials and
followed the Quality Matters process (provided
electronically to subscribers as part of annual
subscriber fee) - Upon successful completion of review and
submission of documents to MOL, course
institution has paid 50 per course for each
registration over those included in annual
subscriber fee - MOL provides access to list of trained Peer
Reviewers to subscribers. Subscribers must
negotiate the peer reviewer stipends paid if any.
(MOL pays 150 per reviewer, 250 to team chair)
26- MOL-Managed Peer Reviews
- MOL will manage the course review process for
subscribing and non-subscribing institutions on a
course-by-course basis. This service includes
Peer reviewers assigned, managed, and paid by
MOL compiled final report and recognition of a
course in the Quality Matters Registry upon its
meeting Quality Matters program review standards. - Quality Matters Program Subscribers
750/course - Non-Subscribers
1,000/course
27Range of Implementation Options
Formal QM Peer Review
Internal QM Peer Review
Internal Review
Dean Review
Colleague Review
Cost
Self Administered
28Multiple Uses of QM
- Guidelines for initial online course development
- Quality assurance of existing courses
- Ongoing faculty professional development
- Institutional reaccredidation packages
- Formation of distance learning policies
steering committees