Interactive Learning Systems Evaluation

1
(No Transcript)
2
Six Facets of Instructional Product Evaluation
Review
Maintenance
Needs Assessment
Formative
Impact
Effectiveness
3
Development Activities
Evaluation Functions
Product Conceptualization
Design
Development
Implementation
Institutionalization
Project Re-conceptualization
4
EMCC Design Document

• Urban Science Course Environment

5
Dimensions of effective technology enhanced
learning environments

• Task-Oriented
• Challenging
• Collaborative
• Constructionist
• Conversational
• Responsive
• Reflective
• Formative

6
Task-Oriented
Academic
Authentic

• The tasks faculty set for students define the
  essence of the learning environment. If
  appropriate, tasks should be authentic rather
  than academic.

7
Task-Oriented Example

• Students in online instructional design courses
  are tasked with designing interactive modules
  for real clients.

8
Challenging
Simple
Complex

• The notion that interactive learning is easy
  should be dispelled. Learning is difficult and
  students should not spoon fed simplified versions
  of their fields of study.

9
Challenging Example

• In a Masters of Public Health program, students
  confront problems as complex and difficult as the
  ones theyll face in the real world.

10
Collaborative
Unsupported
Integral

• Web-based tools for group work and collaboration
  can prepare students
• for team work in
• 21st Century
• work environments.

11
Collaborative Example

• Art, dance, and music students are collaborating
  to produce online shows with digital versions of
  their works and performances for critique by
  international experts.

12
Constructionist
Replication
Origination

• Faculty should engage students in creating
  original knowledge representations that can be
  shared, critiqued, and revised.

13
Constructionist Example

• Students in fields ranging from aero-engineering
  to zoo management are producing digital
  portfolios as integral components of their
  academic programs.

14
Conversational
One-way
Multi-faceted

• Students must have ample time and secure spaces
  for in-depth discussions, debates, arguments, and
  other forms of conversation.

15
Conversational Example

• New knowledge and insight are being constructed
  in conversation spaces such as the e-learning
  forums found in BlackBoard, WebCT,
  Desire2Learn,and other online learning
  authoring platforms.

16
Responsive
Superficial
Genuine

• In learning communities, both faculty and
  students have a mutual responsibility to respond
  quickly, accurately, and with respect.

17
Responsive Example

• This is an area where RD are needed. Some
  universities are seeking to establish supportive
  online networks that will continue throughout a
  career, indeed throughout a life.

18
Reflective
Shallow
Deep

• Both faculty and learners must engage in deep
  reflection and metacognition. These are not
  instinctive activities, but they can be learned.

19
Reflective Example

• Teacher preparation students are keeping
  electronic journals to reflect upon the children
  they teach, and their roles as advocates for
  children.

20
Formative
Fixed Assessment
Developmental

• Learning environments can be designed to allow
  students to develop prototype solutions over time
  rather than to find one right answer that someone
  else has defined.

21
Formative Example

• Faculty should engage their students in ongoing
  efforts to evaluate and refine their work related
  to authentic tasks to encourage lifelong learning.

22
Traditional Course
Online Course

• Task-Oriented
• Challenging
• Collaborative
• Constructionist
• Conversational
• Responsive
• Reflective
• Formative

23
Heuristic Review
24
What is usability?

• The concern with designing software applications
  which people find easy to use and personally
  empowering.
• Usable computer programs are logical, intuitive,
  and clear to the people who use them.

25
Web Site Usability

• The most common user action on a Web site is to
  flee.
• at least 90 of all commercial Web sites are
  overly difficult to use..the average outcome of
  Web usability studies is that test users fail
  when they try to perform a test task on the Web.
  Thus, when you try something new on the Web, the
  expected outcome is failure.

Edward Tufte
Jakob Neilsen
26
Typical Web Usability Problems

• bloated page design
• internally focused design
• obscure site structures
• lack of navigation support
• writing style optimized for print

Jakob Neilsen
http//www.useit.com/
27
Key Usability Principles

• Structure - organize meaningfully
• Simplicity - make common tasks easy
• Visibility - all data needed for a task
• Feedback - keep users informed
• Tolerance - allow cancel, back
• Reuse - reduce the users' need to remember

28
Nielsens Web Usability Rules

• Visibility of system status
• Match between system and real world
• User control and freedom
• Consistency and standards
• Error prevention
• Recognition rather than recall
• Flexibility and efficiency of use
• Help users recognize, diagnose, and recover from
  errors
• Help and documentation
• Aesthetic and minimalist design

29
Two Major Ways to Evaluate Usability

• Heuristic Review
• quick and relatively inexpensive
• based on expert analyses
• no user involvement
• Usability Testing
• finds more problems
• user involvement increases validity
• when designers see problems live, it has a huge
  impact

30
Heuristic Review

• Several experts individually compare a product to
  a set of usability heuristics

• Typical heuristic
• Visibility of system status
• The system should always keep users informed
  about what is going on, through appropriate
  feedback within reasonable time.

31
Heuristic Review

• Violations of the heuristics are evaluated for
  their severity and extent

Severity Scale 1 Cosmetic fix if possible. 2
Minor fixing this should be given low
priority. 3 Medium fixing this should be given
medium priority. 4 Major fixing this should be
mandatory before the system is launched. If the
problem cannot be fixed before launch, ensure
that the documentation clearly shows the user a
workaround. 5 Catastrophic fixing this is
mandatory no workaround possible. Extensiveness
Scale 1 Single case 2 Several places 3
Widespread
32
Heuristic Review

• At a group meeting, violation reports are
  categorized and assigned

• Heuristics violated are identified
• Average severity and extensiveness ratings are
  compiled
• Opportunities for improvement are clarified
• Feasible solutions are recommended

33
Heuristic Review

• Example of Opportunity For Improvement

34
Heuristic Review

• Advantages

• Quick Do not need to find or schedule users
• Easy to review problem areas many times
• Inexpensive No fancy equipment needed

• Disadvantages

• Validity No users involved
• Finds fewer problems (50 less in some cases)
• Getting good experts can be challenging
• Building consensus with experts is sometimes
  difficult

35
Another Weakness

• Some people believe that heuristic evaluation is
  too subjective.
• Human judges are prone to poor judgment at times.

36
Usability Standardshttp//www.astd.org/ASTD/marke
tplace/ecc/ecc_home

• ASTD offers certification of e-learning courses,
  including 8 usability standards
• Navigation
• Orientation
• Feedback cues
• Link cues
• Links labeled
• Help
• Legibility
• Text quality

37
Heuristics for E-Learning Evaluation

• 1. Visibility of system status The e-learning
  program keeps the learner informed about what is
  happening, through appropriate feedback within
  reasonable time.

• red for a problem
• yellow for a warning
• green for OK

38
Heuristics for E-Learning Evaluation

• 2. Match between system and the real world The
  e-learning programs interface employs words,
  phrases and concepts familiar to the learner or
  appropriate to the content, as opposed to
  system-oriented terms. Wherever possible, the
  e-learning program utilizes real-world
  conventions that make information appear in a
  natural and logical order.

39
Heuristics for E-Learning Evaluation

• 3. Error Recovery and Exiting The e-learning
  program allows the learner to recover from input
  mistakes and provides a clearly marked exit to
  leave the program without having to go through an
  extended dialogue.

40
(No Transcript)
41
(No Transcript)
42
Heuristics for E-Learning Evaluation

• 4. Consistency and standards When appropriate to
  the content and target audience, the e-learning
  program adheres to general software conventions
  and is consistent in its use of different words,
  situations, or actions.

43
Heuristics for E-Learning Evaluation

• 5. Error prevention The e-learning program is
  carefully designed to prevent common problems
  from occurring in the first place.

44
Heuristics for E-Learning Evaluation

• 6. Navigation support The e-learning program
  makes objects, actions, and options visible so
  that the user does not have to remember
  information when navigating from one part of the
  program to another. Instructions for use of the
  program are always visible or easily retrievable.

45
Heuristics for E-Learning Evaluation

• 7. Aesthetics Screen displays do not contain
  information that is irrelevant, and bells and
  whistles are not gratuitously added to the
  e-learning program.

46
(No Transcript)
47
(No Transcript)
48
Heuristics for E-Learning Evaluation

• 8. Help and documentation The e-learning program
  provides help and documentation that is readily
  accessible to the user when necessary. The help
  provides specific concrete steps for the user to
  follow. All documentation is written clearly and
  succinctly.

49
Heuristics for E-Learning Evaluation

• 9. Interactivity The e-learning program provides
  content-related interactions and tasks that
  support meaningful learning.

50
Heuristics for E-Learning Evaluation

• 10.Message Design The e-learning program
  presents information in accord with sound
  principles of information-processing theory.

51
(No Transcript)
52
(No Transcript)
53
Heuristics for E-Learning Evaluation

• 11.Learning Design The interactions in the
  e-learning program have been designed in accord
  with sound principles of learning theory.

54
Heuristics for E-Learning Evaluation

• 12.Media Integration The inclusion of media in
  the e-learning program serves clear pedagogical
  and/or motivational purposes.

55
Heuristics for E-Learning Evaluation

• 13.Instructional Assessment The e-learning
  program provides assessment opportunities that
  are aligned with the program objectives and
  content.

56
Heuristics for E-Learning Evaluation

• 14.Resources The e-learning program provides
  access to all the resources necessary to support
  effective learning.

57
Review

• The purpose of review is to ensure that the
  development team is well-informed about previous
  work done in the area during the early stages of
  product conceptualization.
• Designers must avoid recreating the wheel.

58
Review
I can do better than this!

• The two primary methods used are reviewing the
  related literature and reviewing competing
  products.
• Regularly reviewing competing products is a great
  professional development practice.

59
Needs Assessment

• The purpose of needs assessment is to identify
  the critical needs that an instructional product
  is supposed to meet.
• Needs assessment provides essential information
  to guide the design phase of the development
  process.

60
Needs Assessment

• The primary methods are
• task analysis,
• job analysis, and
• learner analysis.
• One of the most important results is a list of
  specific goals and objectives that learners will
  accomplish with the new product.

61
Formative Evaluation

• The purpose is to collect information that can be
  used for making decisions about improving
  interactive learning products.
• Formative evaluation starts with the earliest
  stages of planning and continues through
  implementation.

62
Formative Evaluation

• Provided the results are used, formative
  evaluation usually provides the biggest payoff
  for evaluation activities.
• Clients may be reluctant to accept the results of
  formative evaluation, especially as a program
  nears completion.

63
Effectiveness Evaluation

• The purpose is to estimate short-term
  effectiveness in meeting objectives.
• It is a necessary, but insufficient, approach to
  determining the outcomes of interactive learning.

64
Effectiveness Evaluation

• Evaluating implementation is as important as
  evaluating outcomes.
• If you dont understand how instructional
  products were actually implemented, you cant
  interpret results.

A connection with the server could not be
established?
65
Impact Evaluation

• The purpose is to estimate the long-term impact
  on performance, both intended and unintended.
• It is extremely difficult to evaluate the impact
  of interactive learning products.

66
Impact Evaluation

• Evaluating impact isincreasingly critical
  because of emphasis on the bottom line.
• More and more clients expect impact evaluation to
  include return-on-investment (ROI) approaches.

67
Maintenance Evaluation

• The purpose of maintenance evaluation is to
  ensure the viability of an interactive product
  over time.
• Maintenance is one of the weakest links of
  web-based learning environments.

68
Maintenance Evaluation

• Document analysis, interviews, observations, and
  automated data collection are among the methods
  used in maintenance evaluation.
• Very few education and training agencies engage
  in serious maintenance evaluation.

69
Planning is the key to successful instructional
product evaluation.

• Evaluation requires good planning, careful
  implementation, and systematic follow-up.
• A major challenge is getting clients to identify
  the decisions they face.
• Clear decisions drive the rest of the planning.

70
Heuristics for E-Learning Evaluation

• 15.Feedback The e-learning program provides
  feedback that is contextual and relevant to the
  problem or task in which the learner is engaged.

What the heck?