The Internet Master Evaluation Process

1
The Internet Master Evaluation Process

• Who, what, when, where, why,
• and how

2
Goals of this session

• Why evaluate?
• Instruments
• Process
• Data entry and analysis
• Samples of reporting

3
What this session is not

• Educational statistics class
• The mechanics of using
  SPSS for the PC

4
Why evaluation?

• Formative - to do a better job next time
• feedback on how to improve AND what you have done
  well
• Summative
• accountability to the funder
• accountability to your system
• personal satisfaction

5
For Formative Feedback

• I learned
• I relearned
• I now need to
• I need to tell you that
• (In a pinch, this will let you do fairly good
  reporting. i.e. 100 of the participants had
  written goals at the conclusion of the workshop.)

6
The Primary Focus of the IM Evaluation is
Summative

• Short term outcomes (Knowledge and behavior
  change)
• knowledge and use of Internet
• knowledge and skills to facilitate volunteer
  community education
• Longer term impact (Behavior change and
  consequences to society)
• volunteer hours
• people reached

7
Levels of Evaluation

• 1. Awareness

8
Levels of Evaluation

• 2. Attitudes
• 1. Awareness

9
Levels of Evaluation

• 3. Knowledge change
• 2. Attitudes
• 1. Awareness

10
Levels of Evaluation

• 4. Behavior change
• 3. Knowledge change
• 2. Attitudes
• 1. Awareness

11
Levels of Evaluation

• 5. Consequences to society
• 4. Behavior change
• 3. Knowledge change
• 2. Attitudes
• 1. Awareness

12
Levels of Evaluation

• 5. Consequences to society
• 4. Behavior change
• -----------------------------------
• 3. Knowledge change
• -----------------------------------
• 2. Attitudes
• 1. Awareness

13
Our 1 Recommendation

• Make it someones job in your state to sweat the
  details about Internet Master evaluation
• Best if it is someone involved, but not the
  central mover and shaker for programming
• Our model - for field staff by field staff

14
Whether you are the state evaluation person, or
you will be supportive of that person, it is
important to know the evaluation methodology
15
How does the Internet Master Evaluation Shop
operate in Missouri?
16
The Evaluation Process

• Repeated Measures Design - comparing the
  responses of the learners. The same exact
  questions are asked
• beginning of class 1
• conclusion of class 10
• conclusion of volunteer community education
  (approximately one year after concluding class)
• Copies of the instruments in the green OSU folder
  (you may photocopy)

17
Additional data

• LTM instrument - learning/teaching styles,
  watching/doing scale, some demographics
• Additional data - collected at session 10
• The data collection is both qualitative (open
  ended questions) and quantitative. The analysis
  focuses on the quantitative data.

18
Details

• If someone misses the 1st or the 4th sessions
  should I have them fill out the forms they
  missed? Yes.
• If someone misses the 10th session do I have to
  track them down to get the 10th session
  evaluation filled out? Yes.
• Should we identify people by name or code for the
  evaluation?

19
We want to do our own evaluation thing

• Can we tweak the questions in the repeated
  measures forms? NO!
• The idea is to add up the Internet Master
  programming for multiple states.
• Also have the potential of adding it to Minnesota
  and Missouri
• You can add questions!

20
The Data Analysis

• We have been pleased with SPSS for the PC
  -particularly for someone without access to a
  mainframe
• Requirements for PC
• 486 PC or better
• 12 MB RAM minimum, 16 better
• Windows 95 or Windows NT 3.51 or 4.0
• 48 MB hard disk space to install

21
Steps in Data Analysis Process

• Final data collection
• conclusion of volunteer community education
• one year after concluding class
• again at 1 1/2 years
• offer incentives?
• Color printers
• mouse pad
• ???
• Final class analysis
• Later, can add up impact with other classes/states

• Enter data as it is collected during classes -
  Session 1, LTM, Session 10
• Can do analysis and reporting at this point

22
How we set up a template to enter data from
classes in Missouri...
23
Analysis and Reporting

• Narrative report (word processing)
• Related analysis (SPSS for the PC)