Educational Project Evaluation

1
Educational Project Evaluation
Integrating Simulation Technology in
Undergraduate Engineering

• Center for Evaluation and Assessment
• Donald B. Yarbrough, Director
• d-yarbrough_at_uiowa.edu

2
Overview

• Project Evaluation Definitions Purposes
• Evaluation Inquiry Modes/Methods
• ISTUE Project Descriptions
• ISTUE Evaluation Methods Findings
• Student Learning Theories Assessment
• ISTUE Future Evaluation Activities

3
Educational Project Evaluation Definition

• Evaluation as collaborative subproject of the
  ISTUE project
• Quality discovery development
• Monitoring reporting for improvement
• Documentation for accountability

4
Educational Project Evaluation Purposes

• Assessing Reporting Merit
• Assessing Reporting Worth
• Improvement/Betterment
• (Organizations, Projects, Learning processes,
  Outcomes)
• Oversight, Compliance, Accountability
• New Knowledge, Research Findings (Discovery,
  Translation, Dissemination)

5
Project Evaluation Inquiry Modes

• Description
• Categorization
• Causal Analyses
• Effectiveness Efficiency Analyses
• Values Inquiry

6
ISTUE Project Descriptions

• Needs, Purposes, Goals
• Resources, Activities, Procedures
• Outputs, Outcomes, Impacts

7
Project Objective

• Integrate simulation technology into
  undergraduate engineering curricula focused on
  fluid dynamics
• Variety of sub-purposes

8
Instruction and Practice

• Lectures on CFD and EFD methodology, standard
  procedures, UA
• Interfaces for academic use of commercial
  software and supporting software (Flowlab for
  Fluent)
• Student laboratory work and practice

9
Institutional Contexts

• Small, large, public, private, geographically-dist
  ributed IHE
• Faculty from A, M, C Engineering teaching
  different level courses
• Students from a wide variety of backgrounds with
  various entering skill sets, different
  motivations and cognitive preparation and
  different professional aspirations (workforce,
  graduate training)

10
ISTUE Project Evaluations

• Designs
• Methodologies
• Findings

11
Ideal Project Evaluation Design and Methodology

• Technically adequate measures aligned with goals
  and objectives
• Measures for both cognitive affective outcomes
  (students, faculty, administrators, other
  decision makers)
• Outcomes Impact variety of randomly selected
  and assigned field test sites (students,
  instructors, institutions)
• Comparative design for efficiency analysis
• Cost and resource monitoring

12
Goals for ISTUE Project Evaluation

• Design and plan good initial evaluations (proper,
  feasible, accurate, useful)
• Implement and report the good-enough evaluation
  at demonstration sites
• Seek resources to conduct a more thorough
  evaluation in timely fashion

13

Evaluation Resources

• There are never enough resources to answer all
  worthwhile evaluation questions or to serve all
  evaluation purposes thoroughly. We must choose
  those questions and purposes that are valuable
  enough to justify the resources involved.

14
Selected Evaluation Questions

• What do students learn and how can their learning
  be improved?
• What can be done to improve the teaching
  interfaces templates, instruction, classroom
  and lab experiences?
• How can different students and different levels
  of learning be best accommodated?

15
Selected Designs Methods

• Formative Summative Purposes
• Surveys
• Lab Reports
• Achievement Tests (Y3, UI, Outcome Design)

16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
Theories of Student Learning and Assessment

• Project Impact Theory
• Student Evaluation Standards
• Overlap of Student and Project Evaluations

20
Project Impact Assumptions/Theories

• Theories of Engineering Student Learning Skill
  Acquisition
• Cognitive Science Theories about Expertise Its
  Development
• Fluid Dynamics and Its Role in Engineering
  Knowledge Professional Practice
• Theories about the Roles and Value of Simulation
  Technology

21

• How to evaluate the processes that students
  experience as they learn the desired skills and
  knowledge bases
• How to make those processes more effective and
  efficient for the varieties of students we serve

22
Student Evaluation Standards

• Propriety
• Utility
• Feasibility
• Accuracy
• Student evaluation/assessment in the service of
  effective and efficient student learning.

23
Instructional Designers and Instructors Duties
for Learner Outcomes
Giving Feedback
Selecting Worthy Learner Outcomes
Investigating Changes in Learners
Communicating the Selected Learner Outcomes
Implementing the Interface in the Context of
Supportive Curriculum Instruction
24
Evaluation Duties for Learner Outcomes
Evaluating the Informal and Formal Reporting of
Actual Outcomes
Evaluating the Selected Learner Outcomes
Second Order Metaevaluation
Evaluating the Communication about Intended
Outcomes
Evaluating the Actual Learner Outcomes
Evaluating the Interface in the Context of
Curriculum and Instruction
25
ISTUE Future Evaluation Activities