FacultyIndustry Collaborations for Assessing Student Learning

1
Faculty-Industry Collaborations for Assessing
Student Learning

• Flora McMartin
• University of California - Berkeley
• Jack McGourty
• Columbia University

2
Why Involve Industry?

• Insures what is assessed is valued
• Strengthens accountability
• Addresses accreditation mandate

3
Campus-Industry Relationship
ACE Campus Trends Survey - 1996
4
Synthesis Coalition
Cornell
Iowa State
Berkeley
Hampton
Stanford
Northern Arizona
Cal Poly
Tuskegee
Southern

• Synthesis Coalition Goals
• Institutionalize reforms
• Assess disseminate reforms
• Develop NEEDS
• Institutionalize K-12 outreach

5
Gateway Coalition
- Drexel - Cooper - USC - Ohio State - Academic
Associates - Polytechnic - NJIT - Columbia

• Major Gateway Goals
• Curriculum
• Assessment
• Underrepresented Populations
• Instructional Technology
• Professional Development
• Linking Sharing

6
Assessment Components

• Stakeholder participation (faculty, students,
  industry)
• Faculty assessment training program
• Campus (local) control of assessment process
• Flexible, valid, and reliable tools

7
Assessment Goals

• A. Involve stakeholders to measure effects of
  coalition program reforms
• B. Develop/promote valid reliable performance
  assessment tools
• C. Promote the institutionalization of
  assessment at participating campuses
• D. Satisfy needs of individual faculty,
  colleges, coalition program evaluation
• E. Provide a foundation for ABET accreditation

8
ABET 2000 Learning Outcomes

• 1. math, science engineering knowledge
• 2. experiment, analyze interpret data
• 3. design a system, component or process
• 4. function on multi-disciplinary teams
• 5. identify, formulate solve engineering
  problems
• 6. understand professional ethics
• 7. communicate effectively
• 8. understand global/societal impact of
  engineering
• 9. engage in life-long learning
• 10. knowledge of contemporary issues
• 11. use modern engineering techniques, tools
  skills

9
Synthesis Case Study
10
Student Learning Outcomes

• Open Ended problem solving
• Teamwork
• Multi-disciplinary design
• Hands-on facility with hardware
• Communication skills - (oral, written,
  interpersonal)

11
Goals for Involving Industry

• Determine common learning outcomes valued by
  industry engineering education
• Articulate student learning outcomes in a common
  language
• Bridge industry engineering educational needs
• Energize the assessment process through active
  participation of stakeholders

12
Mechatronics Industrial Board

• Berkeley Process Control Motorola
• Maytag Rockwell International
• Xerox PARC Ford
• Raychem Hewlett-Packard

13
The MIB, continued

• Membership
• primarily project engineers
• observed needs vs. self reports
• Role
• define what industry values in educational
  outcomes
• try out review assessment tools
• try out review assessment rubrics measures
• evaluate assessment results

14
Planning Process Track 1
ALL stakeholders define articulate Student
learning outcomes (SLOs) classroom activities
related to Synthessis goals
Identifying Learning Outcomes
SLOs reduced refined
SLOs reviewed by ALL stakeholders - draft of
Synthesis Assessment Framework (SAF)
Draft SAF compared to scenario activities
Overlap between SLOs activities MOST VALUED
OUTCOMES Final version of SAF
15
Track 1 Learning Outcomes

• Industry

Engineering Ed.
Most valued student learning outcomes
Synthesis Assessment Framework
16
Planning Process Track 2
MIB writes scenarios
Validating the Outcomes
Scenarios edited, reviewed by MIB for accuracy
Scenarios analyzed to ID activities related to
SLOs
Draft SAF compared to scenario activities
Overlap between SLOs activities MOST VALUED
OUTCOMES Final version of SAF
17
Track 2 Validating with Scenarios

• Scenario
• a description of common or critical situations
  faced by an engineer
• actual past, present, or future situation
  experienced by the MIB member
• includes the context and actions of the engineer

18
Validation

• approximately 80 overlap
• Differences
• ability to estimate resources (costs, time)
• ability to supervise others
• ability to take risks
• ability to deal with ambiguity
• ability to decide if project is worth pursuing

19
Assessment Planning Process
Track 1 Identification
Track 2 Validation
ALL stakeholders define articulate Student
learning outcomes (SLOs) classroom activities
related to Synthesis goals
MIB writes scenarios
Scenarios edited, reviewed by MIB for accuracy
SLOs reduced refined
Scenarios analyzed to ID activities related to
SLOs
SLOs reviewed by ALL stakeholders - draft of
Synthesis Assessment Framework (SAF)
Draft SAF compared to scenario activities
Overlap between SLOs activities MOST VALUED
OUTCOMES Final version of SAF
20
Assessment Tools
21
Scenario-Based Assessment

• Assignment developed from MIB scenarios
• Scoring rubric based on SAF
• Assignment rubric tested by MIB
• MIB responses established baseline expert score
• MIB analysis of rubric used to refine measurement
  criteria

22
Scenario Assessment Process
Faculty administer pre post test

• Syn. HQ assess
• holistic
• analytic

• Faculty evaluate
• diagnostic
• course grade

faculty revise course
dept/col - revise curriculum Syn. - longitudinal
study
23
Synthesis - Lessons Learned

• Collaboration created a bridge between industry
  faculty regarding engineering education
• Clarified what is/is not possible to teach
  learn
• validated measurable outcomes
• Identifying articulating learning outcomes
  valued by ALL stakeholders built commitment to
  the assessment process results.
• Vitalized assessment process for all stakeholders

24
Gateway Case Study
25
NJIT/NCE Assessment Statement
Institute an assessment plan and process that
moves from the current anecdotally based
assessment to a formal, rigorous, valid, and
useful assessment and continuous improvement
process.
26
NJIT/ NCE Industrial Advisory Board
Lucent Technologies
PSEG
Foster Wheeler
PrimeMedia
SIAC
Raytheon
US Army
Curtiss Wright
CompUSA
Becton Dickinson
27
Industrial Sub-Committees
NJIT/NCE Advisory Board
Strategic Planning
Curriculum Review
Assessment
Gateway National Visiting Committee
28
Industrial Assessment Committee

• Charter
• To support the development of a comprehensive
  qualitative and quantitative curriculum
  assessment process that incorporates inputs and
  feedback from the employer and takes into account
  the fast pace of technological change.

29
Industrial Assessment Committee

• Short-term objectives
• Identify/prioritize the core competencies
  required for entry-level engineers
• Work with faculty to identify educational
  requirements based on core competencies
• Assess current curricula as to its effectiveness
  in providing students with the required core
  competencies
• Establish employer-level metrics to measure
  graduates skill proficiency and provide a
  baseline for continuous improvement

30
Industry Partner Role

• Participating member of faculty assessment team
• Bring in assessment best practices from industry
  such as planning, TQM processes, metrics,
  multi-source feedback, assessment centers, etc
• Form relationships between school and
  engineering/human resources organizations
• Act as liaison with Gateway Coalition National
  Visiting Committee

31
Assessment Design Steps
Step 1 Define Objectives, Strategies,
Outcomes
Step 2 Identify Assessment Methods
Step 5 Apply Results
Continuous Improvement
Step 3 Develop/Pilot Assessment Processes
Step 4 Implement/Expand Assessment Processes
32
Focus Group Objectives

• Identify outcomes, objectives performance
  criteria for each course and program represented
• Discuss existing assessment tools
• Review potential use for continuous improvement

Design Step 1
33
Participating Programs

• Curriculum Reform
• FE/FED - Freshman course in design
  manufacturing
• General University Requirements
• Learning Tools
• Computer-aided learning

• Industrial Interaction
• Intern program
• Advanced Manufacturing Laboratory (AML)
• Seminars Series - What Do Engineers Do?
• Advance Design Engineering in Product Teams
  (ADEPT)

34
People Involved

• 35-45 faculty members involved
• Multidisciplinary - 3 of 4 colleges involved
• Across all levels with enthusiastic involvement
  from tenured faculty
• Covers courses programs in all four years
• Deans and department chairs
• Members of NJIT/NCE Advisory Board

35
Learning Outcomes
What critical knowledge, skills, and behaviors
must students acquire in this course?
Question
Student will demonstrate an ability to actively
participate, listen and collaborate with other
team members while working on a technical design
project.
Example
36
Results 9 Core Learning Outcomes

• Analytical Thinking
• Communication Skills
• Creative Problem Solving
• Project Management

• Research Skills
• Self-Learning
• Systems Thinking
• Teamwork
• Technical Competence

37
Learning Outcome Definitions

• Analytical Skills
• Applies logic in solving problems and analyzes
  problems from different points of views.
  Translates academic theory into practical
  applications and recognizes interrelationships
  among problems and issues.

38
Learning Outcome Survey
NCE Learning Outcomes Instructor/Coordinator
The purpose of this survey is to gather your
ratings on the importance of each of the nine
broad learning outcomes and their specific
objectives as identified by you during the focus
group sessions conducted recently by the NCE
Assessment Team. Please review each statement
carefully and rate it using the scale provided.
In addition, you are provided with extra space to
add and rate objectives within each broad
learning outcome. Technical Competence Not at
all Minor Moderate Considerable
Critical How important are the following
student important importance importance
importance importance learning objectives to
the satisfactory completion of your course or
program Demonstrates a basic knowledge of
1 2 3 4
5 fundamental engineering principles in the
specific disciplines focused on in
this course/program Integrates basic knowledge
of other 1 2 3
4 5 engineering disciplines
within the scope of the courses project
39
Ratings by Faculty and Industry Partners

• Research Skills (3.82, 3.54)
• Systems Thinking (3.84, 4.65)
• Self-Learning (3.82, 3.22)
• Teamwork (4.23, 4.67)
• Technical Competence (3.61, 3.54)

• Analytical Thinking (3.83, 3.52)
• Communication Skills (3.99, 4.62)
• Problem Solving (3.62, 4.20)
• Project Management (4.10, 4.44)

Other analyses show that learning outcome
emphasis changes as one moves to upper division
curricula.
40
Advisory Board - Post Review

• Discuss/answer the following questions
• Think about an engineer that has recently entered
  your organization and is regarded as highly
  effective (up to speed quickly, little
  retraining, etc.)
• Does he or she demonstrate the selected
  competencies? Give examples of how they
  demonstrate each selected competency. Would you
  add specific statements to the definition of a
  selected competency?
• What performance criteria would you use to
  evaluate successful demonstration of the selected
  competencies in your organization?
• Would you add any other competencies that new
  engineers should have based on your experience
  with the successful engineer above?

41
Other Outcomes

• Creation of several competency-based assessment
  processes
• Revised portfolio process for freshman design
  experience
• Team Developer version for engineering learning
  outcomes
• Senior Exit, Alumni, and Employer surveys
• Development of new assessment planning process
  diffused to all Coalition Schools and beyond

42
Gateway - Lessons Learned

• Industrial partners add value to assessment
  process, especially in providing a real-word
  perspective on educational objectives and
  learning outcomes
• Process fostering comparison helps to promote new
  thinking
• Important for sub-group of industrial partners to
  work closely with faculty throughout assessment
  cycle
• Must keep all participants actively involved in
  planning and decisions

43
Conclusions

• INPUT
• Fresh perspectives real world insights to the
  assessment enterprise.
• New ways of thinking about education
• Commitment to improving education

• OUTCOME
• Vitalizes planning process boosts faculty
  commitment/involvement to assessment
• Increased campus support for innovative teaching
  assessment practices
• Industry partnerships focused on education in
  both sectors

44
The Future of Collaboration

• Current research shows
• In most cases, industry kept to the periphery of
  assessment
• Faculty campus administrators fear industry
  partners will make unreasonable demands because
  they dont understand the challenges facing
  higher education
• Colleges do not know how to capitalize on the
  expertise of industrial partners
• Assessment is still in fledgling stages
• The conclusions from case studies confirmed by
  Colleges who integrated industry into their
  assessment processes