Peggy Brouse, PhD.

1
Measuring Outcomes and Objectives for ABET
Accreditation in a Systems Engineering
Undergraduate Program

• Peggy Brouse, PhD.
• Ariela Sofer, Ph.D.
• Systems Engineering and Operations Research
  Department
• Vogenau School of Information Technology and
  Engineering
• George Mason University

2
0utline

• Background
• Descriptions of the processes used to assess the
  outcomes of our program
• Measurement results of assessing our outcomes
• Descriptions of the processes used to assess the
  objectives of our program
• Measurement results of assessing our objectives
• A summary of noteworthy feedback gained through
  the assessment processes
• A list of improvements made as a result of the
  feedback

3
Background

• The Accreditation Board for Engineering and
  Technology, ABET, is an accreditation
  organization for college and university programs
  in applied science, computing, engineering, and
  technology
• ABET provides a sets of goals and objectives for
  accreditation and evaluates and certifies
  engineering related education
• The purpose of this INCOSE paper is to present
  the process used by the Systems Engineering and
  Operations Research Department at George Mason
  University to accredit our Systems Engineering
  undergraduate program with special emphasis on
  measurement of outcomes and objectives

4
Outcomes Processes 1

• Achievement of Program Outcomes is determined by
  the assessment of appropriate evidence supporting
  the departments outcomes
• samples of student work
• surveys and focus groups of students
• surveys of alumni and employers
• informal interaction between faculty, students,
  alumni and industry representatives

5
Outcomes Processes 2

• Bulk of our assessment activity comes through the
  evaluation of students performance in courses
• Samples of student work, in the form of exams,
  homework, project and laboratory reports,
  presentation and project evaluation, are
  collected and archived in Course Notebooks
• Material from Course Notebooks is extracted and
  compiled into Outcome Notebooks for each of our
  12 Program Outcomes
• Each Outcome Notebook is reviewed and evaluated
  by a faculty member and a member of the SEOR
  Industry Advisory Committee
• Recommendations are compiled, evaluated by the
  SEOR Undergraduate Curriculum Committee, and used
  as input for changes to the curriculum

6
Outcomes Processes 3

• Additional mechanisms by which Program Outcomes
  are assessed
• Senior surveys are done at the School and
  University level in conjunction with the
  graduation application process. Results are
  subsequently provided to the program
• A focus group interview is conducted at the end
  of the spring semester with students completing
  the senior design sequence
• The University conducts course evaluations at the
  end of each semester and the results are provided
  to the department midway through the following
  semester
• Informal feedback is obtained through the
  advising process, through ongoing informal
  contacts between faculty and students, and
  through informal contacts with outside
  constituencies in industry, government and
  academia

7
Outcomes Processes 4

• The Systems Engineering Undergraduate Curriculum
  Committee (SUCC) reviews the evidence for
  accomplishment of the outcomes and the
  instruments used for collecting evidence
• Recommendations for changes in the BSSE Program
  are developed by the SUCC and considered for vote
  by the faculty as a whole
• Proposed changes are reviewed by the SEOR
  Department Industry Advisory Committee

8
Outcomes Processes 5
Mason SEOR Outcomes Process Table
9
Outcomes Measures

• The measures used by the department to determine
  achievement of program outcomes are
• Course exams and homework exercises
• Project and laboratory reports
• Oral presentations given in class and evaluated
  using Presentation Evaluation Form
• Results of group projects Systems Engineering
  undergraduates have won several major national
  awards for their group projects in their Senior
  Design SYST490/495 class
• Results from surveys of students and alumni
• Responses from industry who serve as clients for
  our senior design project. Our SYST490/495
  projects are sponsored by industry groups they
  attend both the final SYST490 and SYST495
  presentations
• Responses from our advisory board Advisory Board
  meetings have been conducted on a regular basis,
  feedback for the BS/SE program has been very
  positive

10
Outcomes Results 1

• Average 2004/2005 Program Outcomes Results
  (scale 1-7)
• 5.20 - Knowledge of mathematics through
  differential and integral calculus and advanced
  topics in differential equations, linear algebra,
  statistical analysis, and complex variables
• 6.60 - Knowledge of core systems engineering
  theory and methods
• 5.40 - An ability to use modern engineering
  techniques, skills and tools, including computer
  based tools for analysis and design
• 5.20 An ability to apply knowledge of
  mathematics, science and engineering to the
  analysis of systems engineering problems
• 5.80 - An ability to design scientific and
  engineering experiments
• 5.60 - An ability to conduct scientific and
  engineering experiments
• 6.40 - An ability to analyze and interpret data

11
Outcomes Results 2

• Average 2006 Program Outcomes Results
  (scale 1-7)
• 6.20 - An ability to identify, formulate, and
  solve engineering problems, including the
  planning, specification, design, implementation
  and operation of systems, components and/or
  processes that meet performance, cost, time,
  safety and quality requirements
• 6.40 - An ability to function on
  multi-disciplinary teams
• 6.40 - An understanding of professional and
  ethical responsibility
• 6.60 - An ability to convey technical material
  through oral presentation and interaction with an
  audience
• 6.00 - An ability to convey technical material
  through formal written papers/reports that
  satisfy accepted standards for writing style.
• 6.20 - A recognition of the need for and an
  ability to engage in life-long learning.
• 6.20 - The broad education and knowledge of
  contemporary issues necessary to understand the
  impact of systems engineering solutions in a
  global and societal context

12
Objectives Processes 1

• Program Educational Objectives are evaluated
  through a process that includes
• Collating and evaluating evidence from our alumni
  who have graduated three to six years from date
  of survey
• Surveying the employers of this group of students
• Evaluation of the objectives themselves and the
  degree to which they are being met by our
  educational program involves two components
• A global evaluation aimed directly at the
  objectives
• A detailed examination using the mapping of
  program outcomes to the program educational
  objectives

13
Objectives Processes 2

• The Program Educational Objectives are presented
  for comment and review to the students of the
  introductory freshman systems engineering course,
  SYST 101, and also to the students in the senior
  seminar, SYST 489
• The Systems Engineering Undergraduate Curriculum
  Committee (SUCC) reviews the evidence for
  accomplishment of the objectives and the
  instruments used for collecting evidence
• Recommendations for changes in the BSSE Program
  are developed by the SUCC and considered for vote
  by the faculty as a whole
• Proposed changes are reviewed by the SEOR
  Department Industry Advisory Committee

14
Objectives Processes 3
Mason SEOR Objectives Process Table
15
Objectives Measures

• The measures used by the department to determine
  achievement of program objectives are
• Feedback from alumni through information
  collected via questionnaires in conjunction with
  the Mason Alumni Office
• Feedback from the employers of our alumni through
  information collected via questionnaires
• Responses from our advisory board Advisory Board
  meetings have been conducted on a regular basis,
  feedback for the BS/SE program has been very
  positive

16
Objectives Results 1

• Results for SEOR graduates graduating from Mason
  in the past 3 to 5 years are summarized below
•  How do you rate your current competency in
  scale 5 choices (very unsatisfied to very
  satisfied)
  
• a. Applying fundamental concepts of mathematics
  to contemporary and future systems Satisfied
• b. Applying fundamental concepts of science to
  contemporary and future systems Very satisfied
• c.  Applying fundamental concepts of information
  technology to contemporary and future
  systems Very satisfied
• d.  Applying fundamental concepts of engineering
  to contemporary and future systems Very
  satisfied
• e. Contributing to the development of systems
  using systems engineering methods and
  processed Very satisfied
• f. Contributing to the development of systems
  using systems engineering models and tools Very
  satisfied

17
Objectives Results 2

• g. Working effectively as a member of
  multi-disciplinary teams
  Very satisfied
• h. Knowing issues related to ethical behavior and
  professional responsibility
  Very satisfied
• i. Possessing skills and establishing a
  foundation for life-long learning systems
  engineering and related areas Very satisfied
• j. Communicating effectively with team members
  and decision makers orally
  Very satisfied
• k. Communicating effectively with team members
  and decision makers in writing Very satisfied
• l.  Overall ability as a Systems
  Engineer Very satisfied
• m. Contribution of your Mason SEOR education to
  your overall ability as a Systems Engineer
  Very satisfied
• n. Overall ability to be successful in your
  professional career
  Very satisfied
• o.Contribution of your Mason SEOR education to
  your overall success
  Very satisfied

18
Assessment Feedback and Actions Taken 1

• SYST489, Senior Seminar, students are required to
  conduct a curriculum review. Some recommendations
  from the students and actions taken are
• Instruction regarding Configuration Management
• Topic is covered in SYST489 but should be covered
  in more detail early in the curriculum
• Faculty is considering where this might be
  included.
• Increased communication between faculty and
  students
• A concerted effort has been undertaken to
  increase communication via email contact, and
  more comprehensive advising

19
Assessment Feedback and Actions Taken 2

• Some recommendations from the students and
  actions taken are
• Review of the SYST101 course
• An extensive review of SYST101 was undertaken and
  significant changes have been made. The
  evaluation of the new course content has been
  very favorable
• Teach students how Systems Engineering is applied
  in the real world.
• Significant progress has been made in this area
  both in the use of sponsored real-world problems
  in the SYST490/SYST495 Senior Design and in the
  use of guest lecturers from commercial firms in
  various classes including SYST489

20
Assessment Feedback and Actions Taken 3

• Some recommendations from the students and
  actions taken are
• Allow for more and varied concentration tracks
• Since this recommendation, new tracks have been
  added
• It is also possible for students to design their
  own track with advisement from their academic
  advisor
• Increased availability of career advisors for the
  students in the program.
• This is an ongoing process
• We have sponsored design projects wherein the
  students are in contact with commercial
  representatives
• Students receive advice throughout their senior
  year

21
Assessment Feedback and Actions Taken 4

• Improvements and actions taken as a result of
  feedback from the Advisory Board
• Board recommended SEOR refer students to their
  companies
• Undergrads with a 3.5 or more GPA will be
  considered for eligibility
• Students will get tuition reimbursement and
  stipend
• This is a very worthy recommendation. We have
  tried to facilitate the interface of students and
  local companies
• Recruit our BS/SE students into the MS/SE degree
• We have a joint BS/MS program that is recommended
  to qualifying students

22
Assessment Feedback and Actions Taken 5

• Improvements and actions taken as a result of
  feedback from the Advisory Board
• Increase number of required credits from 120 to
  132 to include more domain information and
  preparation for the FE exam
• The SEOR faculty is limited to 120 hours by
  school requirements
• A review of capabilities necessary to pass the FE
  exam is being taught by faculty to students on
  Saturdays for weeks before the exam

23

• Questions?