Developing Assessment Plans that Work

1
Developing Assessment Plans that Work

• Quality Assessment Program
• Program Assessment and Evaluation Matrix
• What we need to Accomplish?
• ABET, What we have do First?
• What is Urgent Now?
• Whose responsibility?
• Program Workshops to help your faculty use
  assessment to improve teaching and learning
• ? September 2005?
• WEB PAGE
• Self-study report
• A list of 206 Accredited Computing Programs

2
ABET Quality Assessment Program

• Successful assessment
• Flows from the institution's mission and
  educational purposes.
• Emerges from organized conceptual framework.
• Is marked by faculty ownership and
  responsibility.
• Has institution-wide support.
• Relies on multiple measures.
• Provides feedback to students and the
  institution.
• Is cost-effective.
• Does not restrict or inhibit goals of access,
  equity, and diversity established by the
  institution.
• Leads to improvement.
• Includes a process for evaluating the assessment
  program.

• What are we trying to do?
• Review/Develop
• Assess/Evaluate
• Document Results
• Incorporate results
• Continuously Improve
• Satisfy customers
• Satisfy ABET
• Continue/Stop
• How well can we do it?
• A comprehensive, systematic, ongoing cycle of
  assessment is crucial to continuous improvement
• Understand the purpose of assessment is not to
  compare programs, to point fingers at
  programmatic problems, or to publish scorecards.
• Quality assessment promotes continuous
  improvement at all levels of the university by
  providing the necessary evidence to guide
  effective decision making in many
  areas--programmatic changes, classroom teaching
  modifications, support service modifications,
  policy or procedure changes, structural
  reorganization.
• How can we improve what we are doing?
• Believe in it
• Do yours
• Cooperate
• Help other

3
1
Programs Mission
PEOs POs
4
Program Assessment and Evaluation Matrix

• Goals
• What are the overall goals of the program?
• How do they complement institutional and
  accreditation expectations?
• Program Objectives
• What are the program objectives?
• What should your students know and be able to
  do?
• Performance Criteria
• How will you know the objectives have been met?
• What level of performance meets each objective?
• Implementation Strategy
• How will the objectives be met?
• What program activities (curricular and
  co-curricular) help you meet each objective?

• Evaluation Methods
• What assessment methods will you use to collect
  data?
• How will you interpret and evaluate the data?
• Logistics
• When will you measure?
• How often? Who will collect and interpret the
  data and report the results?
• Feedback
• Who needs to know the results?
• How can you convince them the objectives were
  met?
• How can you improve your program and your
  assessment process?

5
General ABET Criteria

• 3. Program Outcomes and Assessment
• Assessment Process with Documented Results to
  Measure Outcomes
• Results Applied to Improvement of the Program
• Demonstration (incl. Process Measurements) that
  Graduates have
• ability to apply knowledge of math, engineering,
  and science
• ability to design and conduct experiments as well
  as to analyze and interpret data
• ability to design system, component or process to
  meet needs
• ability to function on multi-disciplinary teams
• ability to identify, formulate, and solve
  engineering problem
• understanding of professional and ethical
  responsibility
• ability to communicate effectively
• Broad education
• recognition of need and ability to engage in
  life-long learning
• knowledge of contemporary issues
• ability to use techniques, skills, and tools in
  engineering practice

• Students
• Admit
• Advise
• Evaluate
• Monitor
• 2. Program Educational Objective
• Published and Consistent with Mission and These
  Criteria
• Process in Place to Determine and Periodically
  Evaluate
• Based on of ConstituenciesCurriculum and
  processes to ensure achievement
• Evaluation to Determine Achievement
• Results Used to Improve Effectiveness of the
  Program

6
General ABET Criteria

• 4. Professional Component
• Major Design Experience
• One year of Mathematics and Basic Science
• One and one-half Years on Engineering Topics
• General Education Component, Consistent with
  Program Objectives
• 5. Faculty
• Sufficient Number and Competencies to Cover All
  Curricular Areas (FTE)
• Levels of Student- Faculty Interaction
• Advise and Counsel students
• Actively involve in service (College
  University)
• Interact with Practitioners and Employers
• Professional development
• Monitor and evaluate effectively

• 6. Facilities
• Classrooms
• Laboratories
• Equipment and Tools
• Computing and Information
• Infrastructure
• Certified Technicians and qualified operators
• 7. Institutional Support Financial Resources
• Sufficient to Assure Quality and Continuity of
  the Program
• Sufficient to Attract and Retain a Well-Qualified
  Faculty
• Sufficient to Acquire, Maintain, and Operate
  Facilities and Equipment
• 8. Program Criteria
• Curricular Topics
• Faculty Qualifications
• Other

7
CS ABET Criteria

• Objectives and Assessments (ABET 2 3)
• The program has documented, measurable
• objectives, including expected outcomes for
• graduates. The program regularly assesses its
• progress against its objectives and uses the
  results
• of the assessments to identify program
• improvements and to modify the programs
• objectives.
• Student Support (ABET 1)
• Students can complete the program in a reasonable
• amount of time. Students have ample opportunity
  to
• interact with their instructors. Students are
  offered
• timely guidance and advice about the programs
• requirements and their career alternatives.
  Students
• who graduate the program meet all program
• requirements.
• Faculty (ABET 5)
• Faculty members are current and active in the
• discipline and have the necessary technical
  breadth

• Curriculum (ABET4,8)
• The curriculum is consistent with the program's
• Documented objectives. It combines technical
• requirements with general education
• requirements and electives to prepare students
• for a professional career in the computer field,
  for
• Further study in computer science, and for
• functioning in modern society. The technical
• requirements include up-to date coverage of
• basic and advanced topics in computer science
• as well as an emphasis on science and
• mathematics.
• Laboratories and Computing Facilities (ABET 6)
• Laboratories and computing facilities are
• available, accessible, and adequately supported
• to enable students to complete their course work
• and to support faculty teaching needs and
• scholarly activities.

8
CS ABET Criteria

• Institutional Support and Financial Resources
  (ABET 7)
• The institutions support for the program and the
  Financial resources available to the program are
• sufficient to provide an environment in which the
  program can achieve its objectives. Support and
• resources are sufficient to provide assurance
  that the program will retain its strength
  throughout
• the period of accreditation.
• Institutional Facilities (ABET 6 )
• Institutional facilities including the library,
  other electronic information retrieval systems,
  computer
• networks, classrooms, and offices are adequate to
  support the objectives of the program.

9
What We Have To Do First?

• Departmental Accreditation Committee (DAC)
• Programs Mission
• Program Constituencies ?
• Program Educational Objectives (PEOs)
• Program Outcomes (POs)
• Program Industrial Advisory Committee (PIAC).
• Program ABET Web Page.
• Electronic Assessment Database.
• Plan of Improvement Assessment (PIA)
• Develop performance criteria for each
  objective/outcome.
• Conduct Assessments and Write Up Results
• Document Use of Results Incorporated into
  Program Improvement

10
Programs Mission

• Example Mission
• To prepare intellectual, professional, and
  ethical graduates, capable of meeting challenges
  in the field of Computer Science and to
  coordinate with other parts of the university to
  facilitate the effective use of educational
  resources, including courses

11
Programs Mission

• Example Mission
• The mission of the Department of Computer
• Science is four-fold
• To conduct scholarly research.
• To provide an instructional environment that
  leads to careers and research in computer science
  and information systems.
• To contribute to the liberal education mission of
  the University.
• To serve the community, the Kingdom, region, and
  the profession.

12
Programs Mission
Programs Mission
Example Mission The mission of the Department
of Computer Science is 1. To produce graduates
with a strong grasp of fundamentals of computer
science, knowledge in technical specialty areas,
and an appreciation of the power of collaborative
effort applied to problem solving. 2. To offer
courses and programs which stimulate innovation
and enhance the ability of graduates to achieve
high levels of professional development and to
succeed in a competitive marketplace. 3. To
conduct research in selected areas and to
integrate research results with teaching
activities. 4. To provide service to the
profession and community and forge strategic
alliances with other professions.
13
Program Educational Objectives (PEOs)

• PEOs are consistent with the mission
• Ensure that PEOs are well-stated and measurable.
• PEOs can be linked with the curriculum for the
  Program
• Involve constituencies

14
Program Educational Objectives (PEOs)

• Example PEOs (CS)
• The program objectives are as follows
• Graduates will have an understanding of the
  fundamental mathematical, logical, statistical
  and scientific principles underlying computing
  and information processing.
• Graduates will have a solid foundation in the
  principles of computer science and will have
  applied that knowledge to a variety of problems.
• Graduates will have an understanding and
  appreciation of the context in which professional
  computing activities occur.
• Graduates will have an in-depth knowledge of a
  wide range of topics spanning the field of
  computer science

15
Program Educational Objectives (PEOs)

• Example PEOs (CS)
• Graduates of the Computer Science Program will
  have the necessary technical knowledge and skills
  both in breadth and depth, to pursue the practice
  or advanced study of computer science.
• Graduates of the Computer Science Program will
  understand the importance of life-long learning,
  and be prepared to learn and understand new
  technological developments in their field.
• Graduates of the Computer Science Program will
  understand the ethical and technical context of
  their computer science contributions and their
  obligations therein.
• Graduates of the Computer Science Program will
  develop the communication, teamwork, and
  leadership skills necessary to function
  productively and professionally.

16
Program Educational Objectives (PEOs)

• Example PEOs (CS)
• 1. Preparation for Practice Graduates will be
  prepared for entry-level positions in their
  discipline and for graduate/professional studies.
• 2. Tools for Creativity Graduate will
  experience the creative and design processes and
  their application to typical engineering
  situations.
• Societal Awareness Graduates will receive the
  breadth of education necessary to integrate
  practice in their discipline with the interests
  of a diverse modern society.
• 4. Leadership Skills Graduates will be
  prepared for leadership in their discipline.

17
How to Involve Program Constituencies?
18
Program Outcomes (POs)

• POs satisfy the 11 ABET criteria
• POs must be measurable and specify an
  appropriate method for measurements
• Outcomes must specify a criterion for judging
  findings.
• satisfies professional component
• For academic programs, include both of the
  followinga. Curricular outcomesb. Learning
  outcomes at the course level
• Direct as well as indirect methods of measurement
  are to be used.
• Map POs into PEOs
• Map POs into ABET a k criteria
• Curriculum achieves POs
• Curriculum satisfies professional component
• Curriculum meets program specific criteria

19
Program Outcomes (POs)

• ABET 11 Outcomes
• ability to apply knowledge of math, engineering,
  and science
• ability to design and conduct experiments as well
  as to analyze and interpret data
• ability to design system, component or process to
  meet needs
• ability to function on multi-disciplinary teams
• ability to identify, formulate, and solve
  engineering problem
• understanding of professional and ethical
  responsibility
• ability to communicate effectively
• Broad education
• recognition of need and ability to engage in
  life-long learning
• knowledge of contemporary issues
• ability to use techniques, skills, and tools in
  engineering practice

20
Program Outcomes (POs)

• Example POS
• Students understand the mathematics and
  statistics that underlie scientific applications.
  
• Students can design, develop, and analyze
  significant software systems.
• Students understand the fundamentals of computer
  organization and architecture, data structures
  and related algorithms, and programming
  languages.
• Students can apply computer science principles
  and practices to a variety of problems.
• Students can work independently and also work
  effectively in teams.
• Students can communicate effectively both orally
  and in writing.
• Students understand social, professional and
  ethical issues related to computing.
• Students are knowledgeable of contemporary issues
  in the arts, social sciences, and humanities.
• Students understand the scientific method and can
  apply this mode of inquiry in a laboratory
  setting. (CS Program)
• Students have a broad perspective of the business
  world. (IST Program)

21
Program Outcomes (POs)

• Program Outcomes (PO's)
• Graduates will have a reasonable level of
  understanding of each of the subject areas that
  define the discipline as well as the
  interrelationships that exist among them
  algorithms, architecture, artificial intelligence
  and robotics, data structures, database and
  information retrieval, human-computer
  interaction, operating systems, programming
  languages, and software engineering.
• Graduates will have the ability to utilize
  appropriate theoretical constructs definitions,
  and axioms, theorems, proofs, and interpretation
  of results.
• Graduates will have the ability to utilize
  appropriate abstractive constructs hypothesis
  formation, data collection, modeling and
  prediction, experimental design, and analysis of
  results.
• Graduates will have the ability to utilize
  appropriate design constructs requirements
  analysis and specification, design,
  implementation, and testing.
• Graduates will be exposed to ethical and societal
  issues associated with the computing field.
• Graduates will be familiar with recent
  technological and theoretical developments,
  general professional standards, and have an
  awareness of their own strengths and limitations
  as well as those of the discipline itself.
• Graduates will be aware of the history of
  computing, including those major developments and
  trends - economic, scientific, legal, political,
  and cultural - that have combined to shape the
  discipline.
• Graduates will be able to appreciate the
  intellectual depth and abstract issues that will
  continue to challenge researchers in the future.
  They should have a strong foundation on which to
  base lifelong learning and development.
• Graduates will have the necessary background for
  entry into graduate study.
• Graduates will have the ability to communicate
  effectively.

22
Program Outcomes (POs)

• Program Outcomes (PO's)
• The intended Educational Outcomes of the program
  (that
• support the above objectives)
• that the graduates of the Computer Science
  undergraduate program will have Proficiency in
  the areas of software design and development,
  data structures, and operating systems
• 2. An ability to plan and execute an problem
  design to meet an identified need
• 3. Proficiency in mathematical and scientific
  principles relevant to computer science
• 4. An ability to communicate effectively
• 5. An understanding of the overall human context
  in which computing activities take place
• 6. A knowledge of contemporary issues and an
  ability to use modern tools and techniques in
  engineering practice.

23
Relationship of PEOs to POS
The relationship between the Departmental
Educational Objectives and the Educational Outcome
s is shown in the following table. The matrix
presented in the table may appear a bit densely
populated, but, in fact, all of the Educational
Objectives support all of the Outcomes.
24
Relationship of PEOs to EC2000 Criteria
25

26

27

Test Grade Distribution

28
WEB PAGE Contents
UOB Mission COE Mission COIT Mission Programs
Missions
ABET Documentations CE, CHE, ET, ME CS, CEIT, MIS
Programs Curriculum Program Faculty Students
Information
Record of All JAC/DAC Programs Meetings
Minutes
Final Results Programs Self-study Reports
On-line Employer Survey Alumni Survey Senior
Exit Survey Faculty Survey Industrial Advisory
Committee
29
Example Programs Self-study Reports

• B. Accreditation Summary
• 1. Students
• 2. Program Educational Objectives
• 3. Program Outcomes and Assessment
• 4. Professional Component
• 5. Faculty
• 6. Facilities
• 7. Institutional Support and Financial Resources
• 8. Program Criteria

• Table of Contents
• A. Background Information
• 1. Degree Titles
• 2. Program Modes
• 3. Actions to Correct Previous Deficiencies
• 4. Department Culture and Administrative
  Structure
• 5. Department Constituencies and Feedback Loops
• 6. Examples of the Feedback Loops Working

30

• Appendix I - Additional Program Information
• A. Tabular Data for Program
• Table A.1. Basic Curriculum
• Table A.2. Course and Section Size Summary
• Table A.3. Faculty Workload Summary
• Table A.4. Faculty Analysis
• Table A.5. Support Expenditures
• B. Course Syllabi
• C. Faculty Resumes
• D. Student exit survey comments for the past two
  years
• E. Alumni survey comments for the past two years
• F. Employer survey comments for the past two
  years
• G. Faculty survey for the past two years

31

• Accredited Computing Programs
• Alabama in Huntsville, The University of Computer
  Science (BS) 1988 Huntsville, AL
• Alabama, The University of Computer Science
  (BSCS) 1990 Tuscaloosa, AL 
• Alaska Fairbanks, University of Computer Science
  (BS) 1991 Fairbanks, AK
•  American University in Cairo, The Computer
  Science (BS) 1999 Cairo, Egypt
• Appalachian State University Computer Science
  (BS) 1988-19961997 Boone, NC
• Arizona State University Computer Science (BS)
  1992 Tempe, AZ
• Arkansas at Little Rock, University of Computer
  Science (BS) 1990 Little Rock, AR
•  Armstrong Atlantic State University Computer
  Science (BS) 1991 Savannah, GA 
• Auburn University Computer Science (BS)
  1987 Auburn, AL
•  Baylor University Computer Science (BSCS)
  1987 Waco, TX
•  Boise State University Computer Science (BS)
  1994 Boise, ID 
• Bowie State University Computer Science (BS)
  1998 Bowie, MD 
• Brigham Young University Computer Science (BS)
  1989 Provo, UT
•  Bucknell University Computer Science (BS)
  1991 Lewisburg, PA
•  Bucknell University Computer Science and
  Engineering (BS) 1997 Lewisburg, PA 
• California Polytechnic State University, San Luis
  Obispo Computer Science (BS) 1986 San Luis
  Obispo, CA 
• California State Polytechnic University,
  Pomona Computer Science (BS) 1994 Pomona, CA

32
(No Transcript)
33

• Hampton University Computer Science (BS) 1989
  Hampton, VA
• Houston, University of Computer Science (BS)
  1987 Houston, TX
• Houston-Clear Lake, University of Computer
  Science (BS) 2002 Houston, TX
• Howard University Systems and Computer Science
  (BS) 1988 Washington, DC
• Idaho, University of Computer Science (Moscow
  and Idaho Falls) (BS) 1993 Moscow, ID
• Illinois at Chicago, University of Computer
  Science (BS) 1997 Chicago, IL
• Illinois at Urbana-Champaign, University of
  Computer Science (BS) 2002 Urbana, IL
• Illinois Institute of Technology Computer
  Science (BS) 2003 Chicago, IL
• Illinois State University Applied Computer
  Science, Computer Science (BS) 2000 Normal, IL
  
• Indiana University-Purdue University Fort Wayne
  Computer Science (BS) 2004 Fort Wayne, IN
• Iowa State University Computer Science (BS)
  1986 Ames, IA
• Jackson State University Computer Science (BS)
  1991-19941996 Jackson, MS
• Kansas State University Computer Science (BS)
  1992 Manhattan, KS
• Kansas, The University of Computer Science (BS)
  1995 Lawrence, KS
• Kennesaw State University Computer Science (BS)
  2004 Kennesaw, GA
• Lafayette College Computer Science (BS) 2003
  Easton, PA
• Lamar University Computer Science (BS) 2002
  Beaumont, TX
• Lehigh University Computer Science in the
  College of Engineering and Applied Science (BS)
  1987 Bethlehem, PA

34

• Nebraska at Omaha, University of Computer
  Science (BS) 2004 Omaha, NE
• Nevada-Las Vegas, University of Computer Science
  (BS) 1993 Las Vegas, NV
• Nevada-Reno, University of Computer Science (BS)
  2000 Reno, NV
• New Hampshire, University of Computer Science
  (BS) 1987 Durham, NH
• New Haven, University of Computer Science (BS)
  2000 West Haven, CT
• New Jersey Institute of Technology Computer
  Science (BA) 1995 Newark, NJ
• New Jersey Institute of Technology Computer
  Science (BS) 1986 Newark, NJ
• New Jersey, College of Computer Science (BS)
  1997 Ewing, NJ
• New Mexico, University of Computer Science (BS)
  1988 Albuquerque, NM
• New Orleans, University of Computer Science (BS)
  1987 New Orleans, LA
• New York at Binghamton, State University of
  Computer Science (BS) 1989 Binghamton, NY
• New York at Brockport, State University of
  Computer Science (BS) 1994 Brockport, NY
• New York at New Paltz, State University of
  Computer Science (BS/BA) 1991 New Paltz, NY
• New York, College of Staten Island, City
  University of Computer Science (BS) 1989
  Staten Island, NY
• Nicholls State University Computer Science (BS)
  1995 Thibodaux, LA
• Norfolk State University Computer Science (BS)
  1991 Norfolk, VA
• North Carolina Agricultural and Technical State
  University Computer Science (BS) 1994
  Greensboro, NC
• North Carolina at Greensboro, University of
  Computer Science (BS) 1995 Greensboro, NC

35

• Rowan University Computer Science (BS) 2001
  Glassboro, NJ
• Salem State College Computer and Information
  Studies (BS) 2004 Salem, MA
• San Diego State University Computer Science (BS)
  1994 San Diego, CA
• San Francisco State University Computer Science
  (BS) 1993 San Francisco, CA
• San Jose State University Computer Science (BS)
  1994 San Jose, CA
• Scranton, University of Computer Science (BS)
  1990 Scranton, PA
• South Alabama, University of Computer and
  Information Sciences, Computer Science
  Specialization (BS) 1988 Mobile, AL
• South Alabama, University of Computer and
  Information Sciences, Information Systems
  Specialization (BS) 2003 Mobile, AL
• South Carolina Spartanburg, University of
  Computer Science (BS) 2003 Spartanburg, SC
• South Carolina State University Computer Science
  (BS) 2004 Orangeburg, SC
• South Carolina, University of Computer Science
  (BS) 1990 Columbia, SC
• South Dakota School of Mines and Technology
  Computer Science (BS) 1993 Rapid City, SD
• South Florida, University of Computer Science
  (BS) 1989 Tampa, FL
• Southeastern Louisiana University Computer
  Science (BS) 2001 Hammond, LA
• Southern California, University of Computer
  Engineering and Computer Science (BS) 2004 Los
  Angeles, CA
• Southern California, University of Computer
  Science (BS) 1988-1994 2004 Los Angeles, CA
• Southern Connecticut State University Computer
  Science (BS) 1992 New Haven, CT
• Southern Illinois University-Edwardsville
  Computer Science (BS) 2003 Edwardsville, IL
• Southern Maine, University of Computer Science
  (BS) 1994 Portland, ME

36

• Tennessee at Chattanooga, University of Computer
  Science, Scientific Option (BS) 2002
  Chattanooga, TN
• Tennessee at Chattanooga, University of Computer
  Science-IS Concentration (BS) 2002
  Chattanooga, TN
• Texas A M University Computer Science (BS)
  1993 College Station, TX
• Texas at Arlington, University of Computer
  Science (BSCS) 2004 Arlington, TX
• Texas at Arlington, University of Computer
  Science and Engineering (BS) 1995 Arlington,
  TX
• Texas at El Paso, University of Computer Science
  (BS) 1988 El Paso, TX
• Texas Christian University Computer Science (BS)
  1990 Forth Worth, TX
• Texas State University-San Marcos Computer
  Science (BS) 1999 San Marcos, TX
• Texas-Pan American, The University of Computer
  Science, Broad Field Major (BSCS) 2003
  Edinburg, TX
• Toledo, The University of Computer Science and
  Engineering (BS) 1991 Toledo, OH
• Towson University Computer Science (BS) 1994
  Towson, MD
• Tulane University Computer Science (BS) 1990
  New Orleans, LA
• Tulsa, The University of Computer Science (BS)
  1988 Tulsa, OK
• United States Air Force Academy Computer Science
  (BS) 1986 USAFA, CO
• United States Military Academy Computer Science
  (BS) 1997 West Point, NY
• United States Naval Academy Computer Science
  (BS) 1987 Annapolis, MD
• Utah State University Computer Science (BS)
  1998 Logan, UT
• Utah Valley State College Computer Science (BS)
  2003 Orem, UT
• Vanderbilt University Computer Science (BS)
  1998 Nashville, TN