Assessment Workshop College of Education

1
Assessment WorkshopCollege of Education

• Katherine Perez
• Jacqueline Peña
• Office of Planning and Institutional
  Effectiveness
• January 2009

2
Why Focus on Assessment?

• SACS Accreditation (http//www.sacs.org)
• Key SACS Deadlines
• September 10, 2009Report due
• March 8-12, 2010Onsite Review
• Key IE Deadlines
• February 1, 2009
• Fall 2008 SLO matrices (with results and use of
  results)
• June 1, 2009
• Spring 2009 SLO matrices (with results and use of
  results)
• 08-09 PO matrices (with results and use of
  results)
• 08-09 Operational Objectives (with results and
  use of results)
• Continuous improvement
• Institutional Effectiveness
• Sound, research-based and assessment-based
  decisions

3
Blank SLO Matrix 08-09
Link to the Mission
Student Learning Outcome (Stated in Measurable Terms) Assessment Method Results (Data Summary and Analysis)

Use of Results for Improving Student Learning Use of Results for Improving Student Learning Use of Results for Improving Student Learning

http//w3.fiu.edu/irdata/portal/inst_effectiveness

4
Clear SACS Guidelines

• Student learning outcomes (SLO) matrix
• Undergraduate programs 3 SLOs minimum
• Graduate programs 3 SLOs minimum
• Program outcomes (PO) matrix
• All program 3 POs
• 1 outcome per page
• Align each outcome with an assessment, set of
  results, and a clear use of results
• Reliable and valid assessment of each outcome
• Appropriate use of results to enhance student
  learning for the outcome

5
SLO Matrices More SACS Guidelines

• There must be some difference among degree
  programs.
• The BA and BS in chemistry cannot have the same
  SLOS and the same assessment procedures (i.e.
  artifacts).
• Similar Programs?
• Outcomes Show different levels (Blooms
  Taxonomy)
• Assessments Use different artifacts, rubrics,
  or criteria
• Principle 3.6.1
• The institutions post-baccalaureate professional
  degree programs, masters and doctoral degree
  programs, are progressively more advanced in
  academic content than its undergraduate programs.

6
Writing the slo

• Criteria
• Can be observed and measured
• Relates to student learning towards the end of
  the program
• Reflects an important concept
• Formula Who Action Verb What
• As Stewards of the Discipline, students will
  apply reading education pedagogical and content
  knowledge and skills in a K-12 learning
  environment.
• As Stewards of the Discipline, students will
  apply reading education pedagogical and content
  knowledge and skills in a K-12 learning
  environment.
• Students will analyze and reflect on students
  language abilities and develop appropriate lesson
  plans to address their specific language needs.
• Students will analyze and reflect on students
  language abilities and develop appropriate lesson
  plans to address their specific language needs.

7
Technology SLO

• Principle 3.4.12
• The institutions use of technology enhances
  student learning and is appropriate for meeting
  the objectives of its programs. Students have
  access to and training in the use of technology.
  (Technology use)
• Required for all undergraduate programs
• Optional for 2007-2008 SLO matrices
• Required for 2008-2009 SLO matrices

8
Sample Technology outcomes

• Too General
• Students will use information technology to
  gather and disseminate information.
• More Specific
• Students will be able to effectively demonstrate
  information technology skills by locating and
  retrieving information on education topics and
  issues and published research in education and
  related fields.
• Students will write and present a capstone
  project that requires the use of Word, Excel,
  PowerPoint, and information technology.

9
Action Verbs Who Action Verb What
Knowledge Comprehension Application Analysis Evaluation Synthesis
List Summarize Apply Analyze Assess Combine
Define Describe Demonstrate Separate Rank Integrate
Describe Interpret Illustrate Explain Test Modify
Identify Predict Solve Connect Measure Create
Show Distinguish Examine Classify Judge Design
Examine Estimate Modify Arrange Critique Compose
State Paraphrase Classify Compare Discriminate Prepare
Find Compare Calculate Contrast Support Write
Label Contrast Compute Infer Hypothesize Formulate
Infer Construct Order Construct
Implement
10
SLO Activity

• Graduate students will learn basic concepts in
  their field.
• Students will do an oral presentation.
• Students will communicate effectively in an oral
  format.
• Students will take all the courses in order.
• Graduate students will apply and analyze various
  statistical concepts in an appropriate
  quantitative study.

11
Assessing the SLO

• Artifact
• Paper or project
• Presentation or behavioral observation
• Data Collection / Collection of the Artifact
• Where, when, how
• Census versus sampling
• Sampling technique (if applicable)
• Criteria
• Minimum standards on a rubric/scale or the
  percentage of correct items
• Evaluation of the Artifact
• Faculty panel or external evaluators
  (reliability)
• Rubrics or embedded questions

12
Direct and Indirect Measures
Common Direct Measures Common Indirect Measures
Standardized exams Exit examinations National or regional exams Portfolios Pre-tests and post-tests Locally developed exams Papers Oral presentations Behavioral observations Thesis/dissertation Surveys or questionnaires Student perception Alumni perception Employer perception Focus groups Interviews Student records
13
Sample SLO Assessment
Student Learning Outcome (Stated in Measurable Terms) Assessment Methods Results (Data Summary and Analysis)
Undergraduate English students will communicate effectively in written format. During their final semester, in their senior seminar course (PPP 4677), students will write a 10-page research paper on a critical topic as applied to an English, Caribbean, or American novel. 20 of the students papers will be randomly selected and evaluated by a faculty panel of 3 members using a rubric that measures (1) Grammar, (2) Essay Structure, (3) Coherence of Argument, (4) Application of MLA. Students will achieve a 3 or better on a 4-point rubric 1 novice 2 apprentice 3 practitioner 4 expert
Use of Results for Improving Student Learning Use of Results for Improving Student Learning Use of Results for Improving Student Learning

14
Reporting the results

• Summary of Results
• Direct measures
• Test items
• Performance as determined by rubrics
• Indirect measures
• Surveys and questionnaires
• Interview and focus group data
• Format
• Narrative
• Tables or charts
• Every student learning outcome must have at
  least
• One set of results
• One student learning improvement strategy (use of
  results)

15
Results Activity

• 75 of the students met the criteria for success.
• Our students passed the dissertation defense on
  the first attempt.
• All the students passed the national exam.
• 75 of the students (n15) achieved a 3 or better
  on the 5 rubric categories for the capstone
  course research paper.

16
Sample SLO Assessment Results
Student Learning Outcome (Stated in Measurable Terms) Assessment Methods Results (Data Summary and Analysis)
Undergraduate English students will communicate effectively in written format. During their final semester, in their senior seminar course (PPP 4677), students will write a 10-page research paper on a critical topic as applied to an English, Caribbean, or American novel. 20 of the students papers will be randomly selected and evaluated by a faculty panel of 3 members using a rubric that measures (1) Grammar, (2) Essay Structure, (3) Coherence of Argument, (4) Application of MLA. Students will achieve a 3 or better on a 4-point rubric 1 novice 2 apprentice 3 practitioner 4 expert 100 students took the seminar in 07-08. 20 papers were sampled. 100 met the minimum criterion for success (i.e. scoring a 3 or better on all four components).
Use of Results for Improving Student Learning Use of Results for Improving Student Learning Use of Results for Improving Student Learning

1 2 3 4
G 0 0 12 8
E 0 0 10 10
C 0 0 15 5
M 0 0 20 0
71 29
17
Masters Thesis Results Table
Frequency of Student Results for all Four Categories of the Research Paper (n20) Frequency of Student Results for all Four Categories of the Research Paper (n20) Frequency of Student Results for all Four Categories of the Research Paper (n20) Frequency of Student Results for all Four Categories of the Research Paper (n20) Frequency of Student Results for all Four Categories of the Research Paper (n20) Frequency of Student Results for all Four Categories of the Research Paper (n20)
1 Novice 2 Apprentice 3 practitioner 4 expert Percentage meeting criteria
Grammar 2 (10) 0 (10) 8 (40) 8 (40) 80 (n16) met the criteria.
Essay Structure 4 (20) 0 (15) 11 (55) 2 (10) 65 (n13) met the criteria.
Coherence of Argument 2 (10) 0 (35) 10 (50) 1 (5) 55 (n11) met the criteria.
Application of MLA Rules 3 (15) 0 (25) 12 (60) 0 (0) 60 (n12) met the criteria.
Criteria Students will achieve a 3 or better on
a 4-point rubric on all five sections of the
masters level thesis.
18
Explaining your use of results
DO DONT
DO focus on making specific improvements based on faculty consensus. DONT focus on simply planning for improvements or making improvements without faculty feedback.
DO address specific program improvements that will impact student learning. DONT address assessment improvement plans such as revising the rubric.
DO use concrete ideas. DONT write vague ideas.
DO state strategies that are sustainable and feasible. DONT use strategies that are impossible to complete in one year considering your resources.
DO use strategies that can improve the curriculum and help students learn outside of courses. DONT focus on only improving the curriculum.
19
Use of Results Activity

• Target met. Will continue to monitor.
• The faculty will meet and revise the three
  introductory courses.
• Students demonstrating difficulty writing
  research papers will be referred to the Academic
  Center for Excellence.
• A larger sample will be obtained.
• We will revise the rubric and have a calibration
  session with the faculty prior to evaluating the
  student papers each semester.
• A capstone course will be created that emphasizes
  research and thesis writing methods.

20
Sample Completed SLO matrix
Student Learning Outcome (Stated in Measurable Terms) Assessment Methods Results (Data Summary and Analysis)
Undergraduate English students will communicate effectively in written format. During their final semester, in their senior seminar course (PPP 4677), students will write a 10-page research paper on a critical topic as applied to an English, Caribbean, or American novel. 20 of the students papers will be randomly selected and evaluated by a faculty panel of 3 members using a rubric that measures (1) Grammar, (2) Essay Structure, (3) Coherence of Argument, (4) Application of MLA rules. Students will achieve a 3 or better on a 4-point rubric 1 novice 2 apprentice 3 practitioner 4 expert 100 students took the seminar in 07-08. 20 papers were sampled. 100 met the minimum criterion for success (i.e. scoring a 3 or better on all four components).
Use of Results for Improving Student Learning Use of Results for Improving Student Learning Use of Results for Improving Student Learning
Undergraduate English students will be required to visit the Center for Academic Excellence once each semester to work with a writing tutoring for at least one 60-minute session. Undergraduate English students who have difficulty with grammar, essay structure, coherence of argument, or application of MLA rules at any point during their academic program will be referred to a writing tutor at the Center for Academic Excellence. Every undergraduate English course will require at least one 5-page paper that requires the use of MLA rules and principles. Undergraduate English students will be required to visit the Center for Academic Excellence once each semester to work with a writing tutoring for at least one 60-minute session. Undergraduate English students who have difficulty with grammar, essay structure, coherence of argument, or application of MLA rules at any point during their academic program will be referred to a writing tutor at the Center for Academic Excellence. Every undergraduate English course will require at least one 5-page paper that requires the use of MLA rules and principles. Undergraduate English students will be required to visit the Center for Academic Excellence once each semester to work with a writing tutoring for at least one 60-minute session. Undergraduate English students who have difficulty with grammar, essay structure, coherence of argument, or application of MLA rules at any point during their academic program will be referred to a writing tutor at the Center for Academic Excellence. Every undergraduate English course will require at least one 5-page paper that requires the use of MLA rules and principles.
1 2 3 4
G 0 0 12 8
E 0 0 10 10
C 0 0 15 5
M 0 0 20 0
71 29
21
The summary page

• Overview and Comparisons
• Give an overview or summary of all the outcomes
  together
• Discuss trends that you have seen over the years
• Explanations
• Provide qualitative explanations for poor results
  or exceptionally high results
• Notes and documentation affecting results
• Response rate (e.g. Only 50 of the students
  completed the project.)
• Inter-rater reliability (e.g. 2 faculty members
  reviewed the artifacts and the inter-rater
  reliability was only 60.)
• Assessment Improvement Plans
• Revise or create instrument (e.g. artifact,
  rubric)
• Modify assessment methods (e.g. data collection,
  sampling, criteria, evaluation process)

22
Program Outcomes

• Program-level outcomes
• Focus on student success (not student learning)
• Formula Who Action Verb What
• Examples
• Graduates seeking employment in the field will
  find such employment within 6 months of
  graduation.
• Candidates will pass the FTCE and score higher
  than the state average.
• Program graduates will be satisfied with advising
  and mentoring services.

23
Sample Completed po matrix
Program Outcome (Stated in Measurable Terms) Assessment Methods Results (Data Summary and Analysis)
Full-time students will graduate from the English doctoral program within 7 years of program admission. The program administrative assistant will keep track of students progress in the program from date of admission, making updates for each student at the end of each academic semester. 10 students graduated from the English doctoral program during the 2008-2009 academic year. The program assistant analyzed the completion timeframe for each individual student and discovered that all of the graduating students spent over half of their graduate time working on the dissertation (3-6 years).
Use of Results for Improving Student Learning Use of Results for Improving Student Learning Use of Results for Improving Student Learning
Doctoral advisors and dissertation chairs will be required to meet with their students twice a semester to go over the program of study, the students progress, and the students future goals and timelines. The college will support the graduate student organization in developing dissertation work/study groups, for all incoming students in the program. The dissertation groups should meet once a month to share drafts, establish new goals, create completion timelines, and do any other activities that the group members find helpful for successful completion of the program. Students will be mailed and emailed warning letters after 4 years in the program reminding them that they need to graduate within 7 years of program admission. Doctoral advisors and dissertation chairs will be required to meet with their students twice a semester to go over the program of study, the students progress, and the students future goals and timelines. The college will support the graduate student organization in developing dissertation work/study groups, for all incoming students in the program. The dissertation groups should meet once a month to share drafts, establish new goals, create completion timelines, and do any other activities that the group members find helpful for successful completion of the program. Students will be mailed and emailed warning letters after 4 years in the program reminding them that they need to graduate within 7 years of program admission. Doctoral advisors and dissertation chairs will be required to meet with their students twice a semester to go over the program of study, the students progress, and the students future goals and timelines. The college will support the graduate student organization in developing dissertation work/study groups, for all incoming students in the program. The dissertation groups should meet once a month to share drafts, establish new goals, create completion timelines, and do any other activities that the group members find helpful for successful completion of the program. Students will be mailed and emailed warning letters after 4 years in the program reminding them that they need to graduate within 7 years of program admission.
Time of Completion for the 10 Students Time of Completion for the 10 Students
5 students 6 years
2 students 7 years
2 students 8 years
1 student 9 years
24
The Ie Team (http//w3.fiu.edu/irdata/portal/inst_
effectiveness)

• Marta Perez
• Director
• perezma_at_fiu.edu
• 305-348-2733
• Maria Corrales
• Coordinator
• corrales_at_fiu.edu
• 305-348-0459
• Katherine Perez
• Coordinator
• katherine.perez_at_fiu.edu
• 305-348-1418
• Jacqueline Peña
• Coordinator
• jpena_at_fiu.edu.
• 305-348-1367

• Mayelin Felipe
• Computer Specialist
• mfelip01_at_fiu.edu
• 305-348-0115
• Karla Felipe
• Computer Specialist
• kgarcia_at_fiu.edu
• 305-348-0115
• Amanda Berhaupt
• Graduate Assistant
• aberh001_at_fiu.edu
• 305-348-2731
• Randhir Kaur
• Graduate Assistant
• rkaur001_at_fiu.edu
• 305-348-2731

25
Our Role at IE and in FIU

• Institutional Effectiveness
• Improve student learning
• Student, program, and operational levels
• Improve efficiency and effectiveness of the
  university
• Assist university with the assessment process
• Assessment Coordinators
• Assist with the assessment process, including
• Education concerning assessment
• Articulation of outcomes and assessments
• Institutionalization of assessment practices
• Translation of successful assessment work for the
  SACS world
• Dissemination of assessment and accreditation
  information