Teaching Basic Skills Mathematics: Outcome Assessment

1
Teaching Basic Skills Mathematics Outcome
Assessment
Xi Zhang, Campus Based Researcher, City College
Jenny Kimm, Associate Professor, City College
San Diego Community College District
Presented at the 2009 Strengthening Student
Success Conference San Francisco, CA October 7,
2008
2
Introduction of the District

• San Diego Community College District
• 2nd largest district in the state
• Three 2-year colleges and eleven Continuing
  Education campuses
• Serves approx. 100,000 students each semester

3
Introduction of the San Diego City College

• San Diego City College is the first established
  community college in San Diego.
• San Diego City College is a public, two-year
  community college administered by the San Diego
  Community College District.
• Serving as the educational cornerstone of
  downtown San Diego, the college offers more than
  100 majors, 100 certificate programs and 1,500
  classes each semester to 16,000 students.

4
Purpose of the Presentation

• The primary purpose of this presentation is to
  share the research methodology and innovative
  techniques for data analysis and instrument
  refinement for SLO assessment rather than to
  disseminate results of the study.
• A second purpose is to support the teaching of
  Basic Skills Math in community colleges.

5
Purpose of the Study

• Demonstrate SLO assessment cycle in the Math
  Department at San Diego City College
• Draw attention to measurement issues in assessing
  SLOs.
• Demonstrate methods of item analysis that has
  multiple advantages.

6
Research Design

• SLO assessment cycle
• A repeated measure design pre and post design

7
SLO Assessment in the Math Department
City College Student Learning Outcomes Assessment
Cycle 6 Column Form For
Developmental Math Program- Math 35, 95, 96
Year 2008/2009
8
Research Methodology

• Rasch Model
• Estimate both item difficulty and person ability
• Map both parameters on the same scale
• Inform instrument refinement
• Paired Sample t-test
• Statistically significant improvement from
  pretest to posttest

9
Target Population and Sample Size

• Developmental math courses Pre-Algebra,
  Beginning Algebra, and Intermediate Algebra.
• This totals to over 1000 students taking the
  developmental math courses
• In Fall 2008, we collected pre/post paired data
  from about 250 students

10
Data Collection Instrument

• Developed one pre test and a similar post test
• TestGen testbank software
• 8 multiple choice questions
• Topics of the questions
• A Sample Instrument

11
Data Collection Test Administration

• Pre-test administration
• Pre-test grading
• Post-test administration
• Post-test grading

12
Data Management

• For each student, itemized response per question
  and a total score were entered and paired in
  Excel
• Data then were exported to Winsteps for model
  fitting
• Estimates of item difficulty and student ability
  were analyzed in SPSS to compare pre test results
  to the post.

13
Data Analysis

• Fit itemized responses rather than the total
  scores with the Rasch Model to obtain estimates
  of item difficulty and student ability.

14
Data Analysis

• Map both item difficulties and person abilities
  on the same scale to produce the Item-Person Map.
• Compare Pre test and Post test
• Paired sample t-test
• Anchoring item difficulties

15
Measures

• Item difficulty
• Student ability

16
Results (Pre-algebra FALL 2008 DATA)

• Pre-test
• Item difficulty
• Student ability

17
Results (Pre-algebra FALL 2008 DATA)

• Pre-test
• Person-item map

18
Results (Pre-algebra FALL 2008 DATA)

• Post-test
• Item difficulty
• Student ability

19
Results (Pre-algebra FALL 2008 DATA)

• Post-test
• Person-item map

20
Results (Pre-algebra FALL 2008 DATA)

• Pre and post comparison
• Anchoring item difficulties to produce a new set
  of student ability estimates

21
Results (Pre-algebra FALL 2008 DATA)

• Pre and post comparison
• Anchoring item difficulties to produce a new set
  of student ability estimates

22
Results (Pre-algebra FALL 2008 DATA)

• Pre and post comparison
• Paired sample t test

23
Findings

• Results revealed that students scored
  statistically significantly higher in the post
  test compared to their performance in the
  pretest.
• Content areas that the instructors need to
  emphasize for teaching.
• Information for instrument refinement.

24
INSTRUMENT REFINEMENT
25
Use of Results for Programmatic Improvement

• Imbed the post test into the final exam to
  increase sample size. Also provide online version
  of pre test to collect data from online
  developmental classes.
• Rewrite or revise test questions based on results
  of item analysis.
• Identify difficult topics and disseminate the
  information to developmental math instructors for
  future teaching.
• Also disseminate the information to instructors
  of higher level math course for their preparation
  and planning.

26
Discussion

• Advantages of item analysis
• Solve measurement issues
• Conduct meaningful comparisons
• Bank good test items for constructing future
  tests
• Diagnostic function provides insight of the
  strength and weakness of student content
  knowledge.

27
Limitations of the Research

• Small sample size
• Small number of test items
• Item stability
• Generalizability of the results

28
Questions?