Accountability Reporting for California Community Colleges

1
Accountability Reporting for California Community
Colleges

• Patrick Perry
• Vice Chancellor of Technology, Research, Info.
  Systems
• CCC Chancellors Office

2
Data Preamble

• Information is the currency of democracy.
  -Thomas Jefferson
• Get your facts first, then you can distort them
  as you please. -Mark Twain
• In the twenty-first century, whoever controls
  the screen controls consciousness, information
  and thought. -Timothy Leary

3
The CCC System

• 109 campuses, 72 districts, all locally governed
• 2.6 million students (annual unduplicated)
• 1.1 million FTES (annual)
• 35 white half over age 25 70 part-time
• No admissions requirements
• 20/unit 40 get fees waived
• Highest participation rate of any CC system in
  US 25 of all CC students are CCC

4
Participation (and Fees)
5
CCC Chancellors Office

• Weak authority powers vested locally
• Unitary MIS data collection (1992)
• Student, faculty, course, section, session, grade
  level detail
• Data collected end of term, 3x/yr
• Used for IPEDS, apportionment, accountability,
  research, online data mart

6
History of CCC Accountability

• Simple reporting, fact books until 1998
• 1998 State provides 300m ongoing in exchange
  for accountability reporting
• Partnership for Excellence was born
• CCC developed report in isolation
• CCC allowed to determine adequate progress
• Contingent funding never triggered
• Used 5 metrics to measure system and
  college-level performance

7
PFE Metrics

• Annual volume of transfers to CSU/UC
• Annual volume of awards/certificates
• Rate of successful course completions
• Annual volume of Voc. Ed. Course completions
• Annual volume of basic skills improvements (lower
  to higher level)
• 4 of 5 are volume metrics, only 1 rate

8
The State Said

• Your metrics allow for no adequate college
  comparisons
• Your method of determining adequate progress is
  suspicious
• You only look good because you are growing
• Partnership over (2001), but keep reporting,
  (until 2004)
• we have to spend your money buying energy from
  Enron

9
What Happened Next

• Gov. Gray Davis recalled for spending money
  buying energy from Enron
• Replaced by
• The Governator

10
The Governator

• Likes Community Colleges
• Comes from a country that has European academic
  bifurcation (Austria)-university vs trade paths
• Attended Santa Monica Community College
• Took ESL, PE, bookkeeping, micro/macroeconomics
• Transferred to U. Wisconsin-Superior

11
And Arnold Said

• We shall haves deez accountabeelity seeztem for
  de community collegez.
• A bill was passed to create the framework, and
  eventually the framework was enacted.
• Named Accountability Reporting for Community
  Colleges (ARCC).

12
Arnold Said

• There shall be no pay for performance, but there
  will be the ability to compare performance.

13
We Said

• Some metrics will be system only others will be
  at college-level
• College metrics will be rates (to mitigate size
  for comparison)
• No rankingswe will compare colleges against
  their peers
• No ARCC is a dashboard accountability
  report.

14
Arnold Said

• Colleges need to address their performance
  annually to the State.

15
We Said

• Colleges are more responsive to their local
  district Board annual requirement to take local
  ARCC results to local Board and submit minutes to
  State
• Colleges must submit 500 word response, which
  becomes a part of the final report.

16
Arnold Said

• The report shall be done in collaboration with
  the State, not in isolation.

17
We Said

• The Dept of Finance, Leg Analyst, and Secretary
  of Education shall be a part of the technical
  advisory committee (along with CCC researchers
  and stakeholders).
• We will either succeed or fail together.
• This was a really smart move.

18
ARCC

• The Model
• Measures 4 areas with 13 metrics
• Student Progress Achievement-Degree/Certificate/
  Transfer
• Student Progress Achievement-Vocational/Occupati
  onal/Workforce Dev.
• Pre-collegiate improvement/basic skills/ESL
• Participation
• Process is not measured

19
Student Prog. Achievement Degree/Cert/Xfer

• College
• Student Progress Achievement Rate(s) (SPAR)
• 30 units Rate for SPAR cohort
• 1st year to 2nd year persistence rate
• System
• Annual volume of transfers
• Transfer Rate for 6-year cohort of FTFs
• Annual of BA/BS grads at CSU/UC who attended a
  CCC

20
Student Prog. Achievement Voc/Occ/Wkforce Dev

• College
• Successful Course Completion rate vocational
  courses
• System
• Annual volume of degrees/certificates by program
• Increase in total personal income as a result of
  receiving degree/certificate

21
Precollegiate Improvement/Basic Skills/ESL

• College
• Successful Course Completion rate basic skills
  courses
• ESL Improvement Rate
• Basic Skills Improvement Rate
• System
• Annual volume of basic skills improvements

22
Participation

• College
• None yetbut coming.
• System
• Statewide Participation Rate (by demographic)

23
Major Advancements of ARCC

• Creating a viable alternative to the GRS Rate for
  grad/transfer rate.
• Finding transfers to private/out of state
  institutions.
• Doing a wage study.
• Geo-mapping district boundaries.
• Creating peer groups.

24
Defining Grad/Transfer Rate

• Student Progress Achievement Rate (SPAR Rate)
• IPEDS-GRS for 2-yr colleges stinks
• No part-timers
• How do you define degree-seeking?
• Tracking period too short
• Outcomes counting methodology terrible
• AA/AS/Cert counted before transfer
• Transfer to 2-yr college is counted

25
SPAR Rate

• Defining the cohort
• Scrub first-time by checking against past
  records (CCC, UC, CSU, NSC)

26
SPAR Rate

• Define degree-seeking behaviorally for CC
  populations
• Not by self-stated intent this is a poor
  indicator
• Behavior did student ever attempt
  transfer/deg-applicable level math OR English (at
  any point in academic history)
• Students dont take this for fun

27
Defining Degree-Seeking Behaviorally

• Separates out remedial students not yet at
  collegiate aptitude
• Measure remedial progression to this threshold
  elsewhere
• Creates common measurement bar of student
  aptitude between colleges
• Same students measuredviable comparison

28
SPAR Rate-Unit Threshold

• CCC provides a lot of CSU/UC remediation
• Lots of students take transfer math/Eng and
  leave/take in summer
• Should not count these as success or our
  student
• Set minimum unit completed threshold (12) for
  cohort entrance
• Any 12 units in 6 years anywhere in system

29
SPAR Denominator

• First-Time (scrubbed)
• Degree-seeking (at any point in 6 years, attempt
  transfer/degree applicable math or English)
• 12 units (in 6 years)
• This represents about 40 of students in our
  system

30
SPAR Numerator

• Outcomes the State wants
• Earned an AA/AS/certificate OR
• Transfer to a 4-yr institution OR
• Become transfer-preparedOR
• Completed 60 xferable units
• Became transfer-directed
• Completed both xfer level math AND English
• No double-counting, but any outcome counts
• SPAR Rate51

31
Tracking Transfers

• SSN-level matches with CSU, UC
• Natl Student Clearinghouse for private,
  proprietary, for-profit, out of state
• Match 2x/yr, send all records since 1992
• Update internal xfer bucket
• Works great for cohort tracking
• Needed method for annual volume

32
Tracking Transfers

• Annual Volume of Transfers
• CSU/UC they provide these figures based on their
  criteria
• We didnt want to redefine this
• Private/Out of State NSC cross-section cut
  method
• Validated against CSU/UC xfers from NSC source
• Added another 30 to annual volumes

33
(No Transcript)
34
(No Transcript)
35
Transfer Sector of Choice
36
Demography of Transfer
37
The Rise of The Phoenix
38
Who Transfers to Phoenix?
39
Wage Study

• What was the economic value of the degrees
  (AA/AS/certificate) we were conferring?
• Required data match with EDD
• Had to pass a bill changing EDD code to allow
  match

40
Wage Study

• Take all degree recipients in a given year
• Subtract out those still enrolled in a CCC
• Subtract out those who transferred to a 4-yr
  institution
• Match wage data 5 years before/after degree

41
Wage Study

• Separate out two groups
• Those with wages of basically zero before degree
• Those with gt0 pre wage
• The result The Smoking Gun of Success

42
(No Transcript)
43
Mapping Districts

• CC Districts in CA are legally defined, have own
  elections, pass own bonds
• We did not have a district mapping for all 72
  districts
• So we couldnt do district participation rates

44
Mapping Project

• Get a cheap copy of ESRI Suite
• Collect all legal district boundary documents
• Find cheap laborno budget for this

45
(No Transcript)
46
Peer Grouping

• Peers historically have been locally defined
• My neighbor college
• Other colleges with similar demography
• Other colleges with similar size

47
Peer Grouping

• Taking peering to another level
• Peer on exogenous factors that predict the
  accountability metrics outcome
• Thus leaving the endogenous activity as the
  remaining variance
• Cluster to create groups
• We picked 6 clusters, with a min of 3 in a
  cluster
• Each metric produces different factors, peers,
  clusters

48
Peer Grouping Example

• Peering the SPAR Rate
• 109 rates as outcomes
• Find data for all 109 that might predict
  outcomes/explain variance
• Perform regression and other magical SPSS things
• See how high you can get your R2

49
Finding Data

• What might affect a grad/transfer rate on an
  institutional level?
• Student academic preparedness levels
• Socioeconomic status of students
• First-gen status of students
• Distance to nearest transfer institution
• Student age/avg unit load

50
Finding Data

• We had to create proxy indices for much of these
  (142 tried)
• GIS system geocode student zipcode/ZCTA
• Census lots of data to be crossed by zip/ZCTA
• Create college service areas based on weighted
  zip/ZCTA values
• Different than district legal boundaries

51
(No Transcript)
52
Finding Data

• The Killer Predictor
• Bachelor Plus Index, or what of service area
  population of college has a bachelors degree or
  higher
• Bachelor Plus Index a proxy for
• First gen
• Academic preparedness
• Socioeconomic status
• Distance to nearest transfer institution

53
Peering SPAR Rate

• Exogenous factors that predict SPAR Rate
• Bachelor Plus Index
• older students
• students in basic skills
• R2 .67
• Whats left is implied institutional variance
• Demo

54
Peering Whats Bad

• Its complex and somewhat confusing and labor
  intensive.
• Colleges traditional notion of peer is shaken
• Multiple peers for multiple metrics can change
  every year
• You could do well vs. State average, increasing
  over time, but last in your peer group

55
Peering Whats Good

• Its complex and somewhat confusing
• You will likely look good in some areas, OK in
  others, and low in others
• Its not very likely anyone will be high or low
  in all 6 metrics
• It eliminated rankings.

56
The ARCC Report

• Is almost 800 pages.
• Comes out every March.
• Takes 4 PYs to complete (about 6 months/yr)
• Is generally regarded highly in CA academic and
  Legislative circles.
• DOF and LAO and Sec. of Ed love it.
• Local Trustees/Boards love it.

57
The ARCC Collaboration

• Has brought the system more money
• 33 mil in basic skills
• Increased noncredit reimbursement rates by
  300/FTE
• Has brought about trust between system and State
  stakeholders.
• Has educated both sides tremendously.

58
No More Girlie-Man Accountability!