OYA Recidivism Risk Assessment (ORRA) OYA Recidivism Risk Assessment

1
OYA Recidivism Risk Assessment (ORRA)OYA
Recidivism Risk Assessment Violent
Crime(ORRA-V)
Research Evaluation
2
Defining Recidivism

• Felony adjudication (conviction) within 3 years
  of release from closed custody or commitment to
  probation.

3
Defining the Assessments

• ORRA OYA Recidivism Risk Assessment. Identifies
  the likelihood a youth will recidivate after
  release from closed custody or commitment to
  probation.
• ORRA-V OYA Recidivism Risk AssessmentViolent.
  Identifies the likelihood a youth will VIOLENTLY
  recidivate after release from closed custody or
  commitment to probation. Recognizes the
  propensity for violence or threatening crimes
  that may result in physical harm.

4
Why Develop a Risk Assessment

• Program evaluation
• Placement and treatment decisions
• Parole Decisions
• Sentencing practices
• RNA fails to differentiate risk populations
• Poor predictive accuracy

5
The Data

• 15,968 total youth
• Date range of population January 2005 to May 2007
• Four Youth Populations
• County Probation
• Committed to OYA Probation
• Released from OYA Close Custody Facility
• Released from OYA Close Custody Facility to
  Supervision in the adult system

6
Dozens of Variables were Considered

• Age at first referral
• Total prior sex offense referrals
• Total prior felony referrals
• Total prior theft referrals
• Total prior runaway referrals
• Total prior property referrals

7
Dozens of Variables were Considered (cont.)

• Total burglary referrals
• Total prior misdemeanor referrals
• Total prior robbery referrals
• Total prior violation referrals
• Total prior dependency referrals
• Total prior harassment referrals

8
Variables contributing to the ORRA Scores and
their effects

• Prior felony AOD referral (Y/N)
• Prior weapon referral (Y/N)
• Age
• Criminal mischief referral (Y/N)
• No. prior misdemeanor referrals
• No. prior theft referrals
• Adjudicated delinquent (Y/N)
• No. prior AOD referrals
• Current sex offense (Y/N)
• No. prior runaway referrals
• Gender (male higher risk)
• Interactions mischief referral by
• No. prior misdemeanors
• No. prior theft referrals
• No runaway referrals

• 25.5
• 21.2
• 4.6
• 83.2
• 10.3
• 5.2
• 21.6
• 11.1
• - 39.5
• 11.4
• 20.4
• Flattens out
• Increases
• Flattens out

9

• So What exactly does this mean?

10
Meet the Twins
Age 15
Male
1 Prior Misdemeanor Referral
3 Runaways
1 Prior Felony Drug Referral
11
Interpreting Scores

• Each youth has a score between 0 and 100
• The score approximates the probability that the
  youth will recidivate
• For example, a youth with a score of 40 has a
  40 probability they will recidivate
• This also means the youth has a 60 approximate
  score that the youth will NOT recidivate.

12
Model Accuracy

• Overall Accuracy for ORRA 73
• Accurate for all subpopulations
• Accuracy of 50 suggests poor predictive accuracy
• Accuracy of 100 suggests perfect predictive
  accuracy

13
Comparison of the Four Populations
14
Differences in Youth Populations
15
Predictive Accuracy
16
Interpreting Scores

• ORRA and ORRA-V scores can also be evaluated for
  a specific population
• The average score for a group of youth estimates
  the expected recidivism rate for the group

17
Program EvaluationActual vs. Expected Recidivism
Providers Serving 30 or more youth from 1/1/2000 to 12/13/2007 36-Month Recidivism Providers Serving 30 or more youth from 1/1/2000 to 12/13/2007 36-Month Recidivism Providers Serving 30 or more youth from 1/1/2000 to 12/13/2007 36-Month Recidivism Providers Serving 30 or more youth from 1/1/2000 to 12/13/2007 36-Month Recidivism
Treatment Provider Expected Rate Actual Rate Percent Increase or Decrease
Provider A 18.8 13.0 -31.0
Provider B 20.3 16.3 -19.5
Provider C 21.9 17.9 -18.2
Provider D 26.8 22.2 -17.1
Provider E 16.5 14.0 -15.4
Provider F 22.4 19.2 -14.3
Provider G 27.6 25.0 -9.3
Provider H 14.9 13.6 -8.4
Provider I 30.2 29.7 -1.7
Provider J 28.2 28.0 -0.7
Provider K 36.8 40.4 9.9
Provider L 24.5 27.1 10.6
Provider M 26.4 29.9 13.2
Provider N 26.1 30.0 14.7
Provider O 25.3 37.1 46.5

• Calculate risk scores
• Expected (based on average risk of youth served)
• Actual (based on recidivism of youth served)
• Determine Relative increase or decrease
• Facilitates meaningful comparisons across
  providers

18
Things Done and Things Still to Do

• Done -- Test for all OYA youth groups
• Males/Females
• Minorities
• Crime Type
• Still to Do -- Make the ORRA dynamic
• Incidence
• Revocations
• Programming

19
ORRA-V

• Used the same dataset
• Used violent recidivism a subset of
  recidivism
• Violent recidivism includes murder,
  arsonrobbery, assault, and burglary

20
Variables contributing to the ORRA-V Scores and
their effects

• Male
• Prior weapon referral (Y/N)
• No. prior misdemeanor referrals
• No. prior felony referrals
• Prior felony assault referral (Y/N)
• Prior felony theft referrals (Y/N)
• Misdemeanor theft referrals (Y/N)
• Prior curfew violation (Y/N)
• No. prior runaway referrals
• Interactions
• Weapons X felony theft
• Misdemeanor Referrals X felony referrals

• 178.1
• 62.0
• 13.5
• 31.3
• 32.1
• 36.1
• 20.1
• 22.1
• 8.6
• Flattens out
• Flattens out

21
Differences between the ORRA and the
ORRA-VVariable ORRA ORRA-V
Male 20 178
Weapon offense 21 62
Misdemeanor referrals 10 13
Runaway referrals 11 31
Felony referrals 9
Felony assault referrals 32
Felony theft referrals 36
Misdemeanor theft referrals 20
Curfew violation 22
Felony AOD referral 26
Age 5
Mischief referral 83
Number theft referrals 5
Prior adjudication 22
Number AOD referrals 11
Current sex offense - 40
22
OVIRA and ONIRA

• OVIRA measures the likelihood a youth will engage
  in a violent act in the first six months of
  closed custody
• ONIRA measures the likelihood a youth will engage
  in numerous non-violent incidents in the first
  six months of closed custody

23
Data for OVIRA and ONIRA

• Youth admitted to OYA between November 2007 and
  December 2009
• N 1,258
• 90 male and 10 female
• 27 property crime, 25 sexual offenses, and 9
  robbery
• 64 YCF, 11 DOC, and 11 revoked

24
Variables considered for OVIRA and ONIRA

• ORRA and ORRA-V
• RNA data aggression, drugs/alcohol, mental
  health, employment, relationships, attitudes,
  etc.
• Gender
• Age
• Sexual offender
• Special education and learning disability
• Other variables

25
OVIRA OYA Violent Incident Risk Assessment

• Violence considered an assault or peer fight
  resulting in isolation/segregation
• Considered immediately threatening to life,
  health, or facility safety, security, or good
  order.

26
ONIRA OYA Nuisance Incident Risk Assessment

• Considered four or more non-violent incidents in
  the first months of closed custody

27
Variables contributing to OVIRA and ONIRA
scoresVariable OVIRA ONIRA
Age at admission -20 - 27
Male -43
SED 55 139
Sex offender -45 - 49
Mental health protective - 9
Full relationship risk 29
Belief in fighting/aggression 49
RNA prescreen social score - 11
Mental health risk 28
Aggression protective - 22
Parental authority/control 50
ORRA 1224 (HR13.2)
ORRA-V - 95 (HR.05)
28
Typologies
29
Typologies
C
E
A
F
B
D
30
Type A Description

• Highest need population
• AOD use is high both current and historical
• Poor relationships and likely lack relationship
  skills
• Highest on aggression and attitude issues
• History of Mental Health ADD/ADHD or mental
  health diagnosis recommend analysis of RNA
  items 15.5 and 15.6 to differentiate ADD/ADHD
  versus Formal MH Diagnosis
• Education issues are prominent recommend
  analysis of RNA item 3.1 for potential
  responsivity issues
• 3.1 Special Education or Formal Diagnosis of
  Special Education Need (LD, SED, MRDD Indicators)

31
Treatment Recommendation

• Estimated to require longest dosage of treatment
  (e.g., 12-18 months)
• Group may require more stabilization than other
  groups due to co-occurring mental health and
  learning concerns
• AOD Treatment (longer in duration due to
  persistency)
• MH treatment with QMHP
• Educational intervention
• Social Skills/Relationship Skills development
  (intensive)
• Engagement in prosocial activities that can
  foster protective factors
• Potential family therapy component
• Aggression Replacement Training (intensive)
• Cognitive Behavioral program to address thinking

32
Type E Description

• 66 of this cluster is SO
• Highest on protective factors
• Low need for MH ADD/ADHD or mental health
  diagnosis recommend analysis of RNA items 15.5
  and 15.6 to differentiate ADD/ADHD versus Formal
  MH Diagnosis
• Education issues are low recommend analysis of
  RNA item 3.1 for potential responsivity issues
• 3.1 Special Education or Formal Diagnosis of
  Special Education Need (LD, SED, MRDD Indicators)

33
Treatment Recommendations (Type E)

• Sex Offender Treatment when appropriate
  (Abbreviated Kaufman or general cognitive
  behavioral treatment)
• Capitalize on whatever activities youth engaged
  in prior to coming as leverage for treatment
  engagement
• Seek opportunities for continued engagement

34
Optimal Length of Stay

• Calculated length of stay in months
• Plotted LOS against recidivism for the overall
  sample
• On average, providers reduce recidivism by
  approximately 3 per month of supervision
• But, there may be a window of time where
  providers are most effective

35
Program Evaluation Continuum
36
Summary

• ORRA
• ORRA-V
• OVIRA
• ONIRA
• Typology (being completed)
• Optimum dose (next project)
• Program continuum (being developed)
• LOS report
• Recidivism report
• Timing study for JJPOs
• Revocation (being completed)
• Culture climate survey (data collection
  completed)
• Staff-management/leadership survey (data
  collection now)
• PREA identifying vulnerable youth (surveyed
  thru October)

37
(No Transcript)
38
Close Custody Populations Making comparisons
while controlling for risk
39
Why

• Problems with the RNA
• Not valid for OYA females
• Approximately 85 of the youth in Close Custody
  were High Risk little practical information
• The Area Under the Curve (AUC) was .56
• DOC had the solution
• Methodology for developing risk tool based on
  local data
• The AUC for their risk tool was .78

40
How

• Methodology
• Subjects
• N 28,431 dispositions (19,309 unique youth)
• Qualifying events occurred between 1/1/2005 and
  5/14/2010
• Youth qualified if they were
• Placed on county probation
• Committed to OYA probation
• Released from an OYA close custody facility
• Release from OYA close custody to supervision in
  the adult system

41
Whats Next

• ONIRA OYA Nuisance Incident Risk Assessment
• OVIRA OYA Violent Incident Risk Assessment

42
How

• Methodology continued
• Omitted disposition records of youth
• Supervised under interstate compact
• Returned to DOC to complete their sentences in
  adult institutions
• Committed to OYA or county probation who were
  subsequently committed to an OYA YCF without
  recidivating
• Randomly selected one disposition per youth
• Final dataset N 15,986

43
How

• Methodology continued
• Dependent (Outcome) Variable
• Recidivism event OYA official recidivism measure
• Felony Adjudication or Conviction
• Disposition of formal supervision
• Groups
• Tracking Dates
• Tracking Periods 12-, 24-, 36-, 48-Month

44
How

• Methodology continued
• Independent Variables
• Over 50 starting variables
• Bootstrap Re-sampling
• Run 1000 randomly sampled logistic regressions
  for each tracking period
• Lists the proportion of time each variable is
  significantly related to the outcome variable
• Selected the top 30 of the variables to develop
  the model
• Run stepwise Logistic Regression for each
  tracking period

45
How

• Methodology continued
• Developing the Model
• Run stepwise Logistic Regression for each
  tracking period
• Determine the concordance rate for each model
• Test for interactions
• Run stepwise Logistic Regression for each
  tracking period including significant interaction
  variables

46
How

• Methodology continued
• Selecting and refining the final model
• 36-Month tracking period had the highest
  concordance rate (.73) and included
• 12 predictor variables
• 3 interaction terms

47
How

• Results
• Model Accuracy
• AUC .72
• Estimates Actual Recidivism

48
How

• The Model

PREDICTOR VARIABLES VALUES ODDS RATIO
Prior felony drug or alcohol referral No 0, Yes 1 1.255
Prior weapon offense referral No 0, Yes 1 1.212
Age at start tracking Age at probation start or release to community from close custody 1.046
Prior criminal mischief referral No 0, Yes 1 1.832
Total prior misdemeanor referrals Sum (maximum 20) 1.103
Total prior theft referrals Sum (no maximum) 1.052
Prior delinquency adjudication No 0, Yes 1 1.216
Total prior drug or alcohol referrals Sum (no maximum) 1.111
Current sex offense disposition No 0, Yes 1 0.605
Total prior runaway referrals Sum (maximum 20) 1.114
Total prior felony referrals Sum (maximum 6) 1.204
Male Female 0, Male 1 2.628
Interaction prior criminal mischief referral total prior misdemeanor referrals Product of the two variable terms specified 0.897
Interaction prior criminal mischief referral total prior theft referrals Product of the two variable terms specified 1.108
Interaction prior criminal mischief referral total prior runaway referrals Product of the two variable terms specified 0.935
49
What for

• Interpreting ORRA Scores
• Each youth get a score between 0 and 1
• The score represents the probability that the
  youth will recidivate
• For example, a youth with a score of .42 has a
  42 probability they will recidivate
• The average score for a group of youth estimates
  the expected recidivism rate for the group
• For example, the average ORRA score for females
  on OYA probation was 13.1 and the actual
  recidivism rate was 13.0.

50
What for

• ORRA has multiple uses
• Placement and treatment decisions
• Parole decisions
• Program evaluations
• Sentencing practices
• Foundation for future improvement in risk
  assessment

51

• What for
• Making comparisons while controlling for risk

52

• What for
• Making comparisons while controlling for risk

53

• What for
• Making comparisons while controlling for risk

54
Other risk equations

• ORRA
• ORRA-V
• Risk of being involved in a violent incident in
  the first year in OYA close custody

55
Implementing ORRA ScoresAn Example

• Used ORRA Scores in Evaluating the Effectiveness
  of Residential Programs
• Is the actual recidivism rate different than the
  predicted recidivism rate?
• Is there an optimal length of stay?
• With which youth is a program most effective?
• All youth in residential programs from 2000 to
  2007
• Used official OYA definition for recidivism

56
Actual vs. Expected Recidivism
Providers Serving 30 or more youth from 1/1/2000 to 11/1/2007 36-Month Recidivism Providers Serving 30 or more youth from 1/1/2000 to 11/1/2007 36-Month Recidivism Providers Serving 30 or more youth from 1/1/2000 to 11/1/2007 36-Month Recidivism Providers Serving 30 or more youth from 1/1/2000 to 11/1/2007 36-Month Recidivism Providers Serving 30 or more youth from 1/1/2000 to 11/1/2007 36-Month Recidivism
Treatment Provider Youth Served Expected Rate Actual Rate Percent Increase or Decrease
A 80 14.7 13.8 -6.1
B 125 14.0 13.6 -2.9
C 345 27.4 30.4 10.9
D 287 25.8 29.3 13.6
E 211 32.0 37.4 16.9
F 306 37.0 44.1 19.2
G 490 31.1 37.1 19.3
H 141 15.4 18.4 19.5
I 641 21.9 26.2 19.6
J 141 32.9 39.7 20.7
K 211 19.6 23.7 20.9
L 184 21.9 26.6 21.5
M 121 25.0 33.9 35.6
N 117 28.6 39.3 37.4
O 480 25.2 35.2 39.7

• Calculated risk scores
• Expected (based on average risk of youth served)
• Actual (based on recidivism of youth served)
• Determined Relative increase or decrease
• Facilitates meaningful comparisons across
  providers

57
Next step? Right YouthRight Program

• In depth analysis about who programs are most
  effective with
• Potential variables include sex, age, offense
  type
• In addition to understanding which youth are most
  effectively served by a specific program, this
  analysis may identify gaps and determine which
  youth are not served well by current provider
  resources

58
Questions

• Contact Research
• Lance Schnacker [phone]
• Lance.schnacker_at_oya.state.or.us
• Paul Bellatty
• Paul.T.Bellatty_at_doc.state.or.us

59
Current OYA Population
60
Why

• Problems with the RNA
• Not valid for OYA females
• Approximately 85 of the youth in Close Custody
  were High Risk little practical information
• The Area Under the Curve (AUC) was .56
• DOC had the solution
• Methodology for developing risk tool based on
  local data
• The AUC for their risk tool was .78

61
How

• Methodology
• Subjects
• N 28,431 dispositions (19,309 unique youth)
• Qualifying events occurred between 1/1/2005 and
  5/14/2010
• Youth qualified if they were
• Placed on county probation
• Committed to OYA probation
• Released from an OYA close custody facility
• Release from OYA close custody to supervision in
  the adult system

62
How

• Methodology continued
• Omitted disposition records of youth
• Supervised under interstate compact
• Returned to DOC to complete their sentences in
  adult institutions
• Committed to OYA or county probation who were
  subsequently committed to an OYA YCF without
  recidivating
• Randomly selected one disposition per youth
• Final dataset N 15,986

63
How

• Methodology continued
• Dependent (Outcome) Variable
• Recidivism event OYA official recidivism measure
• Felony Adjudication or Conviction
• Disposition of formal supervision
• Groups
• Tracking Dates
• Tracking Periods 12-, 24-, 36-, 48-Month