Adult Consumer Assessments of Care in New York

1
Adult Consumer Assessments of Care in New York

• Chip Felton, Senior Deputy Commissioner
• Jeff Kirk
• Doug Dornan
• New York State Office of Mental Health
• Center for Information Technology and Evaluation
  Research

2
Overview

• Development of the NYSOMH Mental Health Services
  Survey (MHSS) and Brief Quality of Life
  Assessment (BQLA)
• Annual Adult Consumer Survey Project
• Recovery Oriented Systems Indicators (ROSI)
  Project

3
Development of the NYSOMH Mental Health Services
Survey (MHSS)

• Version 1 (1996-98)
• MHSIP-informed items, response choices and
  domains (access, approp/support for recovery,
  outcomes/perceived efficacy) BASIS-32
  Rosenberg SE SF12 self-help involvement big
  mail survey (150 items!)
• Administered 3x (6 month intervals) to over 6,000
  adults receiving state-operated outpatient
  services response rate 47.
• Developed web-based report card for management
  and print report card for recipients
• Version 2 (1999-present)
• Shortened instrument by limiting content to
  assessment of care domains (access,
  appropriateness, perceived efficacy)
• Adopted neutral wording for items and 4-point
  poor - excellent scale in effort to eliminate or
  reduce satisfaction effect (ceiling effect)
• Developed in anticipation of need to monitor
  Medicaid managed care plans (but plans ultimately
  never implemented)
• Several administrations
• 1) 2001-2002, NYC area state-operated outpatient
  services (approx 3,000) (methods online, peer
  specialist guided)
• 2) 2003-2004, statewide, 5-county sample, group
  sessions (approx 300/year)
• Succeeded in getting lower ratings ! -)

4
Item Construction Access Examples

• MHSIP (and MHSS version 1)
• Staff were willing to see me as often as I felt
  it was necessary
• I do better in school and or work
• MHSS How would you rate
• The ease of getting services when you needed
  them
• Helpfulness of services on your involvement in
  work or school

5
Development of the NYSOMH Brief Quality of Life
Assessment (BQLA)

• Rationale for development of a brief QOL
  instrument
• QOL an important outcome and social indicator
  domain
• Existing measures not very parsimonious, hence
  not suitable for routine large scale
  surveillance, yet much content never used (e.g.
  Lehmans QOL interview 1 hour, face-face,
  objective QOL and subjective
  satisfaction-like QOL in multiple life domains
  published studies rarely reported data other than
  subjective item means subjective items
  highly inter-correlated)
• Combination of experience using and analyzing
  results from longer QOL assessments, analyses of
  item correlations indicating much redundancy,
  desire for very brief measure yielded global
  indicator approach to QOL assessment.
• BQLA developed in late 1990s 1 global rating
  item per life domain 4-point poor-excellent
  scale 10-15 items
• BQLA now administered along with the MHSS

6
Example ItemsBrief Quality of Life Assessment

• Overall, how would you rate
• Your involvement in work, employment
• How you feel about yourself, your self-esteem

7
Annual Adult Consumer Survey Project

• Need to conduct an annual consumer assessment of
  care survey
• Current methodology to survey consumers across
  New York State developed in 2003

8
Survey Methodology

• Stratified random sample of counties (2 counties
  in each of 4 regions)
• Partnered with local peer groups to conduct
  surveys
• Trained peer advocates in survey administration
• Local peer organizations assisted in recruiting
  participants and setting up evaluation meetings

9
Survey Instruments(2004 Survey)

• Mental Health Services Survey (MHSS)
• 32 fixed items 3 open-ended response item
• MHSIP
• 28 fixed items 1 open-ended response item
• Quality of Life Assessment (QOLA)
• 15 fixed items 1 open-ended response item
• Recovery Oriented Systems Indicators (ROSI)
• 42 fixed items 1 open-ended item

10
Comparison Between MHSS and MHSIP

• Comparison of Content
• Mean and Standard Deviation of Subscales Consumer
  Responses
• Percent Positive Responses
• Cross tabs
• Cronbachs Alpha on Subscales
• Factor Analysis
• Correlation with QOLA and ROSI

11
Comparison of Content MHSS and MHSIP
MHSS MHSIP (v1.1) 28-items
Domains Access (10 items) Appropriateness (12) Outcomes (8) Overall Assessment (2) Access (6 items) Appropriateness (11) Outcomes (8) General Satisfaction (3)
Response Scale 1 Poor 2 Fair 3 Good and 4 Excellent 9 Does not apply 1 Strongly Agree 2 Agree 3 Neutral 4 Disagree 5 Strongly Disagree 9 not applicable
12
Means and Standard Deviations for Subscales MHSS
and MHSIP (Adjusted Scores for 1-5 scale, N
651)
MHSS MHSS MHSIP MHSIP
Mean STD Mean STD
Access 3.60 .862 3.71 .847
Appropriateness 3.72 .879 3.81 .758
Outcomes 3.59 .890 3.77 .816
Overall Assessment/ Satisfaction 3.85 1.02 3.89 .963

13
Percent Positive Responses MHSS and
MHSIP(Adjusted Scores for 1-5 scale, N651)
MHSS MHSIP
Access 60 68
Appropriateness 64 72
Outcomes 55 70
Overall Assessment/ Satisfaction 76 74

14
Cross tabs MHSS and MHSIPAdjusted scores for
1-5 scale, 3.5() lt3.5(-)
Comparison of Responses Comparison of Responses Comparison of Responses Comparison of Responses Comparison of Responses
Sub-Scale / -/- Agree MHSS / - MHSIP - MHSS / MHSIP
Access 327 (51) 144 (22) 73 63 (10) 113 (17)
Appropriate 369 (57) 131(20) 77 45 (7) 100 (16)
Outcomes 310 (48) 143 (22) 70 48 (7) 144 (23)
Overall Assessment 420 (66) 98 (15) 81 68 (11) 53 (8)
15
Cronbachs alpha MHSS and MHSIP Subscales
(2003-2004 data, N 651)
MHSS ? MHSIP ?
Access .898 .852
Appropriateness .926 .908
Outcomes .888 .907
Overall Assessment/ Satisfaction .795 .883
One item was added to the 2004 MHSS survey to assess consumer inclusion in treatment planning One item was added to the 2004 MHSS survey to assess consumer inclusion in treatment planning One item was added to the 2004 MHSS survey to assess consumer inclusion in treatment planning
16
MHSS Factor Analysis (1 factor)
17
MHSIP Factor Analysis (1 Factor)
18
Pearson Correlations of Scales
MHSS MHSIP QOLA
MHSS ---
MHSIP .75 (n650) ---
QOLA .53 (n387) .50 (n386) ---
ROSI .72 (n388) .77 (n387) .48 (n387)
plt.01 plt.01 plt.01 plt.01
19
Conclusions Next Steps

• MHSS performs equally or better than the MHSIP
  (v.1.1)
• Scales have comparable variability but MHSS has
  greater variability
• The scales have similar consumer ratings, but
  MHSS may be less likely to illicit positive
  responses
• Both have good internal consistency
• Both scales measure a single construct, but
  outcomes may be considered a distinct factor
• Both are correlated with the QOLA and the ROSI
• Re-evaluate MHSS against MHSIP 2.0
• Implement web-based version of MHSS in
  state-operated facilities (pilot scheduled for
  May 2005)

20

• Recovery Oriented Systems Indicators (ROSI)
  Project

21
Mental Health RecoveryWhat Helps What Hinders

• A means to assess the recovery orientation of
  state and local mental health systems
• 10 focus groups in nine states
• 1000 pages of transcripts qualitatively analyzed,
  coded, and reduced to a 42 item self-report
  survey and a 30-item administrative profile,
  which together combine to make the ROSI

22
Mental Health RecoveryWhat Helps What Hinders

• Survey and administrative profile are
  consumer-driven and recovery focused
• ROSI is compatible with either MHSS or MHSIP
  (v1.1)
• Correlated with MHSS and MHSIP
• Not designed to replace, but to augment MHSS or
  MHSIP

23
Mental Health RecoveryWhat Helps What Hinders

• Factor analyses of the prototype test led to 8
  dimensions in assessing recovery orientation (N
  219 in 7 states)
• Person-centered decision making choice
• Self-care wellness
• Basic life resources
• Invalidated personhood
• Access
• Meaningful activities roles
• Staff treatment knowledge
• Peer advocacy

24
Mental Health RecoveryWhat Helps What Hinders

• Selected results form preliminary analysis of the
  ROSI NYS Self-Report Pilot
• Staff use pressure, threats, or force in my
  treatment 76 responded often, almost always,
  or always
• Mental health staff interfere with my personal
  relationship 73 responded often, almost
  always, or always

25
Mental Health RecoveryWhat Helps What Hinders

• Selected results form preliminary analysis of the
  ROSI NYS Self-Report Pilot (cont.)
• Staff see more as an equal partner in my
  treatment program 56 agreed often, almost
  always, or always
• Mental health staff help me build on my
  strengths 59 agreed often, almost always, or
  always
• Staff see me as a whole person 80 agreed or
  strongly agreed

26
Mental Health RecoveryWhat Helps What Hinders

• Next Steps
• National dissemination and pilot of the ROSI

27
(No Transcript)