IAPSRS Annual Conference

1
IAPSRS Annual Conference

• June 10, 2002
• 130-500 p.m.
• Toronto

2
CRITICAL CHARACTERISTICS OF COMMUNITY SUPPORT
PROGRAMS FOR PEOPLE WITH A SEVERE MENTAL ILLNESS

• Progress on the Road to Measurement and
  Understanding

3
Presenters

• Ross Norman, Ph.D. Clin.Psych., London Health
  Sciences Centre, University of Western Ontario,
  London, Ontario
• Bonnie Kirsh, Ph.D. , University of Toronto,
  Toronto, Ontario
• Ellen Tate, M.A., Centre for Addiction and Mental
  Health, Toronto, Ontario

4
Agenda

• Introductions
• Background and rationale
• Identification of important domains and small
  group work
• Break
• Development of specific items and small group
  work
• Where we go from here
• Discussion and wrap-up

5
Learning Objectives

• To understand the rationale for identifying
  critical program characteristics.
• To understand the process used for identifying
  major domains of characteristics.
• To examine agreement between results of a formal
  approach to identifying important domains and
  participants perceptions based on experience.
• To identify and appraise methodological
  challenges in developing and refining items.
• To understand potential applications of the
  instrument being developed.

6
Research Team Project Staff

• Brian Rush
• Ross Norman
• Bonnie Kirsh
• Cam Wild
• Steve Lurie

• Ellen Tate
• Meghan Prosser

7
FUNDING

• Ontario Mental Health Foundation (OMHF)
• One of six projects under the larger provincial
  Community Mental Health Evaluation Initiative
  (CMHEI) concerning the evaluation of community
  mental health programs and supports
• Five year funding (1999-2004)

8
COLLABORATIVE PROJECT

• Research Committee of the Ontario Federation of
  Mental Health and Addictions Programs
• Centre for Addiction and Mental Health
• Other researchers
• Advisory Committee

9
Advisory Committee Composition

• Range of Stakeholders
• Consumers, Family Members, Service Providers from
  Community
• Researchers
• Geographic diversity
• Range of program backgrounds and experience

10
Advisory CommitteeRole

• Advice including conceptual and methodological
  issues, user friendliness, etc.
• Identification of limitations and potential
  problems.
• Facilitation of pilot studies.
• Review and consultation regarding each phase of
  the plan and its results.

11
Why Is This Important?
12
TRADITIONAL COMPARATIVE EVALUATION
Program Model B
Program Model A
Outcomes
Outcomes
13
TRADITIONAL PROGRAM EVALUATION

• INTERPRETATIVE PROBLEMS
• Adherence to models?
• Overlap in content across models?
• Differences in programs unrelated to models
• Non-program content contributors to outcome?
• Small sample sizes

14
Other Factors to Consider

• Stage and nature of illness (acute, recovery,
  etc.)
• Age of person
• Length of experience with illness
• Individual interests and abilities
• Social support
• Community context

15
GOALS OBJECTIVES

• To develop an instrument package that will
  measure critical program characteristics
• Longer-term goal is to help explain outcomes and
  offer practical, informed advice on how to
  improve services to achieve better outcomes

16
Multiple Perspectives

• Consumers
• Family
• Service providers
• Program managers

17
PROJECT OVERVIEW 5 PHASES

• Phase I ? Determining Content Domains
• Phase II ? Initial Development of Items
• Phase III ? Pre-testing
• Phase IV ? Pilot-testing (current)
• Phase V ? Product Development

18
Whats Been Done

• Phase I ? Determining Content Domains
• qualitative analysis of literature review and
  interviews with consumers, family members, and
  service providers
• Phase II ? Initial Development of Items
• development of package of questionnaires
• Phase III ? Pre-testing
• 6 programs -

19
Qualitative Research

• Assumes there is not one single objective reality
• Attempts to understand phenomena in terms of the
  meanings people bring to them
• Based on experiences, "moments and meanings" in
  people's lives

20
Phase I ? Determining Content Domains

• Qualitative analysis of 42 semi-structured
  interviews with consumer/survivors, family
  members, and service providers
• Purposive selection of interviewees to reflect
  the diverse experience of living in urban, rural
  or frontier areas, of different service models,
  and cultural affiliation in Ontario
• Grounded theory used to analyze text and identify
  themes and domains

21
Grounded Theory

• Theory is generated inductively, i.e., grounded
  in data
• Data analysis constant comparative method
• Unitizing
• Categorizing
• Forming themes

22
Content of Interviews

• The type of community support program a person is
  involved with and what this provides for
  consumers
• How consumers became involved in the program
• How the program benefits consumers
• Specific program components they feel are most
  helpful to consumers
• How specified program components work in practice
• How staff of the program are helpful to consumers
• Other ways in which the program might be helpful

23
Overview of Literature Selection Review

• Included multiple perspectives
• Selection of key words
• Limited by date country (Canada, United States,
  Britain, Australia)
• Reviewed 7000 citations to select articles
  according to specific criteria
• Categorized by type of article
• Selected approximately 200 articles

24
Overview of Process
25
Overview of Process
26
Sample of Quotes Basic Needs

• " Yeah and some people... the sense of
  achievement and sense of autonomy and the fact
  that they have their own apartment and they're
  able to manage it in terms of self esteem, it's
  just tremendous... in terms of for some people
  what that's like... to be able to do that as
  opposed to... living in a boarding home, and
  everything... you know, your meals are provided,
  and rules that you have to follow it's just a
  tremendous... effect on people to be able to
  achieve having their own apartment...."
  (Source service provider)

27
Sample of Quotes Productivity

• Clinicians can contribute their insight to
  appropriate job matches that will support their
  clients illness management as well as vocational
  needs. Clinicians who know what strategies their
  clients use to manage their illnesses can assist
  in planning effective placements. For example,
  some clients with psychotic illnesses keep
  themselves isolated because they cannot be around
  people without feeling paranoid and persecuted.
  When this is true, clinicians can help vocational
  specialists to find isolated evening work for
  their clients. Other clients feel insecure when
  alone and need to be with others to feel safe
  (Source Torrey et al, 1998).

28
Sample of Quotes Personal and Social Supports

• now my family comes to the self-help program
  space andone of my sisters had a placement
  here, and she has an understanding about it but
  its been . like ten years and nobody
  understoodId given pamphlets, um, tried to
  talk to my family of origin about things I
  collected from the mental health association
  and things that are laying around the hospital
  and its like they dont even want to read
  it.and now, after all this time, my family
  have become members of self-help group and they
  dont see it asthey call me normal nowthey see
  that there are levels of function that people
  have and they see that Im not ill all the
  time (Source consumer)

29
Sample of QuotesService Climate

• Therapeutic goals are highly individualized and
  flexible. Because goals are set and specified
  collaboratively by the patient, family and
  clinical team, treatment objectives can vary
  widely from case to case. Similarly, because
  plans for achieving identified goals are based on
  the strengths and weaknesses of each patient and
  family, such plans vary from case to case.
  Flexibility in goal setting and intervention
  efforts is needed to respond to changing
  patient/family circumstances and outcomes
  (Source Santos et al, 1995,p.1119)

30
Sample of Quotes Services Provided

• When my son was in hospital, um his counselor
  visited him, about two or three times a week, and
  had a pretty good idea of how he was getting on -
  if he was becoming better or not - and at one
  point they were going to release him and um, the
  counsellor let them know that he really wasnt
  well, quite well enough. So I think that they
  helped him a lot (Source family member)

31
Sample of Quotes Organizational Functioning

• And we minute resolutions to problems so that
  if things occur in future we can refer back to
  them and say remember this is what we agreed so
  lets everyone stick to this unless we want to
  reopen the issue, and discuss it again. (Source
  service provider)

32
Sample of QuotesSystems Issues

• In addition to sabotaging treatment
  effectiveness, Drake et al. (1996) state that the
  lack of cooperation between services forces
  clients with co- occurring addictive and mental
  disorders to assume responsibility for
  coordinating care in the parallel model, the
  burden of integrating different philosophies,
  recovery models, and interventions fell on the
  patient in other words, it rarely occurred
  (Drake et al., 1996, p. 45). As a result,
  patients are usually excluded from one or both
  types of programs, and are rarely engaged in
  concurrent mental health and substance abuse
  treatments (Drake et al., 1996 Inderlin
  Belisle, 1991)" p.74. (Cited from El mallakh
  (1998)).

33
Phase II ? Initial Development of Items

• To synthesize information to question format...

34
Brainstorming for Items
35
Sample of Quotes from Interviews Housing

• " They'll branch you out anywhere you want to go,
  anything you need, uh... all you got to do is
  ask, and like youve finally got housing, got a
  place, they could make you call a place. They'll
  get you furniture, they'll get you moved, you
  know they'll get you bedding. Here, they have
  bedding, dishes, things like that..... it's
  absolutely incredible, that resources, are just
  like, for the asking really. You know that old
  saying, ask and you shall receive, it happens....
  -laughter-, it makes a really big difference. "
  (Source Consumer)

36
From data to question
Basic Needs, Housing, Suitability, Living
conditions gt
Item on Consumer Questionnaire My
worker/program helped me make my house feel like
a home (e.g., getting furniture, decorating)
37
Phase II - (continued)

• Development of separate instruments for each
  group
• Review of existing instruments in the field
• Organized around domains and sub-domains

38
Phase III ? Pre-testing

• Tested draft questionnaires with six programs
• Revisions based on feedback

39
Who was Involved in Pre-test?

• Variety of programs
• model type (ACT, Case Management, Housing
  Support, Employment Support, Self-Help,
  Clubhouse)
• urban/rural
• N39 (2 consumers, 2 family members, 2 service
  providers and 1 program administrator from each
  of 6 programs)

40
Feedback from Consumers

• Liked opportunity to reflect on service
  experience and personal needs
• Liked opportunity to speak well of program
• Length not a problem
• Some fatigue factor but overall seen as
  worthwhile

41
Family Members

• Desire to help others through sharing experience
• Didnt enjoy as much as consumers
• More difficulty responding because of less
  familiarity

42
Service Providers/Managers

• Too long for us
• But it is all potentially important
• Gee, we sure do a lot
• Do we do this? Should we do this?
• Some statistical information not available

43
Feedback by Program Type

• Issue of how the questions fit with different
  program contexts
• We always refer for that (e.g .housing)
• Some domains dont work for all
• Volunteering
• We are a non-medical model
• Extent of activity (what is the threshold)

44
Some of the issues which keep arising

• What is the unit to be described i.e. the
  program?
• How much can people remember assimilate in
  responding?
• Should we focus on what the program provides or
  the individual receives?
• How and when to aggregate?
• How much agreement between perspectives?
• Where to go for lunch?

45
Further Issues to be Addressed in Developing
Instruments

• Reduce length
• Establish response distributions, reliability and
  validity for individuals
• Assess degree of agreement on program
  characteristics across sources
• Establish reliability and validity for use at
  program (aggregate) level

46
Objectives of Phase IVPilot Testing Stage One

• Elimination of redundant items or items unlikely
  to be useful and thereby reduce scale length 
• Examination of extent of agreement of information
  provided from different perspectives
• Preliminary investigation of the extent to which
  program characteristics may be linked to outcomes
  for individual consumers.

47
Phase IV - Pilot Testing Stage One

• Testing of package of instruments with 6
  programs
• Cross-sectional design
• Includes preliminary linking of results to
  outcome data
• hospitalization
• satisfaction with services

48
Sample

• 6 programs
• Stratified random selection of 25 consumers per
  program (n150)
• gender
• length of time in program
• diagnosis
• All clinical staff (est. 15/program) program
  manager per program (n96)

49
Will Select Programs which

• Offer long-term individualized support to people
  with a severe mental illness
• Are sufficiently large that gaining a sample of
  25 consumers is realistic
• Offer a range of services (i.e., not restricted
  only to housing or employment support)

50
Selection of Programs (contd)

• Are using the PSR Toolkit or a similar outcome
  tracking system
• Represent different parts of the province
• Are able to provide a lead hand to assist

51
Objectives of Phase IVPilot Testing Stage One

• Elimination of redundant items or items unlikely
  to be useful and thereby reduce scale length 
• Examination of extent of agreement of information
  provided from different perspectives
• Preliminary investigation of the extent to which
  program characteristics may be linked to outcomes
  for individual consumers.

52
Overview of AnalysisItem reduction

• Will examine
• location and percent of missing data
• response distributions
• variation between programs
• inter-item correlations
• feedback regarding clarity, feasibility, etc.

53
Overview of AnalysisAgreement in perspectives

• Will examine
• similarity of metric structure between consumer
  and service provider responses
• agreement in responses appropriate indices

54
Overview of AnalysisTentative Links to Outcomes

• Testing of association between program level and
  aggregated outcome is not feasible
• Correlation and multiple regression to examine
  links to common outcome indices (hospitalization
  and satisfaction with service)
• Exploratory analysis for unique outcome data
  within each program.

55
Summary of Overall Data

• In which domains did consumers want/need
  support?
• How much help do consumers report receiving?
• How much variation is there between programs in
  responses to the instrument?

56
Next Stages of Analysis

• Test-retest
• Sub-study on validity (e.g., use of service log)
• Larger samples of programs and subjects
• aggregation issues
• Sub-study on family perspective?

57
Phase V ? Product Development Dissemination

• Research Products
• Instrument Package
• Manual
• Final Reports Presentations
• Program/System Checklist to
• raise questions
• identify needs for enhancement
• suggest strategies
• Peer evaluation instrument
• Possibly a Provincial Tool for Program Description

58
Template for Program/System Assessment

Examples Housing Physical Needs Employment Educati
on
59
Project Information

• www.ontario.cmha.ca (CMHEI link)
• Ellen Tate, Project Coordinator
• Centre for Addiction and Mental Health
• 33 Russell St.
• Toronto, ON M5S 2S1
• Tel [phone] ext. 6575
• email ellen_tate_at_camh.net