From Clinical Observation to Outcome Measurement

1
From Clinical Observation to Outcome
Measurement

• Presented by Virginia Wright (Bloorview Research
  Institute, Toronto) at the OACRS Conference,
  Waterloo, ON, October 23, 2007
• Virginia Wright is supported by a Career
  Development Award (2007 to 2011) through the
  Canadian Child Health Clinician Scientist
  Program, a Canadian Institutes of Health Research
  Strategic Training Initiative.
•  

2
Illustration through results from the OACRS
Measuring Change Study

• Co-investigators The OACRS Steering Committee
  (Vicky Earle, Gill Davis, Jenny Greensmith,
  Dorothy Harvey, Karen Horgan, Laurie Lessard,
  Donna Litwin-Mackey, Helene Mercier) and Virginia
  Wright.
• Funded by Ontario Ministry of Children and Youth
  Services October 2006 to March 2007. MCYS
• The opinions expressed are the consultants' and
  should not be considered reflective of those of
  the Ministry of Children and Youth Services.

3
The broad question for our OACRS project

• What are the outcome indicators that should form
  the main components of a core outcomes set for
  use across OACRS childrens treatment centres
  (CTCs)?

4
Goals of our Project(1st of 3 main goals)

• 1. Produce an inventory of current outcome
  measurement practice across CTCs based on
  clinical practice, based on what is being done.
• Summarize key goal domains that clinicians
  identified as the focus of their intervention
  work
• Identify clinicians methods of measuring change
• Summarize the value of the identified measures in
  the evaluation of change

5
Methods

• Our study was ethnographic in approach.
• It used reflective interviews with clinicians as
  a means of learning about measurement practice
  with out-patient clients.
• Health records were selected at random within
  designated program areas so that the study
  results were representative of overall outcome
  practice.
• Each clinician reflected on a selected current
  out-patient clients health record as the basis
  for their responses to the interview questions
  about measurement.

6
Results
7
We achieved representation of programs/services

• We had representation from all areas in Ontario,
  from Windsor to Sudbury to Ottawa.
• The work represented various clinician groups,
  diagnostic groups, and program/service areas
• 60 of clients seen in treatment
  facility/satellite centre, 30 in community, and
  10 in both locations
• Balanced sample of active intervention and
  consultation clients

8
Clinicians Interviewed (n 408)
Other music therapists, behaviour therapists,
infant development specialists, audiologists,
communication disorder assistants, resource
teachers, family counsellors, service
co-ordinators, kinesiologists, psychologists,
audiologists, and recreation therapist all
self-identified according to role with client
9
Primary diagnosis of client (n 408 interviews)
10
Question 2

• What were the main goals for your work with this
  client based on the assessment findings?

11
ICF Goal Areas (n 408 interviews)
Note no specific goals relating to quality of
life
12
Goal category by clinical profession
13
Goal setting is a key part of practice within
OACRS CTCs

• Goals and associated measures focused on change
  in relation to impairment and activity goals
  rather than on participation
• This focus on impairment and activity was to
  be expected since these tend to be issues that
  clinicians and parents can easily identify
• Both of these areas lend themselves well to
  evaluation by observation
• Changes in participation may have been seen as
  consequences of changes in activity and
  impairment rather than changes that are under the
  direct control of the treating clinician.

14
ICF Focus for Measurement

• Increasing recognition that family-centred
  pediatric rehabilitation practice should be
  grounded in the framework of the International
  Classification of Function and Disability (ICF)
• Both interventions and outcome indicators need to
  take this framework into account body structure
  and functions (impairment), activity (carrying
  out tasks) and participation (involvement in life
  situations).

15
Examples of goal areas

• Body structure/functions
• Jaw control for developing vowels and bilabials
• Develop lateral tongue movement
• Increase hand strength (to do buttons and snaps
  independently)
• Improve lower extremity alignment in standing
• Decrease muscle pain in legs
• Activity
• Ability to use a voice output device in the
  community
• Increase left hand use through instrument playing
• Improve ability to ascend stairs Increased
  ability to ride bike with training wheels
• Dribble tennis ball with right hand
• Play functionally and appropriately with toys for
  a specified period of time

16
Examples of goal areas

• Participation
• Participation in girls group -e.g. increased
  self-esteem, leadership skills, increased social
  skills and network, increased life skills
• Money management (process of using money i.e.,
  choose an item, purchase it )
• Play dodge ball with friends on street after
  school
• Will participate in band class at school (on
  clarinet)
• Education
• Increase familys understanding of a sensory diet
• Teach parents how to work on above speech sounds
  at home
• Inform mother of community resources and supports

17
Examples of goal areas

• Equipment/environment
• Provide a seating and mobility system that is
  safe
• Adaptations in classroom to make it accessible
• Increase use of right ankle foot orthosis
• Development of a low tech communication system -
  picture based
• Trials of voice output communication aid
• Process
• Compile a financial request form
• Develop a behaviour management strategy
• Obtain funding for participation in sledge
  hockey
• Academic program (make recommendations)
• Refer to psychology for full assessment

18
Question 3

• How did you go about assessing child or family
  abilities or status in each of these goal areas
  when you started working on them?
  List informal and formal
  approaches used

19
Examples of approaches used

• Observe behavior/skills related to
• Formal testing on SPAT-D
• Test with oral motor hierarchy checklist
• Review current equipment and writing needs with
  family
• Goal Attainment Scaling (GAS) with child and
  family and school team
• Timed trials for 2 device access methods
• Informal, dialogue with parent
• GMFM, GAS, timed walk tests on treadmill

20

• Measurement of change usually involved some
  combination of use of clinical observation of
  targeted behaviours/skills, parent/client report
  and use of a standardized measure(s)

21
Q 5 Ratings of useful of change measurement
approaches

• Score each with a number of 1 through 5 on the
  five point response scale from not at all
  useful to very useful.
• Identification of issues _____
• Setting goals _____
• Evaluation of change _____

22
Examples of response set for Q 5

• Talking with family
• Identification of issues very useful (5)
• Setting goals very useful (5)
• Evaluation of change very useful (5)
• Goal Attainment Scaling
• Identification of issues somewhat useful (3)
• Setting goals very useful (5)
• Evaluation of change very useful (5)
• Observation with check box scoring
• Identification of issues very useful (5)
• Setting goals very useful (5)
• Evaluation of change somewhat useful (5)

23
Usefulness of measurement approaches overall
There were differences between approaches for
their usefulness in evaluation change when
compared for the 15 diagnostic groupings (lower
for ABI, developmental delay, learning, and
seizure disorder).
24
Standardized measures were used as part of the
assessment approach

• A wide variety of measures were seen as very
  useful to establish a baseline and set goals.
• BUT
• Clinicians frequently highlighted the lack of fit
  of fixed item tools with a childs or familys
  characteristics, issues and goals.
• Thus, they had to find other ways to measure
  change

25
Clinical observation frequency and usefulness
for evaluating change
26
What did clinical observation consist of?

• Clients ability to get down to floor and stand
  up
• Client sitting in desk in new seating set-up
  -observe arm and leg position
• Looked at ability to negotiate various terrain
  within daycare
• Number of stairs he could climb and how much
  assistance he required to accomplish
• During structured play activities-watch how
  marker was held during multiple opportunities
  graded assistance in order to achieve success)
• Observe client's printing sample and observe
  classroom notebooks

27
Approaches to scoring clinical observation

• Use an extent of accomplishment rating
  yes, partial, no
• Take a skill description and associated scoring
  options from a standardized measures such as the
  GMFM and test that skill in isolation of the full
  test, e.g., test the childs ability to climb 4
  steps without use of railing and use the GMFMs
  4-point scale to describe new abilities.

28
Advantages of clinical observation (when not in
context of a larger measure)

• Client-centred allows clinician to zero in on
  the individual issues and goals and focus on
  outcomes that were the target of service
  provision
• More focused than fixed item measures
• Pragmatic approach - time efficient
• Adaptable to any type of observable behaviour

29
Limitations of clinical observation

• Does not use any standard means of documentation
• Clinicians acknowledged that results cannot be
  rolled up to look at changes across children for
  a particular intervention or program not at all
  useful for a systems-level evaluation!
• Unknown reliability as far as scoring of outcomes
  a big issue!

30
Use of client/parent report was the next most
popular method of evaluation of change
31
Use of parent/client report

• In the case of parent, client, teacher or other
  clinician report/interview, accomplishment scores
  were linked directly to the rating of the stated
  goals, and were typically done by a verbal report
  (e.g., yes, no accomplished, partially
  accomplished) in response to targeted questions
  from the clinician about the childs abilities.
• Within the report/interview category, the parents
  were the respondent group that was most
  frequently cited as sources for information. It
  is likely that the report occurred within the
  context of an intervention session.

32
Limitations of parent/client report
(when not in context of a larger measure)

• No standard mean of documentation or reporting
• Means that results cannot be rolled up to look at
  changes across children for a particular
  intervention or program
• Unknown reliability as far as scoring of outcomes

33
Discussion

• Making measurement work in real-life practice
• Use of individualized outcome measures as a
  structural framework for clinical observations
  and parent/client report.

34
Recommendation to MCYS (one of 6 recommendations)

• Use of a validated individualized goal-based
  approach such as Goal Attainment Scaling (GAS) or
  the Canadian Occupational Performance (COPM) for
  each clinician/client dyad.
• Use of GAS or the COPM would allow integration of
  impairment, activity, participation, education or
  environment/equipment goals within the same goal
  evaluation template.

35
Primary Limitation of Fixed-item Outcome
Measures such as PEDI, GMFM, WeeFIM,

• Work well when evaluating group effects (clinical
  trials, program evaluation)
• May work well when evaluating a single clients
  progress over long periods of time
• May not contain enough emphasis on activities of
  priority to client/family or those that are
  targets of an intervention block

36
Comments from OACRS administrators about
individualized goal evaluation

• Not all of the children qualify for standardized
  tests and the children may not make sufficient
  changes within expected time frames as to render
  the use of the assessment tools effective. We do
  believe that dynamic goal setting with the
  individual child and family has been generally
  effective.

37
Comments / Considerations

• It is very interesting to note that clinicians
  really value parent/caregiver report and clinical
  observations as sensitive measures of change.
  Standardized measures need to be interpreted
  cautiously when the family has noted changes but
  in fact the results (standard scores and
  percentiles) may actually have worsened. The
  child may have made many gains but may not have
  closed the gap (a child who makes 6 months change
  in 12 months time according to standardized
  measures).

38
Comments / Considerations

• In terms of the ICF classification definitions,
  we certainly see that goal areas under Impairment
  are easier to isolate and measure however beyond
  pre-school years our goal areas are more linked
  to Participation and Education which are more
  challenging to measure. How do we isolate the
  specific variables which have contributed to the
  client/family changes? 

39
Individualized-item Measures...

• The problem areas being assessed are specific
  for each individual and may be set by either the
  client or the health professionals (Donnelly and
  Carswell 2002, p.85).
• Goals are the foundation of all individualized
  measures

40
Individualized Outcome Measures (in order of
release)

• Goal Attainment Scaling (GAS) (1968)
• McMaster Toronto Arthritis Questionnaire (MACTAR)
  (1987)
• Canadian Occupational Performance Measure (COPM)
  (1988)
• Patient Specific Functional Scale (1995)
• Assessment of Motor and Process Skills (1997)
• Perceived Efficacy and Goal Scale (PEGS) (2000)
• Self-identified Goals Assessment (2002)
• Individually Prioritised Problem Assessment
  (2002)
• Described by Donnelly and Carswell, 2002 or
  Jolles et al, 2005 Missiuna and Pollock,
  2001,

41
What Can Individualized Measures Tell Us?

• 1. The extent of accomplishment of client-based
  goals
• 2. The types of goals typically set within a
  program/team/type of intervention/stage of care

42
Benefits of Individualized-item Measures...

• Evaluate the clients abilities on a set of
  goals/tasks defined just for them
• Are useful across all disabilities, developmental
  levels, settings and across a variety of goal
  domains work well within a team approach!
• Reduce the ceiling effect commonly seen in
  outcome measurement
• Can identify priority problems not covered by
  standardized measures (used as part of the
  problem-defining process prior to intervention)

43
Benefits ...

• Can be used as the structural framework of
  discussions at family-team conferences
• Assist decision making re short and longer term
  planning
• Give message to clients that their goals matter
• Enhance client participation in the rehab process
  and may produce better rehab outcomes? (Gange
  Hoppes, 2003)

44
Would the COPM and GAS be a systematic way to go
about measuring outcomes related to goals?

• Both fit well with observational and
  parent/client-report measurement approaches, and
  also with family-centred care and individualized
  goal setting.
• From a practice standpoint, why were these
  validated, individualized goal approaches rarely
  mentioned within the context of clinical
  observation and family report?
• Their sporadic use was unexpected given the
  indication from several centres that they had
  received training in these approaches, and that
  their use was encouraged within certain program
  areas.

45
Reasons for minimal use?

• COPM and GAS approaches are labour-intensive
  (especially in the early stages of their use) and
  require training to ensure that they are used
  properly and without modification.
• Could it be that not all clinicians were
  sufficiently familiar with these goal-setting
  approaches and their potential for quantitative
  measurement of change?
• Perhaps clinicians do not perceive that there is
  sufficient value over and above informal clinical
  observation or report on goal accomplishment.
• We may have under-sampled children from life
  skills classes in which GAS/COPM may be used more
  often

46
GAS is Not New!

• Developed by Kiresuk and Sherman, 1968
• Published reports in OT/PT journals started
  appearing in the 1980s
• Kiresuk et al, 1994 wrote entire text on GAS
• Notable pediatric work for school-based therapy
  services (King et al, 1999)
• Innovative work for adults for brain injury in
  combination with COPM (Trombly et al, 19982002)

47

• Goals pertain to clients stated needs/hopes and
  the outcome possibilities
• Can be used by any profession for any
  intervention goal
• Permits direct measurement of the clients
  achievement (change)
• In the short term, more responsive to meaningful
  change than standardized/fixed item measures
• Not intended to replace standardized measures
  A complimentary relationship instead!

48
GAS Psychometric Properties

• Reliability (Stolee et al, 1992 Rockwood et al,
  1993 Joyce et al, 1994 Rushton et al, 2002)
• Concurrent validity (Stephens and Haley, 1991
  Malec et al, 1991 Palisano et al, 1992 Joyce
  et al, 1994 Yip et al, 1998 Rushton et al,
  2002Fisher and Hardie, 2002)
• Responsiveness (Palisano et al, 1992 Gordon et
  al, 1999 Trombly et al, 2002 Rockwood et al,
  1993 and 1997 Becker et al, 2000 Fisher and
  Hardie, 2002 Rushton et al, 2002)

49
The 5 GAS levels when positive change is expected

• -2 Ability at time goal is set (baseline
  level)
• -1 Less than expected outcome
• 0 Expected outcome after intervention
• 1 Greater than expected outcome
• 2 Much greater than expected outcome
• 0 level is the short-term goal and 2 level
  may be the longer-term goal

50
(No Transcript)
51
What did clinical observation consist of?

• Volume of feed, time it took to feed and weight
  gain
• Ability to demonstrate tripod grasp 20 of the
  time
• Spinal alignment in sitting and independent
  sitting balance
• Ability to walk on challenging surfaces i.e., on
  gravel
• Observation (informal) of drawing, beading, doing
  up zipper, cutting food, knife and fork, ability
  to steer bike in therapy room and how well she
  used hand brakes
• Observation of letter formation techniques, hand
  positioning on pencil with and without grip,
  ability to cut basic shapes, simple designs,
  seating

52
Parent Well-being Goal

53
Outcome question Was the goal accomplished?

• Aiming for an accomplishment level of 0
• The 5-level GAS scheme allows acknowledgement of
  changes that are less or more than expected
• Gives a chance to describe acquired skills in
  behavioural, measurable terms
• Gives a chance to think about what higher levels
  of accomplishment might be possible
• Extent of accomplishment of all goals set for
  each client can be combined into a single summary
  standardized T-score for use in program
  evaluation

54
Would GAS also help us to evaluate at the group
(systems) level?

• T-scores for all clients in a program can be
  evaluated together
• Aim for a group T-score around 50
• The 50 is the BENCHMARK built into GAS
  indicating that accomplishment of goals occurred
  for the clients overall at targeted 0 level
• Look at distribution of the groups T-scores
  Is there a tendency to under- or over-
  achievement of goals?

55
Example of Program T-scores for first and
second set of goals
56
Functional and Psychosocial Goals in a
School-based Rehab Setting (n 45 children)
57
The COPM

• Based on the model of occupational performance
• Clients own perception of their functioning
  related to self-care, productivity and leisure
• Client takes part in semi-structured interview
  with clinician (usually, but not necessarily, an
  OT)
• Client generates a list of goals and ranks
  importance
• Client rates top 5 goals in terms of current
  level of performance and satisfaction with
  ability (1 to 10 scale)

58
COPMs Psychometric Properties

• Clinical utility (Law et al, 1994 Toomey et al,
  1995 Wressle et al, 2002 2003 Swedish
  version Chen et al, 2002)
• Reliability for item generation or for ratings
  (Sevell and Singh, 2001 Cup et al, 2003)
• Content validity (Chen et al, 2002)
• Concurrent validity (Ripat et al, 2001 Chen et
  al, 2002 McColl et al, 2000 Cup et al, 2003)
• Responsiveness (Carpenter et al, 2001 Wressle et
  al, 1999 Trombley et al, 2002 Chen et al, 2002)

59
Examples of COPM Goals

• Putting winter coat on and do up fasteners
  (Self-care)
• Using a computer to do home work (Productivity)
• Play basketball with friends on the street
  (Leisure)
• ? Each chosen goal is rated by client in terms of
  ability to perform (1 to 10), and satisfaction
  with performance (1 to 0)

60
What can we learn abut outcome by using the COPM?

• Average changes in perception of performance of
  skill
• Average changes in perception of satisfaction
  with performance
• COPM developers say that change of 2 or more
  points ( or -) is considered clinically
  important
• Earn about types of goals set (like GAS)
• Learn about frequency of use of various goals and
  goal domains overall (like GAS)

61
COPM Example

• LEISURE
• Play basketball with friends while wearing new
  sports prosthesis
• Before new sports prosthesis
• Performance 2 Satisfaction 1
• After 3 months of use
• Performance 5 Satisfaction 9
• Change in abilities and satisfaction
• Performance 3 points Satisfaction 8
  points
• What skills exactly do these changes in
  performance and satisfaction represent?
• How well does the child now do the activity?

62
Would the COPM also help us to evaluate at the
group level?

• Scores for all clients in a program can be
  evaluated together
• Aim for a change score of at least 2 points for
  satisfaction and performance
• Look at distribution of the groups scores

63
Publications in pediatrics profiling the COPM or
GAS (Tam et al., submitted)
64
What do OTs think about the COPM for clinical
practice? (Tam et al., under review)

• Three major themes evolved from the interview of
  13 pediatric OTs
• the COPM facilitates client/family-centered care
  
• the benefits of COPM outweigh its limitations,
  and
• the COPM facilitates reflective practice.

65
The Bridge Between the COPM and GAS

• Combined use provides two perspectives
  i) Client viewpoint of
  what happened and its value (COPM), and
• ii) specification of actual accomplishment
  (GAS)

66
The Bridge Between the COPM and GAS

• Trombley et al. started to explore this link in
  their brain injury work
• Established problems (goals) using COPM interview
  process
• Scaled the accomplishment levels using GAS
• Client rated satisfaction and performance at
  baseline and follow-up using COPM scoring system
• Clinician/client rated goal accomplishment at
  follow-up using GAS

67
COPM Example

• BICYCLING GOAL
• Ride up Market St hill with only a little help at
  the bottom
• At baseline
• Performance 1 Satisfaction 1
• After 2 months of practice in the summer
• Performance 8 Satisfaction 8
• Change in abilities and satisfaction
• Performance 7 points Satisfaction 7
  points
• What skills exactly do these changes in
  performance and satisfaction represent? How well
  does the child now do the activity?
• Could integrate with GAS for same goal

68
Community Bicycling Goal
69
Recommendations from OACRS study

• Resources and a clearly identified implementation
  process must be added to the system to permit the
  within-centre and network capture of outcome
  results for the selected measures. Pertains to
  use of GAS and COPM for sure!
• At present, much of the outcomes information
  still resides in the client chart and is
  inaccessible even at the program evaluation level
  within a CTC.
• Not always reportable at a higher level even with
  a system in place! Particularly the case with
  clinical observation and parent report outcomes
  if not done using a standard measurement system.

70
Take home message

• Bottom line given its importance, the
  individualized outcomes information should also
  be captured in a way that can be rolled up.
• GAS and COPM could help us with this! Without a
  systematic approach to goal-based outcome
  measurement, we may be losing more than 60 of
  our outcomes information

71
Acknowledgements

• We thank
• The Ontario Ministry of Children and Youth
  Services (MCYS) for their funding for the project
• The Bloorview Research Institute for supporting
  Virginia Wrights time on the project
• The huge group of clinicians across the 16 CTCs
  who participated in the interviews
• Susan Cohen for her work as research co-ordinator
• Christina Hay and Angela McDonald for their
  assistance with data entry and analysis
• Carolyn Hicks and Wendy Moyle for their strong
  support in the OACRS office..

72
We thank our Research Assistants (centres listed
in alphabetical order)

• Susan Cohen and Joan Walker (Bloorview Kids
  Rehab)
• Janet Coppold (Child Development Centre, Hotel
  Dieu Hospital)
• Joanne Leclair Peterson (Childrens Treatment
  Centre)
• Theresa Hudson (Erinoak Kids)
• Angela Harrison (Five Counties Childrens Centre)
  
• Helene Mercier (George Jeffrey Childrens Centre)
• Karen Koseck (Grandview Childrens Treatment
  Centre)
• Julie Bially (John McGivney Childrens Centre)
• Bridget OBrien (KidsAbility Centre for Child
  Development)

73
We thank our research assistants

• Heidi Nelson (Lansdowne Childrens Centre)
• Linda Wallman (Niagara Peninsula Childrens
  Centre)
• Andree Cornish (One Kids Place/La place des
  enfants)
• Bonnie Lowry Bagshaw (Ottawa Childrens Treatment
  Centre)
• Cathy Dufort Gibbs (Pathways Health Centre for
  Children)
• Janice Dekker (Prism Centre for Audiology and
  Childrens Rehabilitation), and
• Lisa Peacock (Childrens Treatment Network of
  Simcoe York).

74
Thoughts/Questions?
Virginia Wright vwright_at_bloorview.ca