BIGSPD talk: CCAT

1

2
Therapist competence and therapy outcome in the
cognitive-analytic therapy of severe complex
mental health problems

• Glenys Parry Dawn Bennett
• University of Sheffield
• School of Health Related Research
• and Sheffield Care Trust
• Sheffield Health Social Research Consortium
  Conference
• 9th November 2006

3
Acknowledgments

• Mental Health Foundation for funding the study
• Mark Evans and Sarah Littlejohn for help in
  developing the CCAT.
• Therapists and members of consensus panels Gill
  Aitken, Brigitta Bende, Gill Bloxham, Kate
  Cahill, Phil Clayton, Sarah Cluley, Louise
  Cooke, David Crossley, Mark Evans, Kate
  Freshwater, Deborah Gamsu, Yvonne Harris, Anna
  Jellema, Sarah Littlejohn, Carol Lomax, Judy
  Merchant, Sue Pethen, Debby Pickvance, Rachel
  Pollard, and Lawrence Welch.

4
Why study competence?

• A significant source of variance in outcomes.
• Service users think therapist errors are
  important (Castillo, 2003)
• Major challenge for training courses and brand
  name therapies if no relationship between
  specific (technical) competence outcome

5
Why study competence?

• We lack an understanding of how to foster
  competence and reduce therapist errors.
• Re. implementing NICE guidelines, vital part of
  transferability of randomised trials results.
• Studies of competence will have direct
  implications for design delivery of both
  therapist training and psychological therapy
  services

6
Therapist competence

• Different from adherence (i.e. sticking to the
  manual) this has been linked to poorer outcomes
• Not only what was done but how well
• Differing concepts of competence
• in a therapist (i.e. a competent therapist)
• performance (process variable)
• quality of an interaction (context-dependent)

7
Challenges for research

• Different patterns of outcomes for different
  therapists (Luborsky et al 1986)
• Most therapists have mixed outcomes only
  minority generally better or poorer results
  (Huppert et al 2001)
• Relationship between training (or experience) and
  client outcomes is weak (Stein Lambert 1995)
• This may reflect methodological difficulties,
  e.g. differential attrition case mix

8
Challenges for research

• Earlier, we found evidence that therapist skill
  in resolving alliance threats may have an impact
  on outcome in people with more severe complex
  problems (personality disorder) (Bennett, Parry
  Ryle, 2006)
• Repeated or unredeemed errors have been linked to
  drop-out or deterioration

9
Severe complex problems

• We predict relationship between competence and
  outcome stronger for people with severe complex
  mental health problems including personality
  disorder
• Tend not to improve spontaneously in the short
  term (Stone, 1993)
• Sensitive to therapist errors iatrogenic
  deterioration (Shearin Linehan, 1993)
• More difficulty in therapeutic alliance
  (Waldinger Gunderson, 1984 )
• More likely that maintenance of good working
  relationship (hence good outcome) requires
  therapist skill.

10
Aims of this study

• For a sample of patients with severe complex
  mental health problems who received cognitive
  analytic therapy
• To investigate the relationship between levels of
  experience and therapist competence (null
  hypothesis there will be no relationship)
• To investigate the relationship between therapist
  competence and therapy outcome (null hypothesis
  there will be no relationship)

11
Sample

• Patients with severe complex problems seen for
  CAT in routine services using naturalistic data
• N112 patients in total (not all available for
  all analyses)
• All patients had minimum of two year history of
  symptoms and service use
• Methodological limitation descriptive not
  purposive sampling retrospective assignment of
  Axis II borderline personality disorder Axis I
  depression
• n25 therapists
• Full range of levels of training and experience
  from novice to expert

12
Measures CCAT

• Measure of competence in CAT unit of assessment
  is whole session (Bennett Parry, 2004)
• Empirically developed from NCVQ/UKCP competence
  framework for cognitive and humanistic therapies
• 64 items of observable competencies grouped into
  10 domains each with overall score (0-4 Likert
  scale) total score 0-40.
• Distinction between
• early late sessions
• technique specific non-specific competence

13
CCAT continued

• Interactional view of competence takes account of
  task difficulty
• Three reasons for an item of competence not being
  observed
• Inappropriate in that session
• Difficult to perform
• Missing could should have happened but didnt
• Acceptable reliability
• Inter-rater agreement fair (Cohens Kappa
  0.63-0.67)
• Item consistency (Cronbachs alpha 0.96, 0.98)

14
CCAT measure

• 1. Phase-specific tasks
• a) early engagement, induction remoralisation
• b) review and evaluation of process and outcome
• 2. Using theory to inform practice
• 3. CAT-specific tools and techniques
• 4. Establishing and maintaining the external
  framework
• 5. Common factors basic supportive good practice
• 6. Respect, collaboration mutuality
• 7. Assimilation of warded-off, problematic states
  and emotions
• 8. Making links and hypotheses
• 9. Identifying and managing threats to the
  therapeutic alliance
• 10.Being aware of and managing ones own
  reactions and emotions

15
Level of experience

• Four levels of therapist experience
• Novice (n5) new to CAT and to psychological
  therapy (e.g. DClinPsych or MRCPsych trainee)
• Intermediate (n8) some experience of CAT and of
  psychological therapies (e.g. counsellor after
  one year of supervised experience)
• Experienced (n7) gt2 years psychological
  therapist at CAT practitioner level
• Trainer (n5) gt10 years psychological therapist
  ACAT accredited supervisor trainer in CAT

16
Assessment of outcome

• Subgroup of patients for whom both pre- and post-
  therapy measures were available
• CORE
• BDI
• SCL-90R
• Therapy outcome assigned to one of three levels
  (n77 sessions)
• Good (n37) statistically reliable improvement
  to a clinically significant criterion (Jacobson,
  1988)
• Moderate (n27) statistically reliable
  improvement
• Poor (n13) either no change or deterioration

17
Method

• Three rater pairs established reliability on 27
  sessions
• Further sessions rated by single judge
• Each therapy session rated on CCAT
• CCAT ratings blind to outcome of therapy
• Two separate issues
• Do therapy sessions of experienced therapists
  show higher CCAT scores?
• Do therapy sessions from good outcome cases have
  higher CCAT scores?

18
Results competence experience

• 3-way ANOVA therapist x case x experience showed
  no main effect of therapist
• But significant main effect of experience
  (F7.66, plt.001)
• Significance of linear trend tested by 1-way
  ANOVA with between-groups sum of squares
  partitioned into linear and non-linear variance
• Significant relationship of experience with CAT
  competence (plt.0001)

19
CCAT scores across four levels of therapist
experience
20
What does it mean?

• Therapy competence is related to experience in
  this sample
• However
• Not a longitudinal study least talented novices
  may later stop practising CAT
• Variance reduces with experience levels
• Some novices are demonstrating competence with
  little training and experience

21
CCAT scores across four levels of therapist
experience
22
What does it mean?

• Therapy competence is related to experience in
  this sample
• However
• Not a longitudinal study least talented novices
  may later stop practising CAT
• Variance reduces with experience levels
• Some novices are demonstrating competence with
  little training and experience
• Evident ceiling effect between intermediate and
  trainer level therapists

23
CCAT scores across four levels of therapist
experience
24
Results competence outcome

• Significantly higher level of CAT competence
  demonstrated by therapists in sessions from CAT
  therapies with good outcomes compared with
  sessions from therapies with poor outcomes
• Bonferroni post-hoc test showed that all the
  means of all three groups are significantly
  different from each other.

25
CCAT score for three outcome groups
26
What does this mean?

• For this sample, very low levels of CAT
  competence were associated with poor outcomes

27
CCAT score for three outcome groups
28
What does this mean?

• For this sample, very low levels of CAT
  competence were associated with poor outcomes
• None of the good outcome therapies had poor
  levels of competence

29
CCAT score for three outcome groups
30
What does this mean?

• For this sample, very low levels of CAT
  competence were associated with poor outcomes
• None of the good outcome therapies had poor
  levels of competence
• BUT it is possible to show moderate levels of
  competence and still have a poor outcome

31
CCAT score for three outcome groups
32
Conclusions

• Reject the null hypotheses Positive findings of
  orderly relationships
• Experienced therapists are more competent
• More competent therapists get better outcomes
• BUT methodological limitations of practice-based
  evidence
• heterogeneous sample retrospective diagnosis
• confound between experience and training
• different measures of outcome
• Replication part of current randomised
  controlled trial of CAT, CBT CMHT in helping
  people with borderline personality disorder

33