The Competency Assessment Toolkit for Professional Psychology

1
The Competency Assessment Toolkit for
Professional Psychology

• Catherine L. Grus, Ph.D.,
• Associate Executive Director for
• Professional Education and Training
• Education Directorate
• American Psychological Association

2
THANK YOU

• Many thanks to
• Nadya Fouad, University of Wisconsin
• Nadine Kaslow, Emory University
• Benchmarks Workgroups
• Assessment Toolkit Workgroup

3
Overview

• Considerations in the assessment of competence
• Development of the Toolkit
• Putting the toolkit into practice

4
Competence

• the habitual and judicious use of
  communication, knowledge, technical skills,
  clinical reasoning, emotions, values, and
  reflection in daily practice for the benefit of
  the individual and community being served.
• From Epstein and Hundert (2002)

5
Competence

• is developmental depending on stage of
  professional functioning
• context dependent
• executed according to ethical principles,
  guidelines, standards, and values of the
  profession and
• requires public verification

6
Competencies

• Observable
• Measurable
• Containable
• Practical
• Derived by experts
• And flexible

• are elements of competence that are .

7
Assessment of Competence - Considerations

• Use of both formative and summative
  evaluation methods
• Authentic evaluations
• Real world tasks, meaningful activities
• High Fidelity Degree to which
• the assessment represents
• actual performance

8

• Employ a range of perspectives (multi-informant)
  and methods
• Challenge of evaluating attitudes and values
• Psychometric issues
• Feasibility/costs
• Lifelong self-assessment

9
What methods are commonly used to assess
students/ trainees?

• In class examinations and course grades
• Faculty/supervisor completes rating scale
• Letters of recommendation
• EPPP, state jurisprudence exams

10
Developing Faculty/Supervisor Competencies

• Training in difficult conversations
• How to critique constructively
• Develop awareness of potential dual roles in
  evaluation and how that could influence results
• Benefits of having other faculty evaluate
  students performance
• Explicit obligations to make evaluations accurate
• Grade inflation
• Letters of recommendation to market vs.
  evaluate
• Knowledge of a range of assessment tools, access
  to resources toolkits

11
Competency Assessment for Toolkit for
Professional Psychology

• Charge from APA Board of Educational Affairs
  Develop a Toolkit for professional psychology
• Purpose Promote broader implementation of
  competence assessment and provide information
  about application of assessment methods to the
  assessment of competence
• Coordinated with Benchmarks Work Group

12
Benchmarks Core Competencies

• Foundational Competencies
• Professionalism
• Reflective practice
• Scientific knowledge and methods
• Relationships
• Individual and cultural diversity
• Ethical and legal standards and policy
• Interdisciplinary systems

• Functional Competencies
• Assessment
• Intervention
• Consultation
• Research and evaluation
• Supervision
• Teaching
• Administration
• Advocacy

13
How are the Benchmarks Organized?

• Core Foundational and Functional Competencies
• Essential Component what are the critical
  elements of/what knowledge/skills/attitudes that
  make up this competency?
• Behavioral Anchor what would it look like if you
  saw it (essential component)?

14
Components of Toolkit

• Background and Introduction
• Assessment Method Fact Sheets
• Grid of Assessment Methods and Competencies/Essent
  ial Components Best used for

15
Fact Sheet Portfolio Review Description A
portfolio is a collection of products, gathered
by the person being assessed Application
Competency Applies to1 Predominant Use of2 Developmental Level3
Professionalism E F,S 2,3,4
Reflective Practice E F,S 1,2,3,4

• Implementation
• Deciding on form (e.g., web-based or hard copy)
• Psychometric Properties
• Reliability has not been well established due to
  the variable content included
• in a portfolio
• Strengths
• Provides a low cost assessment strategy
• Challenges
• Requires intense commitment of time and labor
  intensive for the assessors
• and the person being assessed
• Future directions
• Create and implement training modules for
  assessors and people being
• assessed in the portfolio assessment strategy

16
Grid of Possible Assessment Methods (Toolkit)
Annual review Case reviews Process/outcome data Rating Forms Consumer surveys OSCE Portfolio reviews Ratings Performance Record reviews Self-assessment Simulations/role plays Stand. patient interview Stand. oral exams Written exams 360 Evaluations
Intervention Knowledge interventions 2 3 3 2     1     2     1 1  
  Intervention planning 2 1 2 1 2 2   3 1   2 2 1    
  Skills 2 2 1 3 2 1 2 1 2   1 1 2    
  Implementation 1 1 1 1 1 1 2 1 2 1 2 2 2 3 1
  Progress evaluation 3 1 2 2 2   3 2 2 1     2   2
17
Assessment Methods in Toolkit

• 360 evaluation
• Portfolio
• OSCE
• Structured Written Oral Exams
• Case Presentation
• Simulation/Role Play
• Competence Evaluation Rating Form
• Self-Assessment

• Ratings of live or recorded performance
• Standardized Client Interview
• Client/Patient Process/Outcome Measure
• Consumer Satisfaction Survey
• End of Rotation Performance Review

18
Putting it all Together
Benchmarks
Toolkit
Using the Toolkit and Benchmarks to Guide the
Assessment of Student Learning Outcomes
19
Identify expected student learning outcomes
using competency model (e.g., Benchmarks)
Expected level of competence attainment and
training plan developed with trainee
Select assessment method(s)
Student demonstrates problems with competencies
acquisition
Remediation plan developed
Formative assessment
Student demonstrates emerging competencies
Training plan continues
Summative evaluation to assess readiness for next
level of training
20
Identify expected student learning outcomes
using competency model (e.g., Benchmarks)
21

• Program Outcome Students will demonstrate the
  following intervention (functional) competencies
  prior to going on internship.
• Knowledge of scientific, theoretical, empirical
  and contextual bases of intervention, including
  theory, research, and practice
• Formulates and conceptualizes cases and plan
  interventions utilizing at least one consistent
  theoretical orientation
• Clinical skills
• Implements evidence-based interventions that take
  into account empirical support, clinical
  judgment, and client diversity (e.g., client
  characteristics, values, and context)
• Evaluate treatment progress and modify treatment
  planning as indicated, utilizing established
  outcome measures

22
Identify expected student learning outcomes
using competency model (e.g., Benchmarks)
Select assessment method(s)
23
Grid of Possible Assessment Methods (Toolkit)
Annual review Case reviews Process/outcome data Rating Forms Consumer surveys OSCE Portfolio reviews Ratings Performance Record reviews Self-assessment Simulations/role plays Stand. patient interview Stand. oral exams Written exams 360 Evaluations
Intervention Knowledge interventions 2 3 3 2     1     2     1 1  
  Intervention planning 2 1 2 1 2 2   3 1   2 2 1    
  Skills 2 2 1 3 2 1 2 1 2   1 1 2    
  Implementation 1 1 1 1 1 1 2 1 2 1 2 2 2 3 1
  Progress evaluation 3 1 2 2 2   3 2 2 1     2   2
24
Identify expected student learning outcomes
using competency model (e.g., Benchmarks)
Expected level of competence attainment and
training plan developed with trainee
Select assessment method(s)
25
Developmental Level Developmental Level Developmental Level
A. Knowledge of Interventions A. Knowledge of Interventions A. Knowledge of Interventions
Readiness for Practicum Readiness for Internship  
Essential Component Basic knowledge of scientific, theoretical, and contextual bases of intervention and basic knowledge of the value of evidence-based practice and its role in scientific psychology Behavioral anchor Articulates the relationship of EBP to the science of psychology Identifies basic strengths and weaknesses of intervention approaches for different problems and populations Essential Component Knowledge of scientific, theoretical, empirical and contextual bases of intervention, including theory, research, and practice Behavioral Anchor Demonstrates knowledge of interventions and explanations for their use based on EBP Demonstrates the ability to select interventions for different problems and populations related to the practice setting Investigates existing literature related to problems and client issues Writes a statement of ones own theoretical perspective regarding intervention strategies  
26
Identify expected student learning outcomes
using competency model (e.g., Benchmarks)
Expected level of competence attainment and
training plan developed with trainee
Select assessment method(s)
Student demonstrates problems with competencies
acquisition
Remediation plan developed
Formative assessment
Student demonstrates emerging competencies
Training plan continues
27
Remediation Plan
Competency Domain/ Essential Components Problem Behaviors Expectations for Acceptable Performance Trainees Responsibilities/ Actions Supervisors/ Faculty Responsibilities/ Actions Timeframe for Acceptable Performance Assessment Methods Dates of Evaluation Consequences for Unsuccessful Remediation

28
Identify expected student learning outcomes
using competency model (e.g., Benchmarks)
Expected level of competence attainment and
training plan developed with trainee
Select assessment method(s)
Student demonstrates problems with competencies
acquisition
Remediation plan developed
Formative assessment
Student demonstrates emerging competencies
Training plan continues
Summative evaluation to assess readiness for next
level of training
29

• For more information
• http//www.apa.org/ed/graduate/competency.html
• Coming soon
• Kaslow, N. J., Grus, C. L., Campbell, L. C.,
  Fouad, N. A., Hatcher, R. L., Rodolfa, E. R.
  (in press) Competency Assessment Toolkit for
  Professional Psychology. Training and Education
  in Professional Psychology.