Ready REAIM Fire

1
Ready ? RE-AIM ? Fire!

• My HealtheVet Performance Evaluation
• Class 221

Kim Nazi ? Kathleen Charters
2
Agenda

• In this session
• A Brief Overview of My HealtheVet
• Why Evaluation is Critical
• Evaluation Goals
• The RE-AIM Framework
• Developing the Indicators
• Current Initiatives Results
• Whats Next
• How You Can Participate

3
Brief Overview of My HealtheVet
My HealtheVet Pilot Program

• Medical Record Extracts
• Self-entered Metrics
• Patient Health Education Libraries
• Delegation

• Demographics, Admissions, Appointments,
  Prescriptions
• Vitals Allergies
• Progress Notes, Discharge Summaries
• Basic Problem List information
• Lab Reports Chemistry, Microbiology, Microscopy,
  Cytology, Pathology
• ECG Radiology reports
• Wellness Reminders
• Co-payment Balances

www.health-evet.va.gov
N 7500 pilot participants at nine VAMCs
4
Brief Overview of My HealtheVet
My HealtheVet National Program

• Online Prescription Refills
• Health Education Information
• Condition Centers
• Self-assessment Tools
• Health Journals e-Logs
• Veteran Specific Conditions
• Seasonal Health Reminders
• Wellness Calendar
• One Stop Benefits/Services
• and More

www.myhealth.va.gov
More than 440,000 Registered Users
Incremental releases of new features (VistA EHR,
Delegation, Secure Messaging)
5
Brief Overview of My HealtheVet
Healthy Living Centers Healthy Eating Physical
Activity Center Smoking Tobacco Use
Cessation Mental Health Screening Tools Alcohol
Use Screening (AUDIT-C) Depression Screening
(PHQ-9) PTSD Screening (PCL) Substance Abuse
Screening (ASSIST) Medical Library MedlinePlus
HealthWise Service Related Conditions
(VHI) Spinal Cord Injury Post Traumatic Stress
Disorder Traumatic Brain Injury Hearing Impairment
Common Conditions Age-Related Macular
Degeneration Bipolar Disorder Cataracts Colds
Colorectal Cancer Depression Diabetes
Diabetic Retinopathy Glaucoma Coronary Heart
Disease Heart Failure High Blood Cholesterol
Hypertension Influenza Post Traumatic Stress
Disorder Schizophrenia Stroke
6
Evaluation Is Critical
The issue of measuring investments in IT is
critical. There is still a widespread lack of
understanding of IT and information systems as a
major capital asset. rely on both tangible
and intangible measures to determine a systems
contribution to an organization. Murphy
Simon (2002, p. 246)
7
Why Evaluation is Critical

• Ex-ante evaluation
• Justify IT investment before it is made
• Ex-post evaluation
• Justify costs that have been incurred
• Guide future IT expenditures
• Challenge
• Adequate measurement of costs benefits
• Across system life cycle
• With sensitivity analysis

8
Why Evaluation is Critical

• First most important criteria
• Does the system contribute to strategic
  objectives of the organization?
• Improved information quality
• Improved access to information/services
• Improved use
• Able to adapt to change
• Supports high quality decision-making

9
Why Evaluation is Critical

• Promotional
• Basis for clinical adoption
• Extend knowledge
• Structure, function, impact
• Pragmatic
• What works best
• Ethical
• Medico-legal
• Safe and effective

10
Evaluation Goals

• Optimize program to improve veterans health care
• Evidence-based knowledge to support clinical
  adoption
• New collaborative relationships with VA Research
• Agency requirements (i.e., OMB 300 Program)
• Inform external partnerships (i.e., AHIC)
• Contribute to field of health care informatics

11
RE-AIM Framework
RE-AIM Framework
Russell Glasgow and colleagues www.re-aim.org
12
RE-AIM Framework
RE-AIM

• Number of registrations visits over time
• Demographic profiles of registered users
• Target populations who can most benefit from use

Reach Effectiveness/Efficacy Adoption Implementati
on Maintenance
13
RE-AIM Framework
RE-AIM
Identify and assess key targeted outcomes
Quality (clinical outcomes) Access (utilization
management) Value (cost) Satisfaction
(perceptions)
Reach Effectiveness/Efficacy Adoption Implementati
on Maintenance
14
RE-AIM Framework
RE-AIM

• Track PHR adoption by patients providers
• Compare adoption across sites settings
• Identify how PHR tools offer clinical utility in
  practice

Reach Effectiveness/Efficacy Adoption Implementati
on Maintenance
15
RE-AIM Framework
RE-AIM
Reach Effectiveness/Efficacy Adoption Implementati
on Maintenance

• System Availability
• Cost Analysis
• Component Utilization
• Implementation Benchmarks
• Best Practices

16
RE-AIM Framework
RE-AIM

• Longer term impacts of targeted outcomes
• ( 6 months)
• Program milestones
• Clinical adoption patterns
• Sustainability

Reach Effectiveness/Efficacy Adoption Implementati
on Maintenance
17
RE-AIM Framework (Extended!)
18
Putting the Pieces Together
19
Putting the Pieces Together
20
Web Analytics

• 440,565 Registered Users as of July 07

21
Web Analytics

• 11.2 Million Visits as of July 07
• Yearly Comparisons for 2004, 2005, 2006, 2007
  1st QTR

22
Web Analytics
3.3 Million Online Prescription Refill Requests
Processed since August 31, 2005
227,630 Refills processed in June 2007 Up 97
over last June Averaging about 6,900
prescription refills daily On June 25, processed
over 10,000 Rx requests, the biggest one-day
volume ever
Monthly Total September 2005 June 2007
23
Data Mining
My HealtheVet Registrant Age Distribution March
2007
24
Data Mining
My HealtheVet Registrant Gender Distribution
June 2007
12
88
25
Customized Indicators

• Conceptual indicators
• Operationalize using template
• Requirements analysis

Collect Data, Analyze, and Report
Optimize Program
26
Satisfaction Surveys
ACSI
My HealtheVet Results June 2007
27
Satisfaction Surveys
American Customer Satisfaction Index (ACSI)

• Industry standard tool for measurement of
  customer satisfaction
• Provides scores, priority maps, benchmarks
• Advanced computational methodology
• OneVA Initiative My HealtheVet segmentation
• Measures over 100 federal programs 100 federal
  websites
• E-Gov index published quarterly
• Government index published in December

28
Satisfaction Surveys
Measure Customer satisfaction on a continuous
basis Focus on drivers of satisfaction and the
resulting behaviors Identify improvements
Diagnose areas of improvement Predict impact of
change on satisfaction behavior Prioritize
improvements based on their ability to drive
ROI Benchmark With leading agencies/companies
in your category and across industries
29
Satisfaction Surveys

• oneVA Data Collection Period
• February 1, 2007 April 30, 2007
• N 13,017 respondents
• Sampling Parameters
• Sampling percentage 1.35
• Loyalty factor 6
• Response Rates
• VA average 8.5
• ForeSee average 6.9
• My HealtheVet Segmentation
• Opt-In links and Help Desk Receipt
• Links added April 1, 2007

30
Satisfaction Surveys
My HealtheVet Site

• Scores are calculated for drivers of
  satisfaction, overall satisfaction and future
  behaviors
• Each driver of satisfaction has multiple
  attributes (multiple questions)
• Impacts are calculated (Latent Variable Partial
  Least Squares Regression)
• Impacts show resulting improvements in
  satisfaction and loyalty based on enhancements
• Impacts help prioritize website enhancements

31
Satisfaction Surveys
When compared to the aggregate, visitors are more
likely to return to the site when they receive
the survey via site delivery on the My HealtheVet
pages. However, visitors who received the survey
on My HealtheVet pages exhibited more difficulty
with Site Performance than the aggregate.
My HealtheVet Site delivery
VA Aggregate
32
Satisfaction Surveys
Segmentation AnalysisMy HealtheVet Site delivery
The majority of MyHealtheVet visitors received
the survey via site delivery Satisfaction did
not differ between the two delivery methods The
analysis focuses on those who received the survey
via site delivery since this group represents the
largest percentage
33
Satisfaction Surveys
1. What best describes you? 
Description
92 veterans 5 family member 1 VSO
33
34
Satisfaction Surveys
Highlights

• The majority are veterans
• Content scored highest, navigation scored lowest
• Active duty vs. Congress vs. News Media (small n)

35
Satisfaction Surveys
2. What is your age range? 
36
Satisfaction Surveys
37
Satisfaction Surveys
3. How frequently do you visit this site? 
38
Satisfaction Surveys
39
Satisfaction Surveys
4. What information were you primarily looking
for? 
40
Satisfaction Surveys
Highlights

• As in the other VA segments, visitors are
  primarily searching for information relating to
  prescriptions
• For this group content had the highest score,
  navigation had the lowest score
• Health care related information is also a
  subject of interest

41
Satisfaction Surveys
5. How were you trying to locate the information? 
42
Satisfaction Surveys
43
Satisfaction Surveys
6. Did you find what you were looking for? 
44
Satisfaction Surveys
Highlights

• Most visitors found what they were looking for
• However, one out of four visitors did not this
  group is 30 points less satisfied than those who
  did
• Those who did not are much less likely to return
  and recommend
• Those who did not rated navigation search
  lowest

45
Satisfaction Surveys
7. What online services would you most like to
see? 
46
Satisfaction Surveys
47
Satisfaction Surveys
8. If you used the search feature, what type of
difficulty if any did you encounter with the
search functionality? 
48
Satisfaction Surveys
Highlights

• Those who did not encounter difficulties with
  search were much more satisfied
• Further segmentation needed to drill into
  other
• Satisfaction much lower if difficulties
  encountered in search

49
Satisfaction Surveys
Priority Map
Site Performance, Search and Navigation are the
primary drivers of customer satisfaction for
visitors who received the survey via site
delivery on the My HealtheVet pages. VA should
focus on these areas in order to improve customer
satisfaction for this group of visitors.
50
Satisfaction Surveys
ForeSee
Visitors who received the survey via site
delivery on the My HealtheVet pages are primarily
concerned with the degree to which the number of
steps to get where they want is acceptable and
their ability to find information. They also
expressed difficulty with the speed of loading
the pages, consistency of speed on the site, how
the site provides comprehensive results and how
the search feature helps to narrow the
results. Improvements in these Top Priority
areas will provide the greatest opportunities to
increase visitors overall satisfaction with the
site.
51
Satisfaction Surveys
VA Aggregate Benchmarking
Drivers of Customer satisfaction
52
Satisfaction Surveys
VA Aggregate Benchmarking
Customer Satisfaction and Future Behaviors
53
Satisfaction Surveys
One out of three visitors who could not find what
they were looking for were primarily searching
for information relating to prescriptions.
Fourteen percent were looking for health care
information.
VA Aggregate
Note Visitors representing less than 4 percent
are not shown.
54
Satisfaction Surveys

• Access to medical records
• Agent orange info
• Aid attendance benefits
• Appointments
• Burial benefits
• Claim status
• Contact information
• Dental benefits
• Detailed job descriptions
• VA medical records
• Trying to refill prescription
• To refill my medications
• Prescription name to match Rx number
• Names of my medication
• The name of the prescription
• Test results
• Refills
• Nursing positions

VA Open Ended Comments
Open-ended comments relating to specific items
that visitors did not find..
55
Satisfaction Surveys
ForeSee
VHA Homepage
One of the primary reasons why visitors come to
the VA site is to look for information relating
to prescriptions. Since this is a very important
topic for site visitors, it would be very
beneficial if information relating to
Prescriptions was made more visible to the
eye. Consider adding a Prescription link on the
left or top navigation bars. Or instead of
having the subtitle Renew Prescriptions under
Information for Patients, consider adding a link
titled Prescriptions on the home page as well.
56
Satisfaction Surveys
VA Homepage
httpwww.myhealth.va.gov/prescription https//ww
w.myhealth.va.gov/mhvPortal/anonymous.portal?_nfpb
true_pageLabelrxRefill
VS
57
Satisfaction Surveys
MHV Rx Refill page
Add informative information about IPA and
medication names
Additional usability testing for navigation
58
Satisfaction Surveys

• Action Items to Improve Satisfaction
• Follow up on open ended comments
• Add more direct link to Prescription refills info
  on VA and VHA Homepages (create an alias to
  shorten the portal link) couple link with
  content about IPA and medication names
• Add information about IPA and medication names to
  Rx Refill page
• Add to appropriate MHV section, additional
  information about when lab results and
  appointments will be available

59
Satisfaction Surveys

• Add link to MHV Contact Us and VA IRIS form in
  close proximity to open ended survey question.
  Include text warning against submitting personal
  information via the survey.
• Develop a VA News Release to enhance awareness of
  Prescription refills, IPA, and Medication names
• Add custom questions to drill down into
  prescriptions and other areas
• Follow up with date of next MHV Release for
  portal integration
• Take steps to improve navigation and relevancy of
  search results

60
Satisfaction Surveys

• MHV Pilot Evaluation
• Veteran Survey
• Provider Survey
• Focus Groups 

VHA Survey of Healthcare Experiences of Patients
(SHEP)
National My HealtheVet Satisfaction Survey
61
Research Collaborations

• Content development
• Formalize MHV Research Agenda
• Responding to requests
• Development of projects aligned with priorities
• Policy development
• RFP
• Scientific evidence to drive clinical adoption

62
Research Collaborations

• Research Agenda examples
• Identify target populations who can most benefit
  from use of the personal health record
• Evaluate clinical performance indicators for
  users vs. nonusers, e.g., chronic diseases
• Compare Quality of Life Measures (VR12) for
  users vs. nonusers

63
External Collaborations

• AHIC Workgroups
• PHR Collaborators Workgroup
• Markle Consumer Authentication Workgroup
• Markle Consumer Access Workgroup
• Kay Center
• Paralyzed Veterans of America
• DoD
• Collaborative Communications Summit
• Society for the Internet in Medicine

64
Using this Information

• Clinicians
• Identify content that should be added or modified
• Submit clinical content using the Product Web
  site
• Educate patients about using My HealtheVet
• POCs
• Assist users with known challenges

65
Field Participation
66
Field Participation
67
Field Participation

• Encourage users to participate in the surveys
• Take the survey when offered that option
• Contact the CAB Performance Evaluation sub-group
  if you wish to help
• Kim.Nazi_at_va.gov
• Kathleen.Charters_at_va.gov

68
Summary

• You have learned about
• My HealtheVet
• Why Evaluation is Critical
• Evaluation Goals
• The RE-AIM Framework
• Indicators
• Current Initiatives Results
• Whats Next
• How You Can Participate

69
Check it Out!

• My HealtheVet National Site
• www.myhealth.va.gov
• My HealtheVet National Demo Account
• www.myhealth.va.gov
• Username mhvuser
• Password mhvdemo1
• My HealtheVet Product Site vaww.va.gov/MyHealtheV
  et/
• My HealtheVet Training Site
• vaww.vistau.med.va.gov/vistau/mhv/

70
Questions Answers
Suggestions? Questions?