Ellen Zager Hill, MS

1
Refugee and Immigrant B Notification Program
EvaluationIdaho

• Ellen Zager Hill, MS
• TB Program epidemiologist
• Idaho department of health welfare

2
Idaho Geography (1)

• Geographically large western state with a low
  population
• 19.2 people per square mile (ranked 44th in US)
• Population 1.58 million people
• 37 of population is in the Boise metro area (Ada
  Canyon Counties)

3
Idaho Geography (2)

• 44 counties divided into 7 multi-county regional
  public health districts
• 95 of refugees and immigrants settle in Boise
  (Ada County) and Twin Falls (Twin Falls County).

4
Background

• For a state with the size population that Idaho
  has, the state receives an unusually large number
  of refugees and immigrants
• 2 local public health districts receive
  essentially all of the refugees arriving in the
  state PHD4 (90) and PHD5 (10)
• Immigrants arriving in Idaho are pretty evenly
  distributed in SFY 20072009, 83 immigrants with
  TB notifications arrived

SFY Idaho Population No. of Refugees Arriving Per capita ratio(per 10,000 people) No. with TB notifications
2007 1,499,402 834 5.6 30 (3.5)
2008 1,523,816 1,080 7.1 54 (5.0)
2009 1,545,801 1,291 8.4 76 (5.8)
5
Evaluation Plan
6
Logic Model
7
Evaluation Plan (1)

• Evaluation Goal

• Evaluation Team

• Baseline analysis of medical screen activities of
  refugees and immigrants arriving in Idaho with a
  TB notification

• Lead Evaluator
• Ellen Zager Hill TB PEN Focal Point/TB Epi
• Team Members
• Christine Hahn TB Controller/State Epi
• Adele Smith MPH student from ISU

8
Evaluation Plan (2)
Big Picture Questions Corresponding Specific Questions
Are the medical evaluations for refugees and immigrants with TB notifications done in a timely manner? What is the proportion of refugees or immigrants that were evaluated within 90 days of arrival?
Are the PHDs sending notifications back to the OEFI in a timely manner? What is the proportion of TB notification forms that are received within 120 days of receiving the notification?
What is the process that the PHDs go through to contact the refugees and immigrants with TB notifications for medical evaluations? 7 out of 7 PHDs have a process to contact refugees or immigrants with TB notifications
9
Evaluation Plan (3)
Big Picture Questions (continued) Corresponding Specific Questions (continued)
Do the PHDs have a different process for evaluating and treating refugees and immigrants with TB notifications? 7 out of 7 PHDs have no difference in evaluation or treatment of refugees and immigrants with TB notifications
What is the process the PHDs perform to evaluate refugees and immigrants with TB notifications? 7 out of 7 PHDs have a process to evaluate refugees or immigrants with TB notifications.
What is the process the PHDs perform to treat refugees and immigrants with TB notifications? 7 out of 7 PHDs have a process to treat the refugees or immigrants with TB notifications who have LTBI.
10
Evaluation Plan (3)

• Data Sources

• Execution Needs

• Idaho Refugee Immigrant Log (MS Excel
  spreadsheet)
• CDCs Electronic Disease Notification System
• Interviews with PHD staff who coordinate TB
  activities (via phone)

• Time
• Survey questions
• Phone
• Software to map processes (MS Visio)
• (relatively) Clean TB notification data
• Data analysis tool (MS Excel and SAS)

11
Carrying Out theEvaluation Plan
12
Carrying Out the Evaluation Plan (1)

• Developed a set of questions to ask the PHD staff
  who carry out TB activities
• Developed a script to go with the questions

13
Carrying out the Evaluation Plan (2)

• A flow of TB notification data from the
  state-level perspective was developed (shown at
  left)
• Based on survey questions, flow charts of the
  evaluation and treatment processes were developed
  for each PHD

14
Carrying out the Evaluation Plan (3)

• Data Cleaning

• Data Analysis

• Started with our Idaho Refugee Immigrant Log
• Took out all records that were not for TB
  notifications
• Filled in what missing data we could from hard
  copy records and the EDN system

• Initially done with MS Excel
• Final analysis done with SAS
• Benefit of using a program like SAS is that you
  have a record of exactly how the data was
  cleaned, what analysis was done, and how it was
  done.

15
Carrying out the Evaluation Plan (4)

• Some lessons learned
• Logs (especially those with no data
  standardization programmed in) are a difficult
  data source to analyze
• (A lesson re-learned) small denominators can be a
  problem
• Clearly document how you collect, clean, and
  analyze data
• An evaluation report is like a lab report.
• Your methods section may be your most important
  section. (Sometimes even more important than your
  results.)
• SAS or other statistical analysis programs are a
  great way to document how you are cleaning and
  analyzing data

16
Evaluation Results
17
Evaluation Results (1)

• Are the medical evaluations for refugees and
  immigrants with TB notifications done in a timely
  manner?
• What is the proportion of refugees or immigrants
  that were evaluated within 90 days of arrival?

• Overall (SFY20072009) 67 were evaluated within
  90 days of arrival.

18
Evaluation Results (2)

• Are the PHDs sending notifications back to the
  OEFI in a timely manner?
• What is the proportion of TB notification forms
  that are received within 120 days of receiving
  the notification?

• Overall 76 of TB notification forms were
  returned within 120 days of receiving the
  original notification.

19
Evaluation Results (3)
Process Questions Results
7 out of 7 PHDs have a process to contact refugees or immigrants with TB notifications All 6 PHDs interviewed have a process (the 7th is known (anecdotally) to have a process. Most have a process for contacting hard-to-reach R/Is and a defined point when they declare someone lost-to-follow-up
7 out of 7 PHDs have no difference in evaluation or treatment of refugees and immigrants with TB notifications Of the 6 PHDs interviewed only 4 have received notifications for both refugees and immigrants. Of these 4, only 1 reported a difference in protocol (a different staff member follows each group)
7 out of 7 PHDs have a process to evaluate refugees or immigrants with TB notifications All PHDs have a process for evaluating refugees and immigrants There is significant variation in these processes between PHDs
7 out of 7 PHDs have a process to treat the refugees or immigrants with TB notifications who are diagnosed with LTBI. Of the 6 PHDs interviewed, 5 actively follow R/Is on treatment for LTBI. The remaining PHD prefers that R/Is diagnosed with LTBI are treated and followed in their medical home.
20
Evaluation Results (4)

• Extra analysis tied to differences in processes

Excludes data from 1 PHD (the PHD with the most
notifications) because we were unable to
interview them.
21
Recommendations

• PHD-level

• State-level

• Ensure there is a system in place to track TB
  notifications
• Review process for coordinating medical
  evaluation on a semi-regular schedule
• Consult OEFI TB staff as needed when a problem
  related to medical evaluation arises

• Consider adding language to TB contracts
  stipulating a time frame for returning TB
  follow-up forms
• Update B Notifications chapter of Idaho TB manual
• Provide at least 1 NTIP-like report a year to
  each PHD that contains an analysis of their TB
  notification statistics
• Review data from calendar years 2009, 2010, and
  2011 for improvement. If improvement is not seen,
  then consider re-evaluating this activity.

22
What has been done as a result of the evaluation

• We are no longer using the Idaho Refugee and
  Immigrant log to monitor performance for this
  objective. We are using solely EDN data.
• We added language to contracts stipulating when
  TB follow-up data needs to be submitted.
• B Notifications chapter of TB manual has been
  reviewed an updated, but has not yet been
  finalized.

• Completion of medical evaluation statistics as of
  1/31/2012 (calendar years)

2008 2009 2010 2011
N 48 93 88 71
Mean (days) 147 132 143 70
Std Dev (days) 162 163 155 43
Low CI (days) 99 98 111 60
High CI (days) 194 166 176 80
Min (days) 15 8 15 4
Max (days) 885 795 743 190

1. This data reflects only notifications for which
   data has been received.
2. Please note that data for all years are
   incomplete for various reasons.
3. The improvements in data are mainly due to
   process changes at the PHD that receives 90 of
   refugees in the state. (This PHD was not
   interviewed for the evaluation.)

23
Questions