Readability of Participant Information Sheets: what can User Testing contribute

1
Readability of Participant Information Sheets
what can User Testing contribute?
School of Healthcare

• Jon Silcock and Peter Knapp
• NRES Northern Conference 26th March 2009

2

• Jon Silcock is chair of an NHS Research Ethics
  Committee, but the views expressed are not
  necessarily those of the National Research Ethics
  Service.
• Peter Knapp is a Director of Luto Research
  Limited, a Leeds University spin-out company that
  writes and tests patient information.

3
Learning outcomes

• Understand the importance of the readability of
  Participant Information Sheets.
• Distinguish between content-based and
  performance-based assessments of readability.
• Understand how User Testing might identify
  strengths and weaknesses in Participant
  Information Sheets.

4
Workshop plan

• Introduction the participant information sheet
  readability
• Group work Whats good and bad about this PIS?
• Suggested PIS problems evidence and expert
  opinion
• Assessing readability content
• Assessing readability performance
• Group work User Testing
• User Testing in practice
• Summary

5
The role of the PIS in consent

• IRB / REC approval
• Participant Information Sheet opportunity to
  ask questions (Helsinki Declaration)
• Role of the PIS
• Pre-consent informs explains provokes
  questions
• Post-consent reminder record
• Phase 1 trials PIS complex, long, technical new
  medicine compensation payment data protection
  
• Phase 3 trials PIS how the trial treatments vary
  from standard NHS care impact on other care
  follow-up frquency length

6
Problems with consent

• Important information not always stated
• Patient understanding not checked (Jenkins et al
  1999)
• 40 participants did not know they could
  withdraw (Lynoe et al 1991)
• 33 participants did not know the trial was
  primarily for research (Lavori et al 2007)
• TGN1412 Phase 1 trial information consent
  questioned in reports (Expert Scientific Group
  2006 Royal Statistical Society 2007)

7
Literacy issues

• Many written at the college graduate level
  (Paasche-Orlow et al 2003)
• Most forms require a level of literacy that is
  too high (Burman et al 2003)

8
TGN1412 PIS (Northwick Park)

• Group work
• Read the two pages of this Phase 1 trial
  Participant Information Sheet.
• In groups of 3
• Discuss what you think are the problems of the
  PIS with regards to readability

9
Problems with TGN1412 PIS

• Content
• Long, difficult or unfamiliar words
• Long sentences
• Layout
• Headings
• Dense text
• Paragraphs
• No bullets or lists

10
Assessing readability

• Expert opinion
• Test against criteria (eg NRES)
• Content-based readability assessment
• Performance-based readability assessment

11
Content-based assessment

• Many US IRBs require a readability formula test
• Easy to use
• Readily available, e.g. http//www.editcentral.c
  om/gwt/com.editcentral.EC/EC.html
• Generate a score, often based on word and
  sentence length

12
Version 1

• This study will involve the use of a new compound
  (QDG1234, described as the study drug in the
  remainder of this consent form) being developed
  for the treatment of various inflammatory
  diseases such as rheumatoid arthritis, and
  possibly also for a particular type of leukaemia.
  This study drug is a "monoclonal antibody" which
  means that it is an antibody (a naturally
  produced protein) that has been designed by
  scientists to target a particular type of cell
  in this case a type of white blood cell called T
  cells.

13
Version 2

• This is a study of a new drug called QDG1234. We
  will call it the study drug in the rest of this
  information sheet.
• It is being developed to treat illnesses caused
  by inflammation, such as rheumatoid arthritis.
• It may also be used for a type of leukaemia.
• This study drug is an antibody called a
  "monoclonal antibody".
• An antibody is a protein which is produced
  naturally in your body.
• Antibodies help the body fight infections, such
  as those caused by bacteria or a virus.
• The study drug is designed to target white
  blood cells called T cells. These cells help
  the body produce antibodies and help the body
  fight infection.

14
Content comparison

15
Helpful

• ...but crucially
• Cant indicate meaning
• Intravenously given is drug the The drug is
  given intravenously
• Cant indicate performance

16
Alternative assessment

• Performance-based readability testing
• EU law for medicine package leaflets (EC. Draft
  Guideline on the readability of the label and
  package leaflet...Sept 06)
• Can people find and understand key points of
  information?

17
User Testing process

• Key points of safety and efficacy selected
• Questionnaire written (12-15 Qs)
• Rounds of 10 people
• 90 items found and, of those, 90 understood
• Any written text can be User Tested

18
Group work

• In groups of 3
• User test the snippet of a trial PIS using the
  questionnaire
• One person is the interviewer
• One person is the interviewee
• One person observes
• Discuss what you find

19
Recent projects

• User testing of 3 trial PIS
• TGN1412 (Phase 1)
• Poor Responders IVF (Phase 3)
• SHARP cholesterol reduction in renal disease
  (Phase 3)
• Test original
• Re-write and re-design
• Test revised version.

20
TGN1412 original PIS

• 1 round of 10 participants
• Many items not found (12 / 210)
• Many items not understood (21 / 210)
• Six items found and understood lt8/10
• Emergency telephone number
• Presence of placebo
• Action required if have insurance
• Informing the GP
• Pre-drug fasting
• Number of clinic visits
• Would have failed a readability test

21
TGN1412 revised PIS

• Re-wording
• Short sentences
• Lay language
• Remove repetition
• Clear structure headings
• Table of contents
• Headlines section
• Re-design
• A4 booklet
• Page numbers
• Use of bullets for lists

22
TGN1412 revised PIS

• 2 rounds of 10 participants
• Few items not found (2 / 420)
• Few items not understood (4 / 420)
• No item found and understood by less than 19/20
• Would have Passed a readability test
• Asked which PIS version they preferred 17/20
  (85) chose the revised version

23
Comparison

• Original PIS Revised PIS
• Total items 12 / 210 2 / 420
• not found
• Total items 21 / 210 4 / 420
• not understood
• Article in press at Journal of Medical Ethics,
  Knapp, Raynor, Silcock Parkinson.

24
SHARP Phase 3 trial PIS

• Original version
• Some difficult words and sentences
• Performed generally well
• No items below 90 thresholds
• Many items difficult to find
• Revised version (revised twice)
• Most items found and understood by all
  participants
• No item below 90 threshold
• Fewer found with difficulty ratings
• Most participants preferred the revised version

25
Poor Responders IVF Phase 3 trial PIS

• Original version
• Some difficult words and sentences not well laid
  out looks amateurish
• Complex dosing schedule
• Did not perform well
• Information mostly found, but not understood
• Revised version
• Most items found and understood by all
  participants
• Improved understanding scores
• 17/20 participants preferred the revised version

26
Summary

• The Participant Information Sheets role in
  consent
• Different methods of assessing readability
• Content-based assessment quick but very limited
• Performance-based assessment (User testing) used
  for licensed medicines
• User testing can identify strengths and
  weaknesses small numbers needed
• Need for expert writing and design to make
  information usable
• j.silcock_at_leeds.ac.uk p.r.knapp_at_leeds.ac.uk