The Bethesda Interobserver Reproducibility Study

1
The Bethesda Interobserver Reproducibility Study

• Ritu Nayar, MD, MIAC
• Associate Professor of Pathology
• Director of Cytopathology
• Northwestern University Chicago, IL

2
The Bethesda System (TBS)

• Goals
• Communicate clinically relevant information from
  the laboratory to the health care provider
• 2. Terminology should be uniform and reasonably
  reproducible, and flexible enough to be adopted
  in a variety of laboratory and geographic
  settings
• 3. Terminology must reflect the most current
  biologic understanding of cervical neoplasia

3
The Bethesda Workshops

• 1988
• NCI sponsored workshop held in December 1988
• Established a bipartite division of HPV effect
  and categories of dysplasia into LSIL/ HSIL
• 1991
• In April 1991 a second workshop was held to
• Evaluate the impact of TBS in actual practice
• Amend and modify it where needed
• One major recommendation was formulation of
  precise criteria for diagnostic terms and
  descriptors of adequacy
• Subsequently, the first Bethesda monograph was
  published (1994) to disseminate this information

4
The Bethesda System (TBS)

• Used by gt 90 of US laboratories
• Introduction of ASCUS atypical squamous
  cells of undetermined significance
• Most problematic and controversial terminology
  for clinicians
• Major problem in the USA
• Approx. 60 million Pap tests done annually in the
  US
• Approximately 7 reported as abnormal
• Among these 5 are ASCUS
• Another 1.2 are LSIL

5
The Bethesda System (TBS), HPV Research, and ALTS

• 1990s
• Advances in the understanding of the biology of
  cervical cancer and HPV carcinogenesis showed
  that the majority of LSIL regress
• Introduction of liquid based cytology
• HPV testing became a viable option for cervical
  cancer screening
• Realizing the optimal management of ASCUS and
  LSIL was unclear, the NCI sponsored the ASCUS/
  LSIL Triage Study (ALTS)

6
ASC-US/ LSIL Triage Study (ALTS)

• Randomized multicenter trial to compare 3
  management strategies for women with ASCUS and
  LSIL cytology (1996-1998)
• Avoid over-referral and over treatment for the
  majority of women with clinically insignificant
  changes
• Provide safety for the small of women with
  under diagnosed high grade lesions

7
Bethesda 2001 Workshop

• Pre-Workshop internet bulletin board discussion
  forum established for drafting of recommendations
  for presentation at the face to face workshop
• Between Oct 2000-April 2001, gt1000 comments were
  posted (http//bethesda2001.cancer.gov)
• Bethesda 2001 workshop held April 30th-May 2nd at
  the Natcher conference center on the NIH campus
  in Bethesda, Maryland
• gt 500 individuals from over 24 countries meet and
  finalized the 2001 Bethesda terminology
• Solomon D, et al, The 2001 Bethesda System
  Terminology for Reporting Results of Cervical
  Cytology. JAMA 2002 287 2114-2119.

8
American Society for Cytopathology-National
Cancer Institute (ASC-NCI) Task Force
9
Bethesda 2001 and the American Society For
Colposcopy and Cervical Pathology (ASCCP)

• In concordance with TBS 2001 terminology and
  based on results from ALTS and other studies, the
  ASCCP drafted consensus guidelines for the
  management of women with cervical cytologic and
  histologic abnormalities
• In September 2006, the second ASCCP consensus
  conference was held in Bethesda publication due
  April 2007
• WEBSITE http//www.asccp.org
• 2001 Consensus Guidelines for the Management of
  Women with
• Cytological Abnormalities JAMA
  (20022872120-2129)

10
ASC-NCI Task Force Visions

• Bethesda Atlas, 2nd Edition
• Illustrate 2001 terminology and include liquid
  based technologies.
• Reorganize into chapter format
• Keep price lt40
• Bethesda System Educational Website
• Create a user friendly web-based atlas with a
  gallery of images linked notes
• Bethesda Interobserver Reproducibility Study
  (BIRST)
• To gauge interobserver variability in the
  interpretation of cervical cytology

11
(No Transcript)
12
ASC Website www.cytopathology.org
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
The Bethesda Interobserver Reproducibility Study
(BIRST)

• Objectives
• Web-based interpretation of a subset of images
  from the Bethesda atlas prior to its publication
• Study of the correlates of interobserver
  reproducibility
• Evaluate concordance among participants with
  varied training and experience
• Identify specific cytomorphologic features and
  cytologic categories that represent sources of
  poor interobserver agreement
• NOT a mechanism to filter out images with poor
  reproducibility
• NOT a proficiency test!

19
(No Transcript)
20
Precision in Gynecologic Cytology Interpretation

• Bethesda Study (BIRST)
• LSIL most reproducible borderline categories
  (ASC-US/ ASC-H and AEC/ AEmC) least reliable
• A priori stratification of image difficulty was
  an excellent predictor of concordance between
  BIRST raters TBS panel
• Implies that among qualified raters,
  cytomorphology is a stronger determinant of
  agreement than reviewers characteristics

21
Precision in Gynecologic Cytology Interpretation

• CAP PAP Program Pathologists interpretations for
  validated (25,745) and educational (14,353)
  conventional slides in 2002
• Interpretations of Negative, Candida, Herpes,
  Trichomonas, and LSIL had a high degree of
  matching
• Interpretations of Repair, HSIL, Adenocarcinoma,
  Squamous cell carcinoma had lesser
  reproducibility
• CAP PAP Program Arch of Pathol Lab Med 2003
  127 1413

22
ALTS Diagnostic Reproducibility Study Stoler
JAMA 2001

• Cytology and histology are interpretative
  processes with similar reproducibility
  performance characteristics (moderate)
• ASCUS is the biggest source of cytologic
  disagreement
• CIN 1 is the biggest source of histopathologic
  discrepancy
• Findings bring into question concept of gold
  standard being the histology diagnosis,
  particularly for CIN 1

23
Negative for Intraepithelial Lesion or Malignancy
(NILM)

• BCC and WNL combined into one negative category
• No more mixed signals to clinicians/ patients
• Avoid confusion between BCC and reactive/
  reparative change
• NILM
• Organisms
• Other Non-Neoplastic findings
• Reporting optional BUT NILM should always be
  header in the General Category or Interpretation

24
Negative for Intraepithelial Lesion or Malignancy
(NILM)

• Areas with lower inter-observer reproducibility
• Atrophy
• Squamous metaplasia
• Parakeratosis
• Histiocytes
• Repair
• Reactive squamous cells
• Reactive endocervical cells

25
Comparison of Performance of Conventional (CS)
and ThinPrep (TP)

• Performance of interpretations in year 2002
  CS 89, 815 and TP 20, 886
• Overall, interpretations on TP had significantly
  lower FPR (1.6) and FNR (1.3) than CS as
  assessed by concordance with correct diagnostic
  series
• However participant responses indicated some
  difficulty in recognizing HSIL and SCCA (i.e.
  higher FNR) on TP compared to CS
• CAP PAP Program Arch Pathol Lab Med 2004
  128 17-22

26
NILM Significance of Repair/Reactive

• Potential for misinterpretation of repair or
  other reactive conditions in cytology
• Among all NILM interpretations, repair most often
  elicits a false positive response
• (CAP PAP program 1998 data Arch Pathol Lab
  Med 2001 125 134-140)
• BIRST analysis suggests that defining upper
  limits of reactive nuclear enlargement and
  distinguishing repair from high grade lesions
  represents an important concern in cytology

27
NILM

• Which non-neoplastic cellular changes need to be
• reviewed by a pathologist ?
• Personal comfort and laboratory policy
• Individual thresholds- interobserver variablilty
• Repair/Reactive terms yes in the USA
• Other entities ambiguous
• -glandular cells status post hysterectomy
• Valuable to look at trends in ones own laboratory

28
Endometrial Cells in Women gt 40

• What Cells Should be Reported as
• Endometrial Cells?
• Exfoliated cell clusters
• Intact cells
• Not abraded cells
• Not histiocytes

29
How to Report Endometrial Cells

• General Categorization
• OTHER See Interpretation/Result
• Descriptive Interpretation
• Endometrial cells present (See Note)
• Negative for squamous intraepithelial lesion
• OR
• Descriptive Interpretation
• Endometrial cells present (See Note)
• Negative for squamous intraepithelial lesion

30
What comment or note to use?

• Endometrial cells in a woman after age 40,
  particularly out of phase, or after menopause,
  may be associated with benign endometrium,
  hormonal alterations, and less commonly,
  endometrial/uterine abnormalities. Clinical
  correlation is recommended.
• If the LMP is provided and endometrial cells are
  seen in cycle in women 40 or older
• Endometrial cells correlate with the menstrual
  history provided.

31
Do endometrial cells have to be referred to a
pathologist for review?

• Not by CLIA regulations (Other)
• Individual lab practice
• Exclude atypical cells
• Assure endometrial cells are exfoliated, not
  abraded
• Examples
• Refer all with bleeding history
• Refer out of cycle 40-50 year old
• Refer all after a specified age

32
Atypical Squamous Cells (ASC)

• Atypical squamous cells of undetermined
  significance (ASC-US)
• changes s/o SIL but lack criteria for definitive
  interpretation 90-95 of ASC cases
• Approximately 50 are HPV
• Overall 10-17 have CIN 2 or 3
• Atypical squamous cells, cannot exclude HSIL
  (ASC-H)
• changes s/o HSIL, but lack criteria for
  definitive interpretation 5-10 of all ASC
• Approximately 85 HPV in women lt35 yrs and 40
  HPV in womengt 35 yrs
• Overall 25-50 CIN 2 or 3

33
IS THE FUSS ABOUT ASCUS WORTH IT ?

• Outcome of ASCUS Pap Tests (ALTS data)
• 26 have CIN follow-up over 2 years
• 15 LSIL
• 11 HSIL ( 6 CIN 2, 5 CIN3)
• Annually approx. 300,000 women with ASCUS harbor
  HSIL
• Reducing or eliminating ASCUS
• Sensitivity for detecting SIL/ HSIL was 100/
  100 utilizing ASCUS vs. 39/ 41 eliminating
  ASCUS
• Pitman M et al in Cancer Cytopath 2002 96128-34

34
ALTS Two-year Cumulative Dx of CIN by Original
Cytology
IC and HPV Arms
35
Atypical Squamous Cells, Cannot Exclude HSIL
(ASC-H)

• Cytologic changes suggestive of HSIL, but lacking
  criteria for definitive interpretation
• Not a biologic or morphologic entity but an
  expression of uncertainity
• Possible causes of cytologic interpretation of
  ASC-H include metaplasia, repair, endometrial
  cells, endocervical cell, HSIL and histiocytes
• Positive predictive value for high grade lesion
  ( gt CIN 2) is greater than ASC-US, but less
  than HSIL

36
Interobserver Variability in Sub Classification
of Squamous Intraepithelial Lesions

• Results of LSIL and HSIL referenced slides from
  the CAP PAP Program, 1996 and 1997
• Only 30 of SIL slides in circulation showed
  perfect discrimination between LSIL and HSIL
• Arch of Pathol Lab Med 1999 123 1079-1084
• CAP Q-probe study of 22, 439 cytology-histology
  correlations in 348 labs
• Biopsy F/up shows 18 LSIL have CIN 2/ 3 and 15
  of HSIL have CIN 1 on biopsy
• Arch Pathol Lab Med 1996 120 523-531

37
HSIL Interpretations

• Two important patterns to keep in mind
• Single small cells
• Easily missed
• Look in empty spaces in LBP, especially
  ThinPrep
• Streams in mucus, especially in conventional
• Hyperchromatic crowded groups
• Atrophy, benign endocervical cells, lower uterine
  segment/ endometrial cells
• Endocervical neoplasia, HSIL extending into
  glands

38
Detection of Squamous versus Glandular Lesions

• AIS Adenoca HSIL SCCA
• cases 213 2821 7535 1886
• False Neg
• Rate 11.7 8.9 4.6 3.3
• CAP Interlaboratory Comparison Program in
• Cervical Cytology (PAP) Program. Pathologists
• reviews in 2001 2002.
• Arch Pathol Lab Med 2004 128 153-157

39
BIRST classification of squamous versus
glandular lesionsSherman M, et al. The Bethesda
Interobserver Reproducibilty study. Cancer
Cytopathol 2007.

• Cytologic recognition of HG endocervical lesions
  is less sensitive than detection of HG squamous
  lesions
• Cytologic detection of endocervical
  adenocarcinoma poses many challenges in both
  range and subtlety of morphologic changes
• Improved detection of glandular lesions is an
  important outstanding challenge for cervical
  cancer screening

40
Cancer Cytopathology Publicationhttp//www3.inter
science.wiley.com/cgi-bin/jissue/91015807

• Original Articles
• Gynecologic Cytopathology
• The Bethesda interobserver reproducibility study
  (BIRST) A web-based assessment of the Bethesda
  2001 system for classifying cervical cytology
  (p NA) Mark E. Sherman, Abhijit Dasgupta, Mark
  Schiffman, Ritu Nayar, Diane Solomon Published
  Online 21 Dec 2006 DOI 10.1002/cncr.22423
• The Bethesda Interobserver Reproducibility Study,
  a web-based investigation of cervical cytology
  performance, demonstrated that among participants
  with more than 1 year of experience,
  cytomorphology is a more important determinant of
  interobserver agreement than participants'
  academic/professional degrees.
• Abstract