Building of Large Public Databases For Validation of Software Tools

1
Building of Large Public DatabasesFor Validation
of Software Tools

• Laurence P Clarke
• Barbara Croft, Carl Jaffe, Gary Becker, John
  Freyman and Dan Sullivan
• NCI CIP
• John Haller
• NIBIB

2
Rationale Todays ImagingCancer Research
Practice
Undeveloped PotentialSoftware Tools

• Detection
• Tedious
• Inefficient
• Observer-dependent
• Not reproducible
• Diagnosis
• Inaccurate
• Invasive confirmation required
• Response to therapy
• RECIST criteria inadequate
• Delayed decisions on individual care
• Development of new therapeutics
• Drug studies lack imaging surrogates
• Trials costly too many patients
• Long path to market patient benefit

• Automated / semi-automated
• Efficient
• Less observer-dependent
• Reproducible
• Accurate classification
• Noninvasive
• Improved objective measures
• Early individual care decisions
• Early go/no-go decisions
• Trials less costly, fewer patients
• Shorter path to market

3
Image database objectives
Rationale

• Make image data trustworthy. How?
• Validated analytic software tools for
• Lesion detection, classification
• Accelerated diagnostic imaging decision
  throughput
• Quantitative imaging assessment of drug response
• Image guidance software for interventions
• Missing ingredient Image Database Resources

4
Stakeholders
Anticipated BenefitsOf Image Databases

• NCI-FDA IOTF
• Imaging equipment companies
• Pharmaceutical companies
• Insurance carriers
• Radiology practitioners
• Researchers
• Oncology practitioners
• Patients
• Healthcare policymakers
• Foundation of NIH

• Public resources
• Benchmarking on common cases
• Accelerated CAD development
• Efficient, reliable detection
• Accurate, noninvasive Dx
• Objective assessment of Tx
• Confident clinical decisions
• Endpoints acceptable to FDA
• Streamlined new drug approvals
• Informed coverage decisions

5

• Two Demonstration Projects

6
Lung Image Database Consortium (LIDC)

• NCI ACRIN/NLST (2002-2008 200M/9 years) is
  supporting a large lung cancer screening trial
  using helical CT, where subsets of images can be
  collected.
• LIDC (7M/5 years 5 Academic Sites 2001-2006).
  Goals Develop an image database for the
  comparison of CAD methods, and encourage
  standards for assessment of application-specific
  software.
• LIDC Steering Committee Includes academic
  members, FDA CDRH and NIST scientists and
  consultants to ensure a broad understanding of
  all the design issues involved.
• Project Progress Population of database started,
  data on CIP webpage.

7
Annotation Consensus Nodule?
Calcified Nodule
Spiculated Nodule
Scar
8
Annotation Nodule Characteristics
LibraryStandardize Radiologist Interpretation.
9
Multi site Un-blinded Reads from four Readers
4/4 Markings
2/4 Markings
2/4 Markings
Multi Site Annotation Label every object in the
3D image
10
LIDC Timeline
Data Accrual 23 nodules Evaluate Segmentation
Methods Completed
Imaging Protocol Complete
Project initiated LIDC
Project Scale Up / 100 Cases New Annotation
Sites
Process Model Annotation Method Completed
Nodule Drawing Expt Completed
Lung Nodule Library Completed
Project Completion 400 cases
Data Accrual 30 cases May 2005
9/01
9/06
6/03
12/04
2/05
5/05
6/05
12/04
12/04
11
REFERENCE IMAGE DATABASE to EVALUATE
RESPONSE to Drug Therapy in Lung Cancer

• RIDER Pilot Project Aims
• Pilot database 200 advanced lung CA patients
  serial CT exams.
• A step toward NCI imaging informatics
  infrastructure
• Expert consensus on database design
• Enable industry academia to develop, test,
  compare semi-automated, automated software tools
  for change analysis

12
Barrier to consistent data Drug Response using CT
Database Design Accommodate different methods
for assessment of change analysis? Probability
map of boundary?
13
Diameter17.7 mm
Diameter17.1 mm
MSKCC DATA
14
RIDER Pilot Project Timeline
1st Steering committee meeting Use-Cases Infrastru
cture design Prototype developed MIRC Field
Center created IRB approvals
Test case material selected Server installs Field
center installed, tested at sites IRB approvals
RIDER MIRC Implementation complete IRB
Approvals Work begins on database design 30
cases CIP webpage
RSNA/MIRCNCI meeting NCI-RIDER Group charts
course De-Identification schema developed
SEPTEMBER GOALS 1) 200 Cases on webpage 2)
Consensus on database design
Project initiated RIDER team created
9/04
10/04
11/04
12/04
9/05
1/05
15
Database Demonstration Projects as PPPs.

• Lung Image Database Consortium (LIDC) Database
  to permit the benchmarking of CAD methods for
  lung nodule detection and diagnosis in a
  screening and early diagnosis context.
• Public Private Partnership (PPP) in final stages
  of completion with 8 CT and/or PACS industries
  with NEMA guidance. Start Date 4/27/05.
• Reference Image Database to Evaluate Response
  (RIDER) Database to permit the benchmarking of
  software metrics such as volumetric measures of
  tumor response to drug or radiation therapy.
• Public Private Partnership (PPP) being explored
  with the CT, PACS and Pharmaceutical industry.
  Targeted date 9/05 -12/05.
• caBIG Possible Long Term Objective Creation of
  an array of reference databases for imaging and
  genomics through a broad based PPP.

16
Business Model Leveraging of Resources

• Clinical Data Collection Costs are covered by on
  going clinical investigations. Support is
  required for archiving of data sets and related
  image annotation to access software performance.
• CaBIG Web accessible methods to query this
  public resource are being developed as an
  integral part of the caBIG Imaging Workspace
• Public Private Partnerships (PPPs) Engage
  cancer center, academic, and the device and drug
  industry communities to develop and support
  public databases. Includes FDA and NIST
  scientists with a goal to accelerate PMA
  approval, and to standardized assessment of
  informatics tools.
• Developers Engage the broader scientific
  community to develop more advanced software tools
  without concern about data collection.
• Physician End Users Encourage this community
  (RSNA etc) to require more standardized methods
  for software evaluation so that informatics tools
  will be widely accepted by the radiology
  community.

17
NCI Business ModelCurrent and Future Clinical
InvestigationsPotential Sources of Data and
TimelinesI
Demonstration Project LIDC
Lung Drug Response Pilot Project
Digital breast- ACRIN
Enhance hardware and software infrastructure with
NCICB
Digital breast- ACRIN
Enhance hardware and software infrastructure with
NCICB
Colonoscopy collection ACRIN
Liver mets - MSK
PET alliances
Colonoscopy collection ACRIN
Liver mets - MSK
Ultrasound
MRI - ACRIN
Ultrasound
MRI - ACRIN
NTROI optical database
Colonoscopy collection - Navy
NTROI optical database
Year 1
Year 2
Year 3
CTEP Reengineered Clinical trials Source of new
data. IOTF NCI and FDA Biomarkers for Drug
response
18
LIDC RIDER IT caBIG and RSNA MIRC
RIDER Investigative Sites
NCI
RIDER caIMAGE Web Portal
DICOM Images/ Annotations
Researchers
CT Scanner
caIMAGE Annotations Database Meta Data
DICOM Annotations
caIMAGE Server
Image
Cancer Centers
PACS
Parser
DICOM
FTP
RSNA MIRC Field Center
NCI MIRC Encrypted Files
Academia
DICOM
DICOM/ HTTPS
CIP PACS
De-identification
Firewall
Firewall
DICOM Images/ Annotations
Quality Review NCI Workstations 3rd Floor EPN
Industry
19
Cancer Centers

• Open Standards
• Data Base Interoperability

Pharmas
Web Engineering Data Queries
NCI Repository
Academia

• Communication
• Common
• Data Elements

Device Mfg.
Software Validation Annotation?
FDA

• Development of Consensus
• On Methodology

• PAs Targeted applications such as informatics
  tools for drug response

20
caBIG Imaging Working Space
Long Term Plans.

• Web access to image and all related meta data
  from targeted clinical, small animal and
  pathological data.
• Engage the cancer center, academic and industry
  community in annotation of images and related
  data.
• Engage the cancer center, academic and industry
  community in development of open source tools to
  access and analyze data.
• Develop consensus process for evaluation of
  software tools such as image noise suppression,
  registration, image enhancement and display.
• Goal Develop a resource to stimulate the
  development of informatics tools and harmonize
  methods for analysis

21
NCI Informatics Infrastructure (caBIG) Image
Database to be Fully Integrated into caBIGLong
Term Validation of Data Integration Tools.
GRID Principles

• Open source
• Open access
• Open development
• Federated

Group Nodes Cancer Centers SPORES CGAP MMHCC Othe
rs
22
(No Transcript)
23
Panel Discussion Points

• Imaging Protocols Can we harmonize across
  platforms for targeted applications such as for
  imaging in drug response trials?
• Plug and Play Software can we implement on
  imaging platforms to harmonize image processing
  methods (ITK, others)?
• Open Source Software Is there a business model?
• Industry-Academic Partnership Software
  Development?
• Software Validation Is there a business model
  for engagement in the development of web
  accessible resources?
• caBIG Working Space Potential interest to
  industry?
• Role of Federal Government NCI, NIBIB, FDA,
  NIST?
• Other NEMA, FNIH, RSNA IHE etc.

24
(No Transcript)
25
(No Transcript)
26
Trans NIH BECON/BISTI 2004 Symposium Goals
Informatics for Clinical Decision Support

• Symposium Goals The development and
  standardized validation of
• software tools for clinical decision
  support. Scope included
• Biomedical imaging, genomic, gene expression, and
  patient
• medical records data for personalized medicine.
• Data integration, knowledge extraction, and
  clinical interpretation,
• of heterogeneous clinically relevant data.
• Linking imaging and other databases with software
  tools.
• Database development for software validation/ FDA
  approval.