Supplement 145 Whole Slide Imaging background and design decisions

1
Supplement 145 Whole Slide Imaging background
and design decisions

• Harry Solomon GE Healthcare

2
DICOM Basics
3
DICOM Image Information Object Definition
DICOM Composite Information Model Hierarchy
Patient Information
Study Information
Series Information
Image (Instance) Information
Dwight Simon
4
Data Element Encoding
Attributes are the logical concepts associated
with an information entity Data elements are how
attributes are encoded in an information object
Data Set
order of transmission
Data Elem.
Data Elem.
Data Elem.
Data Elem.
Data Element
ValueRepresen-tation
Similar to TIFF
Value
Tag
Value Field
Length
optional field - dependent on
negotiated Transfer Syntax
0020000Dhex
UI
1.2.840.1.113709.9.0.0.5743.14575602.1
26hex
Study InstanceUnique Identifier(0020,000D)
Instance UID encoded as dotted decimal
5
Part of a DICOM object
Tags in increasing numeric order Value length
always an even number Attributes related to
modules and information model levels all jumbled
up
6
Attributes

• Logical concepts in the description of an
  Information Entity
• May have 0, 1 or many Values
• 0 (empty) means the creating application doesnt
  know the value of the attribute, e.g. Accession
  Number (0008,0050)
• Multi-value example Specific Character Set
  (0008,0005)
• Value Multiplicity (VM) specified in Part 6
  (possibly further constrained in Part 3)
• Attribute value will be a complex data structure
  for a Sequence attribute

7
Sequence Attributes and Items

• Sequence attribute has a value of a structure
  of subsidiary attributes
• Sequence Attribute name typically includes word
  Sequence
• Subsidiary attributes specified in Part 3 with gt
  character
• Each instantiated set of attributes is a Sequence
  Item
• Number of allowed Items specified in Part 3
• For editorial convenience the attributes of a
  Sequence are often documented in a separate Table
  as a Macro
• Include x Macro Table m-n
• Facilitates reuse of structure in other sequence
  attributes

8
Example Scheduled Protocol Code Sequence
attribute
9
Scheduled Protocol Code Sequence attribute
10
Sequence attribute encoding

• Sequence Items are the values of Sequence
  attributes
• Structure placed in the Data Element Value Field
• Item structure is a nested data set of
  attributes
• Attributes in each Item in tag order
• Item wrapped using special data elements
  specified in Part 5
• Sequence attributes and wrappers may have an
  undefined length flag
• Length of Sequence or Item terminated by explicit
  Delimiter data elements

May be undefined length
Sequence Data Element
ValueRepresen-tation SQ
Value
Tag
Value Field
Length
Item Introducer
Attribute 1
Attribute 2
Item Delimiter
Item Introducer
Attribute 1
Attribute 2
Item Delimiter
Sequence Delimiter
Specifies length of Item, or may say undefined
length
Required if undefined length Item
Required if undefined length Sequence Attribute
11
Image Compression

• Pixel data can be monochrome, color (RGB or
  YCbCr), or palette color (monochrome colorized
  through LUT)
• No definitions yet for hyperspectral, but it has
  been discussed
• Pixel data can be native DICOM (with color
  by-plane or by-pixel)
• Pixel data can be compressed using standard
  compression schemes, and compressed stream put in
  pixel data element
• JPEG, JPEG-LS, JPEG2000 (each lossy or lossless)
• MPEG2
• Run-Length Encoding (Packbits)
• Private compression schemes can also be used

12
Compressed Image Encoding

• Uses structure similar to Sequence attribute
• Allows undefined length attribute eliminates
  232 byte limitation
• 1st Item is Basic Offset Table - pointers to
  individual frames of a multi-frame image
  (optional)
• JPEG and JPEG2000Part1 encode each frame of a
  multi-frame image in a separate Encapsulated
  Stream Fragment
• JPEG2000Part2 (multicomponent) allows arbitrary
  mapping of frames to stream fragments to allow
  component collections (inter-frame compression)

May be undefined length
Pixel Data Element
ValueRepresen-tation OB
Tag
Value
Value Field
Length
(7FE0,0010)
Item Introducer
Basic Offset Table
Sequence Delimiter
Item Introducer
Encapsulated Stream Fragment 1
Item Introducer
Encapsulated Stream Fragment 2
Specifies length of Basic Offset Table
Specifies length of Stream Fragment
Required if undefined length Pixel Data
Attribute
13
Multiframe Images
14
Enhanced Multi-frame paradigm

• Basic concept used for all new multiframe IODs
• MR (Image and Spectroscopy), CT, XA, US, PET
• Multi-frame object to support 1000 image studies
• Dynamic image header supports functional or
  acquisition attributes changing during scan
• Dimensions allow multiple views of data
• File size flexibility through concatenations

15
Single-frame to MultiFrame
N Objects, N Headers
N Frames, One Header
16
Functional Groups and the Per-Frame Header
17
Functional Groups

• Collection of set of closely related attributes
• A mini Module
• Structured as a sequence of (usually 1) item
  under a main Sequence attribute
• Invoked as a Macro in either Shared Functional
  Groups Sequence or Per-Frame Functional Groups
  Sequence
• Keeps items together in encoding under the main
  Sequence attribute

18
Dimensions properties that may change
echo
cardiac phase
b-value
orientation
time
position
volume
time
19
Multi-phase / Multi-slice
Phase (Time) Position Index
1
2
3
Slice Order for phase 1
Phase order for slice 2
Frame number 1-6
Frame number 7-12
Frame number 13-18
Image frames can be sorted/displayed independent
of encoded frame order
20
Concatenations

• What is a concatenation?
• set of image objects
• in the same series
• with the same dimension indexes
• uniquely identified with a Concatenation UID
  (0020,9161)
• contained image objects must have the same
  Instance Number
• Why?
• file system limits e.g., 600 MB CD-R
• pseudo real-time transfer of a stream of images
• workstation needs to post process images in near
  real time to figure out when the scan is to be
  terminated

21
Concatenations
An object may be split up into two or more SOP
Instances, using the same concatenation UID
22
Image Retrieval
23
DICOM Query/Retrieve

• Allows a system to query another system for a
  list of available images (query)
• Also allows a system to request another system to
  send images (retrieve)

24
Hierarchical Query

• DICOM query is not a full SQL-type feature
• Limited attributes, no Join capability
• Directed toward production imaging department
  requirements
• Hierarchical data structure
• (Patient), Study, Series, Image levels
• Patient attributes typically subsumed in Study
  level
• Query at any level requires specification of
  unique entity at each higher level

25
Typical Hierarchical Query
26
Classical Hierarchical Retrieve

• Retrieve can be at any hierarchical level
• (Patient), Study, Series, Image
• Retrieve at any level requires unique ID of
  entity at each higher level
• Object transfer can be on separate Association
  (C-MOVE) or on same Association (C-GET)
• C-MOVE object transfer can be directed to third
  party
• Examples
• Retrieve all objects under Study UID
  1.2.789.87.11
• Retrieve all objects under Study UID
  1.2.789.87.11 / Series UID 1.2.405.31.1
• Retrieve single object Study UID 1.2.789.87.11 /
  Series UID 1.2.405.31.1 / Instance UID
  1.2.405.31.1.99.1
• Retrieved objects sent and confirmed as wholes

27
Interactive JPIP Retrieve

• Image Store SCU and SCP can negotiate a JPEG 2000
  Interactive Protocol (JPIP) Transfer Syntax
• Image header (i.e., entire object minus pixel
  data) transferred and confirmed as usual
• Pixel data replaced by URL to JPIP service for
  this image
• Limitations
• Pixel data must be in JPEG 2000 format
• Storage Commitment not allowed
• Duration of availability of JPIP not specified or
  guaranteed
• Capabilities
• Retrieve subset of image (ROI)
• Retrieve at a lower resolution (e.g., for quick
  navigation)

28
Frame-based retrieve
New in 2009Supplement 119

• Retrieve subset of frames from a multi-frame
  image
• Selected frames of a volumetric stack (ROI)
• Decimated volume (e.g., every 10th slice)
• Single dimension of a multi-dimensional image
• Time snippet of motion image (video)
• SCU SCP negotiate Instance Root Retrieve SOP
  Class
• SCU specifies selected frames or time interval
• SCP creates new multi-frame image with derivation
  attributes
• Frame Derivation Module and Contributing
  Equipment Sequence
• Correct subset of Functional Group Sequence Items

29
Vocabulary and Structured Reporting
30
Vocabulary-intensive messaging

• Theres a lot of things we want to say about
  imaging that cannot be pre-defined in fixed DICOM
  attributes
• E.g., specimen processing
• How do we define message attributes to handle
  what we need to say?

31
Name-value pairs
lt BodyPartExamined ABDOMENPELVIS /gt
ltelgt ltname BodyPartExamined /gt ltvalue
ABDOMENPELVIS /gt lt/elgt
ltelgt ltname code00180015 systemDICOM
meaningBody Part Examined /gt ltvalue
codeR-FAB57 systemSNOMED meaningAbdomen and
pelvis /gt lt/elgt
Why would we want to do this?
32
External coded/concept terminologies

• Flexibility and extensibility
• Leverage externally defined/maintained concepts
• Semantic rigor through referenced dictionary/
  ontology
• General structure higher layer of abstraction
• Allows generalized messaging applications
• Shared vocabulary across disparate systems

33
SNOMED

• Systematized Nomenclature of Medicine
• Most comprehensive clinical healthcare
  terminology
• 375,000 concepts 900,000 relationships between
  concepts
• Multi-hierarchically organized
• Primary external vocabulary system for DICOM
• Anatomy
• Procedures (including radiographic views and
  methods)
• Clinical findings
• Originally developed by the College of American
  Pathologists, now managed by an international
  consortium of governmental agencies (IHTSDO)

34
LOINC

• Logical Observation Identifier Names and Codes
• Standard coding system for laboratory and
  clinical observations
• Hosted by Regenstrief Institute
• Supported by National Library of Medicine
• Particularly focused on names of laboratory and
  clinical tests
• 50,000 codes over 275,000 relationships
• Major external code system for DICOM and HL7

35
Code Sequences
DICOMPart 3
Triplet coding code value, scheme,
meaning (version seldom used)
36
Context Groups (Value Sets)
DICOMPart 3
DICOMPart 16
37
Content Items

• Generic NameValue pair using external coding for
  Name concept
• Encoded as Item in Sequence attributes
• Acquisition Context Sequence (in image IODs)
• Protocol Context Sequence (in Modality Worklist)
• Content Sequence (in Structured Reporting IODs)
• Specimen Preparation Step Sequence (in Specimen
  Module)

Person Name Value (0040,A123)
DateTime Value (0040,A120)
Referenced SOP Sequence(0008,1199)
Text Value (0040,A160)
UID Value (0040,A124)
SOP Class UID(0008,0050)
SOP Instance UID(0008,0055)
Content Item
Concept Name Sequence (0040,A043)
Value Type (0040,A040)
Concept Value Sequence (0040,A168)
Numeric Value (0040,A30A)
Measurement Units Sequence(0040,08EA)
Code(0008,0100)
Scheme(0008,0102)
Meaning(0008,0104)
Code(0008,0100)
Scheme(0008,0102)
Meaning(0008,0104)
Code(0008,0100)
Scheme(0008,0102)
Meaning(0008,0104)
38
Templates

• Structure for Content Items - like Modules are a
  structure for Attributes
• Specified in DICOM Part 16

39
Annotation and Segmentation
40
DICOM annotation principles

• Annotations are conveyed in information objects
  separate from the original image
• Annotations may be created at a time much later
  than the image acquisition, and in a completely
  different environment
• Multiple annotation objects can reference the
  same image
• Selection of an annotation object for display
  implicitly invokes display of the referenced image

41
Annotation types

• Presentation States
• Structured Reporting
• Segmentation

42
Presentation State

• Softcopy Presentation States define how
  referenced image(s) will be displayed
• Transforms to device independent grayscale/color
  space (LUTs)
• Selection of display area (ROI) of the image
• Image rotate or flip
• Graphical and textual annotations, overlays,
  shutters
• Grayscale, color, and pseudo-color SPSs
• Blending SPS overlays a pseudo-color image on a
  grayscale image
• E.g., for PET/CT
• Blending on grayscale originals (currently no
  standard for blending of color originals)

43
Structured Reporting

• Presentation State annotations are for human
  reading, not interoperable for automated
  applications
• No controlled and coded vocabulary, no structural
  semantics (relationships between annotations)
• SR important for (semi-)automated imaging
  analysis and review processes

44
Key Image Note

• SR-type object that provides a classification and
  a textual comment for a referenced object
• Formally known as Key Object Selection, but
  commonly denoted Key Image Note after IHE use
  case and profile
• Classifications typically identify intended
  subsequent use of referenced objects
• For Referring Provider, For Research, For
  Report Attachment
• Rejected for Quality Reasons, Signed Complete
  Study Content

45
Segmentation

• Derived image object
• Uses enhanced multi-frame mechanism
• Multiple segments per object
• Each segment linked to a categorization
• Pixels show presence of category at pixel
  location
• Binary (1-bit/pixel) or fractional (probability
  or occupancy)
• Segmentation object is typically in same Frame of
  Reference as source image
• Segments can be displayed as overlays on source
  image

46
Segmentation Example
47
Pathology in DICOM Specimen and Workflow
48
Whats NOT in Sup145

• All the modules already standardized
• Patient, Study, Series, Equipment, General Image
• Multi-Frame Functional Groups and Dimensions
• Sup122 Specimen Module
• Explicit description of workflow
• Use of Modality Worklist, Modality Performed
  Procedure Step, Image Availability Notification,
  etc.

49
Sup 122 Specimen Identification

• Support for pathology lab workflow,
  specimen-based imaging
• Gross specimens, blocks, vials, slides
• Image-guided biopsy samples
• Specimen Module at image level of hierarchy
• Identification, processing history
• May be used with current Visible Light image
  object definitions
• Update to Modality Worklist to convey Specimen
  Module
• Enables automated slide scanning devices to fully
  populate header
• Update to Modality Performed Procedure Step to
  identify imaged specimen
• Allows LIS/APLIS to track images for specimens

50
Specimen Imaging Information Model
Disambiguates specimen and container Container is
target of image Container may have more than one
specimen Specimens have a physical derivation
(preparation) from parent specimens When more
than one specimen in an imaged container, each
specimen is distinguished (e.g., by position or
color-coding)
Basic DICOM Information Model
51
Preparation Step

• 0-n Preparation Steps per Specimen
• Each Preparation Step described by 1-n structured
  Content Items (namevalue pairs)
• Acquisition Context plus structuring into steps
• DICOM Template 8001 Specimen Preparation

52
Preparation steps example
53
Managed Workflow Concepts (IHE)
ORDER A request for departmental service
REQUESTED PROCEDURE Unit of work resulting in
one Report with associated codified, billable
acts

• PROCEDURE STEP The smallest unit of managed
  work in the workflowScheduled Procedure Step
  A unit of work to doPerformed Procedure Step
  A unit of work done

54
Simple Workflow

• One Order One Procedure One Study One Report

Imaging Department
ORDER A request for DepartmentalService
Set of Codifiable, Billable, Acts
Report
Requested Procedure
Charles Parisot - IHE
55
Multiple Modality Steps
Imaging Department
ORDER A request for DepartmentalService
Set of Codifiable, Billable, Acts
Requested Procedure
56
Anatomic Pathology Imaging Workflow
Slide preparation
Workstation
Gross specimen accessioning
Whole Slide Scanner
Surgical or biopsy procedure
57
Sup145 Whole Slide Imaging Proposal
58
Sup145 multi-frame tiling concept
Use multi-frame image objects (not object per
tile)
Single frame image
Thumbnail Image
Multi-frame image (single object)
Intermediate Image Tiles
Multi-frame image (single object)may include
multipleZ-planes, color planes
Baseline Image Tiles
In 1 or more DICOM Series
59
Alternate approach (not in draft!)
Remove 64k2 image matrix restriction Can leverage
JPEG2000 Part2 multi-component compressionUse
JPEG Interactive Protocol capabilities
JPIP low-res view of baseline image
JPIP medium-res view of baseline image
Baseline Image
Multi-frame (Z-planes, colors) image (single
object)
60
Total Pixel Matrix
Total Pixel Matrix Origin

• Total pixel matrix origin at top left hand corner
  of imaged volume
• Frame (tile) rows and columns align with total
  pixel matrix rows and columns
• Frames limited to 216 columns and rows
• Total pixel matrix limited to 232 columns and
  rows

Columns ?
Rows
?
Frame Pixel Matrix Origin
61
Z-planes

• Z-planes are identified as nominal physical
  height of image focal plane above reference
  surface (µm)
• Z-plane information is used for relative spatial
  positioning of image planes, and nominal
  inter-plane distance
• An image plane may track variable specimen
  thickness / surface contour, but only one Z-value
  used

? Z
Cover slip
? Z
Specimen
Slide substrate (glass)
62
Z planes track curved surface

• Z plane 1, Z plane 2, Z plane 3, Z plane 4

Viktor Sebestyén Varga 3DHISTECH Ltd.
63
Organization of tiles into objects

• All valid

Single Multi-frame image
Multi-frame image per Z-plane
Multi-frame image per spatial region
64
Sparse tiling
Multi-frame med-res image
Multi-frame hi-res image

• Only selected tiles encoded
• Full image matrix might be encoded at lower
  resolution

65
Localizer Image Flavor

• Thumbnail image (single frame) plus
  multi-resolution navigation links
• Each tile of other resolution images has its
  corresponding area identified in thumbnail
• Full description of target tiles
• Object UID and frame
• Resolution
• Z-plane
• Multiple target frames can overlap
• Different resolution, Z-plane, color, etc.
• Presentation and any interactive behavior is not
  defined in standard

66
Label Image Flavor

• Purpose is to capture slide label
• Any specimen captured is irrelevant
• Image IOD includes Slide Label Module
• Barcode (if deciphered)
• Label Text (if deciphered)
• Burned In Annotation (0028,0301) might be NO if
  the label includes only a specimen identifier and
  not patient identifying data

67
C.8.12.2 Slide Coordinates

• Used in VL Slide-Coordinates Microscopic Image
  IOD
• Single frame image, typically from
  microscope-mounted camera
• Used to localize center of VL SCM Image
• DICOM Frame of Reference associated with slide
  corner origin
• Reproducibility not guaranteed across different
  mountings of slide, even on same equipment

Label
? Y
Specimen
VL SCM Image area
X ?
Slide Coordinates Origin
Cover slip
? Z
Specimen
Slide substrate (glass)
68
C.7.4.1 Frame Of Reference Module

• When a Frame of Reference is identified, it is
  not important how the imaging target is
  positioned relative to the imaging equipment or
  where the origin of the Frame Of Reference is
  located. It is important that the position of the
  imaging target and the origin are constant in
  relationship to a specific Frame Of Reference
• The Position Reference Indicator may or may not
  coincide with the origin of the fixed frame of
  reference related to the Frame of Reference UID.
  The Position Reference Indicator shall be used
  only for annotation purposes and is not intended
  to be used as a mathematical spatial reference.

69
WSI Image Pixel Matrix
Columns ?

• Image Matrix not necessarily aligned to slide
  edge, nor to Slide-Coordinates
• Image Matrix origin (top left hand corner)
  located relative to Slide-Coordinates Frame of
  Reference origin (X,Y in mm)
• Direction of rows and columns given as cosines in
  Slide-Coordinates Frame of Reference
• Each tile (frame) TLHC located relative to Image
  Matrix origin (column, row)
• Each tile center located relative to
  Slide-Coordinates origin (X,Y in mm)

Rows ?
X ?
? Y
Image Matrix Origin
Slide-Coordinates Origin
Cover slip
? Z
Specimen
Slide substrate (glass)
70
Optical paths

• Each combination of light source, lenses,
  illumination method, detected wavelengths, etc.
  used in an acquisition is an optical path
• Each path described in an Item of the Optical
  Path Sequence
• Examples
• Full spectrum light, transmission, RGB color
  sensors
• uV light, excitation, blue monochrome sensor
• Each frame may specify a different optical path
• Allows different colors in a single object,
  including hyperspectral (n monochrome planes)
• Identified in Optical Path Functional Group by
  reference to Optical Path Sequence Item

71
Optical paths

• What parameters are needed?
• To be added in Part 16 Context Groups
• Is a macro image simply a selected optical path?

Illumination Color(s) Intensity Type (laser)
Filters Color(s) Polarization
Lens
Illumination Method Transmission Reflection Scatt
er Excitation
Lens
Filters Color(s) Polarization
Sensor Color(s)
72
WSI Functional Groups

• Standard
• Pixel Measures (pixel spacing, layer thickness)
  shared
• Frame Content (datetime, dimensional location)
  per-frame
• Referenced Image, Derivation (if required for
  individual frames)
• WSI Specific
• Plane Position (relative to total matrix and to
  SCM FoR)
• Optical Path
• Specimen Reference (if multiple specimens on
  slide are automatically distinguishable)

73
Pixel Measures functional group
Redefinition
74
Plane Position (Slide) functional group
Do we need to separate to two attributes to
support independent dimensions?
Do we really need this sequence introducer
(consistency w/ C.8.12.2)? Could just specify X
and Y as center point coordinates
75
Dimensions

• Based on attributes in functional groups (i.e.,
  values that change on a per-frame basis)
• Typical dimensions for WSI
• Total Matrix Column Origin
• Total Matrix Row Origin
• Z-Plane
• Optical Path (color/polarization)
• Attributes used for Dimensions specified in
  Multi-frame Dimension Module
• Each frame specifies its dimensional indexes in
  Frame Content functional group
• Index values (ordinals) mapped to dimensional
  attribute values
• E.g., with (Column, Row) dimensions, and 40962
  pixel frames, frame with index value (2,3) would
  have origin column\row values of 8193\12289

76
Annotations of WSI - Segmentations

• Segmentations can be created frame-by-frame /
  pixel-by-pixel against selected frames of
  original image
• Reference through Derivation Image Functional
  Group
• 1-bit/source-pixel, or 8-bits/source-pixel
• Segmentations can be created against arbitrary
  areas within a specified Frame of Reference
• Requires Plane Position and Plane Orientation
  Functional Groups may not be usable with slide
  coordinates Frame of Reference
• Display of segmentation can implicitly invoke a
  non-standardized overlay or blending with source
  image
• Segmentation can specify its preferred color

77
Annotations of WSI - Presentation States

• Color Presentation State supports annotation of a
  source image
• Displayed Area Selection allows up to 231-1
  rows/columns, currently relative to frame-based
  rows/columns
• Proposed enhancement (with new attribute and new
  enumerated value) to allow Displayed Area
  Selection and annotation location relative to WSI
  total matrix, rather than to frame
• Implicitly applies to all dimensions (Z-planes,
  colors), only constrained by explicit frame
  numbers should there be a general mechanism to
  limit by dimension (as is done for segments)?
• Placement of annotations limited to 24-bit
  precision (IEEE 754 32-bit float)
• May be image relative or selected display area
  relative
• Allows sub-pixel resolution up to 8M rows/columns
  for image relative

78
Presentation State - Displayed Area Selection
79
Presentation State - Graphic Annotation Units
80
Modality Worklist

• Scheduled Specimen Sequence added to MWL in
  Sup122
• Allows query by Container ID (slide barcode)
• Allows return from SCP of complete Specimen
  Module (slide processing history to be used for
  imaging set up and/or inclusion in WSI header
• Other parameters can be passed in Protocol
  Context Sequence
• Template specification for Content Items
• Proposed Protocol Optical Paths Sequence
• Parallel to Protocol Context Sequence
• General VL attributes