ECSE6963, BMED 6961 Cell

1
ECSE-6963, BMED 6961Cell Tissue Image Analysis

• Lecture 5 3-D Multi-Spectral Microscopy
• Badri Roysam
• Rensselaer Polytechnic Institute, Troy, New York
  12180.

2
Important Announcement!

• On Monday, Sept 15th, I will be at a conference.
  Here is a link if you are curious
• http//www.hhmi.org/janelia/conf-021.html
• computer vision image analysis are closely
  related fields
• Voice-annotated lectures will be on the course
  website
• Please download and play them on a computer with
  sound turned on (just type F5)
• Save your questions until our Sept 18th class

3
Recap

• 3-D Scanning Microscopy
• The multi-photon effect is a powerful basis for
  3-D imaging
• Second Harmonic Generation Imaging (SHG) is a
  lossless kind of multi-photon microscopy for many
  molecules (e.g., collagen)
• Fluorescent Proteins and multi-photon is a
  magical combination that allows live-cell imaging
• Confocal Microscopy can do 3-D microscopy without
  the multi-photon effect
• Significantly improves z-axis resolution (axial
  resolution) compared to ordinary widefield
  microscope
• Today
• Multi-Spectral Imaging (for fluorescence
  multiplexing)

4
Faster Confocals

• Spinning Disk Systems
• Nipkow Disks
• Scans lots of points at once using a rotating
  disk with a spiral array of holes, and a CCD
  camera instead of photomultipler tubes
• The basis for all modern high-throughput
  microscopes

5
Multi-Spectral Imaging

• Basic Motivation
• How can we capture multiple fluors at once?
• Fluorescence multiplexing
• We want to capture the relative spatial context
  of two or more fluorescently labeled structures
• Solution
• Build instruments that allow us to adjust two
  things in unison
• the excitation wavelengths
• Put a filter wheel in front of a broadband source
  to select illumination wavelengths
• Use multiple and/or tunable light sources
• the wavelengths that our detector is sensitive to
  (spectrally resolved detection)
• Use optical filter wheels in front of detectors
• Use a prism or a diffraction grating to split the
  detected beam, and an array of light detectors

Array of detectors
6
5-D Multi-photon Microscope
Image Acquisition Control Computer
Image Signal
PMT Gain Controls
Wavelength Control
Pulsed fs Ti-Sapphire Laser
PMT 1
PMT 2
PMT 3
PMT 4
Power Control
Mirror
Power Attenuator
Visible Emission Light
Sync Signal
560nm
495nm
515nm
Resonant Scanning Mirrors
Short pass dichroic
Long-pass dichroics (typical wave length cutoffs)
Near-IR Excitation Light
Objective Lens
Piezo Z control
Specimen
7
(No Transcript)
8
The Zeiss META System
Diffraction Grating
Detector Array

• The fluorescence light after the pinhole is
  passed through a grating to an array of 32
  detectors (PMTs)
• Produces a lambda stack I(x, y, z, ?)
• A spectrum at each pixel!

9
5-Label Immunohistochemistry

• Nuclei
• Blood vessels (EBA)
• Neurons (Nissl)
• Astrocytes (GFAP)
• Microglia (Iba1)

Excitation spectra
Emission spectra
50 ?m
10
5-Label Immunohistochemistry

• Nuclei
• Blood vessels (EBA)
• Neurons (Nissl)
• Astrocytes (GFAP)
• Microglia (Iba1)

Excitation spectra
Emission spectra
50 ?m
11
Dealing with Overlapping Spectra
1
2
Nucleus histone GFP Fusion Actin filaments
fluorescein conjugated phalloidin The peaks are
separated by only 7nm !
12
Dealing with Overlapping Spectra
Reference Spectra
Unmixing Result A1 and A2
Compute A1 and A2 at each pixel subject to
constraint A1 A2 1
13
Ultimate Optical Microscope of the Future

• Isotropic and high-resolution sampling of 3-D
  space (x, y, z)
• Recent microscopes have broken past the Rayleigh
  resolution limit
• No wasted photons 100 detection
• Recent microscopes perform 4 pi imaging
• Complete spectrum at each pixel
• Measure absorption emission spectrum
• Complete flexibility to shape the excitation
  spectrum
• Complete flexibility to capture and analyze the
  emission spectrum
• Complete lifetime response at each pixel
• Photon counting hardware at each detector
• Time response at different spectral wavelengths
• Multiple modalities looking at the same specimen
• One of the holy grails that continues to be
  pursued

14
Recap Image data

• 2D image I(x, y)
• Matrix of point measurements
• Point pixel
• Pixels have
• Size ?x, ?y
• Non-isotropic images ?x ? ?y
• Dynamic range 2N
• 3D image I(x, y, z)
• Point voxel
• Axial extent ?z
• Image sequence I(x, y, z, t)
• Time sequence of 2D/3D images
• Temporal interval ?t
• Multi-spectral image I(x, y, z, ?)
• Each pixel/voxel is vector valued
• Each element spectral band
• Multi-modal image I(x, y, z, c)
• Each pixel/voxel is vector valued
• Each element imaging modality

Pixel intensity
15
Image Files Metadata

• Some practical issues
• The file is a linear data structure
• We need to pay attention to how the
  multi-dimensional data is expanded into a linear
  array
• Need to know the bit ordering within each data
  point
• Meta data data about the image data
• Usually stored in the file header
• Currently, image file headers can be quite
  elaborate
• They can store information on where the data came
  from (provenance), microscope settings user to
  record the image, etc.
• TIFF allows a free text field in the header where
  one can store additional information
• OME TIFF uses XML file formats
• Lots of tools available to convert between file
  formats, viewing images, etc.
• For our purposes ImageJ MATLAB are adequate.

Header
Image Data
16
Quantitative Image Analysis

• The process of generating measurements of
  biological interest from image data
• Can be thought of as the generation of additional
  metadata
• Nature of measurements
• We are interested in measurements at the level of
  objects, and groups of objects, rather than at
  the level of pixels
• Objects usually correspond to biologically
  meaningful entities
• Implies that we need to extract objects from
  images first!
• This is the hardest task, the rest is much easier.

17
Steps From Images to Insight

• Step 1 Image pre-processing
• Cleanup image data (suppress imaging artifacts)
• Unmix the data into channels
• Step 2 Delineate objects (segmentation)
• Accurately delineate all valid objects, reject
  invalid objects
• Step 3 Validation Morphometry
• Correct segmentation errors
• Compute intrinsic object measurements
• Step 4 Object classification
• Step 5 Associative object measurements
• Step 6 Statistical Graphical analysis
• Concise, insightful summary

Very Important!!
18
Recap Reasons to Prefer Fluorescence Imaging

• Simplicity
• Assuming that we our fluorescent label is
  specific enough, we can image a specific
  substance, structure, or a class of substances/
  structures selectively
• Images only show these labeled things
• Bright pixels tell us where the structures of
  interest are, and dark pixels are background
• Multiplexing
• You can use multiple fluorescent labels to image
  several related things simultaneously
• Helps us to measure relationships among objects

19
Common Object Morphologies in Fluorescence Images
F
Foci on Barrier
M
Plate / Barrier
P
Foci in ECM
Extra-cellular matrix
F
C
Tube-associated Foci
Neurites
Nuclear membrane
F
T
Cytoskeleton
S
C
Nucleus
Cell membrane
B
Intra-nuclear foci
S
Cytoplasmic foci
F
F
Microvasculature
T
20
Pure Channels

• Channel
• The image data from each fluorescent label (or
  imaging modality) is commonly referred to as a
  channel
• Pure Channel
• A channel is considered pure if it only
  contains one type of object
• The fluorescent label is sufficiently specific to
  the object
• There is negligible spectral overlap (cross
  talk)
• Major advantages
• Software for making measurements is much
  simplified
• Specialized segmentation algorithms for each type
  of object can be much simpler since they only
  need to be able to handle one object type
• High-performance software possible
• We can exploit the specialization to develop
  highly reliable segmentation algorithms

21
Impure Channels

• There are basically two kinds of impurities to
  consider
• Case 1 Morphologically impure
• The fluorescent label is not specific enough
• We can get two or more types of things in a
  channel
• Solution 1 (preferred) work with the biologist
  to either choose different things to label, or
  different labels if at all possible
• Solution 2 develop algorithms that can handle
  morphologically mixed data
• Case 2 Spectrally impure
• The fluorescent labels are specific, but their
  spectra overlap heavily
• Solution 1 (preferred) seek out alternative
  fluorescent labels
• Solution 2 computationally unmix the data

22
Example from Neuroscience Research

• The niche (microenvironment) in which adult
  neural stem cells live
• A Complex Dynamic System
• Multiple interacting cell types
• 3-D vascular relationships
• 3-D Spatial polarity
• Axes of asymmetry for divisions
• Lineage relationships
• Multiple molecules of interest
• Signaling relationships
• Transport phenomena
• Molecular gradients
• Cell migration dynamics
• Gene regulation mechanisms

Ependymal
Immature Precursor
Migrating Neuroblasts
Astrocyte
B
C
A
GFAP- Dlx2 LeX
GFAP- Dlx2 PSA-NCAM
GFAP LeX
23
4-Color Imaging of the Adult Neural Stem-Cell
Niche
Collaboration Sally Temple (AMC)
24
Segmentation

• Goal
• Label each pixel/voxel as belonging to a specific
  biological object, or part thereof
• e.g., surface of cell nucleus 30
• Comments
• Establishes a higher level of abstraction for
  further image analysis
• The hardest step
• Tries to mimic our visual system
• Depends on the nature of the object(s) in the
  image
• Morphology, appearance, expected distortions
• Model based image segmentation algorithms
  generally the most effective

25
Segmentation methods

• Manual / computer assisted
• Use the pattern recognition abilities of the
  human visual system
• Still unbeatable!
• Use a computer to record data
• Great for small-scale image analysis
• Tedious, costly, and impractical for large-scale
• Subjectivity is a problem
• Multi-observer analysis can help
• Limited by hand unsteadiness attention span
• Automated systems much better
• Limited 3D capability
• Stereo viewing is the best you can do

26
Tricks to make manual analysis practical

• Randomly subsample the image extrapolate to the
  full data
• unbiased stereology
• Defines methodical ways to minimize bias
• Big user community
• Software packages available
• Drawbacks
• Variance can be high
• Assumes tissue is homogeneous
• Advocates extraction of small numbers of
  measurements from large numbers of animals
• Cannot handle multi-dimensional data
• Bottom line
• Manual analysis is good for small-scale studies
• Automated methods have much more to offer
• They can also be used in conjunction with
  stereology

27
How we treat the objects we segment

• Compartments
• Usually defined by cell/tissue structures
• A compartment is a region of space occupied by
  the structure of interest
• One compartment can be included in another
• Surfaces
• Think of them as membranes
• Usually separate two or more compartments
• Functional Signals
• Mobile molecules of interest
• Do not define a region of space
• Usually indicate an activity of interest,
  either directly or indirectly
• Biologist must specify how each channel must be
  interpreted, and what it contains
• Compartment / surface / functional signal
• What kind of shape

28
Divide-and-Conquer Strategy
Blob Segmentation
Tube Segmentation
Compartments
Compute Associations
Shell Segmentation
Output
Un-mixing
Microscopy Data
Plate Segmentation
Surfaces
Man-made Objects
Signals
Foci Segmentation
Cloud Segmentation
Pure Channels (common case)
Morphological Unmixing
Mixed Channels (rare case)
29
Basic Types of Measurements

• Intrinsic Measurements
• These measurements are specific to each type of
  thing
• Associative measurements
• These measurements are based on associations
  between one or more of these things

30
Channel 1 pure Fluorescent label DAPI Molecule
of interest Nuclear DNA Type of thing
compartment Compartment Morphology Blobs
DAPI
Blobs
31
Intrinsic Blob Measurements

• Measures of Location
• x, y, and z of centroid
• Measures of size
• Volume, diameter
• Measures of shape
• Eccentricity, shape factor, irregularity
• Measures of appearance
• Average brightness, texture

32
Channel 4 pure Fluorescent label Alexa
546 Molecule of interest Lewis-X a carbohydrate
found in the extra-cellular matrix surrounding
neural stem cells A functional signal Type of
thing Irregular Cloud
Alexa 546
Clouds
33
Intrinsic Cloud Measurements

• Measurements of signal strength
• Brightness
• Intensity per unit volume
• Measurements of variation and organization
• Brightness variance
• Texture and flow
• Measures of size
• Volume, diameter
• Measures of shape
• Spatial compactness
• Stellateness

34
Channel 2 Morphologically spectrally impure!
Fluorescent label Alexa 647 (Cy5) Molecule of
interest laminin that forms the basal lamina
of blood vessels. Compartment type Hollow
Tube Comment Laminin is also part of another
structure (bulbs)
Tubes
Alexa 647 (Cy5)
Shadows of Nuclei
Blobs
35
Bulbs
Traces
Rejected
36
Channel 3 Morphologically impure! Fluorescent
label Alexa 488 Molecule of interest GFAP
(glial fibrillary acidic protein) Compartment
type Tube Comment GFAP also found in soma of
cells
Soma
Alexa 488
Tubes
37
Traces
38
Intrinsic Tube Measurements

• Measurements of Location
• (x, y, z) locations of centerlines
• Locations of branch points
• Metric Measurements
• Thickness (diameter) at each location
• Curvature
• Tortuosity (curvature change per unit length)
• Topological Measurements
• Branching frequency as a function of branching
  order

39
Associative Measurements
Association (1, 2) ltList of Associative
Measurements gt
Object 1 ltList of intrinsic featuresgt
Object 2 ltList of intrinsic featuresgt

• Quantify relationships between segmented objects
• Numerous associations can be imagined
• Even the simplest of these are immediately useful
• Examples
• Proximity, orientation, connectivity
• Adjacency and Neighborhood relationships
• Marker-based object classification
• e.g., Tracking change analysis
• Can be summarized in familiar ways
• e.g., Time course / dose response / Spatial
  Variations
• Conditional histograms and distributions

40
Associative Measurements are a General Concept

• For a fixed point in time
• Establish associations between compartments,
  surfaces, and functional signals
• Each association leads to a measurement of signal
  localization, structural relationships, etc.
• Across points in time
• We first need to track compartments, surfaces,
  and signals over time
• Measuring changes for tracked objects over time
  yield measurements of dynamic phenomena such as
  morphological dynamics, molecular transport,
  signaling, cell movement,

Time
41
Associative Measurements for Blobs

• Signal Associations
• Measure the amount of signal in another channel
  relative to each blob
• Within the blob volume
• On the surface
• Within a defined distance around the blob
• Spatial Associations
• Measure the location of the blob relative to
  other things
• Other blobs, tubes, plates, man-made
  structures,etc.
• Spatio-temporal Associations
• Track the movements of blobs over time

42
Example
43
Cell-Cell Adjacency Features

• Computed by associating cytoplasmic labels with
  nearest nucleus, and segmenting the space
  associated with each nucleus
• Number of neighbors that are in contact
• Contact areas between neighbors
• Number of membrane/other barriers and their
  signal strength
• Spatial organization patterns

44
Tube-Cell Associative Features

• Computed by associating nuclei and associated
  structures with nearest tube
• Distances to nearest tube
• Center distance or surface distance
• Analysis by branching order of tube

45
Associating Nuclei Vessels

• Draw a perpendicular line to nearest vessel
  segment for each segmented nucleus, and measure
  its length.

46
Cell Network analysis

• Computed by associating nuclei with nearest other
  nuclei
• Distances to nearest neighbor
• Integrate secondary label(s) along the path from
  one nucleus to the other
• Yields a labeled graph structure
• Nodes cells
• Arcs relationships

47
Putting it all Together
48
Sample Table of Measurements
3D Location
Distance to Vessel
Eccentricity
Nuclear ID
Convexity
Intensity
Gradient
Texture
Laminin
Lewis-X
Volume
Shape
GFAP
Collaboration Sally Temple (AMC)
49
Example Plot
50
Summary

• Basic Steps and Image Analysis Terminology
• Divide Conquer Segmentation
• Fluorescence imaging simplifies image analysis by
  enabling a divide and conquer strategy
• Pure channel One channel, one morphology
• Basic Types of Measurements
• Intrinsic one set for each channel
• Associative relates objects across channels
• Next Class
• Well follow the divide and conquer road map from
  here on, starting with
• Blob segmentation methods

51
Homework - Part I

• 1. Using the formulas described today, calculate
  the best-achievable lateral (x - y) resolution
  and the axial (z) resolution of a microscope with
  a numerical aperture of 0.9, at a wavelength of
  550nm, and a water immersion medium
• The axial direction is along the optical axis,
  and the lateral direction is perpendicular to it.
• Along which direction does the microscope have
  poorer resolution and by how much?
• Describe at least 3 potential methods to improve
  the axial resolution, and discuss their
  limitations
• 2. The Chameleon Titanium-sapphire laser from
  Coherent Inc. (http//www.coherent.com/Downloads/A
  CF12E4.pdf) is widely used for multi-photon
  imaging.
• If this laser is delivering 1 watt of energy at
  a pulse repetition rate of 90 MHz, how many
  joules of energy does it deliver in each pulse?
  (1 watt 1 Joule/sec, 1MHz 1 million/sec)
• If the duration of a pulse is 140 femtoseconds
  (1 femtosecond 10-15 secs), what is the wattage
  during the pulse?
• If the above excitation is concentrated on a
  cube of side 0.2??m in the specimen (1 ?m
  10-6m), calculate the number of watts per cubic
  micrometer that we are applying.

52
Homework Part II

• 3. Download the image AVO20429_03ah that was
  obtained with a Zeiss LSM META microscope, from
  the course web page
• This is a compressed zip file, so you need to
  unpack it first
• Explore the different ImageJ viewing options, and
  options for data import/export on this software
• Make sure you can see an x-y, x-z, and y-z cut
  through the data
• Make sure that you can visualize each channel by
  itself
• 4. Answer the following questions
• What the size of the image in the x, y, and z
  dimensions?
• How many channels does it contain?
• What can you say about the shapes of objects in
  each channel?
• See any diversity of shapes within a channel?

53
Instructor Contact Information

• Badri Roysam
• Professor of Electrical, Computer, Systems
  Engineering
• Office JEC 7010
• Rensselaer Polytechnic Institute
• 110, 8th Street, Troy, New York 12180
• Phone [phone]
• Fax [phone]
• Email roysam_at_ecse.rpi.edu
• Website http//www.ecse.rpi.edu/roysam
• Course website http//www.ecse.rpi.edu/roysam/CT
  IA
• Secretary Laraine Michaelides, JEC 7012, (518)
  276 8525, michal_at_.rpi.edu
• Grader

Center for Sub-Surface Imaging Sensing