Automatic in vivo Microscopy Video Mining for Leukocytes

1
Automatic in vivo Microscopy Video Mining for
Leukocytes
Chengcui Zhang, Wei-Bang Chen, Lin Yang, Xin
Chen, John K. Johnstone
2
Background Information

• What is in vivo microscopy?
• Images of the cellular and molecular processes in
  a living organism
• Why video-mine leukocytes?
• To Predict Inflammatory response
• Rolling velocity and magnitude of adhesion of
  leukocytes are the main predictors
• Currently analyzed manually
• Time consuming / Expensive
• Subjective

3
Objectives

• Given a sequence of in vivo images,
• Track the moving leukocytes
• Calculate their average velocity
• Find the magnitude of adherent leukocytes

4
Challenges

• Server Noise
• Background movement
• Due to movement of the living organism
• Deformation of leukocytes
• Change of contrast in different frames

5
Previous Work

• Eden et al. use local features (e.g. color) for
  a tracking system
• Assume that leukocytes roll along the vessel
  centerline
• Acton et al. Background removal
  morphological filter
• Assumes the shape/size leukocytes does not change

6
Suggested Approach

• Three main steps
• Frame Alignment
• To correct the camera/subject movement
• Detect Moving Leukocytes
• Detect Adherent Leukocytes
• After moving leukocytes are removed

7
Step 1- Frame Alignment

• 1.1- Detect Camera/Subject Movement
• Define a (dis)similarity measure between
  consecutive frames

• This allows for some tolerance within radius r
• If S(ft-1, ft) is larger than a threshold, then
  ft requires frame alignment

8
Step 1- Frame Alignment

• 1.2- Frame Matching
• Generate a number of high dimensional, local
  scale-invariant features SIFT for the frame and
  its predecessor
• Use nearest-neighbor to find a match for each
  feature point
• Calculate the transformation matrix H, such that

• For every matched point x and x

9
Step 1- Frame Alignment

• Use Random Sample Consensus (RANSAC) to correct
  the mismatches

10
Step 2 - Detecting Moving Leukocytes

• Approach 1 - Probabilistic Learning
• For pixel j in the image, let x1j, x2j, ..., xNj
  be the intensity of the pixel over N frames
• Assume that P(xtj) has a normal distribution over
  time with mean xtj

• If P(xtj) is smaller than a threshold, then it is
  a foreground pixel
• Problem Difficult to find a threshold

11
Step 2 - Detecting Moving Leukocytes

• Approach 1 - Probabilistic Learning
• Problem Difficult to find the threshold value
• Solution Use One-Class SVM to classify
  background and foreground pixels

12
Step 2 - Detecting Moving Leukocytes

• Approach 2 - Neural Network
• Train a neural net to learn the predictable
  pattern of the background pixels
• Input x(t-m), x(t-m1),... , x(t-1)
• A sliding window of the intensity sequence
• Output x(t)
• Prediction for the intensity of the pixel at the
  next frame
• If the neural-net prediction and the real pixel
  intensity are very different, the pixel in the
  current frame is in foreground

13
Step 2 - Detecting Moving Leukocytes

• Approach 2 - Neural Network

14
Step 2 - Detecting Moving Leukocytes

• Calculating the leukocytes velocity
• Find the centroid of each group of connected
  foreground pixels
• For each centroid, find the closest centroid in
  the previous frame
• If their distance is smaller than a threshold,
  they are a match
• Compute the mean velocity

15
Step 3- Detecting Adherent Leukocytes

• First, remove the moving leukocytes
• Three main types of regions left
• Tissues
• Vessels
• Adherent Leukocytes
• These three have different intensity values

16
Step 3- Detecting Adherent Leukocytes
17
Step 3- Detecting Adherent Leukocytes

• Finding the threshold values
• Fit an 8th degree polynomial to the histogram
  curve
• The real part of the second largest root is the
  ideal threshold
• Justification?
• Problem with false positives and false negatives

18
Experimental Results

• Test video of 148 frames
• Detecting moving leukocytes
• 1 false positive for probabilistic learning(?)
• 49 false positive for neural-net approach
• 50 recall
• Detecting Adherent leukocytes
• 2 false positive
• 95 recall

19
Final Remarks

• Paper is mainly related to Vision
• The algorithms require many magic parameters
  that need hand tuning
• Would the current parameters work as well for a
  new video sequence from a new equipment?
• Do we want to pursue more video-mining papers?