Spatially Constrained Segmentation of Dermoscopy Images

1
Spatially Constrained Segmentation of Dermoscopy
Images

• Howard Zhou1, Mei Chen2, Le Zou2, Richard Gass2,
• Laura Ferris3, Laura Drogowski3, James M. Rehg1

1School of Interactive Computing, Georgia
Tech 2Intel Research Pittsburgh 3Department of
Dermatology, University of Pittsburgh
2
Skin cancer and melanoma

• Skin cancer most common of all cancers

Image courtesy of An Atlas of Surface
Microscopy of Pigmented Skin Lesions Dermoscopy

3
Skin cancer and melanoma

• Skin cancer most common of all cancers
• Melanoma leading cause of mortality (75)

Image courtesy of An Atlas of Surface
Microscopy of Pigmented Skin Lesions Dermoscopy

4
Skin cancer and melanoma

• Skin cancer most common of all cancers
• Melanoma leading cause of mortality (75)
• Early detection significantly reduces mortality

Image courtesy of An Atlas of Surface
Microscopy of Pigmented Skin Lesions Dermoscopy

5
Clinical View
Dermoscopy view
Image courtesy of An Atlas of Surface
Microscopy of Pigmented Skin Lesions Dermoscopy

6
Dermoscopy

• Improve diagnostic accuracy by 30 in the hands
  of trained physicians
• May require as much as 5 year experience to have
  the necessary training
• Motivation for Computer-aided diagnosis (CAD) in
  this area

7
First step of analysisSegmentation

• Separating lesions from surrounding skin
• Resulting border
• Gives lesion size and border irregularity
• Crucial to the extraction of dermoscopic features
  for diagnosis
• Previous Work
• PDE approach Erkol et al. 2005,
• Histogram thresholding Hintz-Madsen et al.
  2001,
• Clustering Schmid 1999, Melli et al. 2006
• Statistical region merging Celebi et al. 2007,
  

8
Domain specific constraints

• Spatial constraints
• Four corners are skin (Melli et al.2006, Celebi
  et al. 2007)
• Implicitly enforcing Local neighborhood
  constraints on image Cartesian coordinates
  (Meanshift)

9
Domain specific constraints

• Spatial constraints
• Four corners are skin (Melli et al.2006, Celebi
  et al. 2007)
• Implicitly enforcing Local neighborhood
  constraints on image Cartesian coordinates
  (Meanshift)

10
We explore

• Spatial constraints arise from the growth pattern
  of pigmented skin lesions

11
We explore

• Spatial constraints arise from the growth pattern
  of pigmented skin lesions radiating pattern

12
Embedding constraints

• Radiating pattern from lesion growth
• Embedding constraints as polar coords improves
  segmentation performance

Polar (k 6)
13
Embedding constraints

• Radiating pattern from lesion growth
• Embedding constraints as polar coords improves
  segmentation performance

Polar (k 6)
Meanshift
Polar
14
Comparison to the Doctors

• Radiating pattern from lesion growth
• Embedding constraints as polar coords improves
  segmentation performance

Meanshift
Polar
15
Dermoscopy images Common radiating appearance
16
Growth pattern of pigmented skin lesions

• lesions grow in both radial and vertical
  direction
• Skin absorbs and scatters light.
• Appearance of pigmented cells varies with depth
• Dark brown ? tan ? blue-gray
• Common radiating appearance pattern on skin
  surface

Image courtesy of Dermoscopy An Atlas of
Surface Microscopy of Pigmented Skin Lesions
17
Radiating growth pattern on skin surface

• Difference in appearance more significant along
  the radial direction than any other direction.

18
Radiating growth pattern on skin surface

• Difference in appearance more significant along
  the radial direction than any other direction.

19
Embedding spatial constraintsFeature vectors

• Each pixel ? feature vector in R4
• 3D R,G,B or L, a, b in the color space
• 1D polar radius measured from the center of the
  image (normalized by w)

20
Embedding spatial constraintsGrouping features

• Each pixel ? feature vector in R4
• Clustering pixels in the feature space
• Replace pixels with mean for compact
  representation

21
Radiating pattern Dermoscopy vs. natural images

• Derm dataset (216)

BSD dataset (300)
22
Embedding spatial constraintsGrouping features

• Mean per-pixel residue average per-pixel color
  difference of each pair

23
Dermoscopy vs. natural images Polar vs. Cartesion

• Mean per-pixel residue (k-means, k 30)

24
Dermoscopy vs. natural images Polar vs. Cartesion

• Mean per-pixel residue (k-means, k 30)

25
Polar vs. Cartesian

• The regions appear more blocky in the Cartesian
  case

Polar (k 30)
Cartesian (k 30)
26
Six super-regions

• 30 clusters ? 6 super clusters (K-means)

Polar (k 6)
Cartesian (k 6)
27
Final segmentation
Polar
Cartesian
28
Polar vs. Meanshift

• The regions appear more blocky in the Meanshift
  case

Polar (k 6)
Meanshift (c 32, s 8)
29
Final segmentation
Polar
Meanshift
30
Algorithm overview

• Given a dermoscopy image

31
Algorithm overview

• Given a dermoscopy image

32
Algorithm overview

• 1. First round clustering K-means (k 30)

33
Algorithm overview

• 2. Second round clusters(30)? super-regions(6)

34
Algorithm overview

• 3. Apply texture gradient filter (Martin, et al.
  2004)

35
Algorithm overview

• 4. Find optimal boundary (colortexture)

36
1. First round clustering

• First round clustering K-means (k 30)
• Reduce noise
• Groups pixels into homogenous regions a more
  compact representation of the image
• Artuhur and Vassilvitskii, 2007
• R4 Lab (3D), w polar radius (1D)

37
1. First round clustering

• First round clustering K-means (k 30)
• Reduce noise
• Groups pixels into homogenous regions a more
  compact representation of the image
• Artuhur and Vassilvitskii, 2007
• R4 Lab (3D), w polar radius (1D)

38
2. Second round clustering

• K 6 clusters(30)? super-regions(6)
• Account for intra-skin and intra-lesion
  variations
• Avoid a large k
• Super-regions correspond to meaningful regions
  such as skin, skin-lesion transition, and inner
  lesion, etc.

39
2. Second round clustering

• K 6 clusters(30)? super-regions(6)
• Account for intra-skin and intra-lesion
  variations
• Avoid a large k
• Super-regions correspond to meaningful regions
  such as skin, skin-lesion transition, and inner
  lesion, etc.

40
3. Color-texture integration

• Incorporating texture information can improve
  segmentation performance.
• Severely sun damaged skin texture variations at
  boundaries in addition to color variations

41
3. Color-texture integration

• Incorporating texture information can improve
  segmentation performance.
• Severely sun damaged skin texture variations at
  boundaries in addition to color variations
• Apply texture gradient filter (Martin, et al.
  2004)

42
3. Color-texture integration

• Incorporating texture information can improve
  segmentation performance.
• Severely sun damaged skin texture variations at
  boundaries in addition to color variations
• Apply texture gradient filter (Martin, et al.
  2004)
• Texture edge map pseudo-likelihood

43
4. Optimal boundary

• Optimal skin-lesion boundary
• Color Earth Movers Distance (EMD) between every
  pair of super-regions

44
4. Optimal boundary

• Optimal skin-lesion boundary
• Color Earth Movers Distance (EMD) between every
  pair of super-regions
• Texture Texture edge map

45
4. Optimal boundary

• Optimal skin-lesion boundary
• Color Earth Movers Distance (EMD) between every
  pair of super-regions
• Texture Texture edge map
• Minimizing the integrated color-texture measure

46
Validation and results

• Our collaborating dermatologist Dr. Ferris
  manually outline the lesions in 67 dermoscopy
  images
• The border error is given by
• Computer binary image obtained by filling the
  automatic detected border
• ground-truth obtained by filling in the
  boundaries outlined by Dr. Ferris

47
Typical segmentation result
48
Comparison
To account for inter-operator variation, we also
asked Dr. Alex Zhang to manually outline
boundaries on the same dataset
49
Additional results
Error 5.80
50
Additional results
Error 13.61
51
Additional results
Error 16.60
52
Additional results
Error 34.09
53
Limitation

• Assumption that lesions appear relatively near
  the center may not hold
• Fairly low number of super regions (6) may limit
  the algorithm to perform well on lesions with
  more colors

54
Conclusion

• Growth pattern of pigmented skin lesions can be
  used to improve lesion segmentation accuracy in
  dermoscopy images.
• An unsupervised segmentation algorithm
  incorporating these spatial constraints
• We demonstrate its efficacy by comparing the
  segmentation results to ground-truth
  segmentations determined by an expert.

55
Future work

• Extend to meanshift?

56
Comparison to other methods
57
Color and texture cue integration

• Apply texture gradient filter (Martin, et al.
  2004)
• Pseudo-likelihood map - edge caused by texture
  variation is present at a certain location