Featurepreserving Artifact Removal from Dermoscopy Images

1
Feature-preserving Artifact Removal from
Dermoscopy Images

• Howard Zhou1, Mei Chen2,
• Richard Gass2, James M. Rehg1,
• Laura Ferris3, Jonhan Ho3, Laura Drogowski3

1School of Interactive Computing, Georgia
Tech 2Intel Research Pittsburgh 3Department of
Dermatology, University of Pittsburgh
2
Skin cancer and melanoma

• Skin cancer most common of all cancers

3
Skin cancer and melanoma

• Skin cancer most common of all cancers

Hemangioma
Basal Cell Carcinoma
Compound nevus
Seborrheic keratosis
Image courtesy of An Atlas of Surface
Microscopy of Pigmented Skin Lesions Dermoscopy

4
Skin cancer and melanoma

• Skin cancer most common of all cancers
• Melanoma leading cause of mortality

Hemangioma
Melanoma
Basal Cell Carcinoma
Compound nevus
Seborrheic keratosis
Melanoma
Image courtesy of An Atlas of Surface
Microscopy of Pigmented Skin Lesions Dermoscopy

5
Skin cancer and melanoma

• Skin cancer most common of all cancers
• Melanoma leading cause of mortality
• Early detection significantly reduces mortality

Hemangioma
Melanoma
Basal Cell Carcinoma
Compound nevus
Seborrheic keratosis
Melanoma
Image courtesy of An Atlas of Surface
Microscopy of Pigmented Skin Lesions Dermoscopy

6
Clinical View
Dermoscopy view
Image courtesy of An Atlas of Surface
Microscopy of Pigmented Skin Lesions Dermoscopy

7
Dermoscopy

• Skin surface microscopy
• Improve diagnostic accuracy by 30 for trained,
  experienced physicians
• Requires 5 or more years of experience
• Computer-aided diagnosis (CAD) to assist less
  experienced physicians

8
Artifacts in dermoscopy images

• Hair, air-bubbles,
• Interfering with computer-aided diagnosis

Image courtesy of Grana et al. 2006
9
Artifacts in dermoscopy images

• Hair, air-bubbles,
• Interfering with computer-aided diagnosis

Image courtesy of Grana et al. 2006
10
Artifacts in dermoscopy images

• Hair, air-bubbles,
• Interfering with computer-aided diagnosis

Hair ? lesion boundary
Image courtesy of Grana et al. 2006
11
Artifacts in dermoscopy images

• Hair, air-bubbles,
• Interfering with computer-aided diagnosis

Hair ? lesion boundary
Image courtesy of Grana et al. 2006
12
Artifacts in dermoscopy images

• Hair, air-bubbles,
• Interfering with computer-aided diagnosis

Hair ? lesion boundary
Hair ? pigmented network
Image courtesy of Grana et al. 2006
13
Previous work

• Hair detection and tracing
• Fleming et al. 1998
• Thresholding and averaging
• DullRazor, Tim K. Lee et al. 1997
• Schmid et al. 2003
• Thresholding and inpainting
• Paul Wighton et al. 2008 (right here in the
  conference)

14
Schmid et al.

• Detection thresholding
• Removal morphological operations

15
Schmid et al.

• Thresholding ? false detection
• Accidental removal of diagnostic features

Thresholding
Schmid et al. 2003
16
Schmid et al.

• Morphological operation (neighbors average)
  ?blurring

Morphological operation
Schmid et al. 2003
17
Feature-preserving artifact removal (FAR)

• Detection Explicit curve modeling
• Removal Exemplar-based inpainting

Our method (FAR)
Schmid et al. 2003
18
FAR

• Curve modeling ? more accurate hair detection

Thresholding
Curve modeling
Our method (FAR)
Schmid et al. 2003
19
FAR

• Exemplar-based inpainting ? preserving features

Curve modeling
Morphological operation
Exemplar-based inpainting
Thresholding
Our method (FAR)
Schmid et al. 2003
20
FAR

• Exemplar-based inpainting ? preserving features

Curve modeling
Morphological operation
Exemplar-based inpainting
Thresholding
Our method (FAR)
Schmid et al. 2003
21
FAR

• Exemplar-based inpainting ? preserving features

Our method (FAR)
Schmid et al. 2003
22
FAR

• Exemplar-based inpainting ? preserving features

Our method (FAR)
Schmid et al. 2003
23
FAR

• Exemplar-based inpainting ? preserving features

Our method (FAR)
Schmid et al. 2003
24
System overview
25
Input dermoscopy image
26
Enhancing dark-thin structure

• Luminosity channel in CIE Luv
• Difference b/a morphological closing

Schmid-Saugeona et al. 2003, Towards a
computer-aided diagnosis system for pigmented
skin lesions
27
Detecting line points
Curve B(t)
Steger 1998, An Unbiased Detector of
Curvilinear Structures
28
Detecting line points
Curve B(t)
Cross section
n(t)
f(x)
Steger 1998, An Unbiased Detector of
Curvilinear Structures
29
Detecting line points
Cross section
Curve B(t)
n(t)
f(x)
Steger 1998, An Unbiased Detector of
Curvilinear Structures
30
Detecting line points
Cross section
Curve B(t)
f 0
f large
n(t)
f(x)
Steger 1998, An Unbiased Detector of
Curvilinear Structures
31
Detecting line points
Cross section
Curve B(t)
f 0
f large
n(t)
f(x)
n(t) direction - curve B(t)
eigenvector corresponding to the maximum absolute
eigenvalue of the local Hessian
Steger 1998, An Unbiased Detector of
Curvilinear Structures
32
Detecting line points
n(t)
Steger 1998, An Unbiased Detector of
Curvilinear Structures
33
Detecting line points
Steger 1998, An Unbiased Detector of
Curvilinear Structures
34
Linking line points

• Link the neighboring points to get line segments
  (sets of ordered line points)

35
Fitting polynomial curves

• A set of ordered points Pi s

P
36
Fitting polynomial curves

• A set of ordered points Pi s
• Parametric curve

P
37
Fitting polynomial curves

• A set of ordered points Pi s
• Parametric curve

P
B(t)
38
Fitting polynomial curves

• A set of ordered points Pi s
• Parametric curve
• Minimize sum of squared distance

P
B(t)
39
Fitting polynomial curves

• A set of ordered points Pi s
• Parametric curve
• Minimize sum of squared distance
• Linear system (can be solved by Gaussian
  elimination)

P
B(t)
40
Handling hair intersection
Configurations

41
Before curve fitting and linking
Line segments
42
After curve fitting and linking
Parameterized curves
43
After curve fitting and linking
Parameterized curves
44
After curve fitting and linking
Hair mask
45
After curve fitting and linking
Hair mask
46
Exemplar-based inpainting

• Fill in with patches from the image itself
• Patch ordering ?structure propagation.

Image courtesy of Criminisi et al. 2003
Criminisi et al. 2003, Object removal by
exemplar-based inpainting
47
Exemplar-based inpainting

• Fill in with patches from the image itself
• Patch ordering ?structure propagation.

Criminisi et al. 2003, Object removal by
exemplar-based inpainting
48
Exemplar-based inpainting

• Fill in with patches from the image itself
• Patch ordering ?structure propagation.

Criminisi et al. 2003, Object removal by
exemplar-based inpainting
49
Exemplar-based inpainting

• Fill in with patches from the image itself
• Patch ordering ?structure propagation.

Criminisi et al. 2003, Object removal by
exemplar-based inpainting
50
Exemplar-based inpainting

• Fill in with patches from the image itself
• Patch ordering ?structure propagation.

Criminisi et al. 2003, Object removal by
exemplar-based inpainting
51
Exemplar-based inpainting

• Fill in with patches from the image itself
• Patch ordering ?structure propagation.

Criminisi et al. 2003, Object removal by
exemplar-based inpainting
52
Exemplar-based inpainting

• Fill in with patches from the image itself
• Patch ordering ?structure propagation.

Criminisi et al. 2003, Object removal by
exemplar-based inpainting
53
Exemplar-based inpainting

• Fill in with patches from the image itself
• Patch ordering ?structure propagation.

Criminisi et al. 2003, Object removal by
exemplar-based inpainting
54
Before FAR
55
After FAR
56
More results

• Explicit curve modeling
• Exemplar-based inpainting

Our method (FAR)
Schmid et al. 2003
57
More results

• Explicit curve modeling
• Exemplar-based inpainting

Our method (FAR)
Schmid et al. 2003
58
FAR

• Exemplar-based inpainting ? preserving features

Our method (FAR)
Schmid et al. 2003
59
When is FAR not suitable ?

• Oops, too much hair!

60
When is FAR not suitable ?

• Too much hair
• Makes explicit modeling difficult

Schemid et al. 2003 (DullRazor)
Our method (FAR)
61
Conclusion

• Automatic system that detects and removes
  curvilinear artifacts
• Feature-preserving artifact removal
• Explicit curve modeling
• Exemplar-based inpainting

62
Future work

• Speed up exemplar-based inpainting

63
Future work

• Speed up exemplar-based inpainting
• Handle hair with arbitrary intensity

64
Future work

• Speed up exemplar-based inpainting
• Handle hair with arbitrary intensity
• Extend to removing air bubbles

65
Questions ?
66
Additional results
Our method (FAR)
Original Dermoscopy image