Introduction%20to%20Image%20Processing%20and%20Computer%20Vision

1
Introduction toImage Processing andComputer
Vision
Rahul Sukthankar Intel Research Laboratory at
Pittsburgh and The Robotics Institute, Carnegie
Mellon rahuls_at_cs.cmu.edu
2
Image Processing vs. Computer Vision

• Image processing
• Image ? image
• e.g., de-noising, compression, edge detection
• Computer vision
• Image ? symbols
• e.g., face recognition, object tracking
• Most real-world applications combine techniques
  from both categories

3
Outline

• Operations on a single image
• Operations on an image sequence
• Multiple cameras
• Extracting semantics from images
• Applications

4
Outline

• Operations on a single image
• Operations on an image sequence
• Multiple cameras
• Extracting semantics from images
• Applications

5
What is an Image?

• 2D array of pixels
• Binary image (bitmap)
• Pixels are bits
• Grayscale image
• Pixels are scalars
• Typically 8 bits (0..255)
• Color images
• Pixels are vectors
• Order can vary RGB, BGR
• Sometimes includes Alpha

6
What is an Image?

• 2D array of pixels
• Binary image (bitmap)
• Pixels are bits
• Grayscale image
• Pixels are scalars
• Typically 8 bits (0..255)
• Color images
• Pixels are vectors
• Order can vary RGB, BGR
• Sometimes includes Alpha

7
What is an Image?

• 2D array of pixels
• Binary image (bitmap)
• Pixels are bits
• Grayscale image
• Pixels are scalars
• Typically 8 bits (0..255)
• Color images
• Pixels are vectors
• Order can vary RGB, BGR
• Sometimes includes Alpha

8
What is an Image?

• 2D array of pixels
• Binary image (bitmap)
• Pixels are bits
• Grayscale image
• Pixels are scalars
• Typically 8 bits (0..255)
• Color images
• Pixels are vectors
• Order can vary RGB, BGR
• Sometimes includes Alpha

9
What is an Image?

• 2D array of pixels
• Binary image (bitmap)
• Pixels are bits
• Grayscale image
• Pixels are scalars
• Typically 8 bits (0..255)
• Color images
• Pixels are vectors
• Order can vary RGB, BGR
• Sometimes includes Alpha

10
Canny Edge Detector
cvCanny()
Images courtesy of OpenCV tutorial at CVPR-2001
11
Morphological Operations

• Simple morphological operations on binary images
• erosion any pixel with 0 neighbor becomes 0
• dilation any pixel with 1 neighbor becomes 1
• Compound morphological operations(composed of
  sequences of simple morphological ops)
• opening
• closing
• morphological gradient
• top hat
• black hat
• Aside what is the right definition of
  neighbor?

12
Morphological Operations
Dilatation I?B
Opening IoB (I?B)?B
Erosion I?B
Image I
Closing IB (I?B)?B
TopHat(I) I - (I?B)
BlackHat(I) (I?B)-I
Grad(I) (I?B)-(I?B)
Images courtesy of OpenCV tutorial at CVPR-2001
13
Hough Transform
Goal Finding straight lines in an edge image
Canny edge Hough xform cvHoughLines()
Original image
Images courtesy of OpenCV tutorial at CVPR-2001
14
Distance Transform

• Distance for all non-feature points to closest
  feature point

cvDistTransform()
Images courtesy of OpenCV tutorial at CVPR-2001
15
Flood Filling
cvFloodFill() grows from given seed point
Images courtesy of OpenCV tutorial at CVPR-2001
16
Image Statistics

• Statistics are used to summarize the pixel values
  in a region, typically before making a decision
• Some statistics are computed over a single image
• Mean and standard deviation cvAvg(),
  cvAvgSdv()
• Smallest and largest intensities cvMinMaxLoc()
• Moments cvGetSpatialMoment(),
  cvGetCentralMoment()
• Others are computed over pairs/differences of
  images
• Distances/norms C, L1, L2 cvNorm(),
  cvNormMask()
• Others are computed over pairs/differences of
  images
• Histograms
• Multidimensional histograms (many functions to
  create/manipulate)
• Earth mover distance compare histograms
  cvCalcEMD()

17
Image PyramidsCoarse to Fine Processing

• Gaussian and Laplacian pyramids
• Image segmentation by pyramids

Images courtesy of OpenCV tutorial at CVPR-2001
18
Image PyramidsCoarse to Fine Processing

• Original image Gaussian Laplacian

Images courtesy of OpenCV tutorial at CVPR-2001
19
Pyramid-based Color Segmentation
Images courtesy of OpenCV tutorial at CVPR-2001
20
Outline

• Operations on a single image
• Operations on an image sequence
• Multiple cameras
• Extracting semantics from images
• Applications

21
Background Subtraction

• Useful when camera is still and background is
  static or slowly-changing (e.g., many
  surveillance tasks)
• Basic idea subtract current image from reference
  image. Regions with large differences correspond
  to changes.
• OpenCV supports several variants of image
  differencing
• Average
• Standard deviation
• Running average cvRunningAvg()
• Can follow up with connected components
  (segmentation)
• could use union find or floodfill
  cvFloodFill()

22
Optical Flow

• Goal recover apparent motion vectors between a
  pair of images -- usually in a video stream
• Several optical flow algorithms are available
• Block matching technique cvCalcOpticalFlowBM()
• Horn Schunck technique cvCalcOpticalFlowHS()
• Lucas Kanade technique cvCalcOpticalFlowLK()
• Pyramidal LK algorithm cvCalcOpticalFlowPyrLK()

23
Active ContoursTracking by Energy Minimization

• Snake energy
• Internal energy
• External energy

cvSnakeImage()
Images courtesy of OpenCV tutorial at CVPR-2001
24
Camera Calibration

• Real cameras exhibit radial tangential
  distortion causes problems for some algorithms.
• First, calibrate by showing a checkerboard at
  various orientationscvFindChessBoardCornerGuesse
  s()
• Then apply an undistorting warp to each image
  (dont use a warped checkerboard!)cvUndistort()
• If the calibration is poor, the undistorted
  image may be worse than the original.

Images courtesy of OpenCV tutorial at CVPR-2001
25
Outline

• Operations on a single image
• Operations on an image sequence
• Multiple cameras
• Extracting semantics from images
• Applications

26
Stereo Vision

• Extract 3D geometry from multiple views
• Points to consider
• feature- vs area-based
• strong/weak calibration
• processing constraints
• No direct support in OpenCV, but building blocks
  for stereo are there.

27
View Morphing
Images courtesy of OpenCV tutorial at CVPR-2001
28
Outline

• Operations on a single image
• Operations on an image sequence
• Multiple cameras
• Extracting semantics from images
• Applications

29
Face Detection
Images courtesy of Mike Jones Paul Viola
30
Classical Face Detection
Images courtesy of Mike Jones Paul Viola
31
Viola/Jones Face Detector

• Technical advantages
• Uses lots of very simple box features, enabling
  an efficient image representation
• Scales features rather than source image
• Cascaded classifier is very fast on non-faces
• Practical benefits
• Very fast, compact footprint
• You dont have to implement it!(should be in
  latest version of OpenCV)

32
Principal Components Analysis
High-dimensional data
Lower-dimensional subspace
cvCalcEigenObjects()
Images courtesy of OpenCV tutorial at CVPR-2001
33
PCA for Object Recognition
Images courtesy of OpenCV tutorial at CVPR-2001
34
PCA for Object Recognition
Images courtesy of OpenCV tutorial at CVPR-2001
35
Outline

• Operations on a single image
• Operations on an image sequence
• Multiple cameras
• Extracting semantics from images
• Applications

36
Examples of Simple Vision Systems

• Shadow Elimination
• Idea remove shadows from projected displays
  using multiple projectors
• OpenCV Techniques
• Image differencing
• Image warping
• Convolution filters
• Matrix manipulation

• PosterCam
• Idea put cameras in posters and identify who
  reads which poster
• OpenCV Techniques
• Face detection
• Face recognition
• Unsupervised clustering

37
Single Projector Severe Shadows
display screen
P
38
Two Projectors Shadows Muted
display screen
P-2
P-1
39
Dynamic Shadow Elimination
display screen
camera
P-2
P-1
40
Shadow Elimination Challenges

• Occlusion detection what does a shadow look
  like?
• Geometric issues which projectors are occluded?
• Photometric issues how much light removes a
  shadow?
• Performance how can we do this in near real-time?

41
Shadow Elimination Solutions

• Occlusion detection difference image analysis
• Geometric issues single shadow-mask for all
  projectors!
• Photometric issues uncalibrated feedback
  system
• Performance only modify texture map alpha values

42
Shadow Removal witha Single Mask
43
Shadow Elimination Algorithm
Camera images
Projected
44
PosterCam Overview

• PosterCam Hardware
• Camera in each poster
• Embedded computer in each poster ( iPAQ)
• Network connection to other posters

45
PosterCam Details

• Face detection Viola/Jones (no float ops)
• Lighting compensationhistogram equalization
• Pose variationadditional synthetic faces
• Unsupervised clusteringk-means and nearest
  neighbor with non-standard distance metric

46
Tips on Image Processing and Coding with OpenCV

• Use the OpenCV documentation only as a guide(it
  is inconsistent with the code)
• Read cv.h before writing any code
• OpenCV matrix functions work on images e.g.,
  cvSub()
• Beware camera distortion cvUnDistort() may help
• Beware illumination changes
• disable auto gain control (AGC) in your camera if
  you are doing background subtraction
• histogram equalization (often good for object
  recognition)
• Image processing algorithms may require parameter
  tuning collect data and tweak until you get good
  results

47
Reference Reading

• Digital Image ProcessingGonzalez
  Woods,Addison-Wesley 2002
• Computer VisionShapiro Stockman,Prentice-Hall
  2001
• Computer Vision A Modern ApproachForsyth
  Ponce,Prentice-Hall 2002
• Introductory Techniques for 3D Computer
  VisionTrucco Verri,Prentice-Hall 1998

48
The End

• Acknowledgments
• Significant portions of this lecture were derived
  from the Intel OpenCV tutorial by Gary Bradski et
  al. at CVPR-2001
• Thanks to my former colleagues at Compaq/HP CRL
  for additional slides and suggestions Tat-Jen
  Cham, Mike Jones, Vladimir Pavlovic, Jim Rehg,
  Gita Sukthankar, Nuno Vasconcelos, Paul Viola
• Contact rahuls_at_cs.cmu.edu if you need more
  information