Digital Camera and Computer Vision Laboratory

1
Computer and Robot Vision II

• Chapter 18
• Object Models And Matching

Presented by ??? ??? 0911 246
313 r94922093_at_ntu.edu.tw ???? ??? ??
2
18.1 Introduction

• object recognition one of most important aspects
  of computer vision

3
18.2 Two-Dimensional Object Representation

• 2D shape analysis useful in machine vision
  application
• medical image analysis
• aerial image analysis
• manufacturing

4
18.2 Two-Dimensional Object Representation

• 2D shape representation classes
• global features
• local features
• boundary description
• skeleton
• 2D parts

5
18.2.1 Global Feature Representation

• 2D object can be thought of as binary image
• value 1 pixels of object
• value 0 pixels outside object
• 2D shape features area, perimeter, moments,
  circularity, elongation

6
18.2.1 Global Feature Representation

• Shape Recognition by Moments
• f binary image function
• 2D shape
• digital th moment of S
• area of S number of pixels of S

7
18.2.1 Global Feature Representation

• moment invariants functions of moments invariant
  under shape transform
• prefer moment invariants under translation,
  rotation, scaling
• skewing center of gravity of S
  

8
18.2.1 Global Feature Representation

• central th moment of S
• central moments translation invariant
• normalized central moments of S

9
18.2.1 Global Feature Representation

• seven functions that are rotation invariant

10
18.2.1 Global Feature Representation

• Shape Recognition with Fourier Descriptors
• Fourier descriptors another way for extracting
  features from 2D shapes
• Fourier descriptors defined to characterize
  boundary
• The main idea is to represent the boundary as a
  function of one variable , expand in
  its Fourier series, and use the coefficients of
  the series as Fourier descriptors (FDs).
• finite number of FDs can be used to describe the
  shape

11
18.2.1 Global Feature Representation
12
18.2.1 Global Feature Representation
13
18.2.1 Global Feature Representation
14
18.2.2 Local Feature Representation

• 2D object characterized by local features,
  attributes, relationships
• most commonly used local features holes, corners
• holes found by connected component procedure
  followed by boundary tracing
• holes detected by binary mathematical
  morphology, if hole shapes known
• hole properties areas, shapes
• corner detection can be performed on binary or
  gray tone image
• corner property angle at which lines meet

15

• joke

16
18.2.3 Boundary Representation

• boundary representation most common
  representation for 2D objects
• 3 main ways to represent object boundary
• 1. sequence of points
• 2. chain code
• 3. sequence of line segments

17
18.2.3 Boundary Representation

• The Boundary as a Sequence of Points
• boundary points from border-following or
  edge-tracking algorithms
• interest points boundary points with special
  property useful in matching

18
18.2.3 Boundary Representation

• The Chain Code Representation
• chain encoding can be used at any level of
  quantization
• chain encoding saves space required for row and
  column coordinates
• boundary encoded first quantized by placing over
  square grid
• grid side length determines resolution of
  encoding
• marked points grid intersections closest to
  curve and used in encoding
• marks starting point of curve

19
18.2.3 Boundary Representation

• chain encoding of boundary curve

20
18.2.3 Boundary Representation

• line segments links to be used to approximate
  the curve
• encoding scheme eight possible directions
  assigned integer between 0, 7
• chain chain encoding in the form
• or

21
18.2.3 Boundary Representation

• length of chain code with n chains can be simply
  estimated as n
• number of odd chain codes
• number of even chain codes
• number of corners
• unbiased estimate of perimeter length
• Freeman suggested

22
18.2.3 Boundary Representation

• The Boundary as a Sequence of Line Segments
• line segment sequence after boundary segmented
  into near-linear portion
• line segment sequence used in shape
  recognition or other matching tasks
• coordinate location where pair of
  lines meet
• angle magnitude where pair of lines meet
• sequence of junction
  points to represent line segment sequence

23
18.2.3 Boundary Representation

• sequence of junction
  points representing test object T
• 
  an association
• goal given O, T, to find F satisfying i lt j
  F(i) lt F(j) or F(i) missing or F(j) missing

24
18.2.4 Skeleton Representation

• strokes long, sometimes thin parts forming
  shapes
• line segments that characterize the strokes of
  set of characters

25
18.2.4 Skeleton Representation

• symmetric axis transform set of maximal circular
  disks inside object
• symmetric axis locus of centers of these maximal
  disks
• symmetric axes of the characters

26
18.2.4 Skeleton Representation

• symmetric axis one example of skeleton
  description of 2D object
• symmetric axis of rectangle consists of five
  line segments not single line
• symmetric axis extremely sensitive to noise
• symmetric axis difficult to use in matching

27
(No Transcript)
28
18.2.4 Skeleton Representation

• axis of smoothed local symmetries separate
  definition for skeleton
• local symmetry midpoint P of line segment BA
  joining pair of points A, B
• angle between BA and outward normal at
  A
• angle between BA and inward normal at
  B

29
18.2.4 Skeleton Representation

• point P that is local symmetry with respect to
  boundary points A and B

30
18.2.4 Skeleton Representation

• axes spines loci of local symmetries maximal
  w.r.t. forming smooth curve
• cover of axis portion of shape subtended by axis
• axis cover properly contained in another cover
  second axis subsumes first

31
18.2.4 Skeleton Representation

• symmetric axes of local symmetry of a rectangle

32
18.2.4 Skeleton Representation

• axes of smoothed local symmetries of several
  objects

33
18.2.5 Two-Dimensional Part Representation

• parts, attributes, interrelationships form
  structural description of shape
• nuclei regions where primary convex subset
  overlap
• nuclei shaded areas of overlap

34
18.2.5 Two-Dimensional Part Representation

• decomposition of shape into primary convex
  subsets and nuclei

35
18.2.5 Two-Dimensional Part Representation

• near-convexity allows noisy distorted instances
  to have same decompositions
• , two points on object boundary
• relation visibility relation
• if line completely interior to object
  boundary,
• the graph-theoretic clustering to determine
  clusters of visibility relation

36
18.2.5 Two-Dimensional Part Representation

• decomposition of three similar shapes into
  near-convex pieces

37

• joke

38
18.3 Three-Dimensional Object Representations
39
18.3.1 Local Features Representation

• range data obtained from laser range finder,
  light striping, stereo, etc.
• from depth, try to infer surfaces, edges,
  corners, holes, other features
• 3D matching more difficult than 2D because of
  occlusion

40
18.3.2 Wire Frame Representation

• wire frame model 3D object model with only edges
  of object

41
18.3.2 Wire Frame Representation

• two-color hyperboloid and its line drawing

42
18.3.2 Wire Frame Representation
43
18.3.2 Wire Frame Representation

• Necker cube lower-vertical face or upper
  vertical face closer to viewer
• Schroder staircase viewed either from above or
  from below

44

• two well-known ambiguous line drawings

45

• two well-known ambiguous line drawings

46

• two well-known ambiguous line drawings

47

• inherent ambiguity of line drawing owing to
  complete loss of depth

48
18.3.2 Wire Frame Representation

• general-viewpoint assumption none of the
  following situations
• 1. two vertices of scene objects represented at
  same picture point
• 2. two scene edges seen as single line in picture
• 3. vertex seen exactly in line with unrelated
  edge

49
18.3.2 Wire Frame Representation

• general-viewpoint assumption heart of
  line-drawing interpretation
• viewpoint in perspective projection center of
  projection
• viewpoint in orthographic projection direction
  of projection

50

• subjective contours of Kanizsa white occluding
  triangle in space

51
(No Transcript)
52
18.3.2 Wire Frame Representation

• line labels for visible projections of
  surface-normal discontinuities

53
(No Transcript)
54

• four basic ways in which three planar surfaces
  can form polyhedron vertex

55

• all possible distinct appearances of trihedral
  vertices of polyhedra

56

• complete junction catalog for line drawings of
  trihedral-vertex polyhedra

57

• complete junction catalog for line drawings of
  trihedral-vertex polyhedra

58

• joke

59
18.3.3 Surface-Edge-Vertex Representation

• VISIONS system Visual Integration by Semantic
  Interpretation of Natural Scenes
• PREMIO system Prediction in Matching Images to
  Objects
• PREMIO 3D object model hierarchical, relational
  model with five levels
• 5 levels world, object, face/edge/vertex,
  surface/boundary, arc/2D, 1D piece

60
18.3.3 Surface-Edge-Vertex Representation

• world level arrangement of different objects in
  world
• object level arrangement of different faces,
  edges, vertices forming objects
• face level describes face in terms of surfaces
  and boundaries
• surface level specifies elemental pieces forming
  surfaces

61
18.3.3 Surface-Edge-Vertex Representation

• 2D piece level describes pieces and specifies
  arcs forming boundaries
• 1D piece level describes elemental pieces
  forming arcs
• SDS spatial data structure
• A/V attribute-value table

62
(No Transcript)
63
18.3.4 Sticks, Plates, and Blobs

• sticks, plates, blobs model rough models of 3D
  objects used in rough-matching
• sticks long, thin parts with only one
  significant dimension
• stick cannot bend very much
• stick has two logical endpoints, set of interior
  points, center of mass

64
18.3.4 Sticks, Plates, and Blobs

• plates flattish wide parts with two nearly flat
  surfaces
• plates have two significant dimensions
• plate surfaces cannot fold very much
• plate has set of edge points, set of surface
  points, center of mass
• blobs parts with three significant dimensions
• blob can be bumpy but cannot have concavities
• blob set of surface points and center of mass
• sticks, plates, blobs near-convex

65
(No Transcript)
66
18.3.4 Sticks, Plates, and Blobs

• attribute-value table contains global attributes
• simple-parts relation lists the parts and their
  attributes
• connects-supports relation gives connections
  between pairs of parts
• triples relation specifies connections between
  three parts at a time
• parallel relation lists pairs of parts that are
  parallel
• perpendicular relation lists pairs of parts that
  are perpendicular
• TYPE 1 for stick, 2 for plate, 3 for blob

67

• full relational structure of sticks-plates-blobs
  model of chair object

68
(No Transcript)
69
18.3.5 Generalized Cylinder Representation

• generalized cylinder volumetric primitive
  defined by axis and cross-section
• cross section swept along axis, creating a solid
• e.g. actual cylinder generalized cylinder whose
  axis is straight-line segment and whose cross
  section is circle of constant radius
• e.g. cone generalized cylinder whose axis is
  straight-line segment and cross section is circle
  with radius initially zero to maximum

70
18.3.5 Generalized Cylinder Representation

• e.g. rectangular solid generalized cylinder
  whose axis is straight line segment and cross
  section is constant rectangle
• e.g. torso generalized cylinder whose axis is
  circle and whose cross section is constant circle
• generalized cylinder representation uses
  generalized cylinders as primitives

71
18.3.5 Generalized Cylinder Representation

• surface-edge-vertex model very precise
• sticks-plates-and-blobs model very rough
• generalized cylinder model somewhere in between

72
18.3.5 Generalized Cylinder Representation

• person modeled roughly as cylinders for head,
  torso, arms, legs
• dotted lines axes of cylinders

73
18.3.6 Superquadric Representation

• superquadrics lumps of clay deformable and can
  be glued into object models
• superquadric models mainly used with range data

74
18.3.6 Superquadric Representation

• Superquadrics are a flexible family of
  3-dimensional parametric objects, useful for
  geometric modeling. By adjusting a relatively few
  number of parameters, a large variety of shapes
  may be obtained.

75

• range data image of (a) a doll, (b) its
  superquadric fit (c), (d) wire frame

76

• joke

77
18.3.7 Octree Representation

• octree encoding geometric modeling technique
  used to represent 3D objects
• octree encoding used in computer vision,
  robotics, computer graphics
• octree hierarchical 8-ary tree structure
• each node in octree corresponds to cubic region
  of universe
• full if cube is completely enclosed by 3D object
• empty if cube contains no part of object

78
18.3.7 Octree Representation

• partial if cube partly intersects object
• full, empty, partial correspond to black,
  white, gray in quadtrees
• node with label full or empty has no children
• partial has eight children representing
  partition of cube into octants

79
(No Transcript)
80
18.3.8 The Extended Gaussian Image

• 3D object collection of surface normals, one at
  each point of object surface
• planar surface all points on surface map to same
  surface normal
• convex with positive curvature everywhere
  distinct surface normal everywhere
• Gaussian sphere unit sphere
• set of surface normals mapped to Gaussian sphere
  tail at center head outward
• Gaussian image of object resultant set of points
  on Gaussian sphere

81
(No Transcript)
82
18.3.8 The Extended Gaussian Image

• for planar objects Gaussian image not
  invertible, not precise enough for use
• small surface patch of object
• corresponding surface patch on Gaussian
  sphere
• Gaussian curvature K

83
18.3.8 The Extended Gaussian Image

• point on Gaussian sphere
  corresponding to point (u, v) on object surface
• extended Gaussian image
• planar region Gaussian curvature 0, point mass
  in extended Gaussian image

84
18.3.9 View-Class Representation

• view classes each representing set of viewpoints
  sharing some property
• property e.g. same object surfaces visible
• property e.g. same line segments visible
• property e.g. relational distances between
  relational structures are similar
• characteristic views sets producing
  topologically isomorphic line drawings

85
18.3.9 View-Class Representation

• three view classes of cube producing
  topologically isomorphic line drawings

86
(No Transcript)
87
18.3.9 View-Class Representation

• aspect graph of object graph structure where
• 1. each node represents topologically distinct
  view of object
• 2. a node for each such view of object
• 3. each arc represents a visual event at
  transition
• 4. there is an arc for each such transition

88
(No Transcript)
89
18.4 General Frameworks for Matching

• matching finding correspondence between two
  entities
• consistent labeling procedures examples of
  matching algorithms

90
18.4.1 Relational-Distance Approach to Matching

• relational distance compares two structures and
  determines similarity
• Relational-Distance Definition
• relational description
• sequence of relations
• set of parts of entity being described
• relation indicating various relationships
  among parts
• relational description with part set A
• relational description with part set B

91
18.4.1 Relational-Distance Approach to Matching

• assumption A B, otherwise add dummy parts
  to smaller set
• f any one-one, onto mapping from A to B
• N positive integer
• composition R f of relation R with
  function f

92
18.4.1 Relational-Distance Approach to Matching

• f maps parts from set A to parts from set B
• structural error of f for ith pair of
  corresponding relations in ,
• total error of f with respect to ,
• relational distance between
  ,

93
18.4.1 Relational-Distance Approach to Matching

• best mapping from to mapping f
  that minimizes total error

94
18.4.1 Relational-Distance Approach to Matching

• Relational-Distance Examples
• best mapping from to
  is
• for this mapping

95
18.4.1 Relational-Distance Approach to Matching

• two digraphs whose relational distance is 3

96
(No Transcript)
97
18.4.1 Relational-Distance Approach to Matching

• Relational Distance as a Metric
• relational distance used to determine similarity
  of unknown object to model
• relational distance used to compare object
  models to group models
• f relational isomorphism if f one-one, onto from
  A to B and E(f) 0
• f A B relational isomorphism ,
  isomorphic
• GD relational-distance measure

98
18.4.1 Relational-Distance Approach to Matching

• arbitrary relational descriptions
• , , metric property of GD
• 1. isomorphic
• 2.
• 3.

99
18.4.1 Relational-Distance Approach to Matching

• Attributed Relational Descriptions and Relational
  Distance
• extend relational description and relational
  distance to include
• properties of parts
• properties of the whole
• properties of these relationships

100

• joke

101
18.4.2 Ordered Structural Matching

• definition of ordering on primitives greatly
  reduces complexity of search

102
18.4.3 Hypothesizing and Testing with Viewpoint
Consistency Constraint

• viewpoint consistency constraint
• The locations of all projected model features
  in an image must be consistent with projection
  from a single viewpoint.

103
18.4.4 View-Class Matching

• if 3D object represented by view-class model,
  matching divided into 2 stages
• 1. determining view class of object
• 2. determining precise viewpoint within that view
  class

104
18.4.4 View-Class Matching

• Determining View Class
• relational pyramid hierarchical relational
  structure to represent view class
• Level-1 primitives straight- and curved-line
  segments
• Level-2 relations junctions and loops
• Level-3 relations adjacency, collinearity,
  junction parallelness, loop-inside-loop

105
18.4.4 View-Class Matching

• Pose Determination within View Class
• relational pyramid hierarchical, relational
  structure to constrain matching

106
18.4.5 Affine-Invariant Matching

• set of interest points
  lying in plane
• rotation matrix relating model reference frame to
  camera reference frame
• translation of object reference frame to camera
  reference frame

107
18.4.5 Affine-Invariant Matching

• f distance between image plane and center of
  perspectivity
• observed image data points
  by perspective projection
• when translation in z-direction large
  compared with

108
18.4.5 Affine-Invariant Matching

• A 2 x 2 (scaling, rotation, skewing) matrix
• b 2D (translation) vector
• affine 2D correspondence Aw b

109
18.4.5 Affine-Invariant Matching

• Affine Transformation of Points in a Plane
• necessary and sufficient to define plane
  uniquely 3 noncollinear points

110
18.4.5 Affine-Invariant Matching

• The Hummel-Wolfson-Lamdan Matching Algorithm
• to match noncollinear triplets in model interest
  points with scene
• preprocessing convert model interest points into
  affine-invariant model
• recognition match model against image using
  affine representation

111
18.4.5 Affine-Invariant Matching

• Shortcomings of the Affine-Invariant Matching
  Technique
• affine-invariant matching technique
  mathematically sound in noiseless case
• shortcomings of affine-invariant matching in
  practice
• 1. if three noncollinear points not numerically
  stable, points not reliable
• 2. coordinates of detected interest points noisy
  in real image
• 3. partial object symmetries may cause wrong
  matching

112
18.4.5 Affine-Invariant Matching

• An Explicit Noise Model and Optimal Voting

113
18.5 Model Database Organization

• organize database of models to allow rapid
  access to most likely candidate
• group similar relational models into clusters and
  choose representative
• arrows indicate mapping from parts of object 2
  to parts of other objects
• cluster of object models whose representative is
  object 2

114
(No Transcript)
115

• END