A Versatile Depalletizer of Boxes Based on Range Imagery

1
A Versatile Depalletizer of Boxes Based on Range
Imagery

• Dimitrios Katsoulas, Lothar Bergen, Lambis
  Tassakos

Inos Automation-software GmbH
University of Freiburg
2
Introduction

• Definition of Depalletizing problem.
• Our research focuses on depalletizing of objects
  in distribution centers (boxes, box-like objects,
  sacks).
• This contribution concerns depalletizing of
  boxes.
• Our system deals with pallets containing piled
  boxes of various dimensions.
• Our system is model based.

3
Related Work (1)

• Camera-based systems.
• Range Imagery based
• Katsoulas et.al 2002
• Kristensen et.al. 2001
• Baerveldt 1993
• Vayda et.al. 1990

4
Related Work (2)

• Chen et. Al. 1989
• Model based.
• Hypothesis generation and verification framework.
• Idea object vertices incorporate necessary
  constraints for unique determination of pose
  transform.
• Vertices are represented via surface normals and
  the vertex point.
• Vertex point determined via intersection of
  surfaces.
• 3 surfaces need to be exposed for accurate
  computation of vertex point.

5
Our Approach.

• Data Acquisition via a time of flight laser
  sensor mounted on the hand of the robot.
• Accurate detection of box edges.
• Vertex representation via box edges.
• Hypothesis generation and verification framework
  triggered by detected vertices.

6
The System in Operation
7
Edge Map Creation via Scan Line Approximation

• Edge points are detected by approximating the
  rows and columns of the range image with linear
  segments.
• The algorithm uses infomation from long line
  segments to compute the edge points.
• It is better than local approaches in terms of
  accuracy.
• It is fast.

8
Vertex Detection(1)

• Accurate detection of 3D lines corresponding to
  edges of boxes.
• Grouping of compatible box edges to yield
  Vertices. Representation of a Vertex as a
  triplet (feature set).
• Two-step, 3D line Detection (Inspired from the
  dynamic generalized Hough transform)
• Get rough approximation of the position of the
  lines in space.
• Exploit the sparsity of lines in the 3D space to
  refine the roughly computed line parameters.

9
Vertex Detection(2) 3D Line Detection

• 2D connected components (CCs) are determined.
• 3D line segments are fitted to the points
  defined by the CCs.
• Points in the vicinity of the segments are
  considered.
• Robust Calculation of the lines direction
  vector.
• Robust Calculation of the lines starting point.

10
Vertex Detection(3) 3D Line detection

• Calculation of the direction vector
• Selection of random pairs of points and
  calculation of difference vectors.
• Accumulation of the coordinates of the difference
  vectors in 3 1D accumulators.
• The maxima of the accumulators are the desired
  parameters provided that the standard deviation
  of the accumulators is below a threshold.
• Similar technique for calculating the starting
  point.

11
Object Recognition(1)

• Based on Hypotheses generation and verification.
• A scene vertex is aligned to a model box vertex.
• This alignment produces a transform which brings
  the model to the scene. This is equivalent to
  hypothesis creation.
• Verification of the hypothesis is performed via
  examination of the position of neighbouring
  vertices.

12
Object Recognition (2)- Model Database

• The model database contains triplets of the form
  (model local feature set) which express model
  vertices.
• 6 feature sets per model are stored in the model
  DB.

13
Object Recognition(3)
14
Hypothesis Generation

• If a scene vertex and a model
  vertex, a box location hypothesis is
  generated as follows
• Rotation Matrix
• Translation Vector

15
Hypothesis Verification (1)

• For every adjacent scene vertex, its position in
  the model coordinate system is determined
• If one of the vertices of the model which created
  the hypothesis is close to the back-projected
  adjacent vertex, it is considered compatible to
  the scene vertex which created the hypothesis.
• Verification depends on the number of compatible
  scene vertices found.

16
Hypothesis Verification (2)
17
Hypothesis Verification (3)

• Verification criterio when only one surface is
  exposed detection of one compatible vertex
  which shares no common edge with the hypothesis
  generating vertex.
• This safely verifies the occurence of a box side
  (surface).
• Not a problem, since we are looking for graspable
  surfaces.
• However, the set of necessary constraints for
  verification needs to be further reduced.

18
Hypothesis Verification (4)
19
Hypothesis Verification (5)
20
Hypothesis Verification (6)
21
Plane Fitting Test

• Idea The border points of the hypothesised
  surface can be accurately computed.
• 3D range points lying in the hypothesised surface
  segment are extracted.
• A plane is fitted to the 3D points.
• If the fitting error is below a threshold the
  hypothesis is considered verified.

22
Hypothesis Verification (6)
23
Discussion on the Plane Fitting Test

• It is implemented via polygon rasterization.
• It reduces the number of scene vertices that need
  to be detected for safe verification.
• It could be used as a verification method when no
  compatible scene vertices are detected.
• It ensures that a detected box surface is
  graspable.
• Helps for an even more accurate grasping.

24
Experimental Results (1) Intensity Image
25
Experimental Results (1) Edge Map
26
Experimental Results (1) Robust Lines
27
Experimental Results (1) Vertices
28
Experimental Results (1) Detected Boxes
29
Experimental Results (2) Intensity Image
30
Experimental Results (2) Edge Map
31
Experimental Results (2) Robust Lines
32
Experimental Results (2) Vertices
33
Experimental Results (2) Detected Boxes
34
System Advantages

• Computational efficiency fast vertex Detection,
  fast hypothesis generation and verification.
  Detection of more than one boxes per scan are
  detected. lt15 secs per box.
• Accuracy Accurate hypothesis generation due to
  robust detection of vertices. 4 degrees in
  orientation, 2cm in translation.
• Robustness robust verification criteria. No
  false identifications.

35
System Advantages (2)

• Independence from lighting conditions Employment
  of time of flight laser sensor.
• Versatility Deals with both layered and jumbled
  configurations.
• Ease of Installation Sensor on the hand of the
  robot.
• Low Cost Cost of the sensor 3000 Euro.
• Simplicity.

36
Problems

• System fails when the objects are placed very
  close to each other in distinct layers, because
  no edges points or vertices can be detected.
• Solution additional sensor (e.g. intensity
  camera) to resolve the correct orientation.

37
Future Work

• Exploit vertices detected in previous scans to
  make the recognition process faster.
• Depalletizing of non rigid box-like objects and
  piled sacks.