Title: Vision%20Based%20Motion%20Control
1Vision Based Motion Control
-
- Martin Jagersand
- University of Alberta
- CIRA 2001
2Vision Based Motion Control
- Martin Jagersand
- University of Alberta
- CIRA 2001
3Content
- Vision based motion control
- Programming and solving whole human tasks
- Software systems for vision and control
- Discussion
41. How to go from Visual sensationto Motor
action?gt
- Camera -gt Robot coord Robot -gt Object
5Closed loop traditional visual servoing
EE
- This talk focus on estimating the geometric
transforms
6Lots of possible coordinates
- Camera
- Frame at projection center
- Many different models
- Robot
- Base frame
- End-effector frame
- Object frame
Traditional modeling PP1(ltparamsgt)
P2(ltparamsgt) Pn(ltparamsgt)
7Hand-Eye system
- Motor-Visual function yf(x)
- Jacobian J( dfi / dxj )
8RecallVisual specifications
- Point to Point task error
Why 16 elements?
9Visual Servoing
- Observed features
- Motor variables
- Local linear model
- Visual servoing steps 1 Solve
- 2 Move
10Find J Method 1 Test movements along basis
- Remember J is unknown m by n matrix
- Assume movements
- Finite difference
11Find J Method 2Secant Constraints
- Constraint along a line
- Defines m equations
- Collect n arbitrary, but different measures y
- Solve for J
12Find J Method 3Recursive Secant Constraints
- Based on initial J and one measure pair
- Adjust J s.t.
- Rank 1 update
- Consider rotated coordinates
- Update same as finite difference for n orthogonal
moves
13Trust region of J estimate
- Let be the trust region at time t
- Define a model agreement
- Update the trust region recursively
Where dupper and are dlower predefined constants
14Visual Servoing Steps
- Solve
- Update and move
- Read actual visual move
- Update Jacobian
repeat
15 Visual Servoing Steps
- Solve
- Update and move
- Read actual visual move
- Update Jacobian
repeat
16Jacobians Spline model of underlying non-linear
function
- Over time acquires several Jacobians J
- Each J a hyperplane
- Collection of Js form a (sparse) piecewise
linear spline
17Jacobian based visual model
- Assume visual features mgtgtn motor freedoms
- All visual change restricted to n freedoms by
- Can predict visual change
- Can also parameterize x visually
18Related visual modelAffine model
e1
O
e2
e3
- Affine basis
- Image projection of origin
- Image basis
19Find affine coordinates
e1
q
O
e2
e3
- Observe (track) y through time
- Solve an equation system to find q
- Reprojection Have q,want y
20Relation Affine Jacobian image models
21Composite affine and Jacobian model
- Chain the affine and Jacobian model
- Represents rigid objects in arbitrary motor frame
22Transforms Affine-Jacobian model
- Measurement matrix
- Affine coordinate equation
23ExperimentAffine animation of rigid structure
24Affine vs. Visual-Motor
25Other sensory modalities Force and contact
manipulation
- Accuracy is limited by Visual tracking
- and Visual goal specification
- Specifying well defined visual encodings can be
difficult - Limited to non-occluded settings
- Not all tasks lend themselves to visual
specification.
26Constraint Geometry
- Impact force along surface normal
- Sliding motion
- 3rd vector
27Constraint Frame
- With force frame tool frame we get
- Assume frictionless gt Can update each time step
P2
P1
P3
28Hybrid Control Law
- Let Q Joint -gt Tool Jacobian
- Let S be a switching matrix, e.g. diag(0,1,1)
- Velocity control u
Visual part
Force part
29Accounting for Friction
- Friction force is along motion direction!
- Subtract out to recover surface normal
30Motion Sequence
31Motion Sequence
32Summary of model estimation and visual motion
control
- Model estimation is on-line and requires no
special calibration movements - Resulting Jacobians both model/constrain the
visual situation and provide visual motor transf. - Motion control is direct from image based error
functions to motor control. No 3D world space.
332. How to specify a visual task sequence?
- Grasp
- Move in
- Cut
- Grasp
- Reach close
- Align
- Turn
-
34Recall Parallel Composition Example
Visual error function spelled out
35Serial CompositionSolving whole real tasks
- Task primitive/link
- Acceptable initial (visual) conditions
- Visual or Motor constraints to be maintained
- Final desired condition
- Task
36Natural primitive links
- Transportation
- Coarse primitive for large movements
- lt 3DOF control of object centroid
- Robust to disturbances
- Fine Manipulation
- For high precision control of both position and
orientation - 6DOF control based on several object features
37Example Pick and place type of movement
- 3. Alignment???
- To match transport final to fine manipulation
initial conditions
38More primitives
- 4. Guarded move
- Move along some direction until an external
contraint (e.g. contact) is satisfied. - 5. Open loop movements
- When object is obscured
- Or ballistic fast movements
- Note can be done based on previously estimated
Jacobians
39Solving the puzzle
40Teaching and Programming in Visual Space
- Tele Assistance
- A tele-operator views the scene through stereo
cameras - Objects to be manipulated are pointed out on-line
- Visual Programming
- Off-line
- Like xfig, macpaint, but with a palette of motor
actions. - Teaching by Showing
- A (human) manipulation is tracked in visual space
- The tracked data is used to (automatically?)
generate a sequence of visual goals
41HCI Direct manipulationExample xfig drawing
program
- Icons afford use
- Results visible
- Direct spatial action-result mapping
matlab drawing
line(10, 20,30, 85) patch(35, 22,15, 35,
C) C complex structure text(70,30,'Kalle')
Potentially add font, size, etc
42ExampleVisual programming
43Task control summary
- Servoing alone does not solve whole tasks
- Parallel composition Stacking of visual
constraints to be simultaneously satisfied - Serial composition Linking together several
small movements into a chain of continuous
movements - Vision-based user interface
- Tele-assistance
- Visual Programming
- Teach by showing
44Types of robotic systems
Preprogrammed systems
Autonomy
Programming by demonstration
Tele-assistance
Supervisory control
Generality
453. Software systems for vision-based control
46Hand-EyeSystem
47System requirements
- Solve many very different motion tasks
- Flexible, teachable/re-programmable
- Real time
- On special embedded computers or general
workstations - Different special HW
- Multiprocessors
48Toolbox
49System design
- Interpreted scripting language gives
flexibility - Compiled language needed for speed and HW
interface. - Examples
Matlab
Haskell
PVM
Dyn linking (mex)
Greencard
C, C, fortran
C, C
50Usage example
- Specialize robot
- projandwait(zero3,robotmovehill,A3D,WaitForHill
) - Initialize goals and trackers
- TrackCmd3D,N InitTrackers(1 1,0,1)
- PU GetGoals(1 1,0,1)
- Servo control
- J3s LineMove(projandwait,TrackCmd3D,J3i,PU,Ndi
,err)
51Software systems summary
- Most current demos solve one specific movement
- For solving many everyday tasks we need
flexibility and reprogrammability - Compiled primitive visual trackng and
- Interpreted scripting language
- Higher order functions
52Workshop conclusions
53Workshop conclusions
- Sensing is unreliable and incomplete
- Cant reliably build internal 3D world models,
but can use the real world as an external
reference. - A-priori object and world models uncommon in
human environments - Estimate on-line and only whats needed.
- Human users require human interaction techniques
- Interacting by visual pointing and gestures is
natural.
54Action/Perception division in human and machine
hand-eye syst.
55Open questions?
- For particular tasks what are the most natural
representations and frames? - Global convergence of arbitrarily composed visual
error functions? - Robustness?
- Interaction with other sensing modalities?
56Feedback system
- Fast internal feedback
- Slower external trajectory corrections
57Short and long control loops
58Applications for vision in User Interfaces
- Interaction with machines and robots
- Service robotics
- Surgical robots
- Emergency response
- Interaction with software
- A store or museum information kiosk
59Service robots
- Mobile manipulators, semi-autonomous
DIST TU Berlin KAIST
60TORSO with 2 WAMs
61Service tasks
This is completely hardwired! Found no real task
on WWW
62But
- Maybe first applications in tasks humans cant do?
63Why is humanlike robotics so hard to achieve?
- See human task
- Tracking motion, seeing gestures
- Understand
- Motion understanding Translate to correct
reference frame - High level task understanding?
- Do
- Vision based control