Shared Control for Dexterous Telemanipulation with Haptic Feedback

1
Shared Control for Dexterous Telemanipulation
with Haptic Feedback

• Weston B. Griffin
• Dissertation Defense Presentation
• May 1, 2003

2
Telemanipulation

• First systems developed 1940s
• handling radioactive materials
• Can provide access todangerous environments
• Benefit from natural human abilities

slave
master
operator
environment
The E1 developed by Goertz at Argonne National
Lab
3
Telemanipulation

• Applications include
• underwater salvage
• nuclear waste handling
• space station repair
• minimally invasive surgery

Intuitive Surgical, Canadian Space Agency,
Oceaneering International
4
Telemanipulation Frameworks

• computer controlled electro-mechanical systems
• remote controlled robot ltgt feeding back
  information

• several different architectures

Operator
Master System
Slave Controller
Slave Manipulator
extends a persons sensing and/or manipulation
ability to a remote location
5
Manipulation

• Desire to leverage human manipulation skills
• immersive hand/finger based system

Slave Manipulator
Master System
6
Movie

• Remote Control by Andy Shocken
• filmed 2002 in our lab
• narrated by Mark Cutkosky

7
Issues in Telemanipulation

• operator may feel remotely present
• BUT
• is not getting normal manipulation cues

• Current telemanipulation limitations
• force feedback (limited accuracy and fidelity)
• limited tactile display

8
Contributions

• Development a human-to-robot mapping method
• map glove-based hand motions to a planar robot
  hand
• Development and implementation of a shared
  control framework for dexterous telemanipulation
• combining operator commands with a
  semi-autonomous controller
• Investigation of an experimental telemanipulation
  system
• results demonstrate benefits of shared control
  and need to choose carefully types of feedback to
  achieve a real improvement

9
Outline
Development of dexterous telemanipulation system

• System overview
• Human-to-robot mapping
• Shared control framework
• Experimental investigation

10
Improving Telemanipulation

• Take advantage of the slave controller and local
  sensor information for improved dexterity
• add low-level intelligence

• Why?
• can feedback sensor information by other means
• robot can intervene in certain situations (fast
  response)
• human and robot can share control for improved
  performance

11
Shared Control
bilateraltelemanipulation
high level commands feedback
semi-autonomous dexterous manipulation
12
Shared Control
combining operator high level and low level
commands with a remote controller for improved
manipulation
13
Master System

• CyberGlove instrumented glove
• 22 bend sensors
• calibrated for dexterous manipulation Turner
  2001

• CyberGrasp fingertip force feedback
• lightweight exo-skeleton
• uni-directional force feedback

• Logitech hand tracker
• ultrasonic transducers and sensors
• 6 d.o.f. position and orientation

CyberGlove and CyberGrasp are products of
Immersion Corporation
14
Slave System

• Robot arm
• Adept industrial arm, five d.o.f.
• enlarges task workspace

• Custom built robot hand
• two fingers, two d.o.f. per finger
• low inertia DC motors
• cable capstan drive

• Fingertip sensors
• two-axis force sensors
• contact location sensors

15
System Architecture
Master
CyberGlove
CyberGrasp
Indirect Feedback
Wrist Tracker
GUI
QNX Node-to-Node
QNX-Node 1
QNX-Node 2
Adept Control
Slave
Slave Control
16
Outline
Development of dexterous telemanipulation system

• System overview
• Human-to-robot mapping
• Shared control framework
• Experimental investigation

17
Human-to-Robot Mapping

• Robot is non-anthropomorphic, symmetric, and
  planar

• joint-to-joint mapping not possible
• very different workspace

18
Human-to-robot Mapping

• How do you control a non-anthropomorphic robot
  hand using a human hand and glove?

? ?
19
Virtual Object Mapping

• Interpret human fingertip motions to be imparting
  motions to a virtual object held between the
  fingers
• Virtual object parameters are mapped to robot
• to produce fingertip positions OR motions of a
  grasped object
• Parameters independently modified
• to account for kinematic and workspace differences

20
Virtual Object Mapping

• Match natural human manipulation motions to
  corresponding robot hand motions

• good mapping?
• operator can intuitively control robot and
  utilize robots workspace

21
Outline
Development of dexterous telemanipulation system

• System overview
• Human-to-robot mapping
• Shared control framework
• Experimental investigation

22
Shared Control

• Hannford et al. 1991
• force feedback joystick controlling robot
  arm/gripper
• improved task completion time and resulted in
  lower forces

• Michelman and Allen 1994
• sequencing primitives for dexterous hand control
• joystick control, no provisions for haptic
  feedback
• Williams et al. 2002
• NASAs Robonaut project - robot arm and dexterous
  hand
• force feedback joystick for control
• reduced task peak forces

23
Shared Control

• Next step using shared control in a dexterous
  telemanipulation system with fingertip force
  feedback
• How?
• implement a semi-autonomous controller capable of
  dexterous manipulation
• robot has force and tactile sensors and
  specialized control laws for manipulation

24
Dexterous Manipulation

• What does it mean to autonomously manipulate an
  object?
• with sensors robot can detect the object and
  determine proper fingertip forces for

manipulation
25
Dexterous Manipulation

• What does it mean to autonomously manipulate an
  object?
• with sensors robot can detect the object and
  determine proper fingertip forces for

manipulation
26
Dexterous Manipulation

• What does it mean to autonomously manipulate an
  object?
• with sensors robot can detect the object and
  determine proper fingertip forces for

manipulation
grasp force regulation
27
Object Manipulation Control

• Utilize the Grasp Transform to determine robot
  fingertip forces Mason Salisbury 1985

28
Object Manipulation Control

• Controlling internal force

Velocity Grasp Transform
Ts


ZOH
Tactile Based Object Tracking
-
Tactile Sensing

Object ImpedanceController


Forward Grasp Transform
Finger Controller
RobotFinger
Internal ForceController

-
Internal ForceDecomposition
29
Object Manipulation Control

• Controlling object position

30
Shared Control Telemanipulation

• What are the advantages to programming robot for
  dexterous manipulation?
• robot can monitor operators object manipulation
• if necessary, robot can intervene (take over
  control of object manipulation)
• impedance modification, limit motion, prevent
  release
• robot can warn/inform operator of manipulation
  status through indirect methods
• using other feedback modalities (visual
  indicators, audio, or augmented haptic feedback)

31
Shared Control Telemanipulation

• What are the advantages to letting robot take
  control over force regulation and/or object
  manipulation?
• operator can focus on behavior of grasped object
  or tool
• master commands are no longer essential to
  prevent unwanted slip or damaged objects
• operator can still override to release or grasp
  more tightly

32
Shared Control Telemanipulation

• Shared control implementation issues

• as the robot assumes more control
• concern the operators sense of presence will be
  reduced
• we want to keep the operator in the loop
• preserve operators intent
• what type of indirect feedback is most effective?
• does sharing control improve performance in an
  immersive fingertip force feedback system?
• To answer these questions we perform a set a
  controlled experiments

33
Outline
Development of dexterous telemanipulation system

• System overview
• Human-to-robot mapping
• Shared control framework
• Experimental investigation

34
Previous Experimental Studies

• force feedback evaluation
• Turner et al. 2000 block stacking and knob
  turning
• force feedback with CyberGrasp not always a
  benefit
• Howe Kontarinis 1992 fragile peg insertion
  task
• audio buzzer sounded if grasp force excessive
• operators were not able to reduce force
• shared control evaluation
• Hannaford et al. 1991 peg insertion task
• operators controlled position, shared
  orientation control
• reduction in task completion time and insertion
  forces

35
Experimental Hypothesis

• Addition of a dexterous shared control framework
  will increase an operators ability to handle
  objects delicately and securely compared to
  direct telemanipulation

36
Experiment Description

• Motivating scenario recovering an ancient Greek
  vase on the sea floor

fragile object handling - users asked to carry
an object with minimal force but without dropping
the object
37
Experimental Task
38
Experiment Description

• To assist operator in fragile object handling
  taskthe robot computes the minimum grasp force
  required

39
Shared Controlled Task

• Operator maintains manipulation control

40
Shared Controlled Task

• Operator maintains manipulation control

• Robot and operator share control over internal
  force
• robot monitors excessive force

41
Shared Controlled Task

• Operator maintains manipulation control

• Robot and operator share control over internal
  force
• robot monitors excessive force
• robot can apply minimum internal force required
  to prevent slip

42
Sharing Control in Fragile Task

• Target window with intervention can be wider
  desired force can drop below fint,min without
  adverse effects
• In theory, it is possible to always do better
  without intervention

43
Question that arise...

• Does warning the operator of a possible failure
  help?
• Does task performance improve with robot
  intervention?
• If robot intervenes, is it necessary to inform
  operator?
• Is it helpful to feed back information of
  impending state changes (such as object release)?
• With haptic feedback in a force control task,
  what forces should be fed back?

44
Case Effects

• Audio Alarms - when operators desired force is
  too high or too low

• Robot Intervention - robot assumes control when
  operators desired force falls below a threshold
  (safe minimum internal force)

• Visual Indicator (fingertip LEDs) - to inform the
  operator of robot intervention

• Force Feedback actual vs. commanded - during
  robot intervention, forces to operators
  fingertip are reduced (reduced force feedback)

45
Experiment Cases
46
Case Effects
47
Experimental Procedure

• Diverse set of subjects
• 11 subjects total
• 8 males and 3 females
• Two sessions
• first - calibration and training
• second - four trials for each case
• Case order randomized
• reduce possible learning and fatigue effects

48
Evaluating Performance

• Objective data analysis
• measured internal force applied to object
• fragile object task - lower is better
• task failures (number of drops)
• task completion time
• Subjective data analysis
• operators expressed preference
• operators perceived difficulty

49
Typical Subject Data
50
Data Analysis

• Measured internal force applied to the object
• averages of each subject for each case (trial
  failures excluded)

• Boxplot
• medians and quartiles
• observe trends

• Is there a significant effect?

51
Statistical Analysis

• ANOVA - determines the probability that these
  results (differences in averages) are really due
  to random variation in data
• Apply to averaged measured internal force
• p 0.003 (ltlt 0.05), indicating that there is a
  difference between the means
• but which ones are different
• Cant use a simple t-test for multiple
  comparisons
• increase probability of false-positive
• Dunnetts method - comparison to a control (Case
  1)
• Cases 4, 6, 7 have statistically different mean
  than Case 1
• a reduction of approximately 15

52
Task Failures

• Number of failures that occurred for each case
  (dropped object)

Number of Failures in Each Case - All Subjects
8
6
Case 5 and 6 had least number of failures
Number of Failures
4
2
0
1
2
3
4
5
6
7
Case Number
sub1
sub2
Number of Failures in Each Case
N/A
sub3
sub4
8
sub5
Case failures not dominated by one subject
sub6
sub7
6
sub8
N/A
sub9
Number of Failures
4
sub10
sub11
2
0
1
2
3
4
5
6
7
Case Number
53
Objective Data Analysis Results

• Robot intervention improves performance
• presence and type of direct and indirect feedback
  had an effect
• Cases 4, 6, and 7 had lower internal force
• Case 3 and 5 did not

54
Analysis Results

• Robot intervention improved performance
• presence and type of direct and indirect feedback
  had an effect
• Cases 4, 6, and 7 had lower internal force
• Case 3 and 5 did not
• only informing of intervention not adequate
• Case 7 had most failures
• indicating alarms were helpful

Number of Failures
8
6
4
2
0
1
2
3
4
5
6
7
Case Number
55
Analysis Results

• Robot intervention improved performance
• presence and type of direct and indirect feedback
  had an effect
• Cases 4, 6, and 7 had lower internal force
• Case 3 and 5 did not
• only informing of intervention not adequate
• Case 7 had most failures
• indicating alarms were helpful
• Reduced force feedback
• compare Case 3 to 5
• slight improvement in measured internal force
  (6)
• fewer failures in Case 5
• Cases 4 and 6 show similar results

Number of Failures
8
6
4
2
0
1
2
3
4
5
6
7
Case Number
56
Task Time

• May reveal any physical or mental difficulties
  associated with the various conditions

35
no obvious trends p 0.82 (i.e., no difference
in means)
30
25
Time Sec
shared control did not improve task completion
time BUT did not make it worse
20
15
10
1
2
3
4
5
6
7
Case Number
57
Results

• Given objective data analysis performance
  criteria
• minimizing internal force but preventing
  failures
• provided best overall performance compared to
  bilateral case

Case 6 - shared control with multi-modal feedback

• In post experiment surveys, subjects also
  generally ranked Case 6 highest in preference
  and ease-of-use

58
Conclusions

• Answering our hypothesis
• Can the addition of a dexterous shared control
  framework increase an operators ability to
  handle objects delicately and securely compared
  to direct telemanipulation?

YES, shared control gives better performance but
you need to a) let the operator know when the
intervention is active b) let the operator know
of impending state changes c) feed back force
based on commanded force and not actual forces
(during intervention)
59
Summary of Contributions

• Development a human-to-robot mapping method
• map glove-based hand motions to a planar robot
  hand that allows for intuitive hand control
• Development and implementation of a shared
  control framework for dexterous telemanipulation
• combining operator commands with a
  semi-autonomous controller
• Investigation of an experimental telemanipulation
  system
• results demonstrate benefits of shared control
  and need to choose carefully types of feedback to
  achieve a real improvement

60
Future Work

• Do the benefits of shared control extend to other
  situations and applications?
• assembly tasks
• e.g., steer-by-wire vehicles
• Do the same requirements for shared control
  improvement hold?
• informing the operator of intervention
• notifying of impending state changes
• modifying the forces fed back

61
Acknowledgements

• Mark Cutkosky
• Defense Committee
• Will Provancher
• The DML
• Eric (setting the pace in the final days)

62
Shared Control for Dexterous Telemanipulation
with Haptic Feedback

• Weston B. Griffin
• Dissertation Defense Presentation
• May 1, 2003

63
Backup Slides
64
One Slide Statistics Review

• statistical analysis
• two competing hypotheses
• null cases have no real effect (all the means
  are the same)
• alternate at least one case is different (all
  means are NOT the same)

65
One Slide Statistics Review

• statistical analysis
• two competing hypotheses
• null cases have no real effect (all the means
  are the same)
• alternate at least one case is different (all
  means are NOT the same)
• ANOVA - analysis of variance
• tests if difference in means of several samples
  is significant based on variances

if ratio small then all means are the same
within
if ratio large at least one mean is different

• how likely is it to have a t.s. as extreme as
  observed (p-Value)
• compare to a significance level (95)(e.g.,
  reject null if p lt 0.05)

performance quantity
between
Case
66
Manipulation

• Desire to leverage human manipulation skills
• immersive hand/finger based system

position
Human-to-robot Mapping
Operator
Slave Manipulator
force
Master System
67
Telemanipulation

• Glove based
• Brunner et al. 1994, DLR dexterous robot hand
• Li et al. 1996 - NASA DART project
• Ambrose et al. 2000, NASA Robonaut project
• Teleoperation / telemanipulation
• Lawn and Hannaford 1993
• Lawrence et al. 1993
• Daniel and McAree 1998
• Sherman et al. 2000
• Speich and Goldfarb 2002

68
Control Architectures

• general four-channel one d.o.f. framework

Fh
C3
Ve
C4
-

Ve
Fh
Vm



C1
Zm -1
Zs -1
Fh
-
-
-
-
-
Cm
Cs
Ze
Zh

Fe
Vh
C2

Fe
Fe
Human Operator
MasterSystem
Comm.Link
SlaveSystem
Environ-ment
Lawrence 1993
69
Mapping Background

• anthropomorphic
• linear joint-to-joint Kyriakopoulos et. al 1997
• fingertip position mapping
• scaling Fisher et a. 1998
• semi-anthropomorphic
• pose matching Pao and Speeter 1989
• joint angle transformation matrix
• fingertip position mapping Speeter 1992,
  Rholing et al. 1993
• forward kinematics, inverse kinematics
• non-anthropomorphic
• greater dissimilarities
• grammar based
• functional mapping Speeter 1992

Utah/MIT hand
JPL/Salisbury hand
Dexter hand
70
Point-to-Point Mapping

• initial approach
• planar projection of fingertip positions
• standard planar frame transformation

71
Mapping Implementation

• compute virtual object parameters
• 3D size to capture thumb motion
• planar reduction

• computing robot positions
• based on planar virtual object

72
Transformation to Robot Frame

• must modify and scale parameters for desired
  correspondence

• kinematics
• v.o. orientation
• angular offset
• v.o. midpoint
• frame transformation

• workspace
• v.o. midpoint size
• scaled

73
Parameter Determination

• based on individuals recorded hand motion
• three simple poses/motions
• defining
• orientation offset
• midpoint transformation variables
• midpoint scaling
• size scaling

74
Mapping Results

• Virtual Object Mapping
• improved achievable positions
• pinch-point can be mapped to any point
• fundamentally analytical
• continuous, smooth, and predictable
• fingertip-to-fingertip correspondence

75
Modeling

• Model averaged percent difference in measured
  internal force compared to Case 1

Percent Difference (from Case 1 for each subject)
in Mean Internal Force
Means with Error Bars of Two Standard Deviations
15
10
5
0
Percent Difference,
-5
-10
-15
-20
-25
-30
1
2
3
4
5
6
7
Case Number
76
Model Analysis

• Look at residuals

Residuals due to Task Order (Learning and Fatigue
Effects)
Means with Error Bars of Two Standard Deviations
0.4
0.3
0.2
0.1
Force, N
0
-0.1
-0.2
-0.3
-0.4
1
2
3
4
5
6
7
Order