Gabriella Cortellessa and Amedeo Cesta

1
Toward a Reliable Evaluation of Mixed-Initiative
Systems

• Gabriella Cortellessa and Amedeo Cesta
• National Research Council of Italy
• Institute for Cognitive Science and
  Technology
• Rome, Italy

2
Outline

• Motivations
• Aims of the study
• Users attitude towards the mixed-initiative
  paradigm
• Role of explanation during problem solving
• Evaluation Method
• Results
• Conclusions and future work

3
Motivations

• Need for evaluation methodologies for
  mixed-initiative systems
• Investigation of advantages of mixed-initiative
  approach
• Better performances in problem solving, due to
  the use of complementary strengths
  (human-artificial).
• A higher level of satisfaction for human users
  who can preserve his/her control over the problem
  solving process.

4
Motivations

• Lack of studies that investigate users attitude
  towards this solving paradigm
• Lack of methodologies for evaluating different
  aspects of mixed-initiative problem solving
• This work applies an experimental approach
  (from HCI and Psychology) to the problem of
  understanding users attitude towards the
  mixed-initiative approach and investigating the
  importance of explanation as a means to foster
  users involvement in the problem solving

5
Two alternative Problem Solving approaches
Automated approach
Mixed-Initiative approach
6
Evaluating Mixed-Initiative Systems

• Measuring the overall problem solving performance
• The pair human-artificial system is supposed to
  exhibit better performances (metrics).
• Evaluating aspects related to users requirements
  and judgment on the system.
• Usability, level of trust, clarity of
  presentation, user satisfaction etc.

7
Aims of the study

• Users attitude towards the solving strategy
  selection.
• Automated vs mixed-initiative
• The recourse to explanation during problem
  solving
• Explanations for solvers choices and failures

Differences between experts and non experts
8
Solving strategy selection

• No empirical studies in the mixed-initiative area
  explore the context of strategy selection (who
  and why choose a solving strategy)
• However
• Decision Support Systems
• Empirical evidence of low trust toward automated
  advices during decision making processes (Jones
  Brown, 2002).
• Human-Computer Interaction
• Artificial solver as a competitor rather than a
  collaborator (Langer, 1992 Nass Moon, 2000).

9
Solving strategy selection Hypotheses

• Two variables are supposed to influence the
  selection of the solving strategy (automated vs.
  mixed-initiative) users expertise, and problem
  difficulty
• Hypothesis 1
• It is expected that expert users exploit the
  automated procedure more than non-experts and,
  conversely, non-expert users exploit the
  mixed-initiative approach more than experts.
• Hypothesis 1a
• It is expected that inexperienced users prefer
  the mixed-initiative approach when solving easy
  problems, and the automated strategy when solving
  difficult problems, while expert users are
  expected to show the opposite behavior.

10
Explanation Recourse

• No empirical studies in the mixed-initiative
  research field investigate the role of
  explanations in cooperative problem solving
• However
• Knowledge-Based Systems
• explanation recourse is more frequent in case of
  systems failures (Gilbert, 1989 Schank, 1986
  Chandrasekaran Mittal, 1999).
• explanation recourse is more frequent in case of
  collaborative problem solving (Gregor, 2001)
• individual differences in the motivations for
  explanations recourse (Mao Benbasat, 1996 Ye,
  1995).

11
Explanation Recourse Hypotheses

• The following variables are supposed to
  influence the recourse to explanation users
  expertise, problem difficulty, strategy
  selection, failure.
• Hypothesis 2
• The access to explanation is more frequent in
  case of failure than in case of success.
• Hypothesis 3
• Access to explanation is related to the solving
  strategy selection.
• In particular participants who choose the
  automated solving strategy access more frequently
  to explanation than those who use the
  mixed-initiative approach.

12
Explanation Recourse Hypotheses

• Hypothesis 4
• During problem solving non experts access
  explanations more frequently than experts.
• Hypothesis 5
• Access to explanation is more frequent in case
  of difficult problems.

13
Evaluation Method

• Participants
• 96 participants balanced with respect to gender,
  education, age and profession, subdivided in two
  groups based on the level of expertise (40
  experts and 56 non experts).
• Experimental apparatus
• COMIREM problem solver
• Planning and scheduling problems
• Procedure
• Web-based apparatus
• Stimuli Problems solution
• Questionnaires

14
A mixed-initiative problem solver COMIREM

• COMIREM Continuous Mixed-Initiative Resource
  Management
• Developed at Carnegie Mellon University

Automated Solver
Interaction Module
User
(Smith et al, 2003)
15
Planning and Scheduling problems
event
16
Procedure

• Training session
• Two experimental sessions presented randomly
• Session 1 easy problems
• Questionnaire 1
• Session 2 difficult problems
• Questionnaire 2
• For each session participants were asked to
  choose between mixed and automated strategy

Web-based
17
Tasks

• Stimuli
• 4 scheduling problems defined in the field of a
  broadcast TV station resources management
• 2 solvable
• 2 unsolvable
• Questionnaires aiming to
• Assessing the difficulty of the task 5-steps
  Likert scale (Manipulation check of variable
  difficulty)
• Evaluating the clarity of textual and graphical
  representations
• (5-steps Likert scale)
• Investigating the reasons for choosing the
  selected strategy (multiple choice)
• Studying the reasons for accessing the
  explanation (only 2nd questionnaire)

18
Solving Strategy Selection

• Results

19
Influence of expertise on strategy
Choice_auto
Choice_mixed
Influence of expertise on solving strategy
selection (statistics)
20
Influence of expertise on strategy
Hypothesis 1 Solving strategy selection
(automated vs mixed-initiative) depends upon
users expertise VERIFIED p lt .001 Experts
? automated Non experts ? mixed-initiative
21
Influence of difficulty on strategy
strategy
Easy Problems
expertise
Automated
Mixed
32
24
Chi-square 9.80, df1, plt .01
Non expert
10
30
Expert
42
54
Total
strategy
expertise
Difficult Problems
Automated
Mixed
32
24
Chi-square 3.6 , df1, n. s.
Non expert
15
25
Expert
47
49
Total
22
Influence of difficulty on strategy

• Hypothesis 1a
• Solving strategy selection (automated vs
  mixed-initiative) is related to problem
  difficulty
• PARTIALLY VERIFIED
• Easy problems ? experts automated, non experts
  mixed (plt .01)
• Difficult problems ? (n. s.)

23
Reasons for strategy selection
Automated -- Easy
Automated -- Difficult
Chi-square .92 , df2, n. s.
Chi-square 3.9 , df2, plt .05
Mixed -- Easy
Mixed -- Difficult
Chi-square 1.32 , df2, n. s.
Chi-square 1.15 , df2, n. s.
24
Explanation Recourse

• Results

25
Influence of failures on explanation
26
Influence of failures on explanation
Hypothesis 2 The access to explanation is more
frequent in case of failure than in case of
success. VERIFIED plt .001
27
Influence of strategy on explanation
Easy problems
Difficult problems
28
Influence of strategy on explanation
Hypothesis 3 Access to explanation is related
to the solving strategy selection. Access to
explanation is more frequent in case of
automated strategy choice VERIFIED
Easy problems plt .001
Difficult problems plt .05
29
Influence of expertise and difficulty on
explanation
30
Influence of expertise and difficulty on
explanation

• Hypotheses 4 e 5
• During problem solving non experts rely on
  explanation more frequently than experts
• Access to explanation is more frequent in case of
  difficult problems.

FALSIFIED
plt .01
expertise
31
Reasons for accessing explanation
Non Experts
Experts
Understand the problem
Understand automated solvers choices
Chi-square 2,28 , df1, n. s.
32
Conclusions

• Solving strategy selection depends upon users
  expertise
• Experts ? automated
• Non experts ? mixed-initiative
• The mixed initiative approach is chosen to
  maintain the control over the problem solving
• Explanation during problem solving is frequently
  accessed (73 out of 96 respondents), the access
  being more frequent in case of
• Failures during problem solving
• When using the automated strategy
• Explanation is accessed to understand solvers
  choices

33
Contributions

• Empirical proof that the mixed-initiative
  approach responds to a specific need of end users
  to keep the control over automated systems.
• The study confirms the need for developing
  problem solving systems in which humans play an
  active role
• Need for designing different interaction styles
  to support the existing individual differences
  (e.g., expert vs non experts)
• Empirical proof of the usefulness of explanation
  during problem solving. Failures have been
  identified as a main prompt to increase the
  frequency of access to explanation

34
Remarks

• Need for designing evaluation studies which takes
  into consideration the human component of the
  mixed-initiative system (importing methodologies
  from other fields)
• At present we have inherited the experience from
  disciplines like HCI and Psychology and adapted
  them to our specific case.
• The same approach can be followed to broaden the
  testing of different mixed-initiative features.

35
Future work

• Investigating the impact of strategy (automated
  vs mixed-initiative) and explanation recourse on
  problem solving performance.
• Application of the evaluation methodology to
  measure different features of the
  mixed-initiative systems.
• Synthesis of user-oriented explanations