Can Autonomic Management Systems be Trusted

1
Can Autonomic Management Systems be Trusted?

• A Perspective from Business-Driven IT Management
• Jacques Sauvé - UFCG
• LANOMS 2007 Panel

2
Trust in IT Management

• Why can I talk about this? Whats my special
  perspective?
• I do research in Business-Driven IT Management
  (BDIM)
• Currently looking at ITIL processes and including
  a "business perspective"

3
An experiment closer to home ...

• How many of you mistrust some result presented at
  LANOMS 2007 or other conferences?
• Why?
• How many of you mistrust results presented in
  your own papers?
• Why?

4
Where does mistrust come from?

• Methodology!
• We are deficient in validation
• Questions
• Did you do any kind of validation to see if the
  problem is solved?
• Do you know (versus hope) that your solution is
  useful?
• Did you cover the problem space well?
• Do you know where your solution is good and where
  it is not?
• We plant the seeds of mistrust

5
Why are we deficient?

• Validation easy to do (in theory)
• Formulate a hypothesis "My management software
  improves metric X by Y
• Get a lot of data (from a large user base)
• See if it supports the hypothesis
• If it does, accept the hypothesis
• Your claims will now be valid when selling the
  solution
• Sweep the problem space to see what you can claim

6
But the real world creeps in ...

• Marketing people sell their own hypothesis
• How do you get a large user base to get data?
• Pressure to push products to market before they
  are validated
• How do you test your own baby?
• Of course, its good ... I built it!

7
Good enough solutions

• It takes time to validate and by then you have
  new modules, new problems, ...
• Given the lifecycle of management solutions, how
  much time can we afford to spend validating?
• Our solutions last only a (very few) years
• Validating systems is not like validating general
  theories
• Stop when the validation is good enough

8
Conclusion 1 Full scientific validation is
usually not an option

• But the following remains true ...
• Trust is gained through user experience
• Level of trust changes over time
• Building trust is a process
• Overselling may be good for the bottom line but
  may hurt trust

9
How to Lose Trust?

• Is it helpful to get it right 99 of the time?
• How about 95?
• Getting it wrong once on a while makes managers
  fidgety
• When can I trust it?"
• Losing trust is (almost) immediate and tools get
  boycotted
• Some processes are more critical
• Ex. Getting diagnostics wrong

10
Conclusion 2 Sell the right message

• Sell the message that the systems are useful,
  not always right or best
• Use metrics and baselines to enable comparison
• Use aggregate metrics
• But ...

11
Conclusion 3 Automatic Solutions Should not
Cause Grand Disasters!

• Example automatic scheduler loops and cant
  choose any job to run
• Example Automatic capacity allocation should not
  flip-flop and thrash
• Grand disasters reduce trust to zero

12
What Kind of Metrics Should we Use to Build Trust?

• Are business metrics are better than technical
  metrics?
• Technical BD throughput, resource utilization,
  response time, etc.
• Business business transaction throughput,
  financial gain or loss, risk, business impact,
  etc.

13
Non-Conclusion 1 Are Business Metrics Better for
Trust?

• I think business metrics are more useful but they
  are not directly measurable
• They may or may not help build trust more than
  easy-to-measure technical metrics
• Its hard to trust models, easier to trust
  measurements
• Use of business metrics helps the dialog with
  business people and should help validation and
  thus gain trust
• This is an untested hypothesis

14
The Effects of Scale

• Humans cant handle
• Number of elements and
• Number of dimensions present in todays scenarios

15
Conclusion 4 Autonomic will Rule Because of Scale

• Managers have no choice but to rely to automatic
  solutions because of scale
• Even if trust is not perfect

16
Other Possible Solutions

• Put user in the feedback loop!
• To calibrate tools
• To make tool learn
• To let user decide how much the tool should do
• Automate versus suggest actions

17
Thank you.