How To Start a Metrics Program Without Causing

1
How To Start a Metrics Program Without Causing
!!

• Susan R. Morgan
• IQAA Program Meeting
• June 21, 2007

2

• The views presented in this presentation are that
  of the author and not of Liberty Mutual.

3
Statistician HUMORExample

• There is no truth to the allegation
• that statisticians are mean. They are
• just your standard normal deviates.

4
What are metrics?

• IT METRICS
• the continuous application of measurement-based
  techniques to the development process and its
  products to supply meaningful and timely
  management information, together with the use of
  those techniques to improve that process and
  products
• - Software Metrics, Best Practices for Succesful
  IT Management, Paul Goodman

5
What are metrics?

• Simply stated, metrics answer the question
• HOW ARE WE DOING?
• "The essence of mathematics is not to make simple
  things complicated, but to make complicated
  things simple."      -- S. Gudder

6
What Will Metrics Do For Your Organization?

• Areas of software development which
• can benefit from a well-planned metrics
• program
• Customer/Business Value You can show what was
  delivered and the value.
• Internal Performance of Work Processes What is
  working and what is not?
• Financials Where are we spending our ?
• Culture Training, inflow/outflow

7
Teach Yourself Help Yourself

• RESEARCH, RESEARCH, RESEARCH
• Internet
• Books
• Magazines
• Colleagues
• Conferences
• Network

8
My favorite resources

• The Six Sigma Handbook (Thomas Pyzdek)
• Software Metrics Establishing a Company-Wide
  Program (Robert Grady and Deborah Caswell)
• QI Macros (www.qimacros.com) short demo at the
  end
• SQE Star Conference Metrics track
• Rick Craig
• Local QA group
• Edward Tufte (www.edwardtufte.com)
• In Chicago, IL August 23rd and August 24th
• Completely integrate word, number and image
• Dont get it original get it right
• It is research not creativeness

9
Guiding LightsBecause it can be a soap opera!

• Management Commitment
• Communicate
• Be practical
• Do not measure people
• Make it simple
• Ask Why?
• Multi variable analysis
• Data accuracy and consistency
• Do not use until ready
• Train your organization
• Constant updates and review

10
Guiding Light 1 Management Commitment

• Essential for success
• Look out for the Of course I support metrics
  speech!
• May commit through words only
• No assistance with deploying the program into
  practice
• Educate management on the value of metrics
• If no luck then go to metrics at a project level

11
Guiding Light 2 Communicate, Communicate,
Communicate!

• Those that are closest to the processes and data
  are your friends as well as your customers
• DO NOT HIDE BEHIND EMAIL!!
• Obtain buy in from all parts of the organization
• Understand processes
• Understand data available and current measures
  being produced.
• LISTEN!!!
• Give something back you must deliver SOMETHING
  even if it is not what you had planned.

12
Guiding Light 3 Be Practical and Sensible

• You need a plan!
• Scope
• Requirements
• Timeframe
• Not too big
• 60 pages with 144 metrics JUST SAY NO!
• Not too small
• One metric will not suffice
• Realize nothing will solve all the problems the
  organization is having

13
Guiding Light 4 Do not measure people!

• Most organizations are not mature enough to
  measure people
• Measuring people is the sure way to fail they
  feel like a target
• The Hawthorne Effect
• People will change their behavior to fit the
  right numbers if measured

14
Guiding Light 5 Make it simple

• No single magic measure
• Recognize limitations of measures one will not
  tell the story
• Choose 3-5 metrics to start
• Something SIMPLE!
• Organization needs from discussions
• Defects?
• Delivery of product?
• Where is the biggest bang for the buck?
• Edward Tufte says Dont get it original get it
  right.
• Metrics/Goals Matrix

15
Metrics/Goal Matrix Example
Goals should come from executive mgt
16
Guiding Light 6 Ask WHY?

• Understand your metrics are INQUIRY metrics NOT
  DECISION MAKING metrics. Metrics give you the
  power to ask why things are happening.
• Must relate metrics to why they are important.
• No action from metrics until questions asked and
  analysis.
• Must involve those at all levels who know it best.

17
Guiding Light 7 Multi variable analysis is key

• Multi variable analysis is not SCARY.
• Simply looking at variables together rather than
  singly. May bring a clearer picture and more of
  the story.
• Many ways to show information to get the real
  picture or trends.
• Use business value with IT metrics.

18
Multi Variable ExampleDefects

• Many project managers know how many defects are
  open at any particular time.
• Can they answer the questions SIMPLY
• How many are coming in over time trend?
• What is the closed/open ratio over time?
• What is your backlog?
• How quickly are defects being turned around?

19
Multi Variable ExampleDefects
20
Multi Variable ExampleDefects
21
Multi Variable ExampleDefects
Total Opened
High Severity
Backlog
22
Multi Variable ExampleBacklog Maintenance

• BMI Closed/Open
• Backlog Maintenance Index will help understand
  the backlog trend.

Opened
Closed
BMI
23
Guiding Light 8 Data Collection Musts

• Data must be consistent.
• Data must be accurate.
• Data collection must not add process to your
  organization.
• DOCUMENT EVERYTHING!!!

24
Guiding Light 9 Do Not Open Until Ready

• Do not use the measures until
• Comfortable with data
• Metrics definitions are clear
• All lower level management has reviewed and
  signed off
• NO SURPRISES UPON OPENING
• Clear definitions and documentation of data
  sources.

25
Guiding Light 10 Train your organization

• May seem clear cut to you but you have been
  working on it!
• Train all level of staff on
• What the metrics are definitions and
  calculations
• What they mean
• How they can be used
• Beware of information overload.

26
Guiding Light 11 Constant review and updates

• Constantly review the metrics for usefulness
• Customer satisfaction surveys
• Are goals being measured?

27
Other Things to Remember

• You have one chance to start! DONT BLOW IT!
• You will be seen as the enemy by some in your
  organization.
• STAND BY YOUR ETHICS AND EXPERIENCE.

28
How do I avoid the soap opera?

• Executive management buy in it is a must and
  you may have to be pushy!
• Project team buy in
• Understanding of data available
• Communication!! Communication!! Communication!!

29
Future planning steps

• More statistical analysis
• Add business value data
• More interaction with staff
• More interaction with executive management

COMMUNICATION, COMMUNICATION, COMMUNICATION!!!!
30
Statistician HUMOR Example 2

• A statistician is someone who can have
• his head in an oven and his feet in ice
• and say that on average he feels
• great.

31
The End

• QUESTIONS?
• susan.morgan_at_libertymutual.com