Evaluation of Information Systems GQIM and Ishikawas Tools

1
Evaluation of Information SystemsGQ(I)M and
Ishikawas Tools

• INFO 630
• Glenn Booker

2
Why Care About Measurement?

• To quote Albert Einstein, "Not everything that
  counts can be counted and not everything that can
  be counted counts."
• We are seeking to identify things 1) which can be
  counted, and 2) which count toward achieving our
  goals

3
Reasons for Measurement

• Measurements are required by all major process
  models (CMM, ISO 9000, etc.) for four good
  reasons
• To Characterize or understand the current status
  of activities or products
• To compare that understanding to our objectives,
  and Evaluate whether the current status is good
  or bad

4
Reasons for Measurement

• To Predict future performance, based on past
  trends
• To form the basis for measuring Improvement
• You wont know if you improved if you dont know
  where you started!

5
Where do we get Information?

• Metrics are often used to support decision making
  (the Evaluate step in the previous slide)
• Decisions should be based on quantified
  information
• To get that information, we calculate measures
  from raw data called data elements
• The data elements each come from a data source

6
How do we Choose What to Measure?

• A commonly used method for selecting measurements
  is called GQ(I)M for Goal, Question, Indicator,
  and Measurement
• It is based on GQM work by Victor Basili (first
  reported circa 1988-89)

7
How do we Choose What to Measure?

• The GQ(I)M method uses ten steps to describe
  measurements systematically
• The steps dont have to be followed in the order
  presented their main purpose is to help ensure
  that measurements have been fully thought out, so
  you can start at the top, or the bottom, or the
  middle

8
1. Identify Business Goals

• These are big, vague, lofty desired
  accomplishments or objectives for the
  organization
• Reduce cycle time
• Improve customer satisfaction
• Develop detailed process history
• Respond to changing business environment

9
1. Identify Business Goals

• Reduce overhead
• Improve competitive position
• Increase market share
• Improve product quality
• Think of what youd find cited in a companys
  annual report those are often high level
  business goals

10
2. Identify Desired Knowledge

• Break down each goal into products, resources,
  and activities (processes) needed to meet that
  goal
• Think of questions like
• What activities do I manage or execute?
• What do I want to achieve or improve?
• To meet this, I will need to

11
3. Identify Subgoals

• Set Subgoals (objectives) for each of the
  entities you manage
• What do you want to know about the results of
  step 2? What kind of information is important to
  you?
• How big, fast, expensive, complex, or much time
  will a process, product, tool, or resource take?

12
4. Identify Entities and Attributes

• Formulate questions to identify entities
  (document, product) created by your process, and
  the attributes of them you are interested in
  (size, quality, duration, cost)
• Entities in this sense are often part of the
  inputs, outputs, or process associated with an
  activity

13
4. Identify Entities and Attributes

• Dont get too detailed at this point - just
  identify the type of information desired (what
  are you studying), and the subject of that
  information (what about it do you want to know)
• Then once the entities have been described, find
  the particular characteristics of interest
  (attributes)

14
4. Identify Entities and Attributes
15
5. Define Measurement Goals

• Form structured statements of the measurement
  goals for each attribute
• This step is the heart of defining a metric in
  the form of a sentence
• Two types of goals
• Active goals reduce or improve something
• Passive goals identify, assess, understand

16
5. Define Measurement Goals

• Lower maturity organizations start with passive
  goals, then work on active goals
• First measure existing trends, before predicting
  improvements
• Description of a measurement goal includes the
  target entity, a purpose, a perspective, the
  environment and constraints

17
5. Define Measurement Goals

• It should include the quantity to be measured,
  the active or passive verb, describe the
  independent variable (e.g. time), and for active
  goals, quantify a desired amount or level
• DO NOT report traits of individual people (fear
  of judgment) unless its a known part of their
  job

18
5. Define Measurement Goals

• Also need to balance how often measurements are
  made
• More frequent measurement gives finer control,
  but excessive measurement wastes time and slows
  the process being measured
• Can measure per release, per component, or some
  basis other than time

19
5. Define Measurement Goals

• Examples of measurement goals
• Passive Measure the number of requirements
  which changed each month.Identify the
  voluntary turnover rate per month for programmers
  and software engineers.
• Active Reduce the defect rate of developed code
  over time to under 20 defects/KLOCImprove the
  percent of satisfied customers after 30 days of
  product use to 95 or more

20
5. Define Measurement Goals

• The general format is ltverbgt ltmeasuregt
  ltqualifier(s)gt objectivewhere the
  ltqualifiersgt indicate the scope or time frame of
  the measurements (a.k.a. independent variables),
  and objective is only given for active
  measurement goals

21
6. Quantify Questions and Indicators

• Pose questions to address your measurement goals
  (quantifiable ones, if addressing active goals)
• Active Can we resolve customer emergencies, on
  average, in under 24 hours?
• Passive How many requirements do we have at
  the end of the Requirements Definition phase?

22
6. Quantify Questions and Indicators

• Sometimes the search for a meaningful metric
  starts with a question, and that leads to filling
  out the GQ(I)M from the middle
• Identify indicators to show the results
  effectively, such as
• Pie charts
• Bar graphs
• Scatter plots, etc. (see later examples)

23
7. Identify Data Elements

• Identify the data elements and equation(s) needed
  to prepare (calculate) each indicator
• E.g. defect rate by module each month needs a
  table of defects found, for the last month, with
  the module each came from
• Determine the source for each data element

24
8. Define Measures

• Describe how the data elements will be used to
  produce the indicators (i.e. what is the equation
  for the measure)
• Even the most obvious measure could be defined
  many ways, so state the definition of each
  measure clearly
• E.g. does your turnover rate include all
  project employees, or only salaried ones?

25
8. Define Measures

• Define exactly what you mean by each measure -
  use a checklist if needed to show what is and
  isnt included in its definition
• Include rules, assumptions, constraints, and
  environment
• Cite source if an unusual measure is used or if
  you made it up, explain why

26
An Aside

• The first eight steps of the GQ(I)M approach
  define traceability from the business goals to
  the exact definition of each measure, and the way
  it is calculated from its data elements
• The last two steps focus on the broader issue of
  planning a measurement approach for a project

27
9. Identify Measurement Implementation

• Analyze what measures are currently collected (if
  anything) and how theyre being used
• Diagnose how well the current measurements meet
  your goals ask whats missing?
• Act on implementing new measurements, possibly in
  a phased approach based on priorities

28
10. Prepare Measurement Plan

• Take all of the aforementioned information and
  create a complete plan to identify and implement
  measurement for your organization or project
• This is generally called a Metrics Plan or a
  Measurement Plan

29
Summary of Core Steps

• Goal (the big picture this is describing)
• Subgoal (objective why collect this metric)
• Question(s) (answered by this metric)
• Indicator (how display metric)
• Measurement (the actual metric and its
  definition)
• Data Elements (used to calculate various metrics)
• Source (of each data element)

30
Indicators

• Indicators are the means used to present
  measures, such as charts, graphs, etc.
• (ok, indicator is an odd term for it, but
  pretend it makes sense)

31
Indicators

• To choose a good indicator, consider
• The Amount of Data to be presented for each
  interval (e.g. one measure at a time, or five
  different survey responses at once), and
• The Number of Intervals to be shown, such as time
  units, modules of code, etc.

32
Indicators

• Different indicators are better at different
  Amount or Number characteristics
• Consider also how your data will be presented -
  in color or B/W, live or printed?
• Will your audience see pristine originals, or
  will it be copied and faxed a zillion times?

33
Indicators

• For graphs
• The X-axis of a graph (the horizontal line) is
  the independent variable
• Is often a ltqualifiergt you choose before the
  measurement, such as time, severity, etc.
• If X is time, describe how often measurements
  are made (weekly, monthly, every release, etc.)
• The Y-axis of a graph (the vertical line) is the
  dependent variable the thing you are measuring
  (the Measure)

34
Pie Chart

• The lowly pie chart is good for presenting a
  limited amount of information attractively
• of customers satisfied and not satisfied
• of defects by severity at this moment
• Amount of Data Shows a few data points (2-10)
• Number of Intervals One - it generally shows
  only one moment in time, or one set of
  measurements

35
Ishikawas Seven Basic Tools

• Developed circa 1950 for manufacturing production
• Used widely in manufacturing quality control
• Focuses on project level concerns is this batch
  good enough to accept?
• Not very useful for research has little
  theoretical basis

36
Ishikawas Seven Basic Tools

• From a software perspective, these tools are
  often best for managers focuses on identifying
  process and/or quality control issues
• Not typically helpful for individual developers

37
1. Check sheet

• Used to gather data easily, consistently, and in
  a standard format
• Helps to define key parts of a process, and make
  sure they are all performed
• Examples include code inspection checklist,
  detailed test procedures
• A check sheet used to help the quality of a
  process or product is a checklist

38
2. Pareto Diagram

• Used to identify problem areas - where to fix
  first what are the biggest fires to put out
• Defects tend to cluster in portions of code e.g.
  plot of defects by the type of defect (logic,
  data definitions, etc.), and plot the total of
  defects above it
• Must list categories in descending order of
  frequency

Helps look for the few components where most
defects reside X axis must be a nominal variable
39
3. Histogram

• A bar chart is used to break down data by an
  ordered category (e.g. defect severity,
  satisfaction rating, etc.)
• Can choose to put bars next to each other, or
  stack them. Stack when they add to a constant
  (e.g. 100), or when the total is also a useful
  measure
• Can show limited time spans, e.g. a few time
  intervals

40
3. Histogram
Stacked and cluster bar graphs for the same data
set
41
3. Histogram

• A true histogram uses ranges of X axis values to
  show, for example, the number of events or data
  points in that range of values

42
4. Run Charts

• Purpose is to plot something important versus
  time
• Often compare values to a desired or target
  value, especially at higher process maturity
  levels (CMM Level 3 and up) and for comparing to
  active goals
• Special case The S curve plots (cumulative
  completion of something) versus time

Usually uses a line chart
43
5. Scatter Diagram

• Is used to plot two measures against each other
  to see if theres a correlation between them
• E.g. Defect rate per module versus module
  complexity, or productivity versus experience
• When we say plot Blah versus Ick, Blah is the Y
  axis, and Ick is the X axis

44
5. Scatter Diagram

• If theres enough data, can try to add curve
  fitted lines well cover this in the regression
  analysis discussion
• This and control charts are the most powerful
  types of graphs for understanding your data
• Many estimation rules and equations are derived
  from scatter diagrams

45
6. Control Chart

• Is a key Statistical Process Control tool
• Hard to apply to software development
• Specifications poorly defined
• Each project takes a long time
• Too many uncontrolled process variances
• Process-quality relationship not well defined
• Too many processes used
• Rapidly changing technology
• There are many varieties of control chart

46
6. Control Chart

• Control charts are often used at very high levels
  of process maturity (CMMI levels 4 5)

47
6. Control Chart

• Pseudo-control charts include
• The u chart for defect rates by component, BMI,
  etc. versus time
• The p chart for percentages, such as inspection
  effectiveness or customer satisfaction rating
• Upper and Lower Control Limits (UCL and LCL) are
  /- 3 sigma from the mean (average)
• Can add a warning limit at /- 2 sigma

48
7. Cause and Effect Diagram

• A.k.a. the Fishbone chart
• Is the least used Ishikawa tool, in my experience
  
• Is not an Indicator per se, just a tool for
  capturing thoughts about a problem
• Used with brainstorming to trace the causes of
  some outcome or result (good or bad)
• Ask what causes that or what influences that
  to determine the major types of causes, then
  break them into more detailed events

49
7. Cause and Effect Diagram

• Sample fishbone diagram to analyze causes of
  Incorrect Deliveries

Note the choices of types of causes
(communication, skills, transport, procedures)
usually vary from one problem to the next
50
Relations Diagram

• The Relations Diagram (example on page 155 of
  Kan) tries to examine interaction of causes for a
  problem
• Handles complex causal interaction better than
  the fishbone diagram
• Also serves to help organize thoughts about a
  problem
• Use it to identify causes or interactions
  previously ignored