A framework for software measurement

1

• A framework for software measurement

2

• 3.1 Classifying software measures
• 3.2 Determining what to measure

3

Overview

• The framework is based on 3 principles
• Classifying the measures (process, product or
  resource), (internal or external)
• Determining relevant measurement goals (GQM)
• Identifying the level of maturity of your
  organization

4
3.1 Classifying software measures

• Entities and attributes can be classified into 3
  classes
• Processes (collections of software-related
  activities)
• Products (any artifacts, deliverables or
  documents that result from a process activity)
• Resources (entities required by a process
  activity)

5
3.1 Classifying software measures

• Within each class, can distinguish between
• Internal attributes
• External attributes

6
3.1 Classifying software measures

• Table 3.1 (components of soft. measurement)

• Internal
• Size, reuse
• Size, reuse
• Modularity, coupling
• Time, effort
• Number of specification fault found
• Age, price
• Price, size
• Speed,mememory size

External Comprehensbibility Quality Reliability Q
uality, cost-effectiveness Productivity Experie
nce Usability Reliability quality

• Products
• Specification
• Designs
• Code
• Processes
• Constructing specification
• Detailed design
• Resources
• Personnel
• Teams
• Software
• hardware

7
3.1 Classifying software measures

• External attributes are important whether you are
  a manager, a user.
• External attributes are more difficult to measure
  than internal. reliability can be only measured
  after development is complete.
• Using internals to make judgments about externals
  could be misleading but is needed.

8
3.1.1 Processes

• Directly measurable internal process attributes
• Duration
• Effort
• Number of incidents of a specified type
• Can combined with other measures

9
3.1.1 Processes

• Eg 3.1
• During formal testing, can use indirect measure
  of cost --------------------number of
  errors
• As a measure of the average cost of each error

10
3.1.2 Products

• Code and other artifacts
• Internal product attributes
• provide insight into the likely external
  attributes
• Some products are collections of sub products

11
3.1.2 Products

• Importance of internal attributes
• Most SE assume that good internal structure leads
  to good external quality though is rarely
  established.
• Eg high module cohesion and low module coupling
  is assumed to produce reliable and maintainable
  code.

12
3.1.2 Products

• External product attributes
• Depends on both the product behavior and
  environment
• Eg measuring reliability of code - machine
  mode of operational usage.
• Eg understandability of document -experience and
  credentials of reader

13
3.1.3 Resources

• Any input for software production
• Personnel (individual or team)
• Materials
• Tools (software and hardware)
• Methods
• Measure resources to determine
• Magnitude
• Cost
• Quality

14
3.1.3 Resources

• Cost - how cost of inputs affects cost of output
• Productivity
• external resource attribute
• Usually measured as amount of output
  (product measure) -------------------------
  -------------------- effort input
  (process measure)
• Software development is a creative process

15
3.1.3 Resources

• Other attributes of staff experience, age,
  intelligence
• Team attribute Size, structure and
  communication pattern
• Methods OOP etc
• Techniques manual or automated.
• Tool training or experience.
• Help to understand and control the process

16
3.2 Determining what to measure

• No time to measure everything
• GQM (Goal-Question-Metric)
• Process maturity - the more mature your process,
  the more that is visible and therefore measurable.

17
3.2.1 The Goal-Question-Metric paradigm

• A measurement program can be more successful if
  designed with the goals in mind.
• GWM (Basil and Weiss, Basili and Rombach)
  approach provides a framework with 3 steps
• List the major goals of the development/maintenanc
  e project
• Derive from each goal the questions that must be
  answered to determine if the goals are being met
• Decide what must be measured to answer the
  questions adequately

18
3.2.1 The Goal-Question-Metric paradigm

• Figure 3.2
• Benefits
• generates only those measures relevant to the
  goal
• how to use the resulting data.

19
3.2.1 The Goal-Question-Metric paradigm

• Several measurements may be needed to answer a
  single question and a single measurement may be
  apply to more than one question, Eg 3.8
• GQM must be supplemented by model that express
  the relationships among the metrics.

20
3.2.1 The Goal-Question-Metric paradigm

• templates for goal definition
• Purpose to (characterize, evaluate, predict,
  motivate, etc) the (process, product, mdoel,
  metric, etc) in order to (understand, assess,,
  manage, engineer, learn, improve, etc) it.
• Perspective Examine the (cost, effectiveness,
  correctness, defects, changes, product measures,
  etc) from the viewpoint of the (developer,manager,
  customer, etc)

21
3.2.1 The Goal-Question-Metric paradigm

• Environment The environment consists of the
  following process factors, people factor,
  problem factor, methods, tools, constraints, etc.
• Goal may have subgoals before questions can be
  derived.
• GQM analysis results in a collection of
  measurements related by goal tree and overall
  model.

22
3.2.2 Measurement and process improvement

• Some development processes are more mature as
  noted by SEI.
• 5 levels of process maturity (SEI,figure 3.3)
• Ad hoc (least predictable and controllable)
• Repeatable
• Defined
• Managed
• Optimized (most predictable and controllable)

23
3.2.2 Measurement and process improvement

• Use process maturity as key discriminator
• more visibility -gt higher the maturity, the
  better it can be understood and controlled.
• can measure only what is visible in the process,
  measurement helps to increase visibility
• The five-level maturity scale - used to determine
  what to measure first and how to plan a
  measurement program.

24
3.2.2 Measurement and process improvement

• Table 3.4 (Overview of process maturity and
  measurement)

25
3.2.2 Measurement and process improvement

• Level 1 ad hoc
• Inputs are ill-defined, output are expected,
  transition from input to output is undefined and
  uncontrolled.
• Similar projects vary in productivity and
  quality due to lack of adequate structure and
  control
• Difficult to depict the overall process.
• Visibility is nil and comprehensive measurement
  is difficult.

26
3.2.2 Measurement and process improvement

• Level 1 ad hoc
• Baseline measurements are needed to provide a
  starting point for measuring improvement as
  maturity increases.
• Should impose more structure and control

27
3.2.2 Measurement and process improvement

• Level 2 repeatable
• Able to identify input and output of the process,
  constraints, resource.
• Repeatable in the sense that proper inputs
  produce proper outputs but no visibility how
  outputs are produced.
• Can draw no more than a diagram similar to Figure
  3.4

28
3.2.2 Measurement and process improvement

• Level 3 defined
• Provides visibility into the construct the
  system box.
• Intermediate activities are defined, input and
  output are known and understood
• Input to and from intermediate activities can be
  examined since intermediate products are
  well-defined and visible.
• Fig 3.5 A defined process

29
3.2.2 Measurement and process improvement

• Level 3 defined
• Measurement of product attributes should begin no
  later than level 3.
• Early product measure can be useful indicators of
  later product measures.

30
3.2.2 Measurement and process improvement

• Level 4 managed
• Adds management oversight to a defined process.
• Fig 3.6
• Feedback from early project activities used to
  set priorities for current activities
• Effects of changes in one activity can be tracked
  in the others.

31
3.2.2 Measurement and process improvement

• Level 4 managed
• The basic activities do not change
• can evaluate the effectiveness of process
  activities
• Significant difference between levels 3 and 4 -
  level 4 measurement reflects characteristics of
  the overall process and of the interaction
  between major activities.

32
3.2.2 Measurement and process improvement

• Level 5 optimizing
• Fig 3.7
• Measures from activities are used to improve the
  process
• Change process structure dynamically in response
  to measurement feedback
• Results from on going or completed projects -
  refined, different development process for future
  projects
• Spiral model (Boehm,1988) is an eg of such
  process

33
3.2.3 Combining GQM with process maturity

• Eg, page 94, 95
• Use GQM identify the questions to be answered.
• Before decide on the particular measurement,
  determine the process maturity level.
• The more mature the process is, the richer the
  measurement the more mature the process is the
  more mature the measurement
• GQM combined with process maturity had been used
  in designing measurement program

34
Applying the framework

• Look back to several topics under software
  metrics.
• At which processes, products or resources
  attributes are measured.
• See the focus of the measurement prediction or
  assessment

35
Cost and effort estimation

• It focuses on predicting the attributes of cost
  or effort for the dev process.
• The process includes the dev of spec through
  implementation.
• Most of the estimation techniques present a
  model.
• Boehms original, basic COCOMO model E aSb
• E - effort (person months)
• S - size of software system (thousands of
  delivered source statements)
• a and b - depends on type of software system.

36
Cost and effort estimation

• Used during requirement captures.
• Must predict S in order to predict E.
• More than a model, its a prediction system

37
Cost and effort estimation

• Both basic COCOMO and Albrechts function point
  cost-estimation model assert that effort is an
  indirect measure of a single product attribute
  size.
• Predicting size may be just as hard as predicting
  effort, thus this approach can be unsatisfactory.
• In chapter 7, several ways to address these
  problems will be highlighted.

38
Software measurement validation

• Measurement validation
• The process of ensuring that we are measuring
  what we say we are.
• Validation for measurement and prediction are
  different.

39
Software measurement validation

• Measures/measurement systems
• Used to assess an existing entity by numerically
  characterizing one or more of its attributes
• Prediction systems
• Used to predict some attribute of a future
  entity, involving a mathematical model with
  associated prediction procedures.

40
Validating measures

• Refers to the process of ensuring that the
  measure is a proper numerical characterization of
  the claimed attribute by showing that the
  representation condition is satisfied.

41
Validating measures

• Eg 3.18,
• measuring length of program code
• Combine program P1 and P2, the length of both are
  combined m(P1 P2) m(P1) m(P2)
• If program P1 has greater length than P2 then
  m(P1) gt m(P2)
• Can measure length by counting lines of code and
  since this preserves the above relationships then
  lines of code is a valid measure of length.

42
Validating prediction system

• Refers to the process of establishing the
  accuracy of the prediction system by empirical
  means by comparing model performance with known
  data in the given environment (involves
  experimentation and hypothesis-testing)

43
Validating prediction system

• Eg is COCOMO valid for your development project?
  use data that represent that type of project and
  assess the accuracy of COCOMO in predicting
  effort and duration.
• Acceptable degree of accuracy depends on person
  doing the measurement, whether it is
  deterministic or stochastic.
• Prediction system for cost estimation, effort
  estimation, reliability are very stochastic as
  their margin of errors are large.

44
Validating prediction system

• Acceptable range for the prediction system should
  be specified before using the prediction system.

45
Validation in practice

• A measure must be viewed in the context in which
  it will be used.
• If a measurement for assessment is valid, then it
  is valid in the narrow sense or internally valid.
• It is valid in the wide sense if it is both
  internally valid and a component of a valid
  prediction system.

46
Summary

• Attributes are many.
• GQM framework help to identify what to measure
• CMM and GQM visibility also help to determine
  what to measure
• Validating measurement different measurements
  validation