Measurement of product development

1

• Measurement of product development

2

• 2.1 The representational theory of measurement
• 2.2 Measurement and models
• 2.3 Measurement scales and scale types
• 2.4 Meaningfulness in measurement

3

• We use measurement everyday to understand,
  control and improve what we do and how we do it.
• Measurement devices and techniques developed over
  time based on our improved understanding. Eg
  using mercury to measure temperature.

4

• For software attributes, no comparably deep
  understanding and complex measurement tools.

5
2.1 The representational Theory of Measurement

• Any measurement activity, rules had to be
  followed, that helps us to be consistent in our
  measurement, and provides basis for interpreting
  data.
• Measurement theory depicts the rules.
• Can use rules about measurement to codify our
  initial understanding and then expand our horizon
  as we analyze our software.

6
2.1 Empirical relations

• Data obtained as measures should represent
  attributes of the entities we observe, and
  manipulation of the data should preserve the
  relationships observed among the entities.
• In real world, we tend to understand things by
  comparing them Eg Figure 2.1, page 25.

7
2.1.1 Empirical relations

• Define a binary relation (taller than) on pairs
  of people.
• Taller than is an empirical relation for
  height.
• A (binary) empirical relation is one for which
  there is a reasonable consensus about which pairs
  are in the relation.
• Can define more than one empirical relation on
  the same set, for e.g much taller than

8
2.1.1 Empirical relations

• unary empirical relation, for eg tall.
• Ternary empirical relation, for eg Peter
  sitting on Wondermans shoulders is higher than
  Frankie
• These relations maps from the empirical, real
  world to a formal mathematical world.

9
2.1.1 Empirical relations

• Eg Height (tallness) can be considered as
  mapping from the set of people to the set of real
  numbers.
• If Jack is taller than Jill then any measure of
  height should assign a higher number to Jack than
  to Jill.

10
2.1.1 Empirical relations

• Eg 2.1 can use the survey to define an empirical
  relation greater functionality than
• Program x has greater functionality than program
  y if the survey result for cell (x, y) exceeds
  60.
• thus the relation consists of the pairs (C, A),
  (A, B), (C, B), (C, D), (A, D)
• The empirical relation greater
  user-friendliness is empty.
• This eg shows using simple surveys to gain
  understanding.

11
2.1.1 Empirical relations

• Eg 2.2 Historical advances in temperature
  measurement.
• This eg shows an important characteristic of
  measurement
• begin understanding the world using relatively
  unsophisticated relationships that require no
  measuring tool.
• Once have initial understanding and gather some
  data and analyze, re-evaluate attribute, leads to
  more sophisticated empirical relations
• Improved accuracy and increased understanding.

12
2.1.1 Empirical relations

• Formal definition of measurement a mapping from
  the empirical world to the formal, relational
  world. Thus, a measure is the number or symbol
  assigned to an entity by this mapping in order to
  characterize an attribute.
• Sometimes empirical relations are not agreed upon
  especially when they reflect personal preference.

13
2.1.1 Empirical relations

• Eg ratings preference for design techniques.
• Can still perform subjective measurement but the
  result is not really a measure.
• Figure 2.2 (Subjective rating schemes)
• Capture useful data helps to establish the basis
  for empirical relations, characterizing
  properties so that formal measurement may be
  possible in the future.

14
2.1.2 The rules of the mapping

• Measure is used to characterize an attribute.
• Real world is the domain of the mapping and the
  mathematical world is the range.
• Map the attribute to a mathematical system, have
  many choices for the mapping and the range.
• Can use real numbers, integers and even a set of
  non-numeric symbols.

15
2.1.2 The rules of the mapping

• Eg 2.3 Measuring height
• Measure in inches - defining a mapping from the
  set of people into inches
• Measure in cm - different mapping
• Even when domain and range are the same, the
  mapping definition may be different.
• Eg measuring height without shoes, measuring
  people standing or sitting.
• Thus a measure must specify the domain and range
  as well as the rule for performing the mapping.

16
2.1.2 The rules of the mapping

• Mapping rules must be made explicit
• Measure size of the source code in terms of the
  number of lines of code in a program.
• Thus, the definition of a line of code must be
  made clear.
• The US Software Engineering has developed a
  checklist to decide exactly what is included in a
  line of code, Figure 2.3.

17
2.1.3 The representation condition of measurement

• Representation condition
• The rule to ensure that the mapping preserve the
  relation (behaviour of the measures in the number
  system to be the same as the corresponding
  elements in the real world.)
• It asserts that a measurement mapping M must map
  entities into numbers and empirical relations
  into numerical relations in such a way that the
  empirical relations preserve and are preserved by
  the numerical relations.

18
2.1.3 The representation condition of measurement

• Representation condition
• Eg Figure 2.4, empirical relation taller than
  is mapped to the numerical relation gt.
• A is taller than B if and only if M(A) gt M(B)
• This statement implies that
• Whenever Joe is taller than Fred, then M(Joe)
  must be a bigger number than M(Fred).
• Jill can be mapped to a higher number than Jack
  only if Jill is taller than Jack.

19
2.1.3 The representation condition of measurement

• Eg 2.5, measurement of height
• For the binary empirical relation taller than,
  we can have the numerical relation x gt y.
• Representation condition requires that for any
  measure M,A is taller than B if and only if M(A)
  gt M(B)

20
2.1.3 The representation condition of measurement

• Eg 2.5, measurement of height
• For the unary empirical relation is tall, we
  might have the numerical relation x gt 70.
• Representation condition requires that for any
  measure M,A is tall if and only if M(A) gt 70
• For the binary empirical relation much taller
  than,we might have the numerical relation x gt y
  15
• Representation condition requires that for any
  measure M,A is much taller than B iff M(A) gt
  M(B) 15

21
2.1.3 The representation condition of measurement

• Eg 2.5, measurement of height
• Consider the actual assignment of numbers M given
  in figure 2.5
• M(Wonderman) 72
• M(Frankie) 84
• M(Peter) 42

22
2.1.3 The representation condition of measurement

• Eg 2.5, measurement of height
• With this particular mapping M, the numerical
  relations hold whenever the empirical relations
  hold
• Frankie is taller than Wonderman, M(Frankie) 84gt
  M(Wonderman)72
• Wonderman is tall, and M(Wonderman) 72 gt 70

23
2.1.3 The representation condition of measurement

• Eg 2.5, measurement of height
• Frankie is much taller than Peter, and M(Frankie)
  84 gt 57 M(Peter) 15, similarly Wonderman is
  much taller than Peter M(Wonderman) 72 gt 57
  M(Peter) 15.
• Etc.
• Since all relations are preserved in this way by
  the mapping, we can define the mapping as a
  measure for the attribute. Thus if we think of
  the measure as a measure of height, we can say
  that Frankies height is 84, Peters is 42 and
  Wondermans is 72.

24
2.1.3 The representation condition of measurement

• Not every assignment satisfies the representation
  condition. Eg M(Wonderman) 72, M(Frankie)
  84, M(Peter) 60 does not satisfied much taller
  than relation.
• The mapping that is called a measure is also
  known as representation or homomorphism because
  the measure represents the attribute in the
  numerical world.
• Figure 2.6 Key stages of formal measurement.

25
2.1.3 The representation condition of measurement

• Several conclusions that can be made
• There may be different measure for a given
  attribute. Any measure that satisfies the
  representation condition is a valid measure.
• The richer the empirical relation system, the
  fewer the valid measures.
• Rich if it has a large number of relations that
  can be defined.
• Number of empirical relations increase, the
  number of conditions that a measurement mapping
  must satisfy in the representation condition also
  increase.

26
2.1.3 The representation condition of measurement

• Eg 2.6 study the entity software failures and
  we look at the attribute criticality
• 3 types of failures delayed response, incorrect
  output, data loss.
• Every failure lies in exactly one failure class
  (based on which outcome happens first)

27
2.1.3 The representation condition of measurement

• Empirical relation system consists of
• R1 for delayed response
• R2 for incorrect output
• R3 for data loss
• At this point cannot judge the relative
  criticality of these failure types.
• Choose any 3 distinct numbers and then map
  members from different classes into different
  numbers, Fig 2.7.

28
2.1.3 The representation condition of measurement

• M(each delayed response) 6
• M(each incorrect output) 4
• M(each data loss) 69
• This assignment is a representation because we
  have numerical relations corresponding to R1, R2
  and R3.
• With better understanding, add a new binary
  relation is more critical than data loss more
  critical than incorrect output and delayed
  response, incorrect output more critical than
  delayed response

29
2.1.3 The representation condition of measurement

• x more critical than y contains all those pairs
  (x, y) of failures for which either x is in R3
  and y is in R2 or R1, orx is in R2 and y is in
  R1
• To find a representation in the real numbers for
  this enriched empirical relation system, the
  assignment of numbers had to be more careful
• Also require a numerical relation to correspond
  to more critical than, can use gt

30
2.1.3 The representation condition of measurement

• Table 2.3 examples of specific measures used in
  SE.

31
2.2 Measurement and Models

• A model is an abstraction of reality, where
  detail can be stripped and can be viewed from a
  particular perspective.
• Different forms equations, mappings, diagrams
  (shows how component parts relate to one another)
• Representation condition requires every measures
  to be associated with a model of how the measure
  maps the entities and attributes in the real
  world to the elements of a numerical system.

32
2.2 Measurement and Models

• Model important in
• understanding how measure is derived
• how to interpret behavior of numerical system
  when return to the real world
• Even before begin the measurement process.
• Measuring height, must specify whether shoes to
  be worn, include hair height, certain posture.
• In this sense, actually define a model of a
  person (rather than the person itself) as the
  entity to be measured.

33
2.2 Measurement and Models

• Model of the mapping supplemented with a model of
  the mappings domain.
• Eg 2.7 Measure length of programs using lines of
  code, need a model of a program.
• Process measures are often more difficult than
  product and resource measures because they are
  less understood. Eg 2.8, measuring the
  attributes of the testing process.

34
2.2.1 Defining attributes

• Eg 2.9 Complexity of a program computed using
  cyclomatic number v, proposed by McCabe (Fig
  2.9). This number counts the number of linearly
  independent paths through a program. But it only
  present a partial view of complexity.

35
2.2.2 Direct and indirect measurement

• Direct measurement of an attribute of an entity
  involves no other attribute or entity.
• Eg length of object.
• Indirect measurement
• density of an object can be measured indirectly
  in terms of mass and volume density mass /
  volume.
• Speed distance / time

36
2.2.2 Direct and indirect measurement

• Direct measurement forms the building blocks for
  assessment but many interesting attributes are
  best measured by indirect measurement.
• Direct measurement commonly used in SE
• Length of source code (measured by lines of code)
• Duration of testing process (measured by elapsed
  time in hours)
• Number of defects discovered during testing
  process (measured by counting defects)

37
2.2.2 Direct and indirect measurement

• Time a programmer spends on a project (measured
  by months worked)
• Indirect measures commonly used in SE Table 2.4
• The most common and most controversial is
  programmers productivity (emphasize size of
  output without considering the codes
  functionality and complexity)

38
2.2.2 Direct and indirect measurement

• Indirect measurement is useful in making visible
  the interactions between direct measurement. Eg
  Fig 2.10 and 2.11
• The representational theory of measurement
  described earlier is concerned with direct
  measurement.

39
2.2.2 Measurement for prediction

• Measuring can also be done for prediction.
• Measurement for prediction always involves models
  that show relationships between attributes
• Eg final system is reliable or not, build a
  model of the factors that affect reliability

40
2.2.2 Measurement for prediction

• Eg 2.11 Effort prediction using E aSb , where S
  is the size of system in KLOC, a and b are
  constants.
• distinction between measurement for assessment
  and prediction may not be clear
• Same model can be used for assessment and
  prediction

41
2.2.2 Measurement for prediction

• Need a prediction system rather than a model to
  do prediction need the model parameters and
  procedure to interpret the results

42
2.3 Measurement scales and scale types

• Use of different measurement mappings will affect
  the analysis that can be done.
• measurement scale determines which analyses are
  appropriate.

43
2.3 Measurement scales and scale types

• Representation problem - does the empirical
  relation system has a representation in the
  numerical relation system.
• Uniqueness problem - determine which
  representation is the best.

44
2.3 Measurement scales and scale types

• Five major types of measurement scales (in
  increasing richness)
• Nominal
• Ordinal
• Interval
• Ratio
• absolute

45
2.3 Measurement scales and scale types

• A relational system is richer than the other if
  all relations in the second are contained in the
  first.
• The richer the empirical relation system, the
  more restrictive the set of representations and
  the more sophisticated the scale of measurement.
• Admissible transformation mapping from one
  acceptable measure to another

46
2.3 Measurement scales and scale types

• Eg
• length in inches can be converted to feet by
  multiplying 1/12.
• For length, the class of admissible
  transformation is very restrictive, must be in
  the form of M1 aM
• M1 b aM (b !0), not allowed
• M1 aMb (b ! 1), not allowed
• More restrictive the class of admissible
  transformation, the more sophisticated the scale

47
2.3.1 Nominal Scale

• Major characteristics
• consists only of different classes, classify the
  elements.
• no notion of ordering among the classes.
• Any distinct numbering or symbolic representation
  of the classes is an acceptable measure,
• no notion of magnitude associated with the
  numbers or symbols.

48
2.3.1 Nominal Scale

• Eg 2.15, the location of software faults.
• Fault is the entity
• Location is the attribute
• Specification, design, code
• Clear distinction among the classes and every
  fault belongs to exactly one class.
• Any mapping M that assigns the 3 classes to 3
  different numbers satisfies the representation
  condition and is therefore an acceptable measure.
• Refer to page 48, M1(x), M2(x) are acceptable.
• The class of admissible transformation is the set
  of one-to-one mappings.

49
2.3.2 Ordinal Scale

• Key characteristics
• classes that are ordered with respect to the
  attribute.
• Any mapping the preserves the ordering (any
  monotonic function) is acceptable.
• The numbers represent ranking only, so addition,
  subtraction, other arithmetic operations have no
  meaning.
• Classes can be combined as long as the
  combination make sense with respect to the
  ordering.

50
2.3.2 Ordinal Scale

• Eg 2.16
• Entities - software modules
• Attribute - complexity
• Trivial, simple, moderate, complex and
  incomprehensible.
• Implicit order relation of less complex than
• M be must monotonic increasing function.
• Refer to page 49, M1, M2, M3 are valid measure.
  M4 and M5 are not.
• The class of admissible transformation is the set
  of monotonic mappings.

51
2.3.3 Interval Scale

• Major characteristics
• Preserves order
• Preserves differences but not ratios.
• Addition and subtraction are acceptable but not
  the multiplication and division.
• Eg 2.18

52
2.3.3 Interval Scale

• An attribute is measurable on an interval scale,
  and M and M1 are mappings that satisfy the
  representation condition, thenM aM1 b
  (affine transformation)
• The class of admissible transformation is the set
  of affine transformations.
• EgFahrenheit 9/5 Celsius 32

53
2.3.4 Ratio Scale

• Most useful
• Major characteristics
• Preserves ordering, size of intervals between
  entities, and ratio between entities.
• Zero element (represent total lack of the
  attribute)
• Measurement mapping start at zero and increase at
  equal intervals, known as units.
• All arithmetic can be meaningfully applied to the
  classes in the range of the mapping.
• Key feature is the existence of empirical
  relations to capture ratios.

54
2.3.4 Ratio Scale

• Eg 2.20
• Length can be measured in inches, feet, meters
  etc.
• To convert from one measure to another use the
  transformation of the form (ratio
  transformation)M aM1 where a is positive
  scalar
• Eg convert feet to inches I 12 F
• Eg 2.21 length of software code

55
2.3.5 Absolute Scale

• Major characteristics
• Measurement is made simply by counting the number
  of elements in the entity set.
• Attribute take the form number of occurrences of
  x in the entity
• Only one possible measurement mapping, namely the
  actual count
• All arithmetic analysis of the resulting count is
  meaningful.
• For any two measures, M and M1, there is only one
  admissible transformation the identity
  transformation.

56
2.3.5 Absolute Scale

• Eg number of failures observed during
  integration testing, number of people working on
  a software project.
• Uniqueness of the measure
• Eg 2.22
• Eg 2.21 shows that LOC is a ratio-scale measure
  of length of source code.
• Common mistake is to assume it is absolute-scale
  (because obtained by counting)

57
2.3.5 Absolute Scale

• Eg 2.22
• But length of programs cannot be absolute because
  there are many ways to measure it ( KLOC, number
  of characters, number of bytes)
• However LOC is an absolute scale measure of the
  attribute number of lines of code

58

• Table 2.5 Scales of measurement.
• As the scale carry more info, the defining
  classes of admissible transformations become
  increasingly restrictive.

59
2.4 Meaningfulness in measurement

• Analysis is constrained by scale type
• Important question to ask after doing measurement
  can meaningful statements be deduced about the
  entities being measured.

60
2.4 Meaningfulness in measurement

• Meaningfulness and the truth of the statement
  could be different.
• Formal definition of meaningfulness A statement
  involving measurement is meaningful if its truth
  value is invariant of transformation of allowable
  scales.

61
2.4 Meaningfulness in measurement

• Eg 2.23 Examine the transformation to decide on
  meaningfulness
• Consider the statement Fred is twice as tall as
  Jane.
• Implies that the measure are at least on the
  ratio scale.
• It is meaningful because no matter which measure
  of height is used (inches, feet, etc), the truth
  or falsity of the statement remains consistent.

62
2.4 Meaningfulness in measurement

• If M and M1 are different measures of height,
  then M(Fred) 2 M(Jane)and M1(Fred)
  2 M1(Jane)are either both true or false.

63
2.4 Meaningfulness in measurement

• Consider the statement temperature in Tokyo
  today is twice that in London
• Tokyo 40 Celsius, London 20 Celsius, then this
  statement is true.
• However in Fahrenheit scale, Tokyo is 104
  Fahrenheit and London is 68 Fahrenheit.

64
2.4 Meaningfulness in measurement

• Consider the statement The difference in
  temperature between Tokyo and London today is
  twice what it was yesterday.
• Difference is important, interval scale.
• Meaningful because Fahrenheit and Celsius are
  related by affine transformation, F 9/5C 32
• Yesterday, Tokyo 35 Celsius, London 25 Celsius
  (difference of 10)
• Today, Tokyo 40 Celsius, London 20 Celsius
  (difference of 20)

65
2.4 Meaningfulness in measurement

• Yesterday, Tokyo 95 Fahrenheit, London 77
  Fahrenheit (difference of 18)
• Today, Tokyo 104 Fahrenheit, London 68 Fahrenheit
  (difference of 36)

66
2.4 Meaningfulness in measurement

• Consider the statement failure x is twice as
  critical as failure y.
• Refer to table on page 55.
• Suppose y is in class2 and x is in class3, the
  statement is not meaningful because it is true
  under M but false under M1.

67
2.4 Meaningfulness in measurement

• Not easy to determine meaningfulness for new
  measures.

68
2.4.1 Statistical operations on measures

• Distribution of data set
• Measure of central tendency, average (where the
  middle of the set is likely to be)
• Measure of dispersion (how far the data points
  stray from the middle)
• Figure 2.12 Different ways to compute the central
  tendency of a set of numbers
• Measure of dispersion - max and min values,
  variance and standard deviation

69
2.4.1 Statistical operations on measures

• Cannot be used universally.
• For eg nominal and ordinal measures do not
  permit computation of mean, variance and standard
  deviation.
• Eg 2.25,
• mean is not a meaningful measure of central
  tendency for ordinal scale type
• Median is a meaningful measure in this case.

70
2.4.1 Statistical operations on measures

• Mean if meaningful measure for interval and ratio
  scale data.
• Table 2.8 Summary of measurement scales and
  statistics relevant to each.
• Every meaningful statistic of a nominal scale
  type is also meaningful for an ordinal-scale type
  etc.

71
2.4.2 Objective and subjective measures

• Objective measures ensure consistency , different
  people produce the same measures.
• Subjective measures depend on the environment,
  vary with the person measuring.
• However, subjective measures can be useful if its
  imprecision is understood.

72
2.4.2 Objective and subjective measures

• Eg want to measure quality of requirement before
  turning the spec to test team to define test plan
  from them.
• Can use any of the rating scale (Fig 2.2)
• Table 2.9 shows the result
• Though measurement is subjective, it shows that
  interface requirement may have problem.
• The general picture is more important.

73
2.4.3 Measurement in extended number system

• Sometimes cannot measure an attribute directly.
• Have to measure in terms of component attributes
  (sub-attributes).
• Eg
• Wish to assess the quality of the different types
  of transport
• May not know how to measure quality directly, but
  quality involves two significant sub-attributes
  journey time and cost per mile.

74
2.4.3 Measurement in extended number system

• Eg (continue)
• When there are incomparable entities, we have
  partial order as opposed to strict weak order, so
  cannot measure in the set of real numbers, R.
  What is needed is mapping to pairs of real
  numbers, R x R.
• A strict weak order has two properties
• Asymmetric - if (x, y) is in the relation than
  (y, x) is not in the relation
• Negatively transitive - if whenever (x, y) is in
  the relation, then for every z, either (x, z) or
  (z, y) is in the relation.

75
2.4.3 Measurement in extended number system

• Eg (continue)
• Define m(Transport) (Journey time, cost per
  mile)
• Then define actual pairsm(Car) (3,
  1.5)m(Train) (5, 2) etc
• Numerical relation is(x, y) superior to (x1, y1)
  if x lt x1 and y lt y1
• It is partial order because contains incomparable
  pairs, eg (5, 2) is not superior to (3.5, 3.5)
  (3.5, 3.5) is not superior to (5, 2)

76
2.4.3 Measurement in extended number system

• Eg 2.26
• Want to assess the quality of 4 different C
  compiler.
• Quality is defined in terms of two sub-attributes
  speed (average KLOC compiled per second) and
  resource (min Kbytes of RAM required)
• Collected data and summarized in Table 2.11.
• Similarly had to define a measure for quality
  that takes the two sub-attributes.
• ISO 9126, standard to measure software quality
  based on the combination of 6 distinct
  sub-attributes.

77
2.4.3 Measurement in extended number system

• Eg 2.26
• Widely held view that no single real-valued
  number can characterize such a broad attribute as
  quality.
• N-tuples that characterize a set of n
  sub-attributes.
• Same observation can be made for complexity of
  program, Eg 2.27 (cyclomatic number should not be
  used as a comprehensive measure of complexity)

78
2.4.3 Measurement in extended number system

• In spite of theoretical problem, many situation
  required sub-attributes to be combined to impose
  a strict ranking and hence ordinal scale.
• Eg 2.28,
• purchasing program, decision based on collection
  price, reliability and usability.
• If use only for letter-writing, then price a
  heavier weighting than reliability.
• If for critical system, then reliability would
  get heavier weighting.

79
2.4.4 Indirect Measurement and meaningfulness

• In indirect measurement, also need to pay
  attention to issues of scale type and
  meaningfulness.
• Similar to direct measurement

80
2.4.4 Indirect Measurement and meaningfulness

• Admissible transformation is known as rescaling
  which is define as
• Suppose we measure each of n sub-attributes with
  measure Mi.
• Let M be an indirect measure involving components
  M1, M2, Mn.
• M f(M1, M2, Mn)
• M1 is a rescaling of M if there are rescalings
  M11, M21, Mn1 of M1, M2, Mn respectively such
  that M1 f(M11, M21, Mn1 )
• Rescaling in wide sense.

81
2.4.4 Indirect Measurement and meaningfulness

• Rescaling in narrow sense, have to verify M1
  f(M1, M2, Mn)
• If there is a mixture of ratio, interval and
  nominal scale type, then the scale type of M will
  at best be nominal since it is weakest.

82
2.4.4 Indirect Measurement and meaningfulness

• Eg 2.30
• Indirect measure of testing efficiency T is D/E
  where D is the number of defects discovered and E
  is effort in person months.
• D is absolute scale measure and E is on the ratio
  scale. Hence T is a ratio scale measure.
• Acceptable rescaling of T arise from rescalings
  of E into other measures of effort (person days,
  person years)
• Indirect measures proliferate as prediction
  measures too.