Developing Metrics for agile projects compatible with CMMI

1
Developing Metrics for agile projects
compatible with CMMI

• Graham Collins, UCL
• graham.collins_at_ucl.ac.uk

Research supported by Deutsche Bank
2
Introduction

• UCLs research and projects
• The problems and requests
• EV (Earned Value)
• Agile practices
• Combining EV and agile is it possible?
• Developing suitable metrics compatible with CMMI
• What works, and what reduces the overheads
• Is this approach likely to lead to CMMI Level 5?
• Key changes to projects

3
Background - Requests

• We would like to use earned value
• We would like to predict the outcome of project
  end date, cost and value
• As a project manager I cannot be there all the
  time
• What is the best way to measure progress with
  earned value?
• As developers we would like to experiment with
  some agile approaches
• Your team will be working on other projects as
  well
• We are hoping to achieve metrics suitable for
  CMMI level 3 and higher

4
The Death March Project Style Quadrant
The Death March Project Style Quadrant
high
Mission Impossible
Kamikaze
Happiness
Suicide
Ugly
low
low
high
Chance of success
Edward Yourdon, Death MarchThe complete Software
Developers guide to surviving Mission
Impossible projects, 1997 Prentice Hall
5
Earned Value - Definition
The value of the useful work done at any given
point in a project. The value of completed work
expressed in terms of the budget assigned to that
work. A measure of project progress. Note The
budget may be expressed in cost or labour hours
APM (Association of Project Management) BoK
2006
6
Earned Value Graphical Representation
7
Use of Variance
spic is used in this situation
8
Earned Value compared to Agile Process Planning
Earned Value Agile Development
Based on predictive planning Estimates effort, cost and completion date End-to-end value tracking Adaptive planning. Iteration to iteration tracking Predication of the next iterations effort
Schedule most of the activities Adaptation to unpredictable events is problematic. Changes may require the planned to be revised or baselined Near the beginning, it is not always possible to schedule. Time based iterations allow initial estimate of duration which can be revised through the adaptive driven build-feedback cycle
Estimates based on past performance Estimates are based on progress being made (velocity)
Change rates often low Unpredictable change the norm
Small variations in early measurements of cost and time at the start the project give wide variation in forward predications Unknown team development rates
Some organisations do not chart progress for an initial period Progress is tracked immediately
Earned value well established Prioritization of the value of user stories No earned value approaches in methods
9
CMMI- Process Areas
Category Process Area
Project Management Project Planning Project Monitoring and Control Quantitative Project Management
Support Process and Product Quality Assurance Causal Analysis Resolution
Project planning is a necessity for success, Yet
it still ranks on most surveys within the top
three or four problem areas leading to
failure. Tony Ciorra, Planners Progress,
Project, APM May 2006
10
CMMI Comparative Advantages
Continuous Representation Staged
Representation


Grants explicit freedom to select the order of improvement that best meets the organizations business objectives and mitigates the organisations areas of risk Enables organisations to have a predefined path
Enables increased visibility of the capability achieved in each individual process area Focuses on a set of processes that provide an organization with a specific capability that is characterized by each maturity level
Provides a capability-level rating that is used primarily for improvement in an organisation and is rarely communicated externally Provides a maturity-level rating that is often used in internal management communication, statements external to the organization, and during acquisitions as a means to qualify bidders
Allows improvements of different processes to be performed at different rates Summarizes process-improvement results in a simple form a single maturity-level number
Reflects a newer approach that does not yet have the data to demonstrate its ties to return on investment Builds on a relatively long history of use that includes case studies and data that demonstrate proved return on investment
11
Agile Manifesto
Individuals and interactions over
processes and tools Working
software over comprehensive documentation
Customer collaboration over contract
negotiation Responding to
change over following a plan
That is, while there is value in the items on the
right, we value the items on the left more
Several agile projects have achieved CMMI level
3, example David Anderson, Stretching Agile to
fit CMMI Level 3, Agile Conference 2005
12
The Agile Principles www.agilealliance.com
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software Agile processes promote sustainable development
Welcome changing requirements, even late in development. Agile processes harness change for the customer competitive advantage The sponsors, developer, and users should be able to maintain a constant pace indefinitely
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter time scale Continuous attention to technical excellence and good design enhances agility
Business people and developers must work together daily throughout the project Simplicity - the art of maximizing the amount of work done is essential
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. The best architectures, requirements, and designs emerge from self-organizing teams
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation At regular intervals the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly
Working software is the primary measure of progress
13
Iterative Development (Bittner-Spence)

• Agree with the team the objectives for the
  iteration, including evaluation criteria,
  timescales, and constraints
• Agree on a plan for how the team will achieve the
  objectives
• Execute the plan
• Assess the achievements of the team against the
  initial set of objectives and evaluation criteria
• Assess the impact of the iterations results on
  the project as a whole
• Start the next iteration.

What is iterative development? Part 3 The
management perspective 15 May 2005 www-128.ibm.com
/developerworks/rational/library/may05/bittner-spe
nce/index.html
14
Fundamental shift in measurement
15
Developer Perspective

• Developers are less interested in the business
  value, benefits realization and return on
  investment
• They work on a small number of requirements or
  change requests from their list of outstanding
  work
• They anticipate a decreasing number of
  requirements and change requests as the product
  is developed
• Outstanding requirements and change requests is
  termed the product backlog
• The developer will therefore be aware of progress
  via work completed, product backlog and new work
  allocated.

16
User Satisfaction driving Development
17
Project teams need to adopt some attributes
What is the purpose?
What holds it together?
To develop members capabilities to build and
exchange knowledge
Passion, commitment, and identification with the
groups expertise
Community of practice
To accomplish a specified task
The projects milestones and goals
Project team
Adapted from Communities of Practice The
organizational Frontier, Etienne C. Wenger and
William M. Snyder, Harvard Business Review
p139-145 Jan-Feb 2000
18
Rate of work - velocity
Velocity, gives an indication of the average rate
of work and also a comparison of planned against
delivered, each iteration
19
Often high variation
UCL upper control line
X-bar average
LCL lower control line
Velocity from previous chart, showing UCL, X-bar
average, and LCL
20
Under control
Velocity measures of work rate are useful in that
estimates of the next iteration can be planned in
a rolling process, with plans becoming more
accurate. The use of Process Control and sigma
variation is supportive in this aim. This can be
combined with automated colour coding
21
Types of mean
Story points count Weighted points
0 1 0
5 2 10
18 3 54
14 4 56
24 5 120
27 6 162

14.66667 3.5 19.14286
22
Use of Multipliers
Iterations Completed Low Multiplier High Multiplier
1 0.6 1.60
2 0.8 1.25
3 0.85 1.15
4 or more 0.90 1.10
Multipliers for estimating velocity based on
number of iterations completed from Cohn 2006
23
Work Remaining
This burn-down chart showing can be combined
with an inventory line, showing the cumulative
total of points
24
With the appropriate metrics we can improve
25
Additional Measures

• At different levels, project, release, iteration.
  Velocity and stories delivered (as a burndown
  i.e. remaining), as well as on a daily basis,
  showing tasks agreed on and those remaining
• Graphs for stories can also display inventory, to
  show how much additional work has been added.
  Iteration cumulative flow can also be displayed
  as burn-up charts displaying values for
  inventory and complete
• Issue logs should show, active, resolved, closed
  and blocked
• WIP (Work-In-Progress) inventory charts combined
  with the issue log provide capability to remove
  special cause variation.

26
Earned Value

• EV can be applied to these figures, although
  measuring this, can be complex, especially if
  more stories are added as the work progresses
  (e.g. agile projects).
• This however would be useful if the figures have
  to be shown to senior managers who are used to EV
  figures, or used in comparison to other projects
  where EV figures have been tracked
• When additional story points are added to the
  planned work it does not necessarily affect the
  original plan in terms of EV (or business value).
  In light of business priorities the team may
  commit to a new set of stories and drop other
  stories already planned. Consequently the value
  of stories planned within a specific time frame
  can remain the same. How the planned (budget) is
  calculated, whether based on points or business
  value, needs to be decided in advance.

27
Business Value
More importantly business value (or contribution)
should be considered and evaluated Often units
of measure such as story points can be valued as
0.5 or 1.0 units The key issue in agile project
management is to continually assess with the
client the most important work that should be
done.
Story number Business Value Story Points Points earned Developer hours planned EV (earned value)
1 3 10 10 100 100
2 2 8 8 60 60
3 2 4 4 60 60
4 1 2 0 20 0
total 8 24 22 240 220
28
Moving Range
mR values can make variation more visible
29
Conclusions

• EV can be combined with agile reporting
• The most useful approach to achieve process
  improvement is via understanding of process
  control, the basis for CMMI
• EV is useful for reporting at a programme and
  project level, but even with major scope changes,
  re-planning can give some useful insights
• Acceptance tests are the most useful approach for
  both methods and is the basis for metrics
  outlined. Issue reporting and work-in-progress
  are also valuable
• Changes in features, where story points are
  maintained, on time based iterations, may not
  significantly change the basis for EV reporting.
  Projects indicate that EV can remain a viable
  approach, dependent on the business value
  weighting approach used
• The agile process, the value of time based
  iterations, and that the team are trying to
  achieve maximum business value, even though the
  final goal and plans may evolve during the
  process, needs to be explained to senior
  managers.

30
Key Project Changes

1. Peer reporting - valuing individuals and team,
   moving towards self-determining teams
2. Acceptance testing - working software
3. Ensuring Business Value continual
   prioritisation, estimation and understanding
   there is a cost to development
4. Measuring progress over shorter time periods
   -meeting the needs of process improvement CMMI,
   velocity tracking in agile methods, and better EV
   planning

Individuals and interactions over
processes and tools Working
software over comprehensive documentation
Customer collaboration over contract
negotiation Responding to
change over following a plan