Agile Project Management With SCRUM: Theory and Practice

1
Agile Project Management With SCRUM Theory and
Practice
Jeff Sutherland, Ph.D.Chief Technology
Officer jeff.sutherland_at_computer.org http//jeffsu
therland.com
2
The Zen of SCRUM

• So simple, anyone can implement it!
• So easy, all can benefit!
• So subtle, few achieve transcendent performance
• How is it that a project manager does nothing,
  and achieves everything?
• Interlocking principles emerge product like
  jigsaw puzzle.
• Novice removes one piece -- engine never fires on
  all cylinders
• Who can know why?
• ScrumMaster must understand deeply and practice
  rigorously.
• Only then will team members say, This experience
  changed my life!

3
Complex Adaptive Systems (cas)

• Interacting agents respond to stimuli.
• Stimulus-response behavior is defined in terms of
  rules.
• Agents adapt by changing rules as experience
  accumulates.
• Agents aggregate into meta-agents whose behavior
  is emergent.
• How can a collection of dumb things emerge smart
  system behavior?
• Maamar, Zakaria and Sutherland, Jeff (2000)
  Toward Intelligent Business Objects Focusing on
  Techniques to Enhance Business Objects that
  Exhibit Goal-Oriented Behaviors. Communications
  of the ACM 401099-101.

4
Enterprise Systems are cas

• Business entities are examples of complex
  adaptive systems.
• Modification time is on the order of months or
  years, roughly time required to change software.
• Automating business processes renders parts of
  the business in software.
• Business systems have severely constrained rule
  sets, making ideal test bed for cas concepts. 
• Sutherland, Jeff and van den Heuvel, Willem-Jan
  (2002) Developing and integrating enterprise
  components and services Enterprise application
  integration and complex adaptive systems.
  Communications of the ACM 451059-64.

5
Objects Meet Requirements for Evolution

• Variation there is a continuing abundance of
  different elements (class libraries).
• Heredity or replication the elements have the
  capacity to create copies or replicas of
  themselves (inheritance).
• Differential "fitness" the number of copies of
  an element that are created in a given time
  varies, depending on interactions between the
  features of that element and the features of the
  environment in which it persists (reuse).
• Daniel Dennett. Darwin's Dangerous Idea
  Evolution and the Meanings of Life. Simon and
  Schuster, 1995, p. 343.

6
Do Programmers Meet Evolutionary Requirements?

• Algorithms create movement through design space
• Simple minded repetitive procedure
• Incremental change to adjacent designs
• "Cranes" accelerate movement through design space
  
• Sex and genetic engineers
• Evolutionary prototyping and smart people
• Emergent architecture
• Brooks, Fred. No Silver Bullet Essence and
  Accidents of Software Engineering. IEEE Computer,
  Vol. 20, No. 4, April 1987, pp. 10-19.

7
Change is ImperativeWasserman's 7 Factors
Driving Change

• Criticality of time to market
• Shift in computing economics
• Powerful desktop computers
• Extensive networks and the Web
• Growing availability of object technology
• WIMP (windows, icons, menus, pointers)
• Unpredictability of the waterfall model of
  software development
• Tony Wasserman. Toward a Discipline of Software
  Engineering. IEEE Software 13623-31, Nov 1996.

Too many projects fail
8
"Why Are Systems Late, Over Budget, Wrong?""The
Waterfall Methodology!" (Paul Bassett)

• Analysis Paralysis
• static modeling overused
• specs are stale baked
• Design-from-Scratch
• no generic models
• no standard architectures
• Large Project Teams
• User Intermediaries
• No Early Warning Signals
• Bassett, Paul G. Framing Software Reuse Lessons
  from the Real World. Yourdon Press Computing
  Series, 1997.

9
Wicked Problems Righteous SolutionsOut of a
total cost of 37B for the sample set, 75 of
DOD projects failed or were never used, and
only 2 were used without extensive modification.
Jarzombek. The 5th Annual JAWS S3 Proceedings,
1999.

• Wicked problems have no definitive formulation.
  Each attempt at creating a solution changes your
  understanding of the problem.
• Wicked problems have no stopping rule. The
  problem-solving process ends when resources are
  depleted, stakeholders lose interest or political
  realities change.
• Solutions to wicked problems are not
  true-or-false, but good-or-bad getting all
  stakeholders to agree that a resolution is "good
  enough" can be a challenge.
• There is no immediate or ultimate test of a
  solution to a wicked problem.
• Every implemented solution to a wicked problem
  has consequences.
• Wicked problems don't have a well-described set
  of potential solutions. Various stakeholders have
  differing views of acceptable solutions.
• Each wicked problem is essentially unique. Part
  of the art of dealing with wicked problems is the
  art of not knowing too early what type of
  solution to apply.
• Each wicked problem can be considered a symptom
  of another problem. A wicked problem is a set of
  interlocking issues and constraints that change
  over time, embedded in a dynamic social context.
• The causes of a wicked problem can be explained
  in numerous ways.
• The planner (designer) has no right to be wrong.
• Rittel, H and Webber M. Dilemmas in a General
  Theory of Planning. Policy Sciences, Vol. 4.
  Elsevier, 1973.
• Degrace and Hulet's book, Wicked Problems,
  Righteous Solutions, Prentice Hall, 1990

10
Software Development is an Empirical Process

• Ziv's Uncertainty Principle in Software
  Engineering - uncertainty is inherent and
  inevitable in software development processes and
  products Ziv, 1996.
• Humphrey's Requirements Uncertainty Principle -
  for a new software system, the requirements will
  not be completely known until after the users
  have used it.
• Wegner's Lemma - it is not possible to completely
  specify an interactive system Wegner, 1995.

11
Productivity All at Once Models

• Single Super-Programmer
• Handcuffing two programmers together
• Brooks Surgical Team
• Borland Quattro project
• Goldratts The Goal
• Senges systems thinking
• Hollands complex adaptive systems

12
Team Size Development Effort in Months

• The smaller the better.
• 491 medium sized projects with 35,000-95,000 SLOC
  (source lines of code)
• Putnam, Lawrence H. and Myers, Ware. Familiar
  Metrics Management Small is Beautiful--Once
  Again. IT Metrics Strategis IV812-16, Cutter
  Information Corp., August 1998.

13
Team Size Development Time in Months

• Sweet spot is 5-7 people
• 491 medium sized projects with 35,000-95,000 SLOC
  (source lines of code)
• Putnam, Lawrence H. and Myers, Ware. Familiar
  Metrics Management Small is Beautiful--Once
  Again. IT Metrics Strategis IV812-16, Cutter
  Information Corp., August 1998.

14
Bell Labs Report on most productive project ever
Borland Quattro for Windows
1,000,000 lines of C code BWP Industry standard
Time in months 31 gt50
Staff 8 gt100
Function points per staff month 77 2
Jones, Capers. Applied Software Measurement,
Second Edition. McGraw Hill, 1997.

• BQW

15
James Coplien. Borland Software Craftsmanship A
New Look at Process, Quality, and Productivity.
Proceedings of the 5th Annual Borland
International Conference, Orlando, 1994.

• One of most remarkable organizations, processes,
  and development cultures seen in ATT Bell
  Laboratories Pasteur process research project
• Project management, product management, QA
  integral to team, all making technical
  contributions
• Higher communication saturation than 89 of
  projects
• More even distribution of workload
• Anti-schismogenetic no cliques
• Highly iterative development
• Strong architectural interaction with
  implementation
• More time spent in project team meetings than
  anything else several hours a day
• Gerry Weinberg notes that CMM Level 1 and 2 teams
  need strong managerial direction. Level 3
  paradigm shift is self-directing team. Borland
  team was clearly in this category, although not
  by commonly accepted criteria.

16
Team comments on Quattro project

• We are satisfied by doing real work.
• Software is like a plant that grows. You cant
  predict its exact shape, or how big it will
  grow.
• There are no rules for this kind of thingits
  never been done before.
• Evolutionary development is best technically,
  and it saves time and money.
• Report of the Defense Science Board Task Force
  on Military Software. Oct 1987.

17
History of Iterative and Incremental Development
(IID)

• 1956 Beningtons stagewise model USAF SAGE
  System
• 1957 IBM Service Bureau Corp, Project Mercury,
  IBM Federal Systems Devision Gerry Weinberg
• 1960 Weinberg teaching IID at IBM Systems
  Research Institute
• 1969 - Earliest published reference to IID
• Robert Glass. Elementary Level Discussion of
  Compiler/Interpreter Writing. ACM Computing
  Surveys, Mar 1969
• Larman, Craig and Basili, Vic. A History of
  Iterative and Incremental Development. IEEE
  Computer, June 2003 (in press)

18
History of Iterative and Incremental Development
(IID)

• 1971 IBM Federal Systems Division
• Mills, Harlan. Top-down programming in Large
  Systems. In Debugging Techniques in Large
  Systems. Prentice Hall, 1971
• 1972 TRW uses IID on 100M Army Site Defense
  software
• 1975 First original paper devoted to IID
• Gasili, Vic and Turner, Albert. Iterative
  Enhancement A Practical Technique for Software
  Development. IEEE Transactions on Software
  Engineering. Dec 1975.
• 1977-1980 IBM FSD builds NASA Space Shuttle
  software in 17 iterations over 31 months,
  averaging 8 weeks per iteration
• Madden and Rone. Design, Development,
  Integration Space Shuttle Flight Software.
  Communications of the ACM, Sept 1984.
• Larman, Craig and Basili, Vic. A History of
  Iterative and Incremental Development. IEEE
  Computer, June 2003 (in press)

19
History of Iterative and Incremental Development
(IID)

• 1985 Barry Boehms Spiral Model
• Boehm, Barry. A Spiral Model of Software
  Development and Enhancement. Proceedings of an
  International Workshop on Software Process and
  Software Environments. March, 1985
• 1986 Brooks, Fred. No Silver Bullet. IEEE
  Computer, April 1987
• Nothing has so radically changed my own
  practice, or its effectiveness as incremental
  development.
• 1994 First SCRUM at Easel Corporation
• 1994 DOD must manage programs using iterative
  development
• Report of the Defense Science Board Task Force on
  Acquiring Defense Software Commercially. June
  1994.
• 1995 Microsoft IID published
• McCarthy, Jim. Dynamics of Software Development.
  Microsoft Press, 1995.
• 1996 Kruchten. A Rational Development Process.
  Crosstalk. July.
• Origins of RUP
• Larman, Craig and Basili, Vic. A History of
  Iterative and Incremental Development. IEEE
  Computer, June 2003 (in press)

20
History of Iterative and Incremental Development
(IID)

• 1996 Kent Beck Chrysler Project
• Origin of XP
• 1996 Larman meets with principal author of
  DD-STD-2167
• David Maibor expressed regret for the creation of
  the waterfall-based standard. He had not learned
  of incremental development at the time and based
  his advice on textbooks and consultants
  advocating the waterfall method. With the
  hindsight of iterative experience, he would
  recommend IID.
• 1997 Coad and DeLuca rescue Singapore project
• Origin of Feature-Driven Development
• 1998 Standish Group CHAOS Project
• Top reason for massive project failures was
  waterfall methods. Research also indicates that
  smaller time frames, with delivery of software
  components early and often, will increase success
  rate.
• 1999 Publication of extensive DOD failures
• Out of a total cost of 37B for the sample set,
  75 of projects failed or were never used, and
  only 2 were used without extensive modification.
  Jarzombek. The 5th Annual JAWS S3 Proceedings,
  1999.
• Larman, Craig and Basili, Vic. A History of
  Iterative and Incremental Development. IEEE
  Computer, June 2003 (in press)

21
History of Iterative and Incremental Development
(IID)

• 2001 17 process expert anarchists meet at
  Snow Bird
• Agile Manifesto initiated 100s of books and
  papers on agile development
• 2001 MacCormacks study of key success factors
• MacCormack, Alan. Product-Develoment Practices
  that Work. MIT Sloan Management Review 422,
  2001.
• Larman, Craig and Basili, Vic. A History of
  Iterative and Incremental Development. IEEE
  Computer, June 2003 (in press)

22
Manifesto for Agile Software Development
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value
Individuals and interactions over processes and
tools Working software over comprehensive
documentation Customer collaboration over
contract negotiation Responding to change over
following a plan That is, while there is value
in the items on the right, we value the items on
the left more.
23
MacCormack Process Evolution

• Waterfall model sequential process maintains a
  document trail
• Rapid-Prototyping Model disposable prototype
  helps establish customer preference
• Spiral Model series of prototypes identifies
  major risks
• Incremental or Staged Delivery Model system is
  delivered to customer in chunks
• Evolutionary Delivery Model iterative approach
  in which customers test an actual version of the
  software
• MacCormack, Alan. Product-Development Practices
  That Work How Internet Companies Build Software.
  MIT Sloan Management Review 42275-84, Winter
  2001.

24
MacCormack Success Factors

• Early release of evolving product design to
  customers.
• Daily incorporation of new software code and
  rapid feedback on design changes
• A team with broad-based experience in shipping
  multiple projects
• Major investment in design of product
  architecture
• MacCormack, Alan. Product-Development Practices
  That Work How Internet Companies Build Software.
  MIT Sloan Management Review 42275-84, Winter
  2001.

25
SCRUM Origins Takeuchi and Nonaka Lessons from
Fuji-Xerox, Canon, Honda, NEC, Epson, Brother,
3M, Xerox, HP

• Old model Relay Race
• Speed and flexibility not adequate in todays
  market
• New model - Rugby

Takeuchi, Hirotaka and Nonaka, Ikujiro. 1986. The
new new product development game. Harvard
Business Review 641137-146 (Jan/Feb), reprint
no. 86116.
26
Moving the SCRUM downfield
27
Takeuchi and Nonaka Success Factors

• Built-in instability
• Self-organizing project teams
• Overlapping development phases
• Multilearning
• Subtle control
• Organizational transfer of learning
• These characteristics are like pieces of a
  jigsaw puzzle. Each element, by itself, does not
  bring about speed and flexibility. But taken as a
  whole, the characteristics can product a powerful
  new set of dynamics that will make a difference.

28
Factor 1 Built-in instability

• Top management kicks off development process by
  signaling broad goal.
• Project team is offered extremely challenging
  goals with wide measure of freedom.
• Example Fuji-Xerox gave FX-3500 project team two
  years to come up with a copier that cut costs in
  half
• Top management creates an element of tension in
  the project team through challenging requirements
  with wide freedom to achieve strategic objective.
• Honda Executive Its like putting the team
  members on the second floor, removing the ladder,
  and telling them to jump or else. I believe
  creativity is born by pushing people against the
  wall and pressuring them almost to the extreme.

29
Factor 2 Self-organizing project teams

• A project team takes on a self-organizing
  character as it is driven to a state of zero
  information where prior knowledge does not
  apply.
• Left to stew, the process begins to create its
  own dynamic order.
• The project team begins to operate like a
  start-up company.
• A group possesses a self-organizing capability
  when it exhibits three conditions.
• Autonomy
• Self-transcendence
• Cross-fertilization
• At some point, the team begins
• to create its own concept.

ScrumDown at the Radcliffe Rugby Club
30
Autonomy

• Headquarters involvement is limited to providing
  guidance, money, and moral support at the outset.
• On a day to day basis, management seldom
  intervenes and the team is free to set its own
  direction.
• In a way, top management acts as a venture
  capitalist
• We open our purse and keep our mouth closed.
• Example IBM development of personal computer
• Example Honda City project team, average age 27,
  Develop the kind of car that the youth segment
  would like to drive.

31
Self-transcendence

• The project teams appear to be absorbed in the
  never-ending quest for the limit.
• They elevate their goals through the development
  process.
• By pursuing what appear to be contradictory
  goals, they devise ways to
• override the status quo and
• make the big discovery.
• Example Canon AE-1 team

Flyhalf Sarah Schooler of Radcliffe fending off
Boston College
32
Cross-fertilization

• Team with wide variety of specializations,
  thought processes, and behavior patterns carries
  out new product development.
• Working in one large room is best (Fuji-Xerox).
• When all team members
• are in one room, others
• information becomes yours
• without even trying.

Radcliffe Rugby Football Club
33
Factor 3 Overlapping Development Phases

• Self-organizing character of the team produces
  unique dynamic or rhythm
• Sashimi system Fuji Xerox Rugby system Honda
• Hard merits (demerits)
• Speed and flexibility (watch out for muck and
  mall)
• Soft merits
• Share responsibility and cooperation
• Stimulates involvement and commitment
• Sharpens a problem-solving focus
• Develops initiative and diversified skills
• Heightens sensitivity to market conditions

34
Factor 4 Multilearning

• Learning by doing in two dimensions
• Across organization
• Across specialty
• Enhanced learning opportunities
• 15 of time devoted to dreams 3M
• Peer pressure to study
• Send team to Europe to look around Honda
• Bring in top academics and consultants HP
• Everyone learns multiple skills

35
Factor 5 Subtle Control

• Management establishes checkpoints
• Prevents instability, ambiguity, and tension from
  turning into chaos
• Emphasis on self-control, control by peer
  pressure, control by love subtle control
• Management responsible for
• Selecting team members for balanced team
• Creating an open working environment
• Encouraging engineers to go out in the field
• Establishing rewards based on group performance
• Tolerating and anticipating mistakes
• Encouraging suppliers to become self-organizing

36
Factor 6 Organizational Transfer of Learning

• Transfer knowledge outside group
• Scatter successful team to new projects
• Institutionalize practice (monthly demos at IDX)
• Consciously pursue unlearning
• Next generation must be 40 better
• Cut product cycle by 80
• Scrap old parts, processes, tools

37
Challenges and Opportunities

• Winding the Rubber Band Principle Broad mandate
  and demanding goals create tension.
• Anti-Waterfall Principle Operational decisions
  are made incrementally. Strategic decisions
  delayed to last moment.
• Push/Pull Principle Differentiation in concept
  phase, integration dominates in implementation
  phase
• Spread the Wealth Principle Non-experts take on
  new tasks.
• Cuckoo Principle Successful SCRUMs become
  company models (or they can get crushed because
  they are different).
• Control Anti-Pattern Seniority based companies
  have difficult time.
• But in times of desperation, SCRUMs are easily
  created.

38
Spiral Methodology

• Barry Boehm introduced the Spiral Methodology to
  "fix" problems with the Waterfall Methodology.
  This is the most commonly used variant of the
  Waterfall today.
• The Spiral methodology "peels the onion",
  progressing through "layers" of the development
  process. A prototype lets users determine if the
  project is on track, should be sent back to prior
  phases, or should be ended. However, the phases
  and phase processes are still linear.
• Boehm, B.W. A Spiral Model of Software
  Development and Enhancement. Proceedings of an
  International Workshop on Software Process and
  Software Environments, Coto de Caza, Trabuco
  Canyon, California, March 27-29, 1985.
• Boehm, Barry. A Spiral Model of Software
  Development and Enhancement.  ACM SIGSOFT
  Software Engineering Notes, August 1986. Boehm,
  Barry. A Spiral Model of Software Development and
  Enhancement.  IEEE Computer, vol.21, 5, May
  1988, pp 61-72.

39
Iterative Methology

• The Iterative methodology improves on the Spiral
  methodology.
• Each iteration consists of all of the standard
  Waterfall phases, but each iteration only
  addresses one subsystem. Further iterations can
  add resources to the project while ramping up the
  speed of delivery.
• Improves cost control, reduces risk, ensures
  delivery of (sub)systems, and improves overall
  flexibility.
• Still assumes that the underlying development
  processes are defined and linear.

40
SCRUM Methodology

• The first and last phases (Planning and Closure)
  consist of defined processes.
• The Sprint phase is an empirical process. It is
  treated as a black box that requires external
  controls.
• Sprints are nonlinear and flexible. Sprints are
  used to evolve the final product.
• The project is open to the environment until the
  Closure phase. The deliverable can be changed at
  any time.
• The deliverable is determined during the project
  based on the environment.

41
Methodology Comparison
42
Risk with Current Methodologies

• Any methodology is better than nothing.
• Current approaches rests on the fallacy that the
  development processes are defined, predictable
  processes.
• They lack flexibility needed to cope with the
  unpredictable results and respond to a complex
  environment.
• Schwaber, Ken. SCRUM Development Process.
  Business Object Design and Implementation (Eds.
  Jeff Sutherland et al.). London Springer-Verlag,
  1997.

43
SCRUM Lowers Risk

• Development teams need to operate adaptively
  within a complex environment using imprecise
  processes.
• SCRUM can accelerate closure by inducing the
  phenomenon known as "punctuated equilibrium" seen
  in the evolution of biological species.
• Levy, Steven. Artificial Life A Report from the
  Frontier Where Computers Meet Biology. Vintage
  Books, 1993.
• Lewin, Roger. Complexity Life at the Edge of
  Chaos. Collier Books, 1994.

44
Agile Project Management With SCRUM Theory and
Practice
Jeff Sutherland, Ph.D.Chief Technology
Officer jeff.sutherland_at_computer.org http//jeffsu
therland.com