There is no magic

1
There is no magic Bob Martin bobmartin08008_at_mac.c
om
I assume you will show the alternative ways to
use a cage, a whip, and a chair to manage
software teams? - Stu Feldman, VP Engineering
Google
2
About Me

• Built a lot of software 100M lines of code
• System software
• Operating systems
• Programming languages
• Transaction control
• Network management
• Service control points
• Embedded systems
• Packet switches
• Network equipment
• Testing equipment
• Love technology
• EE and CS trained
• Life long passion now neural science, molecular
  evolutionary biology
• Help me convince Al to work in computational
  biology!
• Not expert in contemporary software
  technology/tools
• Conned into this talk by a great friend of 42
  years

3
Facts of Life

• Failure is the norm
• But it neednt be
• Error correction costs increase exponentially
  during the lifecycle
• Programmers are a special breed
• They like machines not people
• They have high need for approval/recognition
• They must respect the leader
• Beware, complicators and prima donnas lurk in the
  weeds
• Software engineering is complexity management
• Race with technology/tools
• Complexity is still ahead
• Good software engineering is old hat
• But is not always used - why????
• Two things are infinite the universe and human
  stupidity and I'm not sure about the the
  universe - Einstein

Psychology of Computer Programmer - Gerald
Weinberg
4
Why Software Projects Fail
Average overrun 89.9 on cost, 121 on schedule,
with 61 of content
Barry Boehm - A View of 20th and 21st Century
Software Engineering
5
Increase in Software Cost-to-fix vs. Phase (1976)

1000
1000
500
500
200
200
100
Relative cost to fix defect
100
50
50
20
20


Smaller Software Projects
10
10
5
5
2
2
1
1
Requirements Design Code
Development
Acceptance Operation
Requirements Design Code
Development
Acceptance Operation
test
test
test
test
Phase in which defect was fixed
Barry Boehm - A View of 20th and 21st Century
Software Engineering
6
What Year Was This Conference?
NATO Software Engineering Conference - 1968

• Design
• Guidelines
• External function, e.g., user language
• Internal function, e.g., parsers
• Techniques
• Deductive or inductive
• Modularity and interfaces
• Complexity control
• Proscriptions
• Completeness, modularity, efficiency
• Self-monitoring and performance improvement
• Incremental systems
• Balance
• Security, control, vs. cost
• Limited goals to attain excellent performance
• Design problems
• Data structures on system design
• Fixed resources
• Cooperating processes

• Production
• Organization for producing software
• Number and quality of people
• Structure of large groups
• Control and measurement
• Internal communication
• Production techniques
• Monitoring
• Service
• Distribution
• Media
• Acceptance criteria
• Documentation
• Adaptation
• Maintenance
• Error detection reporting
• Response and distribution
• Documentation
• Performance

7
Impact of Increased Discipline

• Low software engineering maturity results in
• Reduced productivity (35)
• Late detection of defects (22)
• Longer time to market (19)
• Higher post-release defect rates (39)
• median annual improvement

CMM process maturity

• Takes several years of focused improvement to
  become a high-performance software team
• Focus on the programmer
• Just management well-controlled junk
• Delicate touch to avoid process imprisonment

CMM Level
Software Engineering Institutes Capability
Maturity Model (CMM)
8
Learning

• Disasters
• Mine
• Others
• Really good companies
• IBM - Discipline inspections
• Sun - Tempo
• Japan - Quality circles reuse
• Microsoft - Reuse
• Apple - User centered design tool kits, Design
  of Everyday Things - Don Norman
• Really smart professionals
• Al Aho Swamp drainer
• Fred Brooks Mythical Man Month, No Silver
  Bullet
• Barry Boehm Estimation and cost of errors
• Michael Cusumano Software factories
• Edsger Dijkstra Importance of time in systems
• Watts Humphrey - Encapsulated team and personal
  discipline - TSP, PSP
• Martyn Thomas - Formal methods
• Ken Thompson - The programming craft

9
Two Troubled Projects

• Loop Maintenance System
• IBM MVS system
• PDP 11/20 backup
• 8 Months to go
• No schedule
• No performance plan
• No requirements
• Weak management

• Loop Assignment System
• Univac system
• Failed trial
• 250 million down the drain
• Demoralized, weak team
• Senior 18-month commitment
• No architecture or requirements
• No technology infrastructure
• Working temporary subsystem
• Unix based

10
Building The Team

• All - domain expertise, discipline simplifiers
• Team leads programming, lateral motivational
  skills
• Architect deep system software computer
  science
• Great ones are really rare
• Everything should be made as simple as possible
  ... but not simpler - Einstein
• Specification - user-centered design customer
  skills
• Design/programming proud craft persons
• Read great books to learn to write, read great
  programs to learn to program
• Lions Commentary on Unix - John Lions
• Testing sadists analysts
• Performance back of envelopers
• Quality - superb communicators
• Tools environment geeks emeritus
• Product manger coordinated extended team
• Sales, marketing, pricing, legal, service,
• Executives owe it to the organization and to
  their fellow workers not to tolerate
  nonperforming individuals in important jobs -
  Drucker

11
Process is not a four-letter word

• Iterative by many names
• Build a little, test a little, refine
  requirements a little
• Entrance/exit criteria
• Templates best cases
• Lightweight
• Web
• Inspections reviews
• Requirements, design, code, test, etc
• Lightweight to heavyweight
• Especially on tough stuff and with new people
• Lead customers
• User-centered design
• Screw-up early detection action
• Jeopardies and speaking up
• Audits with smart friends

I had the worlds greatest group, and I should
have made the worlds two greatest groups. I
didnt realize there are benefits to having real
implementers and real users - Alan Kay
12
Whats your favorite bug?
13
Architecture

• Controlling complexity is the essence of
  computer programming - Kernighan
• Top-down decomposition of a complex problem to
  allow independent, bottom-up development and
  change
• Goals
• Changeability/customizability
• Chunk independence
• Reduce n2 effects
• Inject huge-payoff CS technology
• Tiny languages, protocols, finite state machines,
  search, algorithms data structures, formal
  methods, etc
• Performance
• Reuse
• Check it
• Prototype
• Audit with smart friends
• Crumble with time
• An Architecture History of the Unix System -
  Feldman Usenix 84

14
Calling the shot

• The true art
• Best to wait until specification/design done
• Domain expertise crucial
• Sizing models help, but not magic (still /- 30)
• Let the numbers do the talking - Don Reifer
• Aggressive but doable
• Starvation not gluttony
• Reduce communications, complexity feature creep
  
• Whilst it is true a large programming project
  might require a large programming team, a large
  programming team will always build a large
  project - J. K. Buckle Managing Software
  Projects
• Slips
• If you must, do it once
• Iterative development provides an out
• A customer will always forgive you a slipped
  schedule or missed feature, they will never
  forgive you for bad quality or performance -
  Buckle

15
Which Project Used 15x Staff of the Other?

• Loop Maintenance System
• IBM MVS system
• PDP 11/20 backup
• 8 Months to go
• No schedule
• No performance plan
• No requirements
• Weak management

Switch Test Head PDP 11/20
Repair Service Bureaus
PDP 11/70
IBM 370

• Loop Assignment System
• Univac system
• Failed trial
• 250 million down the drain
• Demoralized, weak team
• Senior 18-month commitment
• No architecture or requirements
• No technology infrastructure
• Working temporary subsystem
• Unix based

Temporary Subsystem
Robust Pipe
Univac 1100 with C
16
How many lines of code?
Determine the frequency count of the occurrence
of words in text, most frequent first

• 1 - Six Unix tools, piped together
• 100 - C, Pascal
• 600 - ACM paper on commenting code
• 1,000 - Cobol, PL/1
• 5,000 - Unisys SW VP
• 10,000 - IBM SW VP
• ??? with very detailed requirements

17
Tempo

• .. the disaster is due to termites, not
  tornadoes and the schedule has slipped
  imperceptibly but inexorably. Indeed, major
  calamites are easier to handle one responds with
  major force - Brooks
• The plan
• Delicate balance of top/down and bottom/up
• PERT to plan, GANTT to manage
• Critical path control
• Dont multitask
• Buffer schedule
• 50 estimates
• Railroad train feature loading
• Leave the station on time
• Features left behind
• The weekly/biweekly project meeting
• Milestones
• All milestones are important, particularly early
  ones and few big ones
• Build team morale, tempo, customer confidence
• No plan survives contact with the enemy -
  Helmuth von Moltke the Elder

18
The Project Meeting
The Screaming
19
Whipping the rowers

• Project manager style
• Most decisions are 50/50
• Avoid acting like A glacier with dignity
• An officer in charge on an Indian agency made a
  requisition in the autumn for a stove costing
  seven dollars, certifying at the same time that
  it was needed to keep the infirmary warm during
  the winter, because the old stove wore out.
• Then, after glacial dignity
• The stove is here. So is spring

An Autobiography - Theodore Roosevelt
20
Testing

• Program testing can be a very effective way to
  show the presence of bugs, but it is hopelessly
  inadequate for showing their absence - Dijkstra
• Full lifecycle activity
• One day build and test (tempo!)
• Multi-level
• Automated
• Domain specific tools
• Field driven test libraries
• What to test
• Requirements design
• Coverage
• Usability
• Performance
• Measure with S curves

21
Get your S-curve You cant make this stuff up
22
Communications Documentation

• Schedule disaster, functional misfits, and
  systems bugs all arise because the left hand
  doesnt know what the right hand is doing -
  Brooks
• Project meetings
• Hierarchical
• Weekly or biweekly
• Jeopardies are good
• Documents
• The act of writing forces hundreds of tiny
  decisions - Brooks
• Few key ones, all template driven, all
  reviewed/inspected
• Requirements, architecture, design, test
  project plan
• Strong version and configuration control
• Code
• Structure - aim for one pagers
• Comments
• This section of code is cursed - Stroustrup C
  
• You are not expected to understand this - Unix

23
Metric Driven Improvement

• Quality
• Measures
• Fault density
• Service responsiveness
• Improvement
• Root cause analysis
• Team individual
• Productivity
• Customer satisfaction
• Huge fault density correlation

24
A Project-Specific Metric

• Project characteristics
• Gluttonously staffed 5,000 people
• Long crumbled architecture - 20 years
• Kryptonite-weight processes
• Humongously successful
• 99.9999 uptime
• Down 30 seconds a year for all causes
• Drug dealer cash flow

• Proposed programmer metric
• Lines of code
• Meetings attended
• Improvement objective exceed 1!

25
Contrarian Views

• Design methodologies
• Great designers make great designs
• Methodologies make lousy designs look good
• Fancy measures
• Confuse precision with accuracy

26
Making It Stick

• People support what they invent
• Thick dictates are good only for cellophane
• Bellcores was ten one-pagers (what not how)
• Participatory multi-day course
• Crusty expert led
• Project team takes course
• Time it with a new release schedule
• Output is their plan their process
• Team mentors
• Senior support

27
My Most Notable Disasters

• Hi ho silver
• Loop maintenance system
• Domain ignorance
• Customer ordering system
• Organizations can only do damage to themselves
  and to society if they tackle tasks that are
  beyond their specialized competence, their
  specialized values, their specialized functions
  - Drucker
• Second system effect
• Switch assignment system
• This second is the most dangerous system a man
  ever designs. The general tendency is to
  over-design the second system, using all the
  ideas and frills that were cautiously sidetracked
  on the first one. - Brooks

28
Maybe a little magic

• High quality, on time useful software can be
  built
• Bellcore after six-years of continuous
  improvement (1994)
• 95 on time/content
• 3-4x competitive productivity
• 10x competitive quality
• Its all about people
• Picking and developing the best and getting rid
  of the worst
• Helping them select and use good software
  engineering practices
• Motivating and rewarding them for
• Using and improving these practices
• Behaving as teams
• Giving them the time and resources to do their
  jobs
• Aggressive but realistic goals
• Making and holding them accountable
• Crowd control can be fun

29
Things you might consider trying(Other than
shooting Alfie the shot-caller)

• Development process
• Iterative development
• Entrance/exit
• Design template
• User-centered design (pair with another team)
• Inspections
• Heavy and lightweight
• Performance model
• Defect metrics root cause analysis
• Tempo control
• Project meeting
• Milestones
• Railroad
• Jeopardy
• Outside audit (pair with another team)
• Changeability/customization
• Automated testing
• System module
• S curves

30
Final
That's All Folks!!!