Why Software Is (Almost) Always Late

1
Why Software Is (Almost) Always Late

• Chuck Connell,
• Boston University and
• CHC-3 Consulting

2
Driving

• Suppose you are taking a car trip... What are
  three planning activities that you can do? (other
  than the driving)
• Estimate the time needed
• Plan the route in detail
• Track progress as you go and possibly make
  changes

3
Software is similar

• These correspond to the three planning activities
  for software development
• Estimate (how long will the project take)
• Schedule (who will do the work and what will they
  do)
• Track/adjust (how are we actually doing).

4
Unfortunately

• Estimating is notoriously difficult for software
  development.
• I believe this is because software projects tend
  to be "new", whereas other kinds of engineering
  tend to repeat known projects.

5
Topics for this talk

• I will look primarily at estimating the time
  required for a software project.
• Ill include some additional information about
  scheduling specific tasks within a software
  development plan.
• Ill assume that estimating happens first. In
  reality, above two are inter-related and often
  are applied cyclically.

6
Some driving examples

• Background
• Worcester is 50 miles and 1 hour away
• Hartford is 100 miles and 2 hours away
• Bridgeport is 150 miles and 3 hours
• New York City is 200 miles and 4 hours
• These times are possible, but pretty tight.

7
What can go wrong 1

• You leave for a meeting in Worcester that starts
  in 60 minutes.
• 55 minutes later, you have fought downtown
  traffic, are in a parking garage, and are getting
  out of the car. 
• Your manager calls on the cell phone Hi! The
  meeting has changed to Hartford, but it starts an
  hour later. So this will be OK, right? You said
  Hartford was a 2 hour trip.
• Can you reasonably make it?

8
Moral 1

• Development times do not always add and subtract
  cleanly.
• The total may be more, or less, then the sum of
  the parts.

9
What can go wrong 2

• You are asked to attend a meeting in Bridgeport
  that starts in 2 hours.
• Can you make it?
• Your manager might saySkip lunch and drive
  fast.Be a team player.At least show that
  you are trying.

10
Moral 2

• Moral -- All of these reasons are beside the
  point. Trying to do something that is impossible,
  still makes it impossible. 
• How would your morale be at the start of the trip
  (and throughout it)?
• Side note -- What would be the worst thing that
  could happen about trying to get to Bridgeport in
  2 hours? (Other than a car crash.)

11
What can go wrong 3

• You have a meeting in NYC in 2 hours.
• Your manager says, I'm a good guy and I know
  this is normally impossible. So, I want you to
  rent a Lear jet and fly.
• Is this a good plan? Can you get there on time?

12
Moral 3

• Moral -- New tools (or extra employees) are good,
  in the long run. They can save you time on future
  projects.
• But there is a startup cost associated with new
  resources. You have to spend significant time now
  to save time later.

13
The right way to drive

• What do professional drivers do? 
• They make accurate, realistic estimates, based on
  experience.
• They allow time for lunch, gas stops, rest
  breaks, and traffic jams (without artificial
  padding).
• They execute the plan efficiently, and (usually)
  get places on time.

14
The right way to code

• The same is true for software.
• Good software estimates are realistic and allow
  for things to go wrong.
• Well look at the top 6 reasons that software
  estimates are inaccurate, based on this
  observation.

15
1. Scope creep

• Expanding the project as it progresses. (I.e.
  Changing the route to Hartford, when you are
  almost to Worcester.)
• This is one of the biggest problems in software
  development.
• I suspect this is more prevalent in software than
  other types of engineering?

16
Scope creep What to do about it

• Try harder to understand the full scope at the
  beginning. Don't listen to one person. Interview
  everyone involved, especially end-users.
• Dont make a firm schedule at first. Create a
  range of estimates and refine the schedule as the
  project becomes more clear. (McConnell)
• Make a realistic assessment of whether scope
  creep will occur. Cant just disallow it. If
  project cannot be nailed down at the start, add
  generous time to the schedule -- 50 or more.

17
2. Using only external deadline

• Must get this product ready for Christmas selling
  season, or the big trade show in April, or beat
  competitors to market.
• What to do about it -- Try not to base estimates
  solely on external deadlines.
• If the external deadline is truly important (and
  sometimes it is) modify the feature set to fit
  the deadline.

18
3. New problem domain or technology

• Example Frank Lloyd Wright buildings. His
  projects were often over-budget and overdue. Why?
  (He was doing things no one had ever done
  before.)
• New technologies are a particular problem because
  you don't know what will go wrong. You don't know
  what you don't know. All estimates are a guess.

19
New problem What to do about it

• If the project involves new technology, add
  substantially to the schedule -- 100 or more.
• Be realistic about whether the project is new or
  well-known. Has this group of engineers really
  done something very similar to this several
  times? (If not, this is new.)

20
4. Assuming all will go well

• Suppose there are 20 tasks in a software project
  (which is low) and each has a 90 chance of going
  well (which is very high).
• How likely is it that everything will go well?
• 12, or 0.920
• What if each of 20 tasks has an 80 chance of
  going well?
• 1

21
Assuming all will go well What to do about it

• Remember these simple examples when estimating a
  project. (Real projects often have 500 tasks,
  each with a 70 chance of going well. You are
  more likely to win the lottery than have a smooth
  project.)
• For small projects, plan on several significant
  things going wrong.
• For big projects (50 people, for 20 months),
  plan on many things going wrong.

22
5. Wanting to get the job

• Deliberately underestimating in order to get
  hired for a project or to get funding within a
  company.
• When I bid on a consulting project, it is to my
  advantage to lie about how long the project will
  take. (I dont do it.) This is a common gripe
  about outside consultants, and is often true.
• Competing groups within a large organization do
  the same thing they want the plum projects.
• It happens all the way up and down the ladder,
  and gets worse as it goes.

23
Wanting to get the job What to do about it

• Tell the truth to your customer, your employees,
  and your manager -- it will help you in the long
  run.
• As a manager, look for evidence that people are
  telling you the truth (rather than what you want
  to hear).

24
6. Motivation tool

• The project is deliberately scheduled too
  aggressively, as a way to get people to work
  hard. 
• The hope is that an unrealistic estimate will get
  everyone moving as fast as possible.
• Project manager knows the schedule will probably
  slip, but believes that the final result will be
  faster than giving people a more relaxed schedule
  in the beginning.
• Reality For any serious development
  organization, unrealistic schedules make projects
  later.

25
Motivation tool What to do about it

• As a manager, don't schedule this way. (It may
  work a couple times, but after that no one will
  believe you.)
• As an engineer, try to avoid these kinds of
  schedules (and managers).

26
Some comments on scheduling

• Scheduling is assigning specific people to
  specific tasks, within an overall project
  timeframe.
• In reality, estimating and scheduling are often
  cyclical activities.

27
Making a schedule for your life

• Suppose you are all college freshmen, and you go
  into a room and write a plan for your life
• You will graduate from college with straight A's.
• You will get a full scholarship to the grad
  school of your choice.
• You will finish your Ph.D. in two years.
• Your starting job will be as vice-president of a
  Fortune 100 company for 500,000/year.
• You will fall in love with the man (or woman) of
  your dreams by age 25, and you will never argue.
• You'll have 3 smiling children, who always listen
  to you.
• And you'll live happily ever after in Greenwich
  CT.

28
Making a schedule for your life

• You emerge from the room, hold up the plan, and
  announce I'm all set. I have it all written
  down.
• What is good about this kind of planning?
• What is bad about it?

29
Applying this to software

• A true story
• A VP of software development was concerned about
  a project, which was slipping later every week.
• He called a summit meeting, for all the project
  managers, to bring in the finish date and nail it
  down.
• They shut the door and agreed to not emerge until
  they had shortened the schedule.
• One of the managers laid a rock on the table,
  turned it over, and said that they would "leave
  no stone unturned" in their effort to shave time
  off the schedule.

30
A true story continued

• They emerged several hours later with a tightened
  schedule. Many tasks were shortened, or made
  concurrent, or eliminated.
• Everyone was happy they had accomplished their
  goal of shortening the schedule.
• What do you think happened?
• What could they have done instead?

31
The final moral

• Making a plan that looks good on paper does not
  get you anywhere.
• You must make realistic plans, and have a
  realistic way to accomplish your plan.

32
Further reading

• Debugging the Development Process, by Steve
  Maguire
• Rapid Development, by Steve McConnell
• Tom DeMarco