The Agile Heartbeat

1
The Agile Heartbeat

• John Graham-Cumming
• Co-founder
• Electric Cloud, Inc.
• Mountain View, CA
• jgc_at_electric-cloud.com

2
Agenda

• The death of the Nightly Build
• Continuous Integration as seen by the build
  manager
  
• The Agile Heartbeat
• Stepping Stones to Continuous Integration
• Tools that can help

3
The Death of the Nightly Build
4
Goodbye Nightly Build

• Build is too long to run overnight
• There are too many variants
• There is no night round the clock
  development
  
• The nightly build is always broken

5
Aside to Enterprise IT

• Automated daily builds are one of the most
  widely disregarded best practices of software
  development. Most software vendors perform
  frequent, and often daily, builds, but for
  enterprise IT organizations, this is the
  exception, rather than the rule.
• Forrester Research report Reining in Runaway
  Builds, February 2006.

6
Aside to Enterprise IT

• Skip the Nightly Build phase altogether
• Go straight to continuous integration
• But if not, nightly builds are still a best
  practice

7
Long builds?
8
While waiting?
9
Real-world Build Trouble

• 2007 survey by Osterman Research
• 352 software companies
• Only 9 said they dont have build problems

10
The Top 3

• Managing Dependencies 50
• Debugging broken builds 40
• Cross-platform builds 29
• Not going to talk about those
• but some pointers

11
Managing Dependencies

• Useful articles
• Dependency ManagementDr Dobbs, April
  2006http//www.ddj.com/dept/architect/184406479
  
• Tips and tricks of the automatic dependency
  generation masterCM Crossroads, October
  2006http//www.cmcrossroads.com/option,com_smf/It
  emid,180/topic,69672/

12
Debugging Broken Builds

• Useful article
• Debugging MakefilesDr Dobbs, March
  2007http//www.ddj.com/dept/debug/197003338

13
Cross-platform Builds

• Useful article
• Cross-platform BuildsDr Dobbs, January
  2005http//www.ddj.com/articles/2005/0501/

14
Infrastructure Problems

• Builds take too long 31
• Maintenance of scripts 30
• Dispersed build systems 28
• Need better reporting 26
• Large volume of builds 24
• Unclear build process 23
• Developers cant do full builds 17

15
Continuous Integration
16
Continuous Integration

• a software development practice where engineers
  integrate frequently, leading to multiple
  integrations per day. Each integration is
  verified by an automated build and test to detect
  integration errors as quickly as possible.
• http//www.martinfowler.com/articles/continuousInt
  egration.html

17
Client Story Intuit

• Intuit implemented Continuous Integration for the
  QuickBooks code base
  
• Reduced build time from 3 hours to 30 minutes
• Complete build and smoke test of latest sources
  every 30 minutes all day
  
• Broken builds result in email to developers
• Last working build of the day becomes the
  nightly
  
• Broken builds are a thing of the past

18
The Five Elements

• Continuous Integration requires
• Automated builds
• Self-testing builds
• Fast builds
• Every engineer commits every day
• Every commit starts a build

19
Automate the Build

• anyone should be able to bring a virgin
  machine, check the sources out of the repository,
  issue a single command, and have a running system
  on their machine

20
Automate the Build

• Implications of the build command
• Build location must be specifiable
• Source location, where to put the binaries
• Build variants must be specifiable
• Debug or release, target platform/processor
• Must set a suitable exit code
• Must know when to halt because of errors
• Needs to build the installer
• Needs to be able to clean up
• Needs to be able to install or verify the tool
  chain

21
IHTM Getting Acquired

• I worked for a start-up that was acquired by A
  Big Co.
  
• ABC didnt have automated builds, they had a team
  of harassed build starters
  
• A week after we merged many of my engineers still
  couldnt build ABCs code on their machines
  
• It Happened To Me

22
Automated Builds

• Some important benefits
• A new employee can be up and running in the time
  it takes to run build.
  
• If build is stored in the SCM old software
  versions can be quickly rebuilt.
  
• Anyone can be the build manager.

23
IHTM The Old Version

• Working at Some British Co. had to patch a
  two-year old version of a driver for an
  Australian Bank.
  
• No way to build the old code.
• Ended up a delivering a patch containing an
  entirely new version.
  
• Crossed our fingers that AB didnt figure it out.

24
Usmans Law

• make clean doesnt
• Achilles Heel of many automated builds systems
  is clean up doesnt work properly.
  
• Essential that can return to virgin state
• Must remove all generated files
• Objects, code, temporary
• Uninstall DLL, OCX, SO

25
Make Your Build Self-testing

• any body of code must have a set of automatic
  tests that can check a large part of the code
  base for bugs
  
• Could be a combination of
• A smoke test
• Developer written unit tests
• Subset of full test suite selected by QA
• Doesnt have to be the full test suite.

26
Everyone Commits Every DayEvery Commit Builds

• Imagine 20 engineers working an 8 hour day
• Each commits once, so 20 builds
• One build and test every 24 minutes
• More engineers means more builds
• Ultimate goal is a build per commit
• No commits unless last build is green

27
Per Commit Builds

• Advantages of One Build Per Commit
• If the build breaks the culprit is immediately
  known
  
• The culprit still has the state of the code in
  their mind
  
• Build breakages a fixed rapidly eliminating build
  crunches
  
• A common goal is 10 minute builds

28
Espresso Builds

• The ultimate Build Per Commit is the Espresso
  Build
  
• Engineer commits code
• Engineer types build
• Engineers gets and drinks a coffee
• Build and test results are ready

29
Keep the Build Fast

• Extreme Programming has a goal of 10 minute
  builds
  
• For most people thats unrealistic
• But implementing CI means having fast builds, and
  the build speed will throttle the integration
  speed

30
Client Story LSI Logic

• Software builds were taking 3 to 4 days (without
  testing)
  
• Decided to be agile
• Needed to build and test 50 unique products every
  day

31
Client Story LSI Logic

• Reduced smoke test time to 1 hour through
  reengineering
  
• Implemented parallel building to reduce a single
  build from 150m to 20m
  
• Replaced home grown build system with commercial
  build automation tool
  
• Can now build everything every day

32
The Agile Heartbeat

• Status of the per-commit build indicates the
  health of the software project
  
• With many commits per day and many builds per day
  its the heartbeat of the software team
  
• The Agile Heartbeat is a useful single measure of
  progress

33
Stepping Stones to CI
34
Stepping Stones

• Fully Automate
• Fast Builds
• Lava Lamps
• Build and Test
• Pre-Flight Builds

35
1. Fully Automate

• Create the build command
• Can probably be created without acquiring new
  tools
  
• Use shell scripts, Perl, or other languages to
  wrap up existing scripts
  
• Make sure it works from a clean machine all the
  way through installation with no intervention

36
1. Fully Automate

• Build when SCM changes
• A common practice is to monitor the SCM system
  for changes
  
• Then wait for a quiet period (no changes) of 15
  minutes
  
• Then launch a build using build

37
1. Fully Automate

• A good opportunity to try out a Build Automation
  tool is to use it to just monitor SCM and kick
  off a build
  
• Lets you try out the tool without implementing
  all of CI first

38
2. Fast Builds

• Set a realistic goal
• 10 minute build and test is unrealistic for most
  teams
  
• Perhaps build and test in 1 hour

39
2. Fast Builds

• Possible approaches
• Upgrade old hardware (particularly slow disks)
• Buy SMP hardware
• Manually restructure build for parallel building
• Use free parallel build tools (e.g. make -j,
  distcc)
  
• Buy commercial parallel build tool (e.g.
  ElectricAccelerator)
  
• Parallel builds will give the best speedups

40
Parallel Build Gotchas

• make -j, distcc and other tools frequently break
  or produce bad builds
  
• Thats not their fault!
• Many build systems are not parallel safe
• They created assuming a serial build

41
Parallel Build Gotchas

• Missing dependencies
• Assuming this happens before that without
  specifying that time dependency
  
• Reuse of generated files
• Example creating the same intermediate generated
  code file in different rules
  
• Writing to a shared file
• Forces the build to run serially

42
IHTM Poor Speedup

• For most organizations Ive dealt with the
  maximum speedup they get from free tools is 2-4x
  
• All of them break down as they push for greater
  and greater speedup when they hit a non parallel
  safe part of the build
  
• At East Coast Co. GNU Make -j achieved a 2.5x
  speedup on a 32-processor machine, but a
  commercial tool produced 5.2x

43
Aside Amdahls Law

• Maximum possible speedup in a parallel build
  is 1 / ( F ( 1 - F ) / P )
  
• P is the number of processors available
• F is the fraction of the build that has to run
  serially

44
Amdahls Law Example

• A build lasts for 4 hours
• The longest portion that must run on a single
  processor is creating the installer which takes
  10 minutes
  
• The maximum possible speedup with 10 processors
  is 7.3x

45
Amdahls Law Graph
46
3. Lava Lamps

• Create a build monitoring tool visible to
  everyone
  
• e.g. Red and Green Lava Lamps indicating build
  status
  
• http//www.artima.com/weblogs/viewpost.jsp?thread
  67492

47
3. Lava Lamps

• Step 1s automatic build sets the build monitor
  status automatically.
  
• Then make the build run as many times as
  possible.
  
• Lava lamps become indication of software health.

48
3. Lava Lamps

• Lava lamps are only practical for small teams
  with one build variant
  
• Real world alternatives
• Internal web page showing build status (make it
  each developers home page)
  
• Flat panel display showing build status mounted
  high up
  
• Email alerts when build is broken

49
Real World Status
50
Culprit Identification

• When the build fails email people who last made
  changes
  
• Before build starts interrogate SCM for names of
  users whove submitted since last build
  
• With frequent, short builds the culprit list will
  be short

51
4. Build and Test

• Integrate smoke test and/or unit tests with build
  script
  
• Now lava lamps show build and test status
• QA can also propose tests that should be included
  automatically

52
IHTM You Broke It, You Own It!

• Working as a partner to Near Seattle, Co. was
  forced to own the build because I broke it.
  
• The build manager floated from person to person.
  If you broke the build you owned it until the
  next person who broke it.
  
• The current build breaker was indicated by a
  skull and crossbones flag on their cube.

53
5. Pre-Flight Builds

• Put full builds in the hands of developers
• Shared server resource for doing builds
• Internal web page for launching builds
• Branch or shared directory to build from
• What to build and test
• Who to email the results to
• Queue of builds on FIFO basis

54
5. Pre-flight Builds

• Problems to contend with
• Long build queue
• Disk space management
• A new engineering rule
• No commits until a pre-flight build has passed on
  the proposed changes

55
Client Story Qualcomm

• 100s of different build configurations
• Individual builds taking 2 hours
• Implemented parallel builds to reduce build time
  to 15 minutes
  
• Now we make incremental changes we might not
  have tried before simply because of the build
  time

56
Tools
57
Tools that can help

• Three major categories
• Build acceleration
• Build automation
• Build visualization
• Wikipedia has a long page on CI
  toolshttp//en.wikipedia.org/wiki/Continuous_int
  egration

58
Build Accleration

• Free accelerators
• GNU Make -j and friends
• distcc
• ccache
• Best known commercial tool is ElectricAccelerator

59
Build Acceleration

• Some tools claim to speed up builds by running
  them regularly
  
• Or by managing a group of machines that run many
  builds
  

60
Build Automation

• Very crowded space
• Wikipedia has extensive list of tools
• Best known free tool is CruiseControl
• Well known commercial tools are IBM Rational
  BuildForge and ElectricCommander

61
Build Automation
62
Build Automation
63
Build Automation
64
Build Visualization

• Typical builds contain hundreds or thousands of
  steps
  
• Dependencies are poorly understood
• Parallel building adds a layer of complexity
• Agile development means multiple builds per day

65
Build Visualization
66
Build Visualization
67
Build Visualization
68
Questions to ask
69
Questions to ask your vendor

• Does the tool support scheduled builds?
• What are the reporting and anaytical tools?
• Does it support pre-flight and per-commit
  builds?
  
• Does the tool support Identity Management?
• Does the tool support multiple teams?

70
Does the tool scale?

• Agile development means lots of fast builds
• How does the tool scale as
• the build load increases
• additional servers, platforms or targets are
  added
  
• the organization grows

71
Is the tool easy to adopt?

• Rewriting Makefiles, build scripts, tests, etc.
  should not be necessary
  
• Avoid any tool that doesnt leverage your
  existing build infrastructure
  
• Should be able to kick the tires without a
  major investment

72
Wrap up
73
Best Practice Review

• Fully automated build and test
• Build when SCM changes
• Visible, automated build status
• Culprit identification
• Pre-flight builds for developers

74
Ultimate Goal 1, 1, 1

• One change
• by one developer
• results on one full build and test

75
Further Help

• CM Crossroads Ask Mr Make
• Not just about Make!
• http//www.cmcrossroads.com/articles/ask-mr.-make.
  html
  
• Agile Journal
• http//www.agilejournal.com/
• www.jgc.org
• All my writing and speaking about GNU Make and
  builds