Removing False Dependencies to Speedup Software Build Processes

1
Removing False Dependencies to Speedup Software
Build Processes

• Yijun YuHomy Dayani-FardJohn Mylopoulos

IBM Toronto Lab
University of Toronto
2
Background

• Phenomenon/Problem large C/C software takes
  much time to be built and rebuilt in the
  development processes, debugging, testing,
  integration, releasing all have to wait for the
  binary image
• Observation from IBM Toronto lab it takes gt 7
  hours to fully compile a system of gt 10 million
  LOC.
• Time is money
• Cost Single build time x Number of People x
  Build freq.
• and more
• Why? Excessive compile-time dependency among
  headers and implementations. Well-known.
• Light weight solution? makedepend cleanup target
  dependencies (0), removing unused header
  inclusions and shorten the inclusion chain
  (estimated by KLOCwork, 37)

3
Our approach

• Finding false dependencies among finest grain
  program units
• Removing them
• Recent case studies (3 months after reported in
  the CASCON paper)
• VIM 6.2, 269 KLOC in C (complete result)
• A large commercial software package, 1580 KLOC
  in C (new result)
• Future work and collaborations

4
1. False dependencies

• A PU X depends on a PU Y if X is using Y and Y
  occurs before X
• We consider a program unit (PU) as a declaration
  or a definition at program or file scope.
• Examples function, variable, typedef, union,
  struct, class, template, enum, macro
• Anti examples data field, member function, auto
  variables, blocks, etc.
• False dependency dependency at inclusion level,
  no dependency at program unit level

5
An analogy

• Chemical properties are kept in molecule
  structures, physical properties are kept in atom
  structures
• Moving molecules doesnt change the smell of the
  code, but replacing atoms in the molecule does!
• Risk of change
• 2H2 O2 2H2O heat
• E m c 2 leads to a nuclear bomb!

6
include foo.hint main() foo()
void foo() void bar()
include
foo.h
main.c
Fileboundary
true
foo
DeclareUnit
main
false
DefineUnit
bar
remove
7
Negative impacts

• Larger preprocessed image
• Longer compilation time
• False couplings among program units, components
  and packages

8
2. Extracting and removing false dependencies

• asg2dep Extracting PU dependencies from abstract
  syntax graph
• based on parser like Datrix, KLOCwork
• dependency graph locations
• header Partitioning the dependency graph
  (algorithm in the paper) and generating code
• Removing unused declarations
• Grouping declarations used by the same
  definitions into the same header files
• Location is output as a line directive
• Regression test and measurements

9
Example (original)
10
Example (restructured)
11
3. Case study VIM 6.2include relations
12
Program units dependencies
13
Preprocessed image size
8.81 MB --gt 4.78 MB
-45.76
14
Build time
96.37 seconds --gt 42.26 seconds
-56.15
15
Synch-and-Stabilize

• Continually synchronize parallel teams
• Periodically stabilize the product in increments
  versus once at the end
• Also known as
• Milestone, daily build, nightly build,
  zero-defect process

16
Case study 2. A C package in IBM preprocessed
image size
361.2 MB--gt 179.1 MB
-50.42
17
Build time
14.02 minutes --gt 4.44 minutes
-68.31
18
Cumulative build time of directly changed .C files
10.68 minutes --gt 3.81 minutes
19
Cumulative build time of recompiled .C files
7.78 hours --gt 2.46 hours
20
4. Summary

• False dependencies can lower productivity, thus
  are harmful to both managers and developers
• Removing false dependencies in C requires
  supporting tools
• Case studies have shown significant improvements
  (demo VIM results can be downloaded from
  http//www.cs.toronto.edu/yijun)

21
5. Future work and possible collaborations

• Integrate with IBM development on demand
• Leverage KLOCworks analysis tools
• Potential collaborations with CSER and CASCON
  members (give the grass, milk the cow)
• UVictoria and UAlberta visualize dependencies to
  assure programmers that there is no MAGIC behind
  the scene
• UWaterloo speedup the parser to meet the
  everyday need in the restructuring Datrix ALV?
  Bring 45 minutes per file down to 45 /184 minute
• UWaterloo and UQueens Quantitative goal
  modelling
• UWindsor web-services compositions and
  repackaging
• UGhent Link-time optimizations Diablo shrink
  VIM memory footprint to 1/3