Open Source Development Tools

1
Lecture 11

• Open Source Development Tools

2
Types of Development Tools

• Editors vi, emacs
• Managing code CVS, Subversion, git
• Compiling make, Ant
• Debugging/Profiling gdb, strace, DTrace, gprof,
  DevelDprof
• Archiving/Packaging tar, cpio, pax, rpm
• Configuration autoconf
• IDE Eclipse, NetBeans

3
Revision Control

• Aka version control, source code management
• Manage changes to source codes (or other
  documents)
• Multiple developers can work on the same project
  and files at the same time
• Changes to the same file are merged

4
Centralized Revision Control

• Central repository
• Client-server
• Lock-Modify-Unlock
• Only one person can write to the central copy
• Source Code Control System (SCCS)
• Revision Control System (RCS)
• Copy-Modify-Merge
• Multiple developers can edit same file at the
  same time
• Changes are merged
• Concurrent Version System (CVS), Subversion

5
Distributed Revision Control

• No single central repository
• Work directly with local repository
• Codes from different repositories are merged
  later
• Advantages
• Can work offline
• Fast operations
• Git, Mercurial, Bazaar

6
Revision Control Concepts

• Commit/Checkin
• Write/Merge changes in working copy to repository
• Checkout
• Create local working copy of repository
• Trunk/Mainline
• Base of the project
• Branch
• Separate copies of code under parallel
  independent development

7
Revision Control Concepts

• Tag/Label
• Snapshot of a project
• May correspond to releases, milestones
• Head
• Most recent commit/latest revision
• Merge
• Reconciling multiple changes to same file
• 3-way merge
• Delta compression
• Record only differences in each revision, similar
  to diff

8
Revision History
1
3
Trunk
Branch
2
4
merge
6
5
Branch
tag
9
7
Release 2.0
8
9
RCS Basic Operations

• Set up a directory for RCS
• mkdir RCS
• Check in a new file into the repository
• ci filename
• Check out a file from the repository for reading
• co filename
• Check out a file from the repository for writing
• co l filename
• Acquires lock
• Compare local copy of file to version in
  repository
• rcsdiff rltIDgt filename

10
RCS Keywords

• Keywords in source files are expanded to contain
  RCS info at checkout
• keyword ? keyword value
• Use ident to extract RCS keyword info
• Author Username of person checked in the
  revision
• Date Date and time of check-in
• Id A title that includes the RCS
  filename, revision number, date, author, state,
  and (if locked) the person who locked the file
• Revision The revision number assigned
• Example Id expands to
• Id proj.sh,v 1.32 2009/11/19 203941 alice Exp
  

11
CVS Features

• No exclusive locks like RCS
• No waiting around for other developers
• No hurrying to make changes while others wait
• Avoid the lost update problem
• Client-server model
• Support branches

12
CVS Repository

• All revisions of a file in the project are in the
  central repository (using RCS)
• Work is done on the checkout (working copy)
• Top-level directories are modules checkout
  operates on modules
• Different ways to connect
• pserver, external rsh/ssh

13
CVSROOT

• Environment Variable
• Location of Repository
• Can take different forms
• Local file system /usr/local/cvsroot
• Remote Shell (set CVS_RSH to ssh for cvs over
  ssh) extuser_at_server/usr/local/cvsroot
• Client/Server pserveruser_at_server/usr/local/cvs
  root

14
Getting Started

• cvs basic-options ltcommandgt cmd-options
  files
• Basic options
• -d ltcvsrootgt Specifies CVSROOT
• -H Help on command
• -n Dry run
• Commands
• import, checkout
• update, commit
• add, remove
• status, diff, log
• tag...

15
Setting up CVS

• Importing source
• Generates a new module
• cd ltsource-dirgt
• cvs dltcvsrootgt import ltnew-modulegt \
  ltvendor-branchgt ltrelease-taggt
• cvs dltcvsrootgt checkout ltmodule-namegt

16
Managing files

• Add files add (cvs add ltfilenamegt)
• Remove files remove (cvs remove ltfilenamegt)
• Get latest version from repository update
• If out of sync, merges changes. Conflict
  resolution is manual.
• Put changed version into repository commit
• Fails if repository has newer version (need
  update first)
• View extra info status, diff, log
• Can handle binary files (no merging or diffs)
• Specify a symbolic tag for files in the
  repository tag

17
Subversion

• Meant to be a better CVS
• Directories, renames, metadata are versioned
• Branching and tagging implemented as copy - no
  branch-point tagging
• Can use database (BerkeleyDB) backend
• Connect using svn, svnssh, http

18
SVN - Setup Repository
On server svnadmin create /path/to/repos On
client svn import myproj \ svnssh//host/path
/to/repos/myproj \ -m "Initial
import Adding myproj/one.cAdding myproj/two.cAd
ding myproj/somedirCommitted revision 1.
19
SVN - Initial Checkout
svn checkout \ svnssh//host/path/to/repos/som
e/proj A proj/one.cA proj/two.cChecked out
revision 1.
20
SVN Commands

• svn add foo
• Schedule file/dir foo to be added to repo.
• svn delete foo
• Schedule foo to be deleted from repo
• svn copy foo bar
• Create duplicate of foo as bar and schedule it
  for addition
• svn move foo bar
• Schedule bar (a copy of foo) for addition and foo
  for removal

21
SVN Commands

• svn mkdir foo
• Same as mkdir foo svn add foo
• svn status
• Overview of your changes - shows what files are
  scheduled for addition, deletion, modification,
  etc.
• svn commit
• Commit your schedule changes to the repo
• svn update
• Update your working copy to include changes
  others made

22
Subversion Example
vi foo.c edit code svn status M
foo.c svn diff Index foo.c
--- foo.c
(revision 27) foo.c (working copy) _at__at_
-7,7 7,7 _at__at_ - for( i0 iltmpt2 i )
for( i1 iltmpt2 i ) svn commit Sending
foo.c Transmitting file data . Committed
revision 28.
23
Subversion Revisions

• svn commit is atomic
• Each commit creates a revision
• Global revision numbers
• Revision N is state of repository after Nth
  commit
• Updates and commits are separate

24
Branching

• Subversion branches are normal filesystem
  directories in the repository
• Use svn copy to create a duplicate in the
  repository
• svn copy \svn//host/path-to-repo/proj/trunk \
  svn//host/path-to-repo/proj/branches/my-branch
  \-m Creating a new branch
• Checkout a working copy to work on your branch
• svn checkout \svn//host/path-to-repo/proj/branch
  es/my-branch

25
Merging

• Keep your branch in sync with trunk regularly
• Use svn merge to include changes to your branch
• svn merge svn//host/path-to-repo/proj/trunk
• Use svn diff to examine the changes
• Resolve conflicts and submit changes to repo
• svn commit -m Trunk changes merged to branch

26
Tags

• Tag and branch are the same in Subversion
• svn copy \svn//host/path-to-repo/proj/trunk \
  svn//host/path-to-repo/proj/tags/rel-1.0 \-m
  Release 1.0 of proj

27
Git

• Distributed version control
• No central server
• Each Git clone is a complete repository with the
  entire development history
• Fast branching and merging
• gitk - Git repository browser

28
Git Example

• Import new project
• tar xzf proj.tar.gz from tarball
• cd proj
• git init
• Create first version
• git add . create snapshot
• git commit
• Make changes to files .
• git add foo bar baz add to index
• git status
• git commit

29
Cloning and Merging

• User A clones user Bs project and makes changes
• git clone /some-path/user-b/proj mycopy
• change some files
• git commit -a
• User B merges user As master branch into Bs
  current branch
• git pull /some-path/user-a/mycopy master

30
Git - SVN
31
Debuggers

• Advantages over the old fashioned way
• you can step through code as it runs
• you dont have to modify your code
• you can examine the entire state of the program
• call stack, variable values, scope, etc.
• you can modify values in the running program
• you can view the state of a crash using core files

32
Debuggers

• Debuggers let you examine the internal workings
  of your code while the program runs.
• Debuggers allow you to set breakpoints to stop
  the program's execution at a particular point of
  interest and examine variables.
• To work with a debugger, you first have to
  recompile the program with the proper debugging
  options.
• Use the -g command line parameter to cc, gcc, or
  g
• Example gcc -g -c foo.c

33
Using the Debugger

• Two ways to use a debugger
• Run the debugger on your program, executing the
  program from within the debugger and see what
  happens
• Post-mortem mode program has crashed and core
  dumped
• You often won't be able to find out exactly what
  happened, but you usually get a stack trace.
• A stack trace shows the chain of function calls
  where the program exited ungracefully
• Does not always pinpoint what caused the problem.

34
GDB, the GNU Debugger

• Text-based, invoked with
• gdb ltprogramfilegt ltcorefilegtltpidgt
• Argument descriptions
• ltprogramfilegt executable program file
• ltcorefilegt core dump of program
• ltpidgt process id of already running program
• Example
• gdb ./hello
• Compile ltprogramfilegt with g for debug info
• Some apps provide graphical frontend to gdb, e.g.

35
Basic GDB Commands

• General Commands
• file ltfilegt selects ltfilegt as the program to
  debug
• run ltargsgt runs selected program with
  arguments ltargsgt
• attach ltpidgt attach gdb to a running process
  ltpidgt
• kill kills the process being debugged
• quit quits the gdb program
• help lttopicgt accesses the internal help
  documentation
• Stepping and Continuing
• continue continue execution (after a stop)
• step step one line, entering called functions
• next step one line, without entering
  functions
• finish finish the function and print the return
  value

36
GDB Breakpoints

• Useful breakpoint commands
• break ltwheregt sets breakpoints. ltwheregt
  can be a number of things, including a
  hex address, a function name, a
  line number, or a relative line offset
• rwatch ltexprgt sets a watchpoint, which will
  break when ltexprgt is written to or read
• info breakpoints prints out a listing of all
  breakpoints
• clear ltwheregt clears a breakpoint at ltwheregt
• delete ltnumsgt deletes breakpoints by number

37
Playing with Data in GDB

• Commands for looking around
• list ltwheregt prints out source code at ltwheregt
• search ltregexpgt searches source code for
  ltregexpgt
• backtrace ltngt prints a backtrace ltngt levels
  deep
• info ltwhatgt prints out info on ltwhatgt
  (like local variables or function args)
• print ltexprgt prints out the evaluation of
  ltexprgt
• Commands for altering data and control path
• set ltnamegt ltexprgt sets variables or arguments
• return ltexprgt returns ltexprgt from current
  function
• jump ltwheregt jumps execution to ltwheregt

38
System Call Tracing

• strace (Linux), truss (OpenSolaris)

strace date execve("/bin/date", "date", /
42 vars /) 0 fstat(1, st_modeS_IFCHR0620,
st_rdevmakedev(136, 0), ...) 0 mmap(NULL,
4096, PROT_READPROT_WRITE, MAP_PRIVATEMAP_ANONYM
OUS, -1, 0) 0x2ab06de16000 stat("/etc/localtime"
, st_modeS_IFREG0644, st_size3519, ...)
0 write(1, "Wed Nov 18 022011 EST 2009\n", 29)
29 close(1)
0 munmap(0x2ab06de16000, 4096)
0 exit_group(0) ?
39
DTrace

• Dynamic tracing framework created by Sun for
  Solaris, ported to other Unix-like systems
• Insert probes into kernel and user applications
  (instrumenting)
• Can be used on observe live production systems
• Very low overhead
• No disabled probe effect
• Highly scriptable
• Scripts in C/awk-like D language

40
DTrace Example
!/usr/sbin/dtrace -s pragma D option
quiet syscallentry /pid 9047/
_at_numprobefunc count()
probe
predicate
action
./mytrace.d C write
1 read 44
pollsys 45 ioctl
134
41
Dtrace Probes

• providermodulefunctionname
• Examples
• syscallreadentry (read() entry point)
• iostart (any physical I/O issued)
• dtrace -l lists all available probes

42
Profilers

• Learn where your program spent its time
• Measure frequency and duration of function calls
• Use gprof to profile C/C application
• Compile and link with -pg to enable profiling
• Execute program to generate call graph profile
  data
• Run gprof to analyze/display data

43
gprof

• g -g -pg -o segment segment.cpp -lm
• ./segment
• gprof segment
• Flat profile
• Each sample counts as 0.01 seconds.
• cumulative self self
  total
• time seconds seconds calls ms/call
  ms/call name 33.36 0.04 0.04
  1 40.03 40.03 void std__introsort_looplted
  ge, longgt(edge, edge, long)25.02 0.07
  0.03 6 5.00 5.00
  convolve_even(imageltfloatgt, imageltfloatgt,
  stdvectorltfloat, stdallocatorltfloatgt
  gt)25.02 0.10 0.03 1 30.02
  120.10 segment_image(imageltrgbgt, float, float,
  int, int)16.68 0.12 0.02 1
  20.02 60.05 segment_graph(int, int, edge,
  float)

44
Profiling for Other Languages

• gprof for C, C, F77 or single-threaded Java
  programs compiled with gcj
• Perl programs can use DevelDProf
• perl -dDprof prog.pl
• Dprofpp
• Netbeans Profiler for Java

45
Make

• make A program for building and maintaining
  computer programs
• developed at Bell Labs around 1978 by S. Feldman
  (now at Google)
• Instructions stored in a special format file
  called a makefile.

46
Make Features

• Contains the build instructions for a project
• Automatically updates files based on a series of
  dependency rules
• Supports multiple configurations for a project
• Only re-compiles necessary files after a change
  (conditional compilation)
• Major time-saver for large projects
• Uses timestamps of the intermediate files
• Typical usage executable is updated from object
  files which are in turn compiled from source files

47
Dependency Graph
48
Makefile Format

• Rule Syntax
• lttargetgt ltdependency listgt
• ltcommandgt
• The lttargetgt is a list of files that the command
  will generate
• The ltdependency listgt may be files and/or other
  targets, and will be used to create the target
• It must be a TAB before ltcommandgt, or it wont
  work
• The first rule is the default lttargetgt for make

49
Examples of Invoking Make

• make -f makefile
• make target
• make
• looks for file makefile or Makefile in current
  directory, picks first target listed in the
  makefile

50
Make Sequence of Execution

• Make executes all commands associated with target
  in makefile if one of these conditions is
  satisfied
• file target does not exist
• file target exists but one of the source files in
  the dependency list has been modified more
  recently than target

51
Example Makefile

• Example Makefile
• CCg
• CFLAGS-g Wall -DDEBUG
• foobar foo.o bar.o
• (CC) (CFLAGS) o foobar foo.o bar.o
• foo.o foo.cpp foo.h
• (CC) (CFLAGS) c foo.cpp
• bar.o bar.cpp bar.h
• (CC) (CFLAGS) c bar.cpp
• clean
• rm foo.o bar.o foobar

52
tar Tape ARchiver

• tar general purpose archive utility (not just
  for tapes)
• Usage tar options files
• Originally designed for maintaining an archive of
  files on a magnetic tape.
• Now often used for packaging files for
  distribution
• If any files are subdirectories, tar acts on the
  entire subtree.

53
tar archiving files options

• c creates a tar-format file
• f filename specify filename for
  tar-format file default is /dev/rmt0 or
  - (stdin/stdout)
• v verbose output
• x allows to extract named files
• z file archive through gzip
• j filter archive through bzip2

54
tar archiving files (continued)

• t generates table of contents
• r unconditionally appends the listed files
  to the archive files
• u appends only files that are more recent
  than those already archived
• L follow symbolic links
• m do not restore file modification times
• l print error messages about links it cannot
  find

55
cpio copying files

• cpio copy file archives in from or out of tape
  or disk or to another location on the local
  machine
• Similar to tar
• Examples
• Extract cpio -idtu patterns
• Create cpio -ov
• Pass-thru cpio -pl directory

56
cpio (continued)

• cpio -i dtum patterns
• Copy in (extract) files whose names match
  selected patterns.
• If no pattern is used, all files are extracted
• During extraction, older files are not extracted
  (unless -u option is used)
• Directories are not created unless d is used
• Modification times not preserved with -m
• Print the table of contents -t

57
cpio (continued)

• cpio -ov
• Copy out a list of files whose names are given on
  the standard input. -v lists files processed.
• cpio -p options directory
• Copy files to another directory on the same
  system. Destination pathnames are relative to the
  named directory
• Example To copy a directory tree
• find . -depth -print cpio -pdumv /mydir

58
pax replacement for cpio and tar

• Portable Archive eXchange format
• Part of POSIX
• Reads/writes cpio and tar formats
• Union of cpio and tar functionality
• Files can come from standard input or command
  line
• Sensible defaults
• pax wf archive .c
• pax r lt archive

59
Distributing Software

• Pieces typically distributed
• Binaries
• Required runtime libraries
• Data files
• Man pages
• Documentation
• Header files
• Typically packaged in an archive
• e.g., perl-solaris.tgz or perl-5.8.5-9.i386.rpm

60
Packaging Source autoconf

• Produces shell scripts that automatically
  configure software to adapt to UNIX-like systems.
• Generates configuration script (configure)
• The configure script checks for
• programs
• libraries
• header files
• typedefs
• structures
• compiler characteristics
• library functions
• system services
• and generates makefiles

61
Installing Software From Tarballs

• tar xzf ltdistgt.tar.gz
• cd ltdist-dirgt
• ./configure
• make
• make install

62
RPM Package Manager

• RPM packages in the form
• name-version-release-arch.rpm
• e.g. gcc-4.1.2-44.el5.x86_64.rpm
• Install rpm
• rpm -i pkgname-ver-rel.arch.rpm
• Or use higher level wrapper tools like yum
• yum install pkgname
• Checks for dependency
• Build rpm from tarball (with a .spec file)
• rpmbuild -tb pkgname-1.0.tar.gz
• RedHat, Fedora, openSUSE

63
Doxygen

• Documentation generator
• Extract documentation from comments in source
• Similar to javadoc
• Comments need to specially marked, e.g. JavaDoc
  or Qt style
• Generates output in HTML, PostScript, PDF, RTF,
  etc.

64
Special Documentation Blocks
/ _at_file _at_author Me ltme_at_test.comgt
_at_version 1.0 _at_section DESCRIPTION This
is just a test... / void do_nothing(int x)
/ Do nothing function _at_param
number of nothing / sleep(1)

• Create configuration file
• Use doxygen -g ltconfig-filegt to generate a
  template for editing
• Run doxygen ltconfig-filegt to generate
  documentation

65
IDE

• Integrated development environment
• Code editing
• Compiling
• Debugging
• Versioning
• Eclipse, NetBeans