Shangli Ou

1
Profiling and Debugging

• Shangli Ou
• PhD, HPC_at_lsu.edu
• Jan, 2006

2
Profiling knowing yourself

• Profiling measure CPU time spent in different
  code block or subroutines
• Reasons for profiling understand the performance
  of your code, locate the bottleneck
  (communication bounded, computationa bounded,
  memory bounded), then optimize limiting code
  blocks.

3
Outlines

• Timing functions do timing statistics on your
  code manually
• Profiling tools on SuperMike and Pelican gprof,
  psrun, vprof/cprof
• Debugging tools on Supermike and Pelican
  dbx/gdb, totalview, pdbx

4
Timing functions (I)

• Sample outputs
• time a.out ...
• real 0m6.176s
• user 0m0.300s
• sys 0m0.210s

• Unix time command run the specified program and
  do timing statistics about the program run

5
Timing functions (II)

• C example
• include lttime.hgt
• ...
• clock_t t0, t1 float dt
• t0clock()
• ...
• t1clock()
• dt(t1-t0)/CLOCKS_PER_SEC

• Timing functions are provided in various
  programming languages to enable users to timing
  their code in flexible ways
• Fortran timef function returns elapsed time in
  ms since the first call to timef
• C clock function returns CPU time used fo far

6
Profiling tool gprof

• common profiling tool prof, gprof
• Intel (SuperMike) ifort -g -p src_file
• IBM (Pelican) xlc -pg src_file
• Ifort -g -p test.f90 -o test
• ./test -----gt generate
  gmon.out
• gprof ./test gmon.out or gprof

7
Profiling tool gprof (II)

• Output from gprof on SuperMike with Intel
  compilers
• ...
• Each sample counts as 0.01 seconds.
• cumulative self self
  total
• time seconds seconds calls s/call
  s/call name
• 92.93 18.40 18.40 200 0.09
  0.09 radiation_force_
• 6.52 19.69 1.29 1 1.29
  19.74 MAIN__
• 0.25 19.74 0.05 1048576 0.00
  0.00 mode_
• 0.25 19.79 0.05
  sqrt.J
• 0.05 19.80 0.01
  __intel_new_memset
• ...

8
Parallel profiling on Pelican

• IBM (Pelican) xlc_r -pg src_file
• Fortran example mpxlf90_r -pg test.f90 -o test
• Set environment variable MP_EUILIBPATH/usr/lpp/pp
  e.poe/lib/profiled/usr/lib/profiled/lib/profiled
  /usr/lpp/ppe.poe/lib
• ./test -----gt generate
  gmon.out.taskid
• gprof ./test gmon.out.0 or
• gprof ./test gmon.out.0 gmon.out.1
• AIX5.3gmon.taskid.out

9
Parallel profiling on SuperMike

• Intel (SuperMike) mpif90 -qp test.f90 -o test
• setenv GMON_OUT_PREFIX gmon.out.
• qsub ...
• gprof -s gout.
• gprof ./test gmon.sum

10
Profiling tools Psrun

• Available on SuperMike
• export PATH/usr/local/packages/psrun-0.6.2a6/bin
  PATH
• export LD_LIBRARY_PATH/usr/local/packages/papi-3.
  2.1/lib/usr/local/packages/psrun-0.6.2a6/libLD_
  LIBRARY_PATH
• Compile source code with -g -p
• psrun -F xmltext a.out

11
Profiling tools vprof

• Available on SuperMike
• export PATH/usr/local/packages/vprof-0.12/binPA
  TH
• export LD_LIBRARY_PATH/usr/local/packages/vprof-0
  .12/libLD_LIBRARY_PATH
• Compile source code with -g, link with vmon
  library,
• then run it

12
Vprof (II)

• C examples
• Modify your program, prog.c
• main()
• vmon_begin()
• ... your code here ...
• vmon_done()

• gcc -g -O2 -o prog prog.c VMONLOCATION/lib/libvmo
  n.a VMONLOCATION/lib/vmonauto_gcc.oPAPILOCATION/
  lib/libpapi.a
• ./prog
• This will generate a file vmon.out. Start the
  visual profiler
• vprof prog vmon.out
• Or run the command line profiler
• cprof -e prog vmon.out
• The -e option to cprof will print out everything.

13
Vprof (III)

• FORTRAN interface
• To use the vmon routines from FORTRAN, call the
  following routines
• VMONBG() ! Starts profiling.
• VMONDN() ! Ends profiling and writes
  vmon.out.
• VMONDT(INTEGER TASK) Ends profiling and
  writes to the file vmon.out.task_id

14
Serial Debugging tools

• dbx/gdb unix/GNU command line debuggers for
  serial programs
• need to compile source code with -g option
• dbx/gdb a.out
• Various commands help, file, where, break, list,
  print, trace, etc.

15
Parallel Debugging tools (I)

• IBM parallel debugger pdbx on Pelican command
  line debuggers similar to dbx but for parallel
  applications
• Compile code with -g option
• pdbx -rmpool 1 -nodes 1 -procs 4 executable
• commands similar to those in dbx
• http//www.hpc.lsu.edu/help/docs/index.php

16
Parallel Debugging tools (II)

• Totalview on SuperMike GUI
• Under /usr/local/packages/totalview
• add totalview-6.7.0-1 in .soft file in your
  home directory or manually setup ENV
• ssh -Y to a head node to ensure X11 forwarding
• Specify machine file to ensure an interactive
  head node to be the master node (launching your
  job from a head node interactively)
• mpirun -np 8 -machinefile ./machinefile -tv a.out

• Sample machinefile
• mike3.cct.lsu.edu
• mike3.cct.lsu.edu
• mike393.cct.lsu.edu
• mike393.cct.lsu.edu
• mike363.cct.lsu.edu
• mike363.cct.lsu.edu
• mike361.cct.lsu.edu
• mike361.cct.lsu.edu

17
A total view of totalview
18
References

• Profiling and Debugging Tools, Chona S. Guiang,
  TACC
• PE Operation and Use, Vol 1, Vol 2, IBM