Grid Applications and Performance

1
Grid Applications and Performance

• Jennifer M. Schopf
• Argonne National Lab NeSC
• Brian Tierney
• Lawrence Berkeley National Lab
• Nov 15, 2005

2
Performance and the Grid

• Applications are being adapted to run over
  multiple administrative domains in a coordinated
  manner
• Rarely achieve even a fraction of the performance
  possible with their underlying systems
• Users dont know what performance could be
  achieved
• No data from which to formulate estimates of
  baseline performance
• Cant compare current application performance
  with that achievable through some degree of
  tuning

3
Grid Performance Workshop 2005

• June 2005, Edinburgh, 45 participants
• Application scientists and tool developers
• Developed a set of recommendations for future
  work
• This BOF will allow further discussion between
  tool and application stakeholders to forward
  performance work in Grids.

4
Workshop Goal

• Tool builders to better understand application
  needs
• Applications to better understand what tools are
  available, pros and cons

5
What we asked theapplication scientists to tell
us

• What are the performance criteria that are most
  important to you/your users?
• How are the hardware resources/services
  monitored? (ie. how do you know if the resources
  are up?)
• How do you detect and manage failures? (eg. how
  do you manage file transfer failures or lost jobs)

6
Applications 2

• What support tools do you use to manage the
  performance of your application
• What off-the-shelf tools are you using?
• What tools have been developed specifically for
  your application?
• What other hacks do you use to monitor/manage
  performance?
• If you could request a new performance managing
  tool what would it do?

7
Meeting Conclusions

• A mismatch exists between application scientists
  and tool developers.
• Application developers want
• simple tools
• for basic problems
• that work reliably
• Tool builders are (often) funded to supply
• complex solutions to novel problems
• Short-term and long-term needs vary strongly
• Tool research often focuses on high-level
  services
• resource brokering, replica location,
  metacatalogues
• Higher level services remain secondary
  requirements for applications (today)

8
Conclusions cont.

• Performance simply is not on the critical path
  for many application projects
• Applications that struggle to get code to execute
  correctly simply do not consider whether they are
  using resources efficiently or performing well
• Reliable solutions are clearly needed
• Preference for reliable solution instead of
  more-performance oriented approach that fails
• Clearer analysis of user requirements is needed
• The needs of application scientists and tool
  builders needs to be clarified
• Additional two-way communication is needed.
• Tool builders need to know more about application
  needs
• Applications should be informed of current tools
  and their uses

9
Recommendations

• Survey application requirements
• Paper surveys or interviews are needed to better
  understand requirements
• Catalogue available tools
• Encapsulate existing catalogues
• Feedback on the entries were enabled
• Identify key areas in Grid performance research
• Short-term and long-term goals of this community
  should be defined, and these recommendations
  should be made available to the appropriate
  funding bodies

10
Recommendations cont.

• Encourage joint application/tool developer
  projects
• Efforts should be made to promote joint projects
  in order for either community to reap the
  benefits of the other
• Identify funding
• Discussions should be held with funding bodies to
  determine how best to support multicountry and
  application scientists / tool developer projects
• Expand community outreach
• Further discussion with, and between, tool and
  application scientists is needed- like this BoF!

11
What does thishave to do with you at SC?

• My favorite SC networking story is the group who
  came to the Portland (1999) SCinet helpdesk and
  complained that their application was running
  very slowly.  They said it ran great in their
  midwest university) lab, but they were seeing
  some 50 milliseconds of latency between the
  midwest and Portland and could we please fix the
  network immediately?
• Mismatch between expectations, knowledge gap as
  well!

12
Horror Stories 2

• A group I'm working with at my national lab was
  absolutely sure that there was a wide area
  network problem because they were only getting 40
  megabits/sec to Seattle and back, but inside
  their lab they were getting over 2.5
  gigabits/second.  Turns out their TCP offload
  engine was only using 64 kilobyte windows, and
  even the 10 millisecond wide area latency was
  killing performance.  Last week I got to watch
  over their shoulders as they increased the window
  size and bettered their performance from 40
  megabits/second to gt5 gigabits/second by merely
  adjusting their TCP offload engine.
• NO way to debug basic performance problems!

13
So the question is-

• Are there applications here that might find this
  interesting?
• Have other people run into this kind of issue?
• Would you like to see another Performance
  Workshop?

14
For More Information

• Jennifer Schopf
• jms_at_mcs.anl.gov
• www.mcs.anl.gov/jms
• Performance Workshop Report
• www.mcs.anl.gov/jms/GPW2005/report.pdf
• Thanks to
• NSF Division of Shared CyberInfrastructure and
  the Division of Computer and Network Systems
• Joint Information Systems Committee (JSIC)
• Microsoft

15

• How can we verify results on foreign systems?
  what software is there to deal with this? How can
  an it shop deal with this?
• Does this relate to provenance?
• This is really 3 problems
• Numerics
• Will the application complete?
• How do I predict how long this will take?
• Will this really take 3 weeks if I run it
  elsewhere?
• Need for a general infrastructure?
• In web services youve got many levels between
  you and your end point

16
Numerics Issue

• Different answers on different platforms/compilers
  how do you pre-vet systems to validate ahead
  of time?
• This is not a Grid specific problem but it may
  be made more difficult if youve got an anonymous
  pool
• Suggestion have communities pool resources but
  resources change too much to make this feasible
  perhaps
• Are we just talking about a resource discovery
  problem? Or a configuration management issue?
• Currently this is one reason why applicatoin
  tools are statically linked

17

• A way to be notified that a system has changed?
• Need for a standard benchmark for this type of
  problem?
• Benchmarks are often inflammatory
• Heres a set of tests used to verify ?
  capability benchmark
• BOINC has done significant testing on results

18
Bringing together appls folks and tools

• It takes multiple tools to get a full picture
  work with open source codes and relationship with
  developers paramount
• One application to multiple tool developers
• (Valerie Taylor work at Oak Ridge as example)
• Perhaps asking each Tool builder to bring an
  Application buddy to the meeting?
• What about the libraries community?

19

• What kind of specs are there?
• What to do about getting credit for making
  tools reliable?
• Gap between whats funded/publishable and what
  application folks must have development vs
  research