Requirements and Services: The Application Point of View

1
Requirements and Services The Application
Point of View

• Jennifer M. Schopf
• Argonne National Lab
• (Steven Newhouse, OMII)

2
The Never EndingRoad Trip

• Jennifer M. Schopf
• Argonne National Lab
• (Steven Newhouse, OMII)

3
What If

• We had a Grid based around scientists, not
  scientists using what they had to from the Grid
• Tools fit into a users normal environment with
  little or no adaptation in work style
• There were simple, reliable, persistent services
  that were easy to use
• So why arent we here already?

4
The Need

• Several projects developing or delivering
  middleware without clear guidance
• Users and uses have grown and changed
• Middleware has grown and changed
• (warning needs to be spread of Stevens move and
  that fact that Im spending a year in the UK .)
• Vital need for information exchange at this
  juncture

5
And Our Guest Today Is

• Many current projects, weve focused on those in
  the UK due to OMII mission (and lack of US
  understanding into these)
• Current application developers with some Grid or
  Web Services experience
• Those with software that might be of broader use
  or interest
• Those who have expressed dissatisfaction with
  current tools

6
So Far

• Oxford Security Workshop
• Networking for Non-Networkers Workshop (NNFN)
• T. Cooper-Chadwich, Southampton, gYacht
• S. Cox, Southampton, GeoDise
• M. Daw, Manchester, AG
• M. Giles, Oxford, gViz
• C. Goble, Manchester, MyGrid Integrative
  Biology Project
• S. Lloyd, Oxford, eDiamond
• J. MacLaren, Manchester, UoM Broker
• A. Martin, Oxford, ClimatePreciction.NET
• M. McKeown, Manchester, OGSIlite and WSRFlite
• S. Pickles, Manchester, TeraGyroid GRENADE
• A. Porter, Manchester, Reality Grid
• A. Rector, Manchester, CLEF
• M. Rider, Manchester, eViz

7
Still To Come

• Edinburgh
• Glasgow
• University College London
• Imperial College
• Southampton (again)
• and others by request

8
The Results So Far

• Need for Training
• Security
• Functionality
• Jobs
• Data
• What isnt mentioned
• What tools should look like
• Infrastructure/Operations

9
Training

• Grid vision still needs to be sold
• What do these tool give me over SSH, scp?
• What if I dont want to stop using my magnifying
  glass to read x-rays?
• Still need basic common practices to be written
  for user, developers and admins
• Web service basics
• Firewalls
• Builds and packaging
• No surprise Communication is still a large
  unsolved problem in Grid computing

10
Security Whats Old News

• If it isnt easy users arent interested
• All users hate firewalls, all system
  administrators love them
• Anonymizing data is hard
• Still need a lot of information sharing
• How fire walls interact with Grid/Web services
• Security audits

11
Security Whats Surprising Us

• User focus on need for data integrity not
  authentication/authorization
• Time and again this was mentioned
• Delegation seems to be the next big question
• GT2-style delegation needed in a services world
• No one has an agreed upon solution yet

12
Questions?

• Need for Training
• Security
• Functionality
• Jobs
• Data
• What isnt mentioned
• What tools should look like
• Infrastructure/Operations

13
Functionality

• Prediction of Pete Beckman (TeraGrid)
• All users want is SSH, scp, and top

14
Functionality

• All users talk about is job submission and file
  transfer capabilities
• When asked about trouble spots they also want
  tools to tell how tools are progressing
• Occasionally would like an application specific
  service, like a viz tool
• Many other tools are on the horizon, but they
  simply dont seem to be on the 6 month horizon
  for the users weve spoken with to date

15
Job Submission No Surprises

• Want simple, dependable run my application
  interfaces
• This was identified at GF1!
• Only resource discovery is small
• How many nodes have a matlab license?
• NOT which cluster should I use?

16
Job Submission Urgent Needs

• Tools to understand errors while a job is
  running- something stopped, where and why?
• TeraGyroids use of SSH for debugging
• Need for Global Job Unique ID
• What to do when a job fails-
• Resubmit or ignore?
• Workflow issues
• Steering

17
File Transfer

• People seem pretty happy with GridFTP
• Some reliability (RFT) would fill out rest of use
• This needs 3rd party transfer (delegation)
• Some projects starting to work with provenance
  issues or access to databases
• Still issues with performance and making sure
  background infrastructure is all as it should be
  more later

18
What (These)Users Arent Talking About

• Notification except for job progress tracking
• Registries or resource discovery
• QoS, reservations, brokering, advanced scheduling
  techniques
• Job migration, checkpointing
• Accounting and pricing (but were talking with
  users, not admins so far)
• Data replication, migration
• Instruments

19
And Why We Think This Is

• A gap still exists between the computer science
  research and tool building community and the
  average user
• Large difference between short term needs and
  long term planning-
• Most users are still trying for basic
  functionality and dealing with todays hurdles
• Most researchers are looking at the greener
  pastures a few years out

20
Questions So Far?

• Need for Training
• Security
• Functionality
• Jobs
• Data
• What isnt mentioned
• What tools should look like
• Infrastructure/Operations

21
Users View of Tools

• Users have strong opinions on tools what a
  surprise!
• Mostly known problems, but the prioritization of
  certain aspects wasnt known

22
Tools Should Be

• Vertical solutions
• End to end use cases, not horizontal pieces that
  dont work together
• Simple
• One job, one tool think unix!
• Work easily for the 80 case, and rest is
  possible if needed
• Ease of use/installs
• Bundle all together so you have entire use
  together
• Dont reinstall things I already have

23
Sometimes There Is No Best Tool

• Same workspace may have many solutions, each with
  slightly different attributes, all valuable
• BPEL vs SCUFL
• Cluster monitors
• Users should not be forced to use a tool from
  another environment if there is no underlying
  scientific need for it
• However, we still need to try to avoid
  replication of tools

24
Composable Functionality

• Lego blocks of basic functions
• Shims to fit between where needed
• API mismatches
• Data translation
• Interfaces to legacy code
• SOAP Lab (wraps command line to look like SOAP)

25
Interfaces

• Need simple APIs at the user level
• eg. SAGA-RG (but then we knew this)
• User API might sit a layer above standard tool
  APIs to mitigate upgrade effects
• eg. HiCog
• The API a user sees and the API the
  infrastructure not only can but should be
  different different goals

26
Environments

• Tools need to fit in with existing user comfort
  zone
• Biologists like Perl
• CFD folks like MatLab
• HEP (EGEE) are used to Python
• This is a sys admins and tool developers
  nightmare but for usability its a must

27
Questions?

• Need for Training
• Security
• Functionality
• Jobs
• Data
• What isnt mentioned
• What tools should look like
• Infrastructure/Operations

28
Builds and Upgrades

• Need for a reproducible build
• Hands off process, works every time
• Verification tools
• Need better understanding of effects of upgrades,
  or else users dont want it
• What will change, what will be affected
• Tools are being used off label
• Tool for usecase A in common use for usecase B
• Scalability becomes an issue
• New/different functions needed

29
Understanding System Stability

• Need for basic tools to verify functionality and
  performance
• I cant transfer my files today
• Whats broken?
• What changed?
• How do I fix it?
• Why couldnt someone find this before me?
• Strong needs for quality assurance tests on all
  platforms clusters, networks, AG, etc.

30
System Tests

• Often system tests dont look like current
  applications
• Tests for firewall functionality dont include
  checks for all ports in current use
• System benchmarks dont look like my
  application
• Ping tests arent enough to assure that a GridFTP
  transfer will work
• Need for better testing, verification- for the
  user, and even by the user!

31
WebMD for Applications

• Basic diagnostics needed for users
• Q - Im trying to transfer a 1 gig file between
  A and B and cant
• A- Is your cert ok? Heres how to check Y/N,
  if Y
• A Is the route between youre hosts up? Heres
  how to run traceroute for your system

32
Topics of Conversation

• Need for Training
• Security
• Functionality
• Jobs
• Data
• What isnt mentioned
• What tools should look like
• Infrastructure/Operations

33
Our Conclusions So Far

• Delegation
• Job tracking
• Easing tool use
• Wrappers
• Composability
• Verification and instability analysis
• Diagnosis

34
The Point

• We cant say it any more simply
• Grid tool developers must continue to talk and
  interact with application scientists without
  them, we are nothing

35
How To Find Us

• Jennifer M. Schopf
• jms_at_mcs.anl.gov
• Will be in Edinburgh
• Steven Newhouse
• s.newhouse_at_omii.ac.uk