ScotGrid

1
Introduction
2
Expression of Interest

• Universities of Aberdeen, Dundee, Edinburgh,
  Glasgow (lead),
• Heriot-Watt, St. Andrews, Stirling, Strathclyde
• wide variety of disciplines
• computing-enabled research
• portals
• training
• accounting
• infrastructure management
• distributed computing provision
• three-year project
• approximately 5m from SFC
• comparable equipment resources from partners

3
Objectives

• Added Value a grid-enabled software
  infrastructure, providing added value to
  institutional hardware
• Integration integrated with existing projects in
  the UK, Europe and internationally
• Leverage standardised middleware will be
  exploited
• Virtual Communities nurture and support
• Outreach and Training reach new research
  communities
• (Scottish companies will also be able to benefit)

4
Development Plans

• Pilot phase prototype grid service
  (www.scotgrid.ac.uk)
• Boost RD at NeSC (www.nesc.ac.uk)
• Prototype e-infrastructure in operation for four
  years
• Growth support for growing number of activities
• Evolution technology development required
• Challenges
• Grid accessible to non-computing researchers
• development of discipline-specific portals
• information management
• integration of different authentication systems

5
Institutional Resources

• Substantial investments in computing resources
• ScotGrid will integrate these
• Dundee 1m life sciences HPC facility (currently
  200 processors, gt 100TB), upgrade planned
• Edinburgh 2m Computing and Data Facility (ECDF)
  planned
• Glasgow 1m dedicated grid facility (first
  components arrived this week) 1 MSI2k, 100 TB
• Stirling 850k in central servers
• Aberdeen, Heriot-Watt, St. Andrews and
  Strathclyde are supportive

6
Accounting and Monitoring

• Ensure fairness and balance of the institutional
  resources between the different research
  communities
• Effective distributed accounting provides the
  basis by which partners can have confidence in
  resource sharing and in additional resource
  provision and consumption
• Annual ScotGrid accounts provided to SFC will
  enable each partner institute to determine their
  usage charges, providing a basis for long-term
  sustainability via research grants and the
  rolling equipment programmes in each university

7
SFC Support

• Institutional equipment investments are
  comparable to the support required from SFC for
  staff to develop and realise the full potential
  of ScotGrid
• In addition, support will be needed for meetings,
  travel, etc
• The full economic cost over three years, starting
  July 2007, is estimated at around 5m
• Project Management (1.2 FTE) A dedicated project
  director plus a fraction of the PI to coordinate
  the complex technical developments and
  specialised discipline requirements
• System Management (4 FTE) Local technical
  management of the key facilities forming part of
  ScotGrid
• Grid Operation (6 FTE) A distributed team of
  technicians to achieve agreed levels of service
  across the entire ScotGrid infrastructure
• Accounting Support (1 FTE) Development of
  accounting support to ensure fair and
  well-managed use of ScotGrid facilities
• Portal Development (4 FTE) A team of specialists
  to develop the specialised portals required by
  client disciplines
• Training and Outreach (2 FTE) A team of
  specialists to determine the development and
  training needs of client disciplines

8
Sustainability

• There is a long-term demand for a Grid over the
  next ten years
• Medium-term sustainability will depend upon close
  co-operation
• The project is designed to be fully inclusive
• The ScotGrid resource will help sustain the
  leading position Scotland has in e-Science and
  Grid research, and ensure that the Scottish
  research community is well positioned for future
  research funding opportunities
• continued support via grant applications for new
  research growing out of the project and from the
  rolling equipment programmes in each University
• A summary report of research income generated
  and/or supported using the infrastructure will be
  compiled in 2009 to enable SFC and the partners
  to determine the most appropriate sustainable
  funding model from 2010 and beyond

9
SFC feedback (17/7/06)

• The proposal was considered outwith the
  discussion of proposals to the SRDG periodic
  competition
• The Panel considered that the proposal met a
  strategic need for Scotland
• It would be ideally suitable for consideration
  for funding under the Enhancing Excellence
  (pooling) funding stream of SRDG
• Two key issues for the proposal
• (i) what parts of infrastructure are provided
  already at the UK level
• (ii) be clear on the pan-Scottish dimension

10
SFC further feedback (26/7/06)

• A. need to understand the relationship and
  synergy with the Informatics Pooling bid
• B. Highlight the "service" aspects of the
  ScotGrid proposition
• C. The outcome of the proposal would be for
  assessment next year
• D. Institutions involved should consider how
  they can contribute
• E. Scottish Enterprise could be involved
  particularly in knowledge transfer and core
  technology aspects
• F. Approach Abertay w.r.t. their formal
  training (linked to informatics pooling)
• G. New senior appointments in this area should
  be considered to build up the area of Grid
  research

11

• A further set of points were identified
• 4. Industrial liaison should be incorporated -
  testbed encouraged
• 5. A gap analysis should be performed
• 6. Investigate Grid-associated studentships
• 7. Service existing research pooling efforts
• 8. An overview document should be prepared
• 9. Indicate how many users are anticipated

12
Broader Issues

• a. Technical issues should be discussed (in the
  appendices)
• b. International review committee will judge
• c. Develop links with co-funders (e.g. Research
  Councils)
• d. EPSRC must be approached re supporting
  academic investments. (PPARC are already aware).

13
Benefits

• Establishment of a Research e-Infrastructure
  leading to
• Expansion in terms of partners and services
• Robust operations and management procedures
• Interoperability with other national and
  international e-infrastructures
• Integration of key data sources
• Improved understanding of data provenance
• Improved and measured ScotGrid reliability
• Supported research across a range of domains
• Access by non-Grid focused researchers
• Service definitions for a sustainable
  infrastructure
• Industrial partners testing/utilising Grid
  services

14
Timeline

• In terms of defining a timeline
• i. need to identify contacts at all Institutes
• ii. SFC (David Gani) will meet with partners on
  November 6th
• iii. focus on enhanced service aspects and
  identifying the core Grid technologies
• Three meetings currently planned
• Today Edinburgh
• November 6th Glasgow
• End-December TBC
• Internal aim (for discussion)
• appendices by the end of the year
• proposal by SFC-requested timescale

15
Summary

• Expression of Interest submitted at end of May
  2006
• Draft proposal developed up to mid-May 2006 was
  used to guide internal discussions
• Received prompt, positive feedback to EoI from
  SFC in July
• Need to build collaboration and review draft
  proposal in light of SFC feedback and todays
  inputs

16
Grids and Business key points

• Trust - Not used to sharing resources
• Security - Sensitive data with sensitive
  applications
• Business models what can be charged for as a
  service
• Guaranteed QoS Service Level Agreements
• Accounting - tracking resources usage in
  multi-admin context
• Standards to encourage long-term investment
• Applications need to support legacy
  applications
• Portability across multiple platforms and
  implementations
• Open source support robust reference
  implementation
• Software license management how to generate
  revenue in a grid context

EGEE-II provides an excellent framework for
collaborating with business on these subjects
17
Background
18
News First Meeting of ScotGrid Partners to
develop Pan-Scottish Grid
News Glasgow prepares with new Grid servers
19
Monday
Wednesday
20
Accounting

• Past CPU performance
• 3,061,049 CPU hours
• 383,858 completed jobs.
• gt 100 users, gt30 VOs
• Need to
• Publish data automatically
• Agree the conversion rates
• CPU, disk, manpower
• Agree internal charging

June 2002 June 2006
21
(No Transcript)
22
PP Grid Overview

• Aim by 2008 (full years data taking)
• CPU 100MSi2k (100,000 CPUs)
• Storage 80PB
• - Involving gt100 institutes worldwide
• Build on complex middleware being developed in
  advanced Grid technology projects, both in Europe
  (Glite) and in the USA (VDT)
• Prototype went live in September 2003 in 12
  countries
• Extensively tested by the LHC experiments in
  September 2004
• 197 sites, 13,797 CPUs, 5PB storage in September
  2005
• Today 177 active sites, 28,574 CPUs, 45PB storage

23
Middleware
Requirement
Storage Element
Basic File Transfer
Reliable File Transfer
Catalogue Services
Data Management tools
Compute Element
Workload Management
VO Agents
VO Membership Services
DataBase Services
Posix-like I/O
Application Software Installation Tools
Job Monitoring
Reliable Messaging
Information System

• 15 Baseline Services for a functional Grid
• Scottish (and National) Grid Service will
  rely upon gLite components
• This middleware (now at v3.0) builds upon
  VDT (Globus and Condor) and meets the
  requirements of all the low-level, basic
  scientific use cases
• Green (amber) areas are (almost) agreed as part
  of the shared generic middleware stack by each of
  the application areas
• Red are areas where generic middleware competes
  with application-specific software.

24
Middleware
Generic User Services
25
What was ?

1. Glasgow/Edinburgh/Durham/ (Dundee) Project
2. Compute-intensive jobs performed at Glasgow and
   Durham
3. Data-intensive jobs performed at Edinburgh
4. First applications using the Grid. Meeting real
   requirements of Grid applications currently
   Particle Physics, Bioinformatics, Computing
   Science, Electrical Engineering
5. Glasgow leading RD in Grid Data Management,
   Security, Portals
6. Edinburgh leading User Training, RD in
   Networking and Storage
7. Management part of a worldwide Grid
   infrastructure through GridPP and EGEE

26
What are the Grid challenges?
2. Software efficiency
1. Software process
3. Deployment planning
4. Link centres
10. Policies
5. Share data
Data Management, Security and Sharing
8. Analyse data
9. Accounting
6. Manage data
7. Install software
27
(No Transcript)
28
Summary

• ScotGrid is part of an evolving National and
  International Grid
• NGS, EGEE and OMII-UK are important partner
  organisations
• Deployment, middleware and application support
  will each raise issues for ScotGrid
• External developments place constraints on the
  hardware and software
• Timeline is right for spin-off

29
Planning
30
Work Plan

• We are developing a more detailed work plan
• International leadership of the participating
  departments, their proven ability to collaborate
  successfully, their needs for large-scale
  computing, and their stated intentions to exploit
  a Scottish Grid
• Each discipline has its own application needs,
  pressing timescales, and expectations of
  large-scale computing
• The project will provide the framework needed for
  synergy and economies of scale
• Deployment plans to
• expand the user base
• include new provider sites
• create a training programme
• enhance current service provision and establish
  user support
• This will be integrated with the work of the UK
  and European Grid services
• A top-down approach will draw on institutional
  strengths and strategic plans for computing
• A bottom-up approach will focus on requirements
  of each discipline, building unified services
  across Scotland

31
Stakeholders
32
Project Management
33
Risk analysis
34
MiddlewareBaseline Services
35
External Relations
36
Management
Need a Shadow Management Structure(to prepare
proposal)