Status and Current Achievements of PLGrid Project

1
Status and Current Achievements of PL-Grid
Project
Polish Infrastructure for Supporting
Computational Science in the European Research
Space

• Jacek Kitowski and Lukasz DutkaACK CYFRONET AGH,
  Cracow, Poland
• Institute of Computer Science AGH-UST
• in collaboration withPL-Grid Representatives
• Michal Turala, Kazimierz Wiatr, Marian Bubak,
  Tomasz Szepieniec,Marcin Radecki, Alex Kusznir,
  Zofia Mosurska, Mariusz Sterzel
• Piotr Bala, Wojciech Wislicki, Norbert Meyer,
  Krzysztof Kurowski, Józef Janyszek, Bartlomiej
  Balcerek, Jaroslaw Rybicki, Rafal Tylman

Cracow Grid Workshop, CGW 2009October 12-14, 2009
2
Outline

• National Grid Initiative in Poland in a nutshell
• Motivation
• Rationales and Foundations
• Status Current Results
• PL-Grid Project Current Results
• Summary

• E-Science approach to research
• Integration activities ongoing in the world

3
Demand from Community for e-Science Approachin
spite of complexity Computing, Storage,
Infrastructure

• E-Science collaborative research supported by
  advanced distributed computations
• Multi-disciplinary, Multi-Site and Multi-National
• Building with and demanding advances in
  Computing/Computer Sciences
• Goal to enable better research in all
  disciplines
• System-level Science beyond individual
  phenomena, components interact and interrelate,
  experiments in silico
• to generate, interpret and analyse rich data
  resources
• From experiments, observations and simulations
• Quality management, preservation and reliable
  evidence
• to develop and explore models and simulations
• Computation and data at all scales
• Trustworthy, economic, timely and relevant
  results
• to enable dynamic distributed collaboration
• Facilitating collaboration with information and
  resource sharing
• Security, trust, reliability, accountability,
  manageability and agility

I. Foster, System Level Science and System Level
Models, Snowmass, August 1-2, 2007
M. Atkinson, e-Science (...), Grid2006 2-nd
Int.Conf.e-Social Science 2006, National
e-Science Centre UK
4
Rationales behind PL-Grid Consortium

• The Consortium consists of five High Performance
  Computing Polish Centres representing Communities
  due to
• Participation in International and National
  Projects and Collaboration
• 35 international projects FP5, FP6, FP7 on Grids
  (50 common)
• 15 Polish projects (50 common)
• Needs by Polish Scientific Communities
• 75 publications in 5 Communities
• Computational resources to date
• Top500 list
• European/Worldwide integration Activities
• EGEE I-III, EGI_DS, EGI, e-IRG, PRACE, DEISA,
  OMII, EU Unit F3 Research Infrastructure
  Experts
• National Network Infrastructure ready
• (thanks to Pionier National Project)

5
PL-Grid Foundations Summary
Polish Infrastructure for Supporting
Computational Science in the European Research
Space Response to the needs of Polish scientists
and ongoing Grid activities in Poland, other
European countries and all over the world

• Motivation
• E-Science approach to research
• EGI initiative ongoing in collaboration with NGIs
• Milestones
• Creation of Polish Grid (PL-Grid) Consortium
  http//plgrid.pl
• Consortium Agreement signed in January 2007
• PL-Grid Project (2009-2012)
• Application in Operational Programme Innovative
  Economy, Activity 2.3 (in Sept. 2008)
• Get funded March 2, 2009 (via European
  Structural Funds)
• Consortium made up of five largest Polish
  supercomputing and networking centres (founders)
• ACK CYFRONET AGH (Cracow) Coordinator

6
PL-Grid Base Points

• Assumptions
• Polish Grid is developing a common base
  infrastructure similar to solutions adopted in
  other countries.
• Specialized, domain Grid systems including
  services and tools focused on specific types of
  applications will be built upon this
  infrastructure.
• These domain Grid systems can be further
  developed and maintained in the framework of
  separate projects.
• Such an approach should enable efficient use of
  available financial resources.
• Creation of a Grid infrastructure fully
  compatible and interoperable with European and
  Worldwide Grids.
• Plans for HPC and Scalability Computing enabled.
• Tighly-coupled activities

7
Elements and Functionality

• PL-Grid software will comprise
• user tools (portals, systems for applications
  management and monitoring, result visualization
  and other purposes, compatible with the
  lower-layer software used in PL-Grid)
• software libraries
• virtual organization systems certificates,
  accounting, security, dynamic
• data management systems metadata catalogues,
  replica management, file transfer
• resource management systems job management,
  applications, grid services and infrastructure
  monitoring, license management, local resource
  management, monitoring.

• Three Grid structures will be maintained
• production,
• reseach,
• development/testing.

8
Planned Realization of Aims
The PL-Grid Project is split into several
workpackages
9
Operational Center Tasks

• Coordination of Operation
• Management and accounting
• EGI and DEISA collaboration(Scalability and HPC
  Computing)
• Users requirements analysis for operational
  issues
• Running infrastructure for
• Production
• Developers
• Research
• Future consideration
• Computational Cloud
• Data Cloud
• Internal and ExternalClouds
• Virtualization aspects

10
Operational Center Current Results

• PL-Grid VO (vo.plgrid.pl) fully operational
• Production resources for PL-Grid users in 4
  centres (Gdansk, Kraków, Poznan, Wroclaw)
• 8kCPUs in total, 142 TB
• Infrastructure monitoring (failure detection)
  tools based on Nagios EGEE SAM in place
• Support teams, tools and EGEE/EGI-compliant
  operations procedures aimed at achieving high
  availability of resources established
• Knowledge sharing tool for operations problems
• Problem tracking tool to be released in November
• Provisional user registration procedure
• User registration portal in preparation (1st
  prototype to be released end of October)
• Accounting integrated with EGEE

PCSS
WCSS
Cyfronet
11
Operational Center Current Results

• PL-Grid VO (vo.plgrid.pl) fully operational
• Virtual production resources for PL-Grid users in
  4 centres (Gdansk, Kraków, Poznan, Wroclaw)
• Infrastructure monitoring (failure detection)
  tools based on Nagios EGEE SAM in place
• Support teams, tools and EGEE/EGI-compliant
  operations procedures aimed at achieving high
  availability of resources established
• Knowledge sharing tool for operations problems
• Problem tracking tool to be released in November
• Provisional user registration procedure
• User registration portal in preparation (1st
  prototype to be released end of October)
• Accounting integrated with EGEE

PCSS
WCSS
Cyfronet
12
Operational Center Next Plans

• User Registration
• access to any person registered as researcher in
  Poland - www.opi.org.pl
• portal being already developed (to be released
  end of October'09)
• access to resources based on
• grants giving rights to use X CPUhours and TB
  of disk space
• access services way of accessing resources in
  particular computing centre
• PL-Grid Research Infrastructure resources on
  demand allows to request machines for different
  purposes
• install a grid middleware for testing purposes
• for developers to build a testing environment
• any other research requiring grid resources
• procedures are being defined
• Enable access to UNICORE resources
• small amount of dedicated UNICORE machines
• integrate UNICORE and gLite make use of the
  same resources

13
Software and Tools Development Current Results

• Establishment of eight groups of research
  developers, 20 people from five Polish
  supercomputing centers, collaborating together to
  improve the quality of existing European grid
  infrastructure services, tools and applications
• Research and development testbed available based
  on dedicated physical and virtual machines from
  supercomputing centers
• Initial evaluation of new user tools,
  environments and grid applications developed and
  supported in Poland
• Basic extensions and improvement procedures
  proposed to various functionalities provided by
  gLite, Unicore and QosCosGrid
• Software reengineering, development and testing
  of useful user tools, such as Migrating Desktop,
  gEclipse, ViroLab/GridSpace, FIVO, Vine Toolkit,
  QCG-OpenMPI, QCG-ProActive, etc.
• Software quality management for new services and
  tools to be deployed and maintained in the second
  phase of Pl-Grid
• Active links to new user communities in Poland
  interested in advanced computing technologies in
  order to collect additional requirements and
  scenarios
• www.plgrid.pl/ankieta

14
Software and Tools Development In numbers

• Research developers 20
• Main research teams 8
• Supercomputing centers involved 5
• Dedicated physical and virtual machines 30
• Tested infrastructures
• EGEE, DEISA
• New infrastructures and extensions
• QosCosGrid, GridSpace
• Current applications under integration and
  testing procedures 10
• Current user tools under integration and testing
  procedures 5
• Active links to new user communities in Poland
  www.plgrid.pl/ankieta

15
Software and Tools Development Next Plans

• Closer collaboration with user and developer
  communities representing various scientific
  domains (based on feedback received from our
  questionnaire)
• Adaptation of virtualization techniques improving
  usability, fault tolerance, checkpointing and
  cluster management
• Fancy web GUIs based on Liferay and Vine Toolkit
  accessing existing and extended e-Infrastructure
  services
• Performance tests of various large scale
  cross-cluster parallel applications using
  co-allocation and advance reservation techniques
• More analysis of grid management, monitoring and
  security solutions
• Common software repository and deployment rules
• Example services, tools and applications
  available as virtual boxes with different
  configurations
• Main checkpoints and timeline

We are here
IX 09
VI 09
IX 10
IV 11
I 12
16
Users Support and Training

• General task
• Running Help-Desk and users training and support
  tasks
• Supporting how to effectively exploit developed
  infrastructure including selection of right
  applications for users problems and seamless
  running computation on the grid.
• Pl-grid is an user oriented infrastructure,
  support and training team will be close to the
  users.
• At the moment
• Support Team is organized
• Internal training for support team already begun
• Selection of licensed application required by the
  users community to be installed on the whole
  PL-Grid infrastructure
• First training for the users delivered.
• Plans
• Many, many trainings for beginners and advanced
  grid users

17
PL-Grid Security Group Current Results

• Defined security guidelines for architecture
  planning
• Defined security policy for local sites
  installations
• Incident reporting systems review to choose best
  solution for PL-Grid
• GGUS -  Global Grid User Support
• RTIR - Request Tracking for Incident Response
• DIHS - Distributed Incident Handling System
• Architecture and prototype version of on-line
  user mapping and credentials distribution system.

18
PL-Grid Security Group Next Plans

• Establish NGI-wide CERT
• Introduce security policies into local sites
• Develop and deploy local sites security policy
  conformity monitoring system
• check if software (kernel, services) is up to
  date
• check listening ports, firewall rules
• monitor suid/sgid lists and integrity
• Develop distributed alert correlation system
  build on top of network and host sensors.

19
Summary Activities

• Short term
• Establish PL-Grid VO using Partners local and
  EGEE resources
• Provide resources for covering operational costs
• Provide resources for keeping international
  collaboration ongoing
• Long term continuously
• Software and tools implementation
• Users support and traning
• Provide, keep and extend the necessary
  infrastructure
• Be prepared / work on approaching paradigms and
  integration development
• HPCaaS and Scalable Computing (Capability and
  Capacity Computing)
• Clouds Computing (internal-external, computing
  clouds, data clouds)
• SOA paradigm, knowledge usage
• Future Internet as defined by EC in
  Workprogramme

20
httpplgrid.pl
Thank you