Introductory course EGEE Grid Security HunGrid Virtual Organisation

1
Introductory courseEGEE Grid Security HunGrid
Virtual Organisation

• Norbert Podhorszki
• MTA SZTAKI

EGEE is funded by the European Union under
contract IST-2003-508833
2
Acknowledgement

• This tutorial is based on the work of many
  people
• Fabrizio Gagliardi, Flavia Donno and Peter Kunszt
  (CERN)
• the EDG developer team
• the EDG training team
• the NeSC training team
• the SZTAKI training team

3
Content

• EGEE
• Introduction
• Components of the infrastructure
• HunGrid Virtual Organisation
• virtual organisations in EGEE
• the HunGrid VO
• How to join?

4
The Grid Vision
The Grid networked data processing centres and
middleware software as the glue of resources.
5
What do we expect from the Grid?

• Access to a world-wide virtual computing
  laboratory with almost infinite resources
• Possibility to organize distributed scientific
  communities (Virtual Organisation)
• Transparent access to distributed data and easy
  workload management
• Easy to use application interfaces

6
CERN Data intensive science in a large
international facility

• The Large Hadron Collider (LHC)
• The most powerful instrument ever built to
  investigate elementary particles physics
• Data Challenge
• 10 Petabytes/year of data !!!
• 20 million CDs each year!
• Simulation, reconstruction, analysis
• LHC data handling requires computing power
  equivalent to 100,000 of today's fastest PC
  processors!

Mont Blanc (4810 m)
Downtown Geneva
7
The EGEE Project

• EU funded project (04/2004 03/2006)
• EGEE offers the largest production grid facility
  in the world open to many applications (HEP,
  BioMedical, generic)
• Existing production service based on LCG (derived
  from EDG software of FP5)
• Next generation open source web-services
  middleware being re-engineered taking into
  account production/ deployment/ management needs
• Well-defined, distributed support structure to
  provide eInfrastructure that is available to many
  application domains

www.eu-egee.org
8
LCG-2/EGEE-0 Status April 2005
Cyprus

• Total
• gt 100 Sites
• 12000 CPUs
• 6.5 PByte

9
Main Logical Machine Types (Services) in LCG-2

• User Interface (UI)
• Information Service (IS)
• Computing Element (CE)
• Frontend Node
• Worker Nodes (WN)

• Storage Element (SE)
• Replica Catalog (RC,RLS)
• Resource Broker (RB)

10
User Interface

• The initial point of access to the LCG-2 Grid is
  the User Interface
• This is a machine where
• LCG users have a personal account
• The users certificate is installed
• The UI is the gateway to Grid services
• It provides a Command Line Interface to perform
  the following basic Grid operations
• list all the resources suitable to execute a
  given job
• replicate and copy files
• submit a job for execution on a Computing
  Element
• show the status of one or more submitted jobs.
• retrieve the output of one or more finished jobs
• cancel one or more jobs
• One or more UIs are available at each site part
  of the LCG-2 Grid

11
Main Logical Machine Types (Services) in LCG-2

• User Interface (UI)
• Information Service (IS)
• Computing Element (CE)
• Frontend Node
• Worker Nodes (WN)

• Storage Element (SE)
• Replica Catalog (RC,RLS)
• Resource Broker (RB)

12
Computing Element

• Computing Element entry
• point into a queue of a batch
• system
• information associated with a computing element
  is limited only to information relevant to the
  queue
• Resource details relates to the system

infoService
gatekeeper
Batch server
Grid Gate node

CPUPIV RAM2GB OSLinux
CPUPIV RAM2GB OSLinux
CPUPIV RAM2GB OSLinux
CPUPIV RAM2GB OSLinux
A CE consist of homogeneous worker nodes
13
Main Logical Machine Types (Services) in LCG-2

• User Interface (UI)
• Information Service (IS)
• Computing Element (CE)
• Frontend Node
• Worker Nodes (WN)

• Storage Element (SE)
• Replica Catalog (RC,RLS)
• Resource Broker (RB)

14
Storage Element (SE)

• A Storage Element (SE) provides uniform access
  and services to large storage spaces.
• Each site includes at least one SE
• They use two protocols
• GSIFTP for file transfer
• Remote File Input/Output (RFIO) for file access
• Storage Resource Manager (SRM) needs to take into
  account
• Transparent access to files (migration to/from
  disk pool)
• Space reservation (on demand and advance)
• File status notification
• Life time management

15
Main Logical Machine Types (Services) in LCG-2

• User Interface (UI)
• Information Service (IS)
• Computing Element (CE)
• Frontend Node
• Worker Nodes (WN)

• Storage Element (SE)
• Replica Catalog (RC,RLS)
• Resource Broker (RB)

16
Information System (IS)

• The Information System (IS) provides information
  about the LCG-2 Grid resources and their status
• The current IS is based on LDAP (Lightweight
  Directory Access Protocol) a directory service
  infrastructure which is a specialized database
  optimized for
• reading,
• browsing and
• searching information.
• the LDAP schema used in LCG-2 implements the GLUE
  (Grid Laboratory for a Uniform Environment)
  Schema

17
Main Logical Machine Types (Services) in LCG-2

• User Interface (UI)
• Information Service (IS)
• Computing Element (CE)
• Frontend Node
• Worker Nodes (WN)

• Storage Element (SE)
• Replica Catalog (RC,RLS)
• Resource Broker (RB)

18
Data Management

• In LCG, the data files are replicated
• on a temporary basis,
• to many different sites depending on
• where the data is needed.
• The users or applications do not need to know
  where the data is located they uselogical files
  names
• lfn/grid/hungrid/pnorbert/mytestfile
• Data Management services are responsible for
  locating and accessing the data
• guid3332ba41-260d-45fe-a84c-dfe9432e7c4b
• sfn//n40.hpcc.sztaki.hu/storage/hungrid/generated
  /2005-07-05/file50807d83-4678-4739-bccf-031c04d030
  8b

19
Replication Services Basic Functionality
Each file has a unique Grid ID. Locations
corresponding to the GUID are kept in the Replica
Location Service.
Users may assign aliases to the GUIDs. These are
kept in the Replica Metadata Catalog.
Files have replicas stored at many Grid sites on
Storage Elements.
Replica Metadata Catalog
Replica Location Service
Replica Manager
The Replica Manager provides atomicity for file
operations, assuring consistency of SE and
catalog contents.
Storage Element
Storage Element
20
Main Logical Machine Types (Services) in LCG-2

• User Interface (UI)
• Information Service (IS)
• Computing Element (CE)
• Frontend Node
• Worker Nodes (WN)

• Storage Element (SE)
• Replica Catalog (RC,RLS)
• Resource Broker (RB)

21
Job Management

• The user interacts with Grid via a Workload
  Management System (WMS)
• The Goal of WMS is the distributed scheduling
  and resource management in a Grid environment.
• What does it allow Grid users to do?
• To submit their jobs
• To execute them on the best resources
• The WMS tries to optimize the usage of resources
• To get information about their status
• To retrieve their output

22
A Simple Configuration
Computing Element 1
Storage Element 1
CLOSE
User Interface Resource Broker Replica
Catalog Information Service
CLOSE
Storage Element 2
Computing Element 2
23
Virtual Organisations (VO) and certificate
request process in practice
24
What is the Virtual Organisation?

• A Virtual Organisation (VO) is a collection of
  people in the same administrative domain
• The EGEE Grid works with Virtual Organisations
  (VO)
• A VO is simply a group of Grid users with similar
  interests and requirements
• who are able to work collaboratively with other
  members of the group
• and/or share resources (data, software, cpu,
  storage space, etc) regardless of geographical
  location
• Need to be a member of a VO before we are allowed
  to submit jobs to the Grid
• There are several VOs already established (Alice,
  Atlas, Babar, HunGrid, Central Europe VO)

25
Virtual Organisation for Grid Users

• I am a Grid user that wants to belong to a VO
• To be authorized to submit jobs to the grid you
  have to belong to a Virtual Organisation (VO)
• The request will be evaluated by the VO manager
  deciding if you can join or not
• To be able to register in one of the VO the user
  has to own a valid certificate, issued by one of
  the known and accepted Certification Authorities
  (CA)
• A list of supported VOs can be found here
• https//lcg-registrar.cern.ch/virtual_organization
  .html

26
The HunGrid Virtual Organisation

• A new virtual organisation (VO) of EGEE
• Created by KFKI-RMKI, SZTAKI and ELTE
• The HunGrid VO is open for anybody in Hungary who
  would like to use the LHC Grid for educational
  purpose and/or scientific research
• The HunGrid provides 7/24 Grid services
• SEQ and MPI job submission
• Storage services
• Information system
• Data management service
• Register at http//www.lcg.kfki.hu
• To register in the HunGrid VO one has to own a
  valid certificate, issued by one of the known and
  accepted Certification Authorities
• NIIFI issues new certificates for members of
  Hungarian institutes

27
The HunGrid Virtual Organisation

• HunGrid is not just a VO within EGEE
• It has new tools extending the usability of the
  Grid
• P-GRADE Portal
• to graphically develop workflow applications
• to execute applications easily on the Grid
• Mercury monitor
• to monitor parallel programs running on the Grid

28
http//www.lcg.kfki.hu
29
HunGrid EGEE magyar verziója
SZTAKI
KFKI-RMKI
ELTE
KKKI

• 250 processzor
• 3.4 TB tárterület

• 26 processzor
• 2 TB tárterület

• 5 processzor
• 1.5 TB tárterület

• 12 processzor
• 1 TB tárterület

További kiépítés Veszprémi E. (6), Miskolci E.
(30), Szegedi E. (50)
30
Get a certificate for yourself

• In order to control the accesses over the Grid,
  every user has to identify her/himself before
  submitting a job
• This is realized via the use of certificates
• The certificates are issued by the Certification
  Authorities
• Obtain a certificate from the accepted CA
• Get a certificate from the NIIF CA at
  http//www.ca.niif.hu
• The NIIF CA provides PKI (Public Key
  Infrastructure) services for the Hungarian
  academic community
• The NIIF CA is operated by the National
  Information Infrastructure Development Office,
  http//www.niif.hu

31
http//www.ca.iif.hu
32
Register in a Virtual Organisation

• You have to be the member of at least one Virtual
  Organisation in order to be able to use the Grid
• After that you can use the resources of all those
  sites which support the VO (in this case the
  HunGrid VO) where you are registered
• For the registering it is necessary to use a WWW
  browser with the user certificate installed for
  the request to be properly authenticated

33
http//www.grid.kfki.hu/Hungrid-Registrar/hungrid.
pl