GridPP2

1
GridPP2

• Data Management Work Area Part 2
• Mass storage local storage mgmt

J Jensen j.jensen_at_rl.ac.uk
2
Data Management Work Area

• Metadata management
• See Gavins talk
• Local Storage Management
• Cleanup temporary space
• Reservations?
• Mass Storage Management
• Enable Grid access to storage at UK Tier 1 and 2

3
Overview
Local Storage
Metadata
Deployment
Support
ARDA
SRM2
EGEE
WSRF?
dCache
Storage management
4
Common

• Links to external middleware groups
• Common support deployment requirements
• Integrate with deployment software
• LCFG-ng / Quattor / Manual install
• Not contractually obliged to deliver
• Must use tools from EGEE JRA1/SA1 (tar/make)
• Can we use EGEE CVS ?
• Savannah
• Work with OMII

5
Storage Group

• Form a Storage Group within GridPP2
• Already set up!
• Requirements, evaluation, development,
  deployment, support, evaluation
• Work with storage staff at sites
• RAL, Bristol, Edinburgh (, Glasgow)
• Will set up web site, mailing list soon
• Regular phone conferences

6
Storage Management Pf2

• Need to support MSS at UK Tier 2 sites
• Evaluate existing solutions in the UK
• Support Tier 1 first, then Tier 2
• Prioritize Tier 2 in order of storage committed
• or ease of implementation
• SRM 1 required
• Move to SRM 2 later?

7
Storage Management Pf3

• SRM version 1 implementations
• Enstore, dCache, CASTOR, EDG SE,
• Pick one to deploy initially
• Initial focus on Tier 1
• Pick one to develop and support
• Which is best for supporting storage systems at
  Tier-2 sites in the UK?

8
Local Storage Management Pf2

• Evaluations
• Clean up temporary files
• Reservations difficult to support
• May be necessary to integrate with WLMS
• May be necessary to integrate with batch systems

9
Local Storage Management Pf3

• Various ways to provide partial functionality
• EDG Zambo
• dCache
• Condor
• Integrate something with batch systems
• Initial deployment
• dCache
• For batch systems, focus on PBS initially, then
  LSF
• Evaluate which is best for long term
• Additional developments may be required

10
Requirements input

• Other GridPP areas
• EGEE Architecture Group
• HEPCAL, in particular HEPCAL I, ARDA,
• EGEE NA4?
• Security - JRA3
• SRM group

11
Before GridPP2

• EDG SE developments
• More work on SRM 1
• Currently testing with dCache client and GFAL
• Improve metadata implementation
• Integrate VOMS, deploy ACL stuff
• Set up support structures (started already!)
• Set up EGEE tracking

12
Support

• Work with Tier-1 and Tier-2 support staff
• Use Bugzilla? Or Savannah?
• Bugzilla to be set up by Edinburgh if reqd
• Provide mailing lists for
• Announcements
• Help integration
• Perhaps internally, for developments/discussions

13
DeliverablesDeployment and Support

• Releases to match EGEE release m and LCGn,
  m?1,2, n ?3,.
• PM03 Initial deployment
• dCache for LSM
• EDG SE (and SRM?)
• PM05 Release for LCG3
• PM07 Release for EGEE release 1
• PM16 Release for EGEE release 2
• PM26 Final post-EGEE release
• For EGEE follow-on project

14
DeliverablesEvaluation and Development

• Evaluations, developments design
• PM08 Storage only Report on mass storage used
  in UK
• PM10 Report on integration with EGEE release 1
• PM14 Local Storage only Report on local
  storage management
• PM20 Report on integration with EGEE release 2

15
DeliverablesManagement and timeline

• Plans, report on interactions with EGEE,
  e-Science, OMII
• PM01 Initial project management plan
• Risk analysis, plan for evaluating and supporting
  storage systems, etc
• PM12 Annual report
• PM24 Annual report
• PM36 Final project report
• Describing delivered and usable software, EGEE
  and LCG deployment and evaluation, dissemination,
  contributions to intl standards, deviations from
  plans,

16
DeliverablesIntegration with EGEE

• Higher level EGEE tracking and integration
  (architecture, security, )
• PM01 Report describing requirements arising from
  EGEE Architecture
• PM11 Report describing requirements arising from
  EGEE Architecture
• PM18 Integration plan for EGEE follow-on

17
SRM 1

• Mass storage control protocol
• GridFTP for data transfer
• SRM 1 reqd for interoperability with DoE Labs
• Some interoperability problems between various
  implementations
• Parts of protocol open to interpretation

18
EGEE

• Integrate directly with
• JRA1 middleware
• Data Management cluster
• Architecture Group
• SA1 support and deployment
• Also watch
• JRA2 QA
• JRA3 security
• And?
• NA2, 3, 4 dissemination, training, use cases

19
LCG

• LCG testbed match LCG3
• Future LCG milestones less well defined (or
  secret)?
• ARDA
• Can provide early prototypes
• What is early pre-GridPP2?
• For Mass Storage, SE to interface with RM
• For Local Storage Management, dCache
• GFAL

20
LCG GFAL

• LCG (will) decide to use GFAL the Grid File
  Access Library
• It was decided to interface to EDG SE using SRM 1
  interface
• For now using EDG RM

POSIX interface
SRM 1 client
EDG SRM
EDG 2.1 Storage Element
Mass Storage
21
Web services

• Already using web services for mass storage
  control interfaces
• Need HTTPS-compatible delegation
• Need to evaluate WSRF

22
SRM version 2

• Protocol (very nearly) defined
• More advanced optional functionality
• Volatile, Durable, and Permanent files
• Volatile, Durable, and Permanent space
• Evaluate need/requirement for this as we go along
• SRM Group to be in GGF as GSM

23
Dissemination

• Work with NA2
• Work with e-Science programme
• We used AHM 2002, 2003 to promote GridPP, EDG SE
  (posters, papers)
• We will be present at AHM 2004 (September)

24
Non-LCG liaison

• e-Science Core programme
• Liaise with e-Science GSC, ETF, STF,?
• ADS used by diverse scientific community
• EGEE WP formerly known as 10
• WP formerly known as 9 is not in EGEE
• Babar
• No Grid plans yet

25
EGEE DICOM server support
The Grid
Metadata
Metadata
Storage Element
WP10 DM2
Store key
Store patient metadata
Encrypt, anonymise
DICOM Server
Access control on metadata required different
ACLs for different types of metadata