Accomplishments in Security, Data Management and Monitoring

1
Accomplishments in Security, Data Management and
Monitoring

• D. Olson, LBNL
• SuperComputing2003SciDAC Booth

2
Contents

• Reminder of PPDG
• Accomplishments in
• Security
• Data Management
• Monitoring
• Talks covering other areas with PPDG
  accomplishments
• For common software stack integration see talk on
  VDT by Miron Livny, U. Wisconsin, Wed. 400pm.
• For job scheduling and production grids see
  Grid2003 talk by Ian Fisk, Fermilab, Thurs.
  1100am.

3
Particle Physics Data GridCollaboratory Pilot
Integrating data grid technologywith HEP and NP
applications to enable world-wide scientific
data analysis, in partnership with GriPhyN,
iVDGL (Trillium).

• PIs - Mount, Livny, Newman
• Coordinators - Pordes, Olson
• HEP
• ATLAS
• BaBar
• CMS
• D0 (CDF)
• NP
• JLab
• STAR (PHENIX)
• ALICE
• CS
• Condor
• Globus
• SDM/SRM
• SRB

4
Elements in this presentationaccomplishments not
covered here

.
O
G
2
N
R

T
S
O
R
E
O
Request
Workflow or
I
E

V
C
O
T

Community
Application-
Consistency Services

I
C
N
I
A
Interpretation
Request
L
T
I
F
I
C
A
Authorization
Specific Data
(e.g., Update Subscription,
V
A
C
I
I
and Planning
Management
C
R
U
M
E
L
Services
Discovery Services
Versioning, Master Copies)
E
T
L
E
P
O
E
Services
Services
L
P
R
S
P
V
D
I
S
I
O
A
V
T
C
C
E
L

G
R
L
1
S

N
O
O
E
I
E
E
L
C
F
T
V

L
C
Data Filtering or
A
Compute
Data
Monitoring/
Data
General Data
I
A
S
DataMover
R
P
R
T
N
I
E
Transformation
Scheduling
Transport
Auditing
Federation
E
Discovery
I
T
U
C
C
D
L
N
O
E
I
Services
(Brokering)
(Brokering)
Services
Services
Services
Services
L
E
V
R
U
S
L
M
G
R
O
E
O
E
O
R
C
S
C
E
L

S
G
E
E
N
C
Resource
Storage
I
C
Compute
Data Filtering or
Database
File Transfer
R
S
R

Monitoring/
U
Resource
Resource
Transformation
Management
Service
U
G
O
N
O
Auditing
Manager
Management
Services
Services
(GridFTP)
S
I
S
R
E
E
R
A
R
H
S
Y
T
I
V
I
Communication
Authentication and
T
C
Protocols (e.g.,
Authorization
E
N
TCP/IP stack)
Protocols (e.g., GSI)
N
O
C
C
Other Storage
I
Mass Storage System (HPSS)
Compute
R
Networks
B
This figure based on the Grid Architecture paper
by Globus Team
Systems
A
systems
F
5
Security

.
O
G
2
N
R

T
S
O
R
E
O
Request
Workflow or
I
E

V
C
O
T

Community
Application-
Consistency Services

I
C
N
I
A
Interpretation
Request
L
T
I
F
I
C
A
Authorization
Specific Data
(e.g., Update Subscription,
V
A
C
I
I
and Planning
Management
C
R
U
M
E
L
Services
Discovery Services
Versioning, Master Copies)
E
T
L
E
P
O
E
Services
Services
L
P
R
S
P
V
D
I
S
I
O
Plusinter-site federation, incident handling
procedures, standards, definition of
responsibilities,
A
V
T
C
C
E
L

G
R
L
1
S

N
O
O
E
I
E
E
L
C
F
T
V

L
C
Data Filtering or
A
Compute
Data
Monitoring/
Data
General Data
I
A
S
DataMover
R
P
R
T
N
I
E
Transformation
Scheduling
Transport
Auditing
Federation
E
Discovery
I
T
U
C
C
D
L
N
O
E
I
Services
(Brokering)
(Brokering)
Services
Services
Services
Services
L
E
V
R
U
S
L
M
G
R
O
E
O
E
O
R
C
S
C
E
L

S
G
E
E
N
C
Resource
Storage
I
C
Compute
Data Filtering or
Database
File Transfer
R
S
R

Monitoring/
U
Resource
Resource
Transformation
Management
Service
U
G
O
N
O
Auditing
Manager
Management
Services
Services
(GridFTP)
S
I
S
R
E
E
R
A
R
H
S
Y
T
I
V
I
Communication
Authentication and
T
C
Protocols (e.g.,
Authorization
E
N
TCP/IP stack)
Protocols (e.g., GSI)
N
O
C
C
Other Storage
I
Mass Storage System (HPSS)
Compute
R
Networks
B
This figure based on the Grid Architecture paper
by Globus Team
Systems
A
systems
F
6
GGF AAA Requirements
7
Site Managed AuthN/AuthZ - Fermilab
Globus gatekeeper was enhanced (standard in
GT2.4) to permit an authentication, authorization
callout to a local module to handle more
elaborate decisions than is possible with the
default behaviour using a grid-mapfile.
8
Site Managed AuthN/AuthZ - Fermilab
9
DOEGrids
10
Problem with end-user management of long term
private keys.
11
In planning stage
12
Data Management

.
O
G
2
N
R

T
S
O
R
E
O
Request
Workflow or
I
E

V
C
O
T

Community
Application-
Consistency Services

I
C
N
I
A
Interpretation
Request
L
T
I
F
I
C
A
Authorization
Specific Data
(e.g., Update Subscription,
V
A
C
I
I
and Planning
Management
C
R
U
M
E
L
Services
Discovery Services
Versioning, Master Copies)
E
T
L
E
P
O
E
Services
Services
L
P
R
S
P
V
D
I
S
I
O
A
V
T
C
C
E
L

G
R
L
1
S

N
O
O
E
I
E
E
L
C
F
T
V

L
C
Data Filtering or
A
Compute
Data
Monitoring/
Data
General Data
I
A
S
DataMover
R
P
R
T
N
I
E
Transformation
Scheduling
Transport
Auditing
Federation
E
Discovery
I
T
U
C
C
D
L
N
O
E
I
Services
(Brokering)
(Brokering)
Services
Services
Services
Services
L
E
V
R
U
S
L
M
G
R
O
E
O
E
O
R
C
S
C
E
L

S
G
E
E
N
C
Resource
Storage
I
C
Compute
Data Filtering or
Database
File Transfer
R
S
R

Monitoring/
U
Resource
Resource
Transformation
Management
Service
U
G
O
N
O
Auditing
Manager
Management
Services
Services
(GridFTP)
S
I
S
R
E
E
R
A
R
H
S
Y
T
I
V
I
Communication
Authentication and
T
C
Protocols (e.g.,
Authorization
E
N
TCP/IP stack)
Protocols (e.g., GSI)
N
O
C
C
Other Storage
I
Mass Storage System (HPSS)
Compute
R
Networks
B
This figure based on the Grid Architecture paper
by Globus Team
Systems
A
systems
F
13
GridFTP

.
O
G
2
N
R

T
S
O
R
E
O
Request
Workflow or
I
E

V
C
O
T

Community
Application-
Consistency Services

I
C
N
I
A
Interpretation
Request
L
T
I
F
I
C
A
Authorization
Specific Data
(e.g., Update Subscription,
V
A
C
I
Bug fixes, feature additions
Bug fixes, feature enhancements in GridFTP client
server, as well as additions to gridftp
protocol.
I
and Planning
Management
C
R
U
M
E
L
Services
Discovery Services
Versioning, Master Copies)
E
T
L
E
P
O
E
Services
Services
L
P
R
S
P
V
D
I
S
I
O
A
V
T
C
C
E
L

G
R
L
1
S

N
O
O
E
I
E
E
L
C
F
T
V

L
C
Data Filtering or
A
Compute
Data
Monitoring/
Data
General Data
I
A
S
DataMover
R
P
R
T
N
I
E
Transformation
Scheduling
Transport
Auditing
Federation
E
Discovery
I
T
U
C
C
D
L
N
O
E
I
Services
(Brokering)
(Brokering)
Services
Services
Services
Services
L
E
V
R
U
S
L
M
G
R
O
E
O
E
O
R
C
S
C
E
L

S
G
E
E
N
C
Resource
Storage
I
C
Compute
Data Filtering or
Database
File Transfer
R
S
R

Monitoring/
U
Resource
Resource
Transformation
Management
Service
U
G
O
N
O
Auditing
Manager
Management
Services
Services
(GridFTP)
S
I
S
R
E
E
R
A
R
H
S
Y
T
I
V
I
Communication
Authentication and
T
C
Protocols (e.g.,
Authorization
E
N
TCP/IP stack)
Protocols (e.g., GSI)
N
O
C
C
Other Storage
I
Mass Storage System (HPSS)
Compute
R
Networks
B
This figure based on the Grid Architecture paper
by Globus Team
Systems
A
systems
F
14
Replica Location Service

.
O
G
2
N
R

T
S
O
R
E
O
Request
Workflow or
I
E

V
C
O
T

Community
Application-
Consistency Services

I
C
N
I
A
Interpretation
Request
L
T
I
F
I
C
A
Authorization
Specific Data
(e.g., Update Subscription,
V
A
C
I
I
and Planning
Management
C
R
U
M
E
L
Services
Discovery Services
Versioning, Master Copies)
E
T
L
E
P
O
E
Services
Services
L
P
R
S
P
V
D
I
S
I
O
A
V
T
C
C
E
L

G
R
L
1
S

N
O
O
E
I
E
E
L
C
F
T
V

L
C
Data Filtering or
A
Compute
Data
Monitoring/
Data
General Data
I
A
S
DataMover
R
P
R
T
N
I
E
Transformation
Scheduling
Transport
Auditing
Federation
E
Discovery
I
T
U
C
C
D
L
N
O
E
I
Services
(Brokering)
(Brokering)
Services
Services
Services
Services
L
E
V
R
U
S
L
M
G
R
O
E
O
E
O
R
C
S
C
E
L

S
G
E
E
N
C
Resource
Storage
I
C
Compute
Data Filtering or
Database
File Transfer
R
S
R

Monitoring/
U
Resource
Resource
Transformation
Management
Service
U
G
O
N
O
Auditing
Manager
Management
Services
Services
(GridFTP)
S
I
S
R
E
E
R
A
R
H
S
Y
T
I
V
RLS used by US ATLAS with simulation production
run using the Chimera/Pegasus virtual data system
from the GriPhyN project.
I
Communication
Authentication and
T
C
Protocols (e.g.,
Authorization
E
N
TCP/IP stack)
Protocols (e.g., GSI)
N
O
C
C
Other Storage
I
Mass Storage System (HPSS)
Compute
R
Networks
B
This figure based on the Grid Architecture paper
by Globus Team
Systems
A
systems
F
15
Replication - integration with application
metadata

.
O
G
2
N
R

T
S
O
R
E
O
Request
Workflow or
I
E

V
C
O
T

Community
Application-
Consistency Services

I
C
N
I
A
Interpretation
Request
L
T
I
F
I
C
A
Authorization
Specific Data
(e.g., Update Subscription,
V
A
C
I
I
and Planning
Management
C
R
U
M
E
L
Services
Discovery Services
Versioning, Master Copies)
E
T
L
E
P
O
E
Services
Services
L
P
R
S
P
V
D
I
S
I
O
A
V
T
C
C
E
L

G
R
L
1
S

N
O
O
E
I
E
E
L
C
F
T
V

L
C
Data Filtering or
A
Compute
Data
Monitoring/
Data
General Data
I
A
S
DataMover
R
P
R
T
N
I
E
Transformation
Scheduling
Transport
Auditing
Federation
E
Discovery
I
T
U
C
C
D
L
N
O
E
I
Services
(Brokering)
(Brokering)
Services
Services
Services
Services
L
E
V
R
U
S
L
M
G
R
O
E
O
E
O
R
C
S
C
E
L

S
G
E
E
N
C
Resource
Storage
I
C
Compute
Data Filtering or
Database
File Transfer
R
S
R

Monitoring/
U
Resource
Resource
Transformation
Management
Service
U
G
O
N
O
Auditing
Manager
Management
Services
Services
(GridFTP)
S
I
S
R
E
E
R
A
R
H
S
Y
T
I
V
I
Communication
Authentication and
T
C
Protocols (e.g.,
Authorization
BaBar using SRB for distributed data management
and has interface to BaBar specific metadata
(NCollDb)
E
N
TCP/IP stack)
Protocols (e.g., GSI)
N
O
C
C
Other Storage
I
Mass Storage System (HPSS)
Compute
R
Networks
B
This figure based on the Grid Architecture paper
by Globus Team
Systems
A
systems
F
16
Storage Resource Management

.
O
G
2
N
R

T
S
O
R
E
O
Request
Workflow or
I
E

V
C
O
T

Community
Application-
Consistency Services

I
C
N
I
A
Interpretation
Request
L
T
I
F
I
C
A
Authorization
Specific Data
(e.g., Update Subscription,
V
A
C
I
I
and Planning
Management
C
R
U
M
E
L
Services
Discovery Services
Versioning, Master Copies)
E
T
L
E
P
O
E
Services
Services
L
P
R
S
P
V
D
I
S
I
O
A
V
T
C
C
E
L

G
R
L
1
S

N
O
O
E
I
E
E
L
C
F
T
V

L
C
Data Filtering or
A
Compute
Data
Monitoring/
Data
General Data
I
A
S
DataMover
R
P
R
T
N
I
E
Transformation
Scheduling
Transport
Auditing
Federation
E
Discovery
I
T
U
C
C
D
L
N
O
E
I
Services
(Brokering)
(Brokering)
Services
Services
Services
Services
L
E
V
R
U
S
L
M
G
R
O
E
O
E
O
R
C
S
C
E
L

S
G
E
E
N
C
Resource
Storage
I
C
Compute
Data Filtering or
Database
File Transfer
R
S
R

Monitoring/
U
Resource
Resource
Transformation
Management
Service
U
G
O
N
O
Auditing
Manager
Management
Services
Services
(GridFTP)
S
I
S
R
E
E
R
A
R
H
S
Y
T
I
V
I
Communication
Authentication and
T
C
Protocols (e.g.,
Authorization
E
N
TCP/IP stack)
Protocols (e.g., GSI)
N
O
C
C
Other Storage
I
Mass Storage System (HPSS)
Compute
R
Networks
B
This figure based on the Grid Architecture paper
by Globus Team
Systems
A
systems
F
17
DataMover (Brokering)

.
O
G
2
N
R

T
S
O
R
E
O
Request
Workflow or
I
E

V
C
O
T

Community
Application-
Consistency Services

I
C
N
I
A
Interpretation
Request
L
T
I
F
I
C
A
Authorization
Specific Data
(e.g., Update Subscription,
V
A
C
I
I
and Planning
Management
C
R
U
M
E
L
Services
Discovery Services
Versioning, Master Copies)
E
T
L
E
P
O
E
Services
Services
L
P
R
S
P
V
D
I
S
I
O
A
V
T
C
C
E
L

G
R
L
1
S

N
O
O
E
I
E
E
L
C
F
T
V

L
C
Data Filtering or
A
Compute
Data
Monitoring/
Data
General Data
I
A
S
DataMover
R
P
R
T
N
I
E
Transformation
Scheduling
Transport
Auditing
Federation
E
Discovery
I
T
U
C
C
D
L
N
O
E
I
Services
(Brokering)
(Brokering)
Services
Services
Services
Services
L
E
V
R
U
S
L
M
G
R
O
E
O
E
O
R
C
S
C
E
L

S
G
E
E
N
C
Resource
Storage
I
C
Compute
Data Filtering or
Database
File Transfer
R
S
R

Monitoring/
U
Resource
Resource
Transformation
Management
Service
U
G
O
N
O
Auditing
Manager
Management
Services
Services
(GridFTP)
S
I
S
R
E
E
R
A
R
H
S
Y
T
I
V
I
Communication
Authentication and
T
C
Protocols (e.g.,
Authorization
E
N
TCP/IP stack)
Protocols (e.g., GSI)
N
O
C
C
Other Storage
I
Mass Storage System (HPSS)
Compute
R
Networks
B
This figure based on the Grid Architecture paper
by Globus Team
Systems
A
systems
F
18
Replica Registration Service
19
Data Filtering Transformation

.
O
G
2
N
R

T
S
O
R
E
O
Request
Workflow or
I
E

V
C
O
T

Community
Application-
Consistency Services

I
C
N
I
A
Interpretation
Request
L
T
I
F
I
C
A
Authorization
Specific Data
(e.g., Update Subscription,
V
A
C
I
I
and Planning
Management
C
R
U
M
E
L
Services
Discovery Services
Versioning, Master Copies)
E
T
L
E
P
O
E
Services
Services
L
P
R
S
P
V
D
I
S
I
O
A
V
T
C
C
E
L

G
R
L
1
S

N
O
O
E
I
E
E
L
C
F
T
V

L
C
Data Filtering or
A
Compute
Data
Monitoring/
Data
General Data
I
A
S
DataMover
R
P
R
T
N
I
E
Transformation
Scheduling
Transport
Auditing
Federation
E
Discovery
I
T
U
C
C
D
L
N
O
E
I
Services
(Brokering)
(Brokering)
Services
Services
Services
Services
L
E
V
R
U
S
L
M
G
R
O
E
O
E
O
R
C
S
C
E
L

S
G
E
E
N
C
Resource
Storage
I
C
Compute
Data Filtering or
Database
File Transfer
R
S
R

Monitoring/
U
Resource
Resource
Transformation
Management
Service
U
G
O
N
O
Auditing
Manager
Management
Services
Services
(GridFTP)
S
I
S
R
E
E
R
A
R
H
S
Y
T
I
V
I
Communication
Authentication and
T
C
Protocols (e.g.,
Authorization
E
N
TCP/IP stack)
Protocols (e.g., GSI)
N
O
C
C
Other Storage
I
Mass Storage System (HPSS)
Compute
R
Networks
B
This figure based on the Grid Architecture paper
by Globus Team
Systems
A
systems
F
20
Data Filtering Transformation

.
O
G
2
N
R

T
S
O
R
E
O
Request
Workflow or
I
E

V
C
O
T

Community
Application-
Consistency Services

I
C
N
I
A
Interpretation
Request
L
T
I
F
I
C
A
Authorization
Specific Data
(e.g., Update Subscription,
V
A
C
I
I
and Planning
Management
C
R
U
M
E
L
Services
Discovery Services
Versioning, Master Copies)
E
T
L
E
P
O
E
Services
Services
L
P
R
S
P
V
D
I
S
I
O
A
V
T
C
C
E
L

G
R
L
1
S

N
O
O
E
I
E
E
L
C
F
T
V

L
C
Data Filtering or
A
Compute
Data
Monitoring/
Data
General Data
I
A
S
DataMover
R
P
R
T
N
I
E
Transformation
Scheduling
Transport
Auditing
Federation
E
Discovery
I
T
U
C
C
D
L
N
O
E
I
Services
(Brokering)
(Brokering)
Services
Services
Services
Services
L
E
V
R
U
S
L
M
G
R
O
E
O
E
O
R
C
S
C
E
L

S
G
E
E
N
C
Resource
Storage
I
C
Compute
Data Filtering or
Database
File Transfer
R
S
R

Monitoring/
U
Resource
Resource
Transformation
Management
Service
U
G
O
N
O
Auditing
Manager
Management
Services
Services
(GridFTP)
S
I
S
R
E
E
R
A
R
H
S
Y
T
I
V
I
Communication
Authentication and
T
C
Protocols (e.g.,
Authorization
E
N
TCP/IP stack)
Protocols (e.g., GSI)
N
O
C
C
Other Storage
I
Mass Storage System (HPSS)
Compute
R
Networks
B
This figure based on the Grid Architecture paper
by Globus Team
Systems
A
systems
F
21
Application Specific Data Discovery Services

.
O
G
2
N
R

T
S
O
R
E
O
Request
Workflow or
I
E

V
C
O
T

Community
Application-
Consistency Services

I
C
N
I
A
Interpretation
Request
L
T
I
F
I
C
A
Authorization
Specific Data
(e.g., Update Subscription,
V
A
C
I
I
and Planning
Management
C
R
U
M
E
L
Services
Discovery Services
Versioning, Master Copies)
E
T
L
E
P
O
E
Services
Services
L
P
R
S
P
V
D
I
S
I
O
A
V
T
C
C
E
L

G
R
L
1
S

N
O
O
E
I
E
E
L
C
F
T
V

L
C
Data Filtering or
A
Compute
Data
Monitoring/
Data
General Data
I
A
S
DataMover
R
P
R
T
N
I
E
Transformation
Scheduling
Transport
Auditing
Federation
E
Discovery
I
T
U
C
C
D
L
N
O
E
I
Services
(Brokering)
(Brokering)
Services
Services
Services
Services
L
E
V
R
U
S
L
M
G
R
O
E
O
E
O
R
C
S
C
E
L

S
G
E
E
N
C
Resource
Storage
I
C
Compute
Data Filtering or
Database
File Transfer
R
S
R

Monitoring/
U
Resource
Resource
Transformation
Management
Service
U
G
O
N
O
Auditing
Manager
Management
Services
Services
(GridFTP)
S
I
S
R
E
E
R
A
R
H
S
Y
T
I
V
I
Communication
Authentication and
T
C
Protocols (e.g.,
Authorization
E
N
TCP/IP stack)
Protocols (e.g., GSI)
N
O
C
C
Other Storage
Phase 2 SBIR with Tech-X and SLAC.
I
Mass Storage System (HPSS)
Compute
R
Networks
B
This figure based on the Grid Architecture paper
by Globus Team
Systems
A
systems
F
22
Moving to Web Services OGSI
JLab
CMS Caltech web servicesClarens
23
Monitoring

.
O
G
2
N
R

T
S
O
R
E
O
Request
Workflow or
I
E

V
C
O
T

Community
Application-
Consistency Services

I
C
N
I
A
Interpretation
Request
L
T
I
F
I
C
A
Authorization
Specific Data
(e.g., Update Subscription,
V
A
C
I
I
and Planning
Management
C
R
U
M
E
L
Services
Discovery Services
Versioning, Master Copies)
E
T
L
E
P
O
E
Services
Services
L
P
R
S
P
V
D
I
S
I
O
A
V
T
C
C
E
L

G
R
L
1
S

N
O
O
E
I
E
E
L
C
F
T
V

L
C
Data Filtering or
A
Compute
Data
Monitoring/
Data
General Data
I
A
S
DataMover
R
P
R
T
N
I
E
Transformation
Scheduling
Transport
Auditing
Federation
E
Discovery
I
T
U
C
C
D
L
N
O
E
I
Services
(Brokering)
(Brokering)
Services
Services
Services
Services
L
E
V
R
U
S
L
M
G
R
O
E
O
E
O
R
C
S
C
E
L

S
G
E
E
N
C
Resource
Storage
I
C
Compute
Data Filtering or
Database
File Transfer
R
S
R

Monitoring/
U
Resource
Resource
Transformation
Management
Service
U
G
O
N
O
Auditing
Manager
Management
Services
Services
(GridFTP)
S
I
S
R
E
E
R
A
R
H
S
Y
T
I
V
I
Communication
Authentication and
T
C
Protocols (e.g.,
Authorization
E
N
TCP/IP stack)
Protocols (e.g., GSI)
N
O
C
C
Other Storage
I
Mass Storage System (HPSS)
Compute
R
Networks
B
This figure based on the Grid Architecture paper
by Globus Team
Systems
A
systems
F
24
Monitoring Deployment
Many sites have deployed Globus MDS, MonALISA and
Ganglia to cover the rangefrom individual
resource attributes toVO-wide usage summaries.
25
Standardizing Info Svcs Schema
GLUE Schema - standardizing information services
schema between US and European collaborators.Many
many many conference calls and meetings for
discussion of details of presenting information,
the actual meaning of the attributes, etc.
26
More Informationwww.ppdg.net

• Reminder of PPDG
• Accomplishments in
• Security
• Data Management
• Monitoring
• Talks covering other areas with PPDG
  accomplishments
• For common software stack integration see talk on
  VDT by Miron Livny, U. Wisconsin, Wed. 400pm.
• For job scheduling and production grids see
  Grid2003 talk by Ian Fisk, Fermilab, Thurs.
  1100am.
• Plenty of items not covered in presentations.
• See also LBNL, SLAC-FNAL, and BNL booths.