xrootd

1
xrootd

• Andrew Hanushevsky
• Stanford Linear Accelerator Center
• 07-June-04
• http//www.slac.stanford.edu/abh/xrootd

2
Goals I

• High Performance File-Based Access
• Scalable, extensible, usable
• Fault tolerance
• Server failures handled in a natural way
• Servers may be dynamically added and removed
• Flexible Security
• Allowing use of almost any protocol
• Rootd Compatibility

3
Goals II

• Simplicity
• Can run xrootd out of the box
• No config file needed for non-complicated/small
  installations
• Generality
• Can configure xrootd for ultimate performance
• Meant for intermediate to large-scale sites

4
High Performance I

• Basic
• Scalable request/response protocol
• Multi-threaded architecture
• Architecture sensitive polling
• MRU scheduling
• Sticky sockets
• Adaptive reconfiguration

5
xrootd Architecture
application
Protocol Manager
xrd
xrootd
Protocol Layer
xroot
Authentication
Filesystem Logical Layer
ofs
Authorization
optional
Filesystem Physical Layer
oss
(included in distribution)
Filesystem Implementation
mss
_fs
6
Scalable Protocol I

• Connection multiplexing
• One connection per client/host
• Multiple logically independent streams
• Request redirection supported
• Similar to http redirection
• Supports dynamic load balancing and fail-over
• Uses an intentional request header
• Can better optimize request processing

7
Scalable Protocol II

• Asynchronous mode allowed
• Multiple processing-order-independent requests
• Optional application-directed pre-read
• Optional application-directed file preparation
• To deal with file pre-staging load balancing
• I/O segmenting
• Able to naturally deal with very large transfers
• Request deferral
• Client waits for resources without using server
  resources

8
Scalable Protocol III

• Unsolicited Reverse Request Mode
• Allows server to manage client for recovery
• Asynchronous redirect, deferral, and messages
• Protocol may be compatibly extended
• Mechanism to send opaque information
• Accommodate things that were forgotten
• Messaging interface
• Cache group
• Request priority
• And so on.

9
Architecture Sensitive Polling

• All POSIX systems support poll()
• Used by default
• Not always an efficient mechanism
• Alternate polling mechanisms allowed
• /dev/poll
• Available on Solaris and patched RH Linux
• Up to an order of magnitude reduction in CPU
• Essential to reduce latency

10
MRU Scheduling

• Connections processed in most recently used order
• Gives priority to active connections
• Reduces polling overhead
• Essentially a fair scheduling algorithm

11
Sticky Sockets

• Connection temporarily binds to a thread
• Avoids polling and scheduling overhead
• Significantly reduces latency
• Connection automatically unbinds
• Client is not sufficiently active
• Number of other requests approaches available
  threads

12
Adaptive Reconfiguration

• Server dynamically adjusts configuration
• Number of threads
• Kept proportionate to number of active requests
• Pre-allocated buffers
• Sizes track actual run-time usage profile
• Pre-allocated objects
• Number tracks recent needs
• High latency connection rescheduling
• Allows for low bandwidth/high latency connections

13
High Performance II

• Optional
• Extended filesystem Layer
• Shared Library plug-in
• Based on proven oofs layer
• Dynamic Load Balancing
• Configurable external program
• Based on proven oolb architecture

14
Versatile sfs Interface

• Integrates performance, scalability, usability
  features
• Shared library plug-in extensions available
• Dynamic load balancing integration
• FD partitioning for reduced polling overhead
• Shared file descriptors for reduced OS overhead
• Non-essential write elimination
• Open file timeout
• Dynamic disk cache for balanced recoverable
  peta-byte file systems
• Agnostic Mass Storage Integration
• Optional RFIO
• Scalable authorization

15
Load Balancing

• xrootd scales in multiple dimensions
• Can run multiple load balanced xrootds
• Architected as self-configuring structured
  peer-to-peer (SP2) data servers
• Servers can be added removed at any time
• Client (XTNetFile) understands SP2 configurations
• xrootd informs client when running in this mode
• Client has more recovery options in the event of
  failure

16
Load Balancing Implementation

• Control Interface (olbd)
• Load balancing meta operations
• Find files, change status, forwarded requests
• Data Interface (xrootd)
• Data is provided to clients
• Interfaces to olbd via the ofs layer
• Separation is important
• Allows use of any protocol
• Client need not know the control protocol

17
Typical SP2 Configuration
olbd
xrootd
Dynamic Selection
olbd
xrootd
client
mss
distinguished
xrootd
olbd
18
Example SLAC Configuration
kan01
kan02
kan03
kan04
kanxx
kanolb-a
bbr-olb03
bbr-olb04
client machines
19
Fault Tolerance I

• Servers may come and go
• Uses load balancing to effect recovery
• New servers can be added at any time
• Servers may be brought down for maintenance
• Files can be moved around in real-time
• Client simply adjust to the new configuration
• XTNetFile object handles recovery protocol

20
Fault Tolerance II

• Whenever client looses connection
• Back to distinguished xrootd(s) for new server
• Limited wait/retry loop on the same server
• All handled in the XTNetFile class
• Disruptions merely delay the client

21
Flexible Security

• Negotiated Security Protocol
• Allows client/server to agree on protocol
• E.g., Kerberos, GSI, AFS Kerberos, etc.
• Can be easily extended
• Multi-protocol authentication support
• Each protocol implemented as a shared library
  plugin

22
rootd Compatibility

• Bilateral compatibility
• XTNetfile reverts to TNetFile for rootd servers
• XRootd reverts to rootd protocol for TNetFile
• Allows for transparent introduction
• Can run mixed mode
• Binary is multi-environment compatible

23
Rootd Compatibility
Client-Side Compatibility
Application
xrootd
XTNetFile
rootd
TNetFile
Server-Side Compatibility
Application
xrootd rootd compability
rootd
TNetFile
client
24
What we have seen

• For a single server
• 1,000 simultaneous clients
• 2,200 simultaneous open files
• Bottlenecks
• Disk I/O (memory next behind)
• We will be experimenting with other machine
  configurations

25
What Have We Heard

• The system is too stable
• Users run extra-long jobs (1-2 weeks) now
• Error not discovered until weeks later
• The system is too aggressive
• New servers are immediately taken over
• Easy configuration but startling for
  administrators

26
The Next Step
SLAC
IN2P3
RAL
IN2P3 proxy
RAL proxy
xrootds
27
Proxy Service

• Attempts to address competing goals
• Security
• Deal with firewalls
• Scalability
• Administrative
• Configuration
• Performance
• Ad hoc forwarding for near-zero wait time
• Intelligent caching in local domain

28
Proxy Implementation

• Uses capabilities of olbd and xrootd
• Simply an extension of local load balancing
• Implemented as a special file system type
• Interfaces in the ofs ayer
• Functions in the oss layer
• Primary developer is Heinz Stockinger

29
Proxy Example
data01
data02
data03
data04
RAL
proxy olb
local olb
4
local olb
proxy olb
SLAC
3
1
red01
data02
data03
proxy01
2
client machines
30
Conclusion

• xrootd provides high performance file access
• Improves over afs, ams, nfs, etc.
• Unique performance, usability, scalability,
  security, compatibility, and recoverability
  characteristics
• Should scale to tens of thousand clients
• A good foundation for native-like file systems
• Will be distributed as part of root package
• Open software, supported by
• SLAC (server) and INFN-Padova (client)