File System Access XRootd

1
File System Access(XRootd)

• Andrew Hanushevsky
• Stanford Linear Accelerator Center
• 13-Jan-03

2
Goals

• High Performance File-Based Access
• Scalable
• Extensible
• Mass Storage integration
• Flexible Security
• Dynamically Configurable
• Rootd Compatable

3
Achieving High Performance

• Multi-threaded server
• Ability to run multiple servers on one host
• MRU polling
• Last active connections serviced first
• Sticky sockets
• Socket/thread bound for active connections
• Using high-performance oofs layer
• Unlocked I/O, sync reduction, file descriptor
  partitioning

4
Achieving Scalability

• Socket multiplexing
• One connection per client/host
• Requires stream id protocol extensions
• Dynamic Load Balancing
• Requires client redirection protocol extensions
• Using scalable oofs layer
• File reuse, idle closeouts, cached file system

5
Achieving Extensibility

• Opaque Information Protocol
• Protocol extension allows client to relay
  outboard information to server
• Messaging interface
• Cache group
• Request priority
• Other things we havent thought of yet

6
Achieving MSS Integration

• Using oofs layer
• Already integrates with almost any MSS
• No brainer

7
Achieving Flexible Security

• Negotiated Security Protocol Extensions
• Allows client/server to agree on protocol
• E.g., Kerberos, GSI, AFS Kerberos, etc.
• Using oofs layer
• Multi-protocol authentication support
• Scalable authorization

8
Achieving Configurability

• Servers must come up/down without harm
• Part of dynamic load balancing
• Requires client retry extension

9
Achieving Compatability

• Client sensitive to server version number
• Rootd servers default to base TNetFile class
• XRootd servers use derived TNetFile class
• Server requires client to turn on extensions
• Base TNetFile class restricted to rootd ops
• Derived TNetFile enables extensions
• Multi-streaming, Redirection, Request Deferral,
  Generic Security, Opaque Information

10
xrootd Architecture
application
Database Protocol Layer
xrootd
Filesystem Logical Layer
oofs
Filesystem Physical Layer
ooss
Filesystem Implementation
mss
xfs
11
Some Details

• Compatible Protocol
• Allows transparent use of rootd xrootd
• Multi-Streaming
• Allows multiple files per connection
• Cache File System
• Dynamic Load Balancing
• Client directed to appropriate server
• Configurability Recoverability

12
Compatible Protocol
Application
xrootd
XTNetFile
xrootd rootd compability
TNetFile
rootd
13
Multi-Streaming

• Each open() assigned a stream number
• Four character id that starts with 0-9
• All requests/responses identify relevant stream
• Non-open file requests use arbitrary id
• Any four characters not starting with 0-9
• E.g., stat(), mv(), unlink(), etc.
• Parallel Multi-Streaming allowed
• Client must explicitly enable it

14
Cache File System
Index Area Optional data cache Default data
area
/databases/mydbfile
symlink
Naming convention allows for audit and index
recovery
Multiple Independent Filesystems
/cache1/databasesmydbfile
Data Area Any number Any Size Chosen based on
free space in LRU order
/cache2
/cache3
15
Dynamic Load Balancing
xrootd
Dynamic Selection
xrootd
client
mss
xrootd
16
DLB Implementation
I do
subscribe
who has the file?
open again
open
Client
wait
try hostport
17
Configurability Recoverability

• Go hand in hand
• Configuration changes may be disruptive
• Must be able to recover from dynamic
  configuration changes
• Dynamic Load Balancing essential part

18
Recoverability

• Whenever client looses r/o connection
• Back to distinguished xrootd(s) for reselection
• Whenever client looses r/w connection
• Limited wait/retry loop on the same server
• We will be working to improve this next year!
• All handled in the XTNetFile class
• Disruptions merely delay the client

19
Configurability

• Since we have recoverability
• Servers may come and go
• New servers can be added at any time
• Servers may be brought down for maintenance
• Files can be moved around in real-time
• Client will recover and adjust to the new
  configuration

20
Conclusion

• Goals for rootd file access can be met
• Most of this work relies on leveraging work done
  for the AMS
• Compatability allows us to lobby the community
  for xrootd to become the default rootd