Implementing the Andrew File System at BII

1
Implementing the Andrew File System at BII

• James Tay
• 2008

2
History of BII file servers

• Initial storage based on Sun A1000 (approx
  420GB).
• Upgraded to Sun 3511 (approx 880GB).
• Centralized architecture.
• File sharing services based on NFS and Samba.
• Easy to setup, but not scalable.

3
AFS Capabilities

• Transactional scalability
• Token based user authentication
• Supports ACLs
• Allows online volume migrations
• Volume quotas
• Optional encryption

4
AFS Architecture

• Distributed across multiple machines
• Distributed across multiple networks
• Database servers are fully redundant
• Dynamically add/remove File servers
• 1 read-write volume, many read-only volumes
• Can promote a read-only volume to a read-write
  volume

5
Server Evaluations

• Solaris 10
• Poor driver support for our hardware. Best IO
  performance
• FreeBSD 6.2
• Very stable platform. Average IO performance
• OpenBSD 3.9
• Very stable platform. Poor IO performance
• Linux 2.6.19
• Stable enough. Good IO performance

6
Initial Implementation

• 3 Database Servers
• 7 File Servers
• Gigabit ethernet
• Laptop as PXEBOOT install server
• 420GB total available storage
• Based on CentOS 4.4
• Custom built 2.6.19 kernel

7
Daily Backups

• Daily full backups onto NFS NAS (night).
• Full backup from NFS NAS to tape library (day).
• Restore from yesterday is simple
• Restore from up to 2 weeks ago is slow

8
Off site clients

• BII users at IMCB building.
• NAT session timeouts messes up AFS callbacks
• AFS client hangs for about a minute
• Looping (failing) write keeps NAT sessions alive

9
Expansion of storage

• Added 2 more File Servers
• Total of 2 TB increase
• Can be quickly increased to 10TB at short notice
• New division shared storage structure
• 200GB per division

10
Additional backups

• 1TB low cost Linux iSCSI target appliance
• Solaris backup server with ZFS on iSCSI
• Daily rsync updates
• Daily ZFS snapshots
• Up to 30 days of snapshots per division

11
Future Upgrades

• 2x StorageTek 6140 arrays (2x 14TB)?
• 3x 2-way Opteron Database Servers
• 4x 4-way Opteron File Servers
• 2x Sun x4500 Storage servers for backups (2x
  36TB)?
• Gigabit network via Extreme Switch

12
Thank Youhttp//web.bii.a-star.edu.sg/james/afs