Embedded Computing Cluster: NFS Server Analysis

1
Embedded Computing ClusterNFS Server Analysis

• Abraham Murray

CS577 Advanced Computer Networks
2
Outline

• Project Abstract
• System Overview
• Measurement Approach
• Measured Results
• Aggregate Results
• Per Client Results
• Per File Results
• Per User Results
• Result Analysis

CS577 Advanced Computer Networks
3
Abstract

• Our Embedded Computing Cluster
• Runs 80 diskless nodes to save cost
• Uses high bandwidth data fabrics
• Allows NFS file systems to perform well
• 1 NFS server for 80 nodes
• Runs proprietary software (light on details)
• Open-source Linux infrastructure
• Intent is to measure and analyze NFS server
  performance

• Any nodes using disproportionate resources?
• What are the five busiest client nodes?
• Is the workload bursty or constant?
• How does the workload change over time?
• What files/directories are most heavily
  requested?

CS577 Advanced Computer Networks
4
System Overview

• System nodes embedded PowerPC
• Combination of Myrinet data fabric, and Gigabit
  Ethernet
• Myrinet is point-to-point switched data fabric,
  2 Gbps full duplex
• Multi-role system, variety of proprietary
  software
• VoIP, communications support, DSP
• General Purpose Processors

Myrinet
Gigabit Ethernet
System Server
Diskless Node
nfsdump
ramdisk
HDD
NFS Client
NFS Server
DisklessNodes
DisklessNode(Router)
DisklessNodes
CS577 Advanced Computer Networks
5
Measurement Approach

• Measurement Tools
• nfsdump modification to support NFS-specific
  attributes
• nfsscan Perl script to automate data analysis of
  nfsdump outputs
• Presented at LISA XVII, October 2003 by Ellard,
  Seltzer
• Measurement Approach
• Task system consistently
• Run tools on NFS server (ISSUE!)
• Anonymize results for presentation

CS577 Advanced Computer Networks
6
Aggregate Server Results
Table 1 Aggregate Server Operation Counts1
TOTAL read write lookup getattr access create remove
371282 79326 938 6663 72049 209737 209 411
1 cmdline ./nfsscan -t 0 -l -o nfsscan_results
./nfsdump_results.txt
Table 2 Server Statistics
Operation Type Operations per Second Percent of Total
TOTAL 240  
read 51 21
write 1 0
lookup 4 2
getattr 47 19
access 136 56
create 0 0
remove 0 0

CS577 Advanced Computer Networks
7
Aggregate Server Results
booting
1st software load
2nd software load
CS577 Advanced Computer Networks
8
Per Client Results
Table 3 15 Busiest Clients Operation Counts1
client TOTAL read write lookup getattr access create remove
0.0.0.70 102658 8440 21 207 26495 67432 5 12
0.0.0.66 66523 6517 26 213 16783 42922 4 11
0.0.0.90 30942 5583 31 251 4561 20446 6 11
0.0.0.82 15720 2145 35 290 3247 9905 8 13
0.0.0.71 12549 6477 16 229 1292 4470 9 13
0.0.0.67 12220 6032 48 239 1352 4445 10 13
0.0.0.72 9203 4394 28 244 1110 3339 8 14
0.0.0.91 8006 4149 31 184 799 2789 1 7
0.0.0.65 7854 1926 56 312 1225 4186 10 18
0.0.0.84 7758 2235 16 313 1296 3818 7 11
0.0.0.80 7681 1820 93 230 1083 4295 7 15
0.0.0.89 7383 1198 43 197 1314 4536 6 16
0.0.0.92 7224 3881 51 187 641 2376 5 14
0.0.0.87 6700 1853 17 230 1092 3449 4 8
0.0.0.85 6372 2079 15 340 1035 2820 8 14
1 cmdline ./nfsscan -t 0 -BC -l -o
nfsscan_results ./nfsdump_results.txt
CS577 Advanced Computer Networks
9
Per Client Results DSP Nodes

• Clients 68, 70, 71 retasked at 18 minutes to run
  concurrent DSP

CS577 Advanced Computer Networks
10
Per Client Results VoIP Node

• Client 90 boots in 1.6 minutes, launches initial
  software from 2-3.5 minutes

CS577 Advanced Computer Networks
11
Per Client Results Router Node

• Client 65 boots in 1.7 minutes. This is a
  Gigabit-Myrinet routing node
• DHCP-relay, IP-forwarding, OSPF routing node

CS577 Advanced Computer Networks
12
Per File Results
Table 4 20 Busiest Files and Directories
Operation Counts1
fileid TOTAL read write lookup getattr access create remove
1544 43733 10 0 0 21972 21751 0 0
249 28167 0 0 31 49 28067 0 0
274 26811 0 0 36 134 26641 0 0
38 25995 0 0 56 6 25914 5 12
340 16343 0 0 68 3 16255 4 11
525 12807 0 0 0 12807 0 0 0
189 10285 0 0 393 37 9793 0 0
1785 7981 0 0 0 7981 0 0 0
166 6996 0 0 90 30 6827 0 0
1245 5936 3162 0 0 1422 1352 0 0
55 5063 0 0 556 908 3599 0 0
161 5042 0 0 118 33 4891 0 0
115 4695 0 0 34 65 4596 0 0
91 4409 0 0 131 24 4254 0 0
7 4385 0 0 13 21 4351 0 0
103 4232 0 0 13 58 4161 0 0
134 4134 0 0 6 54 4074 0 0
640 3656 3563 0 0 50 43 0 0
1821 3520 3516 0 0 0 4 0 0
1244 3476 405 0 0 1561 1510 0 0
646 3145 2947 0 0 97 101 0 0
1 cmdline ./nfsscan -t 0 -BF -f -d -l -o
nfsscan_results ./nfsdump_results.txt
CS577 Advanced Computer Networks
13
Per File Results - Histogram
CS577 Advanced Computer Networks
14
Per User Results
Table 5 Per User Operation Counts1
user TOTAL read write lookup getattr access create remove
root 360790 73265 938 5835 69027 209200 209 411
user 1 10491 6061 0 828 3021 537 0 0
1 cmdline ./nfsscan -t 0 -BU -l -o
nfsscan_results ./nfsdump_results.txt

CS577 Advanced Computer Networks
15
Result Analysis

• The top 5 busiest nodes are responsible for 60
  of NFS traffic
• The NFS workload is bursty, and follows
  operator-system tasking
• Majority of accessed files are touched less than
  80 times
• Once per client
• More regularly accessed files might belong in
  ramdisks
• Reduce NFS Operations by up to 60!

CS577 Advanced Computer Networks