Supermon: Highperformance monitoring for Linux Clusters

1
SupermonHigh-performance monitoring for Linux
Clusters

• Ronald G. Minnich
• 5th Annual Linux Showcase Conference, 2001

2
Introduction

• Problem with large clusters
• Lack of efficient, high performance monitoring
  tools which provide comprehensive information
  about the state of the cluster nodes
• Only one tool available rstatd
• No tool to aggregate information of each node
• rstatd
• Provides limited information about cluster nodes
• Extracting information is very inefficient

3
Introduction

• supermon
• Acts as a server for providing cluster
  performance data for multiple remote programs
• An extensible cluster monitoring system
• Capable of high peak and practical sampling rates
• Based on a generic, self-describing data format

4
rstatd

• rstatd
• Venerable Unix daemon, (1980s)
• Provides status information to remote host via
  SunRPC
• Remote client programs (such as rsysinfo) can
  query rstatd servers for status of system

5
rstatd

• Problems of rstatd
• Provides too little informationnot sufficient in
  cluster environment
• Doesnt provide how much memory a node has and
  how much is in use
• Doesnt provide information about swap usage
  (just rate of swap usage)
• Doesnt provide disk read/write statistics
• Too slow and inefficient
• Takes too long to retrieve information
• It opens and read files in /proc to get kernel
  statistics open/read/close
• Improving rstatd performance
• Use of sysctl interface

6
Supermon

• Supermon
• Improved rstatd resolves problem with getting the
  data from the individual cluster nodes.Remaining
  problem -gt aggregate performance data from each
  node
• Uses vector RPC

7
Output example

• Statistics for startingan MPI application
• Left y axis intr (solid line)
• Right y axis sys CPU (dotted line)
• x axis time

8
Performance

• Performance of improved rstatd is very good
• At 100 samples/second
• Old rstatd consumes at least 12 of CPU
• New rstatd consumptions is not measurable
• Supermon performance is correspondingly good
• Can easily monitor 100 hosts at 100 samples/second

9
Conclusion

• Improved rstatd
• 6 times faster, for more efficient
• New program supermon
• Allows for high-performance, low-overhead
  monitoring of hundreds of cluster nodes