Graybox%20NFS%20Caching%20Proxy

1
Graybox NFS Caching Proxy

• By
• Paul Cychosz and
• Garrett Kolpin

2
What are Graybox Techniques?

• First we need to know a little about how the
  graybox works.
• Inference by observation
• No server or client-side modifications necessary

3
Our Problem

• Figure out what the client is caching
• Maintain low overlap between client and proxy
  caches

4
Our Solution

• Treat the client as a graybox
• Monitor NFS traffic
• Infer from NFS traffic the contents of client
  cache

5
Conclusions

• Cache overlap decreased by 5 to 50 when using
  graybox techniques

6
Outline

• Approach
• Results
• Conclusions

7
Approach
NFS specifics

• Implemented in kernel as another FS type
• Replaces file_operations with its own
  functions for read, write, open, etc.
• Uses VFS abstraction

VFS
RPC / XDR
retrans., data wrapping
NFS kernel code
TCP / UDP
8
Approach
Client cache / Replacement

• Caches on page level
• Not much NFS specific code Uses global
  Linux page cache
• Not a static size, no upper-bound
• Timestamps jiffies, per page.

9
Approach
Reads typically gt 85 of traffic
NFS / RPC read structure

• RPC Packets sent out access to dir, getattr
  to file, access to file, then read, read, read,
• Client use of getattr to revalidate cached
  copies

• Exploit this, proxy explicitly look for these
  to determine client cache contents

10
Approach
Proxy

• Sees
• File handle
• Offset
• size
• --becomes our cache element
• Store in hash table
• Cache on reads, run our cache element
  replacement policy on getattr (proof of cached
  client copy)

11
Approach
NFS Client
Proxy
Replacement (possible force-out)
Read system call
getattr
Revalidate data
hit
Cache lookup
OR
Lookup
read
miss
Read setup
OR
hit
miss
Return data
Add
Data
Typical read()
Forward req. to server
12
Viewing Client Cache Contents

• Disable swap space
• Use mincore() to view pages that client has
  cached
• Use balloon program to keep client cache
  contents manageable

Test Environment
13
Approach
Difficulties

• Logistics Initial setup, client/server/proxy
  all on same PC doesnt work
• Client cache size changes
• -solved compare percentages, normalize
• Repeated RPC calls not idempotent

• Not a significant problem, proxy efficiency
  degrades linearly with NFS performance if network
  traffic is bad

14
Approach
Difficulties

• Ambiguities
• Different FS activities DONT generate unique
  RPC patterns
• getattr used for a few things besides reads
• -- Null RPC packets
• Cannot tell exact blocks client caching, only
  which files
• Need second, local replacement policy on proxy!

15
Approach
Cache Replacement
Global Policies
Local Policies

• LRU
• MRU

• Forcing entire file
• MRU

16
Outline

• Approach
• Results
• Conclusions

17
What We Tested

• Control Experiments
• Global LRU, no local force-out
• Global MRU, no local force-out
• Combinations
• Global LRU, whole file force-out
• Global LRU, block-level local MRU
• Global MRU, whole file force-out
• Global MRU, block-level local MRU

18
How We Tested

• Used Postmark benchmark
• 12 files with sizes randomly selected between 4
  and 50 kilobytes
• Gives us more of a real-world workload

19
Results
20
Results
21
Results
22
Outline

• Approach
• Results
• Conclusions

23
Conclusions

• Decrease in overlap by between 5 and 50 percent
  as compared to control tests
• MRU global file replacement is good by itself
• MRU global policy with whole-file local policy
  gives highest cache fill and lowest overlap

24

• Questions? Feedback?

?
?
?