X-RAY: A Non-Invasive Exclusive Caching Mechanism for RAIDs

1
X-RAY A Non-Invasive Exclusive Caching Mechanism
for RAIDs

• Lakshmi N. Bairavasundaram
• Muthian Sivathanu
• Andrea C. Arpaci-Dusseau
• Remzi H. Arpaci-Dusseau

ADvanced Systems Laboratory Computer Sciences
Department University of Wisconsin Madison
2
Introduction

• Caching in modern systems
• Multiple levels
• Storage 2-level hierarchy
• Level 1 File system (FS) cache
• Software-managed
• Main memory of host/client
• LRU-like cache replacement
• Level 2 RAID cache
• Firmware-managed
• Memory inside RAID system
• Usually LRU replacement

3
Introduction contd.

• LRU
• Replace LRU block
• Cache placement on read

LRU
MRU
39
4
Introduction contd.

• LRU
• Replace LRU block
• Cache placement on read
• 2 levels of LRU
• Redundant contents

FS Cache
..
10
LRU
MRU
RAID Cache
..
MRU
10
LRU
5
Introduction contd.

• LRU
• Cache placement on read
• Replace LRU block
• 2 levels of LRU
• Redundant contents
• Goal
• Exclusive caching

FS Cache
LRU
10
11
12
.
MRU
RAID Cache
10
LRU
11
12
.
MRU
6
Improved RAID Caching

• Multi-Queue (Zhou et al. 2001)
• Add frequency component to cache policy
• Not strictly exclusive!

• DEMOTE (Wong and Wilkes 2002)
• Change interface to disk
• File system issues cache place command
• Has perfect information and hence perfectly
  exclusive caches
• Interface changes difficult to deploy

7
Ideal RAID Cache

• Exclusive caching
• File system and RAID caches should have different
  contents
• Global LRU
• Known to work well
• RAID cache should be a victim cache
• No interface changes

FS Cache
.
MRU
Victim Block
RAID Cache
Block Read

LRU
8
X-RAY

• Observes disk traffic
• Reads and writes to data and metadata
• Builds a model of the FS cache
• Uses semantic knowledge
• Predicts size and contents of FS cache
• Identifies set of exclusive blocks
• Recent victims of the FS cache
• Reads blocks from disk into cache
• Result
• A nearly exclusive cache without interface changes

Host
File system cache
RAID
X-RAY
Model of FS cache
RAID cache
9
Talk Outline

• Introduction
• File Systems
• Information and Inferences
• X-RAY Cache Design
• Results
• Conclusion

10
File System Operation

• Applications perform file reads and writes
• File system (Unix)
• Translates file accesses to disk block requests
• Metadata
• To maintain application data on disk and manage
  disk blocks
• Periodically written to disk
• Examples inodes, bitmap blocks

11
File System Operation

• Inode
• Pointers to data blocks
• File access information

Latest access time
File
Inode
Pointers to data blocks
Data Blocks
12
File System Operation

• File access
• Use inode to obtain pointers to disk data blocks
• Read corresponding blocks from disk if they are
  not in FS cache
• Update the access time information in inode
• Metadata updates
• Periodically check for dirty inodes and write
  to disk

13
The Problem

• To observe disk traffic and infer the contents of
  FS cache
• Why difficult?
• FS cache size changes over time
• Shares main memory with virtual memory system

14
The Problem

• To observe disk traffic and infer the contents of
  FS cache
• Why difficult?
• FS cache size changes over time
• Disk cannot observe all FS-level accesses

10
11
12
Read block
FS Cache
10
11
12
LRU
MRU
Disk Read
RAID
FS Cache Model
10
11
12
MRU
LRU
15
The Problem

• To observe disk traffic and infer the contents of
  FS cache
• Why difficult?
• FS cache size changes over time
• Disk cannot observe all FS-level accesses

10
13
Read block
FS Cache
10
11
12
LRU
MRU
Disk Read
RAID
FS Cache Model
10
11
12
MRU
LRU
16
The Problem

• To observe disk traffic and infer the contents of
  FS cache
• Why difficult?
• FS cache size changes over time
• Disk cannot observe all FS-level accesses

Read block
FS Cache
10
12
13
LRU
MRU
RAID
FS Cache Model
11
12
13
MRU
LRU
17
The Problem

• To observe disk traffic and infer the contents of
  FS cache
• Why difficult?
• FS cache size changes over time
• Disk cannot observe all FS-level accesses
• Key observation
• We need information about accesses that hit in FS
  cache
• File system maintains access information in inodes

Read block
FS Cache
10
12
13
LRU
MRU
RAID
FS Cache Model
11
12
13
MRU
LRU
18
Talk Outline

• Introduction
• File Systems
• Information and Inferences
• X-RAY Cache Design
• Results
• Conclusion

19
Information

• Obtain information from observing disk traffic
• Knowledge of file system structures and
  operations
• File system maintains time of last access in
  inodes
• Periodic inode writes
• Assuming whole file access, all blocks are in FS
  cache
• Assume file system cache policy is LRU

20
Inferences

• Read for data block
• Block will be placed in file system cache (MRU
  block)

• Read for previously read data block
• Block became victim in file system cache
• Blocks with an earlier access time should also be
  victims

• Inode write new access time , no disk read
  observed
• All blocks belonging to file are in FS cache
• Other blocks with later access time should also
  be present

21
Talk Outline

• Introduction
• File Systems
• Information and Inferences
• X-RAY Cache Design
• Results
• Conclusion

22
Design
Block number
Access time

• Recency list (R-list)
• List of data blocks ordered by access time
• Cache Begin (CB) pointer
• Divides R-list into inclusive and exclusive
  regions
• RAID Cache contents
• Subset of blocks in exclusive region

MRU
A, 1
LRU
Inclusive region
Exclusive region
CB
Blocks the RAID should cache
Blocks expected to be in FS cache
23
Disk Read
LRU
MRU
A , 1
B , 1
C , 2
D , 3
E , 3
F , 4
Inclusive region
Exclusive region
CB
24
Disk Read
LRU
MRU
A , 1
B , 1
C , 2
D , 3
E , 3
F , 4
Exclusive region
Inclusive region
CB
25
Disk Read
LRU
MRU
A , 1
B , 1
C , 2
D , 6
E , 3
F , 4
Exclusive region
Inclusive region
CB
26
Inode Write Access time change
LRU
MRU
A , 1
B , 1
C , 2
D , 3
E , 4
F , 5
G , 7
Exclusive region
Inclusive region
CB
27
Inode Write Access time change
LRU
MRU
A , 1
B , 1
C , 2
D , 3
E , 4
F , 5
G , 7
Exclusive region
Inclusive region
CB
28
Inode Write Access time change
D , 6
E , 6
LRU
MRU
A , 1
B , 1
C , 2
F , 5
G , 7
Exclusive region
Inclusive region
CB
29
X-RAY Cache
RAID Cache (size 2 blocks)
LRU
MRU
A , 1
B , 1
C , 2
F , 5
D , 6
E , 6
G , 7
Exclusive region
Inclusive region
CB

• Keep track of additions to window in exclusive
  region

30
X-RAY Cache
RAID Cache (size 2 blocks)
LRU
MRU
A , 1
B , 1
C , 2
F , 5
D , 6
E , 6
G , 7
Exclusive region
Inclusive region
CB

• Read newly-added blocks from disk
• Replace blocks no longer in the window
• Additional disk bandwidth
• Idle time, extra internal bandwidth, freeblock
  scheduling

31
Talk Outline

• Introduction
• File Systems
• Information and Inferences
• X-RAY Cache Design
• Results
• Tracking FS Cache Contents
• RAID Cache Performance
• Conclusion

32
Results Tracking

• Accurate size and content prediction
• Highly responsive to FS cache size changes
• Tolerates changes in inode write interval
• Partial file reads
• X-RAY performs well if percentage of partially
  accessed files is lt 40 (typical traces have less
  than 30)

33
Results Cache Performance

• Performs better than LRU and Multi-Queue
• Close to DEMOTE, in spite of imperfect
  information
• Hit rate advantage translates to lower read
  latency

34
Additional Results

• File system cache policy is not LRU
• Clock, 2Q
• X-RAY performs nearly as well as before
• It performs better than both LRU and Multi-Queue
• Idle time requirements
• X-RAY reads blocks into cache only during idle
  time
• It performs well if idle time is greater than
  one-third of actual idle time observed in the
  trace
• More in the paper

35
Conclusion

• Easy deployment is an important goal in
  developing technology
• Avoid interface changes use non-invasive
  mechanisms
• Higher-level systems maintain various pieces of
  information about data they manage
• Provide low-level systems with basic semantic
  knowledge
• Semantic intelligence for managing RAID caches
• Use access information in metadata to track file
  system cache contents and cache exclusive blocks
• In spite of imperfect information, X-RAY performs
  nearly as well as changing the interface
• Semantically-smart Disk Systems
• Availability, security and performance
  improvements

36
Questions ?

• ADvanced Systems Laboratory (ADSL)
• Computer Sciences, University of
  Wisconsin-Madison
• http//www.cs.wisc.edu/adsl