DADA%20

1
DADA Dynamic Allocation of Disk Area

• Jayaram Bobba
• Vivek Shrivastava

2
Outline

• Introduction
• Existing Framework
• Implementation
• Issues
• Results
• Conclusions

3
Problem

• Disk Volumes
• Allocate physical space on creation
• Multiple Disk Volumes on a physical disk
• Unused space
• Disk space not a premium
• What If?

4
Motivation

• Virtual Machine Environment
• Multiple OS may create pressure on disk space
• Access patterns may warrant dynamic relocation of
  disk blocks
• Storage Area Networks
• Storage area is a bottleneck resource
• Dynamic Physical Allocation (LFS style)
• Group writes from various volumes into a single
  physical write.

5
Related Work

• HP AutoRAID
• Dynamically change redundancy levels
• Hot Data Mirrored
• Cold Data RAID 5
• Dynamic migration of data

6
Outline

• Introduction
• Existing Framework
• Implementation
• Issues
• Results
• Conclusions

7
Existing Framework

• Logical Volume Manager (LVM)
• User space tool that enables creation of logical
  volumes of a logical partition.
• Supports resizing of volumes.
• Device Mapper (DM)
• Kernel driver that provides a level of
  indirection in address translation

8
LVM design

• Physical Volume (PV)
• Volume Group (VG)
• Logical Volume (LV)
• Physical Extent (PE)
• Logical Extent (LE)

9
LVM design
Physical Device
10
LVM design
PV
PE
11
LVM design
LV 1
PV
PE
12
LVM design
LE
LV 1
LV2
PV
PE
13
LVM design
LE
LV 1
LV2
PV
PE
14
LVM design
LE
VG
LV 1
LV2
PV
PE
15
LVM DM interaction
LVM2
libdevmapper
Userspace
Kernel
Device Mapper
16
Outline

• Introduction
• Existing Framework
• Implementation
• Issues
• Results
• Conclusions

17
Modifications

• A trap mechanism from kernel driver to user-level
  LVM code.
• Support for multiple segments in LV
• Support for virtual mappings in driver

18
Initial State
LVM daemon
LVM
User space
Kernel space
Device Mapper
Unallocated Disk Space
19
Volume Group Creation
LVM daemon
LVM
User space
Kernel space
Device Mapper
PE (4MB)
25 Extents
Unallocated Disk Space
20
Current Implementation
LVM daemon
Create LV 32MB
User A
LVM
User space
Kernel space
Device Mapper
Free Disk Space
21
Current Implementation
LVM daemon
User A
LVM
Map 0-7 extents for user A
User space
Kernel space
Device Mapper
Free Disk Space
22
Current Implementation
LVM daemon
User A
LVM
Map 0-7 extents for user A
User space
Kernel space
Device Mapper
Free Disk Space
23
Current Implementation
LVM daemon
User A
LVM
Create LV
User B
16 MB
User space
Kernel space
Device Mapper
Free Disk Space
24
Current Implementation
LVM daemon
User A
LVM
Create LV
User B
16 MB
map 8-11 extents for B
User space
Kernel space
Device Mapper
Free Disk Space
25
Current Implementation
LVM daemon
User A
LVM
User B
map 8-11 extents for B
User space
Kernel space
Device Mapper
Free Disk Space
26
Current Implementation
LVM daemon
User A
LVM
User B
Read/write
User space
Read/write
Kernel space
Device Mapper
Free Disk Space
27
Volume Group Creation
LVM daemon
LVM
User space
Kernel space
Device Mapper
PE (4MB)
25 Extents
Unallocated Disk Space
28
Modified Implementation
LVM daemon
Create LV1 32MB
User A
LVM
User space
Kernel space
Device Mapper
Free Disk Space
29
Modified Implementation
LVM daemon
User A
LVM
Virtually map 0-7 extents for user A
User space
Kernel space
Device Mapper
0-7 error
Free Disk Space
30
Modified Implementation
LVM daemon
User A
LVM
Create LV
User B
16 MB
User space
Kernel space
Device Mapper
LV A 0-7 error
Free Disk Space
31
Modified Implementation
LVM daemon
User A
LVM
Create LV2
User B
16 MB
Virtually map 0-3 extents for B
User space
Kernel space
Device Mapper
LV A 0-7 error
LV B 0-3 error
Free Disk Space
32
Modified Implementation
LVM daemon
User A
LVM
Write 6MB
User B
User space
Kernel space
Device Mapper
LV A 0-7 error
LV B 0-3 error
Free Disk Space
33
Modified Implementation
LVM daemon
User A
LVM
Write 6MB
User B
trap
User space
Kernel space
Device Mapper
LV A 0-7 error
LV B 0-3 error
Free Disk Space
34
Modified Implementation
On Demand allocation of disk area
LVM daemon
User A
LVM
Write 6MB
User B
trap
User space
Kernel space
Device Mapper
LV A 0-7 error
LV B 0-3 error
Free Disk Space
35
Modified Implementation
On Demand allocation of disk area
LVM daemon
User A
LVM
Write 6MB
User B
Allocate 0-1 extents (8MB) LV A
User space
Kernel space
Device Mapper
LV A 0-7 error
LV B 0-3 error
Free Disk Space
36
Modified Implementation
On Demand allocation of disk area
LVM daemon
User A
LVM
Write 6MB
User B
Return
User space
Kernel space
Device Mapper
LV A 0-1 linear,2-7 error
LV B 0-3 error
Free Disk Space
37
Modified Implementation
LVM daemon
User A
LVM
User B
Write 10MB
User space
Kernel space
Device Mapper
LV A 0-1 linear,2-7 error
LV B 0-3 error
Free Disk Space
38
Modified Implementation
LVM daemon
User A
LVM
User B
Write 10 MB
trap
User space
Kernel space
Device Mapper
LV A 0-1 linear,2-7 error
LV B 0-3 error
Free Disk Space
39
Modified Implementation
On Demand allocation of disk area
LVM daemon
User A
LVM
User B
Write 10 MB
trap
User space
Kernel space
Device Mapper
LV A 0-1 linear,2-7 error
LV B 0-3 error
Free Disk Space
40
Modified Implementation
On Demand allocation of disk area
LVM daemon
User A
LVM
User B
LV B Allocate 8-11 extents (12 MB)
Write 10 MB
User space
Kernel space
Device Mapper
LV A 0-1 linear,2-7 error
LV B 0-2 linear, 3 error
Free Disk Space
41
Outline

• Introduction
• Existing Framework
• Implementation
• Issues
• Results
• Conclusions

42
Issues
LVM implementation not efficient
LVM2
libdevmapper
Userspace
Kernel
Device Mapper
43
Issues
LVM implementation not efficient
LVM2
Costly kernel-user space crossing
libdevmapper
Userspace
Kernel
Device Mapper
44
Issues
LVM implementation not efficient
LVM2
Normal function call
libdevmapper
Userspace
Device Mapper
Kernel
45
More Issues

• On a trap
• How much to allocate?
• On demand
• Pre-allocate
• Where to allocate?
• Temporal locality
• Spatial locality
• Free unused space

46
How much?

• On-demand
• Allocate only as many blocks as needed
• Pre-allocate
• Pre-allocate physical space for blocks likely
• to be accessed in the future.
• How much?
• Where?

47
How much?

• Tradeoff Performance vs. Disk Space
• Depends on the workload
• I/O on critical path
• Asynchronous I/O
• Multithreaded Applications

48
Pre-Allocation

• Think Filesystem Block Allocation
• Dumb Allocation
• Stride Allocation
• Multi-Strided Allocation

49
Allocation Strategy
On Demand allocation of disk area
LVM daemon
User A
LVM
Write
Allocate 2 extents (8MB)
trap
User space
Kernel space
Device Mapper
0-1 linear2-7 error
Free Disk Space
50
Allocation Strategy
On Demand allocation of disk area
LVM daemon
User A
LVM
Write
Allocate 2 extents (8MB)
trap
User space
Kernel space
Device Mapper
0-1 linear2-7 error
Free Disk Space
51
Dumb Allocation
On Demand allocation of disk area
LVM daemon
User A
LVM
Write 6MB
Allocate 2 extents (8MB)
trap
User space
Kernel space
Device Mapper
0-1 linear2-7 error
Free Disk Space
52
Strided Allocation (Motivation)
53
Stride Allocation
On Demand allocation of disk area
LVM daemon
User A
LVM
Write
Allocate 4 extents (16MB)
trap
User space
Kernel space
Device Mapper
0-1 linear2-7 error
Free Disk Space
PreAllocated Extent
54
Outline

• Introduction
• Existing Framework
• Implementation
• Issues
• Results
• Conclusions

55
Results

• MicroBenchmarks
• mkfs
• Asynchronous io (aio) based workload
• Trace based analysis
• Trace collected by HP Research Labs used
• in the D-GRAID paper

56
Micro Benchmark mkfs

• Does the traps taken affect performance ?

57
Micro Benchmark Asynchronous I/O
Does the traps taken affect performance ?
58
Trace Based Analysis
Dumb Allocation What is the tradeoff in
choosing extent size?
59
Trace Based Analysis
Dumb Allocation What is the tradeoff in
choosing extent size?
60
Trace Based Analysis
Strided Allocation What is the tradeoff in
choosing extent size?
61
Trace Based Analysis
Strided Allocation What is the tradeoff in
choosing extent size?
62
Trace Based Analysis Dumb vs. Stride (ext2)
63
Trace Based Analysis Dumb vs. Stride (ext3)
64
Outline

• Introduction
• Existing Framework
• Implementation
• Issues
• Results
• Conclusions

65
Conclusions

• Tradeoff between performance penalty and space
• Useful in environments where disk space is the
  bottleneck
• Techniques like Pre Allocation can be used to
  reduce the performance penalty.
• For practical implementation, LVM must be placed
  in kernel address space

66
Acknowledgments

• Vijayan
• Prof. Remzi