Redundant Data Update in Server-less Video-on-Demand Systems

1
Redundant Data Update in Server-less
Video-on-Demand Systems

• Presented by Ho Tsz Kin

2
Agenda

• Background
• Redundant Data Regeneration
• Sequential Redundant Data Update(SRDU)
• Update Overhead Analysis
• Performance Evaluation
• Conclusion and Future Works

3
Background

• SLVoD System

Data Nodes
Receiver
Redundancy Nodes
4
Background

• Data Placement

Data Nodes
Redundancy Nodes
d2
d1
d0
d3
r0
r1
Same Parity Group
v0
v1
v2
v3
c0,0
c0,1
Q-bytes data blocks
v4
c1,0
c1,1
v5
v6
v7
Q-bytes redundant data blocks
v8
c2,0
c2,1
v9
v10
v11
v12
c3,0
c3,1
v13
v14
v15
v16
c4,0
c4,1
v17
v18
v19
5
Background

• New nodes join the system
• Data blocks are reorganized to utilize storage
  and streaming capacity
• Data blocks in Parity Group change
• Redundant data blocks need to update/re-compute

d2
d1
d0
d3
r0
r1
d4
v0
v1
v2
v3
c0,0
c0,1
v4
Redundant data blocks CHANGE!
v5
c1,0
c1,1
v6
v7
v8
v9
Data blocksin parity group CHANGE!
v10
c2,0
c2,1
v11
v12
v13
v14
v15
c3,0
c3,1
v16
v17
v18
v19
6
Redundant Data Regeneration

• Regenerate new redundant blocks by all the data
  blocks
• Data blocks are distributed among nodes
• Send all the data blocks to redundancy node
• Compute the new redundant
• blocks in redundancy node
• Overhead is too high

Data Nodes
Redundancy Node
7
Redundant Data Regeneration

• Data blocks are stored in central archive server
• Regenerate all the new redundant data in server
• Send to redundancy nodes
• Server may be costly, not desirable, not feasible

Server
Redundancy Node
8
Sequential Redundant Data Update

• Idea
• Old redundant data blocks contain valuable
  information on the data blocks
• Data blocks are different after reorganization
• But some blocks in parity group is the SAME
• Possible to reuse for linear code

d2
d1
d0
d3
r0
r1
Old Redundant data blocks
v0
v1
v2
v3
c0,0
c0,1
d2
d1
d0
d3
r0
r1
d4
New Redundant data blocks
v0
v1
v2
v3
c0,0
c0,1
v4
9
Sequential Redundant Data Update

• Linear erasure correction code
• Can be analyzed using linear algebra
• Linear matrix multiplications in computing
  redundant data
• Reed-Solomon Erasure Correction Code
• Redundant data by Vandermonde matrix
• Number of nodes, N
• Number of redundancy node, h
• Element at row i, column j ji-1

Data from data node 0
Redundant data in redundancy node 0
10
Sequential Redundant Data Update

• Rounding or truncating would prevent a correct
  reconstruction of source data
• 1 / 3 0.3333, decoding fails
• Arithmetic over Galois Fields
• Finite Fields
• Finite elements
• Add, subtract, multiply, divide similar to
  algebra
• Closed under these operations
• GF(2w) are integers from zero to 2w-1
• Fixed bits to represent an element
• Assume a Q-bytes data block can be represented by
  a field element

11
Sequential Redundant Data Update

• Sequential Redundant Data Update(SRDU) consists
  of 3 techniques
• Direct reuse of old redundant data
• Parity group reshuffling
• Caching of transmitted data

12
SRDU Direct Reuse

• Construct 1st parity group
• Before node addition

d2
d1
d0
d3
r0
r1
v0
v1
v2
v3
c0,0
c0,1

• After node addition

d2
d1
d0
d3
r0
r1
d4
v0
v1
v2
v3
c0,0
c0,1
v4
Only need to send v4, Save 4 blocks
13
SRDU Direct Reuse

• Construct 2nd parity group
• Before node addition

d2
d1
d0
d3
r0
r1
v4
v5
v6
v7
c1,0
c1,1

• After node addition

d2
d1
d0
d3
r0
r1
d4
v5
v6
v7
v8
c1,0
c1,1
v9
Terms are different. Direct Reuse Fails!
14
SRDU Parity Group Reshuffling

• The order during encoding is not important
• Parity group reshuffling
• Before node addition
• After node addition

Only need to send v4, v8 and v9 Save 2 blocks
15
SRDU Parity Group Reshuffling

• Choices of old redundant data for reusing

• Before node addition
• After node addition

d2
d1
d0
d3
r0
r1
v8
v9
v10
v11
c2,0
c2,1
d2
d1
d0
d3
r0
r1
d4
v5
v6
v7
v8
c1,0
c1,1
v9
5 blocks needed, More overhead
16
SRDU - Caching

• Caching of transmitted data in constructing
  previous redundant block
• Reuse them to construct current redundant blocks
• Construct 1st parity group
• Construct 2nd parity group

v4 is cached in the redundancy node
v4 can be reused from locally cached data,Only
v8 and v9 need to send
17
SRDU - Caching

• The buffer for caching is at most (N-h)
• Data blocks from earlier than (N-h) blocks are
  not useful for reusing
• Construct the 4th parity group

All previous blocks are useless
c4,1
c3,1
Old redundant block
New redundant block
At most 4 data blocks are cached
c3,1
18
SRDU - Summary

• For construction of each new redundant block
• Reuse old redundant block by
• Direct reuse
• Parity group reshuffling
• Caching of transmitted data
• Select the one with lowest overhead to reuse
• Send the required blocks only

19
Update Overhead Analysis

• Total data blocks, B
• Redundant Data Regeneration
• Data blocks are distributed among nodes
• Total overhead B (h-1) B/(N-h)

All data blocks are sent, overhead B
All redundant data are constructed here
overhead B / (N-h)
20
Update Overhead Analysis

• Data blocks are stored in central archive server
• Total overhead h B/(N-h)

All redundant data are constructed in server
overhead B / (N-h)
21
Update Overhead Analysis

• Sequential Redundant Data Update(SRDU)
• Total overhead block movement under SRDU (h
  -1) B/(N-h)

Data blocks are sent under decision of SRDU
Combine the old redundant data and received
partial result
Compute partial result of each new redundant
blocks
overhead B / (N-h)
22
Update Overhead Analysis

• Redundant data Regeneration
• Data blocks are distributed among nodes
• Total overhead B (h-1) B/(N-h)
• Data blocks are stored in central archive server
• Total overhead B/(N-h) (h-1) B/(N-h)
• Sequential Redundant Data Update(SRDU)
• Total overhead blocks movement under SRDU
• (h-1) B/(N-h)
• Sufficient to consider overhead for one redundant
  update as performance metric

23
Performance Evaluation

• Update overhead in continuous system growth
• Number of data blocks 40000
• Grow from 1 node to 400 nodes, one node added
  each

24
Performance Evaluation

• Update overhead in batched redundancy update
• Defer update until a fixed number of nodes, W
• Initial system size 80, W ranges from 1 to 200

25
Conclusion

• Identify the redundant data update problem
• Investigate redundant data regeneration
• Propose SRDU to reduce the overhead substantially
  (about 75)
• Further reduce the overhead by batched redundancy
  update (about 97 for a batch size of 10)

26
Future Works

• Distributed Redundant Data Update and its
  scheduling
• Mathematical Modeling
• Optimality of redundant data update
• Redundant Data Addition