Distributed Lock Manager on a Network Processor

1
Distributed Lock Manager on a Network Processor

• J. Lu and J. Wang
• Department of Computer Science
• University of Massachusetts Lowell

2
Outline

• Motivation
• Background
• Design
• Overview
• Core lock manager
• Deadlock detection
• API
• Future work

3
Motivation

• Parallel applications can be implemented either
  on a parallel computer or on a distributed system
• Mutual exclusion on a single computer has a
  simple and elegant solution
• Distributed mutual exclusion has a significant
  impact on performance of parallel applications
• Looking for new ways to improve mutual exclusion
  in a distributed system, centered on an IXP
  network processor

4
Background

• Centralized solution
• Small and fixed number of messages
• Single point of failure
• Bottleneck at control site
• Distributed solutions
• Non-token based and token based algorithms
• Overcome the drawbacks of centralized approach
• Increase complexity
• Require large number of messages

5
Distributed Lock Manager(DLM)

• Master node grants lock requests
• Directories keep track of the lock-master mapping
  information

Master Node
5. Grant lock
4. Request for lock
Client
3. Reply with master node information

• Find directory node based on requested resource
  name

Directory Node
2. Query for master node
6
Distributed Lock Manager(DLM)

• Compromise between the centralized approaches and
  distributed approaches
• Centralized control for a particular lock
  resource
• Distributed masters and directories
• Widely adopted for parallel applications
• E.g., in the Oracle Parallel Server
• Software implementation at application layer
• Extra communications with directory nodes
• More failure points, with complex recovery
• Complex deadlock detection

7
System architecture
8
Locking model

• Same locking model of DLM
• Each lock resource represents a lockable entity
• Each lock resource has three lock queues
• Grant
• Convert
• Wait
• Six modes of locks

9
Core Lock Manager

• Manages the pool of all lock resources
• Accesses lock resources with a hash table
• Queues up the lock requests and grants locks
  accordingly
• Takes advantage of the IXP built-in hardware
  support for FIFO (ring and queue)
• Unordered thread model on a dedicated ME
• Eight hardware assisted threads provides parallel
  capability
• ME offers hardware features to support
  synchronization mechanisms
• IXP2xxx provides the advanced hardware features

10
Lock request processing

• Packet flow

Receive (ME)
Transmit (ME)
Others
A/C
R
Core Lock Manager(ME)
11
Lock request processing

• Lock acquiring request

Requested lock mode compatible with the most
restrictive granted lock mode ?
Generate hash key
Grant the lock
G
Lock resource
C
Search hash table
Block the lock
W
Lock resource hash table
12
Lock request processing

• Lock converting request

Requested lock mode compatible with other most
restrictive granted lock mode ?
Grant the lock
G
Block the lock
Lock resource
C
Search hash table
W
Lock resource hash table
13
Lock request processing

• Lock releasing request

G
1. Remove lock
Lock resource
C
Search hash table
2. Process convert queue
3. Process wait queue
W
Lock resource hash table
14
Lock request processing

• Lock cancel request

G
Lock resource
C
Search hash table
Remove lock
W
Lock resource hash table
15
Deadlock detection

• Deadlock detection for distributed system is
  complex and affects performance significantly
• Centralized algorithm can be used here
• XScale core is powerful but often sits idle
• Builds and reduces the resource wait-for-graph to
  detect cycle on demand
• Force client to release lock to resolve deadlock
• Does not modify lock resource data structures

16
Client API

• Transparent API for application to create,
  acquire, or convert a lock in either synchronous
  or asynchronous mode.
• Maintains local granted and pending requests
• Lock information
• Client information
• Communicates with core lock manager via protocol

17
Client API

• Error handling
• Sanity check for improper calls
• Monitors aliveness of client and cleans locks if
  a client is abnormally terminated
• Initiates deadlock detection process
• Configurable time out value
• Timestamp each request

18
Future work

• Simulation comparisons on performance
• Running DLM on a PC vs. running DLM on an IXP2400
• Performance enhancement
• Implement cache in ME local memory
• Improve protocol with security consideration
• Extend the approach to other areas
• Distributed shared memory
• Parallel cache management
• Global resource manager

19
Distributed Lock Manager on a Network Processor

• J. Lu and J. Wang
• Department of Computer Science
• University of Massachusetts Lowell