Zap

1
Zap
A System for Migrating Computing Environments

• Steven OsmanDinesh SubhravetiGong SuJason Nieh

2
Benefits of Migration

• Dynamic Load Balancing
• Mobility
• Data Access Locality
• Improved Administration
• Fault Resilience

3
Clustered System Approach

• Single system image across a cluster
• Good for load-balancing
• Examples include, MOSIX, Sprite
• May leave dependency on previous host
• May be new operating system or significant kernel
  changes

4
Middleware/Language Approach

• Object-based approach using special programming
  language or middleware
• Examples include, Abacus, Emerald, Globus,
  Legion, Rover
• Requires applications to be rewritten

5
User-level Approach

• No operating system changes
• Good for long-running applications
• Examples include, Condor, CoCheck, Libckpt, MPVM
• Does not support many common operating system
  services

6
Virtual Machine Monitor Approach

• Support any operating system
• No application changes
• Example, using VMware for migration
• Must migrate the whole operating system
• Potentially higher overhead

7
Introducing Zap

• Transparent migration
• Unmodified legacy applications
• Networked applications
• Commodity operating system
• Minimal operating system changes
• Leaves nothing behind
• Low overhead

8
Outline

• Background Motivation
• Difficulties of Migration
• Zap components
• Virtualization
• Migration
• Experimental Results
• Conclusion

9
Migration Difficulties

• int iChildPID
• if (iChildPIDfork())
• / parent does some work /
• waitpid(iChildPID)
• else
• / child does some work /
• exit(0)

10
Resource Consistency Problem
Host A
Host B
Parent invoked waitpid(20)
11
Resource Conflict Problem
Host B
Host A
PID 20
PID 20
Resources May Conflict With Other Processes
12
Resource Dependency Problem
Host A
Host B
Parent
Parent
Child
Parent and child depend on each other
13
Problem Recap

• Resource consistency
• Names cant change
• Resource conflict
• Names cant be duplicates
• Resource dependency
• Migration must be complete

14
Solution

• Group processes into a POD (Process Domain) that
  has a private virtual namespace
• PODs can contain one process, one group of
  processes, or a whole user session
• PODs are migrated as a unit
• Solves
• Resource consistency problem
• Resource conflict problem
• Resource dependency problem

15
Zap Architecture

• Zap combines
• Thin virtualization layer
• Checkpoint/restart mechanism
• Checkpoint/restart offers
• Easier to implement than demand paging
• Leaves nothing behind
• Suspend sessions
• Easily configure and clone environments
• Dynamic system configuration

16
What Should Zap Virtualize?

• Process identifiers (PIDs)
• Inter-process communication (IPC) keys
• Memory
• File system structure
• Network connections
• Devices

17
PID and IPC Key Virtualization Migration

• Create unique namespace for the POD
• Names are virtualized
• When entering a system call, replace POD virtual
  identifiers with real ones
• When exiting a system call, replace real return
  values with POD virtual ones
• Mask out identifiers that do not belong to the POD

18
Memory Virtualization Migration

• Like IPC, create unique shared memory namespace
• Modern architectures support virtual memory

Thank you modern architectures!
Migration optimization Move only data pages,
code pages can be remapped
19
File System Virtualization Migration

• Some filenames cant conflict
• /var/run/httpd.pid
• Some directories have unique configuration
• /etc
• Some directories depend on the current processes
• /proc

20
File System Virtualization Migration

• Create a directory structure for POD
• Use network file systems
• Create private POD directories
• Good for /tmp, /var POD specific configuration
• Private /proc directory
• Private /dev directory

21
File System Example
Host FS
binetcpod bin ? NFS/pods/bin dev
? Dynamic proc? Dynamic tmp ?
Private POD
POD FS
Use chroot() to map POD root directory
22
Networking Virtualization Migration

• Two network addresses
• Persistent internal address
• Host-dependent external address
• For connection migration
• Transport layer sees virtual address
• Network layer sees real address
• Transport layer independent
• Initial virtual address is real address

23
Virtual Networking
TransportADDR. 1
TransportADDR. 2
24
Device Virtualization Migration

• Device migration is hard
• Pseudo Terminal
• Sound Device
• CDRW During a Recording Session
• Electron Microscope

25
Device Migration Virtualization

• Pseudo Terminal ? Virtual device
  configurationdata
• Sound Device ? Virtual device configuration
• Recording CDRW ? Migrate later
• Electron Microscope ? Communicate with original
  host

26
Device Migration Virtualization

• Unsupported devices do not appear in a PODs /dev
• Zap currently supports pseudo terminals, ensuring
  their names are consistent after migration (e.g.
  /dev/pts/2)

27
Zap Implementation

• Developed for Linux 2.4
• Zap design enables
• Loadable kernel module
• No need to rebuild the kernel
• Intercept system calls for virtualization

28
Zap Implementation
User space
User Processes
kernel space
ZAP Virtualization
System Calls
Zap Migration
Kernel
29
Virtualization Cost

• Created micro-benchmarks
• PID calls (getpid)
• IPC calls (shmget/ctl, semget/ctl)
• Process creation calls (fork, execve, exit)
• Used macro-benchmarks
• Apache
• Build Linux kernel
• Volano

30
Virtualization Results
31
Virtualization Results

• Zap incurs low overhead

32
Migration Cost VNC Session
33
Migration Cost Apache

• Apache 2.0.35
• Default configuration

34
Migration Cost Time
35
Migration Cost Space
36
Migration Cost

• Zap can be fast
• lt1 second checkpoint/restart times
• Includes Zap networking round-trip

37
Zap

• Offers transparent migration of legacy and
  network applications
• Introduces PODs
• Consistency
• Conflict free
• Avoids Unwanted dependencies
• Leaves nothing behind
• Fast and lightweight

38
For more information

• Zap computing
• http//www.ncl.cs.columbia.edu/research/migrate
• Network Computing Laboratory
• http//www.ncl.cs.columbia.edu/

39
Future Work

• Secure migration
• Trusted images, POD sandbox, etc.
• Generalized device support
• Migration policies
• Heterogeneity
• Contextualization
• Resource management