CPS110: Wrapping up memory

1
CPS110 Wrapping up memory

• Landon Cox
• March 6, 2008

2
Traditional OS structure
App
App
App
App
Operating System
Host Machine
3
OS abstractions
Last month of class
Applications
OS
Instructions CPU
Virtual addrs Physical mem
Syst calls I/O devices
Hardware
What are the interfaces and the resources? What
is being virtualized?
4
Courser abstraction virtual machine

• Weve already seen a kind of virtual machine
• OS gives processes virtual memory
• Each process runs on a virtualized CPU
• Virtual machine
• An execution environment
• May or may not correspond to physical reality

5
Virtual machine options

• How to implement a virtual machine?
• Interpreted virtual machines
• Translate every VM instruction
• Kind of like on-the-fly compilation
• VM instruction ? HW instruction(s)
• Direct execution
• Execute instructions directly
• Emulate the hard ones

6
Interpreted virtual machines

• Implement the machine in software
• Must translate emulated to physical
• Java byte codes ? x86, PPC, ARM, etc
• Software fetches/executes instructions

Program (foo.class)
Interpreter (java)
Byte code
x86
What does this picture look like?
Dynamic virtual memory translator
7
Java virtual machine

• What is the interface?
• Java byte-code instructions
• What is the abstraction?
• Stack-machine architecture
• What are the resources?
• CPU, physical memory, disk, network
• The Java programming language
• High-level language compiled into byte code
• Library of services (kind of like a kernel)
• Like C/STL, C

8
Direct execution

• What is the interface?
• Hardware ISA (e.g. x86 instructions)
• What is the abstraction?
• Physical machine (e.g. x86 processor)
• What are the resources?
• CPU, physical memory, disk, network

x86
Program (XP kernel)
x86
x86
Monitor (VMware)
9
Different techniques

• Emulation
• Bochs, QEMU
• Full virtualization
• VMware
• Paravirtualization
• Xen
• Dynamic recompilation
• Virtual PC

10
Virtual machines are hot
VMware IPO 19.1 billion
Xen sale 500 million
11
Views of the CPU

• How is a processs view of the CPU different than
  the OSs?
• Kernel mode
• Access to physical memory
• Manipulation of page tables
• Other privileged instructions
• Turn off interrupts
• Traps
• Keep these in mind when thinking about virtual
  machines

12
Virtual machine structure
Guest App
Guest App
Guest App
Guest OS
Guest OS
Guest OS
Virtual Machine Monitor (Hypervisor)
Host Machine
13
Why are hypervisors useful?

• Code reuse
• Can run old operating systems apps on new
  hardware
• Original purpose of VMs by IBM in the 60s
• Encapsulation
• Can put entire state of an application in one
  thing
• Move it, restore it, copy it, etc
• Isolation, security
• All interactions with hardware are mediated
• Hypervisor can keep one VM from affecting another
• Hypervisor cannot be corrupted by guest operating
  systems

14
Encapsulation

• Say I want to suspend/restore an application
• I decide to write the process mem PCB to disk
• I reboot my kernel and restart the process
• Will this work?
• No, application state is spread out in many
  places
• Application might involve multiple processes
• Applications have state in the kernel (lost on
  reboot)
• (e.g. open files, locks, process ids, driver
  states, etc)

15
Encapsulation

• Virtual machines capture all of this state
• Can suspend/restore an application
• On same machine between boots
• On different machines
• Very useful in server farms
• Well talk more about this with Xen

16
Security

• Can user processes corrupt the kernel?
• Can overwrite logs
• Overwrite kernel file
• Can boot a new kernel
• Exploit a bug in the system call interface
• Ok, so Ill use a hypervisor. Is my data any
  less vulnerable?
• All the state in the guest is still vulnerable
  (file systems, etc)
• So whats the point?
• Hypervisors can observe the guest OS
• Security services in hypervisor are safe, makes
  detection easier

17
Security

• Hypervisors buggy too, why trust them more than
  kernels?
• Narrower interface to malicious code (no system
  calls)
• No way for kernel to call into hypervisor
• Smaller, (hopefully) less complex codebase
• Should be fewer bugs
• Anything wrong with this argument?
• Hypervisors are still complex
• May be able to take over hypervisor via
  non-syscall interfaces
• E.g. what if hypervisor is running IP-accessible
  services?
• Paravirtualization (in Xen) may compromise this

18
VMware architecture
Host World
VMM World
Target App
Target App
VM App
Target OS
Host App
Host OS
Virtual Machine Monitor
VM Driver
Host Machine
19
SimOS (proto-VMware) arch.
Target App
Target App
Target OS
SimOS
Host App
Host App
Host OS
Host Machine
20
SimOS memory
SimOS
SimOS VMemory
SimOS code, data
Target OS
TargOS code, data
Target App
TargApp code, data
Target App
Virtual MMU
SimDisk
Host OS
SimDisk File
Mem File
Host Machine
21
SimOS page fault
SimOS
SimOS Fault handler
SimOS VMemory
What if I want to suspend and migrate the target
OS?
TargOS Fault handler
Target OS
Unmapped addr
Target App
Target App
Virtual MMU
SimDisk
Host OS
SimDisk File
Mem File
Host Machine
22
Full vs interpreted

• Why would I use VMware instead of Java?
• Support for legacy applications
• Do not force users to use a particular language
• Do not force users to use a particular OS
• Why would I use Java instead of VMware?
• Lighter weight
• Nice properties of type-safe language
• Can prove safety at compile time

23
Full vs interpreted

• What about protection?
• What does Java use for protection? VMware?
• Java relies on language features (cannot express
  unsafe computation)
• VMware relies on the hardware to enforce
  protection (like an OS)
• What are the trade-offs? Which protection model
  is better?
• Java gives you stronger (i.e. provable) safety
  guarantees
• Hardware protection doesnt constrain programming
  expressiveness
• What about sharing (kind of the opposite of
  protection)?
• Sharing among components in Java is easy
• (call a function, compiler makes sure it is safe)
• Sharing between address spaces is more work, has
  higher overhead
• (use sockets, have to context switch, flush TLB,
  etc)

24
Singularity (could try both)
25
Virtual machine challenges

• Privilege modes
• Memory management
• Protection
• Performance
• Many more for every architecture

26
Course administration

• Multi-process test cases
• Autograder will test your pager with gt 1 process
• But dont submit any to the autograder
• How to write multi-process test cases
• Use vm_yield
• Can quickly open processes in different windows
• Can use sleep (unsigned int seconds)
• Could use fork

27
Course administration

• Extra office hours after spring break
• Will announce over Blackboard
• Other questions?

28
Sharing machines among users

• PlanetLab (752 nodes at 361 sites)
• Platform for distributed applications
• Research testbed
• Why is this more useful than a cluster?
• See real Internet problems
• Latency, failures, etc
• Service fault tolerance

29
Sharing machines among users

• Consolidate under-utilized servers to reduce
  CapEx and OpEx

X

• Avoid downtime with relocation

• Dynamically re-balance workload to guarantee
  application SLAs

X

• Enforce security policy

What about the enterprise?
30
Sharing machines among users

• When?
• PlanetLab (testbeds, distributed services)
• Data centers (three-tier web applications)
• Scientific computing (protein folding, etc)
• What should the interface be?

31
Shared infrastructure interfaces

• Unmodified OS
• Each app gets a login username
• Access resources through system calls
• Users can see other users files, processes
• Drawbacks of this approach?
• Administration, configuration headaches
• (e.g. which libraries are installed?)
• No performance isolation
• (one process can dominate CPU, buffer cache,
  bandwidth)

32
Shared infrastructure interfaces

• Unmodified OS
• Retrofit resource accounting into OS (V-Servers)
• Access resources through system calls
• Virtualize some resources
• (e.g. each app has own process table, file
  system)
• Drawbacks of this approach?
• How do you know that youve virtualized
  everything you need to
• Especially hard for software resources
• (e.g. what about entries in the file descriptor
  table?)
• (e.g. who gets charged on a page fault?)

33
Shared infrastructure interfaces

• Unmodified OS
• Retrofit resource accounting into OS (V-Servers)
• Virtual machines (Xen, VMware)
• Virtualize hardware interface
• Each app gets to choose its own OS
• (e.g. apps have their own virt. CPU, physical
  memory, disk)
• Drawbacks of this approach?
• Very heavy-weight
• A lot of redundant state (e.g. kernel, libraries,
  executables)
• Might not scale well

34
Amazon EC2
Anyone know what EC2 uses?
Xen
35
Xen challenges

• Kind of the opposite approach of V-Servers
• V-Servers start with OS, virtualize
• Xen start with VM, para-virtualize
• Goals
• Performance isolation
• Support many operating systems
• Reduce performance overhead of virtualization

36
Para-virtualization

• Full virtualization
• Fool OS into thinking it has access to hardware
• Para-virtualization
• Expose real and virtual resources to OS
• Why do we need para-virtualization?
• Mostly because X86 makes full virtualization hard

37
Why para-virtualize?

• Limitations of x86
• Privileged instructions fail silently
• VMM must execute these instructions
• Cannot rely on traps to VMM
• How does VMware deal with this?
• At run-time rewrite guest kernel binary
• Insert traps into the VMM, when necessary

38
Why para-virtualize?

• Limitations of x86
• Timing issues
• May want to expose real time to OS
• TCP time outs, RTT estimates
• Support for performance optimizations
• Superpages
• Page coloring

39
VMware architecture
Host World
VMM World
Target App
Target App
VM App
Target OS
Host App
Host OS
Virtual Machine Monitor
VM Driver
Host Machine
40
SimOS architecture
Target App
Target App
Target OS
SimOS
Host App
Host App
Host OS
Host Machine
41
Xen architecture
Guest App
Guest App
Guest OS
Host App
Guest OS
Xen
Domain 0
Host Machine
42
X86_32 address space
When are each set of virtual addresses are valid?
All address spaces
4GB
All of a VMs address spaces
S
Xen
Kernel
S
3GB
Each guest app
User
U
0GB
When does the hypervisor need to flush the
TLB? When a new guest VM or guest app needs to be
run.
43
Xen physical memory

• Allocated by hypervisor when VM is created
• Why cant we allow guests to update PTBR?
• Might map virtual addrs to physical addrs they
  dont own
• VMware and Xen handled this differently
• VMware maintains shadow page tables
• Xen uses hypercalls
• (update Xen and VMware support both mechanisms
  now)

44
VMware guest page tables
Virtual ? Machine
Update PTE
Guest OS
How does VMM grab control when PTE is
updated? Marks PTE pages read-only, generates
page fault.
Shadow page table
VMM
Hardware
MMU
45
Xen physical memory

• Guest OSes allocate and manage own PTs
• Hypercall to change PT base
• Like a system call between guest OS and Xen
• Xen must validate PT updates before use
• What are the validation rules?
• 1. Guest may only map phys. pages it owns
• 2. PT pages may only be mapped RO

46
Xen guest page tables
Virtual ? Machine
Update PTE hypercall
Guest OS
1) Validation check 2) Perform update
VMM
Hardware
MMU
47
Para-virtualized CPU

• Hypervisor runs at higher privilege than guest OS
• Why is having only two levels a problem?
• Guest OSes must be protected from guest
  applications
• Hypervisor must be protected from guest OS
• What do we do if we only have two privilege
  levels?
• OS shares lower privilege level with guest
  applications
• Run guest apps and guest OS in different address
  spaces
• Why would this be slow?
• VMM must flush the TLB on system calls, page
  faults

48
X86_32 address space
4GB
S
Xen
Ring 0
Ring 1
Kernel
S
3GB
Ring 3
User
U
0GB
What does this assume? Guest OS doesnt need ring
2 (e.g. OS/2).
49
Para-virtualized CPU

• Hypervisor runs at higher privilege than guest OS
• Ring 0 ? hypervisor, ring 1 ? guest OS, ring 3 ?
  guest apps
• Handling exceptions
• Guest registers handlers with Xen (must modify
  guest)
• System calls
• Guests register "fast" handler with Xen
• Xen validates handler, inserts in CPUs handler
  table
• No need to go to ring 0 to execute
• What handler cannot be executed directly and why?
• Page fault handler must read register CR2 (only
  allowed in ring 0)
• CR2 is where the fault-generating address is
  stored

50
Para-virtualization

• Pros
• Better performance
• Scales better than full virtualization
• Cons
• OS needs (minor) changes
• How well does it actually scale? Unclear at this
  point
• Impure abstractions
• Is it important to provide good abstractions?
• I say yes
• Bad interfaces lead to code complexity,
  maintainability issues