Cloud%20Computing%20with%20Nimbus

1
Cloud Computing with Nimbus

• FNAL, January 2009
• Kate Keahey
• (keahey_at_mcs.anl.gov)
• University of Chicago
• Argonne National Laboratory

2
Cloud Computing
Elastic computing, Pay-as-you-go, Capital
expense operational expense
Science Clouds
3
Everything-as-a-Service
SaaS
PaaS
IaaS
4
The Quest Begins

• Code complexity
• Resource control

5
Workspaces

• Dynamically provisioned environments
• Environment control
• Resource control
• Hardware implementations vs virtualization

6
A Brief History of Nimbus
STAR production runs on EC2
Xen released
EC2 goes online
Nimbus Cloud comes online
2003
2009
2006
Research on agreement-based services
First Workspace Service release
Support for EC2 interfaces
EC2 gateway available
7
Nimbus Overview

• Goal open source, extensible, IaaS
  implementation and tools
• Specifically targeting scientific community
• A platform for experimentation with features for
  scientific needs
• Set up private clouds (privacy, expense
  considerations)
• Tools
• IaaS layer (Workspace Service)
• Orchestration layer (Context Broker, gateway)
• http//workspace.globus.org/

8
The Workspace Service
Pool node
Pool node
Pool node
VWS Service
Pool node
Pool node
Pool node
Pool node
Pool node
Pool node
Pool node
Pool node
Pool node
9
The Workspace Service
The workspace service publishes information on
each workspace as standard WSRF
Resource Properties.
Pool node
Pool node
Pool node
VWS Service
Pool node
Pool node
Pool node
Users can query those properties to find
out information about their workspace (e.g. what
IP the workspace was bound to)
Pool node
Pool node
Pool node
Pool node
Pool node
Pool node
Users can interact directly with their workspaces
the same way the would with a physical machine.
Trusted Computing Base (TCB)
10
Workspace Service Interfaces and Clients

• Web Services based
• Web Service Resource Framework (WSRF)
• GT-based
• Elastic Computing Cloud (EC2)
• Supported ec2-describe-images,
  ec2-run-instances, ec2-describe-instances,
  ec2-terminate-instances, ec2-reboot-instances,
  ec2-add-keypair, ec2-delete-keypair
• Unsupported availability zones, security groups,
  elastic IP assignment, REST
• Used alongside WSRF interfaces
• E.g., the University of Chicago cloud allows you
  to connect via the cloud client or via the EC2
  client

11
Security

• GSI authentication and authorization
• PKI credential required
• Works with Grid proxies
• VOMS, Shibboleth (via GridShib), custom PDPs
• Secure access to VMs
• EC2 key generation or accessed from .ssh
• Validating images and image data
• Collaboration with Vienna University of Technology

12
Networking

• Network configuration
• External public IPs or private IPs (via VPN)
• Internal private network via a local cluster
  network
• Each VM can specify multiple NICs mixing private
  and public networks (WSRF only)
• E.g., cluster worker nodes on a private network,
  headnode on both public and private network

13
The Back Story
Workspace WSRF front-end that allows clients to
deploy and manage virtual workspaces
VWS Service
Pool node
Pool node
Pool node
Workspace back-end
Pool node
Pool node
Pool node
Resource manager for a pool of physical
nodes Deploys and manages Workspaces on the
nodes
Pool node
Pool node
Pool node
Each node must have a VMM (Xen)? installed, as
well as the workspace control program that
manages individual nodes
Pool node
Pool node
Pool node
Trusted Computing Base (TCB)
14
Workspace Components
workspace resource manager
WSRF
workspace service
workspace control
EC2
workspace pilot
workspace client
15
Workspace Control

• VM image propagation
• Image management and reconstruction
• Creating blank partitions, sharing partitions
• VM control
• Starting, stopping, pausing, etc.
• Integrating a VM into the network
• Assigning MAC addresses and IP addresses
• DHCP delivery tool
• Building up a trusted (non-spoofable) networking
  layer
• Contextualization information management
• Talks to the workspace service via ssh
• Standalone component
• Some functionality overlap with libvirt
• Implementations in Xen and KVM (queued up for
  release)

16
The Workspace Resource Manager

• Basic slot fitting
• Implements immediate leases
• Extensible vehicle to experiment with different
  leases
• Open source resource manager for multiple
  different VMMs
• Datacenter technology equivalent
• Can be replaced by OpenNebula or other datacenter
  technologies
• Deployment
• University of Chicago, University of Florida,
  Purdue, Masaryk University and all the other
  Science Cloud sites

17
The Workspace Pilot

• Challenge how can I provide a virtualization
  solution without disrupting the current operation
  of my cluster?
• Flying Low the Workspace Pilot
• Integrates with popular LRMs (such as PBS, SGE)
• Implements best effort leases
• Glidein approach submits a pilot program that
  claims a resource slot
• Includes administrator tools
• Deployment
• Testing _at_ U of Victoria (Atlas), Ian Gable and
  collaborators
• Adapting for the use of the Atlas experiment _at_
  CERN, Omer Khalid

18
Cloud Closure
storage service
workspace resource manager
WSRF
workspace control
workspace service
workspace pilot
EC2
workspace client
cloud client
19
IaaS Gateway

• Goals
• Access to different IaaS infrastructures
• Account management
• Facilitate movement between academic and
  commercial clouds and creation of meta-clouds
• Combine higher-level tools and IaaS
• Released as service, not as code
• First online in June 2007, currently in a rewrite
• Used to move e.g., HEP STAR experiments between
  Science Clouds and EC2

20
The IaaS Gateway
storage service
workspace resource manager
WSRF
workspace control
workspace service
workspace pilot
EC2
IaaS gateway
EC2
potentially other providers
workspace client
cloud client
21
One-click Virtual Clusters

• Parameterizable appliance
• Tightly-coupled clusters

IP1
HK1
IP3
HK3
IP2
HK2
MPI
Reciprocal exchange of information networking
and security
22
Context Broker
IP3
HK3
IP1
HK1
IP1
HK1
IP1
HK1
IP2
HK2
IP1
HK1
IP1
HK1
IP1
IP1
IP1
IP1
IP1
HK1
IP2
HK2
IP2
HK2
IP1
IP1
IP1
IP1
IP2
HK2
IP3
HK3
IP3
HK3
IP1
IP1
IP1
IP1
IP3
HK3
Context Broker
23
Goals for Context Broker

• Can work with every appliance
• Appliance schema, can be implemented in terms of
  many configuration systems
• Can work with every cloud provider
• Simple and minimal conditions on generic context
  delivery
• Can work across multiple cloud providers, in a
  distributed environment

24
Status for Context Broker

• Release history
• In alpha testing since August 07
• First released summer July 08 (v 1.3.3)
• Latest update January 09 (v 2.2)
• Used to contextualize 100s of nodes for EC2 STAR
  runs
• Contextualized images on workspace marketplace
• Working with rPath to make contextualizatin
  easier for the user

25
End of Nimbus Tour
storage service
workspace resource manager
WSRF
workspace control
workspace service
EC2
workspace pilot
context broker
IaaS gateway
EC2
potentially other providers
context client
workspace client
cloud client
26
Science Clouds

• Make it easy for scientific projects to
  experiment with cloud computing
• Can cloud computing be used for science?
• Evolve software in response to the needs of
  scientific projects
• Start with EC2-like functionality and evolve to
  serve scientific projects virtual clusters,
  diverse resource leases
• Federating clouds moving between cloud resources
  in academic and commercial space

27
Science Cloud Resources

• University of Chicago (Nimbus)
• first cloud, online since March 4th 2008
• 16 nodes of UC TeraPort cluster, public IPs
• University of Florida
• Online since 05/08
• 16-32 nodes, access via VPN
• Other Science Clouds
• Masaryk University, Brno, Czech Republic (08/08),
  Purdue (09/08)
• Installations in progress IU, Grid5K, others
• Using EC2 for overflow
• Minimal governance model
• http//workspace.globus.org/clouds

28
Cloud Use

• 100 DNs
• Utilization
• Overall 16
• Peak pw 86 (week of 7/14)
• Requests rejected
• None till 7/14
• Lots afterwards -)

Data scaled to the nubmer of days
29
Who Runs on Nimbus?
Project diversity Science, CS, education,
buildtest
30
Hadoop over Many Clouds
U of Florida
U of Chicago
ViNE router
ViNE router

• CS research investigate latency-sensitive apps,
  e.g. hadoop
• Need access to distributed resources, and high
  level of privilege to run a ViNE router
• Virtual workspace ViNE router application VMs
• Paper CloudBLAST Combining MapReduce and
  Virtualization on Distributed Resources for
  Bioinformatics Applications by Andréa Matsunaga,
  Maurício Tsugawa and José FortesFirst, accepted
  to eScience 2008.

31
Alice HEP Experiment at CERN

• CHEP paper in preparation

32
STAR

• STAR a high-energy physics experiment
• Need resources with the right configuration
• Complex environments correct versions of
  operating systems, libraries, tools, etc all have
  to be installed.
• Consistent environments require validation
• A virtual OSG STAR cluster
• OSG cluster
• OSG CE (headnode), gridmapfiles, host
  certificates, NSF, PBS
• STAR worker nodes SL4 STAR conf
• Requirements
• One-click virtual cluster deployment
• Migration Science Clouds -gt EC2

33
STAR (cntd)

• From proof-of-concept to production runs
• 2 years ago proof-of-concept
• Last September EC2 runs of up to 100 nodes
  (production scale, non-critical codes)
• Testing for critical production deployment
• Performance
• Within 10 of expected performance for
  applications
• Work by Jerome Lauret, Doug Olson, Leve Hajdu,
  Lidia Didenko

34
Scalability Testing

• Motivation
• Test scalability of various Globus components
• Test on a different platforms
• Workspaces
• Globus 101 others
• Requirements
• very short-term but flexible access to diverse
  platforms
• Work by various members of the Globus Toolkit
  (Tom Howe and John Bresnahan)
• Resulted in provisioning a private cloud for
  Globus
• Typically very short-lived communities of one

35
Montage Workflows

• Evaluating a cloud from users perspective
• Paper Exploration of the Applicability of Cloud
  Computing to Large-Scale Scientific Workflows,
  C. Hoffa, T. Freeman, G. Mehta, E. Deelman, K.
  Keahey, SWBES08 Challenging Issues in Workflow
  Applications

36
Cloud Computing Ecosystem
Appliance Providers marketplaces commercial
providers communities
Deployment Orchestrator orchestrate the
deployment of environments across possibly many
cloud providers
User Environments
VMM/datacenter/IaaS
37
Open Source IaaS Implementations

• Eucalyptus
• Open source implementation of EC2
• UCSB, R. Wolski team, 06/2008
• OpenNebula
• Open source datacenter implementation
• University of Madrid, I. Llorente team, 03/2008
• Cloud-enabled Nimrod-G
• Monash University, MeSsAGE Lab, 01/2009
• Industry efforts
• openQRM, Enomalism

38
Friends and Family

• Committers Kate Keahey Tim Freeman (ANL/UC),
  Ian Gable (UVIC)
• A lot of help from the community, see
  http//workspace.globus.org/people.html
• Collaborations
• Cumulus S3 implementation (Globus team)
• EBS implementation with IU
• Appliance management rPath and Bcfg2 project
• Virtual network overlays University of Florida
• Security Vienna University of Technology

39
To the Future and Beyond

• Increasing Importance of Appliance Providers
• Cloud computing tools
• Increased interest in cloud interoperability
• Standards rough consensus working code
• Image formats, contextualization capabilities,
  cloud interfaces, etc.
• Cloud markets