Hercules:%20An%20Environment%20for%20Large-Scale%20Enterprise%20Infrastructure%20Testing

1
Hercules An Environment for Large-Scale
Enterprise Infrastructure Testing

• Lan Wang, Charles Ellis, Wei Yin, Dung Luong
• lanwang, ceellis, weiyin, dluong_at_memphis.edu
• Networking Research Lab
• Computer Science Department
• University of Memphis

2
Enterprise Infrastructure

• Hardware equipment
• Routers
• Firewalls
• Load balancers
• Servers
• Operating Systems
• Middleware
• Web server (e.g. iPlanet)
• App server (e.g. WebLogic)
• Messaging service (e.g. JMS)
• Network connectivity

3
Why is infrastructure testing important?

• Why is it necessary? Vendor testing not
  sufficient, interaction between components,
  unexpected failure scenarios
• According to a 2002 Gartner survey, 17 of
  surveyed companies had a downtime cost of 500K
  per hour.

Source Gartner Group 1
Infrastructure testing can improve the
reliability and performance of online services.
4
Hercules Project Objectives

• Develop testing methodologies to
• Uncover reliability and performance issues in any
  given enterprise infrastructure
• Ensure that infrastructure will provide adequate
  support to the applications.
• Develop tools to make infrastructure testing
  effective, efficient and simple.

5
Challenges and Our Solutions

• Test bed
• functionality/complexity/scale?
• change the infrastructure on demand and inject
  failures?
• Solution use emulation to build virtual test
  beds over physical testbed

• Traffic load
• minimal scripting by testers?
• realistic traffic?
• Solution simulate real users behavior

• Server-side Applications
• We are NOT testing the applications.
• Solution (a) use benchmark applications (b) use
  existing tested enterprise applications.

No need for off-shoring the testing!
6
How to Construct a Scalable and Flexible Testbed?

• Build virtual testbeds over one physical testbed.
• Emulate arbitrary topology, link bandwidth,
  delay, loss rate
• Users determine the OS and software.
• Allow users to swap in/out their experiments.

Example U. Utahs Emulab 2
Source www.emulab.net

• Problem existing emulation testbeds built for
  networking and OS research.
• Our contribution investigate how to conduct
  infrastructure testing using emulation testbed
  and develop necessary tools (e.g. inject
  failures).

7
How to Simulate Real Users?

• Model random arrivals of new user sessions
• Model users thinking time
• A user traverses a web site based on the
  popularity of each web page.

8
Typical Testing Scenarios
Will our web service infrastructure be able to
handle the increase in user traffic in the next
two years?
What would happen if we upgrade our application
server software to the new version?

• First, company sets up emulation environment.
• Open source software from U. Utah (a dozen
  universities and companies have set up their own
  Emulab)
• Second, use our tools to create virtual testbed
  and test cases.
• Third, run the tests that automatically launch
• our traffic generator on the clients (no
  scripting required).
• our benchmard applications on the servers.

9
Preliminary Results

1. Testing web service infrastructure using Emulab
2. Web traffic generation using stochastic
   session-based model

10
Experimenting with Emulab
Emulab can be used to create arbitrarily large
network topologies. The Java based management
client can be used to construct and maintain the
experiment layout. Connectivity with Emulab is
maintained via Secure Shell Service (SSH) to
ensure privacy.
11
Web/App Server Tests

• Emulab Virtual Topology
• 5 client machines simulating users from different
  continents
• 1 server running Apache/Tomcat

Server Throughput (requests/sec)
Test Case Avg (ms) Stddev. (ms)
100 Users 682 732
200 Users 702 787
300 Users 869 2737
Response Time
12
Web Graph

• Continuous Markov-chain model for simulating
  requests in web sessions.
• Session arrival rate ?
• Thinking time ti, time user spends on page i
• Transition probability pij, probability for a
  user to jump from one URL to another
• Exit probability 1-?j pij
• Parameters can be estimated using web access log.

13
SessionSim Traffic Generation Tool

• Each client simulates one type of traffic based
  on our model.
• Controller distributes client instances and
  parameters to many machines.
• Each machine generates a fraction of the overall
  traffic.
• The Controller monitors the aggregate traffic
  load and adjust the model parameters if needed.

14
Conclusion and Future Work

• Proposed methodology for enterprise
  infrastructure testing
• Build flexible virtual testbeds using network
  emulation
• Use stochastic session-based model to generate
  realistic user traffic for testing
• Next step develop the proposed tools and
  evaluate them.

15
Related Work

• Network testbed using emulation and overlay
  networks 2, 3, 4 different audience (OS and
  networking researchers)
• Commercial hardware traffic generators 5, 6, 7
  stress testing
• Software traffic generators 8 we model
  detailed application-level user behavior from an
  enterprise webservers point of view

16
References

1. High-Availability Networking Toward Zero
   Downtime, Gartner Group, Sept. 2002
2. University of Utah Emulab Testbed,
   http//www.emulab.net
3. Wisconsin Schooner Router Testbed
   http//www.schooner.wail.wisc.edu/
4. Resilient Overlay Network (RON),
   http//nms.csail.mit.edu/ron/
5. Ixias IxLoad, http//www.eantc.com/fileadmin/eant
   c/downloads/test_reports/2003-2005/EANTC-IxLoad_Ma
   rketingReport.v4.pdf
6. Omnicor, http//www.omnicor.com/netest.htm
7. Spirent, http//www.spirent.com/analysis/technolog
   y.cfm?media7D4WS2SS43plc8
8. Traffic Generators for Internet Traffic,
   http//www.icir.org/models/trafficgenerators.html