Load Testing

1
Load Testing

• Klaus Fellner, Gregor Rechberger

2
Goals of the Session

• Understand general industry information and
  trends
• Learn SilkPerformers capabilities and
  positioning
• Be able to articulate why SilkPerformer is a
  superior solution (Features-Advantages-Benefits)

3
Industry Information and Trends

• Load Testing

4
The Many Objectives of Load Testing

• Benchmarking
• Test QoS of various architectural options
• Regression testing
• Verify QoS from build-to-build
• Application tuning
• Optimize an applications QoS
• Capacity planning
• Accurately predict the required system and
  network capacity
• Service level tests
• Test against service levels to meet them once you
  go live
• Acceptance tests

5
Need for Tuning Data Brings Testing and
Diagnostics Closer Together
GUI Logic
Business Logic
Shows whats going on inside the black-box
Data Logic
Application Stack
6
Need for Earlier Testing Leads to Convergence Of
Development QA

• Fixing a bug late in the dev cycle costs 80 to
  1,000 times more than it would if it was dealt
  with earlier Gartner 2003
• 1/3 of today's costs for fixing software defects
  can be eliminated by earlier testing Gartner
  2003
• Developers need to start testing early
• Development platforms start to integrate test
  management and basic testing capabilities
• Microsoft Visual Studio Team Test
• Project Hyades (Eclipse/Java)

7
What to Load Test?
AppServer
WebServer
Internal User
Firewall
Load Balancer
Database
App Server
...
External User
WebServer
AppServer
Table n
...
...
8
SilkPerformerThe most powerful yet easiest to
use enterprise-class load and stress testing
tool
9
Manual Testing is Problematic
Go!
Difficult to synchronize
ResultAnalysis?

• Resource intensive
• Testing staff
• Client computers

Coordinator
Internet
Repeatability?
System Under Test
Testers
10
Straightforward Test Design via Point Click
Script Recording

• End-to-end Load Test

• Component-level Stress Test

11
High Flexibility, Reuseabilty and Ease-of-use
Through Powerful Test Design Tools
TrueLog provides ease-of-use
BDL provides power flexibility
Project attributesincreasereusability
Define any possible workload scenario quickly
easily
12
What Are The Benefits Of SilkPerformer's Test
Design Capabilities?

• Advantages
• Easily build test scripts from scratch
• Reuse unit test drivers built by developers for
  component-level stress testing
• Reuse and adapt existing test scripts
• Simulate every possible workload scenario,
  including those used to isolate a problem
• Ignore irrelevant errors
• Benefits
• Time saving
• Reduced risk of missing error situations
• Accurate test results

13
Test Performance, Scalability and Reliability of
Enterprise Applications
Analyze performance bottlenecks
Reports
Application
Multi-Machine Controller (MMC)
Server
Performance from the VU perspective
Reliability from the VU perspective
14
What Are The Benefits Of This Test Execution and
Results Analysis Architecture?

• Advantages
• Accurate performance assessment based on true
  end-user experience
• Support for testing service levels to meet them
  once you go live
• Test IP load balanced sites
• Scale up to higher loads than possible from a
  single box
• Immediately get back information for tuning the
  application as well as the underlying system
• Automatically get back custom management reports
• Find out what happened from the end-user
  perspective and easily nail down the root cause
  of an error
• Benefits
• Reduced risk of project failures
• Reduced project cycle times
• Reduced error turnaround time
• Lower ongoing maintenance costs

15
Support For Closed-Loop Testing Closes The
Knowledge Gap Between RD and IT
SCPM
Production
Extendsquality optimization to the completeSLC
SilkPerformerSilkTest
Pre-Production
Reuse application knowledge inside RD for APM
Get realistic test cases through recording
erroneous real-user sessions
16
Support for All Major Enterprise Applications and
Technologies
Use a single tool for load testing all your
enterprise applications

• Client/Server (Databases)
• ODBC
• ADO
• Oracle OCI
• IBM CLI

• Web Browser
• HTTP(S)/HTML
• Macromedia Flex/AMF
• Streaming (MS, Real)

• Internet
• HTTP(S)
• SMTP/POP
• MAPI
• Outlook Web Access
• FTP
• LDAP
• WAP2/MMS Radius
• TCP/IP UDP

• ERP/CRM
• PeopleSoft
• Siebel
• Oracle Applications
• Chordiant
• E.piphany
• Lawson

• Middleware
• CORBA (IIOP)
• J2EE/EJB (Framework)
• .NET Remoting (Framework)
• (D)COM (COM, MTS)
• BEA Tuxedo (ATMI, JOLT)
• Oracle Forms

• Open Interfaces
• DLL
• .NET, Java VB6 Framework
• COM (API)

• Web Services (SOA)
• SOAP (HTTP/XML)
• MS .NET SOAP Stack
• Apache Java SOAP Stack

• Terminal Services (ICA)
• Citrix MetaFrame

17
FeaturesAdvantagesBenefits superior to the
competition
18
Why is SilkPerformer Better?

• More accurate
• More scalable
• Superior ease-of-use
• Superior front-end diagnostics
• Superior lifecycle coverage

19
Why is SilkPerformer More Accurate?

• Features
• Content Verification Under Load
• TrueCache technology
• Agent health control
• Advantages
• Realistic simulation of end-user traffic
• Detection of application failures in addition
  to system failures
• Benefits
• Reduces the risk of project failures through
  accurate test results
• Delivers results that you can TRUST

20
Reliable Error Detection with Content
Verification Under Load
SilkPerformer
System Under Test

• Web recorder scripts content verifications
  automatically
• Verifies activated links and form submissions
  (click path)
• Add more content verifications easily with point
  click
• Define verifications based on what you see in the
  TrueLog
• Verify server response for every virtual user of
  the test
• Application errors often occur only to a subset
  of users

21
Accurate Emulation of Real Users Through
TrueCache Technology
Not emulating a Web browsers cache, a test tool
would simulate too much load
200
200
TrueCache Technology accurately simulates
caching, including conditional requests
150
150
Real user (Realistic load)
Load on the Server (HTTP Hits)
100
100
50
50
Caching every request (ignoring conditional
requests), a test tool would simulate too low a
load
0
0
25
50
75
0
100
Percentage of users who visit Web site more than
once
Only first time users
Only revisiting users
22
Why Is SilkPerformer More Scalable?

• Features
• TrueScale technology
• Advantages
• Simulate twice as many virtual web users on the
  same machine than the closest competitor
• No performance penalty when using IP spoofing
• No performance penalty when using NTLM or
  Kerberos authentication (typically used in
  Intranet applications)
• Benefits
• Reduce resource demand for testing equipment
• Run large scale load tests with reasonable costs

23
Why is SilkPerformer Easier to Use?

• Features
• Built-in guidance
• Integrated Test Environment
• TrueLog script customization
• Advantages
• Always know what step needs to be done next in
  order to successfully complete your test project
• Manage all agents and test assets efficiently
  from a single, central point of control
• Easily grasp what kind of customizations are
  necessary in order to get your test script ready
  for execution
• Benefits
• Reduce project cycle times by improved
  productivity of your testers and flattened
  learning curve of new users

24
Built-In Guidance Simplifies Usage
Guides you through each step of the testing
process
Context sensitive help provides in-depth assitance
25
Single Central Point of Control via an
Integrated Test Environment

• Manage all agents efficiently from a single,
  central location (MMC)
• Installation and configuration
• Automatic detection
• Health monitoring
• Manage all test assets efficiently within a
  project and keep everything in one place
• Scripts workloads
• Agent assignments
• Results
• Project attributes

26
Fast and Easy Script Customizations with TrueLog
Point Click Technology
TrueLog visualizes the results of the test
script
Easy script reuse for other target applications
via project attributes
Simply select user data that needs to be
customized
No manual programmingrequired, but possible
Customize user data with point click
27
SilkPerformer Provides Superior Front-end
Diagnostics

• Features
• TrueLog On Error
• Advantages
• Illustration of user click-path unveils error
  message of the application
• Web page breakdown points out the objects and
  their network timings that have caused the
  performance threshold violation
• Benefits
• Reduce error turnaround time

28
Illustration of User Click-path Unveils Error
Message of the Application
Logging just the error page does not provide any
help
Error message is buried in a successful response
29
Web Page Breakdown Analysis Identifies the Object
Causing a Threshold Violation
Is there a performance problem with the HTML
document or with the embedded objects?
Shows objecttimings for anindividual call
Does caching work?
Is the object too large?
Which individual object took up the most time?
30
SilkPerformer Provides Superior Coverage for the
Software Lifecycle

• Features
• Support for component-level stress testing
• Support for closed-loop testing
• Advantages
• Test middleware components under realistic server
  conditions early in the development cycle
• Quickly get realistic test cases to recreate
  real-life error situations in your QA lab
• Benefits
• Reduce the cost of bugs
• Reduce error turnaround time

31
SilkPerformer Editions

• SilkPerformer
• Enterprise-class load testing
• Provides all features
• SilkPerformer Lite
• Entry-level Web only load testing
• Limited to HTTP(S)/HTML, 100 VUs, and simple
  server monitoring
• SilkPerformer Component Test Edition (CTE)
• Test middleware components under realistic server
  conditions (5 VUs)
• Web services, Java/EJB and .NET middleware
  components

32
Competitive Tools

• Enterprise-class
• Mercury LoadRunner
• Compuware QA Center Performance Edition
• IBM/Rational TeamTest
• Entry-level Web only
• Mercury Astra Load Test
• Empirix eLoad
• RadView WebLoad
• OpenSTA (open source)

33
ROI Framework

• A look at what is under construction.

34
Load Testing and SilkPerformer - Connecting
Features to Customers Business Objectives
Reduce Operational Cost
Improve Productivity
Meet Market Requirements
OBJECTIVES
2. Shorten project cycle time while maintaining/
improving quality
4. Reduce defect repair cost
3. Efficient use of available resources
7. Reduce training cost
5. Lower ongoing maintenance cost
1. Reduce risk of project failures
6. Save license costs for quality tools
STRATEGY
1.1 Accurately emulate load during pre-production
load tests -Flexible, scenario based workload
models -Workload wizard to calculate required
virtual users based on user sessions per hour -IP
address simulation -Agent health
control -TrueCache technology
4.1 Early defect detection Start testing early in
dev cycle with component level stress testing
before a client can be recorded -.Net Explorer,
Java Explorer (SP CTE) -Unit test importer
(JUnit/NUnit integration) (SP CTE) -Database
script API ready for manual scripting from scratch
2.1 Easily create test scripts from scratch or
through re-use -Script recorder -Unit test
importer -Visual Studio .NET Add-in -Visual
out-input correlation wizards for customizing
session dependent data -Built-in guidance
(workflow bar) -TrueLog Script customization
3.2/6.1 Standardize on a single software quality
optimization platform for all your
applications -Support for all major enterprise
applications and technologies
2.4 Communicate test results to management
faster -Management reports that are easy to
share customize (Overview Report)
TACTICS Product Feature Services
2.5/4.2 Identify and repair defects
faster -Artificial workloads such as stress
tests or steep rising edges -Real-time server
monitoring -Integration with Borland Optimizeit
ServerTrace -TrueLog On Error -Automatic client
to server side results correlation (response time
server resource) -Closed-loop testing via
passive monitoring
5.1 Optimal Capacity Planning -Workload model of
a step-wise increase -Client-side performance of
thresholds -Server-side performance diagnostic
tools
3.3 Reuse test units from development -Unit test
importer
1.2 Reliable error detection during
pre-production load tests -Content verification
under load -Ability to ignore irrelevant errors
2.2 Manage complete test tool infrastructure
test assets from single point of
control -Integrated Test Environment
7.1 Flatten learning curve with
ease-of-use -Built-in guidance (workflow bar,
context sensitive help) -TrueLog script
customization
3.4 Reuse results from monitoring production
environment for pre-production load
testing -Closed-loop testing via passive
monitoring
3.1 Optimal use of test environment -Remote
agent technology with load balancing (firewall
ready) -TrueScale Technology
2.3 Easy re-use and adaptation of test
scripts -SilkPerformer BDL scripting
language -SilkPerformer project attributes
1.3 Test and validate service levels before
deploying applications -Client-side performance
thresholds
NOTE Underlined features are competitive
advantages compared to Mercury Availability
METRICS next step
35
Thank You!