Best Practices in Performance Testing

1
Best Practicesin Performance Testing

• Jennifer Turnquist
• Storage Service Line Director
• Lionbridge Technologies

2
Lionbridge Profile
8 of the worlds 10 most valuable companies are
Lionbridge customers BusinessWeek Global 1000,
July 2004

• Public Company (Nasdaq LIOX)
• Nearly 400M in revenues
• Profitable
• Deep expertise across the application life cycle
• Application Development Maintenance
• Testing (Independent VV)
• Content Development, Conversion Enhancement
• Globalization
• Worldwide scale and capability
• Over 4000 employees operating in 25 countries
  (Scale)
• SEI CMM Level 5 certified process model (Quality)

3
Services Designed around our Clients Need
Lionbridge A Trusted Partner Around the World
Global Development Testing Solutions
Global Language Content Solutions
Interps
Application Development
Testing Certification
Maintenance Support
Localization/Translation
Technical Publications
eLearning Courseware
Software Development Lifecycle
Full Content Lifecycle

• Off-shore platforms leverage more than staff in
  China, India, and Eastern Europe
• Global footprint enables local interaction and
  facilitates worldwide release and support
• Trusted, US-based public company protects against
  IP loss

• Localization services spanning more than 80
  languages
• Proprietary web-architected TM and terminology
  solution accelerates production and improves
  consistency
• Authoring and eLearning development services
  integrate seamlessly with localization to address
  global demand

4
VeriTest Setting the Standard in Testing Since
1987

• Worlds largest independent testing company
• Over 400 test architects, engineers, and analysts
  in 11 labs across US, Europe, Asia
• Rapid expansion in VeriTest India
• From PDAs and PCs to 32-way servers
• Data center class storage lab
• Industry leader
• Exclusive provider and architect of
  industry-leading certification programs
• Developer of PC Magazine benchmarks
• Test and publish industry standard ISP benchmarks
• Operate globally-networked onsite to offshore
  model

5
The Lionbridge TeamLocal Connections, Global
Efficiency
4,000 Worldwide Staff Experience and Efficiency
6
Todays Agenda

• Why Test Performance?
• Different Types of Performance Testing
• Performance Testing Roadmap
• Choosing the Right Testing Tools
• Top 10 performance testing pitfalls

7

• "The standard philosophy of 'test to
  destruction'... will probably give you an idea of
  roughly how many users your site can handle at
  once, but it won't always tell you why the site
  fails to function properly. And without knowing
  why, you're not likely to be able to do much
  about it..."
• --Extreme Tech

8
Why Test Performance?

• The internet and IT infrastructure crucial to
  business
• Usersemployees, business partners,
  customersrely on portals, applications, and data
  to do their jobs
• Cost of failure can be devastating
• Performance testing in the enterprise is
  intermittent, cyclical, often prompted by
  upgrades
• Testing is highly specialized

9
The high cost of not conducting performance
testing

• Performance testing overlooked until disaster
  strikes
• Lost and abandoned sales - most visible result of
  poor performance testing but
• Efficiency of mission-critical systems directly
  impacts business productivity
• Preventing problemslost productivity, lost
  business, lost reputation, and even injury or
  deathis a major incentive
• Knowing the vital performance metrics
  ammunition to IT departments when planning and
  justify purchasing decisions
• Provides the ability to demonstrate to investors
  and other critical stakeholders that the
  companys infrastructure is adequate

10
Events that trigger performance testing

• Build vs. buy
• Evolving requirements
• Technology due diligence
• Consolidating servers
• Deploying a SAN
• Deploying or upgrading enterprise application
• Migrating to a new platform

• Addition of features
• Response to public critique
• Enhancements due to buying trends
• Acquiring or merging a business
• Launching new product
• Enhancing web application
• Promoting an offering
• Doing any of the above globally

11

• Performance testing is not a one time event.

12
Conduct the right test to get the right results

• Load Testing
• Determines the response time and throughput
  during typical user load
• Stress Testing
• Determines the peak user load
• Volume Testing
• Determines the problems that occur during
  long-term user activity
• Component Testing
• Determines the performance and behavior of a
  specific component
• Benchmark Testing
• Measures the performance of a system or component
  relative to a standard
• Transaction Cost Analysis
• Determines the system resources consumed by a
  single transaction

13
Performance Testing Roadmap
Script
Test
Plan
Define
Communicate

• Design scripts
• Create scripts
• Validate scripts
• Build script library

• Verify basic functionality
• Generate use cases
• Capture user activity logging information
• Analyze user activity profile
• Model user activity
• Choose the tool(s)
• Identify re-usable script components
• Assign resources neededfor scripting, testing
• Create test environment

• Execute tests
• Collect data
• Analyze test results
• Run possible iterations
• Troubleshoot bottlenecks
• Tune system
• Retest
• Log non-performance failures

• Identify stakeholders
• Agree on goals of testing
• Determine budget
• Determine schedule constraints
• Agree to promotion strategy
• Outline resources available
• Determine staffing plan
• Engage test lab (if needed)

• Outline context
• Draft results
• Provide feedback to stakeholders
• Deliver action items
• Finalize report(s)
• Promote results

14
An overview of the performance testing process

• After initiating the test, the load generator
  systems to begin accessing the system under test
  using the designed usage patterns.
• Depending on whether the test is a global, local,
  or isolated configuration, the load generators
  may be located worldwide or completely contained
  within a test lab.
• The one critical configuration requirement for
  the load-generating systems is that they have
  adequate network bandwidth throughput capability
  to access the system under test in a realistic
  manner without bandwidth constraints.
• If bandwidth constraints become a problem, adding
  additional load generators to the pool of load
  generators will typically fix this problem.
• If the test is global or local, the Internet will
  be an important factor in the configuration. For
  an isolated configuration, the Internet is not a
  factor.

15
An overview of the performance testing process

• Once a performance test is initiated, it can run
  for several minutes to several days, depending on
  the test goal.
• During the test time, the test tool monitors and
  collects performance data from all of the
  components within the system under test, such as
  the Web server, application server, or database
  server.
• All of the monitor data along with the
  performance test data collected at the generating
  client end to determine the overall performance
  as well as the potential system bottlenecks.
• In a typical performance test cycle, the
  performance bottlenecks are located, fixed, and
  iteratively retested to ensure that they are
  fixed as designed.

16
High Level Picture of the Process
Overcome resource limitations Replace testers
with Virtual Users Run many Virtual Users on
few machines Controller manages Virtual Users
Run repeatable tests with scripted actions Get
meaningful results with analysis tools
Load Generation
System under Test
17
You dont have to go it alone
Script
Test
Plan
Define
Communicate
Build and train internal resources Hire
contractors Utilize service offerings from test
tool vendors Rely on application provider Engage
with a consulting firm or SI Partner with an
independent testing company
18
Important considerations for choosing the
resources

• Deadlines
• Testing skills and experience
• Technology and/or application expertise
• Frequency and scale of testing requirements
• Infrastructure requirements
• Risk assessment
• Market factors

19
The vast number of performance testing tools can
be overwhelming
20
Important considerations for choosing the right
tool

• Do you already own the license?
• Do you have the internal resources to script and
  execute?
• Will it meet the test objectives?
• Is it compatible with your technology objectives?
• Does it fit within your budget constraints?
• Do you have the training and expertise to analyze
  the results?
• Does it match the frequency of your testing needs?

21
Leading performance tools
Benchmark Tool Pros Cons
Mercury Interactive LoadRunner Compatible with numerous protocols Excellent data analysis tools WAN emulation Web transaction breakdown monitor Expensive license Requires a unique license for each protocol type
Segue SilkPerformer Offer Lite version for reduced price Excellent data analysis tools Root cause analysis tools included Windows only Expensive license
RadView WebLoad Inexpensive license Good data analysis tools Covers few protocols primarily Web-based automation
Spirent Avalanche/Reflector Uses SST TracePlus to provide record and playback feature No additional hardware required Expensive hardware to purchase
22
Popular Benchmark Tools
Benchmark Tool Workload Simulated
VeriTest Tools VeriTest NetBench VeriTest WebBench CIFS file / network traffic HTTP / Web traffic
Microsoft Tools WCAT WMLS Microsoft LOADSIM DBHammer HTTP(S) traffic streaming media Exchange email traffic SQL database traffic
SPEC benchmarks SPECjbb SPECsfs SPEC Java Business Benchmark Exercises the CPU NFS File / Network Traffic
Other Industry Standard Tools IOMeter IOZone TSScaling block level data transfer, OLTP (database) traffic block level data transfer terminal services traffic
23
Manual testing may be your best tool

• Frequency of testing requirements
• Rate of change
• Limitations of available tools

24
Top 10 performance testing pitfalls
25
Top 10

• Not Testing.
• Lack of clearly defined test objectives and poor
  planning.
• Relying exclusively on beta customers to find
  performance issues.
• Using the wrong tool for the job.
• Introducing too many variables simultaneously
  into a test.
• Failing to test how your product or system is
  actually used.
• Conducting load testing in a vacuum.
• Treating performance testing like a one-time
  event.
• Assume that the scripting effort will be short
  and simple.
• Finding functional bugs during performance runs.

26
Conclusion

• Companies rely on systems to conduct business
  efficiently and effectively
• Performance testing ensures that your users are
  getting reliable and timely access to the
  resources they need
• Performance testing mitigates the risk of lost
  time and money due to poor performance
• A fully integrated performance testing program is
  the preventative medicine that keeps your system
  from becoming an inaccessible and costly
  resource.
• Though it may seem counterintuitive at first to
  slow your deployment for performance test
  planning and execution, the payoff in time,
  money, and quality will be big and will come
  soon.

27
Thank You