Test Automation Lets talk business

1
Test Automation Lets talk business

• Igor Gershovich
• Connected Testing, Inc.
• www.connectedtesting.com
• igershovich_at_connectedtesting.com

2
Overview

• Test Automation Myths
• Return on Investment (ROI)
• Frameworks

3
Test Automation Myths
4
Myth 1 - Test Automation is simple, that every
tester can do it

• This myth is promoted by the tool sales people.
  They are trying to promote the following test
  automation process
• Record the script
• Enhance the script by adding functions and data
  driving
• Run the scripts
• Report results
• Under the influence of this myth the QA manager
  can proudly report All our testers are
  developing test automation.

5
Reality - Test automation is a software
development task

• Automation should be designed, developed and
  tested
• You need to have some kind of a programming
  background to implement test automation. Test
  Automation is not as complex as C/C/Java
  development.
• Test automation standards should be developed
• Automated test components are assets that should
  be treated like application source code

6
Myth 2 Commercial test tools are expensive

• Under the influence of this myth some companies,
  especially the small ones
• Try to develop their own test automation tools
• Use scripting languages like Perl and Ruby
• Use shareware test tools
• Do not consider test automation at all

7
Reality Commercial tools are cheap

• Per seat license for the most expensive
  automation tool is 8K
• This tool will be used for 5 years.
• Maintenance/Support fees are 20 of tool cost or
  1,800 per year
• The cost of this tool is 8K/51,800 3,100
  per year
• The automation developer cost with overhead is
  100K per year
• The cost of this tool is just 3 of the person
  who uses it, but productivity gain can be very
  significant

8
Commercial Tool Benefits

• Customer support. Many of the open source tools
  come and go with little to no support
• Most commercial tools are constantly being
  updated as technologies change
• Most commercial tools usually have more
  functionality (QTP can test various GUI
  applications Web, .Net, Java, VB, C/C,
  PowerBuilder, etc. vs. WATIR Web only)
• Commercial tools usually have a large community
  of users, which translates into better
  availability of qualified resources
• Commercial tools require less advanced
  programming
• More test automation frameworks are available for
  commercial tools
• Commercial tools are integrated with Test
  Management tools which makes reporting and
  execution much simpler.

9
Over 300 Test Tools are available
(http//www.softwareqatest.com)

• Load/Performance tools 54
• Web Functional/Regression 60
• Java Test tools - 48
• Other Web tools 76
• Which tool is right for you?

10
Absolutely Unscientific research Employer
demand by tool expertise(May, 2008)
11
Dice search results across US (30 days)
12
(No Transcript)
13
Excerpt from IDC Report Worldwide Automated
Software Quality 2007-2011 Forecast and 2006
Vendor Shares
14
(No Transcript)
15
Think of yourself like a business

• Facts
• Average IT job tenure is about 2 years
• Competitive intensity is rising
• What you should do
• Invest your time/money in learning mainstream
  automation
• Create a niche for yourself - add Configuration
  Management, Performance Testing or Project
  Management to your skillset
• Market yourself

16
Return On Investment (ROI)
17
Classic ROI Calculation
ROI BENEFIT/COST Automation Cost Price Of HW
Price of SW Development Cost Maintenance
Cost Execution Cost Manual Testing Cost
Development Cost Maintenance Cost Execution
Cost ROI (Manual Testing Cost - Automation
Cost)/Automation Cost Looks right, Doesnt it?
18
Problems with Classic ROI Calculation

• You cant compare Automated Testing and Manual
  Testing. They are not the same and they provide
  different information about the AUT.
• You cant compare cost of multiple execution of
  automated tests vs. manual tests. You would never
  dream of executing that many test cases manually.

19
Automation Real ROI

• ROI value IS NOT the value of Automation vs. Cost
  of executing these tests manually
• Automation ROI value IS the benefit of this type
  of testing, and it can be
• Reducing Time to Market
• Increased Test Efficiency (Productivity)
• Increased Test Effectiveness

20
Reduced Time to Market

• Can get a greater market share
• Makes people available to work on other projects
• Higher margins, if no competitive products are
  currently available

21
Productivity and Effectiveness

• More testing gets done faster, increasing the
  odds of finding defects
• Defects found early have better chances of being
  fixed
• Manual Testers can concentrate on clever ways to
  finding defects, instead of typing test inputs
  and verify output.
• About 7 of bug fixes create new bugs, sometimes
  in already tested parts of the system. With
  automation you can rerun tests for those modules.
  This almost never happens when testing is done
  manually.

22
ROI summary

• Each project requires different types of
  automation - there is often no easy formula
  available to calculate ROI
• Performing ROI calculation can help to determine
  upfront what type of automation, what level of
  skills, what tools will be required.

23
Automation Frameworks
24
Test Automation Framework

• A Test Automation Framework is a set of
  assumptions, concepts and tools that provide
  support for Automated Software Testing.
• A correctly implemented Test Automation Framework
  can further improve ROI by reducing the
  development and maintenance costs.

25
Types of Test Automation Frameworks

• Modular
• Data-Driven
• KeywordDriven
• Model-Based

26
Modular framework

• The Modular framework is the natural progression
  derived from Record-and-Playback
• The modular framework seeks to minimize
  repetition of code by grouping similar actions
  into modules (e.g. login)
• Test Data is in a script or in an internal table
• This is what tool sales people are promoting.

27
Data-Driven/Keyword Driven

• These frameworks are similar in that the data is
  separated from the test script
• The script is just a "driver" or delivery
  mechanism for the data.
• The difference
• In keyword-driven testing, the navigation data
  and test data are contained in the data source
• In data-driven testing, only test data is
  contained in the data source.

28
Model-Based testing

• Model-based testing is software testing in which
  test cases are derived in whole or in part from a
  model that describes some (usually functional)
  aspects of the system.
• Model-based testing for complex software systems
  is still an evolving field.

29

• Well be concentrating on Keyword-Driven testing
  since it is the most beneficial framework for
  large-scale test automation.

30
Advantages of Keyword-Driven Test automation

• This Framework addresses the most common problem
  with test automation
• Automation Engineers do not have domain
  knowledge and the End Users (Subject Matter
  Experts/Test Engineers) usually do not have
  automation expertise.
• When properly implemented and maintained, it
  presents a superior ROI because each business
  event is designed, automated and maintained as a
  discrete entity.
• Keywords can then be used to design test cases,
  but the design and automation overhead for the
  keyword has already been paid.

31
Advantages of Keyword-Driven Test automation

• Reduced the cost and time spent maintaining and
  updating tests
• The modular structure of keyword-driven testing
  means that new tests can easily be created from
  pre-existing modules
• The test team is capable of entirely automating
  tests, even without programming knowledge
• Can be easily modified to use with different test
  tool
• Reusability across different projects

32
Classic Keyword-Driven Example
33
Recent Success with Keyword-Driven approach

• Estimate using Modular Framework 1 year
• Estimate with Keyword-Driven Framework 6 months
• Project was completed in 4 months by 1 person, 2
  month ahead of schedule.
• This was the first project for the automation
  developer utilizing the Keyword-Driven Framework.
• 170 components were developed (1 component per
  Web page).
• 109 Regression Test cases were automated.

34
Example of Test Data for Keyword-Driven test
automation developed by Connected Testing Inc.
for TestComplete and QTP
35
Resources

• Everything you want to know about Test
  Automation by Brian Le Suer
• Managers Guide to GUI Test Automation by Yury
  Makedonov
• The ROI of Test Automation by Michael Kelly
• What is my ROI? by mVerify Corporation
• SQA Forums
• An Overview of Test Automation Frameworks by
  Nathaniel Ritmeyer
• The benefits of keyword-based software test
  automation by David W. Johnson
• Wikipedia

36
Igor Gershovich

• Company Connected Testing, Inc.
• E-Mail igershovich_at_connectedtesting.com
• Phone [phone]
• Website www.connectedtesting.com