Tips for Success when Selecting Testing Tools

1
Tips for Success whenSelecting Testing Tools

• Marie Ackley
• ackley _at_ promail.com
• SASQAG panel discussion
• October 18, 2001

2
Different Categories of Testing Tools

• Requirements Management
• Configuration Management
• Test Planning and Test Case Management
• Test Evaluation - Coverage Analysis
• Automated Testing Systems - Front End testing
• Load Simulators and Performance Measurement Tools
• Defect Tracking
• others Cyclomatic Complexity Bounds Checkers
  Comparators - Files, Screens, Reports Data
  Generators - Combinatoric, Cause / Effect Node
  Graphing

3
Leading Cause of SHELF WARE in Testing Tools

• is Putting the CART before the HORSE
• Don't invest in Tools without having good QA
  Methods and Practices.
• A good tool will compliment and augment good
  processes but not even the greatest tool can
  overcome process chaos.

4
Requirements Management

• Supports multi levels of linking and tracing
  Requirements to Specifications to Test Cases,
  including hierarchies within those groups
• Provides traceability - both forward and backward
• Provides coverage analysis
• Provides multi relationships one-to-many,
  many-to-one and coverage is tracked for partials
  when relationship is one-to-many
• Provides Change Management of requirements along
  with audit history and maybe security privileges
  for modifications
• Import and export features so stuff came go and
  come between the tool and word processors or
  spreadsheets

5
Requirements Management

• Links to tool comparisons
• the "Volere" Requirements Method - by Atlantic
  Systems Guild
• http//www.systemsguild.com/GuildSite/Robs/Templat
  e.html
• Automating Requirements Management - by Karl E.
  Wiegers
• Detailed feature comparisons of many requirements
  management tools
• and guidance on selecting a requirements
  management tool
• http//www.processimpact.com/articles/rm_tools.htm
  l

6
Configuration Management

• "To change and to change for the better are two
  different things"
• - German Proverb
• Provides Identifications (indexing of artifacts)
• Controls not only software code but
  documentation, hardware, firmware, and system
  environment configuration
• Provides audit trail and history tracking
• Provides customizable reports
• High ease of use for View Only functions
• Sufficient security levels for the Promote
  functions
• Accounting and reporting for different Status
  Levels" - WIP, Waiting Room, Release/Production

7
Configuration Management

• Links to tool comparisons
• Configuration Management Vendor Links - by UCM
  Central
• http//www.snuffybear.com/ucmcentral_new_vendorlin
  ks.htm
• config-mgmt Newsgroup
• http//www.faqs.org/faqs/by-newsgroup/comp/comp
  .software.config-mgmt.html

8
Test Planning and Test Case Management

• Import test cases from other file formats such as
  word processing or spreadsheet
• Customizable reporting
• Tracks history and status of test cases
• Allows file attachments
• Integration - can tool link to other tools such
  as your Requirements tool and your Automated
  Testing tool
• Provide a Test Execution Results repository
• Maintains Master Set of test cases and then can
  build subsets for project specific needs

9
Test Planning and Test Case Management

• Links for tool comparisons
• Testing and Test Management Tools
• http//www.methods-tools.com/tools/frames_testing.
  html
• Information About Testing Tools -
• http//www.cs.queensu.ca/home/shepard/testing.dir/
  under.construction/tool_list.html

10
Test Evaluation - Coverage Analysis

• Identifies what "unique" data combinations are
  needed
• Uses an appropriate measurement technique
• Decision Condition/decision Path Predicate
• Types of coverage included in the reporting
• Branch Loop Multi Operator Operand
• Integration - is it compatible with your other
  tools such as Automated Test Execution tools
• Implementation - is the tool "invasive" (must be
  compiled along with the source code)

11
Test Evaluation - Coverage Analysis

• Links to tool comparisons
• Experience With the Cost of Different Coverage
  Goals for Testing - by Brian Marick
• http//www.testing.com/writings/experience.pdf
• Coverage Analysis - by Steve Cornett
• http//www.bullseye.com/webCoverage.html

12
Automated Testing Systems - Front End testing

• Maintenance - object recognition when the
  application changes
• Ease Of Use - complexity matches the programming
  talent of QA persons
• Tool language and syntax matches what company
  uses Visual Basic or C
• Extensible Language - can use external
  subroutines or DLLs
• Integration with other tools - Code Coverage,
  Test Case Management, Defect Tracking
• Compatibilty - platforms (Client/Server, Web,
  Mainframe) Operating Systems(Windows, Mac, Unix,
  OS/2, Mainframe) and GUI Development
  (PowerBuilder, C, Visual Basic, Galaxy

13
Automated Testing Systems - Front End testing

• Supports Data Driven inputs from flat files,
  spreadsheet files, and database connections
• For Evaluation/Proof of Concept - include
  automating a Level One test
• Links to tool comparisons
• Making the Right Choice - by Elisabeth
  Hendrickson
• http//www.qualitytree.com/feature/mtrc.pdf
• Short Sharp Advice about how to choose a testing
  tool - Mark Fewster and Dorothy Graham
• http//www.grove.co.uk/Tool_Information/Choosing_T
  ools.html
• Tool Comparison between Rational, Mercury, Segue,
  and Compuware - by Ray Robinson
• http//www.qacity.com/testtype/AUTOMATION_TOOLS.do
  c

14
Automated Testing Systems

• Links to articles
• Lessons in Test Automation - by Elfriede Dustin
• http//www.sqa-test.com/articles.html
• The Truth About Automated Test Tools - by Linda
  Hayes
• http//www.worksoft.com/resources/4-97.htm
• http//www.worksoft.com/resources/5-97.htm
• Test Automation - by Bret Pettichord
• http//www.io.com/wazmo/qa/test_automation
• Testers' Network - VeriTest website
• http//www.veritest.com/testers'network
• Test Automation Frameworks - by Carl J. Nagle
• http//members.aol.com/sascanagl/FRAMESDataDrivenT
  estAutomationFrameworks.htm

15
Load Simulators and Performance Measurement

• Reporting includes graphical charts
• Debugging features for script creation
• Reporting includes graphical charts and logs of
  performance for server speed and responsiveness
• Quantity of simulated users
• Variations in session-specific items such as
  cookies, usernames, passwords, proxies and
  ASP-session IDs
• Support for your middleware - SQLnet, DBLib or
  CBLib, SQL Server, ODBC, Telnet
• Challenges include SSL sessions, cookies, forms,
  proxies, password authentication, dynamic HTML,
  full HTTP, SSL, XML, Javascript, ActiveX, WAP,
  Java
• Platforms and protocols

16
Load Simulators and Performance Measurement

• Link to tool comparisons
• Load and Performance Test Tools - by Rick Hower
• http//www.softwareqatest.com/qatweb1.html
• Software Research - TestWorks
• http//www.methods-tools.com/tools/testing.html

17
Defect Tracking

• Determine how many current users you have and
  consider and scaling for future users
• Web based or client-server
• Features Customizable Fields Customizable
  Reports File Attachments Audit Trail Email
  linking Security Levels
• Provides customizable rules for the workflows,
  security privileges and notifications
• Integration - does it link to your other tools

18
Defect Tracking

• Links to tool comparisons
• Tracking Down a Defect Management Tool - by Hung
  Quoc Nguyen
• http//www.qacity.com/testgen/DefectTrackingNguyen
  STQE701.pdf
• The Problem with Problem Tracking - by Linda
  Hayes
• http//www.worksoft.com/resources/3-99.htm

19
Websites for Tool Information
Vendor contact information http//www.testingfaqs.
org/tools.htm http//www.ondaweb.com/sti/stivend.h
tmL http//www.dbmsmag.com/pctest.htmlP0890 Bul
letin board forums and articles http//www.qaforum
s.com http//community.sqatester.com http//www.qu
alitytree.com/links/links.htm http//www.stqemagaz
ine.com