AUTOMATED TEST GENERATION

1

• AUTOMATED TEST GENERATION
• Muhammed I. KALKAN

2
Automatic Test Generation (ATG)

• Index
• What is ATG?
• ATG Types
• Why do we need ATG?
• What are the main benefits of ATG
• ATG Methods
• Testing Methodologies
• Automatic Test Pattern Generation
• Relevant Examples

3
What is ATG?

• Testing an engineered system is the most
  important part of the product revealed.
• Engineers are working on and putting alot of
  effort into this issue.
• Since testing process is required for all
  obtained engineering products, engineers decided
  to develop an automated test mechanism instead of
  generating test cases manually.

4
What is ATG?

• Automated Test Generation includes test cases ,
  scenarios and the evalution of the obtained
  result such as passed and failed.
• Given inputs and obtained outputs are evaluated
  automatically.
• The Main Idea
• TEST COMPLETELY AUTOMATICALLY

5
ATG or ATPG?

• ATG is an engineering subject which can be
  applied to both Computer Sciences and other
  engineering specialities like Electronics or
  hardware circuits.
• ATG is a term which is generally used for
  Computer Sciences especially software testing.
• On the other hand for hardware testing Automatic
  Test Generation is usually referred as Automatic
  Test Pattern Generation.

6
ATG or ATPG?

• ATPG is defined as follows
• ATPG is an electronic pattern automation
  method/technology used to find an input (or test)
  sequence
• When applied to a digital circuit, it enables
  testers to distinguish between the correct
  circuit behavior and the faulty circuit behavior
  caused by defects.

7
ATG.But WHY??

• As years have passed,computer hardwares and its
  capabilities grow bigger and bigger.
• Result 1 Software types and their capabilities
  are varied.
• Result 2 New software demand on market for jobs.
• Result 3 Bigger income ,more firms to compete
  for market ration.
• Result 4 Requirement for strong,bug free and
  developed in small amount of time softwares

8
ATG.But WHY??

• Result 5 Software Engineering and Testing
  studies.

9
ATG.But WHY??

• Software testing is important
• Software errors cost the U.S. economy about 59.5
  billion each year
• Improving testing infrastructure could save 1/3
  cost
• Software testing is costly
• Account for even half the total cost of software
  development

10
Benefits

• Reduces Testing Time
• Reduces Testing Cost
• Reduces human errors while testing

11
ATG Methods

• Data-Driven ATG
• xUnit Frameworks for ATG
• Examples JUnit and NUnit
• Model-Based ATG
• Keyword-Driven ATG

12
xUnit

• Various code-driven testing frameworks have come
  to be known collectively as xUnit.
• Based on a design by Kent Beck, originally
  implemented for Smalltalk as SUnit, but are now
  available for many programming languages and
  development platforms.

13
xUnit

• The overall design of xUnit frameworks depends on
  several components.
• Test Fixtures
• A test fixture is the set of preconditions or
  state needed for a test to run. Also known as a
  test context.
• Test Suites
• A test suite is a set of tests that all share the
  same fixture.
• Test Execution
• The execution of an individual unit test proceeds
  as follows

14
xUnit

• The setup() and teardown() methods serve to
  initialize and clean up test fixtures.
• Assertions
• An assertion is a function or macro that verifies
  the behavior of the unit under test. Failure of
  an assertion typically throws an exception,
  aborting the execution of the current test.

15
Data-Driven ATG

• Data-driven testing is a methodology used in Test
  automation
• Test scripts are executed and verified based on
  the data values stored in one or more central
  data sources or databases.
• Databases can range from datapools, ODBC sources,
  csv files, Excel files, DAO objects, ADO objects,
  etc.
• Data-driven testing is the establishment of
  several interacting test scripts together with
  their related data results in a framework used
  for the methodology.

16
Data-Driven ATG

• Variables are used for both input values and
  output verification values navigation through
  the program, reading of the data sources, and
  logging of test status and information are all
  coded in the test script. Thus, the logic
  executed in the script is also dependant on the
  data values.
• An example of xUnit Framework Automatic Testing
  by using Data-Driven Test Methodology is as
  follows

17
Example

• Put the data that varies from test to test into
  the Data-Driven Test file that the interpreter
  reads to execute the tests
• For each test it does the same sequence of
  actions to implement the Four-Phase Test

18
Example (FPT)

• Set-up the test fixture
• Try the test case
• Evaluate the result
• Release the fixture

19
Keyword-Driven ATG

• It is also known as table-driven testing,
  action-word testing
• It can be used for manuel testing as well as
  automated testing
• The advantages for automated tests are the
  reusability and therefore ease of maintenance of
  tests which have been created at a high level of
  abstraction.
• The Keyword-Driven ATG methodology divides test
  creation into two stages.
• Planning Phase
• Implementing Phase

20
Keyword-Driven ATG

• Planning Phase
• The application (or the requirements for the
  application) is analyzed to determine which
  operations and objects will need to be tested.

21
Example

• A web-questionnaire application will require a
  large amount of text entries. By identifying
  which operations should be encapsulated into
  keywords, the efficiency and maintainability of
  the tests are maximized.

22
Keyword-Driven ATG

• Implementation Phase
• The implementation stage differs depending on the
  tool or framework used.
• Often, automation engineers implement a
  framework that provides keywords like check and
  enter.
• Testers or test designers (who dont have to know
  how to program) write test cases based on the
  keywords defined in the planning stage, which
  have been implemented by the engineers.
• The test is executed using a driver which reads
  the keywords and executes the corresponding code.

23
Keyword-Driven ATG

• Other methodologies use an all-in-one
  implementation stage.
• Instead of separating the tasks of test design
  and test engineering, the test design is the test
  automation.
• Keywords, such as enter or check are created
  using tools in which the necessary code has
  already been written.
• This removes the necessity for extra engineers in
  the test process, because the implementation for
  the keywords is already a part of the tool.

24
Keyword-Driven ATG

• Benefits
• This methodology requires more planning and a
  longer initial time-investment than going
  directly to the test creation stage and recording
  a test.
• It does make the test creation and test
  maintenance stages more efficient and keeps the
  structure of individual tests more readable and
  easier to modify.
• The more abstract keywords are, the more reusable
  they are, and therefore the easier a test is to
  maintain.
• As well as reducing the cost and time spent
  maintaining and updating tests, the modular
  structure of Keyword-Driven ATG means that new
  tests can easily be created from pre-existing
  modules.

25
Keyword-Driven ATG

• Another advantage is the reduction in technical
  know-how required for the test automation
  process.
• In the first approach, technical know-how is only
  required by the engineers that implement the
  keywords.
• In the second approach, even this is not
  required, which means that the test team is
  capable of entirely automating tests, even
  without programming knowledge.

26
Model Based ATG

• Model-based testing is software testing in which
  test cases are derived in whole or in part from a
  model that describes some (usually functional)
  aspects of the system under test process
• The model is usually an abstract, partial
  presentation of the system under test's desired
  behavior.
• The test cases derived from this model are
  functional tests on the same level of abstraction
  as the model
• Can not run abstracted test cases.Need executable
  ones.

27
Model Based ATG

• Diagram description
• Model-Based Test Systems

28
Model Based ATG

• Many ways to produce test cases.
• No best method for producing tests.
• Use your test interest to define your test cases.
• Design decisions is often known as "test
  requirements", "test purpose" or even "use case"

29
Model Based ATG

• Use model and test requirements to derive
  abstract test cases
• Implementation Extra Information to abstract test
  suit to produce executable test cases

30
Model Based ATG

• Execute the test
• Evaluate the reports

31
Model Based ATG

• There are a few ways to deploy the Model-Based
  Testing
• Online Testing A model-based testing tool
  connects directly to a system under test and
  tests it dynamically.
• Offline Generation of Executable Tests A
  model-based testing tool generates test cases as
  a computer-readable asset that can be later
  deployed automatically.
• A collection of Python classes that embodies the
  generated testing logic

32
Model Based ATG

• Offline Generation of Manually Deployable Tests
  A model-based testing tool generates test cases
  as a human-readable asset that can be later
  deployed manually.
• A PDF document in English that describes the
  generated test steps
• Another Step
• Test Generation

33
Model Based ATG

• Model-Based Testing has a good potential in
  aspect of automation.
• Test case generation is one of the issues that
  ATG is subject to
• To find test cases, Model-Based structure is
  interpreted as a kind of Finite State Machine to
  see different states that the system might be in
• There are a few technics for generating test
  cases.Most knowns are as follows

34
Model Based ATG

• Test case generation by theorem proving
• System model is defined as predicates,kinds of
  logical expressions
• These expression are processed and proved
  afterwards
• Test case generation by constraint logic
  programming
• Selecting test cases satisfying specific
  constraints by solving a set of constraints over
  a set of variables

35
Model Based ATG

• Test case generation by model checking
• We provide a model of the system under test and a
  property we want to test to the model checker
• Test case generation by symbolic execution
• Searching for execution traces in an abstract
  model. In principle the program execution is
  simulated using symbols for variables rather than
  actual values
• Test case generation by using an event-flow model
• With a graphical user-interface (GUI) front-end
  is called the event-flow model that represents
  events and event interactions

36
Example

• TestMaster automates the generation of tests for
  call processing features developed for the
  5ESS-2000 Switch
• This test uses Model-Based Test Generation
• Results obtained here as follows.
• A test generation productivityimprovement of just
  over 90

37
Automatic Test Pattern Generation

• ATPG is an electronic design automation
  method/technology used to find an input (or test)
  sequence that, when applied to a digital circuit,
  enables testers to distinguish between the
  correct circuit behavior and the faulty circuit
  behavior caused by defects
• The generated patterns are used to test
  semiconductor devices after manufacture,
• In some cases to assist with determining the
  cause of failure

38
Automatic Test Pattern Generation

• The effectiveness of ATPG is measured by the
  amount of modeled defects, or fault models, that
  are detected and the number of generated
  patterns.
• We can estimate test quality by effectiveness of
  ATGP

39
Automatic Test Pattern Generation

• A defect is an error introduced into a device
  during the manufacturing process.
• A fault model is a mathematical description of
  how a defect alters design behavior.
• A fault is said to be detected by a test pattern
  if, when applying the pattern to the design, any
  logic value observed at one or more of the
  circuit's primary outputs differs between the
  original design and the design with the fault.
• There are two steps that ATPG should take to
  detect fault.

40
Automatic Test Pattern Generation

• Fault activation Establishes a signal value at
  the fault model site that is opposite of the
  value produced by the fault model
• Fault propagation Moves the resulting signal
  value, or fault effect, forward by sensitizing a
  path from the fault site to a primary output.

41
Sequential ATPG

• Searches for a sequence of vectors to detect a
  particular fault through the space of all
  possible vector sequences
• Vector Sequence A sequence of values for circuit
  input
• In Sequential Circuits due to the presence of
  memory elements, the controllability and
  observability of the internal signals (in
  general)are much more difficult than those in a
  combinational circuit
• Complexity of sequential ATPG much higher than
  that of combinational ATPG.

42
Algorithmic Methods

• Testing very-large-scale integrated circuits with
  a high fault coverage is a difficult task because
  of complexity.
• Many different ATPG methods have been developed
  to address combinatorial and sequential circuits.
• Some examples
• Pseudorandom test generation is the simplest
  method of creating tests. It uses a pseudorandom
  number generator to generate test vectors

43
Algorithmic Methods

• The D Algorithm was the first practical test
  generation algorithm in terms of memory
  requirements. The D Algorithm introduced D
  Notation which continues to be used in most ATPG
  algorithms.
• Fan-Out Oriented Algorithm It limits the ATPG
  search space to reduce computation time and
  accelerates backtracing.

44
Automatic Test Pattern Generation

• ATPG can fail to find a test for a particular
  fault in at least two cases.
• The fault may be intrinsically undetectable
• it is possible that a pattern(s) exist, but the
  algorithm cannot find it

45
References

• Automated Test Generation, (From a Behavioral
  Model), James M. Clarke, Lucent Technologies
• Automatic Test Pattern Generation ,
  http//en.wikipedia.org/wiki/Automatic_test_patter
  n_generation
• Data Driven Tests, Gerrard Meszaros,
  http//xunitpatterns.com/Data-Driven20Test.html,
  2007
• Data-Driven Testing, http//en.wikipedia.org/wiki/
  Data-driven_testing

46
References

• Test Automation, http//en.wikipedia.org/wiki/Test
  _automation
• Test Automation Framework, http//en.wikipedia.org
  /wiki/Test_automation_framework
• A Survay on Automatic Test Case Generation,
  M.Prasanna, S.N. Sivanandam, R.Venkatesan,
  R.Sundarrajan, Department of Computer Science and
  Engineering, PSG College Of Technology,
  http//www.acadjournal.com/2005/v15/part6/p4/
• Keyword-Driven Testing , http//en.wikipedia.org/w
  iki/Keyword-driven_testing

47
References

• Model Based Testing, http//en.wikipedia.org/wik
  i/Model-based_testing
• Improving Automation in Developer Testing
  Achievements and Challenges, Tao Xie, Department
  of Computer Science, North Carolina State
  University
• xUnit , http//en.wikipedia.org/wiki/XUnit

48
The End

• Thank You
• Q A