Fault Detection Techniques

1
Fault Detection Techniques Testing other
approaches

• Chapter 8

2
Why testing?

• Lets look at some well-known software defects
• ??????????????????,??????????????? (no stress
  testing)
• ???????????
• 2000-2005- ?????????-5?????????????????15?????????
  ?
• 2003 -????????????????5000?????,3???
• 2000 ????????,4??? (??????)
• 1997 ????,225??? (????????)
• 1995 ???????????159???(??????)

3
Statistics

• 2002- ?????????-????????595??? (0.6 GDP)
• ??????
• ?????????????
• ?????????????

4
Quality Control and Quality Assurance

• Software is a product.
• Good quality product requires ????quality control
  (QC) and quality assurance (QA) ????

5
Manufacturing in other discipline

• ?????????
• ????(quality)?????????
• ????????????????????????100????100???????
• QC (Quality Control) ????
• ??????????????????????
• ????????????????
• ???????????(process improvement)?????,????????????
  ?

6
How to improve quality?

• ????????????????,?????????????????????????????????
  ???????????????????????????,???????
• Ex. ????????????????
• Question
• ?????????????????,?????????????

7
Assembly line in manufacturing
8
???

• ???????????????????????
• ???????????
• ?????????????????????????????????????
• ??????????.

9
Again, lets review the engineering process of
other engineering fields.

• Idea (??)
• Marketing analysis (Requirement Analysis) (????)
• Analysis and Design (?????)
• Manufacturing (QC) (??)
• Testing (QA)(??)
• Release product (????)

10
Software engineering

1. Idea (??)
2. Requirement analysis and specification (100
   design) (???????)
3. Design and analysis (?????)(100 design, QC
   here?)
4. Implementation (????)(95 design?, QC here?)
5. Manufacturing (??)(compilation no cost
   manufacturing)
6. Testing(??)

11
Question?

• Is coding a design process or a manufacturing
  process
• Is Code a design or a product?
• Again, the analog breaks
• More questions
• If coding is not a design process, we should be
  able to hire many ????? to control our quality
• If coding is a design process, bad news, errors
  are inevitable because programmers are always
  doing complicated design jobs, not simple and
  easy job.

12
Facts

• Now, you should know why software has so many
  bugs or notorious for being flawed

13
The QC story
?????
software
??????
QC ????
14
A Joke

• ???????????????,
• ???????????,
• ?????????????
• ??????,???????????,?????????,?????????????,???????
  ????????
• ?????????,????K??????,???????????,?????????????,??
  ?????????????,???????
• ?????,???????????????,??????????????,????????????
  ?????????????????

• ?????????,
• ????????????????????
• ???????????????,
• ??
• ????????????,???????????????????
• ???????????,??????????????,?????????,?????????????
  ??????????,???????????????

15

• ???????,
• ?????????,
• ??????????????
• ?????????
• (??????????????
• 21????????????????
• ????????)?
• ????????,
• ????????????????
• ?????????,
• ???????????????????
• ??????,
• ?????????????,
• (????????????),

??????????????, ???????????????? ???????????! ??
???????, ????????????????, ???????????-- ?????????
????, ???????, ??????????????......? ???????, ???
????, ???????, ????????, ??????????????????, ?????
?????????????......?
16
Software Quality

• ???????,??????????????,,??????,??????????
• ???? programmers
• This is why testing is so important in matured
  software industry.

17
??????,?????????
18
How do you test a DVD player
19
How do you test a software
20
How do you test an Anti-Virus engine?
21
How do you test an online-game which could have
10000 online at the same time?
22
The complexity of software testing

• ???????????? (including user interface, network
  interface, file interfaces, etc. )
• ?????????????????
• Correct execution paths(?????)
• Incorrect execution paths (??????)
• ??????????????????????,??,??,??????????

23
SDLC (Software Development Life Cycle)

• ???? Requirement analysis (marketing research)
• ?????? Software function/performance
  specification
• ????? Analysis and Design (QA ??????????)
• ??,??PL,?? platform, database, network protocols.
• ????????,???????,??,??
• ????????? Coding and unit testing (by
  programmers)
• ???? Integration testing (by programmers/QAs)
• Alpha testing (by QAs)
• Most software functions and features are
  basically completed
• All functions are tested, no functions will be
  added beyond this point
• Serious flaws (high severity) are solved and
  addressed (show stopper)
• Beta testing (by Beta users)
• sub-serious bugs are all fixed
• Test plan has been completely executed
• Bug discovering rate is lower than bug fixing
  rate
• ???? Release
• Bug discovering rate is lower than bug fixing
  rate for a long period of time.
• The version after fixing bugs has been
  regressively tested (regression test)

24
Common software defects

• 70 occurs in design and hard to correct
• Software specifications are not precise
• Software is complicated
• Coding errors (20)
• Function changes which affects other components
• 3rd party software is flawed

25
? Early detection of errors
Error found Mar 13
Error found Feb 13
Error occurs Jan 13
Work (person-days)
10
20
Time
Work proceeds at (say) 10 person- days per month
10 person-days of work has been done assuming
the error is not there. Now this must be redone.
If error found this late, 20 person-days must be
redone
26
What to test for a QA (?????)

• ??? (completeness)
• ???????????,??????????????
• ??? (correctness)
• ??? (reliability)
• ??? (compatibility)
• ??(efficiency)
• ???? (usability)
• ??? (portability)
• ????? (scalability)
• ??? (testability)

27
??

• ????????????
• ???????????????????,????????????????
• ??????????,???????
• ???????,?????????,????????????
• ??????????????

28
??????

• ?????????
• Big software company has their testing team
• Small software companies which do not have
  resources to test can outsource to 3rd party
  testing company
• ?????????
• Big software company like ????Trend Micro has QA
  team
• Many software companies use programmers as
  testers a sad fact!

29
Company Organization
Developers
Marketings
QAs (testing teams)
30
Management Hierarchy
Director of development
Director of QA
Developer manager
Developer manager
Test manager
Test automation Team manager
Test facilities manager
programmers
Test automation programmers
Test database programmers
programmers
testers
31
Important reasons to make QA team and developer
team equal and separated

• ????????? programmers ?????(????)???
• ? programmer ????,???bugs ???????????,???????????
• ?????????bug ,?????????,????????????
• ??bug??,??????????,????????,???????
• ??????????
• QA ????????? programmer ?????
• QA ???????????? bug,????,QA??????
• QA ??? developer team ??,???????????????????

32
Marketing
????
Developer
QA (testers)
33
????V??
????
??????
????
??????????
???????
????
??????????
??????????
???????
????
???????
????
????
34
Software Defects

• Software defects represent the undesirable
  aspects of a software's quality
• What is undesirable?
• A software defect is software behaviors that do
  not conform to specification
• Software specs define the correctness of the
  program

35
Software defect classification

• Specs error
• A specs that is wrong in the beginning
• e.g., in specs, a teacher can only teach one
  course in practice, two teachers can teach one
  course and share the credits/fee
• design error
• An error that is caused by wrong design
• e.g., in design, UDP is used to transmit the
  holding (owning) of a treasure in practice, UDP
  can lose, program behaviors are incorrect
• coding error
• An error caused by programmer the error type
  you are mostly familiar with
• performance inadequacy failed to meet the
  nonfunctional requirement
• Scalability inadequacy failed to be able to
  serve large amount of users and transactions
• memory management error failed to meet the
  memory resource requirements

36
Severity of Software Defects

• show stopper (severity 1)
• unusable software (severity 1)
• Microsoft ranks the bug severity from 1-30
• killing people, waste a lot of money.

37
How to find software defect

• Dynamic
• testing
• testing, teamwork in a ACM survey are the three
  disciplines that should be trained in CS
• rely on dynamic execution of the program
• a program must be built first before testing
• The major way to test performance or scalability
• Static
• code review (survey shows 70 sever bugs can be
  discovered in this process)
• static analyzer verify software defects without
  running a program but by analyzing program or
  models

38
software defects detection techniques
classification

• optimistic (???)
• testing
• using optimistic approach can only show how good
  your code may be. It can not show the absence of
  faults
• pessimistic (???)
• static analyzer, code review
• mostly want to show the absence of faults
• may report spurious results
• compiler complains about usage of noninitialized
  variable

39
Optimistic v.s. Pessimistic

• Optimistic

40
Comments

• In principle, testing cannot show the absence of
  faults, so it is impossible to prove a program is
  correct by testing
• E.g., you can not prove a sorting algorithm is
  correct totally by testing. To prove a sorting
  program is correct exhaustively, you need to test
  N! cases. If N 100, it intractable.
• Using exhaustive testing to obtain correctness of
  a program is often infeasible for common problem.
  
• Question How Sorting algorithms in Algorithm
  Course are proved correct?

41
Comments

• Proof is difficult. You need to show that your
  solution works for all cases. Even the cases are
  N! and N can be arbitrary
• Testing only works for a comparatively small set
  of input domain under reasonable cost and
  resources.

42
Testing in Software industry

• Testing is still the most effective and mostly
  used fault detection techniques although testing
  in theory can not show the absence of faults.
• Testing only show the presence of faults not the
  absence of faults
• Most errors can still be found in extensive
  testing if you are willing to invest resources in
  it.

43
Testing in software development

• Testing plan in the beginning of the project
• unit testing (programmer)when programmers
  complete their functions, procedures, modules, or
  components, they should perform unit testing.
  Often proceeded as white-box testing
• Integration testing (programmer)The process for
  development team to integrate their modules,
  components, classes, library into a complete
  executable system (a complete build is produced
  for later stages).
• Alpha Testing (QA- Testers)The process for
  internal testers to test a complete build. It is
  often proceeded as blackbox testing. During this
  stages, assertion is often turned on.
• Beta Testing (Power Users)A build is given to
  power users to test. During this stages,
  assertion is often turned off.
• Released

44
Unit Testing (component testing in text book)

• Testing applied on software units. The typical
  types of system units are
• classes
• components
• modules
• library
• functions
• subsystems
• Unit testing is programmers responsibility

45
How to determine a units correctness?

• Since software units are not complete systems, so
  they do not have specification to describe their
  correctness, how can we determine the unit
  behaviors under tested is correct?
• ANS The correctness of units are specified
  in design not in specs. For example, correctness
  of units are specified in module interfaces,
  class interfaces, class descriptions, and so on.

46
Unit testing

• Just like the module testing described before.
• The purpose is to produce test cases to feed the
  interface of software units and exercise the
  software units
• In principle

47
The testing criteria for unit testing

• In principle, exhaustive testing of unit
  interface is also impossible.
• e.g.,void p(short int x, short int y)
• .
• You need to test 65536 65536 cases to make sure
  your unit behavior correctly and each test run
  needed to be judged by human (with eyes)
• So, how much we need to test to (at least)
  guarantee some degree of confidence.?

48
Criteria of confidence

• Test Coverage (from weak to strong)
• statement coverage each statement is at least
  executed once.
• branch coverage each branch should be exercised
  at least once
• path coverage each path should be at least
  exercised at least once

49
Binary search flow graph
50
The weakest criteria

• Statement coverage1 test case to run 1 2 3 8 9
• 1 test case to run 1 2 4 6 7 2
• 1 test case to run 1 2 4 5 7 2 8
• You only need 3 test cases to have each statement
  at least executed once.

51
Branch coverage

• each branch can produce two choices, every choice
  combinations should all be exercised
• there are 3 branches, so at most 2 2 2
  branching choices should be exercised if loop is
  not considered.
• So at least you need to find test cases to meet
  this criteria
• 1 2 8 9
• 1 2 3 8 9
• 1 2 3 4 5 7 2 8 9
• 1 2 3 4 5 7 2 3 8 9
• 1 2 3 4 6 7 2 8 9
• 1 2 3 4 6 7 2 3 8 9
• In the example, there are less than 8 branching
  choices because it is not a complete tree

52
Path Coverage

• The highest coverage
• Try to find test cases which can cover all the
  paths
• equal to exhaustive testing if you want to cover
  them all, which is impossible
• You need to find infinite test cases which have
  finite (if the program must stop) or infinite
  length (if the program may run forever)
• 1 2 8 9 (finite length)
• 1 2 3 8 9 (finite length)
• 1 2 3 4 5 7 2 8 9
• 1 2 3 4 5 7 2 3 4 5 7 2 8 9
• 1 2 3 4 5 7 2 3 4 6 7 2 3 8 9
• 1 2 (3 4 5 7 2) 8 9 ( finite length but
  very long sequence)
• 1 2 (3 4 5 7 2 3 4 6 7 2) 8 9 ( finite
  length with permutations)
• 1 2 (3 4 5 7 2) ? 8 9
• where is finite iteration in automata theory
• where ? is infinite iteration in automata theory

53
Path coverage

• in a program which can run for ever, full path
  testing is equal to exhaustive testing
• it is infeasible in practice
• So, we must discover more practical path coverage
  criteria.

54
Independent paths (a weaker criteria)

• 1, 2, 3, 8, 9
• 1, 2, 3, 4, 6, 7, 2
• 1, 2, 3, 4, 5, 7, 2
• 1, 2, 3, 4, 6, 7, 2, 8, 9
• each test case should explore statements that are
  not explored by the previous test cases to reduce
  redundant test cases
• A dynamic program analyser may be used to check
  that paths have been executed
• In practice, this is often difficult too

55
Test coverage in real world

• Since different coverage criteria involve
  different degree of cost and time
• Often statement coverage is the lower bound.
• Branching coverage or path coverage can be
  applied to important components
• Test coverage testing is widely applied in
  software industry. Although coverage rate mean
  nothing to program correctness. It is only an
  index to show if testing has been done
  adequately. Most people accept that index is
  closely related to software quality since test
  cases are executed.

56
Test coverage

• IMPORTANT
• If you have 100 statement coverage, it does not
  mean your program is correct.
• Sorting algorithm is an example
• you can easy derive only few test cases to have
  100 coverage of your program
• But to prove your program is correct is totally a
  different scale of the program

57
How to proceed test coverage testing?

• You need to have source code to know the
  coverage.
• You need to have tool to display the coverage
  rate
• Each test case, you need to have someone (or in
  very few cases, tools) to judge the correctness
  of program behaviors.

58
White box testing
Test cases
output
You test an equipment by observing the internal
behaviors of the equipment also known as
structural testing (text book)
59
Test coverage

• Test coverage must be done with white-box
  testing. The tester must have source code for
  testing
• Effective test cases must be derived from source
  code, which require programming skills.
• So, test coverage can be done by programmers or
  3rd party testing company.
• In USA, Europe, there are many 3rd party testing
  companies.
• In Taiwan, never heard of it. Maybe you should
  start a new one.

60
Test coverage tools

• Aonix Validator/Req
• BullseyeCoverage
• Clover
• CodeTEST
• CTC
• Dynamic Code Coverage
• GCT
• Glass JAR Toolkit
• Hindsight/TCA
• Hindsight/TPA
• Insure

• Java Test Coverage
• JavaCov
• Koalog Code Coverage
• LDRA Testbed
• Logiscope
• Microsoft Visual Test
• Rational PureCoverage
• Rational Test RealTime Coverage
• TCMON
• TCA
• TCAT C/C
• TCAT for Java
• TCAT-PATH
• TestCenter

61
(No Transcript)
62
Black Box Testing
Test Cases
Outputs
63
Black Box Testing

• A testing process proceed without knowing the
  internal behaviors of the system.
• Often done by QA (Testers)
• QA only interact with system by
• feeding test cases by user interfaces
• feeding test cases by files
• feeding test cases by networking traffic
• feeding test cases by consoles

64
BlackBox testing and Coverage

• In principle, Black Box testing can still combine
  with test coverage tools, if source code is
  available.
• Test coverage tools can be used to show the index
  of how good the test cases are derived by the
  QAs.

65
BlackBox Testing

• In practice, QAs use blackbox testing to assure
  Software quality after unit testing and
  integration testing. They take over from
  developers
• They focus on checking
• whether the program is conformed to specs.

66
A simple ???? of testing

• It is a common knowledge that testing after alpha
  testing should not be done by developers.
• developers tends to feed good test cases only
• developers are not happy to find more bugs (find
  more bugs, more works)
• find more bugs may not get promoted. On the
  contrary, more bugs prove his coding quality is
  low.
• So, to obtain high quality software, a simple
  management rule
• have testers to test the program, not programmers
• find more bugs mean promotion
• just like ????

67
Marketing
????
Developer
QA (testers)
68
BlackBox (alpha testing) Testing

• QAs test program by specs.
• In common cases, specs may describe
• correct behaviors
• incorrect behaviors that should be avoid.
• QAs job goal
• to show the presence of software errors as much
  as possible
• test the software from a users perspective
• test all positive, negative, and boundary cases

69
Black-box testing
70
Equivalence partitioning(partition testing)

• Input data and output results often fall into
  different classes where all members of a class
  are related
• Each of these classes is an equivalence partition
  where the program behaves in an equivalent way
  for each class member
• Test cases should be chosen from each partition

71
Equivalence partitioning
72
Equivalence partitioning with boundary cases

• Partition system inputs and outputs into
  equivalence sets
• If input is a 5-digit integer between 10,000 and
  99,999, equivalence partitions are lt10,000,
  10,000-99, 999 and gt 10, 000
• Choose test cases at the boundary of these sets
• 00000, 09999, 10000, 99999, 10001

73
Equivalence partitions
74
Automation of testing(including blackbox
whitebox)

• If software quality is important to you, of
  course, you want a higher criteria to be met.
• Unfortunately, higher criteria means higher test
  cases, more time, more cost.
• If your test cases outcome must to be judged by
  humans, higher criteria means much higher cost
  and time

75
Testing automation

• In many applications or functionalities, testing
  can be automated at different degrees. In
  contrast, many functionalities and applications
  are not amenable to automatic testing, where
  humans are needed
• examples that is difficult to automate testing
• testing WORD.
• examples that is easy to automate
• testing a sorting algorithm you can easily
  write a program to check if the output is listed
  in ascending or descending order

76
Test oracles

• If test cases can be derived automatically and
  then these test cases can be executed
  automatically and then report the error What a
  perfect world ! We can fire all the testers.
• In practice, this is only a dream
• In many applications, Effective test cases can
  only be derived by human
• In many applications, Executions requires human
  to monitor and validate if program behaviors are
  conformed to specs.

77
Test Oracles

• Oracle In ancient Greece, an oracle was a
  priest or priestess who made statements about
  future events or about the truths
• A test oracle is a program that can determine if
  the program behaviors or program input/output are
  conformed to specs.

78
Testing automation

• For testing automation, test oracle must be
  presented to determine the conformity between
  specs and programs.
• In some applications, test oracles are easy to
  derive
• In most applications, test oracles are difficult
  to implement or in theory, impossible to derive.

79
Testing automation

• In a big map, network games, a test oracle can be
  
• characters movements are controlled by a test
  program. The test program keep feeding the mouse
  events and keyboard events randomly
• A simple test oracle is to see if the program
  crash. This test oracle is easy and close to
  none.
• A more complicated test oracle is to check if the
  virtual money given to the characters are
  controlled under a fixed amount, C How can
  you design this test oracles?
• complicated test oracle often require program
  instrumentation program code that inserted in
  the program only for monitoring purpose.
• The difficulty of test oracles is depending on
  what kind of system properties you want to check.

80
Integration testing

• Tests complete systems or subsystems composed of
  integrated components
• Integration testing should be black-box testing
  with tests derived from the specification
• Main difficulty is localising errors
• Incremental integration testing reduces this
  problem

81
Incremental integration testing
82
Approaches to integration testing

• Top-down testing
• Start with high-level system and integrate from
  the top-down replacing individual components by
  stubs where appropriate
• Bottom-up testing
• Integrate individual components in levels until
  the complete system is created
• In practice, most integration involves a
  combination of these strategies

83
The key to incremental testing

• emulate the functions you will call
• emulate the functions that call you.

84
Top-down testing
85
Top-down testing

• top down testing is used mostly by personal
  programming.
• Test cases are full test cases derived one by one
  from specs.
• it is a hot programming paradiagm called test
  driven programming.

86

• main()
• loadfile(myarray) // load an array from
  file
• sorting(myarray) // sort an array
• input a // get input from
  user
• ret binseary(myarray,a)
• if (ret) dosomethingA()
• else dosomethingB()

87

• An loadfile stubs
• loadfile(int array)
• array1 10
• array2 1
• array3 55

88
Bottom-up testing
89
Bottom up Testing

• Bottom up testing are suitable for teamwork
  project which proceeds concurrently and parallel.
• Test drivers are derived based on modules
  interfaces, parameters, etc. They are typically
  not full test cases derived from specs.

90
Comparisons of top-down and bottom up testing

• top-down test cases are derived from specs. So,
  many errors can be discovered earlier.
• top down testing is often less efficient than
  bottom up regarding teamwork and parallel
  cooperation.
• bottom up testing drawback test cases are
  typically not derived from specs. If design is
  wrong, it can not be discovered earlier.

91
Testing approaches

• Architectural validation
• Top-down integration testing is better at
  discovering errors in the system architecture
• System demonstration
• Top-down integration testing allows a limited
  demonstration at an early stage in the
  development
• Test implementation
• Often easier with bottom-up integration testing
• Test observation
• Problems with both approaches. Extra code may be
  required to observe tests

92
Interface testing

• Takes place when modules or sub-systems are
  integrated to create larger systems
• Objectives are to detect faults due to interface
  errors or invalid assumptions about interfaces
• Particularly important for object-oriented
  development as objects are defined by their
  interfaces

93
Interface testing
94
Interfaces types

• Parameter interfaces
• Data passed from one procedure to another
• Shared memory interfaces
• Block of memory is shared between procedures
• Procedural interfaces
• Sub-system encapsulates a set of procedures to be
  called by other sub-systems
• Message passing interfaces
• Sub-systems request services from other
  sub-systems

95
Interface errors

• Interface misuse
• A calling component calls another component and
  makes an error in its use of its interface e.g.
  parameters in the wrong order
• Interface misunderstanding
• A calling component embeds assumptions about the
  behaviour of the called component which are
  incorrect
• Timing errors
• The called and the calling component operate at
  different speeds and out-of-date information is
  accessed

96
Interface testing guidelines

• Design tests so that parameters to a called
  procedure are at the extreme ends of their ranges
• Always test pointer parameters with null pointers
• Design tests which cause the component to fail
• Use stress testing in message passing systems
• In shared memory systems, vary the order in which
  components are activated

97
Stress testing

• Exercises the system beyond its maximum design
  load. Stressing the system often causes defects
  to come to light
• Stressing the system test failure behaviour..
  Systems should not fail catastrophically. Stress
  testing checks for unacceptable loss of service
  or data
• Particularly relevant to distributed systems
  which can exhibit severe degradation as a
  network becomes overloaded

98
Stress testing

• require considerable cost and efforts.
• often require you to implement a system to test
  the system.

99
Object-oriented testing

• The components to be tested are object classes
  that are instantiated as objects
• Larger grain than individual functions so
  approaches to white-box testing have to be
  extended
• No obvious top to the system for top-down
  integration and testing

100
Testing levels

• Testing operations associated with objects
• Testing object classes
• Testing clusters of cooperating objects
• Testing the complete OO system

101
Object class testing

• Complete test coverage of a class involves
• Testing all operations associated with an object
• Setting and interrogating all object attributes
• Exercising the object in all possible states
• Inheritance makes it more difficult to design
  object class tests as the information to be
  tested is not localised

102
Weather station object interface

• Test cases are needed for all operations
• Use a state model to identify state transitions
  for testing
• Examples of testing sequences
• Shutdown Waiting Shutdown
• Waiting Calibrating Testing Transmitting
  Waiting
• Waiting Collecting Waiting Summarising
  Transmitting Waiting

103
Object integration

• Levels of integration are less distinct in
  object-oriented systems
• Cluster testing is concerned with integrating and
  testing clusters of cooperating objects
• Identify clusters using knowledge of the
  operation of objects and the system features that
  are implemented by these clusters

104
Approaches to cluster testing

• Use-case or scenario testing
• Testing is based on a user interactions with the
  system
• Has the advantage that it tests system features
  as experienced by users
• Thread testing
• Tests the systems response to events as
  processing threads through the system
• Object interaction testing
• Tests sequences of object interactions that stop
  when an object operation does not call on
  services from another object

105
Scenario-based testing

• Identify scenarios from use-cases and supplement
  these with interaction diagrams that show the
  objects involved in the scenario
• Consider the scenario in the weather station
  system where a report is generated

106
Collect weather data
107
Weather station testing

• Thread of methods executed
• CommsControllerrequest WeatherStationreport
  WeatherDatasummarise
• Inputs and outputs
• Input of report request with associated
  acknowledge and a final output of a report
• Can be tested by creating raw data and ensuring
  that it is summarised properly
• Use the same raw data to test the WeatherData
  object

108
Testing workbenches

• Testing is an expensive process phase. Testing
  workbenches provide a range of tools to reduce
  the time required and total testing costs
• Most testing workbenches are open systems because
  testing needs are organisation-specific
• Difficult to integrate with closed design and
  analysis workbenches

109
A testing workbench
110
Tetsing workbench adaptation

• Scripts may be developed for user interface
  simulators and patterns for test data generators
• Test outputs may have to be prepared manually for
  comparison
• Special-purpose file comparators may be developed

111
Three Well Known Approaches

• Testing execution on product or program
  (implementation) to detect faults (optimistic)
• Simulation execution on artifacts (models)
  which is simpler representation of the original
  complexity (optimistic)
• Verification explore all the state space of
  models which is a simpler representation of
  original complexity (pessmistic)

112
Models

• So, we want models can be
• Simulated!
• Verified !
• They can provide more clues of how our design is
  wrong
• By the way, what is design?

113
The foundation behind model simulation and
verification
X ? Y
(X gt -1) gt (Ygt-1)
Preorder (implements)
?
Prog
Model
deadlock exist ? deadlock
exists
114
Fault Detection Tools in Industry