Title: Black-Box Testing
1Black-Box Testing
( FÅP First-year Project Course, ITU, Denmark )
- Claus Brabrand
- brabrand_at_itu.dk
2Reminder Learning Exam Goals
3Outline
- Introduction
- Bugs and Black-box testing
- Testing the Specification
- static black-box testing
- Testing the Code
- dynamic black-box testing Equivalence
partitioning - Other techniques
- Model-checking Static-analysis
- Exercises
- Work on your group exercises
4Definition bug
- Main entry 2bug
- Pronunciation /b?g/
- Function noun
- Etymology origin unknown
- Date 1622
- 1 a an insect or other creeping or crawling
invertebrate (as a spider or centipede) - b any of several insects (as the bedbug or
cockroach) commonly considered obnoxious - c any of an order (Hemiptera and especially
its suborder Heteroptera) of insects that have
sucking mouthparts, forewings thickened at the
base, and incomplete metamorphosis and are often
economic pests called also true bug - 2 an unexpected defect, fault, flaw, or
imperfection ltthe software was full of bugsgt - 3 a a germ or microorganism especially when
causing disease b an unspecified or nonspecific
sickness usually presumed due to a bug - 4 a sudden enthusiasm
- 5 enthusiast lta camera buggt
- 6 a prominent person
- 7 a crazy person
- 8 a concealed listening device
- 9 a weight allowance given apprentice jockeys
5Example of a Bug -)
Hej Claus, A propos test og datoberegninger
Fredag morgen mente min PDA kalender (Microsoft
Windows CE) at vi skrev lørdag 1. marts. Jeg
stillede selvfølgelig tålmodigt uret tilbage til
fredag 29. februar 2008 (dette fandt sted på et
møde hos Microsoft i Vedbæk -) Lørdag formiddag
startede den så med at vise min (ret tomme)
kalender for august 2049. Det indbyggede ur stod
på 1. januar 1601. Ahem. Peter
6Terminology (for Bugs)
- Generic terms
- Problem
- Error
- Bug (? well use this term)
- Severe conditions
- Fault
- Failure
- Defect
- Unintended operation
- Anomaly
- Incident
- Variance
- Feature (intended)
Typically imply blame as in- it was his fault
Famous quote - Its not a bug, its a feature
7The Harvard Mark II Bug
The first documented computer bug was a moth
found trapped between points at Relay 70,
Panel F, of the Mark II Aiken Relay Calculator
while it was being tested Harvard University,
Sep. 9, 1947
Photo of firstactual bug
8Definition Bug
- A bug is a relation between
- Specification
- Implementation
- Whether or not specification is
- Explicit or Implicit
- Written or Oral
- Formal or Informal
- (e.g., product spec vs. back-of-envellope)
9Specification
- A spec states things an impl
- Should do!
- Shouldnt do!
- Unspecified? (unclear, unmentioned, or
left open)
Should do!
Unspecified?
S
Shouldnt do!
Specs are often intentionally under-specified.
Its often better to not prematurely commit to
a particular solution (by specifying exactly how
a task should be done) and instead just state
which overall tasks the system should do.
10Formal Definition Bug
cf. Software Testing, R.Patton, p.15
- A (software) bug occurs when
- 1. Impl doesnt do sth spec says it should
-
- 2. Impl does sth spec says it shouldnt
-
- 3. Impl does sth spec doesnt mention
-
- 4. Spec doesnt mention sth, but should
-
- 5. Impl is hard to understand/use by user(s)
-
(or does it incorrectly)
11Consequences
- Consequently,
- Additional functionality?
- e.g., a calculator which also does square root
(but wasnt supposed to) - Easter egg?
- e.g., hit AltShift2 in Solitaire to win game
is a bug !
is a bug !
12Question
- Question
- A) R.Patton interprets as (spec impl)
- ? trivially no bugs in impl !
- (according to own definition 1,2,3)
- B) More sensible to interpret as no spec
- ? entire impl is essentially one big bug !
- (according to definition 34 p.15)
an impl w/o a spec ?
Because the software is already complete, you
have the perfect specification the product
itself
R.Patton, Software Testing, p.31
13Number one Cause of Bugs
cf. Software Testing, R.Patton, p.17
other
impl
spec
design
- Often due to
- complexity
- tight schedule
- under-specification
- under-documentation
(NB no percentages given)
14Cost of (Fixing) Bugs
cf. Software Testing, R.Patton, p.18
- Cost of bugs increases exponentially (over time)
log
10,000,000
1,000,000
10,000
100
1
spec
design
code
test
release
15Testers Goal
cf. Software Testing, R.Patton, p.19
- The goal of a software tester is
-
- Testers (therefore) often arent the most popular
members of a project team
- 1) to find bugs - 2) find them as early as
possible and - 3) make sure they get
fixed R.Patton, Software Testing, p. 19
16The Pesticide Paradox
- The more you test a software, the more immune
it becomes to your tests - B.Beizer, Software Testing Techniques, 1990
17Bugs
- Bugs often occur in groups
- If you find one, chances are there are more
nearby - Not all bugs will be fixed
- Too costly?
- Too risky?
- Not cost-effective?
- Bug unknown -)
18Programming vs. Testing
- Constructive thinking
- Test-to-pass
- Often not a good idea to test your own code -(
- Destructive thinking
- Test-to-fail
- (aka., error-forcing)
- Often better to test/break someone elses code -)
Advice test-to-pass before test-to-fail (i.e
., test normal usage first)
19Black-Box Testing
- The goal of black-box testing is
- Make sure impl solves problem it is supposed to
- i.e.,
- Point of departure
- spec, not impl
- not a particular program which claims to solve
problem - testing w/o insights of code
impl spec
Static (test spec)
Dynamic (test impl)
20Outline
- Introduction
- Bugs and Black-box testing
- Testing the Specification
- static black-box testing
- Testing the Code
- dynamic black-box testing Equivalence
partitioning - Other techniques
- Model-checking Static-analysis
- Exercises
- Work with your group exercises
21Spec Warning Words (I/III)
cf. Software Testing, R.Patton, p.61
- Unconditionals (universally)
- Always, for every, for each, for all,
- Try to violate (i.e., find exemptions from rule)!
- Unconditionals (never)
- None, never, not under any circumstances,
- Try to violate (i.e., find exemptions from rule)!
- Tautologies
- Trivially, certainly, clearly, obviously,
evidently, - Check assumptions (that nothings swept under the
rug)!
22Spec Warning Words (II/III)
cf. Software Testing, R.Patton, p.61
- Unspecified conditionals
- Some(-times), often, usually,
ordinarilly, mostly, - Unclear spec (under which circumstances)?
- Continuations
- Etcetera, and so forth, and so on,
- Check that spec is comprehensively unambiguous?
- Examples
- E.g., for example, such as,
- Is example representative (what about other
examples)?
23Spec Warning Words (III/III)
cf. Software Testing, R.Patton, p.61
- Positive adjectives
- Good, fast, efficient, small, reliable,
stable, - Subjective (needs objectification if to be used
for testing)! - Alegedly completed
- Handled, processed, taken care of,
eliminated, - Is something hidden?
- Incompleted
- Skipped, unnecessary, superfluous,
rejected, - Is something forgotten?
24Spec Warning Words (if-then)
cf. Software Testing, R.Patton, p.61
- Finally, watch out for
- If Then (with missing Else)
- Check what happens in the Else-case
25Test Sequencing
- Question which is best (and why)?
- A) white-box testing black-box testing
- (i.e., white-box testing first)
- B) black-box testing white-box testing
- (i.e., black-box testing first)
- Answer B
- Settle overall impl spec problems first
- Before zooming in on the impl as in white-box
testing
or
26Outline
- Introduction
- Bugs and Black-box testing
- Testing the Specification
- static black-box testing
- Testing the Code
- dynamic black-box testing Equivalence
partitioning - Other techniques
- Model-checking Static-analysis
- Exercises
- Work on your group exercises
27Test Boundaries
- Programs are vulnerable around the edges
- e.g. testing legal inputs (time, in hours)
- e.g. testing legal inputs (dates, in April)
Property Minimum-1 Minimum Middle
repr. Maximum Maximum1
Expected output invalid valid valid valid invalid
Actual output
Input -1 0 11 (e.g.) 23 24
Property Minimum-1 Minimum Middle
repr. Maximum Maximum1
Input 00/4 01/4 12/4 (e.g.) 30/4 31/4
Expected output invalid valid valid valid invalid
Actual output
28Test Powers-of-Two
- Programs vulnerable around powers-of-two
- e.g. years of age (assume held in a byte)
- e.g. game-spectators (assume held in a 16-bit
word)
Property Minimum-1 Minimum Middle
repr. Maximum Maximum1
Expected output invalid valid valid valid invalid
Actual output
Input -1 0 27 (e.g.) 255 256
Property Minimum-1 Minimum Middle
repr. Maximum Maximum1
Input -1 0 12345 (e.g.) 65535 65536
Expected output invalid valid valid valid invalid
Actual output
29Test Empty Input
- Default / empty / blank / null / zero / none / ?
- e.g., any program
Property No input
Expected output Error message
Actual output
Input ?
30Test Invalid Input
- Invalid / illegal / wrong / garbage / bogus data
- e.g., calculator
Property Invalid input Bogus data!!!
Expected output Error message Error message
Actual output
Input 31 ?!
31Equivalence Partitioning
- Partitions and
- Equivalence Relations
32Testing Infinite process
- Recall testing is an incomplete process
- (i.e., testing cant prove absence of bugs)
- There are infinitely many possible inputs
- (hence, testing will take an infinite amount of
time)
33Crash course on Relations
- Relations
- Equivalence Relations
34Relations
- Example1 even relation
- Written as as a short-hand for
- and as as a short-hand for
- Example2 equals relation
- Written as as a short-hand for
- and as as a short-hand for
- Example3 dist-btwn relation
- Written as
- as short-hand for
_even ? Z
_even 4
4 ? _even
5 ? _even
? Z ? Z
(2,2) ?
2 2
2 ? 3
(2,3) ?
? ? CITY ? Z ? CITY
(Aarhus, 310, Copenhagen) ? ?
35Equivalence Relation
- Let be a binary relation over set A
- ? A ? A
- is an equivalence relation iff
- Reflexive
-
- Symmetric
-
- Transitive
-
?x?A x x
?
?x,y?A x y ? y x
?
?x,y,z?A x y ? y z ? x z
36Exercise Eq.Rel.
- Which relations are equivalence relations?
- and which are not (and why not)?
- a) The "less-than-or-equal-to" relation ' ? '
- (n,m) n,m ? Z, n ? m
- b) The almost-total-relation-on-integers,
(relating all numbers except 42, but relating 42
with 42) - (n,m) n,m ? ( Z \ 42 ) ? (42,42)
- c) The "is-congruent-modulo-three" relation
- (n,m) n,m ? Z, (n 3) (m 3)
- d) The "have-talked-together" relation
- (p,q) n,m ? PEOPLE, p and q have talked
together - e) The "is-in-the-same-group-as" relation
- (p,q) n,m ? PEOPLE, p and q are in same
FÅP group
37Eq. Rel. ? Partition
- Equivalence relation
- (A,A), (B,B), (A,B), (B,A), (P,P), (X,X),
(Y,Y), (Z,Z), (X,Y), (Y,X), (X,Z), (Z,X),
(Y,Z), (Z,Y) - E.g.
- A B, P P, X X, X Z
- A P, B Y, P Z
P
A
X
B
Z
Y
Canonical representatives A B
A, B P P X
Y Z X, Y, Z
Capture the same information i.e., notion of
equivalence
38Testing Infinite process
- Recall testing is an incomplete process
- (i.e., testing cant prove absence of bugs)
- There are infinitely many possible inputs
- (hence, testing will take an infinite amount of
time)
39Equivalence Partitioning
- Partition input
- Finitary partition
- If finite categories (aka. equivalence
classes) - Here 3 zero, pos, neg
- We can now test all equivalence classes
- Using representative elements from each category
zero
0
1 2 3
-3 -2 -1
neg
pos
40Test Sum (contd)
- We can now test all equivalence classes
- Using representative input from each category
- Sum (testing all equivalence classes)
Property Pos, Pos Neg, Pos Zero,Pos Pos, Neg Neg,
Neg Zero,Neg Pos, Zero Neg, Zero Zero,Zero
Expected output 3 1 5 -1 -17 -10 11 -12 0
Actual output
Input (1,2) (-3,4) (0,5) (6,-7) (-8,-9) (0,-10) (1
1,0) (-12,0) (0,0)
41Frequent Partitions for Testing
- Numbers
- Positive, Negative, Zero
- Zero, One, Two, Many (aka. Greenlandic Numbers)
- Lists
- Length-0, Length-1, Length-2, Length-3
- Ascending-elements, Descending-elements,
AscDesc-elements
Advice - consider how problem might be solved -
partition into qualitatively different
categories such that (likely same category ?
same error)
42Outline
- Introduction
- Bugs and Black-box testing
- Testing the Specification
- static black-box testing
- Testing the Code
- dynamic black-box testing Equivalence
partitioning - Other techniques
- Model-checking Static-analysis
- Exercises
- Work on your group exercises
43Model-checking
- Modelling
- Verification
- Model-checking
44Program vs. Model World
(Java)
(modeling lang)
Program world
Model world
abstraction
P
M
concretization
Abstract
Concrete
45Methodology Model-based Design
- Design abstract model
- Decompose model
- Reason/Test/Verify model
- individual parts and whole
- Recompose insights
- make model safe
- Impl. concrete program
abstract
REAL PROBLEM
MODEL
reason
?
?
test
?
?
verify
concretize
SAFE MODEL
SAFE PROGRAM
46Model-checking
(Java)
(modeling lang)
Program world
Model world
1. P ? ?
3. M ? ?
2. abstract
P
M
4. verify!!
5. M ? !
7. P ? !
6. concretize
Abstract
Concrete
If models are finite, we can have a computer
test all possibilities.
Never send a human to do a machines job --
A.Smith (99)
47Static Analysis
- Static Analysis
- Undecidability
48Undecidability
- Most interesting properties are undecidable
- e.g., can program P have a type error (when run)
- Compilers use approximations (computed via
static analyses)
no program can decide this line automatically
(in all cases)
?
incorrect
correct
incorrect
correct
safe (over-)approximation
49Undecidability (self-referentiality)
- Consider "The Book-of-all-Books"
- This book contains the titles of all books that
do not have a self-reference (i.e. don't contain
their title inside) - Finitely many books i.e.
- We can sit down figure out whether to include
or not... - Q What about "The Book-of-all-Books"
- Should it be included or not?
- "Self-referential paradox" (many guises)
- e.g. "This sentence is false"
"The Bible"
"War and Peace"
"Programming Languages, An Interp.-Based
Approach"
...
The Book-of-all-Books
?
?
50Termination Undecidable!
- Assume termination is decidable (in Java)
- i.e. ? some program, halts PROG ? BOOL
- Q Does P0 loop or terminate...? )
- Hence "Termination is undecidable"
- ...for Java, C, C, Pascal, ?-Calculus, Haskell,
...
bool halts(Program p) ...
-- P0.java --
bool halts(Program p) ... Program p0
read_program("P0.java") if (halts(p0)
halts) loop() else halt()
51Example Type Checking
- Will this program have type error (when run)?
-
- Type error ? ltEXPgt evaluates to true
void f() var b if (ltEXPgt) b
42 else b true /
some code / if (b) ...
Undecidable (in all cases)
i.e., what ltEXPgt evaluates to (when run)
52Example Type Checking (contd)
- Hence, languages use static requirements
-
- All variables must be declared
- And have constant type (throughout the program)
void f() bool b / some code /
if (ltEXPgt) b 42 else b
true / some more code /
Static compiler error Regardless of what
ltEXPgtevaluates to when run
53Outline
- Introduction
- Bugs and Black-box testing
- Testing the Specification
- static black-box testing
- Testing the Code
- dynamic black-box testing Equivalence
partitioning - Other techniques
- Model-checking Static-analysis
- Exercises
- Work on your group exercises