Today

1
Todays Agenda

• HW 3 Due, HW 4 Out
• Quick Review
• Finish Race Analysis
• Reachability Testing

2
Quick Review

• What is data race? What is general race?

3
Reachability Testing

• Introduction
• A General Execution Model
• Computing Race Variants
• A RT Algorithm
• Empirical Results
• Conclusion

4
Concurrent Programs

• Contain a set of cooperative threads/processes
• multithreaded programs - threads synchronize by
  accessing shared memory
• distributed programs threads synchronize by
  exchanging messages
• Pervasive in modern software development
• better resource utilization
• increased computing efficiency
• provide simple solutions to certain problems
• Notoriously difficult to test due to their
  non-deterministic behavior
• Multiple executions of the same program with the
  same input may display different behaviors

5
Testing Strategies

• Non-deterministic testing - execute the same
  program with the same input for multiple times
• easy, but inefficient, and some errors cannot be
  detected
• Deterministic testing - select a set of test
  sequences and then force them to be exercised
• can detect more subtle errors, but requires
  additional effort for test sequence selection and
  runtime control
• State exploration explore the state space in a
  systematic manner
• suffers the well-known state explosion problem
• state representation is difficult for programs
  written in a full-fledged programming language

6
Reachability Testing

• Combines non-deterministic and deterministic
  testing
• Generating test sequences dynamically, without
  constructing any static model
• Dealing with partial orders directly, and thus no
  redundant interleavings!
• Can also be considered as a state exploration
  technique
• Exercising every possible synchronization
  sequence of a program with a given input

7
The RT Process

• Execute a program P with input I
  non-deterministically to collect a trace Q
• Identify the race conditions in Q and compute its
  race variants
• For each variant, conduct a prefix-based test run
  to collect a new trace Q
• Repeat 2, 3, and 4 for each newly collected trace
  Q.

8
Assumptions

• For a concurrent program P with a given input X,
  the RT process will exercise every
  partially-ordered SYN-sequence of P with X
  exactly once, under the following assumptions
• P is closed
• Concurrency is the only source of
  non-determinism
• A test driver is used to make every test run
  terminate
• RT can be applied to the commonly used
  synchronization constructs, including message
  passing, semaphores and monitors.

9
Example
10
Reachability Testing

• Introduction
• A General Execution Model
• Computing Race Variants
• A RT Algorithm
• Empirical Results
• Conclusion

11
Event Types

• Message-passing
• When a thread T performs a send (receive)
  operation, a send (receive) event occurs on T
• Semaphore
• When a thread T calls a P or V operation on a
  semaphore s, a call event occurs on T
• When a P or V operation is completed, a semaphore
  completion event occurs on s
• SU Monitor
• When a thread T calls a method of monitor M, a
  monitor call event occurs on T.
• When T eventually enters the monitor, a monitor
  entry event occurs on M
• SC Monitor
• A monitor call event occurs on a thread T when it
  calls a monitor method or tries to re-enter the
  monitor
• A monitor entry event occurs when T eventually
  enters the monitor

12
SYN-sequence

• A pair of sending and receiving events that are
  synchronized with each other
• Sending event Send or call event
• Receiving event Receive, completion, or entry
  event
• Let Q be a SYN-sequence exercised by a
  concurrent execution. Q can be defined as a tuple
  (Q1, Q2, , Qn, Y)
• Qi the totally-ordered sequence of events that
  occurred on a thread or a synchronization object
• Y the set of synchronization pairs
• The outcome of a concurrent execution is
  determined by the program text, the input, and
  the SYN-sequence exercised by the execution.

13
SYN-sequence Equality

• Two SYN-sequences Q and Q are equal, denoted as
  Q Q, if there exists a one-to-one mapping m
  from the events in Q to those in Q that
  preserves the synchronization relation,
• That is, ltm(s), m(r)gt is a synchronization pair
  in Q if and only if lts, rgt is a synchronization
  pair in Q.

14
Event Descriptor (1)

• Sending event (Sender, Destination, Operation,
  Index)
• Sender the sending thread
• Destination the destination thread or
  synchronization object
• Operation the type of the operation performed
  (P, V, send, etc),
• Index the event index
• Receiving event (Receiver, OpenList, index)
• Receiver the receiving thread or object
• OpenList a field to be defined later that
  assists in identifying race conditions
• index the event index

15
Event Descriptor (2)
Event descriptor for sending event
Synchronization construct Sender Destination Operation index
asynchronous message passing sending thread ID Port ID send event index
synchronous message passing sending thread ID Port ID send event index
semaphores calling thread ID semaphore ID P or V event index
monitors calling thread ID monitor ID method name event index
Event descriptor for receiving event
Synchronization construct Receiver OpenList index
asynchronous message passing receiving thread ID the port of r event index
synchronous message passing receiving thread ID open ports (including rs port) event index
semaphores semaphore ID operations (P and/or V) that could be completed at r event index
monitors monitor ID all of the monitors methods event index
16
Example Asynchronous Message-Passing
17
Example Synchronous Message-Passing
18
Example Semaphore
19
Example - Monitor
20
Reachability Testing

• Introduction
• A General Execution Model
• Computing Race Variants
• A RT Algorithm
• Empirical Results
• Conclusion

21
Timestamp Assignment

• Recall that vector timestamps can be used to
  determine the happened-before relation.
• Notations
• T.v the vector clock maintained by a thread T.
• T.vi the ith element in T.v
• e.ts the vector timestamp of an event e.

22
Timestamp Syn. Comm.

• When a thread Ti executes a blocking send event
  s (a) Ti.vi Ti.vi 1 (b) s.ts Ti.v.
• When a thread Tj executes a receive event r that
  receives the message sent by s (a)Tj.vj
  Tj.vj 1 (b) Tj.v max(Tj.v, s.ts) and (c)
  r.ts Tj.v. Thread Tj also sends Tj.v back to
  thread Ti
• When thread Ti receives Tj.v Ti.v max(Ti.v,
  Tj.v).

23
Timestamp Semaphore/Monitor

• When a thread Ti executes a sending event s (a)
  Ti.vi Ti.vi 1 (b) s.ts Ti.v
• When a receiving event r that is synchronized
  with a sending event s occurs on a
  synchronization object O (a) O.v max(O.v,
  s.ts) (b) r.ts O.v.
• Semaphore When a thread Ti finishes executing a
  P() or V() operation on semaphore O Ti.v
  max(Ti.v, O.v).
• SU monitor When a thread Ti finishes
  executing a method of monitor O T.v max(T.v,
  O.v).
• SC monitor When a thread Ti finishes
  executing a method of monitor O, or when Ti is
  signaled while waiting on a condition variable
  of O Ti.v max(Ti.v, O.v).

24
Happened-before Relation

• For a sending or receiving event e, let e.tid be
  the (integer) thread ID of the thread that
  executes e.
• If event e is a completion event on a semaphore
  or an entry event on a monitor, then e.tid is the
  thread ID of the thread that executes the sending
  partner of e.)
• Let e1 and e2 be two events in Q. Then, e1 ? e2
  if and only if
• lte1, e2gt is a synchronization pair or
• e1.tse1.tid ? e2.tse1.tid and e1.tse2.tid lt
  e2.tse2.tid.

25
Control Structure

• The control-structure c-struct(e, Q) of event e
  contains all the events, as well as the
  synchronizations between them, that could
  possibly control whether or not event e is
  executed.
• Empty if e is the first event exercised by a
  thread
• Otherwise, the prefix of Q that contains the
  event f that T exercised immediately before e and
  all the events that happened before f, including
  the synchronizations between these events.

26
Event Equality

• Let CP be a concurrent program. Let Q and Q be
  two SYN-sequences of a concurrent program CP with
  input X.
• An event e in Q is equal to an event e in Q,
  denoted as e e, if c-struct(e, Q)
  c-struct(e, Q).

27
Lead Race

• Let s be a sending event and r be a receiving
  event in an execution such that lts, rgt is a
  synchronization pair. Let s be another sending
  event in the execution.
• There exists a lead race between s and s with
  respect to r if s and r can be synchronized with
  each other during another execution in which all
  the events that happen before s or r in Q are
  replayed.

28
Race Set

• Let s be a sending event and r be a receiving
  event such that lts, rgt is a synchronization pair.
  The race set race_set(r) is the set of sending
  events in Q that have a race with s with respect
  to r.
• A sending event s is in the race set of a
  receiving event r in Q if
• s is open at r
• r does not happen before s
• if lts, rgt is a synchronization pair, r happens
  before r
• if a sending event s has the same source and
  destination as s but happens before s, there
  exists a receiving event r such that lts, rgt is
  a synchronization pair and r happens before r.

29
Race Variant

• Let Q be a SYN-sequence. A race variant V of Q
  is another SYN-sequence that satisfies the
  following conditions
• There exists at least one receiving event r in
  both Q and V such that send(r, Q) ? send(r, V).
• Let r be a receiving event in Q and V. If
  send(r, Q) ? send(r, V), then send(r, V) must be
  in race_set(r, Q).
• Let e be a sending or receiving event in Q.
  Then, e is not in V if and only if there exists a
  receiving event r in Q such that r ? c-struct(e)
  and send(r, Q) ? send(r, V).

30
Race Table

• In a race table, each column represents a
  receiving event whose race set is non-empty, and
  each row represents a unique, partially-ordered
  race variant of Q.
• Let r be a receiving event represented by one of
  the columns and V be a race variant represented
  by one of the rows. The value v in the row for V
  and the column for r indicates how r in sequence
  Q is changed to create variant V
• v -1 indicates that r is removed from V.
• v 0 indicates that no new sending partner is
  specified for r in V.
• v gt 0 indicates that in V, the sending partner of
  r is changed to the v-th event in race_set(r, Q),
  where the sending events in race_set(r, Q) are
  arranged in an arbitrary order and the index of
  the first event in race_set(r, Q) is 1.

31
Race Table - Example
r1 r3 r4
0 0 1
0 1 -1
0 2 -1
1 0 0
1 1 -1
32
A Naïve Algorithm (1)

• Let (r1, r2, , rn) be the heading of the race
  table, which consists of the receiving events
  whose race sets are non-empty, arranged in an
  arbitrary order.
• Let domain(ri) be the set of values that can
  appear in the column with heading ri.
• If the size of the race set for ri is denoted as
  race_set(ri), then the set of values in
  domain(ri) is -1, 0, 1,,race_set(ri).

33
A Naïve Algorithm (2)

• Generates the set T of tuples that represent all
  the possible combinations of changes
• T domain(r1) ? domain(r2) ? ? domain(rn).
• A tuple t ? T to the race table if t passes a
  validity check
• There is at least one value ti, 1 ? i ? n,
  such that tigt 0.
• ti -1, 1? i ? n, if and only if there
  exists an index j, where 1 ? j ? n and j ? i,
  such that tj gt 0 and rj ? c-struct(ri)
• If ti gt 0, there does not exist an index j, 1?
  j ? n, such that tj gt 0 and rj ? c-struct(s),
  where s is the tith sending event in
  race_set(ri),

34
Algorithm Construct-Race-Table

• 1. initialize table (heading, rows) to be an
  empty race table
• 2. R r ? Q race_set(r) gt 0
• 3. let heading (r1, r2, ..., rR be a
  topological order of R w.r.t the happened-before
  relation
• 4. D d1, d2, ..., dR, where di
  race_set(ri)
• 5. let t be an array of length R and
  initialize t with all 0s
• 6. while (true)
• 7. find the largest index i such that ti lt
  di and ti ! -1
• 8. if (such an index i does not exist)
• 9. break
• 10. ti
• 11. if (ti 1) // just changed ti from 0
  to 1
• 12. for (i lt j ? R)
• 13. if (tj ? -1 and ri ? c-struct(rj))
• 14. tj -1
• 15. for (i lt j ? R)
• 16. if (tj dj)
• 17. tj 0 // just changed tj from
  dj to 0
• 18. for (j lt k ? R)
• 19. if (tk -1 and rj ?
  c-struct(rk) and there is no index l, 1? l lt k,
  such that tl gt 0 and rl ? c-struct(rk))

35
Reachability Testing

• Introduction
• A General Execution Model
• Computing Race Variants
• A RT Algorithm
• Empirical Results
• Conclusion

36
S/V Graph

• All the possible SYN-sequence of a program with
  a given input can be organized into a S/V graph
• Each node n is labeled by a SYN-sequence, denoted
  by seq(n).
• An edge from node n to node n is labeled by a
  variant, var(e) of node n, indicating that
  seq(n) can be collected from a prefix-based test
  run with var(e).
• Theorem Let CP be a concurrent program. Let G
  be the S/V-graph of CP with input X. Then, G is
  strongly connected.
• Thus, reachability testing can start with a
  SYN-sequence collected from non-deterministic
  testing.

37
A Graph-Theoretic Perspective

• The goal of RT is to construct a spanning tree
  of the S/V-graph of a concurrent program with a
  given input.
• Existing RT algorithms must save the test
  history to avoid generating the same node for
  more than once.

How to generate a spanning tree without saving
the test history?
38
Example S/V Graph
39
Main Idea

• Two path constraints are identified such that
  given two arbitrary nodes n and n in G, there is
  exactly one acyclic path from n to n that
  satisfies these constraints
• A spanning tree can be constructed by enforcing
  these constraints, i.e., by only generating paths
  that satisfy these constraints.

40
Path Constraints

• Constraint 1 An edge can change the outcome of
  a race only if such change reconciles a
  difference between the source node and the target
  node.
• Constraint 2 Each edge must reconcile all the
  differences that can be reconciled between the
  source node and the target node.

41
Algorithm Reachability-Test

• ALGORITHM Reachability-Testing (CP a concurrent
  program X an input of CP)
• 1. let variants be an empty set
• 2. collect a SYN-sequence Q0 by executing CP
  with input X non-deterministically
• 3. let V0 be the special empty variant that
  contains no events
• 4. variants GenerateVariants(Q, V0)
• 5. while (variants is not empty)
• 6. withdraw a variant V from variants
• 7. collect a SYN-sequence Q by conducting a
  modified prefix-based test run with V
• 8. variants variants ? GenerateVariants(Q,
  V)

42
Function GenerateVariants

• FUNCTION GenerateVariants (SYN-sequence Q,
  Variant V)
• // sequence Q was collected during prefix-based
  testing with variant V
• // prune old sending events
• 1. for each receiving event r in V (and thus in
  Q too)
• 2. race_set(r, Q) race_set(r, Q) -
  race_set(r, V)
• // generate a subset of the race variants
  of Q
• 3. Use algorithm Construct-Race-Table to
  construct a race table with statement 2 in
• the algorithm replaced with the
  following statement
• R r ? Q race_set(r) gt 0 and
  r.color white
• 4. Derive a list variants(Q) of race variants,
  one variant from each row of the race table
• // set the colors of the receiving events
  in the variants of Q
• 5. for (each variant V' in variants (Q))
• 6. for (each receiving event r in V' whose
  sending partner was explicitly changed)
• 7. r.color black
• 8. for (each receiving event r' that
  happens before r in V')
• 9. r'.color black
• 10.
• 11.
• 12.

43
Reachability Testing

• Introduction
• A General Execution Model
• Computing Race Variants
• A RT Algorithm
• Empirical Results
• Conclusion

44
Subject Programs

• BB A solution to the bounded-buffer problem
  where the buffer is protected using either a
  selective wait, semaphores, an SU monitor, or an
  SC monitor.
• RW A solution to the readers and writers
  problem using a selective wait, semaphores, an SU
  monitor, or an SC monitor.
• DP A solution that uses an SU monitor to solve
  the dining philosophers problem
• DME A solution to the distributed mutual
  exclusion problem.

45
Empirical Results
Program Seqs Program Seqs Program Seqs
BB-select 144 RW-select 768 DP-monitorSU (3P) 30
BB-semaphore 324 RW-semaphore 21744 DP-monitorSU (4P) 624
BB-monitorSU 720 RW-monitorSU 13320 DP-monitorSU (5P) 19330
BB-monitorSC 12096 RW-monitorSC 61716 DME 4032
46
Comparison to VeriSoft (1)

• VeriSoft is a state exploration-based tool
  developed by Bell Lab
• Uses partial order reduction to avoid exercising
  redundant interleavings
• VeriSoft is very similar to RichTest in terms
  that it is stateless, i.e., it explores the state
  space by exercising all possible synchronization
  sequences.

47
Dinning Philosophers
48
Reachability Testing

• Introduction
• A General Model
• Computing Race Variants
• A RT Algorithm
• Empirical Results
• Conclusion

49
Conclusion

• RT generates test sequences dynamically, which
  is more scalable than many model-based
  approaches. Our latest tool has almost constant
  memory requirements.
• Compared to JPF and VeriSoft, which try to
  reduce the chances of generating redundant
  interleavings, RT deals with partial orders
  directly.
• While most tools for concurrent programs testing
  rely on access to the thread scheduler and are
  thus platform dependent, our RT tool is platform
  independent.

50
Future Work

• Test oracle automatically evaluate test runs
  against properties specified in some logic
  formalism
• Coverage-based RT - use coverage measures to
  selectively exercise a subset of the
  SYN-sequences
• Model extraction use RT to extract a complete
  model of the synchronization/communication
  behavior of a program
• Real-time testing add explicit time into the RT
  framework