Specification-Based Error Localization

1
Specification-Based Error Localization

• Brian Demsky
• Martin Rinard
• Laboratory for Computer Science
• Massachusetts Institute of Technology

2
Problem
Crash or Unexpected Result
Execution with Broken Data Structure
Error Introduced

• Have to trace symptom back to cause
• Corruption may not cause visible error in test
  suite

3
Problem
Crash or Unexpected Result
Execution with Broken Data Structure
Error Introduced

• Solution discover bugs when
• they corrupt data
• not when effect becomes visible
• Perform frequent consistency checks
• Bug localized between
• first unsuccessful check and
• last successful check

4
Architecture
Abstract Model
Concrete Data Structure
Model DefinitionRules
o1
nodes
o1
value
next
values
o1
o1
value
Consistency Constraints
5
Architecture Rationale

• Why use the abstract model?
• Model construction separates objects into sets
• Reachability properties
• Field values
• Different constraints for objects in different
  sets
• Appropriate division of complexity
• Data structure representation complexity
  encapsulated in model construction rules
• Consistency property complexity encapsulated in
  (clean, uniform) model constraint language

6
Simplified Freeciv Example
Terrain Grid

• tile gridEDGEEDGE
• structure tile
• int terrain
• city city
• structure city
• int population

O Ocean
P
O
M
M
P Plain
O
O
M
P
M Mountain
P
O
M
M
City Structures
P
P
M
P
7
Sets and Relations in Model

• Sets of objects
• set TILE of tilegrid
• set CITY of city
• Relations between objects values of object
  fields, referencing relationships between objects
• relation CITYMAP TILE -gt CITY
• relation TERRAIN TILE -gt integer

8
Model Translation

• Bits translated to sets and relations in abstract
  model using statements of the form
• Quantifiers, Condition ? Inclusion Constraint
• for x in 0..EDGEEDGE, true ? gridx in TILE
• for t in TILE, true ? ?t,t.terrain? in TERRAIN
• for t in TILE, !t.city NULL ? ?t,t.city? in
  CITYMAP
• for t in TILE, !t.cityNULL ? t.city in CITY

9
Model in Example
grid0 grid1 grid2 grid3
terrain 1 2 3 4
city NULL NULL
population 10,000
Tiles
Cities
grid0 grid1 grid2 grid3
city
1 2 3 4
10
Consistency Properties

• Quantifiers, Body
• Body is first-order property of basic
  propositions
• Inequality constraints on numeric fields
• Cardinality constraints on sizes of sets
• Referencing relationships for each object
• Set and relation inclusion constraints
• Example
• for t in TILE, MIN lt t.TERRAIN and
  t.TERRAINltMAX
• for c in CITY, size(CITYMAP.c)1
• for c in CITY, !(CITYMAP.c).TERRAINOCEAN

11
Consistency Violations

• Evaluate consistency properties, find violations
• for c in CITY, size(CITYMAP.c)1

Tiles
Cities
grid0 grid1 grid2 grid3
city
1 2 3 4
12
Slide about checks

• TODO

13
Optimized Implementation

• Compilation (4.7x speedup)
• Fixed point elimination (210x speedup)
• Evaluate model definition rules using simple
  traversal
• Relation construction elimination (500x speedup)
• Evaluate uses of relations directly on data
  structures
• Set construction elimination (3900x speedup)
• Evaluate constraints while traversing data
  structures
• Bottom line
• Interpreted version X times slower than
  uninstrumented
• Optimized version Y times slower than
  uninstrumented

14
Freeciv Case Study

• Multiplayer Client/Server based online game
• Available at www.freeciv.org
• Case study looked at the server
• Server contains 73,000 lines of code
• Added 750 instrumented sites
• 20,000 consistency checks performed in our sample
  execution

15
Case Study

• Created three buggy version of Freeciv
• Two groups of three developers
• One used conventional tools
• One used specification-based consistency checking
• Each participant was asked to spend at least one
  hour on each version
• Both populations given a pre-instrumented version
  of Freeciv

16
Consistency Properties

• Map exists
• size(MAP)1
• Grid of tiles exists
• size(GRID)1
• Tiles have valid terrain values
• for t in TILE, MIN lt t.TERRAIN and
  t.TERRAINltMAX
• Cities are not in the ocean
• for c in CITY, !(CITYMAP.c).TERRAINOCEAN
• Each city has exactly one reference from the grid
• for c in CITY, size(CITYMAP.c)1

17
Bugs Introduced

• Actual errors in buggy versions
• First error creates invalid terrain values
  (violates valid terrain property)
• Second causes two tiles to refer to the same city
  (violates single reference property)
• Third causes a city to be placed on ocean
  (violates cities not in ocean property)

18
Results

• User study shows benefit from approach
• With tool
• All developers found and fixed all bugs
• Mean of 11 minutes required
• Without tool
• Three developers found total of one bug (out of
  nine developer/bug combinations)
• Others spent hour debugging (unsuccessfully)

19
Repair for Deployed Systems

• Consistency specifications for repair
• Input inconsistent data structure
• Output consistent data structure
• Technique enables programs to recover from data
  structure corruption
• And continue to execute successfully
• OOPSLA 03 paper describes this technique
• Full reference here

20
Related Work

• Specification languages such as UML or Alloy
• Specification-based testing
• Korat (Boyapati et. al. ISSTA 2002)
• Testera (Marinov and Khurshid)
• Eiffel (Meyer)
• Invariant inference and checking
• Daikon (Ernst et. al.)
• DIDUCE (Hangal and Lam)
• Carrot (Pytlik et. Al.)

21
Conclusion

• Consistency checking to localize data structure
  corruption bugs
• Good experimental results
• With checker, bugs fixed in minutes
• Without checker, bugs not fixed in an hour
• Optimizations for good performance
• Data structure repair

22
Consistency Properties

• The FAT blocks exist
• FAT contains valid values only
• -1 terminates FAT streams
• -2 indicates free blocks
• Valid disk block index next block in stream
• FAT streams properly terminated
• Free blocks properly marked
• Streams contain valid blocks only
• Streams do not share blocks

23
Pointers

• Sets in model can include
• Primitive types (int, char, )
• Structs (identified by pointer to struct)
• Standard linked list example
• struct node int value node next
• set nodes of node
• relation next node, node
• for n in nodes, true ? n.next in nodes
• for n in nodes, true ? ?n,n.next? in next

24
What About Corrupted Pointers?

• System only allows valid structs in model
• struct must be completely in valid memory
• one struct may be nested inside another struct
  (but must agree on memory format)
• If encounter invalid or null pointer, the
  (invalid) struct does not appear in model
• Implementation must track operations that affect
  valid regions of address space
• malloc, free
• mmap, munmap

25
CTAS in Action
FAST at DFW TRACON
TMA at Fort Worth Center
26
Usage Scenarios

• Reduced development effort
• Invest less effort in finding and fixing bugs
• Rely on repair to deliver reliable system
• Afraid to fix bug
• Cheap insurance policy
• No good quantitative justification
• But repair seems like a good idea

27
Current Work

• Support for recursive data structures
• Support for adding or removing individual
  elements
• Support for acyclicity constraints
• Repairing back-links

28
Issues

• Unclear relationship between repaired bits and
  bits from correct execution of program
• Identifying results involving repaired data
• Characterizing likely errors
• Data races in multithreaded programs
• Failure to update correlated data structures
• Caching inconsistencies
• Unanticipated failures/exit points
• Constraint language expressivity
• Coverage of desired properties
• Limitations from acyclicity requirement
• When to test for consistency and repair

29
What About Corrupted Pointers?

• Sets may contain pointers to structs
• System only allows valid structs in model
• struct must be completely in valid memory
• one struct may be nested inside another struct
  (but must agree on memory format)

Invalid Struct
Valid Struct
Valid Structs
Valid Memory
Invalid Memory
30
Interesting Nuggets

• Small specifications
• Global invariant advantages