Title: Verisim: Formal Analysis of Network Simulations
1Verisim Formal Analysis of Network Simulations
- Karthikeyan Bhargavan, Carl A. Gunter, Moonjoo
Kim, Insup Lee, Davor Obradovic, Oleg Sokolsky,
Mahesh Viswanathan - University of Pennsylvania
2Internetwork Routing Protocols
- Internetwork routing protocols enable
interoperability between physical networks. - Routing protocols for the Internet RIP, OSPF,
and BGP. - Routing protocols for packet radio AODV and DSR.
- Routing protocols and software have growing
importance and complexity.
3Current Analysis Techniques
- Rigorous mathematical models and proofs limited
by complexity. - Testing.
- Testbed expensive.
- Operational risky and inconvenient.
- Simulation.
- Performance attributes throughput, latency,
reliability, etc.
4Limitations of Performance Analysis of Simulations
- Flaws may not be detected if no expected
performance can be used for comparison. - When flaws are suspected, finer means of analysis
are useful. - Some flaws do not manifest themselves as
performance problems (e.g. most security gaps).
5Verisim
- Verisim provides support for logical analysis
of network simulations. - This talk describes its architecture and logic.
- We provide a series of experiments aimed at
assessing the approach.
6Monitoring and Checking (MaC) Framework
NS
MEDL
7Meta Event Description Language (MEDL)
- Expresses properties of traces.
- Extension of Linear Temporal Logic (LTL) with
auxiliary variables. - More expressive than LTL.
- Properties classified into (instantaneous) events
and (enduring) conditions.
S Kannan, M Kim, I Lee, O Sokolsky, M Viswanathan
98
8NS Network Simulator
Instrumented Protocol Code P C
Protocol Agents
P
P
P
Scenario
Configuration Parameters OTcl
Network Model
N
N
N
Topology OTcl
Traffic Agents
src/sink
src/sink
src/sink
Traffic Model OTcl
NS Trace
VINT Project
9Verisim
Properties MEDL
Instrumented Protocol C
Checker
NS
Trace
Metatrace
Scenario OTcl
10Ad Hoc Networks
- Routing for a wireless internetwork without the
aid of a central base station. - Connections are low-bandwidth, lossy, and highly
transient. - Unique routing assumptions
- Most routes are seldom used.
- Bandwidth must be protected.
11Illustration Part 1 of 2
Movement
Routing
12Illustration Part 2 of 2
New Routing
13AODV Protocol
- If a node S needs a route to a destination D and
does not have one, it floods a route-request
(RREQ) packet through the network. - Each recipient R of this RREQ keeps a return
pointer. - R broadcasts the request to its neighbors if it
is not D and does not have a route to D. - If R is D, or has a route to D, it responds with
a route-reply (RREP) packet using the return
pointers for S.
Perkins and Royer 99
14Loop Freedom in AODV
- Routing loops are undesirable.
- AODV uses sequence numbers to indicate freshness
of link information. - Key Invariant If next(n) n, then
- seqno(n) ? seqno(n), and
- if seqno(n) seqno(n), then hops(n) gt hops(n).
- The invariant ensures that there are no loops.
15Outline of Experiment
- Run a scenario of modest complexity.
- Analyze it in Verisim using a list of 9
properties of AODV expressed in MEDL. - First pass Repair First Bug (RFB).
- Second pass tune the MEDL formulas to avoid
rerunning the simulation discovering bugs in the
metatrace. - We instrumented simulation code for AODVv0
supplied by the CMU Monarch Project.
16Experiment Scenario
1
2
3
4
5
17Experiment Scenario
1
2
3
4
5
18Experiment Scenario
1
2
3
4
5
19Experiment Scenario
1
2
3
5
4
20Experiment Scenario
1
5
2
3
4
21Sample MEDL Alarm
alarm LoopInvatnxtdst sendroutatdst
when ((at?nxt) ? (at?dst) ? (nxt?dst) ?
(obs_nexthopatdst nxt) ?
((obs_seqnoatdst gt obs_seqnonxtdst) ?
((obs_seqnoatdst obs_seqnonxtdst)
? (obs_hopcontatdst lt
obs_hopcntatdst))))
This is the negation of the fundamental invariant
ensuring no loops in AODV If the next hop from
node at toward destination dst is node nxt then
the sequence number (for dst) of at is less than
or equal to that of nxt or they are equal and the
hop count (for dest) of at is less than or equal
to that of nxt.
22Repair First Bug Experiment
- Let ? be the MEDL property set.
- Run the simulation to get a trace T.
- Run the checker to get a metatrace T?.
- Repair the first bug in the metatrace to get new
protocol code. - Rerun the simulation with the new protocol code
to get a new trace U. - Rerun the checker to get a new metatrace U?.
- Continue until an empty metatrace is obtained.
23RFB Experiment Statistics
Meta- trace Dest Rep Detect RErr Node Rep Loop Env Total Alarms
T? 4 54 38 43 220
U? 0 54 38 43 216
V? 0 48 39 44 206
W? 0 0 0 0 1
24Tuning Experiment Statistics
Meta- trace Dest Rep Detect RErr Node Rep Loop Env Total Alarms
T? 4 54 38 43 220
T? 0 54 38 43 216
T? 0 0 38 50 166
T? 0 0 21 0 21
No new simulation traces.
25Analysis of Off-The-Shelf (OTS) Simulations
- The prior experiment involved fewer than 10,000
events, and we designed it to exercise key
scenarios. - Can we do useful analysis with OTS performance
simulations? - Aim Verisim can be added with small
modifications, run alongside OTS simulations, and
find logical failures in a practical period of
time.
26Experiment
- Took largest available OTS simulation for AODV
and ran it to create a trace. - Simulation from Monarch uses 50 mobile nodes on
1500x300m grid moving 20 m/s. - 5220 seconds (1.5 hours) to complete simulation.
- 6,446,316 events.
- NaĂŻve effort ran MEDL with MonSeqNo test on all
nodes (2500 relations) using 550Mhz dual
processor machine with 1GB of memory. - Aborted the experiment after 4 days estimate 100
days to complete this analysis.
27Optimizations For Network Routing Simulations
- Population Abstraction test the property for a
subset of the routers. - Packet Type Abstraction prune the trace to
include only relevant events.
28Sample Experiment Results for MonSeqNo
- Population Abstraction for 5 nodes.
- Trace size 6,446,316 events
- Property size 14,543 bytes
- Time 51,054 seconds
- Rate 0.54 micro seconds per event per property.
- Population and Packet Type Abstractions.
- Trace size 6812 events
- Property size 14,543 bytes
- Time 51 seconds (found failures)
- Rate 0.51 micro seconds per event per property.
29Sample Experiment Results for LoopInv
- Population Abstraction for 5 nodes.
- Trace size 69,411 events
- Property size 75,508 bytes
- Time 8064 seconds
- Rate 1.54 micro seconds per event per property.
- Population and Packet Type Abstractions.
- Trace size 48,735 events
- Property size 75,508 bytes
- Time 5912 seconds (found failures)
- Rate 1.61 micro seconds per event per property.
30Related Work
- Communication-based test generation systems.
STRESS and Verisoft. - Test oracles based on formulas or formal
operational specifications. GIL and TETRA. - Formal instance verification of routing
protocols. SPIN/PITHIA for PNNI. - Simulations based on logical specifications.
MTSim and Maude. - Network monitoring systems. Emerald.
31Summary
- Verisim integrates simulation and logical trace
analysis. - The combination provides a more flexible approach
to analyzing network simulations for safety
properties. - It is able to find bugs in existing simulator
studies. - Its flexibility can be exploited to improve
debugging turn-around time.