Cognitive Model Validation

1
Cognitive Model Validation

• David Wood
• Denise Dykstra
• Supervisor Dr. Mili

2
Outline

• Background
• Issues
• Approach
• Specification Methodology
• Tool Support

3
Outline

• Background
• Issues
• Approach
• Specification Methodology
• Tool Support

4
Cognitive Science

• Subset of AI
• Interdisciplinary
• Cognitive Modeling

5
Putting It Together
6
Cognitive Modeling

• Area of Cognitive Science
• Used Academically
• Moving Into Applied Areas

7
Industrial Applications

• Safety Critical
• Military
• Entertainment
• Education Training

8
Outline

• Background
• Issues
• Approach
• Specification Methodology
• Tool Support

9
Correctness

• Correct execution is essential
• Verification Proving or disproving the
  correctness of a system using mathematical
  methods
• Validation Determining the degree to which a
  model is an accurate representation of the real
  world

10
Specification
11
Challenges Inhibitors

• Ill-structured
• Reactive
• Lengthy
• Tedious Error Prone

12
Reasons For Our Project

• Correct Execution Is Essential
• Little Has Been Done
• Formal Specifications Are Necessary
• Formal Specifications Require Support
• Goals
• Formulate A Specification Methodology For
  Cognitive Modeling
• Produce A Tool To Aid In Specification Validation

13
Outline

• Background
• Issues
• Approach
• Specification Methodology
• Tool Support

14
Necessities

• Specification Language
• Patterns Conflicts
• Support For Writing Specifications
• CMSA (Cognitive Model Specification Analysis)

15
Outline

• Background
• Issues
• Approach
• Specification Methodology
• Tool Support

16
Specification Language

• Handles Sequences of Events
• Characterizes Sequences of Events
• Uses Pre Post Conditions
• Applicable To Many Situations

17
Patterns

• Once A Pattern Is Started It Must Be Completed

18
Conflict

• A Conflict May Never Complete

The robot may not put a VHS tape into the DVD
player.
19
Patters Conflicts

• Formal
• Simple
• Visual
• Modular
• Additive

20
Outline

• Background
• Issues
• Approach
• Specification Methodology
• Tool Support

21
Tool Support is Beneficial

• Support for the specification group during the
  specification phase
• Support for the verification group during the
  specification generation
• Support during the
• verification and
• validation phase

22
Specification Phase
23
Purpose

• Positive Traces
• Negative Traces
• Syntactic and Semantic Check
• Redundancy
• Incoherence
• Inconsistency

24
Scenario 1

• Positive Trace

P1 Once the robot goes outside, it must
eventually drop the box.
S1 ltOpen, Doorgt, ltGoto, Outsidegt, ltDrop,
Boxgt S2 ltGoto, Outsidegt, ltGoto, Insidegt, ltDrop,
Boxgt S3 ltPickup, Keygt, ltPickup, Boxgt
25
Scenario 2

• Assume the specifier does not agree with S3. The
  specifier adds C1 as follows

C1 The robot can only hold one thing at a time.
S1 ltPickup, Keygt, ltDrop, Keygt, ltPickup, Keygt,
ltDrop, Keygt S2 ltPickup, Boxgt, ltGoto, Outside gt,
ltDrop, Boxgt, ltGoto, Insidegt, ltPickup, Keygt S3
ltPickup, Keygt, lt Drop, Keygt, ltPickup, Keygt
26
Scenario 3

• Negative Trace

P1 Once the robot goes outside, it must
eventually drop the box.
N1 lt Goto, Outsidegt, ltDrop, Boxgt, ltGoto,
Insidegt, lt Goto, Outsidegt
27
Scenario 4

• Pre-Processing
• Incoherence

P1 Once the robot goes outside, it must
eventually go outside
Incoherence P1 triggers itself in its third
transition ltGoto, Outsidegt
28
Scenario 5

• Pre-Processing
• Inconsistency

P1 Once the robot goes outside, it must
eventually go to the car
C1 If the robot goes outside, It must not go to
the car
Inconsistent Specification P1 conflicts with C1
29
Scenario 6

• Incoherence
• Not found with Pre-Process

P1 Once the robot goes outside, it must
eventually drop a box
P2 Once the robot drops a box, it must
eventually go inside
P3 Once the robot goes inside, it must
eventually go outside
30
Scenario 6 cont.

• P1 starts P2
• P2 starts P3
• P3 starts P1
• Error! No sentences of length less than N found.

31
Heuristics and Strategies
32
Common Mistake 1

• Common Mistake 1
• A then B instead of A then B right away

P1 Once the robot goes outside, it must
eventually drop the box.
S1 ltOpen, Doorgt, ltGoto, Outsidegt, ltDrop,
Boxgt S2 ltGoto, Outsidegt, ltGoto, Insidegt, ltDrop,
Boxgt S3 ltPickup, Keygt, ltPickup, Boxgt
33
Common Mistake 2

• Common Mistake 2
• A then B once instead of A then multiple Bs

P1 Once the robot goes outside, it must
eventually drop the box.
N1 lt Goto, Outsidegt, ltDrop, Boxgt, ltGoto,
Insidegt, lt Goto, Outsidegt
34
Common Mistake 3

• Common Mistake 3
• P1 is incoherent

P1 Once the robot goes outside, it must
eventually go outside
Incoherence P1 triggers itself in its third
transition ltGoto, Outsidegt
35
Common Mistake 4

• Common Mistake 4
• P1 and C1 Conflict

P1 Once the robot goes outside, it must
eventually go to the car
C1 If the robot goes outside, It must not go to
the car
Inconsistent Specification P1 conflicts with C1
36
Strategy

• CMSA does the following for the specification
• Specification is pre-processed
• Positive traces are produced
• Negative traces are produced
• Traces generated according to strategies
• Strategy 1
• Short-random
• Strategy 2
• Overlap

37
Short-Random

• Pros
• Short
• Easy to read and understand
• Gives a quick insight into the specification
• Cons
• Not very specific
• Not intelligent

38
Overlap

• Pros
• Specific to over lapping patterns
• Short
• Easy to understand
• Cons
• Only gives insight about overlapping patterns

39
Screen Shots
40
Assumptions

• Regular Grammar
• Subset of CFG
• Finite Automata
• Alphabet
• ltSubject, Action, Objectgt

41
Summary

• Specification for CM is new
• Specification is needed
• Correctness is essential
• Tool support is very helpful

42
Future Work

• More Sentence Generators
• More Strategies
• Make Alphabet More Versatile
• Pre-Processing

43
Questions
44
References

• http//www.ai.mit.edu/people/jcma/papers/1988-air/
  node7.html
• http//www.wellesley.edu/Psychology/Cogsci/htmlpag
  es/whatiscogsci.html
• http//sitemaker.umich.edu/soar
• http//en.wikipedia.org/wiki/Cognitive_science
• http//plato.stanford.edu/entries/cognitive-scienc
  e/
• Mili, F., MacKlem, A., Adams, C. Dungrani, S.
  Patterns and Conflicts for the Specification and
  Verification of Cognitive Models, 2004.
• Corcoran, P., Gilmer, G., Ziemkiewicz, C. Mili,
  F., Cognitive Model Validation, 2003.