SABER

1
SABER

• Semi-Automatic Building and Evaluation of
  Relational Models
• Team lead Jenni Thompson
• Team members
• Jessica Bradley Jerry Buck
• Rory Smith
• Faculty advisor Prof. Daniel Barbara
• Project type OR

2
SABER TASK STATUS
Adapt APOLLO Generate Survey APOLLO
Prototype Research KE Tech Techniques to
Include Historical Dist Personality
Model Weighting SMEs Prototype Dev User
Interface Research VV Apply VV Field
Test Lessons Learned
6 Apr
2 Mar
9 Mar
4 May
13 Apr
20 Apr
27 Apr
16 Mar
23 Feb
23 Mar
30 Mar
12 May
3
KE Tasks Summary and Status

• Conduct a comprehensive literature search
• Complete
• Analyze selected techniques
• Analysis complete
• Currently reviewing incorporation into
  methodology
• Propose Technique for use in SABER
• Complete
• Functionality of APOLLO mapped to SABER
• Develop Technique(s)
• Questionnaire and software development tasks
  (later slides)
• Incorporate into Methodology
• In progress

4
KE Selected Technique Review

• Review of the advantages and disadvantages of
  questionnaire direct elicitation techniques
• Minimize/eliminate any biases inherent in the
  technique or in our methodology
• Understand the SMEs language
• Understand the type and quality of the
  information we will receive
• Understand what the answers mean
• Conflict resolution ( Aggregation of SME Input)
• Combination of
• Iterated feedback
• Bayesian Model Averaging

5
Prototype Development Task (Software)

• Current task, develop the algorithm that defines
  the connection between the questionnaire results
  and the final Bayes Net
• Mathematical mapping from questions of modeler to
  resulting CPT
• Algorithm is developed (next slide)

6
Prototype Development Task (Questionnaire)

• Qualitative Answers into CPTs
• Based on the technique of the Analytical
  Hierarchy Process (AHP)
• Steps
• Initial Questions Users choose the nodes to be
  included in the net from a predetermined superset
  of nodes
• Next Set of Questions Down Select from a
  supersets of states within each of the selected
  nodes
• Next Set of Questions Pair-wise comparisons of
  the states to determine each standalone node
  (AHP)
• Final Set of Questions Pair-wise comparison of
  the states between parent and child nodes (AHP)
• RESULT BAYES NETS with CPTs

7
Prototype Development Task (Software) (Cont)

• Current Status now that algorithm is developed,
  begin implementation of this mathematical mapping
• Will require some adaptation of the original
  questionnaire but only minor
• Next Step tie this to the visualization
  techniques we have been researching (actual Bayes
  Nets and Digital Dashboard amongst others)

8
Techniques for Translating SME Responses into
Probabilities (Weighting SMEs)

• Researched multiple techniques including
• Bradley-Terry
• Analytical Hierarchy Process
• Paired Comparison
• Swing Weighting

9
Transforming Qualitative Answers Into
Quantitative Probabilities (Weighting SMEs)

• We have decided to utilize Paired Comparison
  Technique to elicit useful knowledge from SMEs to
  rank nodes and states within the network
• Once the Paired Comparison Technique has been
  applied, we will use the Analytical Hierarchy
  Process (AHP) to convert the SME responses into
  probabilities

10
Aggregation of SME Input

• In a group with multiple SMEs, there will be
  varying opinions
• Continued iterations of feedback will improve
  central tendency
• There will still be variability.
• Must define a method to determine reliability of
  SME opinion and weight various inputs differently
• Will employ Bayesian Model Averaging, a method
  similar to multiple hypothesis testing to weight
  and combine multiple expert opinions

11
Historical Data Modeling

• Incorporated sample historical data to develop
  prior likelihood distributions for enemy reaction
  to a given friendly action
• Used second set of sample data to update prior
  likelihoods and develop posterior probabilities
  that will change as new data is gathered

• Using Dirichlet distribution to model enemy
  reaction to possible friendly decisions.

12
Testing SABER

• We discussed testing techniques for our system,
  and decided to apply the APOLLO technique
  manually and compare the resulting Bayes Net
  against a net constructed by using our SABER
  technique for a sample problem
• Buying a House in the DC area
• This technique endorsed by Dr Barbara, Dr Laskey
  and Dr Chang
• We have begun development of this questionnaire

13
What If? - Experimental Design

• Researched ways to validate the resulting
  Bayesian Network model with What If experiments
• Determine the sensitivity of the net to various
  evidence applies determine if its reasonable
• Estimate a full-factorial design will require
  over 30 million runs
• Strategy-
• Run tests on main effects of each evidence node
• Run tests of the factorial combinations of
  evidence within functional frames of the Bayes Net

14
V V Next Steps

• Finish developing a questionnaire that provides
  validation for the system as a whole
• Feedback from SMEs
• Develop a sample data set
• Compare Bayesian Network Outcome to Known Outcome
• Used to validate the Bayesian Network