Columbia University G5043 Cognitive Science and Medical Informatics

1
Columbia UniversityG5043 Cognitive Science and
Medical Informatics

• Towards a cognitive theory of information
• Retrieval
• by
• Alistair Sutcliffe, Mark Ennis
• May 5, 2003
• Class Presentation
• Roman Trakhtenberg

2
Intro

• A framework for constructing a cognitive model
  of users information searching behaviour
• Proposed a taxonomy of components for
• process models of the information seeking task
• Information need types
• Knowledge sources

3
Cognitive theory of information searching
In addition to the proposed framework defined
strategies of search and rules predicting user
behavior based on - information need types - IR
system - user knowledge
4
Evaluation of the theory

• Using Claims analysis
• Based on empirical observations of user
• behavior
• A cognitive walkthroughs of an IR session
• Results Expert strategies at different stages
  of a task are identified,
• actual user behavior is poorly predicted

5
Future Possibilities

• Employing this theoretical model as a tutorial
  advisor for IR
• As an evaluation method for IR systems
• Have some potential in IR and HCI

6
Summary

• Proposed a synthesis theory of IR
• Employs previous cognitive models
• Builds on experimental evidence of
• user information seeking behaviour
• The study sets out a modelling framework for
  studying IR

7
Major components

• Process Model of Information Searching and KR
  necessary for the task
• Contains activities in terms of strategies tasks,
  and actions
• Strategies plans and methods expertise
• Mistakes novices sub-optimal behavior

8
Activities in Retrieval

• Problem Identification
• Need Articulation
• Query formulation
• Results evaluation

9
Problem Identification

• Formulates User Goals or Information Needs
• Information needs describe Query in terms
• of complexity, intended target and specificity
  of expression
• Initial search strategy is based on need type
  definition

10
Need articulation

• Natural language expressions of a need
• Expressions parsed into Knowledge structures
• Knowledge structures should contain
• high level concepts and semantic propositions

11
Query Formulation

• Transformation of conceptual needs into
• keywords
• Query syntax employed by computer system

12
Results Evaluation

• Comparison of retrieved results against
  information needs and adaptation of search
  direction accordingly
• Evaluation strategies Volume, Relevance,
• Precision parameters of the result set
• Iterative cycle is formed, different routes based
  on strategy selections

13
Strategies

• Represent users information searching skill
• 7 Strategy rule sets are used within the process
  model for problem identification
• need articulation query formulation
• results evaluation query reformulation and
• process strategies

14
Knowledge Representation

• 2 Levels of KR in the model
• Knowledge Sources stereotype models
• in user knowledge influence strategy choices
• Query instance models domain specific
  information, propositional content of the query
  (LTM, retrieved documents, system thesaurus)

15
Knowledge sources

• Domain knowledge
• Facts, concepts, relationships in the domain
• Device Knowledge
• Functions provided by the Task supported
  Facilities(UI),
• Knowledge of the syntax and semantics of the
  sytems retrieval language
• Information Resource Knowledge Knowledge of
  Searchable Databases
• IR Knowledge Search Strategies (expert behaviour)

16
Query Instance Models

• 3 Semantic Primitives Concepts, Terms,
  Relationships Used to construct schema of an
  evolving information need
• Concepts high-level aggregations of info,
  relating to goal
• Terms search target in a device query lang
• Relationships Associate Terms and Concepts in
  knowledge structures
• Queries are formed by Extracting Elements from
• Query Instance Models

17
Correspondence Rules

• Integrate KR and behaviour
• Predict Process Model behaviour according
• to users knowledge
• system facilities
• information needs

18
Testing the theory

• Experiments of user information seeking with the
  MEDLINE database
• Observationsto assess the theorys claims
• Scenariosuse the process model to generate
  explanations of behaviour and performance

19
Scenario Analysis

• the theory does explain strategy selection
  during information searching Only at a general
  level
• The theory cannot account for hybrid
• strategies and only partially for sub-optimal
• performance

20
The framework

• Whats New?
• Extends previous models with correspondence rules
  and strategies
• to account for behaviour
• Whats missing? No account for the development of
  the users conceptual model of a query

21
Predictions

• Poor
• Individual user behaviour is beyond the scope of
  the model, theoretical difficulties
• Individual User Searching Strategies may be
  inconsistent
• Strategies from memory
• Different search styles for different IR systems
• More can be achieved for modeling generic groups
  of users

22
Applications

• Tutoring, Intelligent Help system for IR ( using
  process models and rules sets)
• Walkthrough evaluation method for IR
• Predictions of expert strategies for a particular
  search need (Expert System Advisor using human
  intermediary knowledge)
• Modeling Novice Users pointing out mistakes
• expert strategy and system facility suggestions
  for
• each stage of Information Retrieval

23
Previous work

• Belkins Anomalous Knowledge State(ASK)
• Perception of needs change during search with
  feedback
• High-level goals are achieved by
  context-specific search
• Ingwersen modeling IR polyrepresentation theory
  ? need for cognitive theories
• Bates target evolves with feedback
• Kuhlthau six-stage process model with high-level
  goalsInitiation, problem definition, source
  selection, formulating queries, examining
  results, extracting useful information

24
Whats missing?

• No account of users IR behaviour
• No explanation for optimal performance
• No Common theory
• Cognitive theories need to account for
• payoffs, costs of searching, resources
  available, amount of information sought,
  characteristics of data, conflicts among
  documents
• user motivation, importance of information need,
  their impact on search duration and effort
  employed

25
HCI perspective

• Bridging models ranging from human information
  processing architectures (i.e, EPIC) to
  cognitive models of user system interaction
  (computational models and software engineering/
  interaction models)
• Small scale scenarios of interaction obtain
  precise predictions of behaviour
• IR implies broadly based models

26
Proposed approach

• Model cognitive Tasks at a High Level of
  granularity and sacrifice precision
• Based on models of action, reasoning, and levels
  of user knowledge
• Elaboration of action models to describe goal
  directed tasks ( realized in walkthrough methods)

27
Alternative to Modeling

• Task Artefact Cycle of Caroll et al.
• Focus on designed Artefacts
• Embedding HCI theories with Claims Analysis to
  extract knowledge why a particular design should
  have Good Usability
• Doesnt give a comprehensive account of
• psychological process of interaction

28
Modeling Information Searchingstarting points

• Info Search a range of behaviours (goal-directed
  ??exploratory browsing)
• Need to account for Relational and Bibliographic
  DBs
• Goal-directed Info search (specific target)
• Exploratory info search (intention to explore)
• Embedded info search(goal is motivated by
  external task)

29
(No Transcript)
30
Dual Agent Model

• Two agents User and IR system
• Wish to model expert behaviour that incorporates
  intermediarys skills
• Helps evaluate aspects best carried out by
• people or computer systems
• Compare with triple agent model( user, expert
  intermediary, IR system)

31
KR and Behavior Integration

• Use Correspondence Rules
• Predict Process Models behavior
• Using Users knowledge and an Info need
• Large Number of Rules to account for strategies
  and activities
• Solution Modular Design to minimize uncalled
  rules interaction and ease validation

32
Computational Cognitive Architecture

• I.e. SOAR hybrid agent architecture
• (1) Percepts ? Reflexes
• (2) Deliberation ? Knowledge
• (3) Compilation (2) ? (1)
• Difficult to program, complex environment

33
Computational Architecture II

• Authors choice COGENT
• No Problem Solving /Learning
• Models Working Memory, Selective Attention,
  Spreading activation in LTM
• Allows Cognitive Models to be Constructed
• Implemented with GTK, Prolog
• Graphical IDE in Windows/Unix/MacOS

34
Process Model Development

• Behaviour Six rule-sets
• 3 Activity related
• 2 Global process termination and execution of
  other strategy selection rules
• 1 system facility consultation rule-set
• Articulation rules ?Query formation rules ?
  Evaluation rules ? Query reformulation rules ?
  System Consultation and Context rules (need for
  system facilities or search termination )
• Cognitive Resources Rules to be employed

35
(No Transcript)
36
Evaluation

• Medline Searches
• Think aloud protocols/ performance data
• Explanation example
• Premature Termination Problem
• Test theory by backward chaining from the
  observed situation users are satisfied with
  fewer relevant results than available

37
Explanations

• Motivation explanation(motivation low, results
  few ? terminate)
• Poor domain knowledge (results few, domain
  knowledge low, accept sample)
• One poor explanation (device knowledge low,
  results few, accept sample)
• From interviews 2 confirmed, the last is
  doubtful users compensate by more search
  iterations

38
Scenario Analysis

• Another method of validation
• Describe IR scenario
• Account for the development of a search
• Trace through the process model and strategy
  rules
• Theory does explain the strategy selection
• only at a general level ( none for hybrid
  strategies, partially for sub-optimal
  performance)

39
Utilization Interface Design

• Three possible applications
• Tutoring/ Intelligent Help System
• Can be developed from the process model and the
  rules set
• Walkthrough Evaluation used to test IR system
• Decision Support System (using the process model)
  for configurable IR systems or an intelligent
  adaptable UI