BPM Seminar understandability of process models

1
BPM Seminarunderstandability of process models

• A users perspective on learning

2
Topics

1. Context of the research
2. Theoretical Background
3. Theoretical Design Conceptualisation
4. Measurement
5. Questions?

3
1. Context of the research

• Goal
• Defining understanding as a result of learning,
  how do differences in user characteristics
  contribute to differences in understanding
  process models?

4
An example
5
Kick-off

• Curtis et al. (1992)
• Goal
• facilitate human understanding and
  communication
• Sub-goal
• represent processes in forms understandable by
  humans

6
Gap-analysis

• Independent variables
• The effect of
• task complexity
• technology/process model characteristics
• on performance

7
Visualisation
8
Gap-analysis II

• Understandability defined as an intrinsic
  property of a process model
• Complexity, learnability, usability, etc.
• Pragmatic quality

9
Why is this important?

• Practically
• ) helps representing processes in forms
  understandable by humans
• ) additional insights allow for more effective
  training
• Academically
• ) effects of users often averaged out by
  assuming homogeneity (or x vs. y)
• ) Maturity of the discipline will benefit from
  an interdisciplinary approach

10
2. Theoretical background

• To learn is to be human
• (Goward)

11
Theory of Cognitive Load

• Define information integration as a learning
  process
• Control factors of
• Content
• Content Presentation

12
An example
13
Learning Conceptualised
14
Revisions

• Controlled Learning Context
• Limited Mutuality

15
Controlling the Learning Context

• HOW?

16
3. Theoretical Design Conceptualisation
a Cognitive Perspective onLearning
17
Structure

• Gap Analysis
• Introduction of a framework
• Integration into learning theory

18
Prior to Learning
19
User CharacteristicsGap analysis
distal variables affective variables psychosocial variables skills expertise
Agarwal et al., 1999 Cognitive Fit
Aranda et al., 2007 Domain Expertise Language Expertise
Bandara et al., 2005 User Competence
Chiew Wang, 2004 Syntactic Knowledge Semantic Knowledge Schematic Knowledge Strategic Knowledge
Maes Poels, 2007 Perceived Ease of Use Perceived Usefulness Perceived Semantic Quality User Satisfaction
Mendling et al., 2007 Experience Domain Expertise
Patig, 2008 Age Gender Experience Domain Knowledge
Reijers Mendling, 2008 Education Job type Domain Knowledge Company Experience Field Experience
20
Input User Characteristics

• Identification of a framework which
  conceptualises user characteristics in an
  integrative manner
• Problem-solving is inherent to human nature
• Usage of goal-setting behavioural theory

21
Goal-setting Behavioural Theory
22
Eight Variables
23
During Learning
24
Knowledge ConstructionGap analysis

• Static Approach hence limited availability
• Cardoso et al. (2006)
• short- and long-term memory
• chunking and tracing
• Hungerford et al. (2004)
• Task Planning
• Others, e.g. Central vs. Peripheral, Queues

25
Throughput Knowledge Construction

• Identification of a framework that describes
  Knowledge Creation
• Mandviwalla Hovav (1995)
• Motivational process

26
Motivational Hub (Locke, 1991)
27
Three Variables
28
After Learning
29
Learning OutcomeGap analysis
Cognitive Performance Behavioural Performance Affective Performance
Agarwal Karahanna, 2000 Behavioural Intention to Use
Aranda et al., 2007 Correctness of understanding Time to completion Confidence Perceived Difficulty
Goodhue et al., 2000 Time to completion User Evaluation of consistency User Evaluation of training
Igbaria et al., 1997 System Usage
Mendling et al., 2007 Number of questions correct Perceived understandability
Recker Dreiling, 2007 Number of questions correct Cloze Recall test Problem Solving task Time to task completion Ease of Understanding
Venkatesh Bala, 2008 Use Behaviour
30
Output Learning Outcome

• Identification of a framework which
  distinguishes between different types of
  understanding

31
Motivational Hub (Locke, 1991)
Test Performance Test Performance
Learning Outcome Cognitive Description Retention Transfer
No learning No knowledge Poor Poor
Rote learning Fragmented knowledge Good Poor
Meaningful learning Integrated knowledge Good Good
32
Three Variables
33
Time Out
34
Variables Explained

• Eight Presage
• Three Process
• Three Product

35
User Characteristics
Categories Variables Sub-variables
Distal Variables Distal Variables Demographics Culture Attitude towards targets Personality traits Exposure to media Other individual difference variables
Affective Variables Attitude Positive Anticipated Emotions Negative Anticipated Emotions
Psychosocial Variables Subjective Norm Perceived Behavioural Control Self-efficacy
Skills and Expertise Skills and Expertise Syntactic Skills Semantic Skills Pragmatic Skills Expertise
36
Knowledge Construction
Factors Variables Sub-variables
Approach to Learning Goal Motives Fear of failure Aim for qualification Intrinsic interest Commitment to work
Goals Performance Learning
Learning Strategies Surface Deep Non-directed
37
Learning Outcome
Variables
No learning
Rote learning
Meaningful learning
38
4. Measurement
39
Configuration

• What to measure?
• How to measure?
• Who to sample?

40
What to measure?

• Due to model complexity,
• test a part of the model
• a) Personal data prior to learning
• b) Information of their learning approach
  during learning
• c) Questions on understanding post learning

41
a) Personal Data

• What to collect?
• Distal Variables Personality?
• Demographics?
• Culture?
• Skills Reading skills?
• Studying skills?
• Modelling skills?

42
Examples of Frameworks

• Frederiks Weide (2006) Analysis Skills
• incl. Handle implicit knowledge, grammatical
  analysis, abstract sentence structure, think on
  an abstract level
• Bandara et al. (2007) Content for IS Subjects
• From focus group analytical skills, understand
  the problem, ability to communicate with client
• Lindland et al. (1994) Quality
• Syntactic, Semantic Pragmatic Quality
• Vanderfeesten et al. (2007) Complexity metrics
• incl. Coupling, Cohesion, Modularity

43
Skills

• Syntactic Skills a) Word comprehension
• b) Vocabulary syntax
• II) Identify obstacles
• Semantic Skills c) Spatial ability
• c) BMP modelling discourse
• d) experience/past behavioural skills
• I) studying habits
• II) identify obstacles
• Pragmatic Skills e) Working memory
• e) Integration capacity
• f) Real time self-evaluation
• I) Develop effective strategies
• II) Problem-solving
• IV) Emotional stability

44
b) Learning Approach

• Methods available
• Yet, is this useful?

45
c) Understanding

• Methods available
• Only cognitive measurement?
• Indicators Recall Transfer
• Question answered correctly
• Problem solving
• Time?
• Incorporation of affective constructs

Cognitive Performance Behavioural Performance Affective Performance
46
How to measure?

• Qualitative vs. Quantitative

Ability to measure all skills Control over context Scope No additional layer of interpretivism

47
Fixed vs. Loose

• Should respondents be provided with a goal?
• Should meaningful learning be the outcome to aim
  for?

48
Tests, tests, tests

• some examples
• Spatial ability test
• Working memory
• integration test
• Reading ability
• Understanding of story
• Problem solving
• Cloze Recall test
• Cognitive Coupling

49
Who to sample?

• Controlling the Learning Context
• vs.
• homogeneity of population
• Assume group differences (e.g. expert/novice)
• vs.
• differences by measurement

50
Encore un fois
51
5. Questions?

• ?