Measuring User Satisfaction through Experiments

1
Measuring User Satisfactionthrough Experiments

• B. de Vries

2
Domotica
Wildgroei Toekomstdroom Losse deelmarkten
Computable 5 November 2004
3
Innovatie

• Juiste product ?
• Juiste doelgroep ?
• Juiste distributie ?
• Juiste tijd ?
• Juiste marketing ?

4
Experiments

• Observational
• Experimental

5
Characteristics

• Empirical Gather evidence through observation
  and measurement that can be replicated by others
• Measurement
• Replicability
• Objectivity

6
Variables

• Independent Cause
• Dependent Effect

7
Scientific research

• Validity Are you measuring what you claim to
  measure
• ( measuring the right thing)
• Reliability The ability to produce the same
  results under the same condition
• (Measuring thing right)
• Error The difference between our measurements
  and the value of the construct we are measuring

8
Validity

• Internal validity problems
• Group threats, regression to the mean, time
  threats, history, maturation, instrumental
  change, differential mortality, reactive and
  experimenter effects
• External validity problem
• Over-use pf special participants group,
  restricted number of participants

9
Between groups
Treatment (experimental gp.)
Measurement
Random allocation
No Treatment (control gp.)
Measurement
10
Measuring User satisfaction

• Virtual Reality
• Bayesian Belief Networks

11
Desk-Cave
12
(No Transcript)
13
Desk-Cave
14
(No Transcript)
15
Set-UP

• 2 synchronized PCs with dual monitor output
• 4 LCD Projectors

16
Features

• 1 1 Scale
• 3DS import
• Immersion

17
Bayes Theorem
From Evaluation and Decision (7M834)
18
Bayesian Belief Network
19
Node Probability Table
20
NPTs
21
Analyzing a BBN
Marginal probability
p(Norman late) p(Norman late train strike)
p(train strike) p(Norman late no train
strike) p(no train strike) (0.8 0.1) (0.1
0.9) 0.17
Conditional probability
p(TrainNorman late) (p(Norman latetrain
strike) p(train strike))/ P(Norman late)
(0.8 0.1) / 0.17 0.47
22
Measuring User Satisfaction Using Virtual Reality
and Bayesian Belief Networks.
Maciej A. Orzechowski

• 01.11.2004

23
Motivations, aims

• Current techniques for measuring user
  preferences (CA, MM, interview) are artificial,
  lengthy or expensive.
• For good results we need to get the respondents
  more involved in the measurement.
• Can Virtual Reality (VR) improve the quality of
  measuring preferences more involved and higher
  reliability?
• The aim of this project was to develop and test
  an interactive VR tool for measuring housing
  preferences.

24
Solution strategy

• Interactive Virtual Environment (iVE).
• Modification of a design.
• Translation of applied modifications into
  choices.
• Entering this information into a Bayesian Belief
  Network.
• Checking the consistency (if necessary prompting
  for verification).
• Learning (updating) the preference network.

25
VR System

• MuseV3 a Virtual Reality application with
  functionality of a simple CAD system.
• Two categories of modifications
• Structural modifications (change layout).
• Textural modifications (change visual
  impression).

26
Structural Modifications

• Change of internal and external dwellings
  layout.
• The most important for estimating user
  preferences.
• Include following commands create/resize space
  insert openings.
• Direct impact on overall costs of the dwelling.

27
MuseV3 in Desktop CAVE
28
Bayesian Belief Network

• Non-obtrusive interactive method to collect
  housing preferences.
• Potential advantages
• Interaction with the model during the time of
  preferences estimation.
• Incremental learning.
• Possibility to assess
• where the knowledge about preferences is most
  uncertain.
• consistency of measurements.

29
Bayesian Belief Network cont.
A Bayesian Belief Network (BBN) captures believed
relations (which may be uncertain, stochastic, or
imprecise) between variables, which are relevant
to some problem.
Price (?)
30
CPT calculation
31
Learning process
32
Convergence
33
Utility Convergence
34
Experiment

• 1600 letters -gt 100 answers -gt 64 respondents.
• Respondents were people searching for a house or
  who just bought one.
• 4 kinds of 2 types of tasks (2 traditional, 2
  based on MuseV3)
• CA Verbal Description Only (VDO) Multimedia
  Presentation (MM).
• BBN Preset Options (PO) Free Modification (FM).
• Each respondent completed both types of tasks.

35
Experiment Types cont.
36
Analysis

• Estimation of separate models for each task.
• Test for order effect.
• Comparison of CA and BBN models in terms of
• Internal validity.
• Predictive validity.
• Questionnaire.

37
Internal Validity CA vs. BBN

• Roughly similar between CA and BBN.
• Estimated utilities are not identical but
  strongly correlated.
• The difference in scale suggests that the BBN
  has a lower error variance.
• The task order effect suggests VR pre-learning
  improves the validity.

38
External Validity CA vs. BBN

• Models based on BBN on average predicted
  correctly 69 of the choices.
• Models based on CA on average predicted
  correctly 56 of the choices.
• High increase in CA model performance when task
  is preceded by VR task
• for VDO from 56 to 62.
• for MM from 32 to 73.

39
Observed-Predicted
40
Conclusions

• The results support the potential of the
  suggested approach.
• The results suggests higher involvement of
  respondents.
• This approach is non-obtrusive compared to
  different preference measurement techniques.
• The system (tool) can be used to
• To assist individual users in creating their own
  design.
• To derive market potential of housing designs at
  aggregate level.

41
Domotica applications

• Alarmering inbraak, zorg, brand
• Autom. Verlichting
• Autom. Zonwering